Building an Economic Stress Early-Warning System, What the Accuracy Score Did Not Tell Me

June 12, 2026

A walkthrough of how I approached an imbalanced economic-risk prediction problem.

Hey guys!

It has been a while since I last wrote here.

My previous blog post was about how I approach exploratory data analysis: the questions I ask, the checks I perform, and the mistakes that taught me to slow down before modeling.

Since then, I have graduated with my M.S. in Data Science from Pace University, completed several new projects, and spent much more time thinking about a question that sounds simple but is actually very important:

How do we know whether a machine learning model is genuinely useful?

This question became central while I was working on a project I called the - Global Economic Stress Early-Warning System.

The goal was to build a model capable of identifying periods of elevated macroeconomic risk using historical economic indicators.

At first, the project appeared to be a standard binary-classification problem:

But very quickly, I realized that the difficult part was not training a classifier.

The difficult part was deciding what good performance should actually mean.

Let’s get into it.

1. The Problem: Rare Events Are the Events We Care About

Economic crises are important partly because they are unusual.

Most historical periods are not classified as severe crises. That means the dataset naturally contained far more examples of normal conditions than high-stress conditions.

This created a severe class-imbalance problem.

Imagine a simplified dataset with:

A model that predicts normal every single time would achieve 95% accuracy.

That sounds excellent.

But it would detect exactly zero crises.

For an early-warning system, that model would be completely useless.

This was the first major lesson from the project:

A metric can be mathematically correct and still be practically misleading.

Accuracy answers the question:

How often was the model correct overall?

But the problem I cared about was closer to:

How many high-risk periods did the model successfully identify before they were missed?

Those are not the same question.

2. Starting With the Data

The project used historical macroeconomic indicators associated with financial and economic stress.

Before building any model, I spent time examining:

I also engineered features that could represent underlying stress more clearly.

Two examples were:

Yield Curve Inversion Tracking

A yield curve inversion occurs when shorter-term interest rates exceed longer-term rates.

It is frequently discussed as a warning signal of economic slowdown, but using it in a model requires more than simply adding one interest-rate column.

I created features that captured the presence and behavior of yield-curve inversion over time.

Economic Stress Index

I also developed a custom Economic Stress Index, or ESI, to combine signals from multiple macroeconomic variables into a more structured representation of economic pressure.

The purpose was not to replace the original features, but to provide the model with an additional summary of how different indicators were moving together.

Feature engineering became especially important because crisis prediction is not simply about finding one unusually large or small number.

It is often about identifying combinations of changes across several indicators.

3. The First Model Looked Better Than It Was

I began with several common classification models:

These gave me useful baselines.

Some of the early results appeared promising, especially when viewed through accuracy.

But once I examined the confusion matrix and class-specific performance, the picture became less impressive.

A model could correctly classify most normal periods while still missing too many crisis periods.

In a standard classification task, a few false negatives might be acceptable.

In an early-warning system, a false negative means:

The system saw a high-risk period and failed to raise a warning.

That error is much more serious than incorrectly flagging one normal period as potentially risky.

This changed how I evaluated the models.

Instead of focusing mainly on accuracy, I paid much closer attention to:

4. Why Recall Became So Important

Recall measures the proportion of actual positive cases that the model identifies.

In this project, the positive class represented high-stress or crisis conditions.

The basic idea is:

Recall = True Positives / (True Positives + False Negatives)

In simpler terms:

Out of all the actual crisis periods, how many did the model detect?

That made recall especially important.

A higher-recall model may generate more false alarms, but it reduces the risk of completely overlooking real crisis events.

Of course, maximizing recall without any limits is not enough either.

A model that labels everything as a crisis would achieve perfect recall, but it would also produce so many false positives that nobody could trust it.

So the real task became finding a sensible balance:

This is where model evaluation became much more interesting than simply comparing accuracy scores.

5. Handling Class Imbalance

Because crisis examples were limited, I explored ways to improve the representation of the minority class.

Instead of relying only on conventional oversampling, I compared multiple deep generative approaches for tabular data:

The purpose of these models was to generate synthetic high-risk observations that could help the classifier learn patterns associated with rare crisis conditions.

This sounded straightforward in theory.

In practice, synthetic data introduced an entirely new set of questions:

This taught me another important lesson:

Generating more data does not automatically mean generating more information.

Synthetic observations had to be treated as a modeling tool, not as unquestionable ground truth.

I compared the behavior of the generated data, examined distributions, and evaluated whether each approach improved minority-class detection on real validation periods.

6. Why Random Train-Test Splits Were Not Enough

One of the most important decisions in the project was how to validate the models.

For many machine learning problems, we divide the data randomly into training and testing sets.

But macroeconomic data is chronological.

The past occurs before the future.

A model should not be trained using information from a later economic period and then evaluated on an earlier one.

Doing that can create an unrealistic evaluation and, in some cases, introduce temporal leakage.

So I used chronological cross-validation.

The general idea was:

  1. Train on earlier historical periods
  2. Validate on a later period
  3. Expand or move the training window
  4. Evaluate again on the next unseen period

This better approximated how an early-warning system would work in reality.

At any given point, the system would only have access to information available up to that time.

This validation method was more demanding.

Performance sometimes varied across folds because economic conditions do not remain constant forever.

Relationships that appear useful in one period may weaken during another.

But that variation was valuable information.

A slightly lower but realistic score is much more useful than an impressive score produced by an unrealistic evaluation design.

7. The Accuracy–Recall Trade-Off

After testing several approaches, I found that no model dominated every metric.

Some achieved stronger overall accuracy but missed too many high-risk observations.

Others detected more crises but generated more false positives.

The final workflow combined optimized XGBoost modeling with a jittering-based ensemble approach designed to improve robustness and minority-class detection.

The resulting model achieved approximately:

The accuracy was strong.

But the recall figure was more meaningful to me.

A recall of 0.788 meant that the model detected a substantial share of the high-risk observations instead of simply benefiting from the large number of normal periods.

Still, that number should not be interpreted as the problem is solved.

It means the model showed promise under the project’s dataset, feature definitions, and validation framework.

A real economic early-warning system would require much more:

This project was a predictive modeling exercise, not a claim that a machine learning model can perfectly forecast the next economic crisis.

8. What Accuracy Hid From Me

Accuracy hid several important things.

1. It hid which class the model was getting right

A model can achieve high accuracy mainly by performing well on the majority class.

That does not mean it understands the minority class.

2. It hid the cost of false negatives

Missing a crisis period is not equivalent to incorrectly flagging a normal period.

The two errors have different consequences.

3. It hid instability over time

A single aggregate score can conceal large performance differences across validation periods.

4. It hid threshold decisions

Classification probabilities must eventually be converted into class predictions.

The default threshold of 0.5 is convenient, but it is not automatically the best threshold for every problem.

5. It hid whether the model was useful

A model can be statistically impressive and operationally useless.

The final question should always be:

What decision will someone make using this prediction?

9. Mistakes and Challenges Along the Way

Like most of my projects, this one did not follow a perfectly clean path.

Here are some of the challenges that changed how I approached the work.

Mistake 1: Being impressed by accuracy too quickly

At first, seeing accuracy above 90% felt like a major success.

Only after examining class-specific performance did I understand how incomplete that result was.

Mistake 2: Treating imbalance as only a data-volume problem

The minority class did not simply need more rows.

It needed meaningful and realistic representation.

Generating synthetic data was useful only when those observations preserved important economic relationships.

Mistake 3: Comparing models using one metric

A single metric could not capture the complete behavior of the system.

I needed to compare models through multiple views: recall, precision, confusion matrices, stability, and chronological performance.

Mistake 4: Underestimating validation design

The validation strategy was as important as the model architecture.

A powerful model with poor validation can produce very convincing but unreliable results.

Mistake 5: Thinking the most complex model must win

Complexity can help, but it does not guarantee usefulness.

Logistic Regression remained valuable as an interpretable baseline, even when ensemble models achieved stronger predictive performance.

10. What I Would Improve Next

No project is ever truly finished.

If I continued developing this system, I would focus on several areas.

Better Probability Calibration

Instead of evaluating only final class predictions, I would examine whether predicted risk probabilities accurately reflect observed outcomes.

Cost-Sensitive Threshold Selection

The classification threshold should reflect the relative cost of false negatives and false positives.

For a risk-detection system, the default threshold may not be appropriate.

Model Explainability

I would add SHAP-based analysis to better understand:

Regime-Aware Modeling

Economic relationships change.

A future version could explicitly model different economic regimes rather than assuming one stable relationship across the complete dataset.

Expanded External Validation

Testing the system across additional countries, periods, or data sources would provide a stronger picture of generalizability.

Monitoring for Data Drift

A deployed system would need to identify when incoming data no longer resembles the historical data used for training.

11. What This Project Changed About My Approach

Before this project, I understood conceptually that accuracy was not always the best metric.

After this project, I understood it much more practically.

I saw how easy it is to build a model that appears successful when the evaluation metric rewards the easiest behavior.

I also learned that model development is not only about choosing algorithms.

It is about making a chain of decisions:

Every one of those choices affects whether the final system is meaningful.

12. Final Thoughts

The biggest lesson from this project was simple:

The best-looking metric is not always the most important metric.

For an early-warning system, getting most observations right is not enough.

The model must recognize the rare events it was created to detect.

Accuracy told me that the model performed well overall.

Recall told me whether it was actually paying attention when the risk was highest.

And chronological validation told me whether that performance had a chance of holding up beyond one convenient data split.

The project made me more cautious about celebrating model scores and more interested in understanding the behavior behind them.

Now, whenever I evaluate a model, I try to ask:

Because a machine learning model is not useful simply because it is accurate.

It is useful when its evaluation matches the problem it was built to solve.

If you made it this far, here’s a cookie 🍪

~Vibhav