Bear Miss Rate: Measuring Optimistic Forecast Errors on Loss Days

Why optimism on a falling day matters

Picture someone holding a long position. When a model underestimates a drop, the loss on the position exceeds what the forecast led them to expect. If forecasts feed into position size, hedges, liquidity reserves or risk limits, a rosy estimate leaves extra capital exposed and less protection in place. The realized loss comes from the market and not from the forecast, and nothing here computes the profit or loss of a trading strategy. The direction of the error still counts, because for a long holder, understating a fall can do more harm than an equally large pessimistic mistake. Picking a forecasting model therefore means understanding how its errors behave on the days an asset loses value, and not only what they average.

A study of five UK-listed energy securities shows the point. A median model narrowly beat a zero-return benchmark on average accuracy. A downside-oriented HuPin model scored worse on average, yet made far fewer optimistic misses whenever returns were negative. Both outcomes are real, and they answer separate questions.

What the usual scores leave out

RMSE (root mean squared error) and out-of-sample R² summarize how far off forecasts are overall. Neither says how often a forecast understated the size of a loss that actually happened, and two forecasts can score alike in aggregate while differing sharply there. On the common panel used below, R² is a monotone transformation of RMSE, so together they give one central squared-error ranking and not two independent views.

This reporting gap motivated the Bear Miss Rate (BMR) approach, which came out of an MSc dissertation. It bundles three related diagnostics: any optimistic error on a loss day, optimistic errors beyond a practical materiality threshold, and missed severe-return events. The idea is to place a clear downside view alongside ordinary accuracy reporting and not to replace it.

A worked example

Suppose a security logs a daily log return of -2% and a model forecast -0.5%. The sign is right, yet the loss is badly understated, by an optimistic gap of 1.5 percentage points.

A directional hit-rate could score that forecast as a win because both values are negative. RMSE does include the error, but averages it in with all the others. Neither answers the sharper question of how often a forecast ran too optimistic when the asset fell, and how often the optimism was large enough to matter.

BMR brings that conditional question into view. For a long position, "optimistic" means a predicted return above the realized one. So the metric concerns understating the size of a loss, including cases where the forecast gets the negative sign right.

The three diagnostics

Let the realized daily log return be yₜ and its point forecast ŷₜ.

Basic BMR

The simplest version measures how often the forecast is too high on days with a negative realized return:

The comparison is strict, so a forecast equal to the realized return isn't a miss. If no negative-return observations exist, the rate is undefined and not zero.

BMR with a tolerance

Some forecast gaps are too small to matter in practice. BMR_tol counts only optimistic gaps that reach a chosen tolerance ε:

One shared value, ε = 0.01, is applied to every model and security. In daily log-return units that's about one percentage point. It was picked as a round, easy-to-read materiality level for this illustration and isn't presented as a universal optimum. The denominator is the same bearish one BMR uses, and the rate is undefined if that denominator is empty.

BMR for severe losses

BMR_Z targets severe realized losses. For each security, a outlier-resistant cutoff is built from training returns only:

MAD stands for the raw median absolute deviation. The candidate cutoff is kept only if the outlier-resistant scale comes out finite and above zero, and the training set has at least five returns at or below that candidate. When either condition fails, the procedure uses the training quantile Q(0.0228) with linear interpolation. Each security gets its own cutoff, but models don't, so the Martingale, median Quantile and HuPin forecasts are all judged against one event definition.

The miss boundary is strict again: a forecast landing exactly on the cutoff counts as flagging the severe event, not as a miss. When no realized test observation lands in the defined tail, the conditional rate is undefined and not zero.

Once the cutoff is set, the conditional miss ratio equals, algebraically, the false-negative rate (one minus recall) of the event classifier the cutoff defines. That identity covers only the final ratio. The complete BMR_Z diagnostic also includes the training-only cutoff, the sparse-tail fallback and the place it holds in the integrated BMR design.

The experiment

The empirical test uses five securities: BSIF, HTG, SSE, TLW and UKW. Each model is scored on an identical set of 315 dates per security, running 22 April 2024 to 18 July 2025. That yields 1,575 security-days on the common panel, 785 of them with a negative realized return.

The three models play deliberately different roles:

• Martingale: a theoretical anchor that always forecasts zero return.
• Median Quantile Regression: a linear conditional-median baseline fitted at quantile level 0.5.
• HuPin LSTM: a downside-oriented network trained on a blend of Huber and pinball losses, with a selected pinball parameter of 0.05.

These three families were kept for the illustration after looking at their common-panel results. Their pooled RMSE values lie within 1.133% of the lowest, and their out-of-sample R² values all hover near zero, while bear-miss rates separate their downside behavior much more clearly. That's a choice made for presentation, with no claim that central accuracy is statistically equivalent.

Two caveats apply. The additive HuPin objective doesn't by itself make its output a calibrated 5% conditional quantile. And the median Quantile and asymmetric HuPin objectives needn't target the same functional. The comparison below is thus an empirical demonstration of what BMR diagnostics reveal, and not a finished test of a same-functional model-selection rule.

Every retained model uses the same evaluation keys, the same materiality threshold and tail cutoffs derived from training data. A numerical verifier recomputed all five metrics straight from the common-panel output.

The first figure shows RMSE and out-of-sample R² on the 1,575-observation panel.

Results on average accuracy

The median Quantile model posts the lowest RMSE and a slightly positive test R², with an RMSE 0.0603% under the zero-return benchmark's. HuPin's RMSE is 1.133% over the median model's, and its test R² turns negative.

Results on downside misses

Read through BMR, the same forecasts look different. Across the 785 negative-return days, the Martingale misses every one, the median model misses 765 and HuPin 627. With the shared one-percentage-point threshold applied, the counts drop to 394, 365 and 270. In this sample HuPin's BMR_tol therefore lands 12.10 percentage points under the median model's.

Figure two lays out BMR, BMR_tol and BMR_Z side by side using shared definitions and denominators.

A 100% BMR for the Martingale is built in: it always predicts zero, so any negative realized return sits below it. Its 50.2% BMR_tol is more informative, since it says slightly over half the bearish days fell at least 0.01 under the zero forecast in log-return terms.

BMR_Z adds a negative result that's just as useful. The common test panel holds 115 realized tail observations, and under the stated threshold rule all three models miss every one. So the diagnostic can't rank the models here. What it records is a limitation they share: on the realized tail days, no point forecast reached the training-defined severe-loss cutoff.

What this does and doesn't show

HuPin doesn't "win". In this sample it makes fewer optimistic loss-day errors while scoring worse on aggregate accuracy, and the BMR family makes that tradeoff visible. No acceptable-accuracy margin was declared in advance, and the median and HuPin objectives may aim at different functionals, so the table can't establish a preferred model. No inferential equivalence test was run.

BMR is intended as a supplement to central accuracy. A future selection protocol might begin by keeping only forecasts aimed at the same target functional, then rate them with a central score that's consistent for that functional and a prespecified acceptable-accuracy margin. Among the survivors, a chosen BMR measure could act as a tie-breaker favoring fewer optimistic loss-day errors.

That sequence is important. Pushing any forecast lower can cut its optimistic miss rates without supplying any new timing information. It does change error behavior in a genuine way and might fit an investor who cares about the downside, yet its price has to remain visible through central accuracy, target-appropriate bias or calibration, and a clear limit on excess conservatism. A lower BMR by itself isn't evidence of extra predictive information. A decision-focused study would additionally have to put a cost on pointless defensive moves and gains given up.

The diagnostics carry their own limits too. Because they condition on realized losses, they're silent about false-positive downside warnings, and BMR_Z would need a false-positive measure or an operating curve before it could support a full event-detection assessment. No rate here alone demonstrates trading profit, economic value, general model superiority or statistical significance.

Where this leaves things

Average errors stay essential, since they show how far forecasts typically land from outcomes. BMR adds a second layer that asks, when losses happened, whether the forecast understated them, whether the gap was material and whether the forecast missed a defined with outlier-resistant statistics severe-loss state.

The five-security study illustrates the benefit. The model with the best overall accuracy isn't the one making the fewest optimistic loss-day misses, and the severe-event diagnostic reveals a failure all three have in common. Those are real findings about how forecasts behave, even though they stop short of settling a model choice or an investment decision.

Each mathematical component links to familiar statistical objects. What's new is limited: the Bear Miss Rate naming and problem framing, the combined three-diagnostic design, its operational specification and the empirical application.

That leads straight to the follow-up question. For forecasts that target the same functional and meet a prespecified central-accuracy requirement, can BMR work as a dependable secondary criterion that reduces material optimistic loss-day errors while honoring an explicit cap on conservatism?

Answering it requires fresh data, a predeclared comparison design, dependence-aware uncertainty, controls for conservative shifts and comparisons with target-consistent alternatives. What exists so far is the BMR formulation, the integrated three-part design, a reproducible implementation and empirical evidence motivating that next test.