Our backtesting and evaluation framework
See how economic news becomes a forecast and a portfolio, then how we test whether a change improves the result.
Part 1 · How the model works
First understand the investment process. Then see how our framework tests changes to it.
Subtract the consensus forecast from each actual release. Divide by the spread of earlier forecast misses so different releases can be compared.
Surprise = (actual − consensus) ÷ earlier miss standard deviationAverage the available standardized releases into Growth and Inflation signals. Add financial conditions, geopolitical risk and eligible policy information; the reference model also includes specified interactions.
A positive surprise means above consensus; it does not automatically mean “buy stocks.”The reference model uses a separate Ridge regression for each asset’s next-month return. Ridge limits coefficient size. Training-only input limits and policy-history checks address unstable inputs.
The method experiment compares Ridge with simpler forecasts using its own shared input set.Combine the model’s predictions with reference returns implied by the 45 / 35 / 20 mix and historical asset risk. Earlier completed forecast errors determine view uncertainty; greater uncertainty reduces a forecast’s influence.
This produces adjusted return estimates. Portfolio weights are chosen in the next step.Choose weights using adjusted returns, how the assets move together, risk and a penalty for changing holdings. Apply allocation limits, then account for trading costs.
SPY: U.S. stocks · IEF: U.S. Treasuries · LQD: investment-grade corporate bondsIf the actual release is 100, consensus is 90 and the earlier miss standard deviation is 5, the standardized surprise is +2: (100 − 90) ÷ 5.
Illustrative numbers, not an observed release. This measures surprise relative to expectations, not how good the economy is.
The underlying investment process is the starting point. Our contribution is a framework for testing added signals and forecasting methods under controlled conditions, with longer forecast horizons planned.
Inputs and forecasting methods can change within an experiment. The portfolio construction rules stay shared within that comparison.
Asset risk (Σ) describes how SPY, IEF and LQD moved together over the earlier 36 months, with at least 24 required. Forecast uncertainty (Ω) uses the variance of each asset’s earlier completed prediction errors, with at least 12 required.
The reference returns are implied by the chosen benchmark mix and historical risk. They are not a separate economic forecast.
The allocation step considers all three assets together. The corrected reference allows SPY 20–70%, IEF 15–60% and LQD 10–40%, with weights adding to 100%, on a one-percentage-point grid.
Holdings drift between trades. Costs are 5 basis points per dollar traded, using the full absolute change in weights. The 45 / 35 / 20 mix is the initially funded reference, not a reset every month.
Follow one saved decision
Corrected reference model · saved development results only · starts with the earliest decision. This explains one model’s mechanics; the signal and method experiments have different specifications.
1 · Information at the decision
These are input values before the model’s training-window processing. Growth and Inflation average standardized release surprises. Index values and surprise scores use different units.
Economic histories include later revisions and assumed availability dates. This is not a complete reconstruction of what was known in real time.
2 · Learn from earlier outcomes
Missing values, input limits, scaling and Ridge tuning use earlier training data. Unsupported policy features contribute zero.
The inputs alone do not explain why an individual forecast moved; that requires examining the fitted coefficients and processed features.
3 · Predict, then account for uncertainty
All returns below refer to the same upcoming holding period. The bars share one scale across assets, stages and selected dates.
| Asset | Raw Ridge forecast | After Black–Litterman | Earlier error spread |
|---|
Earlier error spread is the square root of the saved error variance used in Ω, expressed in percentage points; it is not a confidence interval or RMSE. Black–Litterman combines the reference, asset covariance and forecast uncertainty. It does not simply clip forecasts or shrink each one toward zero.
4 · Choose and trade weights
“Before trade” includes the previous holdings’ drift. Trading costs are deducted at entry; both sides of a switch count toward traded weight.
5 · Observe the later outcome
The realized return was unavailable when the decision was made. Open it after inspecting the forecasts and weights.
One month illustrates the process. It does not establish skill; use the full matched sample to judge performance.
This walkthrough reads saved results. It does not rerun the model or inspect the reserved test. Download the saved research results.
Part 2 · How we test changes
The saved outputs let us inspect where a result came from rather than looking only at the final return.
Build macro surprises and risk inputs using the observations treated as available by the decision. Keep source dates and availability assumptions.
Learn from earlier completed returns. Fit missing-value treatment, input limits, scaling and tuning within each training window.
Black–Litterman combines forecasts with a historical-risk reference. Each model’s uncertainty comes from its own earlier completed forecast errors.
Choose SPY, IEF and LQD weights within fixed limits. Trade after the decision, account for drift and charge the same cost rule.
Compare prediction errors, returns, daily losses, turnover and costs. Check the full sample before focusing on individual episodes.
Controlled experiments
“Same rules” applies within an experiment. The two completed studies have different input sets and slightly different coverage.
| Study | What changes | What stays fixed | Current sample |
|---|---|---|---|
| Added signals | One added input group: claims, credit or curve | Original wider-input Ridge pipeline, dates and portfolio rules | |
| Forecasting methods | Historical mean, OLS, Ridge, Elastic Net; then separate lag/interaction extensions | Shared eligible rows, training procedure and portfolio rules; core methods share inputs | |
| Forecast horizons | Planned: one-, three- and six-month return targets | Target alignment, completed-target rules and trading design must be specified | One month evaluated; longer targets not run |
A return target must finish before it can enter training or an earlier-error estimate. The same timing check applies inside tuning folds.
Use matched observations within each study. A method with a shorter usable history cannot quietly receive an easier comparison period.
Holdings drift with asset returns and are not reset at episode boundaries. Costs use the full weight change from those drifted holdings.
Two levels of evaluation
Simple benchmarks make the added value of macro information easier to assess.
Compare each asset with a forecast of its past average return.
Use the same construction and cost rules within the study.
Rolling windows overlap, and small episodes are not independent experiments. Bootstrap intervals are exploratory and conditional on the saved portfolio paths; they do not refit models or correct for trying multiple variants.
Reliability is part of evaluation
The original model’s +589% SPY prediction exposed a sparse policy input with a tiny training scale. We traced it, compared repairs, and reran a corrected reference model. This is a quality check within the research—not its whole contribution.
The correction uses earlier-data input limits and explicit policy-history and variation checks. The original 23.99% remains a diagnostic comparison. The initial added-signal runs have not yet been repeated with this corrected specification.
Prediction checks
Compare prediction errors first, then portfolio returns. The original model is shown to explain what changed.
| Version | MAE ↓ | RMSE ↓ | OOS R² vs. mean ↑ | Largest |forecast| | |Forecast| >20% |
|---|
R² = 1 − model squared error ÷ expanding-mean squared error, using the same months. Negative means worse than the historical mean. The 20% flag is an inspection rule, never a return cap. Zero flags does not establish good forecasts.
| Version | Decision | Asset | Forecast | Realized | Largest feature contribution |
|---|
Common portfolio dates, with continuous holdings. Partial years are labeled. These small windows are diagnostics.
| Holding year | Months | Selected RMSE | Mean RMSE | OOS R² vs. mean | Largest |forecast| |
|---|
All rows use the same 44 allocated development months and the same BL, trading costs and weight bounds. Each model uses its own prior completed forecast errors for view uncertainty. Portfolio results do not determine which forecast repair is acceptable.
| Version | Net total return | Daily max drawdown | Monthly turnover | Trading costs* | Months at any weight bound |
|---|
*Sum of monthly trading-cost fractions, in basis points, rather than a compounded return drag. At-bound counts refer to rebalance weights. A lower return can accompany a more reliable forecast.
| Asset | Training target mean | All feature contributions | Reconstructed forecast | 6m change coefficient* | 6m change contribution | Actual return |
|---|
*Coefficient is in decimal return per training standard deviation. Multiplying it by −5,454.09 reproduces the policy-change contribution before percentage display rounding.
| Sheet | Source date | Original level | Change from preceding snapshot |
|---|
| Feature month | Policy source date | Target exit | 6m policy level | 6m policy change |
|---|
The corrected reference rules and evaluation criteria are pinned by the locked-version record below. Confirm with the team whether 2023 onward remains unexamined and obtain separate authorization for the reserved-period run. The development commands reject reserved holding months.
Revised macro histories and assumed publication lags remain a separate limitation. A real-time vintage archive would require additional work, including ALFRED vintages where supported.
Updated model · development results
We fixed the model’s unstable predictions, then tested it against a simple approach: predict each asset’s past average return. Both forecasts use the same portfolio rules and trading costs.
| Asset | Model’s average miss | Past-average forecast’s miss |
|---|
“Average miss” ignores whether the prediction was too high or too low. A 5-percentage-point miss means predicting +2% when the actual return is −3%, for example. These numbers use the same 44 portfolio months.
| Asset | Model MAE | Past-average MAE | Model RMSE | Past-average RMSE | OOS R² · portfolio months | OOS R² · all forecast months | All improvement checks passed? |
|---|
MAE is the average miss shown above. RMSE gives more weight to large mistakes. OOS R² compares the model’s squared errors with the past-average forecast: positive is better, negative is worse. We require positive R² and no worse MAE in both sets of months before describing an asset’s forecast as improved. These development comparisons do not establish statistical significance.
| Model / comparison | Total return after costs | Largest fall from a peak | Average traded each month | Trading costs* |
|---|
The past-average forecast uses the same Black–Litterman step and portfolio rules as the model. Fixed mixes and SPY-only provide additional comparisons, with different risk levels. *Costs are the sum of monthly cost rates; 1 basis point is 0.01%. This is not the compounded reduction in returns.
The original model once predicted a monthly SPY return of about +589%. A policy input had very little observed history and barely changed in training, so a new value was magnified during scaling. We now limit unusual inputs using earlier training data and switch off policy inputs until they have enough history and variation. The updated model has no predictions beyond ±20% in this development sample. The old 23.99% portfolio return remains below for comparison; it is not the updated result.
We require enough original observations and enough variation before using a policy input. Filled-in or carried-forward values do not create new observations. If an input fails these checks, it has no effect on the forecast.
The exact rules are 12 distinct source observations, three distinct values when rounded to one-basis-point bins, and a standard deviation of at least one basis point before and after filling missing values and limiting extreme inputs. One basis point is 0.01 percentage points. These fixed safeguards are repeated in every training and tuning window; they are not proof that an input helps predict returns.
| Policy input | Original observations | Distinct values · 1bp bins | Observed standard deviation · pp | After processing · pp | Used? / reason |
|---|
We chose these rules after reviewing the development data, so the results above are not a final test. This version has not been tested on 2023 onward. Before using those years for a final test, the team needs to confirm that they have not already influenced the model’s design. The data also include later revisions, and some release dates are assumed; fixing the predictions does not remove those limits.
Limits and the final test
The development work supports comparisons and diagnostics. It is not an untouched confirmation of a selected strategy.
Some economic values were revised later. Availability lags approximate historical access; they do not reconstruct all real-time vintages.
There are few independent stress episodes. Year and boundary checks help describe sensitivity but do not establish broad robustness.
The original signal study, compact model comparison and corrected reference answer different questions. Their results should not be pooled into one ranking.
The team must confirm whether 2023 onward remains unexamined before a separately authorized final test. Longer forecast horizons also remain untested.
Read the experiment
Adjusted SPY, IEF, and LQD prices begin in 2010. Training begins February 2015, requires 36 completed targets, and expands each month. BL waits for 12 earlier forecast errors.
The same Ridge → BL → allocation pipeline is rerun for the original and three added-condition versions. Fixed robustness checks change only preprocessing/shrinkage or the inclusion of policy fields. Claims, credit spread, and yield curve are tested separately, without choosing settings for each episode.
Decide at 8 p.m. New York on the last trading day. Enter the next trading day’s close and hold until the next rebalance close. The displayed holding month names the entry month.
Starting mix 45 / 35 / 20. SPY 20–70%, IEF 15–60%, LQD 10–40%. Risk aversion 2.5, BL τ = 0.05. Costs are 5 bp per dollar traded from drifted holdings.
The research baseline is a fresh run, using longer price history, exact next-trading-close execution, completed targets only, and imputation fitted separately inside each tuning fold. Daily drawdowns also differ from the original monthly drawdowns. Compare signal experiments with the baseline on this page; the original MVP remains available in its tab.
Claims, ANFCI, and GPR use revised histories with availability lags, not a complete real-time vintage archive. The workbook’s policy paths start in September 2019; earlier missing policy fields use training-only medians, or zero when the entire training column is missing. These are exploratory development results. No post-2022 strategy decisions are evaluated. The January 3, 2023 quote only closes December 2022’s holding interval.
These episode labels are evaluation windows, not information supplied to the model. The COVID market-shock window is not the same as the official recession chronology. Drawdown recovery is reported only within the selected window.
The combined robust-input sensitivity clips inputs to training-only 1st/99th percentiles and uses the fixed Ridge grid 1, 10, 100, 1000. Preprocessing and target-availability checks are repeated in every tuning fold. Separate clipping-only and shrinkage-only experiments isolate those two changes. The policy coverage safeguard disables each policy feature until it has 12 observed training values, inside every fold. This fixed sensitivity threshold does not certify a stable model. There is no cap on predicted returns. Without-policy sensitivity removes five policy-path fields over identical evaluation dates; it cannot reconstruct missing policy history.