Accuracy
How the what-if engine's predictions held up against what actually happened, quarter by quarter, out of sample
What the backtest shows
- Deal counts: direction right 69% of the time one quarter after a shock, against 50% for a coin flip.
- Valuations: right 71% of the time; most of that skill is price moves persisting rather than reversing.
- Deal values are close to a coin flip (56% right): one megadeal can make or break a sector's quarter.
- Feeding in the last 2 quarters' moves, whose effects are still rippling, lifted direction from 61% to 64%.
Predicted vs what happened
IPO counts, 1 quarter on · average over the five sectors · % versus trendIPO counts, 1 quarter onDirection right 68%Error vs no change −22%150 forecasts
Direction right, by quarter ahead
Share of predicted moves with the right sign · bar: skill against no change| Feature | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| Valuations | —input | |||
| Deal counts | ||||
| Deal values | ||||
| IPOs |
50% is a coin flip. Skill bars: 0 = no better than predicting no change, full width = 0.4 (a 40% smaller squared error).
What tuning changed
153 setups compared on the same 31 quarters- estimated (priors ×0.25 links, ×1 spillovers, ×0.5 macro), 3 rounds, fade 1/1/0.85/0.7, plus the last 2 quarters' shocks0.163
- estimated (priors ×0.25 links, ×0.5 spillovers, ×0.5 macro), 3 rounds, fade 1/1/0.85/0.55, plus the last 2 quarters' shocks0.162
- estimated (priors ×0.25), 3 rounds, fade 1/1/1/0.7, plus the last 2 quarters' shocks0.161
- estimated (priors ×0.25), 2 rounds, fade 1/1/1/0.7, plus the last 2 quarters' shocks0.160
- estimated (priors ×0.5), 3 rounds, fade 1/1/0.85/0.55, plus the last 2 quarters' shocks0.160
- estimated (priors ×0.25 links, ×0.5 spillovers, ×1 macro), 3 rounds, fade 1/1/0.85/0.7, plus the last 2 quarters' shocks0.160
How the backtest works
Out of sample. At each of 31 quarters from 2019Q1 to 2026Q3, the links are re-estimated using only the data available before that quarter. The engine then gets that quarter's real inputs, the changes in Fed funds, the 10-year yield, the Baa spread and the VIX plus each sector's ETF move (less the part it already puts down to those macro changes), and predicts the next four quarters. It also gets the same inputs for the 2 quarters before: their effects are still travelling through the lagged links at the forecast date, so the part still to come counts. That lifted direction from 61% to 64%. Deal and IPO numbers are never inputs.
Scoring. Deal and IPO counts are compared with the average of the four quarters before (plus trend), because a single quarter is noisy, and the calendar pattern is taken out (deals dip in Q1, IPOs bunch in Q2), measured only from earlier quarters. Valuations are compared with the quarter before. Direction is the share of predicted moves of at least 0.5 points that had the right sign. Skill is 1 minus the squared error divided by that of predicting no change from trend: above 0 beats it.
Tuning. 153 setups were compared: hand-set or estimated links (with the priors at a quarter to double width), one to three rounds of effects, and a fade per feature. The best, estimated (priors ×0.25 links, ×1 spillovers, ×0.5 macro), 3 rounds, fade 1/1/0.85/0.7, plus the last 2 quarters' shocks, now runs the Scenarios tab. Choosing settings on the same quarters they are scored on flatters them a little, so the two halves of the period are shown separately.
Tried and rejected. Magnitude calibration (it fitted the 2020 quarters and failed cross-validation), quarter-of-year controls, megadeal-capped deal values, stronger clipping, six extra sector spillovers the data suggested, end-of-quarter macro changes, leaving out the pandemic quarters, feeding past deal and IPO surprises, more than two quarters of history, sector-specific deal links, weighting recent years, capping big market moves and averaging several link sets. None beat the settings above out of sample; the log is in research/experiments.md.
Limits. Deal figures in weak coverage years (2010–2011, 2020 and a few sector-years) are left out of the estimation and can't be scored. The links learn from history back to 2010; the scoring yardstick (trends and the calendar pattern) is measured from 2015. Quarterly deal values hinge on single megadeals and stay close to a coin flip. Re-run with npm run backtest; the full report is in research/backtest.md.