← Back to Flight Club

Scenario Replay Is Not a Backtest

Backtesting is the most abused word in finance: it sounds rigorous, implies discipline, and almost always lies in practice.

The standard backtest takes a strategy, runs it against historical data, and declares victory when the equity curve slopes upward. The problem is that strategy was built with the benefit of that historical data. Rules were tuned, parameters were adjusted, and the universe was selected with full knowledge of what happened next. It’s not a test so much as a fitting exercise dressed up as science.

We replaced backtesting with Scenario Replay, and the difference is fundamental. Backtesting asks: if I had run this strategy in the past, how would it have performed? Scenario Replay asks if a current portfolio, exactly as it stands today, is dropped into a past market environment, what would the experience look like?

That distinction matters because it eliminates the most dangerous form of hindsight bias. If holdings were selected based on today’s signals, today’s valuations, today’s model output, then running Scenario Replay through the 2008 financial crisis or the COVID crash or the 2022 rate shock tests the current book against condition patterns that show us how the current portfolio should handle these events. It is a more honest stress test.

There is another problem with backtesting that rarely gets discussed: the businesses themselves have changed. A backtest that claims to show how your portfolio would have performed in 2008 is pretending that the companies you own today existed in their current form 18 years ago. They did not. Apple in 2008 was a consumer electronics company that had just launched the iPhone. Today it is a services and ecosystem giant with a fundamentally different risk profile. Banks carried different balance sheets, energy companies operated under different regulatory frameworks, and some of the companies in a current portfolio simply did not exist yet.

Backtesting ignores all of this. It treats a ticker symbol as a fixed entity across time, as if the business behind it, its strategy, its management, its competitive landscape, and its capital structure were unchanged. Scenario Replay does not make that mistake. It takes the companies as they are today and asks how they would respond to a historical pattern of market stress. It is not claiming these businesses would have been there in 2008. It is asking what 2008’s market dynamics would do to today’s actual holdings.

We use our proprietary PB/AO™ framework to make Scenario Replay more granular than a simple drawdown simulation. Most stress tests take the portfolio’s beta, multiply it by the market return each day, and produce an equity curve. That tells you almost nothing useful because it assumes the portfolio behaves the same way every day regardless of market conditions. We know that is wrong. On Pure Beta™ days, when correlations spike and the market moves as one block, portfolio returns are almost entirely determined by beta exposure. On Alpha Opportunity™ days, individual stock characteristics dominate and alpha can express itself.

Our Scenario Replay separates these dynamics so that each historical trading day in a scenario is classified and applies the appropriate return model. On PB days, the replay applies the portfolio’s beta-adjusted market return with minimal alpha assumption. On AO days, it layers in the portfolio’s demonstrated alpha generation range based on current conditions. The result is a projection that respects the structural difference between days when stock-picking matters and days when it does not.

We propose that Scenario Replay gives us more realistic drawdown estimates. A pure beta simulation overstates losses for low-beta portfolios during crashes because it ignores the defensive characteristics that show up precisely during stress. It also overstates gains during recoveries for the same reason. By conditioning on regime, the replay captures the actual mechanics of how a diversified stock-picking portfolio behaves across different market environments.

We maintain a library of historical scenarios that cover the major market events of the past 25 years. The 2008 financial crisis. The 2010 flash crash. The 2015 China devaluation. The 2018 fourth-quarter selloff. The 2020 COVID crash and recovery. The 2022 rate shock. Each scenario preserves the full daily sequence of returns, regime classifications, and VIX levels so the replay captures not just the magnitude of the event but its rhythm and duration.

The rhythm matters. A 20% drawdown that happens in three days feels and behaves differently than one that grinds out over three months. The first is almost certainly a fear event. The second is likely structural. Our Scenario Replay preserves that temporal signature because it walks through the historical sequence day by day rather than applying a single shock.

When a client or an allocator asks how the portfolio would handle a repeat of 2008, we do not hand them a backtest built on a strategy that did not exist in 2008, populated by companies that no longer resemble their former selves. We hand them a replay of today’s actual holdings, as those businesses exist right now, through the actual daily sequence of that crisis, with regime-conditional modeling that reflects what we know about how correlations and alpha generation behave under stress.

There are honest limitations. Scenario Replay assumes the portfolio remains static through the event. In reality, we would be making adjustments in a structural event. The replay also uses the portfolio’s recent alpha generation range as an input, which may not persist through extreme conditions. We state both of these assumptions clearly in every replay output because the goal is informed judgment, not false precision.

The broader point is philosophical. Backtesting tells you what you want to hear. It confirms that the strategy you already chose would have worked in the past you already observed. Scenario Replay tells you something harder: here is what your current portfolio would experience if the world turned hostile in a specific, documented way. One flatters while the other informs.

We built this tool because we wanted to answer the question honestly, and we think all investors deserve the same courtesy.