A rigorous portfolio-level backtest follows a structured pipeline: universe definition (which stocks are eligible), signal generation (ranking stocks by your criteria), portfolio construction (determining weights and position sizes), execution simulation (modeling transaction costs, slippage, and market impact), and performance evaluation.
Each stage introduces potential biases. The universe must account for survivorship bias — testing only on companies that still exist today quietly deletes every bankruptcy from history. Signals must use point-in-time data, meaning only information actually available on each historical date. Portfolio construction must respect realistic constraints like position limits, sector caps, and turnover limits. Execution must model realistic costs.
Get any one stage wrong and the whole backtest lies to you — usually in the flattering direction.
Key metrics for evaluating a backtested strategy include the Sharpe ratio (risk-adjusted return), maximum drawdown (worst peak-to-trough decline), Calmar ratio (return divided by max drawdown), turnover (how frequently the portfolio changes), and alpha (excess return over a benchmark).
The formulas are simpler than they sound. Sharpe ratio = (portfolio return minus risk-free rate) divided by the volatility of returns. Volatility here means standard deviation — a measure of how much returns bounce around their average. A Sharpe near 1.0 over a long live period is genuinely good; backtests showing 3.0 or more usually signal a mistake, not a miracle.
The Sortino ratio fixes a quirk of Sharpe: Sharpe punishes big up moves the same as big down moves. Sortino ratio = (portfolio return minus risk-free rate) divided by downside deviation, where downside deviation counts only the bad bounces below your target.
No single metric tells the complete story. A strategy with a high Sharpe ratio but extreme drawdowns may be unacceptable. A strategy with modest returns but very low drawdowns may be preferable for risk-averse investors.
Volatility and maximum drawdown both measure risk, but they measure different pain.
Volatility is day-to-day choppiness — the average size of the wiggles. Maximum drawdown is the worst peak-to-trough loss you'd have lived through: buy at the top, watch it fall, and measure the bottom. A strategy can have modest volatility and still carry a horrifying drawdown if its losses cluster, arriving in one long relentless stretch instead of scattered across time.
Drawdown is usually the risk that actually breaks people. Almost nobody abandons a strategy because daily wiggles were 2 percent instead of 1. People abandon strategies 18 months into a 40 percent hole, usually near the bottom. So when you review a backtest, look past the smooth summary statistics and ask: what was the worst stretch, how long did it last, and how many years did recovery take? Then be honest about whether you'd truly have held on — while the drawdown was still deepening and no one could tell you where the bottom was.
The most seductive backtesting mistake is tuning your strategy on the same data you use to grade it. Tweak parameters until history looks great, and you've built a strategy fitted to the past — a description, not a prediction.
Walk-forward analysis is the antidote. Split history into windows. Tune your parameters on the first window — say, 2000 through 2004. Then test on the next slice, 2005, which the tuning never saw. Slide everything forward and repeat: tune on 2001 through 2005, test on 2006, and so on. Stitch the test slices together and you get a track record built entirely from decisions made without knowledge of the future.
Walk-forward results almost always look worse than the tuned backtest. That's the point — they're closer to what live trading would have felt like. And watch how the best parameters move between windows. If each window picks wildly different settings, your strategy has no stable edge; it's just chasing whatever noise each period happened to contain.
A single backtest shows one path through history — the one that happened. But that exact sequence of returns will never repeat. Monte Carlo resampling asks the deeper question: how much of the result was strategy, and how much was the luck of the ordering?
The method is simple in spirit. Take your backtest's trade or monthly returns, shuffle their order (or draw them randomly with replacement), and rebuild the equity curve. Do this ten thousand times and you get a whole distribution of alternate histories: the lucky versions, the unlucky ones, and everything between.
The outputs worth studying are the ugly tails. Maybe your actual backtest showed a 25 percent max drawdown, but in a fifth of the shuffled histories the same trades produced 40 percent or worse — merely because losses happened to bunch up. That's not a different strategy. That's your strategy on an unlucky day, and the future is under no obligation to deal you the pleasant ordering. Size your positions for the distribution, not the single path.
Frictionless backtests are fiction. Every real trade pays commissions, crosses a bid-ask spread (the gap between the price buyers offer and sellers ask), suffers slippage (the difference between the price you expected and the price you got), and — for larger orders — creates market impact by moving the price against itself.
A workable starting model: charge every trade the commission plus half the typical spread, then add a slippage allowance that grows for smaller, less-traded stocks. Small-cap and micro-cap backtests are where paper edges most often go to die, because the spreads are wide, volume is thin, and the backtest assumed liquidity that wasn't there.
Costs scale with turnover, so a strategy trading weekly might pay twenty times the annual friction of one rebalancing quarterly. A useful stress test: double your cost assumptions and rerun. A strategy that survives doubled costs has a margin of safety. One that dies was probably never alive — its edge was smaller than its friction, and the frictionless backtest just hid the funeral.
Previous: Quantitative Strategies · Next: Commitment of Traders (COT) · Financial glossary
← Back to all investing concepts