Machine learning — software that finds patterns in data instead of following hand-written rules — has conquered image recognition, translation, and protein folding. So surely it can predict stocks? The uncomfortable truth: markets are close to the worst possible environment for it.
The first reason is signal-to-noise ratio, the proportion of meaningful pattern to random static in your data. A photo of a cat is nearly all signal: the cat is really there, and a million photos of cats all contain cats. Daily stock returns are nearly all noise. The predictable component of tomorrow's return, if it exists at all, is a whisper under a roar — competent researchers celebrate models whose predictions correlate with outcomes at a few percent. In most ML fields, a model that weak would be thrown away. In markets, it might be a career.
When signal is that faint, a flexible model does something dangerous: it learns the noise. And noise, by definition, doesn't repeat.
The second problem is non-stationarity: the statistical rules of the game change over time. A cat in 2010 photos is still a cat in 2026. A pattern in 2010 markets — say, small stocks outperforming in January — may be gone by 2026, and for a deep reason.
Markets are made of people and algorithms competing to exploit exactly the patterns you're hunting. When a genuine edge is discovered, capital piles onto it, and the pattern shrinks or vanishes. The market notices being predicted — it's the only data-generating process that actively fights back. This is why anomalies tend to fade after publication; studies tracking dozens of published return patterns found profitability dropped substantially once the papers came out.
Regimes shift for innocent reasons too: rate environments change, market structure changes, the set of dominant traders changes. A model trained on 2010 to 2020 learned the physics of a low-rate world. In 2022 that world ended, abruptly. Any ML approach in markets has to assume its training data describes a country the model may never visit again.
Overfitting is when a model fits its training data so closely that it captures the noise along with the signal, and then fails on new data. Every ML field fights it. Finance loses to it more often, because faint signal plus shifting regimes is the perfect breeding ground — and because the standard test for it can be quietly corrupted.
The corruption is called data leakage: information from the future or from outside the tradeable moment sneaking into training. The classic forms are worth memorizing. Lookahead bias: using a quarterly earnings number on the day the quarter ended, weeks before it was actually published. Survivorship bias: training only on stocks that exist today, silently excluding the ones that went bankrupt — which teaches the model that stocks rarely die. And the subtlest: testing many ideas on the same data. Try 200 strategies on one historical dataset and a few will look brilliant by pure chance; report only the winner and you've laundered luck into a track record. Statisticians call this multiple testing, and it's how most spectacular backtests are manufactured, often by honest people.
Given all that, does anyone serious use machine learning in markets? Yes — heavily. But mostly not as a crystal ball spitting out buy and sell calls. The real uses are quieter.
Feature ranking. A feature is any measurable input — a valuation ratio, a momentum score, an inventory figure. ML methods are genuinely good at sifting hundreds of candidate features and ranking which ones carry information, and at combining weak signals that are individually near-useless.
Text, at scale. Natural language processing — software that extracts meaning from text — reads earnings calls, filings, and news far faster than humans, scoring tone or flagging changed language between reports. Nobody claims the score predicts much alone; it's one more weak input.
Execution and risk. Some of the most reliable wins are unglamorous: models that forecast short-term trading costs so a fund can trade more cheaply, or that estimate volatility and correlation for risk management. Notice the pattern across all of these: professionals aim ML at problems with decent signal-to-noise — cost prediction, text classification — and stay humble about the one problem outsiders assume it solves, which is predicting returns.
Now the honest part. The internet is full of tutorials where a neural network backtests to riches on historical prices. Nearly all of them are curve-fits — models sculpted to the specific wiggles of one dataset.
The tells are consistent. Trained and tested on overlapping or adjacent data. No trading costs, or token ones. Performance that collapses if you shift the start date a year. A suspiciously smooth equity curve — real strategies with real edges still have ugly stretches. And above all, results reported after many silent retries: every tweak of a parameter that was kept because the backtest improved is a small act of overfitting, and dozens of tweaks compound into a model that has memorized one decade.
The arithmetic behind this is sobering. The more configurations you try, the higher the best backtest will look by chance alone, so a strategy's reported performance must be discounted by how hard you searched — and most tutorials searched hard. If a backtest looks too good, the correct prior isn't that you've found a money machine. It's that the machine found your noise. Assume any edge that survives scrutiny is smaller than measured, and shrinks further once real money — yours and competitors' — starts trading it.
None of this means don't experiment. It means experiment like a skeptic. A few guardrails turn ML tinkering from self-deception into education.
Hold out data you never touch. Split history, lock the recent years in a vault, and evaluate there exactly once, after all tuning is finished. Walk-forward testing is stricter still: repeatedly train on the past, test on the next slice, roll forward — mimicking how you'd actually have traded.
Beat a dumb baseline first. Before crediting a neural network, check whether it outperforms buying and holding, or a one-line trend rule, after costs. Most don't, and learning that is worth more than a fake win.
Prefer boring models. Fewer parameters, fewer features, and features with an economic story — a reason grounded in how investors behave or how cash flows work — fail less catastrophically than black boxes fed raw prices.
And size any live experiment as tuition, not as a portfolio. The realistic prize of retail ML isn't beating professional quants at their own game; it's a sharply better understanding of markets, statistics, and your own capacity for self-deception.
Previous: A Short History of Crashes (and Recoveries) · Next: Market Microstructure: How Trades Actually Happen · Financial glossary
← Back to all investing concepts