Your amazing backtest results are probably a lie.
I’ve seen it a hundred times. A trader, brilliant and full of hope, shows me a backtest with a Sharpe ratio of 4.5. The equity curve looks like a rocket ship to the moon. They think they’ve found the holy grail.
99% of the time, they’re just fooling themselves.
I learned this the hard way. Early in my career, I spent six months building what I thought was a revolutionary trading algorithm. The backtest was beautiful. I put real money into it. And in two weeks, I lost 30% of my capital. It was a painful, expensive lesson in the difference between a theoretical backtest and real-world results.
Most traders, even the smart ones, get backtesting wrong. They fall for the same biases and make the same mistakes. They build strategies that look perfect on paper but fall apart the second they touch a live market.
This is my definitive guide to doing it right. I’m sharing the framework I’ve developed over a decade of building, investing in, and advising companies that live and die by their algorithms. If you’re serious about algorithmic trading, this is for you.
The Two Snakes in the Garden: Overfitting and Lookahead Bias
Before we get into the “how-to,” we need to talk about the two biggest reasons backtests fail. Get these wrong, and nothing else matters.
Overfitting is the siren song of algorithmic trading. It’s when you tailor your strategy so perfectly to historical data that it loses its predictive power. Your model essentially memorizes the past instead of learning general patterns. It’s like a student who crams for a test by memorizing the answer key. They’ll ace that specific test, but they’ll fail the real exam.
I once saw a team spend a year building a model to trade VIX futures. They had hundreds of parameters. The backtest was a thing of beauty. But when they went live, the market zigged where the model expected a zag. They had built a perfect map of a world that no longer existed.
Lookahead bias is more subtle, but just as deadly. It’s when your model uses information that wouldn’t have been available at the time of the trade. A classic example is using the closing price of a stock to decide to buy it at the opening price. You’re peeking at the future.
It’s easy to introduce this bias by accident. Using financial data that gets restated later, like earnings reports, is a common one. Or using data from a source that has been cleaned and corrected with the benefit of hindsight.
Avoiding these two pitfalls is 80% of the battle.
The Definitive Guide to Backtesting
Here’s my step-by-step process for backtesting a strategy. This isn’t just theory. This is a battle-tested framework that has made and saved me millions.
Step 1: Start with a Hypothesis
Every good strategy starts with a clear, testable hypothesis. Why should this strategy make money? What is the market inefficiency you are exploiting?
“Buy low, sell high” is not a hypothesis.
“Stocks with high revenue growth and low price-to-sales ratios in the technology sector tend to outperform the market over a 6-month period” is a hypothesis.
I remember when I was an angel investor in a small AI-driven hedge fund. The founder came to me with a black box algorithm. He couldn’t explain why it worked. I passed. A year later, the fund blew up. If you can’t explain your edge in a simple sentence, you don’t have one.
Step 2: Gather Your Data
Garbage in, garbage out. Your backtest is only as good as your data. You need clean, accurate, point-in-time data.
For my own projects, I use multiple data providers and cross-reference them. It’s expensive, but not as expensive as a failed strategy. For a serious trader, paying for quality data from a reputable source like Quandl or Polygon is a no-brainer.
Make sure your data includes:
- Delistings: Companies that go bankrupt or get acquired. Survivorship bias is a huge problem.
- Corporate Actions: Stock splits, dividends, mergers. These need to be adjusted for.
- Accurate Timestamps: To the millisecond, if you’re doing high-frequency trading.
Step 3: Split Your Data
Never, ever, test your strategy on the same data you used to build it. This is the cardinal sin of backtesting.
You need to split your data into at least two sets:
- In-Sample Data: This is the data you use to train your model and optimize your parameters.
- Out-of-Sample Data: This is the data you use to test your strategy. It’s a proxy for how your strategy will perform in the future.
A 70/30 or 80/20 split is common. I prefer to have multiple out-of-sample sets, from different time periods and market regimes.
Step 4: Run the Backtest
Now it’s time to run the backtest. This is where you simulate your strategy on the in-sample data.
Be realistic about your assumptions. Account for:
- Transaction Costs: Brokerage fees, exchange fees, etc.
- Slippage: The difference between the price you expect to trade at and the price you actually get.
- Market Impact: For larger trades, your own order can move the market.
I’ve seen so many backtests that assume zero costs and perfect execution. That’s not the real world. I usually add a conservative buffer to my estimated costs, just to be safe.
Step 5: Analyze the Results
Don’t just look at the final P&L. You need to dig deep into the performance metrics.
- Sharpe Ratio: Measures risk-adjusted return. Anything above 1 is decent, above 2 is great, and above 3 is exceptional (and potentially overfitted).
- Maximum Drawdown: The largest peak-to-trough drop in your equity curve. This tells you how much pain you would have had to endure.
- Win/Loss Ratio: The percentage of trades that are profitable.
- Average Win/Average Loss: Your average winning trade should be significantly larger than your average losing trade.
Look at the distribution of returns. Are you making a lot of small wins and a few huge losses? That’s a risky strategy.
Step 6: Validate on Out-of-Sample Data
This is the moment of truth. Run your strategy, without any changes, on the out-of-sample data.
If the performance is significantly worse than your in-sample results, you’ve probably overfitted your model. It’s time to go back to the drawing board.
Don’t be discouraged if this happens. It happens to everyone. The goal is not to create a perfect backtest. The goal is to build a robust strategy that works in the real world.
Step 7: Stress Test Your Strategy
The past is not a perfect predictor of the future. You need to test your strategy in a variety of market conditions.
- Walk-Forward Analysis: A more advanced version of out-of-sample testing, where you repeatedly optimize your strategy on a rolling window of data.
- Monte Carlo Simulation: Run thousands of simulations with different random variations in your data to see how your strategy holds up.
- Scenario Analysis: Test your strategy against historical market crashes, like the 2008 financial crisis or the 2020 COVID-19 crash.
I have a “pre-mortem” process for every new strategy. I get my team in a room and we brainstorm all the ways it could fail. What if interest rates spike? What if a black swan event happens? This helps us identify weaknesses before they cost us money.
Tools of the Trade
You don’t need a PhD in computer science to backtest a strategy. There are plenty of great tools out there.
- QuantConnect: A popular cloud-based platform for backtesting and live trading. They have a huge library of data and a great community.
- Backtrader: A powerful and flexible Python library for backtesting. It’s what I use for most of my own projects.
- Zipline: Another open-source Python backtesting engine, originally developed by Quantopian.
Start with one of these. Don’t try to build your own backtesting engine from scratch unless you have a very good reason.
A Final Word of Warning
A successful backtest is not a guarantee of future success. It’s a prerequisite. It’s the bare minimum you need to even consider trading a strategy with real money.
The market is a constantly evolving, adversarial environment. Your edge will decay over time. You need to constantly monitor your strategy, re-evaluate your assumptions, and be prepared to adapt.
I’ve made my career by being paranoid. I question everything, especially my own successes. That’s the mindset you need to survive and thrive as an algorithmic trader.
Now go build something great. And for God’s sake, backtest it properly.
Frequently Asked Questions
Is this guide based on real experience?
Every recommendation in this guide comes from direct experience, either from building and selling my own companies, or from patterns I've observed across 200+ angel investments. I don't write about things I haven't personally tested.
What if I disagree with some of the advice?
Good. That means you're thinking critically, which is exactly what a good founder should do. Take what resonates, test it, and discard what doesn't work for your specific situation. No advice is universal.
Who is this guide designed for?
This guide is written for founders and operators who want practical, actionable advice rather than theoretical frameworks. Whether you're just starting out or scaling an existing business, the principles here apply across stages.