Can you trust a backtest?
Below is a simple strategy that uses intermarket signals from the Japanese Yen to trade US 30 Year Bond Futures.
It only has 2 rules for entry. The strategy produces fairly stable in-sample returns especially for being so simple. The in sample period is from January 2003 to August 2012.
*Graph shown shows results based trading 1 contract per signal for demonstration purposes*
The intermarket strategy actually performed quite well out of sample on our unseen data as well. The out of sample data is from September 2012 to October 2016 (highlighted in blue). The ability for a strategy to continue to perform on unseen data is paramount and mission critical for many system traders and strategy architects.
*Graph shown shows results based on 1 contract per signal for demonstration purposes*
However, can you trust this backtest? In Build Alpha, there are many validation and robustness tests that can add confidence to your system development process. There is a special test called the “Noise Test” which creates 1000 new price series by adding and subtracting varying amounts of noise from the underlying price data. The test then “re-trades” the strategy on the new 1000 price series.
The idea is that if a strategy is overfit then it is overfit to the noise of the original data and is not trading the underlying signal we hope it is. Changing the noise will reduce performance if the strategy is overfit to the noise of the original data. Ideally, we want to see a strategy continue to perform well on the new, noise-adjusted price series.
This “Noise Test” can help prevent a backtest from lying to us and hopefully prevent us from wasting further research time or taking a weak strategy live! You can see that our original backtest in blue is significantly better than the 1000 “Noise Test” results. This is worrisome and most likely a sign of an overfit strategy that will surely fail on live data.
Viewing this simple strategy’s performance since October 2016 to June 20, 2017 confirms the value of the Noise Test. The strategy has been in a nose dive since going “live”, but a smart trader would have never taken it live – no matter how pretty the backtest – due to the failed Noise Test.
Any kind of trading – manual or discretionary – should be subjected to these rigorous tests before getting real money allocations. If not, you may be in for a surprise from a lying backtest or unreproducible hypothetical returns!
Thanks for reading.
Dave is the founder of Build Alpha software. Build Alpha is software that automatically creates systematic trading and investing strategies, allows the validation and testing of each strategy, and generates exportable and executable code for each strategy – no programming necessary.
Twitter: @DBurgh
No position in any of the mentioned securities at the time of publication. Any opinions expressed herein are solely those of the author, and do not in any way represent the views or opinions of any other person or entity.
