Curve-fitting is almost certain death for a trading or investment strategy.
So, what is curve-fitting?
Well, you know when you test a trading or investment hypothesis, fall in love with the historical results, and then the idea fails to generate similar (or even positive) returns once you decide to trade it live?
Most of the pitfalls of system trading, and trading in general, can be avoided or mitigated following these three simple techniques or rules of thumb.
- Use out of sample data! Out of sample data is simply withholding some of the data in your “test” period for further evaluation. For example, you have ten years of historical data and opt to put the last 30% in your back pocket. You develop a great trading strategy on the first seven years of the data set and then whip out your “out of sample” data (remaining 30% from your back pocket) and validate your findings. If the strategy fails to produce similar results in the out of sample data then you can be almost certain you have curve-fit to the first seven years of your data set. Below is chart of a strategy built using Build Alpha that highlights the out of sample period. You like to see similar growth (and characteristics) in both the in-sample and out-of-sample data.
- Make sure your strategy has enough occurrences or trades. This can be simply explained with a coin flip example. If you flip a coin 10 times and it lands on heads 7 times you cannot be certain you do or do not have a rigged coin. However, if you flip a coin 10,000 times and it lands on heads 7,000 times you can have very high confidence it is indeed a rigged coin. In trading, if a strategy has 30 trades then it is unlikely you would have high confidence that what you have found is legitimate. However, if a strategy holds up over 1,000 or 3,000 plus trades then you can have higher confidence you’ve discovered true edge.
Below is a photo of only 30 coin flips and below that is a photo of six different trials of 100,000 coin flips. You can see after a large number of occurrences things tend to converge toward the true expectation (also known as the Law of Large Numbers).
- Validate your strategy across other markets. If a strategy works on only one market it has a higher chance of being curve-fit to the data set than if a strategy performs profitably on a handful of markets. I am not saying that a strategy that only works on one market is curve-fit – as there are many nuances, different players, and idiosyncrasies that exist within each market. However, if a strategy performs across markets then you can certainly have higher confidence that it is not curve-fit.
Thanks for reading.
Dave is the founder of Build Alpha software. Build Alpha is software that automatically creates systematic trading and investing strategies, allows the validation and testing of each strategy, and generates exportable and executable code for each strategy – no programming necessary.
Twitter: @DBurgh
No position in any of the mentioned securities at the time of publication. Any opinions expressed herein are solely those of the author, and do not in any way represent the views or opinions of any other person or entity.
