Your Trading System Might Be Broken...
How To Investigate System Stability Over Time
Welcome to the “Systematic Trading with TradeQuantiX” newsletter, your go-to resource for all things systematic trading. This publication will equip you with a complete toolkit to support your systematic trading journey, sent straight to your inbox. Remember, it’s more than just another newsletter; it’s everything you need to be a successful systematic trader.
Introduction:
A very common question I hear from systematic traders is:
“How do you know when your system is broken and how do I know when to turn it off before it’s too late?”
The short answer to that question is: generally you can’t know for sure if something is “broken.” Usually it’s too late before you actually know for sure.
But this answer is a bit lazy.
Maybe we can get some system degradation warning signs before the system completely dies. There are some things we can examine to understand if the system is degrading over time.
While there is no way to know for certain that the degradation will continue (maybe the system comes back to life), at least knowing that some degradation appears to be present is good information for us systematic traders to make decisions around.
Maybe if we could see degradation within a system, we could:
Allocate less capital to it as the confidence in the system persisting slowly decreases with degradation. Then, if the system comes back to life, we can increase capital allocation again.
Or maybe you put the system in a probation period.
Or maybe you just keep the knowledge of degradation in the back of your mind and are ready to pull the plug at a moment’s notice.
These are some of the tricky parts of systematic trading. These are the parts that require discretionary decisions, which we never like. But if we use some informed and unbiased data-driven approaches, then maybe that discretionary decision around “Is my system broken?” becomes a little more systematic.
Why Is It So Difficult To Know When Your System Is Broken?:
Knowing when a trading system is broken is hard for a few reasons:
We don’t know the future. Who’s to say the markets are just experiencing unfavorable conditions for the system and the system will start working again once the markets start to cooperate again.
Systems tend to ebb and flow, moving in and out of drawdown. Who knows if this seemingly normal drawdown is the one that never recovers?
Some drawdowns feel worse than others. Deeper, faster, or more aggressive than any historical drawdown. This can lead to doubt and the feeling that something fundamental changed.
Psychologically, the decision to turn off a trading system due to the belief of it being broken is hard as well because:
We fear that if we keep the system running and it breaks, then we will lose money as the system crashes and burns.
On the contrary, we fear if we turn the system off and it starts working again we will miss out on the gains as the system rips out of drawdown.
We put all this work into researching and developing the system, accepting all that hard work is no longer valid is frustrating and annoying.
Take, for example, the TSX (Canadian) momentum system we developed as part of the Portfolio Development Series. This system has had many difficult periods of time.
The system has had deep drawdowns, flat spots, and volatility. Consider the flat performance between 2014 and 2021. I don’t think anyone would fault you for thinking the system stopped working during that period. Only then for it to come roaring back to life in 2021 and 2024 (and keep in mind, 2019 onwards was used as out-of-sample for the development of this system).
TSX Momentum System:
Knowing if this system was broken or just experiencing normal drawdowns and sideways action is hard.
Very hard.
If this was the only system you were trading, that would have likely been very frustrating at times.
This is one reason why it’s always best to trade a portfolio of many diversified systems together. Then, when this TSX momentum system went sideways for years, you still would have had many other systems pick up the slack.
One of the best ways to know if a system is broken is through a deep understanding of how the system works. For example, this TSX momentum system obviously trades high momentum stocks.
So, if the TSX market does not exhibit traits of momentum for long periods of time, it would be reasonable to think the market conditions are simply unfavorable for this system; rather than the system just stopped working.
TSX Market Index - SPTSX:
Notice how the TSX market index has large drawdowns and sideways periods at the exact same times as the momentum system struggled. So, just through basic reasoning of how this system works, we may be able to convince ourselves it’s the market rather than the system.
A system can only take what the markets give. If the markets are not providing momentum, this momentum system cannot extract returns from the market.
Understanding the underlying system dynamics and why/how it makes money is important to understanding if the system is broken; or instead are the markets just not providing the conditions for this system to extract returns.
On the other hand, if you have a more esoteric strategy and you are not really sure why the system makes money; then it is a lot more difficult to know if it’s market conditions driven or if the system is actually broken.
A system comprised of “indicator soup” that looks nice in a backtest is cool, but going forward in time we have no understanding of why the system works and if the current drawdown is temporary or a full system failure.
We don’t understand what this “indicator soup” system is harvesting. So we have no concept of knowing if the lack of performance is due to the markets simply not being cooperative or if the system was curve fit to begin with, and is now falling apart.
Anyway, the first step in knowing if your system is broken or not is to answer and understand the following things:
Understanding what effect its harvesting.
Understanding why the system should work.
Understanding what market conditions need to be present for the system to work.
Understanding when those conditions tend to exist.
Understanding if the system capturing returns like it’s supposed to when favorable conditions exist.
Once you’ve had a critical thought about your system and can answer those questions, you can move on to the next section where we explore a tool I crated to look “under the hood” of your trading system.