Out-of-sample testing
Out-of-sample testing means measuring a strategy's performance on data that was not used to create it. In-sample results — performance on the data the strategy was built from — carry almost no information about whether it will work.
In plain terms
If you designed a rule by looking at 2018 to 2024 data, then its performance across 2018 to 2024 tells you mainly that you designed it well for 2018 to 2024. The only informative test is data that played no part in the design.
Why it matters
Almost every strategy looks profitable in sample. That is what building it means. The gap between in-sample and out-of-sample performance is the single most useful diagnostic you have: a small gap suggests something real, a large gap suggests you fitted noise.
The contamination problem
Out-of-sample data stops being out of sample the moment you use it to make a decision.
This is subtler than it sounds, and it is where most careful people still go wrong. You test a strategy, it fails out of sample, you adjust it, you test again. The second test is no longer out of sample — you have used the answers to change the question. Do that five times and your "out-of-sample" period has quietly become in-sample.
The only real defences are to declare your test in advance and to account for how many attempts you have made. Which is precisely what a multiple-testing penalty does.
How Mithos handles it
Out-of-sample measurement is built into the walk-forward structure rather than being a separate optional step. And because the platform tracks how many related variations you have tested, repeated attempts on the same underlying idea raise the bar rather than silently eroding it.
Answers to the usual ones
- How much data should be held out?
- There is no universal answer, and the question is less important than whether the held-out data stays genuinely untouched.
- Can I retest after changing a strategy?
- Yes, but honestly — each retest is another attempt, and the statistical bar should rise accordingly.