Multiple testing, in plain English

Every variation you try is another ticket in a lottery. Change the stop, move the entry time, add a filter, widen the target — twenty tries in and one of them will look brilliant by luck alone, on data whose shape you already know.

You cannot feel the difference from inside. The lucky variation arrives with the same confident equity curve a real edge would have. The search that produced the result is invisible in the result itself.

This is the part that catches careful people out. Being disciplined does not help, because nothing about the winning chart tells you how many charts you looked at before it. The effect has a name — multiple testing — and the correction is arithmetic rather than willpower: the more things you tried, the higher the bar a result has to clear before it means anything.

Most backtesting setups don't count the tries at all. The number that would tell you how impressed to be is simply never recorded.

In Mithos, the count is part of the test. When you explore a range of settings rather than a single strategy, the number of combinations you searched is carried forward into the verdict, so a result found after two hundred attempts is judged against a harder standard than the same result found on the first attempt.

That is the whole trick, and it is deliberately unglamorous. Nothing here makes a good strategy better. It just stops a lucky one from passing itself off as a good one.

The concept page on multiple testing bias goes into the mechanics, and Why Mithos sets out the wider argument: a backtest was never the hard part.