I have cannibalised this from another conversation I had earlier but the words are mine so my corespondent won’t mind-

When you are looking back at historical data and performing backtests etc. there are some subtle things that are out there and should be considered.

When you are looking at information that already existed at the time your had your inspired idea you need to note that relative to players who traded those prices in real time you have a spectacular advantage. That advantage is that you did not have to deal with the changing...

.. distributions in real time and also any new statistical outcomes (that occur every day) are in your data set. The historical real time trader had to deal with these things as they developed. These are subtle things but are devastating to the performance of quantitative..

.. strategies when taken into production/ out of sample. This circles back quite nicely to the following : You simply must expect and plan for more variance in prices in the future on data that does not yet exist than you have observed historically. Doing this allows you ..

.. to - if not compensate for - then certainly acknowledge, the advantages you have by looking back and hoping for some Consillience with the future.