As a recovering quant, I share this cautious sentiment about data driven decisions.

Ways data driven decision can be worse than careful thinking: https://twitter.com/shreyas/status/1288674294881427456

Precision is not accuracy

A repeatable and precise measurement of a system may not be the measurement that matters most.

Bias towards the measurable

Tendency to measure what is easy to measure. And ignore what is hard to measure.

Jumping to conclusions

A false assumption about the data, methodology mistake, or bug in the code, can invalidate a conclusion.

Who’s double checking?

Over-optimization of metrics / metrics gaming

“You can’t improve what you don’t measure”. But also, it might become the only thing you improve, as the cost of the main prize.

False sense of confidence

Data, tables, charts, graphs... are great to fool people with. Don’t fool yourself.

Data ultimately needs interpretation

I remember a study giving a bunch of experts the same data, and they all ended up with different conclusions.

The data driven process should supplement intuition.

Intuition has access to more varied data.

Think first, then do data.