This leads me to mention my favorite epistemological question, to which I know no satisfactory answer. Imagine I flip a coin: Hayden says it will fall on heads for sure, Tyler says it will fall on tails for sure, Vera says it's impossible to tell (it's 50-50). •1/10 https://twitter.com/gro_tsen/status/1288881134424784896

The coin falls on heads. So Hayden exclaims “SEE? I WAS RIGHT!” But were they? I tend to think that Vera made the correct analysis: but how can we know this, even in hindsight? Of course, with a coin, it's easy to tell. But maybe the coin was biased. •2/10

With repeated experiments we could make a call. Based on information theory I suggest: for each experiment, ask each predictor to evaluate the probability of the N outcomes, and give them log(N) + log(p) points if they evaluated to probability p the outcome which occurred. •3/10

(So on my example, Hayden would receive log(2) points, Vera 0 and Tyler −∞.) But this method of scoring only gives meaningful results over the long run (when totaling the scores for many predictions), not on an isolated experiment! •4/10

Similarly, we can say “predictions by themselves are worthless: what matters is the reasoning by which the predictions were made”, which is nice in principle, but outside of pure math, science is validated by experiment, so we still need reproducibility. •5/10

For example, many people claim to have correctly predicted Trump's election in 2016, and laugh at someone like @NateSilver538 who placed the odds at something like Clinton 60%, Trump 40%. But events with 40% probability do occur (like… 40% of the time!). •6/10

Of course, people who claimed that Clinton had a sure win were unequivocally wrong. But even those who placed her odds at 99% have some wiggle room in theory: events with 1% probability do occur sometimes! (I'm still tempted to say that in this occasion they were “wrong”.) •7/10

Were the 60-40 odds incorrect? How can we tell? And what does this even mean? I have no good answer to this. The best I have is to run the test I discuss above, or to judge the method, but both require repeated runs. In real life, political coins can be flipped only once! •8/10

This applies to basically anyone who makes predictions about Covid-19. Just about every imaginable prediction has been made by some idiot for stupid reasons. Inevitably, some of these idiots will be right, and they'll laugh at me for claiming we can't predict anything. •9/10

My point is: making a correct prediction isn't enough to be “right” in the broader sense of the word. You have to make the correct prediction either reproducibly or for the right reasons — but this is tremendously difficult to test if we can't repeat experiments. •10/10