1/Now in @NatMachIntell: our Matters Arising paper describing our external validation of a COVID-19 mortality prediction model by Yan et al., in collaboration with Matt Barish, @SiaBolourani, @LawrieLau @sareenshahmd https://www.nature.com/articles/s42256-020-00254-2

2/The initial study was published on 5/14, we submitted our paper on 6/5. In the initial publication by Yan et al., the authors claimed to have developed a novel simplified mortality prediction model to apply in clinical practice. https://www.nature.com/articles/s42256-020-0180-7

3/The model was proposed to be applied to any blood sample, far ahead of the primary clinical outcome, thereby suggesting its use as an admission triage tool. It was simple to use, interpretable, and highly accurate.

4/It also attracted tons of attention,>100k accesses, thousands of tweets, >40 news articles and blogs, and already dozens of scientific citations. We decided to take a closer look at it and validated their proposed decision tree against a large database of COVID-19 patients.

5/Some initial methodological concerns were raised. The first issue we noticed was that the researchers evaluated the model backward, starting from the closest point to death or discharge. While as good as any doctor is, they aren’t fortune tellers.

6/A clinical model that is contingent on knowing in advance the date of outcome is of no use and when validated this way, it most probably won’t work as advertised, when used in real circumstances. Once we validated it with data provided by the authors, performance dropped.

7/When validated with our own data, whether early in hospitalization or later, the model, in some ways, was no better than a coin flip; overall accuracy of .48 percent. We show that recalibrating the model and modifying thresholds of the tree won't increase prognostic utility.

8/In their reply to our paper, the authors mention the need to retrain using our data, which is a completely valid point but goes against all their (multiple) claims on generalizability.

9/Overall, our main goal was to underscore the need for more careful methodological steps, guidance from prediction model experts, and stakeholder engagement. It also showcases the usefulness of external validations