I have several concerns about the generalizability of our paper. The 17% response rate is not a major one. Here is why: 1/ https://twitter.com/Maherjane/status/1334754237004345345

1- Our sample was a very elite sample (i.e. high volume authors, or affiliation with high rank centres). So in effect the concern implies our elite sample is populated by slouches... which is a bit inconsistent 2/

2- We analyzed Brier scores by h-index, seniority and publication volume. No correlation. So the markers of expertise do not seem to correlate w/ f'cast skill 3/

3- Our findings are consistent w/ our neurology study, where we also surveyed co-I's on the study- who by definition shd not be slouches. They did no better than non-co-I's. 4/

4- An aside: ever look at participation rates in clinical trials (i.e. patients approached vs. patients randomized)? Almost no one talks about it, but the equivalent of response rates (which we care about for external validity) is often around 20% 5/

Here is what worries me about the generalizability of our findings: 1) the small sample of trials (maybe we picked really tough trials by chance alone). For example, one trial in our GU sample gave opposite results as a previous trial. Really hurt our GU group performance 6/

Second concern: we had no way to incentivize good prediction. One wonders what skill people might have had if they invested 2 hours in generating forecasts, reading through phase 2 trials, rather than doing a 15-30 minute survey 7/

But who knows- if I've learned anything, I've learned that experts often think they know more about how things work than they actually do. So I might be wrong! END/