Read on Twitter
1/ A somewhat problematic WSJ editorial by @AllyssaFinley. I’ll start with some comments, but hope that some of the experts in clinical trials and observational research that I follow can chime in. https://www.wsj.com/articles/medical-researchs-cross-of-gold-imperils-covid-treatments-11599589020
2/ First, let’s start with a couple of things that are right: “Randomized controlled trials can yield important insights, but it is a medical mistake and a disservice to patients to dismiss other types of evidence.” Exactly right...
3/ In fact, I have dedicated much of my career to observational research and their value. OR leading to real world evidence (RWE) can be a very important way to generate the design and specifics for an RCT. Or, appropriately done, they can even be strongly supportive of an RCT.
4/ But, in the end, even the best observational studies can only control for variables that are known and expected by the researcher. Others may be heavily imbalanced between comparison groups because of selection bias, confounding, and conditioning on colliders.
5/ Second thing that is right, a comment from @donberwick, “Randomized trials for some purposes is the gold standard, but only for some purposes” There is definitely a major role for observational research in many settings. But...
6/ ..the confirmation of the efficacy and safety of a new drug (or new indication for a drug) or vaccine isn’t one of them. Observational research can definitely add to an RCT for these purposes, but can’t stand alone because...
7/ ...of the inability to equally allocate the many potentially confounding factors within the placebo and drug group. Randomization can often (but not always) do this.
8/ OK, some things that are incorrect: First, a big one: “Trials don’t reveal differences in how patients respond to a drug at different dosages or illness severity.” Not true...
9/ This depends on how the trial was designed. At least one Phase II clinical trial for every drug is dose ranging and multiple Phase II trials can be done in different disease severity.
10/ Next up, a reference to the “37% reduction in mortality among patients under 80 who weren’t on a ventilator and received a high-antibody plasma within three days of hospitalizations.” with convalescent plasma. First, it was a comparison with active, not placebo. Second...
11/ ...this kind of cherry picking is exactly what clinical researchers are concerned about. Look at enough “sliced and diced” subgroups and you are bound to find an effect.
12/
Then, the editorial goes on to say “A randomized trial might have obscured these nuances and complexities, denying doctors important information about treatment options.” This is just not right...
Then, the editorial goes on to say “A randomized trial might have obscured these nuances and complexities, denying doctors important information about treatment options.” This is just not right...
13/ You can do subgroup analysis in clinical trials like observational research (although smaller numbers of patients in RCTs means you can’t really do as many different comparisons)...
14/ However, the FDA isn’t going to be impressed. You are supposed to create an “a priori” hypothesis and test it. They have too much experience in seeing repeats of clinical trials based on subgroup analysis fail to show any benefit, or worse.
15/ Next, the editorial equates a bunch of observational studies of HCQ/AZ with three RCTs – the observational studies are good for hypothesis generation, but the RCTs specifically address the question...
16/ Once you have well-controlled and designed negative RCTs, the only evidence that is going to cast doubt on the result is going to be another RCT with a different result (because, even RCTs are subject to variability in the size of the effect measured).
17/ The editorial is concerned that RCTs sometime fail to show an effect for a drug, such as remdesivir. It points to two studies that failed to reach significance because of the size or choice of study group....
18/ Yes, that’s right. RCTs sometimes aren’t designed well or can enroll the wrong patients or fewer patients than needed.
19/ On the other hand, observational analysis can redone until a significant result is reached. That isn’t the way it’s supposed to be done, but it happens. The best RWE is done with pre-specified analysis plans and no data dredging beforehand to figure out what will work.
20/ OK, that’s a start. Let me invite in a few people who could add a great deal to this discussion. @nataliexdean @RFRedberg @califf001 @michaelmina_lab @EpiEllie @angie_rasmussen @EricTopol @jsross119 @f2harrell @ChristosArgyrop @ZoeMcLaren
