Read on Twitter
It’s been rightfully pointed out that OWS protocols have not been made public; suspicious given current political climate.
Here’s a thread with reflections on designs for COVID vax trials that spells out some tricky issues involved in these trials from my perspective.
1/20 https://twitter.com/erictopol/status/1304546058983227392
Here’s a thread with reflections on designs for COVID vax trials that spells out some tricky issues involved in these trials from my perspective.
1/20 https://twitter.com/erictopol/status/1304546058983227392
Disclaimers: views entirely my own but informed by long discussions with many others involved in vaccine development process.
The elements discussed are broadly similar to what I expect most (US) Phase 3’s to look like, but I’m not reporting on exact trial protocols.
2/20
The elements discussed are broadly similar to what I expect most (US) Phase 3’s to look like, but I’m not reporting on exact trial protocols.
2/20
Baseline: ptcpt randomized 1:1 or 2:1 to vaccine or placebo.
Which one?
2:1 = easier (more power) to detect effective vaccines; easier to learn about immune responses induced by vaccine
1:1 = less likely to erroneously conclude vaccine increases severe adverse events
3/20
Which one?
2:1 = easier (more power) to detect effective vaccines; easier to learn about immune responses induced by vaccine
1:1 = less likely to erroneously conclude vaccine increases severe adverse events
3/20
Baseline: Ptcpt have serology at enrollment to detect past infections. If +, participants allowed to be vaccinated.
What to do with +s?
Exclude from analysis: easier to detect vax eff if past inf is sterilizing
Include in analysis: potential for broader indication
4/20
What to do with +s?
Exclude from analysis: easier to detect vax eff if past inf is sterilizing
Include in analysis: potential for broader indication
4/20
Baseline: make sure to enroll high risk participants, and in particular, individuals over 60, racial/ethnic minorities.
Why?
These are individuals who have borne brunt of pandemic.
High risk = more disease endpoints = faster results
5/20
Why?
These are individuals who have borne brunt of pandemic.
High risk = more disease endpoints = faster results
5/20
Follow up: ptcpt are followed actively for symptomatic disease. If sick, come in for a visit, get tested.
Also follow up for asymptomatic inf at scheduled visits via serology.
Active follow up for infection AND disease is ideal, but may be logistically infeasible.
6/20
Also follow up for asymptomatic inf at scheduled visits via serology.
Active follow up for infection AND disease is ideal, but may be logistically infeasible.
6/20
Primary endpoint: gathering consensus for symptomatic dis.
Why not infection? false positives bias VE towards null.
Why not severe dis? Prohibitively large sample sizes.
Why not burden of dis? How to judge death vs severe vs symp vs asmyp dis. (I still like this endpt)
7/20
Why not infection? false positives bias VE towards null.
Why not severe dis? Prohibitively large sample sizes.
Why not burden of dis? How to judge death vs severe vs symp vs asmyp dis. (I still like this endpt)
7/20
Power: assume 60% VE with primary test of a null of 30%. Power calcs computed under assumption that the vaccine takes time to “ramp up”.
This ends up being ~150 disease endpoints required, with best guess of ~30,000 ptcpt enrolled to see that many disease cases.
8/20
This ends up being ~150 disease endpoints required, with best guess of ~30,000 ptcpt enrolled to see that many disease cases.
8/20
Aside: the excellent seqDesign R package can be used to generate these power calcs.
https://cran.r-project.org/web/packages/seqDesign/index.html
9/20
https://cran.r-project.org/web/packages/seqDesign/index.html
9/20
Interim efficacy monitoring: Now to the juicy parts.
Plan for 2-3 interim looks adjusting for multiple testing using alpha spending function (eg O’Brien Fleming-like).
Let’s pause here to say a little bit more about this.
10/20
Plan for 2-3 interim looks adjusting for multiple testing using alpha spending function (eg O’Brien Fleming-like).
Let’s pause here to say a little bit more about this.
10/20
Why look early? To detect highly effective vaccines as quickly as possible.
Is there risk in looking early? Yes, always risk of a false positive signal, but we control the rate of false positives by making the burden of proof more stringent with more looks.
11/20
Is there risk in looking early? Yes, always risk of a false positive signal, but we control the rate of false positives by making the burden of proof more stringent with more looks.
11/20
How likely is a trial to stop early? Unlikely, unless VE very high.
With 2 looks, first after 75 events. Even if vaccine is truly 60% effective (good for licensure) there’s just a 1/4 chance we would stop early.
On other hand, if VE over 90%, about a 9/10 chance.
12/20
With 2 looks, first after 75 events. Even if vaccine is truly 60% effective (good for licensure) there’s just a 1/4 chance we would stop early.
On other hand, if VE over 90%, about a 9/10 chance.
12/20
So what happens if one of these trials stops early? Fascinating question.
Do all participants get unblinded and given the vaccine?
+ equipoise
- can’t learn long term VE
And what of participants in other ongoing phase 3 trials?
There will be some tough decisions.
13/20
Do all participants get unblinded and given the vaccine?
+ equipoise
- can’t learn long term VE
And what of participants in other ongoing phase 3 trials?
There will be some tough decisions.
13/20
What about stopping early for futility? If we look early and the vaccine stinks, stop early to invest $ elsewhere.
Interesting ? is right balance here: if too aggressive stopping early, risk missing eff vax. If not aggressive enough, risk wasting $ on a dud vaccine.
14/20
Interesting ? is right balance here: if too aggressive stopping early, risk missing eff vax. If not aggressive enough, risk wasting $ on a dud vaccine.
14/20
Under two-look approach described above, there’s a 1/5 chance we’d correctly stop early because vaccine is a dud (0% VE) and a 1/20 chance we’d stop early by mistake when vaccine effective enough to be licensed (50% VE).
15/20
15/20
What about adverse event monitoring? This is happening continually throughout the trial. Whenever a serious AE is recorded the DSMB has power to pause or stop trial.
16/20
16/20
So what’s best possible outcome? Stop one of the first trials early and vax is highly effective (>90%).
When could that happen? As early as fall.
How likely is that? Not.
Why? VE must be very high AND fast enrollment AND fast endpt accrual AND we get a little lucky.
17/20
When could that happen? As early as fall.
How likely is that? Not.
Why? VE must be very high AND fast enrollment AND fast endpt accrual AND we get a little lucky.
17/20
What’s worst possible outcome? Middling vaccine is approved early on.
Why? Don’t get to learn about durability, calls into question ethics/feasibility of future RCTs.
Could happen due to bad luck (random high bias) or, as many are worried about, political interference.
18/20
Why? Don’t get to learn about durability, calls into question ethics/feasibility of future RCTs.
Could happen due to bad luck (random high bias) or, as many are worried about, political interference.
18/20
Of course I’m worried about politics in science, but I haven’t seen it *yet*.
Skepticism is justified and > transparency would go a long way.
Companies will (should) want total confidence in their product and I think (hope) will push back on political interference.
19/20
Skepticism is justified and > transparency would go a long way.
Companies will (should) want total confidence in their product and I think (hope) will push back on political interference.
19/20
Maybe that’s naive. We’ll soon see. In the meantime, we will continue to do the best science we can.
Thanks for tuning in.
20/20
Thanks for tuning in.
20/20
