Read on Twitter
1/ Everyone’s talking about RCTs, but I am guessing that it is a bit of a vague concept for many non--medical folks. Here's a quick but longish look at a few important/essential factors of efficacy in RCTs. Safety considerations are different, complex, and beyond this tweetorial.
2/ Specifically I will cover:
Treatment
Outcome/ Measurement
Placebo
Triple Blind – patient, doctor, outcome assessor
Randomization
Pre-specified analysis plan/Null hypothesis
Clinical Effect
Type I error
Type II error
Unblinding
Analysis
Treatment
Outcome/ Measurement
Placebo
Triple Blind – patient, doctor, outcome assessor
Randomization
Pre-specified analysis plan/Null hypothesis
Clinical Effect
Type I error
Type II error
Unblinding
Analysis
3/ Treatment: Everything RCT starts with a treatment and an outcome. Both of these must be exactly specified – for treatment – who is treated (they have to either be at risk, for a preventative therapy or have the disease, for a treatment)?, what dose, how often, for how long?
4/ Outcome: For outcome – preventing (e.g. vaccine) ? reducing (e.g. blood pressure)? shortening (e.g. Tamiflu), each specified for as clear a result as possible. Measurements of outcomes have different amounts of “error", which should be understood in advance.
5a/ Measurement: Let’s make an example. Suppose we found an extract of pepper (PEP-1) that was effective at treating itch. We would need to find people who itch regularly (e.g. eczema) and have a measure for itching that is reliable (itching episodes /day?).
5b/ Lots of pre-work would tell us the dose and treatment regimen for PEP-1, and how long it took to work, etc. so we could design our trial to measure both the baseline amount of itching and itching at specific time points (? one week, one month, three months?).
6/ Placebo: Ideally, a clinical trial is run as placebo-controlled, “triple-blind.” Placebo is a pill or injection that is indistinguishable from the treatment in every way. Easier said than done, but it is important to do if possible.
7/ Blind: Everyone involved in the trial: the patient, those caring for the patient, and those assessing the results and analyzing the data must be unaware of whether a particular person receives PEP-1 or placebo.
8a/ Randomization: Randomization is a scheme where a patient in a clinical trial receives either treatment A or B based on a pre-specified “coin toss.” Nobody knows that A is placebo and B is PEP-1, and nobody should know which patient is on A or B.
8b / Randomization equally distributes factors – known and unknown - that might influence outcomes to the two groups. This means that, in the best circumstances, it should be possible to separate the effect of PEP-1 compared with placebo from all the other factors.
8c/ Coin tosses being what they are, sometime there will be imbalances. If there is a single very important known factor that could influence the outcome, randomization may be done separately for the group that has the factor and the group that doesn’t have it.
9/ Analysis Plan: The protocol must contain an analysis plan that has the statistical methods spelled out. This prevents statisticians from trying different methods to find one that works, after the trial. With an analysis plan, they have to stick to the plan.
10/ Null Hypothesis: The clinical trial depends on the null hypothesis. The null hypothesis states that there is no difference in outcome between the PEP-1 group and the placebo group. A successful clinical trial will reject the null hypothesis that there is no difference.
11a/ Type I Error: In most clinical trials, an error rate is set for the possibility of incorrectly rejecting the null hypothesis. That is, the chance that the results might say that there is an effect when there is none.
11b/ Type I Error A typical clinical trial would set a threshold of 0.05 (or 5%), which means that there is only a 5% probability that an ineffective treatment would be found to be effective. This is the measure of statistical significance.
12/ Clinical Effect: In a large trial, it is possible to reject the null hypothesis with a small effect, e.g. reducing itching from 20 to 19 episodes/day. It is important to chose a level of clinical meaningfullness, e.g a 40% decrease from 20 episodes/day to 12.
13a/ Type II Error: If you spend millions on a clinical trial, you want it to work (i.e. reject the null hypothesis for a clinically meaningful effect). Assuming a given effect size, the probability it will “work” is based on the number of patients enrolled in a clinical trial.
13b/ Type II error is the probability of missing an effect when one actually exists. In designing a clinical trial, the targeted number of patients is chosen to match an acceptable Type II error, usually 20%.
13c/ A Type II error set at 20% means that 1 of 5 trials of an effective drug will fail to reject the null hypothesis. But each patient in a clinical trial is expensive, and few companies can routinely design trials with better Type II error than 20%.
13d/ Here is a way to remember Type I vs. Type II error (these posts are what made me think of writing this tweetorial, actually). https://twitter.com/StotterMD/status/1304895378412711937?s=20
14a/ Unblinding: Once a clinical trial is completed, all data have been collected, and are ready for analysis, it can be unblinded. The list that matches treatment assignment with patient is taken out of the safe, applied to the data, and the analysis programming run.
14b/ Analysis: The result will show the actual effect measured in the clinical trial, which hopefully met the “clinical significance” criteria, and the statistical significance of the effect – whether it rejected the null hypothesis or not. We will now know if PEP-1 reduces itch.
A nice presentation from the MHRA that covers much of this and more with some details:
https://www.ema.europa.eu/en/documents/presentation/presentation-sme-workshop-session-1-statistical-common-types-clinical-trial-design-study-objectives_en.pdf
https://www.ema.europa.eu/en/documents/presentation/presentation-sme-workshop-session-1-statistical-common-types-clinical-trial-design-study-objectives_en.pdf
Addendum - I have spelled out typical parameters of an industry RCT. There are many, many variations on everything I have mentioned, and sometimes it simply isn't possible to meet the ideal in some of the parameters mentioned.
