Read on Twitter
Since this graph is being circulated by MPs I guess it’s worth explaining why it’s a misleading (and generally dumb) graph and how you’d go about presenting the same data if your intent were to inform rather than scaremonger.
Laidlaw has compiled his graph using data on lumbar spine bone mineral density (BMD) from a recent NHS puberty blocker study. https://www.medrxiv.org/content/10.1101/2020.12.01.20241653v1.full-text
Despite his assertion that the graph “shows baseline spine bone density and then the decrease over time” the original data actually show slight growth in BMD. (He goes on to complain that the problem is insufficiently rapid increase in BMD).
If your only graphical presentation of these data is a series of columns getting shorter over time, you’d better be very clear and upfront about the fact that these data describe an overall increase in BMD, otherwise people might question your integrity.
Laidlaw has taken the z-values from Table 3 in the original study and converted them into percentiles. This tells you where the average member of the sample falls within the distribution of BMD for the general population (adjusting for age (and possibly height)).
So if the result for a sample is the 34th percentile then 34% of the general population are below the average member of the sample and 66% are above.
The reason percentiles at follow-up are lower than the baseline is because the participants are increasing their bone density at a slower rate than peers in their age group.
The result may still seem scary, but here’s where Laidlaw’s mistakes become important. He’s made two major errors and at least one minor one.
His first mistake comes from presenting only one baseline percentile to compare the three follow-ups to.
His first mistake comes from presenting only one baseline percentile to compare the three follow-ups to.
A baseline is necessary because your sample may not be typical of the general population. (In this case they started off with a lower-than-average BMD).
You can’t interpret a result after 1/2/3 years of treatment unless you now what you’re comparing against.
You can’t interpret a result after 1/2/3 years of treatment unless you now what you’re comparing against.
But Laidlaw’s mistake is to take the baseline for all n=44 study participants and present it implicitly as the baseline against which all follow-up samples should be compared.
In reality the no. of participants decreased (n=43, 24, 12) at each follow-up stage (because participants were reaching the age of 16 and going onto cross-sex hormones (CSH)).
So the sample being considered at 24 months is a sub-sample of the one at 12 months etc.
So the sample being considered at 24 months is a sub-sample of the one at 12 months etc.
For this reason the result at each stage needs to be compared against a baseline for the sample assessed at that stage, not the overall baseline.
In this case the effect of this error is to make the change at e.g. 24 months look much more dramatic than it is since the baseline for that group was lower than the overall baseline.
The second big mistake is that for some reason Laidlaw doesn’t include the 95% confidence intervals on his graph (despite them being presented in a convenient format in the table he’s copying out).
When considering a sample of a larger population (e.g. participants in this study as a sample of all people who receive PBs) the mean of the sample will differ from the mean of the population. Confidence intervals mark the range within which the true value is likely to lie.
Fixing these two errors I can present a set of corrected graph. They compare the sample at each follow-up with the relevant baseline for that sample.
(Note the close/overlapping confidence intervals).
(Note the close/overlapping confidence intervals).
His third and more minor mistake (probably typographical) is that he seems to have taken the z-value of the baseline from those that have been adjusted for both age and height and the z-values at follow-up from those adjusted for age but not height.
I include this second version in case his intention was for the graph to show height adjusted percentiles.
Finally, Laidlaw and co. are presenting these findings as though they represent a lifelong change in BMD relative to peer-group without any justification.
I'm not a bone guy but a quick google suggests most increase in bone density is occurs post-puberty (even without CSH).
I'm not a bone guy but a quick google suggests most increase in bone density is occurs post-puberty (even without CSH).
P.S. Though I've focused here on the problems with how the data is being presented, I'd quickly add that the implicit premise of the graph that we should be especially concerned about BMD percentile relative to peer group rather than some medically relevant threshold seems silly.
