Read on Twitter
ONS, breaches, + teachers: THREAD
Today I reported an ONS breach to the UK Statistics Authority.
Two breaches in fact: one regarding an “ad hoc” analysis on education staff in part 9 of the 6 Nov ONS covid survey pilot, and one for a related 3 Nov government leak. 1/
Today I reported an ONS breach to the UK Statistics Authority.
Two breaches in fact: one regarding an “ad hoc” analysis on education staff in part 9 of the 6 Nov ONS covid survey pilot, and one for a related 3 Nov government leak. 1/
On 3 Nov, the Department for Education tweeted a video of Chris Whitty saying:
“All the data, including ONS data, do not imply that teachers are a high risk occupation, unlike, for example social care workers, and medical staff like myself.” 2/ https://twitter.com/educationgovuk/status/1323701607159099392
“All the data, including ONS data, do not imply that teachers are a high risk occupation, unlike, for example social care workers, and medical staff like myself.” 2/ https://twitter.com/educationgovuk/status/1323701607159099392
According to the UK Statistics Office “Code of Practice for Statistics,”
https://code.statisticsauthority.gov.uk/wp-content/uploads/2018/02/Code-of-Practice-for-Statistics.pdf
this was a breach of trustworthiness principle T3: Orderly release. 3/
https://code.statisticsauthority.gov.uk/wp-content/uploads/2018/02/Code-of-Practice-for-Statistics.pdf
this was a breach of trustworthiness principle T3: Orderly release. 3/
T3.4 states that “ahead of their publication [...] no indication of the statistics should be made public.”
T3.8 says “Policy, press or ministerial statements referring to regular or ad hoc official statistics should [...] contain a prominent link to the source statistics.” 4/
![T3.4 states that “ahead of their publication [...] no indication of the statistics should be made public.”T3.8 says “Policy, press or ministerial statements referring to regular or ad hoc official statistics should [...] contain a prominent link to the source statistics.” 4/](https://pbs.twimg.com/media/EnYntHsXIAUL9ax.jpg "T3.4 states that “ahead of their publication [...] no indication of the statistics should be made public.”T3.8 says “Policy, press or ministerial statements referring to regular or ad hoc official statistics should [...] contain a prominent link to the source statistics.” 4/")
![T3.4 states that “ahead of their publication [...] no indication of the statistics should be made public.”T3.8 says “Policy, press or ministerial statements referring to regular or ad hoc official statistics should [...] contain a prominent link to the source statistics.” 4/](https://pbs.twimg.com/media/EnYntICWMAIR5sD.jpg "T3.4 states that “ahead of their publication [...] no indication of the statistics should be made public.”T3.8 says “Policy, press or ministerial statements referring to regular or ad hoc official statistics should [...] contain a prominent link to the source statistics.” 4/")
T3.8 says “Policy, press or ministerial statements referring to regular or ad hoc official statistics should [...] contain a prominent link to the source statistics.” 4/
This 3 Nov announcement led to confusion, as before 6 Nov, the only published occupation-differentiated ONS covid data was on deaths during spring lockdown.
And although Whitty said “all the data, including ONS,” he never ended up identifying non-ONS data on this topic. 5/
And although Whitty said “all the data, including ONS,” he never ended up identifying non-ONS data on this topic. 5/
The more important breach, though, has to do with the 6 Nov ad hoc analysis itself, which defied the “Code of Practice for Statistics” in so many ways as to raise serious concerns about independence. 6/
https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/coronaviruscovid19infectionsurveypilot/6november2020
https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/coronaviruscovid19infectionsurveypilot/6november2020
Though the other analyses in this 6 Nov report used data through *31 Oct* & included time dependence, the ad hoc teacher analysis only used 2 Sep-*16 Oct* data, & suppressed time dependence.
This matters: 17 Oct is about when 2-11-yr-olds overtook 35-49-yr-olds in infection. 7/
This matters: 17 Oct is about when 2-11-yr-olds overtook 35-49-yr-olds in infection. 7/
Why omit the 17-31 Oct occupation data—especially when this omission was so likely to bias results?
Did they feel they had run out of time to analyse already-acquired data? Code of Practice says that’s an unacceptable reason, especially considering the 3-day-early leak. 8/
Did they feel they had run out of time to analyse already-acquired data? Code of Practice says that’s an unacceptable reason, especially considering the 3-day-early leak. 8/
OTOH, with teacher training, many schools started Mon 7 Sep instead of Thu 2 Sep. Infection takes time to incubate before testing +.
So up to 10 days worth of their early data predated the option for school-acquired infection, diluting total positivity results for teachers. 9/
So up to 10 days worth of their early data predated the option for school-acquired infection, diluting total positivity results for teachers. 9/
Wait, it gets worse: just LOOK at the error bars!
Data was disaggregated in such a way as to make the study totally underpowered.
Error bar width is
>2 times the positivity value for most edu categories,
>33% higher than positivity for ALL BUT ONE edu category.
10/
Data was disaggregated in such a way as to make the study totally underpowered.
Error bar width is
>2 times the positivity value for most edu categories,
>33% higher than positivity for ALL BUT ONE edu category.
10/
But there was one exception: “teacher of unknown type,” with error bar 35% SMALLER than the positivity—meaning most respondents simply answered “teacher.”
THIS teacher category had
28% higher positivity than key workers,
16% higher positivity than other professions. 11/
THIS teacher category had
28% higher positivity than key workers,
16% higher positivity than other professions. 11/
So really, the chart should have just looked like the one below, with more sensible error bars,
and with teachers in the highest positivity category by far, even without the 17-31 Oct data.
12/
and with teachers in the highest positivity category by far, even without the 17-31 Oct data.
12/
It was only the disaggregation of teachers into underpowered buckets that made it plausible for ONS to claim:
“Data [...] show no evidence of differences in the positivity rate...” in the text and “there is no evidence of difference in the positivity rate” in the caption. 13/
![It was only the disaggregation of teachers into underpowered buckets that made it plausible for ONS to claim:“Data [...] show no evidence of differences in the positivity rate...” in the text and “there is no evidence of difference in the positivity rate” in the caption. 13/](https://pbs.twimg.com/media/EnYnvTZXcBMigfW.jpg "It was only the disaggregation of teachers into underpowered buckets that made it plausible for ONS to claim:“Data [...] show no evidence of differences in the positivity rate...” in the text and “there is no evidence of difference in the positivity rate” in the caption. 13/")
![It was only the disaggregation of teachers into underpowered buckets that made it plausible for ONS to claim:“Data [...] show no evidence of differences in the positivity rate...” in the text and “there is no evidence of difference in the positivity rate” in the caption. 13/](https://pbs.twimg.com/media/EnYnvTiXMAEDwKi.jpg "It was only the disaggregation of teachers into underpowered buckets that made it plausible for ONS to claim:“Data [...] show no evidence of differences in the positivity rate...” in the text and “there is no evidence of difference in the positivity rate” in the caption. 13/")
“Data [...] show no evidence of differences in the positivity rate...” in the text and “there is no evidence of difference in the positivity rate” in the caption. 13/
These “no evidence of differences” comments don’t mean what the audience is intended to think they mean.
Rather, they mean, “we disaggregated and restricted the data in such a way as to make the analysis too underpowered to generate significant evidence.” 14/
Rather, they mean, “we disaggregated and restricted the data in such a way as to make the analysis too underpowered to generate significant evidence.” 14/
This badly violates the Code for Practice.
And saying “no evidence of difference” vs “evidence of no difference” isn’t a get-out-jail-free card.
Context matters. Readers expect that if you publish an ONS analysis, the verdict is not: this was underpowered by design. 15/
And saying “no evidence of difference” vs “evidence of no difference” isn’t a get-out-jail-free card.
Context matters. Readers expect that if you publish an ONS analysis, the verdict is not: this was underpowered by design. 15/
The ONS are also obligated to provide rationale for substantial deviations from standard methodology, such as why they omitted 17-31 Oct data and suppressed time dependence for this one analysis, in a report that consistently provided time-dependent data through 31 October. 16/
On a side note, it contradicts Swedish findings.
In spring, Swedish teachers of upper sec schools (fully remote) had median infection rates.
Teachers of (fully open) lower sec schools had nearly twice the % of confirmed cases, and were among the most infected professions. 17/
In spring, Swedish teachers of upper sec schools (fully remote) had median infection rates.
Teachers of (fully open) lower sec schools had nearly twice the % of confirmed cases, and were among the most infected professions. 17/
It is not the role of the ONS to manufacture false evidence for purposes of state propaganda. This message of this one analysis was
-announced by Chris Whitty on 3 Nov,
-tweeted by the Department for Education on 3 Nov, ...
18/ https://twitter.com/educationgovuk/status/1323701607159099392
-announced by Chris Whitty on 3 Nov,
-tweeted by the Department for Education on 3 Nov, ...
18/ https://twitter.com/educationgovuk/status/1323701607159099392
-invoked by SAGE in the 4 Nov (published 13 Nov) report “TFC: Children and transmission 4 Nov, 2020,” where it was the only included “evidence” on teacher risk—in fact they (predictably) changed “no evidence of difference” to “show no difference”,...
https://www.gov.uk/government/publications/tfc-children-and-transmission-4-november-2020
19/
https://www.gov.uk/government/publications/tfc-children-and-transmission-4-november-2020
19/
-tweeted on 13 Nov by the Department for Education (this time by invoking the 4 Nov SAGE report) as new evidence of school safety, even though much of the actual child-related data in the report involved strong evidence of substantial transmission,...
20/ https://twitter.com/educationgovuk/status/1327246747102552066
20/ https://twitter.com/educationgovuk/status/1327246747102552066
-and invoked 18 Nov by Scottish Government as part of their evidence of school safety from covid,...
https://www.gov.scot/news/coronavirus-covid-19-in-schools/
https://www.gov.scot/publications/coronavirus-covid-19-evidence-on-children-schools-early-learning-and-childcare-settings-and-transmission-from-covid-19---summary-report/
(Again, “no evidence of difference” became “found there is no difference.”) 21/
https://www.gov.scot/news/coronavirus-covid-19-in-schools/
https://www.gov.scot/publications/coronavirus-covid-19-evidence-on-children-schools-early-learning-and-childcare-settings-and-transmission-from-covid-19---summary-report/
(Again, “no evidence of difference” became “found there is no difference.”) 21/
As an American-born, this all brings back too many bad memories of George W Bush’s “weapon’s of mass destruction.”
If you push hard enough for the manufacture of evidence, evidence will be produced, whether it’s valid or not.
22/
If you push hard enough for the manufacture of evidence, evidence will be produced, whether it’s valid or not.
22/
That’s why ONS independence is so incredibly important.
We can’t let this happen again.
23/end
We can’t let this happen again.
23/end
Since this kind of blew up, I wrote a follow-up thread. Wanted to clarify several things. Also, sorry for instances of inflammatory language in the above thread. Trying to be a good Twitter communicator, but still learning. 24/ https://twitter.com/SarahDRasmussen/status/1330744740711698433
Still, that follow-up thread mostly points out inconsistency in how the 6 Nov ONS Infection Survey Pilot presented/discussed this ad hoc school study vs other studies.
Like examples where captions in that report point out apparent trends with less confidence than this one,.. 25/ https://twitter.com/SarahDRasmussen/status/1330744754234126336
Like examples where captions in that report point out apparent trends with less confidence than this one,.. 25/ https://twitter.com/SarahDRasmussen/status/1330744754234126336
and I discuss in detail inconsistent use of the phrase “no evidence.”
In all 28 ONS ISPs, only 4 other individual analyses (sometimes repeated over weeks) use the phrase “no evidence.” Unlike the 6 Nov ad hoc, they all explain this term, and all mention error bars (CIs). 26/ https://twitter.com/SarahDRasmussen/status/1330744770629677063
In all 28 ONS ISPs, only 4 other individual analyses (sometimes repeated over weeks) use the phrase “no evidence.” Unlike the 6 Nov ad hoc, they all explain this term, and all mention error bars (CIs). 26/ https://twitter.com/SarahDRasmussen/status/1330744770629677063
Also, the Code of Practice for Statistics emphasises that
quality assurance, and discussion of strengths + limitations (eg inadequate data, uncertainty, and factors that might distort or bias results)
should be proportionate to the *importance* and *use* of the data. 27/ https://twitter.com/SarahDRasmussen/status/1330744789218824192
quality assurance, and discussion of strengths + limitations (eg inadequate data, uncertainty, and factors that might distort or bias results)
should be proportionate to the *importance* and *use* of the data. 27/ https://twitter.com/SarahDRasmussen/status/1330744789218824192
