How should we interpret #COVID19 case counts given the enormous variation in testing? A quick cautionary thread.

As everyone knows the # of "confirmed cases" for COVID19 are a small fraction of actual infections. Most early estimates put it at 1/10 (1/5-1/30) of infections.

But things have changed. Testing has ramped up everywhere; in some places a ton. The fraction of total infections detected is higher (probably 1.5 to 5x higher so 1/5-1/10). But it's variable and in some places there are now large groups of asymptomatic people also being tested.

In my county in CA, a state testing center ("Optimserve" on webpage below) opened only for asymptomatic people (later mildly symptomatic people were allowed). It accounted for 1/2-1/3 of all test/day in the county over the past 6 weeks.
http://santacruzhealth.org/HSAHome/HSADivisions/PublicHealth/CommunicableDiseaseControl/CoronavirusHome/SAVELivesSantaCruzCounty.aspx

It will be discontinued within a month as CA stops paying for it; it tried to offload costs to county but they are far too high for County afford. It now has very slow turn around times and has limited re-agents so not a huge loss in its current form but was helpful for a while.

What does this mean for interpreting trends in cases and % of tests positive?
Testing is highly variable making cases & % positive hard to interpret. A quick example w/ data.

Before asymptomatic testing started (March- mid-May), just symptomatic people were getting tested by hospitals. Testing increased in mid-April b/c hospitals bought new machines & people were trained how to use them. Cases stayed low b/c of lockdown through early May. % + low.

Lifting lockdown led to increase in transmission (Mother's day, May 10th: many large family gatherings led to 55+ cases detected over next 3 weeks). But % tests + spiked & then decreased due to Asymptomatic testing site also opening and ramping up to 100+ tests/day.

Most of these tests were negative b/c asymptomatic people.
In June w/ more re-opening, cases increased faster than increase in testing, finally leading to increase in % positive.
What was pattern of infections over this whole period? Very difficult to say!

If tests were characterized as for "asymptomatic", "mild symptoms", "symptomatic", and "contact tracing" one could try to sort out these changes in testing and better interpret cases. But will this data be shared by counties & states? Not so far.

What do we do with highly variable testing capacity and criteria that is not reported clearly?
We can use hospitals and deaths of course but those occur 13.5&27.2d ( https://www.medrxiv.org/content/10.1101/2020.04.12.20062943v1), on average after infection so very delayed.

Also hospital transfers are now complicating those data. My county received hospital transfers from prison outbreak in SF. In TX hospitals are transferring patients b/c no beds remaining. If these cross county lines, data becomes tough to interpret.

Statewide data less likely to include hospital transfers (but some do occur) but doing so mixes counties with very different infection trajectories (much like combining states to look at whole US). In short - it's a mess. If everything points in 1 direction, ok. If not, ?????