"Since releasing the update, your product has changed. Your users have changed. The world has changed. Can you be confident that the initial feature update is the root cause?"

That's a classic external validity problem. Can you rely on your A/B tests from before Covid-19? https://twitter.com/patrickdoupe/status/1278583785970438144

"I ran an A/B test: are the results valid now? [...] will the results be valid once 'normal' life resumes?"

In our paper (w/ @eliasbareinboim), we present transportability techniques from the causal AI literature that can shed light on these questions. https://arxiv.org/abs/1912.09104 

These techniques allow you to leverage the power of graphical causal models to transport experimental results from one domain (pre-Covid-19) to another (now) based on expert knowledge about the differences in causal mechanisms across domains.

This framework is extremely flexible. It allows you to incorporate observational data, surrogate experiments, and leverage causal knowledge from several different source domains (meta-transportability). Also, the transportability task can be fully automated based on do-calculus.