Read on Twitter
A list of tech companies and their experimentation platforms. If you're an engineer and use (or want to use) experimentation / AB tests/feature flags, this is worth a read. A thread.
1. Uber. Lots of articles on Uber's engineering blog, from the platform itself ( https://eng.uber.com/experimentation-platform/), through analyzing outcomes ( https://eng.uber.com/analyzing-experiment-outcomes/) and a talk on decoupling experimentation logs from business metrics ( https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69126.html)
2. Doordash. Reading most of this was "deja vu: this is *so* similar to what we are doing at Uber!" It's a good writeup : https://doordash.engineering/2020/09/09/experimentation-analysis-platform-mvp/
Considering how fast Doordash went from <10% market share to market leader in the US, they definitely know how to experiment well.
Considering how fast Doordash went from <10% market share to market leader in the US, they definitely know how to experiment well.
3. Spotify's brand new experimentation platform: part 1 ( https://engineering.atspotify.com/2020/10/29/spotifys-new-experimentation-platform-part-1/) and part 2( https://engineering.atspotify.com/2020/11/02/spotifys-new-experimentation-platform-part-2/). It's neat to see UX like the experiments planner, even if the data is obviously mock.
4. LinkedIn shipped an improved experimentation engine a year ago ( https://engineering.linkedin.com/blog/2020/making-the-linkedin-experimentation-engine-20x-faster). What's great about this article is they share load numbers: 800K QPS (!!), 35K concurrent experiments (!).
5. Zalando. Listing this as it's "just out" a few days ago ( https://engineering.zalando.com/posts/2021/01/experimentation-platform-part1.html). Pretty cool overview of the adoption and a story you can relate to even if you're not working at a massive-sized company.
6. Airbnb ( https://medium.com/airbnb-engineering/https-medium-com-jonathan-parks-scaling-erf-23fd17c91166). This one is from 3 years ago, it gives a high-level overview and a neat UI visualization. Precomputed dimensional cuts seems like a big win (and an idea to copy).
7. Netflix. They were one of the first to share on how they did experimentation and this article is a classic (and highly read one): https://netflixtechblog.com/its-all-a-bout-testing-the-netflix-experimentation-platform-4e1ca458c15
I can't recall seeing a follow-up since: I wonder if thete's been any major changes since?
I can't recall seeing a follow-up since: I wonder if thete's been any major changes since?
8. Grab ( https://engineering.grab.com/feature-toggles-ab-testing). Nice to see an article that shows some actual code and configuration. If you build your own, this is good inspiration. Unfortunately, they don't specify what platforms and use cases this SDK is for: backend? Mobile? Web? All of them?
9. Intuit. I came across this one accidentally: Intuit open sourced their A/B testing platform ( https://github.com/intuit/wasabi ) and wrote this article about it ( https://www.intuit.com/blog/technology/engineering/meet-wasabi-an-open-source-ab-testing-platform/). Used in 120 apps(!!) across web, mobile and desktop.
Anyone used this project?
Anyone used this project?
10. Pinterest's platform. Also an early, 2016 writeup ( https://medium.com/pinterest-engineering/building-pinterests-a-b-testing-platform-ab4934ace9f4). I love how experiment changes need to go through code reviews!! Not the case in most places I've seen.
11. Of course, let's talk about the thing you're reading it this thread on: Twitter. This article is from 2015 ( https://blog.twitter.com/engineering/en_us/a/2015/twitter-experimentation-technical-overview.html) so I assume the infra probably changed since (in Manhattan still the DB powering Twitter?). The article is less of an overview, though.
These are some more interesting ones. Know of other notable mentions, and ones I've missed? Please reply!
All of this & more will be part of my book, Building Mobile Apps at Scale- and you can grab the PDF version for free (special shoutout to @bitrise). https://www.mobileatscale.com/
All of this & more will be part of my book, Building Mobile Apps at Scale- and you can grab the PDF version for free (special shoutout to @bitrise). https://www.mobileatscale.com/
