Read on Twitter
My thoughts on baselines, a concept that is *extremely* relevant in industry ML but does not exactly translate from academic ML: 1/9
In academic ML projects, my classmates and I would code up logistic regression or simple models as baselines before hacking our way to make whatever complicated neural network architecture work. 2/9
I believed a good baseline is a simple off-the-shelf model that you can throw at a dataset. In my first few months in industry, I felt “woke” because I learned that, across the board, baseline models can be most effective. 3/9
So from scratch, I trained and deployed my xgboost classifiers on different tasks, disregarding the concept of baselines and adopting the “simple models work best” strategy. 4/9
As time passed, I wondered whether my models were generating any real value. Turned out that a single if statement / decision stump outperformed my model’s recall score. 5/9
I learned about this if statement because it was what the client was doing before using ML. I felt quite dumb; I should have inquired about the client’s existing policy and treated that as a baseline. 6/9
The concept of a baseline is super important in industry ML, otherwise you cannot quantify the value your ML solution provides. And you cannot build any product these days without measuring the value it adds. 7/9
Maybe this is obvious to most, but it took me some time to understand what a baseline truly means: it is the strategy currently used to solve a task. Maybe it’s an ML model; maybe it’s nothing. 8/9
I would love to see MLOps tools be more prescriptive about how people should iterate on modeling. Baselines in industry are the “SOTA”s in research — nothing is worth deploying unless it beats, in some way, what existed before. 9/9
