Read on Twitter
How to add new classes to your ML model? 
... 
?
You have a large multi-class NN in production.
You discover a new important class and want to add support for it *quickly* and with *low* risk.
Example: traffic signs recognition for self-driving cars
Thread
... ?You have a large multi-class NN in production.
You discover a new important class and want to add support for it *quickly* and with *low* risk.
Example: traffic signs recognition for self-driving cars
Thread
The naive approach 
Collect examples of the new class (for example a new traffic sign), label them and retrain the whole NN.
It will probably work
It will be time consuming, especially for big models.
Risk for unintended regressions
Collect examples of the new class (for example a new traffic sign), label them and retrain the whole NN.
It will probably work It will be time consuming, especially for big models. Risk for unintended regressions
Freezing the first layers 
Typical CNNs learn generic image features in the initial layers and they will likely apply to the new sign as well.
You can freeze the weights of the initial layers and only retrain the last fully connected layer(s).
Typical CNNs learn generic image features in the initial layers and they will likely apply to the new sign as well.
You can freeze the weights of the initial layers and only retrain the last fully connected layer(s).
Faster retrain, because only the last layers are updated Lower risk for regressions For signs that look very differently, the CNN may lack the required features, leading to poor performance Still some risk for regression, because we touch all classes.
Extracting high-level features 
Train the NN to extract high-level features suitable for traffic signs, like shape, color, text or digits.
Define rules to classify each sign based on these features.
(Special credits to @ernestomancebo and @KwasiRansom for this idea)
Train the NN to extract high-level features suitable for traffic signs, like shape, color, text or digits.
Define rules to classify each sign based on these features.
(Special credits to @ernestomancebo and @KwasiRansom for this idea)
No need to retrain Low risk because classes can be separated well Will work well for speed limits Won't work for some sign categories, like for example warning signs that only differ by their pictogram Won't work for signs with new shapes
One-shot learning 
Transform sign images into a latent space, where they can be compared. Each class image is then described by a feature vector.
Siamese NNs can trained to transform images of the same class to very similar feature vectors in a latent space.
Transform sign images into a latent space, where they can be compared. Each class image is then described by a feature vector.
Siamese NNs can trained to transform images of the same class to very similar feature vectors in a latent space.
How it works?
Choose examples for the known classes and precompute their feature vectors
Transform new images to the latent space and find the best match to a known class
When a new class is found, simply add an example image to the set of known classes.
Choose examples for the known classes and precompute their feature vectors Transform new images to the latent space and find the best match to a known class When a new class is found, simply add an example image to the set of known classes.
No retrain needed Low risk for regression Only few examples of the new class needed May not work well if the new sign is very different and not supported well by the encoder
And finally some examples of traffic signs that you may not think exist until you encounter them... 
