Read on Twitter
COVID-19, and the subsequent economic effects it triggered, is a major factor behind widespread MLmodel degradation across all kinds of models in the industry today.
Here is how MLOps Monitoring and Explainability can help. /thread
Here is how MLOps Monitoring and Explainability can help. /thread
A Machine Learning model is developed to construct relationships between target and feature variables using historical data.
Put differently, we analyze the past in order to predict the future. /2
Put differently, we analyze the past in order to predict the future. /2
The degree of difference between the future and the past is the primary reason most models degrade over time and must eventually be replaced.
ML Engineers call this difference "Model Drift". /3
ML Engineers call this difference "Model Drift". /3
Often times, when faced with performance issues ML engineers retrain the models with corrective actions or overlay code to factor in its weaknesses and limitations.
COVID-19 has seen more and more adjustments to models to compensate for their performance degradation. /4
COVID-19 has seen more and more adjustments to models to compensate for their performance degradation. /4
Ideally, we want every model not just to have high accuracy, but also high stability so that it can be applied to different time periods of data without significant loss in its prediction accuracy. /5
A model stability measure evaluates if the distribution and value range of inputs and outputs have shifted over time, such as between the production data and the development sample, or between different windows of production data. /6
Model robustness analysis also becomes important as it ensures conclusions hold under different assumptions.
Robustness can be viewed as a model's resilience measurement given dramatic changes to model inputs.
This is where the model explainability becomes super critical. /7
Robustness can be viewed as a model's resilience measurement given dramatic changes to model inputs.
This is where the model explainability becomes super critical. /7
COVID-19 has caused many models to break in record time, casting doubt on model reliability and the organization's reliance on models when navigating through the crisis. /8
Example: https://fortune.com/2020/06/09/instacart-coronavirus-artificial-intelligence/
Example: https://fortune.com/2020/06/09/instacart-coronavirus-artificial-intelligence/
ML models are essentially stochastic entities and they fail silently. Unlike traditional software, we cannot monitor obvious issues like 404s, high latencies, etc.
Often times labeled data may not be available, to know if the model accuracy is deteriorating. /9
Often times labeled data may not be available, to know if the model accuracy is deteriorating. /9
And when things go wrong, corrective actions are not clear, it is not as simple as adding more servers.
Careless corrective actions can further worsen the model performance and increase model risk. /10
Careless corrective actions can further worsen the model performance and increase model risk. /10
COVID-19 pandemic has led to increased suspicions around the accuracy and reliability of ML in an organization.
By monitoring 5 key aspects of a Model, @fiddlerlabs helps ML teams minimize the risks while maximizing the ROI from their ML apps. /end https://blog.fiddler.ai/2020/04/explainable-monitoring-stop-flying-blind-and-monitor-your-ai/
By monitoring 5 key aspects of a Model, @fiddlerlabs helps ML teams minimize the risks while maximizing the ROI from their ML apps. /end https://blog.fiddler.ai/2020/04/explainable-monitoring-stop-flying-blind-and-monitor-your-ai/
