ML Drift - How to Identify Issues Before They Become Problems // Amy Hodler // MLOps Meetup #89

MLOps Community Meetup #89! Last Wednesday we talked to Amy Hodler, Evangelist, Responsible AI of Fiddler. //Abstract Over time, our AI predictions degrade. Full Stop. Whether it's concept drift where the relationships of our data to what we're trying to predict as changed or data drift where our production data no longer resembles the historical training data, identifying meaningful ML drift versus spurious or acceptable drift is tedious. Not to mention the difficulty of uncovering which ML features are the source of poorer accuracy. Catch this meetup to understand the key types of machine learning drift and how to catch them before they become problems. // Bio Amy helps organizations see how they can achieve more responsible AI by improving machine learning explainability, accuracy, and bias detection. As the AI evangelist for Fidder Labs, she educates data scientists on the use of continuous monitoring for modern MLOps. Amy is the co-author of O’Reilly books on Graph Algorithms and Knowledge Graphs as well as a contributor to the upcoming book, AI on Trial. Amy has consistently helped teams apply novel approaches to generate new opportunities working at companies such as Microsoft, Hewlett-Packard (HP), Hitachi IoT, Neo4j, and Cray. Amy has a love for science history and a fascination for complexity studies. // Related links Responsible AI & Graph Technology -    • Responsible AI: Amy Hodler, Analytics & AI...   Slides from the session - https://www.slideshare.net/aehodler/m... ---------- ✌️Connect With Us ✌️------------ Join our Slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, Feature Store, Machine Learning Monitoring and Blogs: https://mlops.community/ Connect with Demetrios on LinkedIn:   / dpbrinkm   Connect with Amy on LinkedIn:   / amyhodler   Timestamps: [00:00] Introduction to Amy Hodler [02:49] Thank you Fiddler! [04:08] ML Drift [04:42] Chasing the real world [06:23] Fiddler [06:50] Model Drift world impact [07:36] Don't get too caught up in terminology [08:49] How we experience ML Drift [10:00] Key types of Drift [14:14] Drift examples for a Loan Application Model [16:33] Label and feature drift as part of Data Drift [17:13] Triggers of ML Model Drift [19:58] But really... What's really important? [20:20] Detect issues. Analyze root cause. Fix it! [21:01] Detecting issues [22:19] Performance monitoring and supervised learning [25:09] Data Drift Monitoring [27:39] Unsupervised Learning [30:09] Data Integrity and Outlier Monitoring [31:06] Getting the root cause [33:36] Feature importance neural network-based models of Fiddler [34:43] Fix it! [37:22] Q&A [37:30] Is how to Fix it a machine learning engineer job description? [39:37] Integration of Fiddler with MLFlow and other MLOps tools [41:47] Recommendations on resources to learn more about how to Fix it [44:50] All about Josh [47:30] Monitor different stages of machine learning life cycle [52:42] How to approach model performance monitoring? [55:40] Wrap up