Continuous Evaluation of Deployed Models in Production

Download Slides

Many high-tech industries rely on machine-learning systems in production environments to automatically classify and respond to vast amounts of incoming data. Despite their critical roles, these systems are often not actively monitored. When a problem first arises, it may go unnoticed for some time. Once it is noticed, investigating its underlying cause is a time-consuming, manual process. Wouldn’t it be great if the model’s output were automatically monitored? If they could be visualized, sliced by different dimensions? If the system could automatically detect performance degradation and trigger alerts? In this presentation, we describe our experience from building such a core machine-learning services: Model Evaluation.

Our service provides automated, continuous evaluation of the performance of a deployed model over commonly-used metrics like the area-under-the-curve (AUC), root-mean-square-error (RMSE) etc. In addition, summary statistics about the model’s output, their distributions are also computed. The service also provides a dashboard to visualize the performance metrics, summary statistics and distributions of a model over time along with REST APIs to retrieve these metrics programmatically.

These metrics can be sliced by input features (e.g. Geography, Product type) to provide insights into model performance over different segments. The talk will describe various components that are required in building such a service and metrics of interest. Our system has a backend component built with spark on Azure Databricks. The backend can scale to analyze TBs of data to generate model evaluation metrics.

We will talk about how we modified Spark MLLib for computing AUC sliced by different dimensions and other optimizations in Spark to improve compute and performance. Our front-end and middle-tier, built with Docker and Azure Webapp provides visuals and REST APIs to retrieve the above metrics. This talk will cover various aspects of building, deploying and using the above system.

« back
About Deepak Pai

Adobe, Inc.

I am a Software Engineer and Data Scientist with 14 years of experience. I have published papers in top peer reviewed conferences and have been granted patents. As part of my current job, I manage a team of data scientists and engineers developing core ML services at Adobe. Our services are used by various Adobe Sensei Services that are part of Experience Cloud. I hold a Masters and Bachelor degree in Computer Science from a leading universities in India.

About Vijay Srivastava

Adobe, Inc.

I am working as Staff Data Scientist in the ML Core Services team. I have 13+ years of industry experience, 11 of which are at Adobe. As one of the initial members of the team, I am currently responsible for building and maintaining the core machine-learning service infrastructure that are part of Experience Cloud Intelligent Services.