MLflow helps organizations manage the ML lifecycle through the ability to track experiment metrics, parameters, and artifacts, as well as deploy models to batch or real-time serving systems. The MLflow Model Registry provides a central repository to manage the model deployment lifecycle, acting as the hub between experimentation and deployment.
A critical part of MLOps, or ML lifecycle management, is continuous integration and deployment (CI/CD). In this post, we introduce new features in the Model Registry on Databricks [AWS] [Azure] to facilitate the CI/CD process, including tags and comments which are now enabled for all customers, and the upcoming webhooks feature currently in private preview.
Today at the Data + AI Summit, we announced the general availability of Managed MLflow Model Registry on Databricks, and showcased the new features in this post. You can read more about the enterprise features of the managed solution in our previous post on MLflow Model Registry on Databricks.
Annotating Models and Model Versions with Tags
Registered models and model versions support key-value pair tags, which can encode a wide variety of information. For example, a user may mark a model with the deployment mode (e.g., batch or real-time), and a deployment pipeline could add tags indicating in which regions a model is deployed. And with the newly added ability to search and query by tags, it’s now easy to filter by these attributes so you can identify the models that are important to your task.
Tags can be added, edited, and removed from the model and model version pages, as well as through the MLflow API.
Adding Comments to Model Versions
With the latest release of the Model Registry, your teams now have the ability to write free-form comments about model versions. Deployment processes often trigger in-depth discussions among ML engineers: whether to productionize a model, examine any cause of failures, ascertain model accuracies, reevaluate metrics, parameters, schemas, etc. Through comments, you can capture these discussions during a model’s deployment process, in a central location.
Moreover, as organizations look to automate their deployment processes, information about a deployed model can be spread out across various platforms. With comments, external CI/CD pipelines can post information like test results, error messages, and other notifications directly back into the model registry. Also, in conjunction with webhooks, you can set up your CI/CD pipelines to be triggered by specific comments.
Comments can be created and modified from the UI or from a REST API interface, which will be published shortly.
Notifications via Webhooks
Webhooks are a common mechanism to invoke an action via a HTTP request upon an occurrence of an event. Model registry wehbooks facilitate the CI/CD process by providing a push mechanism to run a test or deployment pipeline and send notifications through the platform of your choice. Model registry webhooks can be triggered upon events such as creation of new model versions, addition of new comments, and transition of model version stages.
For example, organizations can use webhooks to automatically run tests when a new model version is created and report back results. When a user creates a transition request to move the model to production, a webhook tied to a messaging service like Slack could automatically notify members of the MLOps team. After the transition is approved, another webhook could automatically trigger deployment pipelines.
The feature is currently in private preview. Look for an in-depth guide to using webhooks as a central piece to CI/CD integration coming soon.
Monitoring Events via Audit Logs
An important part of MLOps is the ability to monitor and audit issues in production. Audit logs (or diagnostic logs) on Databricks [AWS] [Azure] provide administrators a centralized way to understand and govern activities on the platform. If your workspace has audit logging enabled, model registry events, including those around comments and webhooks [AWS] [Azure], will be logged automatically.
Get Started with the Model Registry
To see the features in action, you can watch today’s keynote: Taking Machine Learning to Production with New Features in MLflow.
You can read more about MLflow Model Registry and how to use it on AWS or Azure. Or you can try an example notebook [AWS] [Azure].
If you are new to MLflow, read the open source MLflow quickstart. For production use cases, read about Managed MLflow on Databricks and get started on using the MLflow Model Registry.