Detecting Financial Crime Using an Azure Advanced Analytics Platform and MLOps Approach
- Data Science, Machine Learning and MLOps
- 40 min
Applying MLOps principles, which are embodied by ML pipelines, enables fast development, increased feedback, and a culture of continual experimentation to machine learning. MLOps, as a set of principles can help automate the ML lifecycle such that machine learning models reach production and add the desired value. We have implemented these principles with some of the industry standard technologies such as Azure Databricks, MLflow, Azure Data Factory and Azure DevOps. We have developed a training pipeline template in close collaboration with data scientists which is easily scalable to many different use cases within DFC. The template consists of the usual components of training a model such as data preparation, data validation, model training and monitoring. Data scientists utilize this template and adjust different components according to model specific requirements. Our repo structure is inspired by CI/CD templates, as recommended by Databricks. Adhering to this structure and using YAML templates has enabled us to implement additional bank standard checks to validate the build and code before deploying to different environments. A successful run of our training pipeline results in registering a model in our central model registry which can then be consumed by our inference pipelines.
The MLOps approach streamlines the development of ML models and at the same time ensures that models stay relevant and valuable in a dynamic world such as the financial sector. We illustrate how the MLOps principles has contributed to a better cooperation between data scientists and IT Operation teams. Following these principles has created an environment that facilitates fast development, increased feedback and a culture of continual experimentation that is very much needed.