Automating Model Lifecycle Orchestration with Jenkins
- MLOps and DataOps
- Moscone South | Upper Mezzanine | 160
- 35 min
The ML model lifecycle involves multiple steps and is extremely varied. A key part of the lifecycle involves bringing a model to production. In regular software systems, this is accomplished via a CI/CD pipeline such as one built with Jenkins. However, integrating Jenkins into a typical DS/ML workflow is not straightforward as data scientists use a completely different set of tools, like Airflow, Jupyter notebooks and custom model development platforms. This leads to a separation between the software world and the ML world, which increases friction and slows the model delivery process. Many teams end up considering models as simple binaries that their system consumes. Moreover, best practices that the CI/CD industry learned over the years are not leveraged for the ML lifecycle, leading to higher uncertainty about the quality of models deployed.
In this hands-on talk, I will talk about what Jenkins and CI/CD practices can bring to your ML workflows, demonstrate a few of these workflows, and share some best practices on how a bit of Jenkins can level up your MLOps processes. I will show how to integrate your regular ML development flow with an existing software development flow without significant overhead for your DS/ML team.