SESSION

SciCLOps: Databricks Quick Start for Machine Learning, Powered by DABs

OVERVIEW

EXPERIENCEIn Person
TYPEBreakout
TRACKData Science and Machine Learning
INDUSTRYEnterprise Technology, Retail and CPG - Food
TECHNOLOGIESAI/Machine Learning, Developer Experience, MLFlow
SKILL LEVELIntermediate
DURATION40 min

Early in our journey, Databricks users at our company implemented workflows individually, leading to inconsistency and slow onboarding for teams. We have since built a tool (that we call “SciCLOps”) powered by Databricks Asset Bundles (DABs) to allow for better workflow reuse and standardization across data engineering, ML, and data science. DABs combined with Terraform, Github Actions, and a service catalog has enabled:

  • Users to request Databricks environments with no manual intervention by platform teams
  • Much simpler adoption of changes to workflow standards
  • Day two support for projects (such as automatic PAT rotation)
  • Faster promotion of models and science to production due to consistent workflows

SciCLOps has made significant impact to our onboarding speed and time to production. We will give a demo of our SciCLOps tool, as well as explain our journey in building it and the observations we have made along the way.

SESSION SPEAKERS

Connor Brown

/Cloud Engineer
84.51˚

Tara Enright

/Data Engineer
84.51˚