Data Management and Governance with Unity Catalog

In this course, you will develop an in-depth understanding of how to design, implement, and govern scalable machine learning systems that operate effectively at enterprise scale. The curriculum is organized into three experiential modules: developing distributed ML workflows with frameworks such as Apache SparkML and Ray, transitioning local ML development to distributed compute using tools like Pandas on Spark, and operationalizing and governing production models with Databricks’ MLOps ecosystem.

Through hands-on projects, you will construct end-to-end distributed ML pipelines using the SparkML workflow, applying Transformers, Estimators, and the fit/transform paradigm for both classification and regression tasks. You will version, compare, and manage experiments using MLflow 3.0 to ensure reproducibility and governance, capturing lineage between data, features, and model artifacts. Additionally, you will apply scalable Hyperparameter Optimization frameworks to improve model performance at scale.

The course concludes by demonstrating complete lifecycle management, from experimentation to production deployment, using Unity Catalog and Model Serving. You will learn to operationalize trained models, monitor their performance, and implement strong governance over models, features, and Delta assets within the Databricks environment.

In this course, you will explore how Unity Catalog enables secure, centralized data governance and fine-grained access control on Databricks. You will learn about table and volume types, catalog and schema configuration, group-based access management, and strategies for migrating existing access controls into Unity Catalog. The course also explains how to design and apply fine-grained controls such as row-level security, column masking, and attribute-based access control, how to combine these mechanisms across data and AI assets, and how to align them with broader governance requirements for compliant, scalable access management.

Languages Available: English | 日本語 | 한국어

This course is meant to provide guidance on best practices for executing a successful migration from Cloud Data Warehouses (CDWs) - e.g. Amazon Redshift, Snowflake, and Synapse Analytics - to Databricks. It covers a wide range of migration processes starting from initial discovery all the way through to delivery.