Governing Lakebase: Unity Catalog + Postgres

In this course, you will learn to design and implement machine learning workflows that scale from a single machine to distributed compute on Databricks. The curriculum is organized into three modules: foundations of model development (when and why to distribute, and how Spark executes work), scalable model development with Spark ML and MLflow, and distributed model training with Ray and PyTorch — closing with a framework for choosing the right tool for a given workload.

Through hands-on demos and labs, you will build distributed Spark ML pipelines using Transformers, Estimators, and the fit/transform paradigm; scale existing pandas code with the Pandas API on Spark; and track, compare, and govern experiments with MLflow 3 and Unity Catalog. You will then train models across a cluster with Ray Train and PyTorch (TorchDistributor and DeepSpeed), and learn to match each workload to the right framework based on data size, model size, hardware, and cost.

The course concludes with a capstone that takes a problem end to end — engineering features, training a model at scale, tracking it with MLflow, and registering it in Unity Catalog with a governance alias. (Production deployment, serving, and monitoring are covered in the separate Advanced MLOps course.)

By the end, you will be able to choose, build, and justify distributed, reproducible, and governed model-development workflows on Databricks.

Note: For SCORM lecture files, please ensure that you close the SCORM window after completing the content. Do not click the ‘Next Lesson’ button, as doing so may prevent the SCORM module from being marked as complete.

Establishes what Lakebase is and why it exists as fully managed Postgres on the lakehouse with unified governance. Built on the lecture-demo pattern (teach a concept, immediately run it), the course anchors four canonical use cases — each proven with a named customer story — and a three-signal decision framework for choosing between Lakebase, Delta, and the Lakehouse. It is the entry point for the Lakebase pathway; the use cases and decision tree introduced here are expanded by every later course in the series.

本课程全面介绍了 DevOps 原则及其在 Databricks 项目中的应用。课程首先概述了核心 DevOps、DataOps、持续集成（CI）、持续部署（CD）和测试的基本概念，并探讨了如何将这些原则应用于数据工程管道。

课程随后聚焦于 CI/CD 流程中的持续部署，介绍用于项目部署的工具，包括 Databricks REST API、SDK 和 CLI。您将了解声明式自动化包（DAB）及其在 CI/CD 流程中的作用，深入学习其关键组件、文件夹结构，以及它们如何简化在 Databricks 中跨各种目标环境的部署流程。您还将学习如何使用 Databricks CLI 为多个具有不同配置的环境添加变量、修改、验证、部署和执行声明式自动化包。

最后，课程介绍如何将 Visual Studio Code 作为交互式开发环境（IDE），用于在本地构建、测试和部署声明式自动化包，从而优化开发流程。课程最后介绍如何使用 GitHub Actions 自动化部署管道，以增强声明式自动化包的 CI/CD 工作流。

完成本课程后，您将具备使用声明式自动化包自动化 Databricks 项目部署的能力，并能够通过 DevOps 实践提升工作效率。

注意：

Databricks Academy 正在将所有自定进度课程从视频讲座转为更简洁的 PDF 格式，其中包含幻灯片和讲义。请注意，演示视频仍将以原始格式提供。我们非常希望听到您对这一变化的看法，因此请通过课程末尾的课程调查分享您的反馈。感谢您成为我们学习社区的一员！本课程是 ‘Advanced Data Engineering with Databricks’ 系列的第四门课程。