Architecting Data Warehouses for Large-Scale Deployments

In this course, you will develop an in-depth understanding of how to design, implement, and govern scalable machine learning systems that operate effectively at enterprise scale. The curriculum is organized into three experiential modules: developing distributed ML workflows with frameworks such as Apache SparkML and Ray, transitioning local ML development to distributed compute using tools like Pandas on Spark, and operationalizing and governing production models with Databricks’ MLOps ecosystem.

Through hands-on projects, you will construct end-to-end distributed ML pipelines using the SparkML workflow, applying Transformers, Estimators, and the fit/transform paradigm for both classification and regression tasks. You will version, compare, and manage experiments using MLflow 3.0 to ensure reproducibility and governance, capturing lineage between data, features, and model artifacts. Additionally, you will apply scalable Hyperparameter Optimization frameworks to improve model performance at scale.

The course concludes by demonstrating complete lifecycle management, from experimentation to production deployment, using Unity Catalog and Model Serving. You will learn to operationalize trained models, monitor their performance, and implement strong governance over models, features, and Delta assets within the Databricks environment.

In this course, you will learn how to integrate a Databricks App with Lakebase, Databricks' fully managed, cloud-native PostgreSQL OLTP database. This course assumes some familiarity with Databricks Apps and Lakebase and will therefore not cover the basic of those pieces; instead we'll focus on integrating the two to deliver an app that performs basic data collection and captures it in a Lakebase table. An overview of PostgreSQL and safety best practices will be emphasized.

Note: For SCORM lecture files, please ensure that you close the SCORM window after completing the content. Do not click the ‘Next Lesson’ button, as doing so may prevent the SCORM module from being marked as complete.

In this course, you will explore the Databricks Data Intelligence Platform from the perspective of platform architecture, specifically related to the platform foundation in lakehouse architecture. You will learn about the scope, vision, and capabilities of a lakehouse-based platform, examine how Databricks integrates with major cloud providers, and discover the key features of a successful lakehouse implementation through the well-architected lakehouse framework. The course also covers essential architectural principles, best practices, and data architecture strategies to help accelerate your organization’s data and AI initiatives using Databricks.