Monitoring and Optimizing Apache Spark Workloads on Databricks

In this course, you will gain a comprehensive understanding of how to design, scale, and operationalize end-to-end feature engineering pipelines on the Databricks platform. The curriculum is structured across three progressive modules: mastering the fundamentals of Spark’s distributed execution and optimization, implementing scalable data ingestion with Auto Loader and declarative Lakeflow pipelines, and advancing to production-grade MLOps with the Databricks Feature Store.

You will engage in hands-on learning experiences such as debugging Spark performance with the Catalyst Optimizer and Spark UI, building robust Bronze-Silver-Gold medallion architectures with automated quality checks, and implementing scalable feature transformations using SparkML. The course culminates in deploying real-time feature serving through Online Feature Stores, defining FeatureSpecs with on-demand transformations, and applying governance and lineage tracking with Unity Catalog.

By the end of the course, you will be equipped to manage the complete feature lifecycle for machine learning at enterprise scale—designing pipelines that are performant, reliable, governed, and production-ready.

This course introduces users to the essential concepts and skills needed to build data pipelines using Lakeflow Declarative Pipelines in Databricks for incremental batch or streaming ingestion and processing through multiple streaming tables and materialized views. Designed for data engineers new to Lakeflow Declarative Pipelines, the course provides a comprehensive overview of core components such as incremental data processing, streaming tables, materialized views, and temporary views, highlighting their specific purposes and differences.

Topics covered include:

- Developing and debugging ETL pipelines with the multi-file editor in Lakeflow using SQL (with Python code examples provided)

- How Lakeflow Declarative Pipelines track data dependencies in a pipeline through the pipeline graph

- Configuring pipeline compute resources, data assets, trigger modes, and other advanced options

Next, the course introduces data quality expectations in Lakeflow, guiding users through the process of integrating expectations into pipelines to validate and enforce data integrity. Learners will then explore how to put a pipeline into production, including scheduling options, and enabling pipeline event logging to monitor pipeline performance and health.

Finally, the course covers how to implement Change Data Capture (CDC) using the AUTO CDC INTO syntax within Lakeflow Declarative Pipelines to manage slowly changing dimensions (SCD Type 1 and Type 2), preparing users to integrate CDC into their own pipelines.

Note: Databricks Academy is transitioning from video lectures to a more streamlined PDF format with slides and notes for all self-paced courses. Please note that demo videos will still be available in their original format. We would love to hear your thoughts on this change, so please share your feedback through the course survey at the end. Thank you for being a part of our learning community!

Languages Available: English | 日本語 | Português BR | 한국어

Welcome to the AWS Databricks Platform Architect Accreditation

This is a 20-minute assessment that will test your knowledge about fundamental concepts related to Databricks platform administration on AWS. Questions will assess your knowledge about cloud-specific elements of the platform, including integration with managed services and security best practices.

After successfully completing this assessment, you will be awarded an AWS Databricks Platform Architect badge.

Courses to help you prepare for the Accreditation:  AWS Databricks Platform Administration Fundamentals.AWS Databricks Networking and Security Fundamentals.AWS Databricks Cloud Integrations.Learning path This course is part of AWS Databricks Platform Architect Learning Plan.

Proof of completion Upon an 80% pass rate, you will receive a badge from Databricks.