Get Started with Databricks for Data Engineering

The Databricks Accredited Platform Administrator is a 20-minute, unproctored accreditation exam that will test your knowledge about managing and administering the Databricks Lakehouse Platform using Unity Catalog. It will assess your ability to perform the fundamentals of administering the platform, administer identities, configure and secure the platform, manage compute resources, administer and secure data within the platform, and administer CI/CD tooling within the platform. Test takers that pass the exam will be awarded the Databricks Accredited Platform Administrator badge and certificate.

In this course, you will gain a comprehensive understanding of how to design, scale, and operationalize end-to-end feature engineering pipelines on the Databricks platform. The curriculum is structured across three progressive modules: mastering the fundamentals of Spark’s distributed execution and optimization, implementing scalable data ingestion with Auto Loader and declarative Lakeflow pipelines, and advancing to production-grade MLOps with the Databricks Feature Store.

You will engage in hands-on learning experiences such as debugging Spark performance with the Catalyst Optimizer and Spark UI, building robust Bronze-Silver-Gold medallion architectures with automated quality checks, and implementing scalable feature transformations using SparkML. The course culminates in deploying real-time feature serving through Online Feature Stores, defining FeatureSpecs with on-demand transformations, and applying governance and lineage tracking with Unity Catalog.

By the end of the course, you will be equipped to manage the complete feature lifecycle for machine learning at enterprise scale—designing pipelines that are performant, reliable, governed, and production-ready.

This course introduces users to the essential concepts and skills needed to build data pipelines using Lakeflow Declarative Pipelines in Databricks for incremental batch or streaming ingestion and processing through multiple streaming tables and materialized views. Designed for data engineers new to Lakeflow Declarative Pipelines, the course provides a comprehensive overview of core components such as incremental data processing, streaming tables, materialized views, and temporary views, highlighting their specific purposes and differences.

Topics covered include:

- Developing and debugging ETL pipelines with the multi-file editor in Lakeflow using SQL (with Python code examples provided)

- How Lakeflow Declarative Pipelines track data dependencies in a pipeline through the pipeline graph

- Configuring pipeline compute resources, data assets, trigger modes, and other advanced options

Next, the course introduces data quality expectations in Lakeflow, guiding users through the process of integrating expectations into pipelines to validate and enforce data integrity. Learners will then explore how to put a pipeline into production, including scheduling options, and enabling pipeline event logging to monitor pipeline performance and health.

Finally, the course covers how to implement Change Data Capture (CDC) using the AUTO CDC INTO syntax within Lakeflow Declarative Pipelines to manage slowly changing dimensions (SCD Type 1 and Type 2), preparing users to integrate CDC into their own pipelines.

Note: Databricks Academy is transitioning from video lectures to a more streamlined PDF format with slides and notes for all self-paced courses. Please note that demo videos will still be available in their original format. We would love to hear your thoughts on this change, so please share your feedback through the course survey at the end. Thank you for being a part of our learning community!

Languages Available: English | 日本語 | Português BR | 한국어