Advanced Techniques with Spark Declarative Pipeline

This course explores Databricks' Lakeflow Spark Declarative Pipelines (SDP) for building production-grade streaming pipelines. You will learn advanced design patterns, robust data quality enforcement, and cross-platform integration essential for real-world lakehouse engineering.

Throughout the course, you will dive into modern data ingestion and processing techniques, mastering tools like Liquid Clustering for layout optimization and the Multiplex Streaming pattern for mixed-schema events. By the end of the modules, you will know how to confidently handle schema evolution, automate Change Data Capture (CDC), and ensure data integrity.

Through lectures and hands-on demos, you will:

• Build multi-flow pipelines to ingest multi-source data into a unified Bronze table.

• Apply Liquid Clustering and Data Quality Expectations across Silver and Gold layers.

• Implement the Multiplex pattern with Iceberg UniForm for cross-platform data access.

• Automate SCD Type 2 history tracking using AUTO CDC INTO.

• Design zero-data-loss quarantine pipelines to audit and manage invalid records.

Note:

1. This course is the first in the 'Advanced Data Engineering with Databricks' series.

2. For SCORM lecture files, please ensure that you close the SCORM window after completing the content. Do not click the ‘Next Lesson’ button, as doing so may prevent the SCORM module from being marked as complete.

Skill Level

Professional

Duration

Prerequisites

The content was developed for participants with these skills/knowledge/abilities:

• Spark Declarative Pipelines — Completion of the "Build Data Pipelines with Lakeflow Spark Declarative Pipelines" course, or familiarity with CREATE OR REFRESH STREAMING TABLE, CONSTRAINTS, and the Pipelines UI

• Delta Lake Fundamentals — Understanding of Delta tables and how Delta manages data files and transaction logs

• Streaming Concepts — Knowledge of micro-batch streaming, checkpointing, and event-time processing in SDP

• SQL Proficiency — Ability to read and write SQL, including SELECT, JOIN, MERGE, CASE WHEN, and common aggregate functions

• Python in Databricks Notebooks — Comfort with reading and running Python code in Databricks notebooks

• Unity Catalog Basics — Understanding of catalogs, schemas, tables, and volumes in Unity Catalog

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Customer registration Partner registration

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Platform Administrator

Data Governance at Scale

In this course, you will learn how to implement data governance at scale on Databricks using Unity Catalog, with a focus on attribute-based access control, observability, and federated sharing. You will configure ABAC with governed tags, migrate from legacy fine-grained controls, enable and use system tables for audit and cost monitoring, deploy Lakehouse Monitoring for data and model quality, interpret lineage for impact and compliance, and apply federated governance and Delta Sharing patterns for secure cross-cloud collaboration.

Paid & Subscription

Lab

Associate

Data Warehousing Practitioner

Building Semantic Models in Databricks with UC Metric Views

In this course, you’ll learn how to empower your data consumers to get validated business insights with less burden on you and the rest of your enterprise's data and AI team. Through building a semantic layer on top of your gold layer, you'll offer decision makers and business analysts the opportunity to explore key business performance indicators in novel ways, while still maintaining efficient data operations and enterprise-vetted metrics.

SQL Analytics on Databricks

In this course, you'll learn how to effectively use Databricks for data analytics, with a specific focus on Databricks SQL. As a Databricks Data Analyst, your responsibilities will include finding relevant data, analyzing it for potential applications, and transforming it into formats that provide valuable business insights. You will also understand your role in managing data objects and how to manipulate them within the Databricks Data Intelligence Platform, using tools such as Notebooks, the SQL Editor, and Databricks SQL. Additionally, you will learn about the importance of Unity Catalog in managing data assets and the overall platform. Finally, the course will provide an overview of how Databricks facilitates performance optimization and teach you how to access Query Insights to understand the processes occurring behind the scenes when executing SQL analytics on Databricks.

Note: Databricks Academy is transitioning from video lectures to a more streamlined PDF format with slides and notes for all self-paced courses. Please note that demo videos will still be available in their original format. We would love to hear your thoughts on this change, so please share your feedback through the course survey at the end. Thank you for being a part of our learning community!

Languages Available: English | 日本語 | Português BR | 한국어

Paid & Subscription

Lab

Associate