Build Data Pipelines with Apache Spark Declarative Pipelines

This course introduces users to the essential concepts and skills needed to build data pipelines using Apache Spark™ Declarative Pipelines (SDP) in Databricks for incremental batch or streaming ingestion and processing through multiple streaming tables and materialized views. Designed for data engineers new to Spark Declarative Pipelines, the course provides a comprehensive overview of core components such as incremental data processing, streaming tables, materialized views, and temporary views, highlighting their specific purposes and differences.

Topics covered include:

• Developing and debugging ETL pipelines with the multi-file editor in Spark Declarative Pipelines using SQL (with Python code examples provided)

• How Spark Declarative Pipelines track data dependencies in a pipeline through the pipeline graph

• Configuring pipeline compute resources, data assets, trigger modes, and other advanced options

Next, the course introduces data quality expectations in Spark Declarative Pipelines, guiding users through the process of integrating expectations into pipelines to validate and enforce data integrity. Learners will then explore how to put a pipeline into production, including scheduling options, and enabling pipeline event logging to monitor pipeline performance and health.

Finally, the course covers how to implement Change Data Capture (CDC) using the AUTO CDC INTO syntax within Spark Declarative Pipelines to manage slowly changing dimensions (SCD Type 1 and Type 2), preparing users to integrate CDC into their own pipelines.

Note: For SCORM lecture files, please ensure that you close the SCORM window after completing the content. Do not click the ‘Next Lesson’ button, as doing so may prevent the SCORM module from being marked as complete.

Skill Level

Associate

Duration

Prerequisites

In this course, the content was developed for participants with these skills/knowledge/abilities:

• Basic understanding of the Databricks Data Intelligence platform, including Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture, Lakeflow Jobs and Unity Catalog.

• Experience ingesting raw data into Delta tables, including using the read_files SQL function to load formats like CSV, JSON, TXT, and Parquet.

• Proficiency in transforming data using SQL, including writing intermediate-level queries and a basic understanding of SQL joins.

• Understanding of ETL concepts, and batch/streaming workflows.

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Customer registration Partner registration

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

AI Engineer

Get Started with AI Agents on Databricks - Mandarin Chinese

本课程介绍 AI 智能体的概念、其在现代 AI 应用中的作用，以及如何在 Databricks 平台上构建 AI 智能体应用程序。您将学习 AI 智能体的基本原理、它们与传统 AI 系统的区别，并浏览其核心键组件。通过互动演示和实践实验，您将了解如何使用 Databricks AI 和 Agent Bricks 在 Databricks 上构建、部署和评估 AI 智能体。在本课程结束时，您将掌握如何使用 Databricks Data + AI Platform 创建和部署 AI 智能体。

注意：Databricks Academy 正在将 Databricks 环境中的课堂课程转为基于 notebook 的形式，并停止在讲座中使用幻灯片。您可以在 Vocareum 实验环境中访问课程 notebook。

Get Started with AI Agents on Databricks

This course is an introduction to AI agents, their role in modern AI applications, and how to build AI agent applications on the Databricks platform. You'll learn about the principles of AI agents, how they differ from traditional AI systems, and explore their key components. Through interactive demos and hands-on labs, you'll see how to build, deploy, and evaluate AI agents on Databricks using Databricks AI and Agent Bricks. By the end of this course, you'll understand how to create and deploy AI agents using the Databricks Data + AI Platform.

Paid & Subscription

Lab

Onboarding

Model Development at Scale

In this course, you will learn to design and implement machine learning workflows that scale from a single machine to distributed compute on Databricks. The curriculum is organized into three modules: foundations of model development (when and why to distribute, and how Spark executes work), scalable model development with Spark ML and MLflow, and distributed model training with Ray and PyTorch — closing with a framework for choosing the right tool for a given workload.

Through hands-on demos and labs, you will build distributed Spark ML pipelines using Transformers, Estimators, and the fit/transform paradigm; scale existing pandas code with the Pandas API on Spark; and track, compare, and govern experiments with MLflow 3 and Unity Catalog. You will then train models across a cluster with Ray Train and PyTorch (TorchDistributor and DeepSpeed), and learn to match each workload to the right framework based on data size, model size, hardware, and cost.

The course concludes with a capstone that takes a problem end to end — engineering features, training a model at scale, tracking it with MLflow, and registering it in Unity Catalog with a governance alias. (Production deployment, serving, and monitoring are covered in the separate Advanced MLOps course.)

By the end, you will be able to choose, build, and justify distributed, reproducible, and governed model-development workflows on Databricks.

Paid & Subscription

Lab

Professional