Skip to main content

Advanced Data Engineering with Databricks

This course serves as an appropriate entry point to learn Advanced Data Engineering with Databricks. 


Note: Databricks Academy is transitioning to a notebook-based format for classroom sessions within the Databricks environment, discontinuing the use of slide decks for lectures. You can access the lecture notebooks in the Vocareum lab environment.


Below, we describe each of the four, four-hour modules included in this course.

Advanced Techniques with Spark Declarative Pipelines

This course explores Databricks' Apache Spark Declarative Pipelines (SDP) for building production-grade streaming pipelines. You will learn advanced design patterns, robust data quality enforcement, and cross-platform integration essential for real-world lakehouse engineering.


Throughout the course, you will dive into modern data ingestion and processing techniques, mastering tools like Liquid Clustering for layout optimization and the Multiplex Streaming pattern for mixed-schema events. By the end of the modules, you will know how to confidently handle schema evolution, automate Change Data Capture (CDC), and ensure data integrity.


Databricks Data Privacy

Databricks Data Privacy teaches data engineers how to find, protect, delete and account for personal data on the Databricks Data Intelligence Platform. It treats privacy as a set of technical decisions and controls built on Unity Catalog and Delta Lake. The course separates privacy from security, governance and compliance, then organizes the work around an identify, protect, manage and verify lifecycle and a short decision model that learners apply to every technique.


Learners start by finding sensitive columns with Data Classification and recording how each should be treated with a governed tag. They protect those columns with tag-driven attribute-based access control (ABAC) policies and use manual column masks and row filters where a central policy doesn't fit. De-identification goes further with salted hashing, tokenization through a separately governed vault and column encryption, with the salt and key held in Databricks Secrets. For the right to erasure, learners track a request, find the subject's data with classification and lineage, delete and propagate the change with DELETE and MERGE, then purge the underlying files with REORG TABLE ... APPLY (PURGE) and retention-aware VACUUM. The course closes by combining lineage, audit logs, query history and Delta table history into an evidence trail that shows whether each control worked.


All data is synthetic. The demos follow one multinational retailer's data estate through the lifecycle and end each control with evidence that it worked, such as a masked query result, an empty erasure check or a table history entry. Two scenario-based labs reinforce the workflow. In the first, learners repair broken protection on a subsidiary's customer table. In the second, they finish a half-completed erasure request and assemble the evidence that it is complete. The course is technical and does not give legal advice.


Databricks Performance Optimization

Databricks Performance Optimization teaches data engineers and analysts how to diagnose, measure, and fix performance bottlenecks on the Databricks Data Intelligence Platform, and how to tie performance improvements back to cost. The course follows a measurement-first, platform-first workflow: let the platform's automatic optimizations do the heavy lifting, verify what was applied using Query Profile and system tables, then manually tune where needed.


Learners build a foundation for measuring performance with Query Profile, Performance Insights, and query history system tables. From there, they explore key optimization techniques, including data layout and self-tuning managed tables, liquid clustering, caching and intermediate results, shuffle, skew, spill, row explosions, driver performance, Python UDFs, serverless compute, Photon, and cost attribution.


Throughout the course, learners explore deliberately slow queries over synthetic retail data and see how different optimization techniques affect performance. They use evidence such as file pruning, task execution time, Photon coverage, and cost to evaluate improvements. Two scenario-based labs reinforce the optimization workflow from diagnosis through validation.


Automated Deployment with Declarative Automation Bundles

This course provides a comprehensive review of DevOps principles and their application to Databricks projects. It begins with an overview of core DevOps, DataOps, continuous integration (CI), continuous deployment (CD), and testing, and explores how these principles can be applied to data engineering pipelines.


The course then focuses on continuous deployment within the CI/CD process, examining tools like the Databricks REST API, SDK, and CLI for project deployment. You will learn about Declarative Automation Bundles (DABs) and how they fit into the CI/CD process. You’ll dive into their key components, folder structure, and how they streamline deployment across various target environments in Databricks. You will also learn how to add variables, modify, validate, deploy, and execute Declarative Automation Bundles for multiple environments with different configurations using the Databricks CLI.


Finally, the course introduces Visual Studio Code as an Interactive Development Environment (IDE) for building, testing, and deploying Declarative Automation Bundles locally, optimizing your development process. The course concludes with an introduction to automating deployment pipelines using GitHub Actions to enhance the CI/CD workflow with Declarative Automation Bundles.

By the end of this course, you will be equipped to automate Databricks project deployments with Declarative Automation Bundles, improving efficiency through DevOps practices.


Languages Available: English | 日本語 | Português BR | 한국어

Skill Level
Professional
Duration
16h
Prerequisites

Prerequisites

• Spark Declarative Pipelines — Completion of the "Build Data Pipelines with Apache Spark Declarative Pipelines" course, or familiarity with CREATE OR REFRESH STREAMING TABLE, CONSTRAINTS, and the Pipelines UI

• Delta Lake Fundamentals — Understanding of Delta tables and how Delta manages data files and transaction logs

• Streaming Concepts — Knowledge of micro-batch streaming, checkpointing, and event-time processing in SDP

• SQL Proficiency — Ability to read and write SQL, including SELECT, JOIN, MERGE, CASE WHEN, and common aggregate functions

• Python in Databricks Notebooks — Comfort with reading and running Python code in Databricks notebooks

• Unity Catalog Basics — Understanding of catalogs, schemas, tables, and volumes in Unity Catalog

• Ability to perform basic code development tasks using the Databricks Data Engineering and Data Science workspace (create clusters, run code in notebooks, use basic notebook operations, import repos from git, etc.)

• Intermediate programming experience with PySpark

• Extract data from a variety of file formats and data sources

• Apply a number of common transformations to clean data

• Reshape and manipulate complex data using advanced built-in functions

• Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions, etc.) 

• Beginner experience configuring and scheduling data pipelines using the Apache Spark Declarative Pipelines UI 

• Beginner experience defining Apache Spark Declarative Pipelines using PySpark 

• Ingest and process data using Auto Loader and PySpark syntax

• Process Change Data Capture feeds with APPLY CHANGES INTO syntax

• Review pipeline event logs and results to troubleshoot Declarative Pipeline syntax

• Strong knowledge of the Databricks platform, including experience with Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture, Unity Catalog, Apache Declarative Pipelines, and Workflows. In particular, knowledge of leveraging Expectations with Apache Declarative Pipelines. 

• Experience in data ingestion and transformation, with proficiency in PySpark for data processing and DataFrame manipulation. Candidates should also have experience writing intermediate-level SQL queries for data analysis and transformation.

• Proficiency in Python programming, including the ability to design and implement functions and classes, and experience with creating, importing, and utilizing Python packages.

• Familiarity with DevOps practices, particularly continuous integration and continuous delivery/deployment (CI/CD) principles.

• A basic understanding of Git version control.

• Prerequisite course DevOps Essentials for Data Engineering Course

• Navigating the Databricks workspace and running notebooks

• Writing intermediate SQL, including joins and aggregations

• Writing PySpark using DataFrame transformations and actions

• Working with Delta Lake tables and core Unity Catalog concepts, including catalogs, schemas, tables, and managed tables

• Understanding core Spark execution concepts, including partitions, transformations, and lazy evaluation

• Basic familiarity with Delta Lake tables, including table history and time travel

• Familiarity with layered data pipelines such as the medallion (bronze, silver, gold) architecture

Outline

Advanced Techniques with Apache Spark Declarative Pipelines

• Introduction to Multi Flows, Expectation and Liquid Clustering in SDP

• Demo: Multi Flow SDP with Liquid Clustering and Data Quality

• Introduction to Multiplex Streaming, Delta Sinks and  Iceberg Reads

• Demo: Multiplex Streaming SDP with Delta Sinks and Iceberg Reads

• Change Data Capture (CDC) Review

• Demo: Automating SCD Type 2 with AUTO CDC in Apache Spark Declarative Pipelines

• Advanced Data Quality Checks and Expectations in SDP

• Demo: Advanced Data Quality Checks and Expectation in SDP

• Lab - Building Multi-Source Ecommerce Pipeline with SDP


Databricks Data Privacy

• Demo:  REQUIRED - Project Overview and Setup

• Privacy, Security, Governance, Compliance

• Demo: Discover and Classify Sensitive Data

• Demo: ABAC and Manual Masks

• Lab: Discover, Tag and Protect PII

• Demo: De-identification and Key Management

• Demo: Right to Erasure - Delete, Propagate, Purge

• Demo: Privacy Evidence with System Tables

• Lab: Fulfil and Verify an Erasure Request


Databricks Performance Optimization

• The Performance Landscape

• Demo: Measure First - Query Profiles and Benchmarks

• Demo: Data Layout

• Demo: Liquid Clustering vs Partitioning

• Demo: Caching and Intermediate Results

• Lab 1: Layout

• Demo: Shuffle, Skew, Spill and Explosion

• Demo: Python, Driver and UDF Pitfalls

• Serverless and Photon

• Demo: Cost & Cost Attribution

• Lab 2: Cost


Automated Deployment with Declarative Automation Bundles

• DevOps Review

• Continuous Integration and Continuous Deployment/Delivery (CI/CD) Review

• Demo: Course Setup and Authentication

• Deploying Databricks Projects

• Introduction to Declarative Automation Bundles (DABs)

• Demo: Deploying a Simple DAB

• Lab: Deploying a Simple DAB

• Variable Substitutions in DABs

• Demo: Deploying a DAB to Multiple Environments

• Lab: Deploy a DAB to Multiple Environments

• DAB Project Templates Overview

• Lab: Use a Databricks Default DAB Template

• CI/CD Project Overview with DABs

• Demo: Continuous Integration and Continuous Deployment with DABs

• Lab: Adding ML to Engineering Workflows with DABs

• Developing Locally with Visual Studio Code (VSCode)

• Demo: Using VSCode with Databricks

• CI/CD Best Practices for Data Engineering

• Next Steps: Automated Deployment with GitHub Actions

Upcoming Public Classes

Date
Time
Your Local Time
Language
Price
Nov 17 - 18
10 AM - 06 PM (Asia/Kolkata)
-
English
$1500.00
Nov 17 - 20
01 PM - 05 PM (Europe/Paris)
-
English
$1500.00
Dec 08 - 09
09 AM - 05 PM (Europe/Paris)
-
English
$1500.00
Dec 15 - 16
09 AM - 05 PM (Europe/Paris)
-
English
$1500.00
Dec 15 - 16
09 AM - 05 PM (Europe/London)
-
English
$1500.00
Jan 05 - 08
09 AM - 01 PM (Asia/Kolkata)
-
English
$1500.00
Jan 05 - 06
09 AM - 05 PM (Europe/Paris)
-
English
$1500.00
Jan 19 - 20
10 AM - 06 PM (Asia/Singapore)
-
English
$1500.00
Jan 19 - 20
10 AM - 06 PM (Asia/Kolkata)
-
English
$1500.00
Jan 19 - 20
09 AM - 05 PM (America/New_York)
-
English
$1500.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Data Engineer

Databricks Performance Optimization - Mandarin Chinese

Databricks Performance Optimization 课程教导数据工程师和分析师如何在 Databricks Data Intelligence Platform 上诊断、衡量和修复性能瓶颈,以及如何将性能提升与成本关联起来。本课程遵循"先衡量、先平台"的工作流:让平台的自动优化功能承担繁重工作,通过 Query Profile 和系统表验证已应用的优化,再在必要时进行手动调优。

学员将以 Query Profile、Performance Insights 和查询历史记录系统表为基础,学习如何衡量性能。在此基础上,他们将探索关键优化技术,包括数据布局与自调优托管表、liquid clustering、缓存与中间结果、shuffle、数据倾斜、溢出、行爆炸、驱动程序性能、Python UDF、serverless compute、Photon 以及成本归因。

在整个课程中,学员将针对合成零售数据中刻意设计的慢查询进行探索,观察不同优化技术对性能的影响。他们将利用文件裁剪、任务执行时间、Photon 覆盖率和成本等依据来评估优化效果。两个基于场景的实验室将强化从诊断到验证的完整优化工作流。

注意:Databricks Academy 正在将 Databricks 环境中的课堂教学转为基于 notebook 的形式,不再使用幻灯片进行授课。您可以在 Vocareum 实验环境中访问课程 notebook。

Languages Available: English | 日本語 | Português BR | 한국어 | Français | Español | 普通话

Paid
4h
Lab
instructor-led
Professional

Databricks Data Privacy - Mandarin Chinese

Databricks 数据隐私课程面向数据工程师,讲解如何在 Databricks Data Intelligence Platform 上发现、保护、删除个人数据并对其负责。课程将隐私视为建立在 Unity Catalog 和 Delta Lake 之上的一组技术决策与控制。课程把隐私与安全、治理和合规区分开来,然后围绕识别、保护、管理、验证的生命周期,以及学习者可应用于每项技术的简短决策模型来组织工作。

学习者首先使用 Data Classification 查找敏感列,并用治理标签记录每列应如何处理。他们用基于标签的属性访问控制(ABAC)策略保护这些列,并在中央策略不适用时使用手动列掩码和行过滤器。去标识化进一步涵盖加盐哈希、通过单独治理的保险库进行令牌化,以及列加密,盐值和密钥保存在 Databricks Secrets 中。对于删除权,学习者跟踪请求,借助分类和血缘查找主体数据,用 DELETE 和 MERGE 删除并传播变更,再用 REORG TABLE ... APPLY (PURGE) 和考虑保留期的 VACUUM 清除底层文件。课程最后将血缘、审计日志、查询历史和 Delta 表历史组合成证据链,证明各项控制是否生效。

所有数据均为合成数据。演示沿着一家跨国零售商的数据资产走完生命周期,并以已掩码的查询结果、空的删除检查或表历史记录等证据结束每项控制。两个基于场景的实验强化该工作流。第一个实验中,学习者修复子公司客户表上损坏的保护。第二个实验中,他们完成一份只做了一半的删除请求,并汇总完成证据。本课程为技术课程,不提供法律建议。

注意: Databricks Academy 正在将 Databricks 环境中的课堂教学过渡为基于 Notebook 的形式,不再使用幻灯片进行讲座。您可以在 Vocareum 实验室环境中访问讲座 Notebook。

Paid
4h
Lab
instructor-led
Professional

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.