Advanced Data Engineering with Databricks
In this course, students will build upon their existing knowledge of Apache Spark, Structured Streaming, and Delta Lake to unlock the full potential of the data lakehouse by utilizing the suite of tools provided by Databricks. This course places a heavy emphasis on designs favoring incremental data processing, enabling systems optimized to continuously ingest and analyze ever-growing data. By designing workloads that leverage built-in platform optimizations, data engineers can reduce the burden of code maintenance and on-call emergencies, and quickly adapt production code to new demands with minimal refactoring or downtime.
The topics in this course should be mastered prior to attempting the Databricks Certified Data Engineer Professional exam.
Languages Available: English | 日本語 | Português BR | 한국어
- Experience using PySpark APIs to perform advanced data transformations
- Familiarity implementing classes with Python
- Experience using SQL in production data warehouse or data lake implementations
- Experience working in Databricks notebooks and configuring clusters
- Familiarity with creating and manipulating data in Delta Lake tables with SQL
The prerequisites listed above can be learned by taking the Data Engineering with Databricks and Apache Spark Programming with Databricks instructor-led courses (can be taken in either order) and validated by passing the Databricks Certified Data Engineer Associate and Databricks Certified Associate Developer for Apache Spark certification exams.
Outline
Incremental Processing with Spark Structured Streaming and Delta Lake
- Streaming Data Concepts
- Introduction to Structured Streaming
- Aggregations, Time Windows, Watermarks
- Delta Live Tables Review
- Auto Loader
Streaming ETL Patterns with DLT
- Data Ingestion Patterns
- Data Quality Enforcement Patterns
- Data Modeling
- Streaming Joins and Statefulness
Data Privacy Patterns
- Store Data Securely
- Streaming Data and CDF
- Deleting Data in Databricks
Performance Optimization with Spark and Delta Lake
- Spark Architecture
- Designing the Foundation
- Introduction of Spark UI
- Fine-Tuning - Choosing the Right Cluster
- Code Optimization
- Shuffles
- Spill
- Skew
- Serialization
SWE Practices for Delta Live Tables Pipelines
Automate Production Workflows
- Introduction to REST API and CLI
- Deploy Batch and Streaming Jobs
- Working with Terraform
Upcoming Public Classes
Date | Time | Language | Price |
|---|---|---|---|
Oct 02 - 03 | 09 AM - 05 PM (Europe/Paris) | English | $1500.00 |
Oct 07 - 10 | 09 AM - 01 PM (Asia/Kolkata) | English | $1500.00 |
Oct 07 - 08 | 09 AM - 05 PM (America/New_York) | English | $1500.00 |
Oct 07 - 10 | 02 PM - 06 PM (America/New_York) | English | $1500.00 |
Oct 08 - 09 | 09 AM - 05 PM (Europe/London) | English | $1500.00 |
Oct 14 - 17 | 09 AM - 01 PM (Asia/Singapore) | English | $1500.00 |
Oct 14 | 09 AM - 01 PM (Asia/Kolkata) | English | $1500.00 |
Oct 21 - 24 | 01 PM - 05 PM (America/Los_Angeles) | English | $1500.00 |
Oct 30 - 31 | 09 AM - 05 PM (Europe/Paris) | English | $1500.00 |
Nov 04 - 07 | 01 PM - 05 PM (Australia/Sydney) | English | $1500.00 |
Nov 04 - 05 | 09 AM - 05 PM (Europe/London) | English | $1500.00 |
Nov 07 - 08 | 09 AM - 05 PM (America/New_York) | English | $1500.00 |
Nov 12 - 14 | 09 AM - 01 PM (Asia/Kolkata) | English | $1500.00 |
Nov 12 - 15 | 09 AM - 01 PM (Asia/Kolkata) | English | $1500.00 |
Nov 12 - 15 | 01 PM - 05 PM (America/Los_Angeles) | English | $1500.00 |
Nov 27 - 28 | 09 AM - 05 PM (Europe/Paris) | English | $1500.00 |
Dec 02 - 05 | 11 AM - 03 PM (Asia/Singapore) | English | $1500.00 |
Dec 02 - 03 | 09 AM - 05 PM (Europe/London) | English | $1500.00 |
Dec 02 - 03 | 09 AM - 05 PM (America/New_York) | English | $1500.00 |
Dec 09 - 12 | 01 PM - 05 PM (America/Los_Angeles) | English | $1500.00 |
Dec 16 - 19 | 09 AM - 01 PM (Europe/Paris) | English | $1500.00 |
Dec 17 - 20 | 09 AM - 01 PM (Asia/Kolkata) | English | $1500.00 |
Jan 06 - 09 | 01 PM - 05 PM (Australia/Sydney) | English | $1500.00 |
Jan 07 - 08 | 09 AM - 05 PM (Europe/Paris) | English | $1500.00 |
Jan 13 - 16 | 01 PM - 05 PM (America/Los_Angeles) | English | $1500.00 |
Jan 20 - 23 | 01 PM - 05 PM (Asia/Singapore) | English | $1500.00 |
Jan 27 - 28 | 09 AM - 05 PM (Europe/London) | English | $1500.00 |
Jan 27 - 30 | 01 PM - 05 PM (America/Chicago) | English | $1500.00 |
Public Class Registration
If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.
Private Class Request
If your company is interested in private training, please submit a request.
Registration options
Databricks has a delivery method for wherever you are on your learning journey
Self-Paced
Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos
Register now
Instructor-Led
Public and private courses taught by expert instructors across half-day to two-day courses
Register now
Blended Learning
Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase
Purchase now
Skills@Scale
Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details
Upcoming Public Classes
Career Workshop/
March 20
