Skip to main content

Data Engineering with Databricks

This is an introductory course that serves as an appropriate entry point to learn Data Engineering with Databricks. 

Below, we describe each of the four, four-hour modules included in this course.


1. Data Ingestion with Lakeflow Connect

This course provides a comprehensive introduction to Lakeflow Connect as a scalable and simplified solution for ingesting data into Databricks from a variety of data sources. You will begin by exploring the different types of connectors within Lakeflow Connect (Standard and Managed), learn about various ingestion techniques, including batch, incremental batch, and streaming, and then review the key benefits of Delta tables and the Medallion architecture.


From there, you will gain practical skills to efficiently ingest data from cloud object storage using Lakeflow Connect Standard Connectors with methods such as CREATE TABLE AS (CTAS), COPY INTO, and Auto Loader, along with the benefits and considerations of each approach. You will then learn how to append metadata columns to your bronze level tables during ingestion into the Databricks data intelligence platform. This is followed by working with the rescued data column, which handles records that don’t match the schema of your bronze table, including strategies for managing this rescued data.


The course also introduces techniques for ingesting and flattening semi-structured JSON data, as well as enterprise-grade data ingestion using Lakeflow Connect Managed Connectors.


Finally, learners will explore alternative ingestion strategies, including MERGE INTO operations and leveraging the Databricks Marketplace, equipping you with foundational knowledge to support modern data engineering ingestion. 


2. Deploy Workloads with Lakeflow Jobs

Deploy Workloads with Lakeflow Jobs course teaches how to orchestrate and automate data, analytics, and AI workflows using Lakeflow Jobs. You will learn to make robust, production-ready pipelines with flexible scheduling, advanced orchestration, and best practices for reliability and efficiency-all natively integrated within the Databricks Data intelligence Platform. Prior experience with Databricks, Python and SQL is recommended.


3. Build Data Pipelines with DLT

This course introduces users to the essential concepts and skills needed to build data pipelines using DLT in Databricks for incremental ingesting and processing through multiple streaming tables and materialized views. Designed for data engineers new to DLT, the course provides a comprehensive overview of core components such as incremental data processing, streaming tables, materialized views, and temporary views, reviewing their specific purposes and differences. 


Topics covered include:

Developing and debuging ETL pipelines with the multi-file editor in DLT using SQL (with Python code examples provided)

How DLT tracks data dependencies in pipelines within the pipeline graph

Configuring pipeline compute resources, data assets, trigger modes, and other advanced options


Next, the course introduces data quality expectations in DLT, guiding users through the process of integrating expectations into pipelines to validate and enforce data integrity. Then learners will then explore how to put a pipeline into production, including scheduling options, and gain basic knowledge in analyzing pipeline event logs to monitor a pipeline.


Finally, the course covers the implementation of Change Data Capture (CDC) using the APPLY CHANGES INTO syntax to manage slowly changing dimensions (SCD Type 1 and Type 2), preparing users to integrate CDC into their own pipelines.


4. Data Management and Governance with Unity Catalog

In this course, you'll learn about data management and governance using Databricks Unity Catalog. It covers foundational concepts of data governance, complexities in managing data lakes, Unity Catalog's architecture, security, administration, and advanced topics like fine-grained access control, data segregation, and privilege management.


* This course seeks to prepare students to complete the Associate Data Engineering certification exam, and provides the requisite knowledge to take the course Advanced Data Engineering with Databricks.

Skill Level
Associate
Duration
16h
Prerequisites
1. Data Ingestion with Lakeflow Connect

Basic understanding of the Databricks Data Intelligence platform, including Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture and Unity Catalog.

Experience working with various file formats (e.g., Parquet, CSV, JSON, TXT).

Proficiency in SQL and Python.

Familiarity with running code in Databricks notebooks.


2. Deploy Workloads with Lakeflow Jobs

⇾ Beginner familiarity with basic cloud concepts (virtual machines, object storage, identity management)

⇾ Ability to perform basic code development tasks (create compute, run code in notebooks, use basic notebook operations, import repos from git, etc.)

⇾ Intermediate familiarity with basic SQL concepts (CREATE, SELECT, INSERT, UPDATE, DELETE, WHILE, GROUP BY, JOIN, etc.)


3. Build Data Pipelines with DLT

⇾ Basic understanding of the Databricks Data Intelligence platform, including Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture and Unity Catalog.

⇾ Experience ingesting raw data into Delta tables, including using the read_files SQL function to load formats like CSV, JSON, TXT, and Parquet.

⇾ Proficiency in transforming data using SQL, including writing intermediate-level queries and a basic understanding of SQL joins.


4. Data Management and Governance with Unity Catalog

⇾ Beginner familiarity with cloud computing concepts (virtual machines, object storage, etc.)

⇾ Intermediate experience with basic SQL concepts such as SQL commands, aggregate functions, filters and sorting, indexes, tables, and views.

⇾ Basic knowledge of Python programming, jupyter notebook interface, and PySpark fundamentals.

Outline

1. Data Ingestion with Lakeflow Connect

Introduction to Data Engineering in Databricks

Cloud Storage Ingestion with LakeFlow Connect Standard Connector

Enterprise Data Ingestion with LakeFlow Connect Managed Connectors

Ingestion Alternatives


2. Deploy Workloads with Lakeflow Jobs

⇾ Introduction to Data Engineering in Databricks

⇾ Lakeflow Jobs Core Concepts

⇾ Creating and Scheduling Jobs

⇾ Advance Lakeflow Jobs Features


3. Build Data Pipelines with DLT
Introduction to Data Engineering in Databricks
Pipeline Fundamentals
Building Pipelines

4. Data Management and Governance with Unity Catalog
Data Governance Overview
Demo: Populating the Metastore
Lab: Navigating the Metastore
Organization and Access Patterns
Demo: Upgrading Tables to Unity Catalog
Security and Administration in Unity Catalog
Databricks Marketplace Overview
Privileges in Unity Catalog
Demo: Controlling Access to Data
Fine-Grained Access Control
Lab: Migrating and Managing Data in Unity Catalog

Upcoming Public Classes

Date
Time
Language
Price
Jul 01 - 02
09 AM - 05 PM (Asia/Kolkata)
English
$1500.00
Jul 07 - 08
09 AM - 05 PM (America/Chicago)
English
$1500.00
Jul 21 - 24
02 PM - 06 PM (Europe/Paris)
English
$1500.00
Jul 21 - 24
02 PM - 06 PM (America/New_York)
English
$1500.00
Jul 28 - 31
11 AM - 03 PM (Asia/Singapore)
English
$1500.00
Aug 04 - 05
09 AM - 05 PM (Asia/Kolkata)
English
$1500.00
Aug 04 - 05
09 AM - 05 PM (Europe/Paris)
English
$1500.00
Aug 04 - 05
09 AM - 05 PM (America/New_York)
English
$1500.00
Aug 18
09 AM - 05 PM (Europe/London)
English
$1500.00
Aug 18 - 19
09 AM - 05 PM (America/Los_Angeles)
English
$1500.00
Aug 25 - 28
11 AM - 03 PM (Asia/Singapore)
English
$1500.00
Aug 25 - 28
02 PM - 06 PM (Europe/Paris)
English
$1500.00
Aug 25 - 28
02 PM - 06 PM (America/New_York)
English
$1500.00
Sep 01 - 02
09 AM - 05 PM (Asia/Kolkata)
English
$1500.00
Sep 01 - 02
09 AM - 05 PM (Europe/Paris)
English
$1500.00
Sep 02 - 03
09 AM - 05 PM (America/New_York)
English
$1500.00
Sep 22 - 25
11 AM - 03 PM (Asia/Singapore)
English
$1500.00
Sep 22 - 25
02 PM - 06 PM (Europe/Paris)
English
$1500.00
Sep 22 - 25
02 PM - 06 PM (America/New_York)
English
$1500.00
Sep 29 - 30
09 AM - 05 PM (America/Los_Angeles)
English
$1500.00
Oct 01 - 02
09 AM - 05 PM (Asia/Kolkata)
English
$1500.00
Oct 01 - 02
09 AM - 05 PM (Europe/Paris)
English
$1500.00
Oct 06 - 07
09 AM - 05 PM (America/New_York)
English
$1500.00
Oct 13 - 14
09 AM - 05 PM (Europe/London)
English
$1500.00
Oct 20 - 21
09 AM - 05 PM (America/Los_Angeles)
English
$1500.00
Oct 27 - 30
11 AM - 03 PM (Asia/Singapore)
English
$1500.00
Oct 27 - 30
02 PM - 06 PM (Europe/Paris)
English
$1500.00
Oct 27 - 30
02 PM - 06 PM (America/New_York)
English
$1500.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.