Data Ingestion with Lakeflow Connect
This course provides a comprehensive introduction to Lakeflow Connect as a scalable and simplified solution for ingesting data into Databricks from a variety of data sources. You will begin by exploring the different types of connectors within Lakeflow Connect (Standard and Managed), learn about various ingestion techniques, including batch, incremental batch, and streaming, and then review the key benefits of Delta tables and the Medallion architecture.
From there, you will gain practical skills to efficiently ingest data from cloud object storage using Lakeflow Connect Standard Connectors with methods such as CREATE TABLE AS (CTAS), COPY INTO, and Auto Loader, along with the benefits and considerations of each approach. You will then learn how to append metadata columns to your bronze level tables during ingestion into the Databricks data intelligence platform. This is followed by working with the rescued data column, which handles records that don’t match the schema of your bronze table, including strategies for managing this rescued data.
The course also introduces techniques for ingesting and flattening semi-structured JSON data, as well as enterprise-grade data ingestion using Lakeflow Connect Managed Connectors.
Finally, learners will explore alternative ingestion strategies, including MERGE INTO operations and leveraging the Databricks Marketplace, equipping you with foundational knowledge to support modern data engineering ingestion.
⇾ Basic understanding of the Databricks Data Intelligence platform, including Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture and Unity Catalog.
⇾ Experience working with various file formats (e.g., Parquet, CSV, JSON, TXT).
⇾ Proficiency in SQL and Python.
⇾ Familiarity with running code in Databricks notebooks.
Outline
- Introduction to Data Engineering in Databricks
- Data Engineering in Databricks
- What is Lakeflow Connect?
- Delta Lake Review
- Exploring the Lab Environment
- Cloud Storage Ingestion with LakeFlow Connect Standard Connectors
- Introduction to Data Ingestion from Cloud Storage
- Appending Metadata Columns on Ingest
- Working with the Rescued Data Column
- Ingesting Semi-Structured Data: JSON
- Enterprise Data Ingestion with LakeFlow Connect Managed Connectors
- Ingesting Enterprise Data into Databricks Overview
- Enterprise Data Ingestion with Lakeflow Connect
- Ingestion Alternatives
- Ingesting Data with Databricks Marketplace
- Ingesting into Existing Delta Tables
- Data Ingestion with MERGE INTO
Upcoming Public Classes
Date | Time | Language | Price |
|---|---|---|---|
Jul 22 | 09 AM - 01 PM (America/New_York) | English | $750.00 |
Jul 23 | 01 PM - 05 PM (Asia/Kolkata) | English | $750.00 |
Jul 25 | 09 AM - 01 PM (Europe/London) | English | $750.00 |
Aug 05 | 01 PM - 05 PM (Australia/Sydney) | English | $750.00 |
Aug 26 | 01 PM - 05 PM (Europe/London) | English | $750.00 |
Sep 22 | 01 PM - 05 PM (Asia/Kolkata) | English | $750.00 |
Sep 23 | 01 PM - 05 PM (America/New_York) | English | $750.00 |
Oct 29 | 01 PM - 05 PM (Asia/Kolkata) | English | $750.00 |
Oct 30 | 01 PM - 05 PM (America/New_York) | English | $750.00 |
Public Class Registration
If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.
Private Class Request
If your company is interested in private training, please submit a request.
Registration options
Databricks has a delivery method for wherever you are on your learning journey
Self-Paced
Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos
Register now
Instructor-Led
Public and private courses taught by expert instructors across half-day to two-day courses
Register now
Blended Learning
Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase
Purchase now
Skills@Scale
Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details
