Data Ingestion with Lakeflow Connect
This course provides a comprehensive introduction to Lakeflow Connect, a scalable and simplified solution for ingesting data into Databricks from a wide range of sources. You’ll begin by exploring the different types of Lakeflow Connect connectors (Standard and Managed) and learn various data ingestion techniques, including batch, incremental batch, and streaming ingestion. You'll also review the key benefits of using Delta table and the Medallion architecture
Next, you’ll develop practical skills for ingesting data from cloud object storage using Lakeflow Connect Standard Connectors. This includes working with methods such as CREATE TABLE AS SELECT (CTAS), COPY INTO, and Auto Loader, with an emphasis on the benefits and considerations of each approach. You’ll also learn how to append metadata columns to your bronze-level tables during ingestion into the Databricks Data Intelligence Platform. The course then covers how to handle records that don’t match your table schema using the rescued data column, along with strategies for managing and analyzing this data. You’ll also explore techniques for ingesting and flattening semi-structured JSON data.
Following this, you’ll explore how to perform enterprise-grade data ingestion using Lakeflow Connect Managed Connectors to bring in data from databases and Software-as-a-Service (SaaS) applications. The course also introduces Partner Connect as an option for integrating partner tools into your ingestion workloads.
Finally, the course wraps up with alternative ingestion strategies, including MERGE INTO operations and leveraging the Databricks Marketplace, equipping you with a strong foundation to support modern data engineering use cases.
Note: Databricks Academy is transitioning to a notebook-based format for classroom sessions within the Databricks environment, discontinuing the use of slide decks for lectures. You can access the lecture notebooks in the Vocareum lab environment.
• Basic understanding of the Databricks Data Intelligence platform, including Databricks Workspaces, Apache Spark, Delta Lake, the Medallion Architecture and Unity Catalog.
• Basic understanding of data ingestion workflows (batch, streaming, incremental) and general ETL principles
• Experience working with various file formats (e.g., Parquet, CSV, JSON, TXT).
• Proficiency in SQL and Python.
• Familiarity with running code in Databricks notebooks.
Outline
• Data Engineering in Databricks
• Exploring the Lab Environment
• Data Ingestion from Cloud Storage
• Demo - Data Ingestion with CREATE TABLE AS and COPY INTO
• Demo - Create Streaming Tables with SQL using Auto Loader
• Appending Metadata Columns on Ingest
• Demo - Adding Metadata Columns During Ingestion
• Working with the Rescued Data Column
• Demo - Handling CSV Ingestion with the Rescued Data Column
• Lab - Creating Bronze Tables from CSV Files
• Ingesting Semi-Structured Data: JSON
• Demo - Ingesting JSON Files with Databricks
• Lab - Creating Bronze Tables from JSON Files
• Ingesting Enterprise Data Overview
• Demo - Enterprise Data Ingestion with LakeFlow Connect
• Additional Features and Ingesting into Existing Delta Tables
• Demo - BONUS - Data Ingestion with MERGE INTO
Upcoming Public Classes
Date | Time | Your Local Time | Language | Price |
|---|---|---|---|---|
Aug 04 | 09 AM - 01 PM (Asia/Singapore) | - | English | $750.00 |
Aug 04 | 09 AM - 01 PM (Europe/Paris) | - | English | $750.00 |
Aug 04 | 09 AM - 01 PM (America/Los_Angeles) | - | English | $750.00 |
Sep 01 | 01 PM - 05 PM (Australia/Sydney) | - | English | $750.00 |
Sep 01 | 01 PM - 05 PM (Europe/Paris) | - | English | $750.00 |
Sep 01 | 09 AM - 01 PM (America/New_York) | - | English | $750.00 |
Oct 01 | 09 AM - 01 PM (Asia/Kolkata) | - | English | $750.00 |
Oct 01 | 09 AM - 01 PM (Europe/Paris) | - | English | $750.00 |
Oct 01 | 01 PM - 05 PM (America/New_York) | - | English | $750.00 |
Public Class Registration
If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.
Private Class Request
If your company is interested in private training, please submit a request.
Registration options
Databricks has a delivery method for wherever you are on your learning journey
Self-Paced
Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos
Register now
Instructor-Led
Public and private courses taught by expert instructors across half-day to two-day courses
Register now
Blended Learning
Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase
Purchase now
Skills@Scale
Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details
