Skip to main content

Databricks Performance Optimization

In this course, you’ll learn how to optimize workloads and physical layout with Spark and Delta Lake and and analyze the Spark UI to assess performance and debug applications. We’ll cover topics like streaming, liquid clustering, data skipping, caching, photons, and more.

Note: This course is part of the 'Advanced Data Engineering with Databricks' course series.


Languages Available: English | 日本語 | Português BR | 한국어

Skill Level
Professional
Duration
4h
Prerequisites

The content was developed for participants with these skills/knowledge/abilities:  

  • Ability to perform basic code development tasks using Databricks (create clusters, run code in notebooks, use basic notebook operations, import repos from git, etc.)
  • Intermediate programming experience with PySpark
  • Extract data from a variety of file formats and data sources
    • Apply a number of common transformations to clean data
  • Reshape and manipulate complex data using advanced built-in functions
  • Intermediate programming experience with Delta Lake (create tables, perform complete and incremental updates, compact files, restore previous versions, etc.)


Outline

  • Spark Architecture

Spark UI Introduction

  • Designing the Foundation File Explosion 

 Introduction to Designing Foundation

 File Explosion

 Data Skipping and Liquid Clustering

 Lab: Data Skipping and Liquid Clustering 

  • Code Optimization

 Skew

 Shuffle 

 Spill

 Lab: Exploding Join 

 Serialization 

 User-Defined Functions

  • Fine-Tuning - Choosing the Right Cluster

 Fine-Tuning: Choosing the Right Cluster

 Pick the Best Instance Types

Upcoming Public Classes

Date
Time
Your Local Time
Language
Price
May 21
11 AM - 03 PM (Asia/Singapore)
-
English
$750.00
May 21
09 AM - 01 PM (America/New_York)
-
English
$750.00
Jun 12
01 PM - 05 PM (Europe/London)
-
English
$750.00
Jun 16
08 AM - 12 PM (Asia/Kolkata)
-
English
$750.00
Jul 17
01 PM - 05 PM (Australia/Sydney)
-
English
$750.00
Jul 17
09 AM - 01 PM (America/New_York)
-
English
$750.00

Public Class Registration

If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.

Private Class Request

If your company is interested in private training, please submit a request.

See all our registration options

Registration options

Databricks has a delivery method for wherever you are on your learning journey

Runtime

Self-Paced

Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos

Register now

Instructors

Instructor-Led

Public and private courses taught by expert instructors across half-day to two-day courses

Register now

Learning

Blended Learning

Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase

Purchase now

Scale

Skills@Scale

Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details

Upcoming Public Classes

Building Reliable Conversational Agents with Genie

This course teaches you how to design, build, and maintain a Databricks Genie Space, a natural language interface that enables business users to ask questions about governed data and receive SQL-backed answers without writing code.

You will learn how Genie fits into the Databricks AI/BI product family and how it translates natural language into reliable SQL queries. The course focuses on what it takes to create a Genie Space that delivers accurate, consistent, and trustworthy results.

You will follow a complete end-to-end workflow, from understanding source data and defining benchmarks to configuring and refining a Genie Space using the full set of Knowledge Store curation tools. These include metadata, synonyms, prompt matching, SQL logic, example queries, and text instructions.

You will also learn how to share Genie Spaces with business users through Databricks One, understand how Unity Catalog governance is automatically enforced, and use monitoring and user feedback to continuously improve quality over time.

By the end of the course, you will be able to create and manage a production-ready Genie Space that delivers governed, self-service conversational analytics at scale.

Note: Databricks Academy is transitioning to a notebook-based format for classroom sessions within the Databricks environment, discontinuing the use of slide decks for lectures. You can access the lecture notebooks in the Vocareum lab environment.

Paid
4h
Lab
instructor-led
Associate

Questions?

If you have any questions, please refer to our Frequently Asked Questions page.