Apache Spark Programming with Databricks
This course serves as an appropriate entry point to learn Apache Spark Programming with Databricks.
Below, we describe each of the four, four-hour modules included in this course.
Note: Databricks Academy is transitioning to a notebook-based format for classroom sessions within the Databricks environment, discontinuing the use of slide decks for lectures. You can access the lecture notebooks in the Vocareum lab environment.
Introduction to Apache Spark
This beginner-friendly course covers the fundamentals of Apache Spark for large-scale data processing. You will explore Spark’s distributed architecture, master the DataFrame API, and learn to read, write, and process data using Python. Through hands-on exercises, you will build the skills needed to execute Spark transformations and actions efficiently.
Developing Applications with Apache Spark
Master scalable data processing with Apache Spark in this course. Learn to build efficient ETL pipelines, perform advanced analytics, and optimize distributed data transformations using Spark’s DataFrame API. Explore grouping, aggregation, joins, set operations, and window functions. Work with complex data types like arrays, maps, and structs while applying best practices for performance optimization.
Stream Processing and Analysis with Apache Spark
This course guides learners on how to develop scalable, fault-tolerant stream processing applications using the Spark Structured Streaming API on Databricks.
Monitoring and Optimizing Apache Spark Workloads on Databricks
This course covers core Databricks platform components and Apache Spark with Delta Lake, focusing on the Lakehouse architecture, Unity Catalog, and data management techniques like ACID transactions and table maintenance. Additionally, it teaches students how to optimize query performance through partitioning, shuffles, and tuning tools, while utilizing the Spark UI to monitor applications and resolve bottlenecks.
The content was developed for participants with these skills/knowledge/abilities:
• Basic programming knowledge
• Familiarity with Python
• Basic understanding of SQL queries (SELECT, JOIN, GROUP BY)
• Familiarity with data processing concepts
• No prior Spark or Databricks experience required
Outline
1. Introduction to Apache Spark™
Apache Spark Runtime Architecture
• What is Apache Spark
• Spark Runtime Architecture
• Demo: Exploring Spark Architecture in Databricks
Spark DataFrames and SQL
• Introduction to DataFrames
• Reading and Writing Data
• Demo: Reading and Writing Data with DataFrames
Distributed Systems Programming Fundamentals
• Distributed Systems Programming Fundamentals
ETL with the DataFrame API
• Basic ETL Operations with the DataFrame API
• Demo: Flight Data ETL with the DataFrame API
• Lab: Analyzing Transaction Data with DataFrames
2. Developing Applications with Apache Spark
DataFrame API Basics
• DataFrame API Basics
• Demo: Basic ETL with the DataFrame API
Grouping and Aggregating Data
• Grouping and Aggregating Data
• Demo: Grouping and Aggregating Data
• Lab: Grouping and Aggregating E-Commerce Data
DataFrame Relational Operations
• DataFrame Relational Operations
• Demo: DataFrame Relational Operations in Spark
Working with Complex Data
• Working with Complex Data
• Demo: Working with Complex Data Types in Spark
• Lab: Working with Complex Data Types in E-Commerce Data
3. Stream Processing and Analysis with Apache Spark
Introduction to Stream Processing
• Introduction to Stream Processing
• Demo: Your First Streaming Query
Spark Structured Streaming
• Spark Structured Streaming
• Demo: Introduction to Spark Structured Streaming
• Lab: Building Streaming Queries
Stream Aggregations, Windows and Watermarks
• Stream Aggregations, Windows and Watermarks
• Demo: Window Aggregation in Spark Structured Streaming
• Lab: Window Aggregation and Late Data
Real-Time Mode
• Real-Time Mode in Structured Streaming
• Demo: Kafka to Real-Time Spark with Live Visualization
4. Monitoring and Optimizing Apache Spark Workloads on Databricks
Apache Spark and Databricks
• Apache Spark and Databricks
• Demo: Exploring Spark on Databricks
Using Apache Spark with Delta Lake
• Using Apache Spark with Delta Lake
• Demo: Working with Delta Lake
• Lab: Working with Delta Lake
Optimizing Apache Spark
• Optimizing Apache Spark
• Demo: Monitoring and Optimizing Spark Workloads
• Lab: Optimizing Spark Workloads
Upcoming Public Classes
Date | Time | Your Local Time | Language | Price |
|---|---|---|---|---|
Nov 17 - 18 | 10 AM - 06 PM (Asia/Singapore) | - | English | $1500.00 |
Nov 17 - 18 | 09 AM - 05 PM (Europe/London) | - | English | $1500.00 |
Nov 17 - 18 | 09 AM - 05 PM (America/Los_Angeles) | - | English | $1500.00 |
Public Class Registration
If your company has purchased success credits or has a learning subscription, please fill out the Training Request form. Otherwise, you can register below.
Private Class Request
If your company is interested in private training, please submit a request.
Registration options
Databricks has a delivery method for wherever you are on your learning journey
Self-Paced
Custom-fit learning paths for data, analytics, and AI roles and career paths through on-demand videos
Register now
Instructor-Led
Public and private courses taught by expert instructors across half-day to two-day courses
Register now
Blended Learning
Self-paced and weekly instructor-led sessions for every style of learner to optimize course completion and knowledge retention. Go to Subscriptions Catalog tab to purchase
Purchase now
Skills@Scale
Comprehensive training offering for large scale customers that includes learning elements for every style of learning. Inquire with your account executive for details
