SESSION

An In Depth Look at the New Features of Apache Spark 3.5

Register or Login

OVERVIEW

EXPERIENCEIn Person
TYPEBreakout
TRACKData Engineering and Streaming
INDUSTRYEnterprise Technology
TECHNOLOGIESApache Spark
SKILL LEVELIntermediate
DURATION40 min

This session will delve into the latest advancements in Apache Spark™ 3.5, highlighting its pivotal role in pushing the boundaries of big data processing and AI. We'll discuss Spark Connect's role in enhancing accessibility, DeepSpeed's integration for AI efficiency, and performance optimizations. Additionally, we'll delve into the new PySpark and SQL features, including built-in functions for array manipulation, SQL IDENTIFIER clause enhancements, expanded API support, and Arrow-optimized Python UDFs, highlighting their impact on building scalable, efficient, and robust data-driven applications.

SESSION SPEAKERS

Daniel Tenedorio

/Sr. Staff Software Engineer
Databricks

Xiao Li

/Engineering Director
Databricks