Session

No-Code Change in Your Python UDF for Arrow Optimization

Overview

ExperienceIn Person
TypeLightning Talk
TrackData Engineering and Streaming
IndustryEnterprise Technology
TechnologiesApache Spark
Skill LevelBeginner
Duration20 min

Apache Spark™ has introduced Arrow-optimized APIs such as Pandas UDFs and the Pandas Functions API, providing high performance for Python workloads. Yet, many users continue to rely on regular Python UDFs due to their simple interface, especially when advanced Python expertise is not readily available.

 

This talk introduces a powerful new feature in Apache Spark that brings Arrow optimization to regular Python UDFs. With this enhancement, users can leverage performance gains without modifying their existing UDFs — simply by enabling a configuration setting or toggling a UDF-level parameter.

 

Additionally, we will dive into practical tips and features for using Arrow-optimized Python UDFs effectively, exploring their strengths and limitations. Whether you’re a Spark beginner or an experienced user, this session will allow you to achieve the best of both simplicity and performance in your workflows with regular Python UDFs.

Session Speakers

Hyukjin Kwon

/Staff Software Engineer
Databricks