Session

Simplify Data Ingest and Egress with the New Python Data Source API

Overview

ExperienceIn Person
TypeBreakout
TrackData Engineering and Streaming
IndustryEnergy and Utilities, Manufacturing, Financial Services
TechnologiesApache Spark, Delta Lake, LakeFlow
Skill LevelAdvanced
Duration40 min

Data engineering teams are frequently tasked with building bespoke ingest and/or egress solutions for myriad custom, proprietary, or industry-specific data sources or sinks. Many teams find this work cumbersome and time-consuming. Recognizing these challenges, Databricks interviewed numerous companies across different industries to better understand their diverse data integration needs. This comprehensive feedback led us to develop the Python Data Source API for Apache Spark™.

Session Speakers

Craig Lukasik

/Sr. SSA
Databricks