Session
Simplify Data Ingest and Egress with the New Python Data Source API
Overview
Experience | In Person |
---|---|
Type | Breakout |
Track | Data Engineering and Streaming |
Industry | Energy and Utilities, Manufacturing, Financial Services |
Technologies | Apache Spark, Delta Lake, LakeFlow |
Skill Level | Advanced |
Duration | 40 min |
Data engineering teams are frequently tasked with building bespoke ingest and/or egress solutions for myriad custom, proprietary, or industry-specific data sources or sinks. Many teams find this work cumbersome and time-consuming. Recognizing these challenges, Databricks interviewed numerous companies across different industries to better understand their diverse data integration needs. This comprehensive feedback led us to develop the Python Data Source API for Apache Spark™.
Session Speakers
Craig Lukasik
/Sr. SSA
Databricks