Apache Spark™ Connect: The Path to Modern, AI-Ready Spark

Spark Connect is more than a new client interface for Apache Spark™ — it is the foundation for a modern, AI-ready, and production-friendly Spark experience. As AI assistants generate more Spark code, the defaults they learn will shape how the next generation of data applications is built — that makes Spark Connect a strategic shift, not just a technical one. In this session, we will share our vision for making Spark Connect the default way developers build with Spark. We will discuss how DataFrame-first guidance, better AI-generated code compatibility, and machine-readable API discovery can make Connect the easiest path for developers. We will also cover the work needed to build production trust, including closing migration gaps, improving observability, and increasing ecosystem compatibility. Attendees will leave with a clear view of why Spark Connect matters now and what it will take to make it the standard interface for modern Spark applications.