Session
Building With Community Connectors: Custom Data Sources, Managed Reliability
Overview
| Experience | In Person |
|---|---|
| Track | Data Engineering & Streaming |
| Industry | Enterprise Technology |
| Technologies | Lakeflow |
| Skill Level | Intermediate |
Are you tired of building fragile, costly API scripts for niche data sources that lack enterprise-grade features like lineage, retries and unified governance?Community Connectors are built on the foundational power of Lakeflow Connect and bridge the gap between "DIY" and “fully managed." These new AI-powered, open-source templates help you build, manage and host your own ingestion paths while inheriting many of the robust, built-in features of the Databricks Platform.In this lightning talk, we’ll explore:
- Built-in tooling: Use out-of-the-box observability, SCD Type 2 handling, automatic retries and more
- Flexible customization: Use open-source templates to meet requirements that go beyond standard managed connectors
- Unified governance: Centrally manage credentials, pipelines, data and lineage via Unity Catalog
Join us to get started and learn how they offer the flexibility of a DIY approach with the scale and reliability of a managed Databricks service.
Session Speakers
Peter Pogorski
/Staff Product Manager
Databricks