Session
Building With Community Connectors: Custom Data Sources, Managed Reliability
Overview
| Experience | In Person |
|---|---|
| Track | Data Engineering & Streaming |
| Industry | Enterprise Technology |
| Technologies | Lakeflow |
| Skill Level | Intermediate |
Are you tired of building fragile, costly API scripts for niche data sources that lack enterprise-grade features like lineage, retries and unified governance?
Community Connectors are built on the foundational power of Lakeflow Connect and bridge the gap between "DIY" and “fully managed." These new AI-powered, open-source templates help you build, manage and host your own ingestion paths while inheriting many of the robust, built-in features of the Databricks Platform.
In this lightning talk, we’ll explore:
- Built-in tooling: Use out-of-the-box observability, SCD Type 2 handling, automatic retries and more
- Flexible customization: Use open-source templates to meet requirements that go beyond standard managed connectors
- Unified governance: Centrally manage credentials, pipelines, data and lineage via Unity Catalog
Join us to get started and learn how they offer the flexibility of a DIY approach with the scale and reliability of a managed Databricks service.
Session Speakers
Peter Pogorski
/Staff Product Manager
Databricks