Session

Building With Community Connectors: Custom Data Sources, Managed Reliability

Overview

ExperienceIn Person
TrackData Engineering & Streaming
IndustryEnterprise Technology
TechnologiesLakeflow
Skill LevelIntermediate

Are you tired of building fragile, costly API scripts for niche data sources that lack enterprise-grade features like lineage, retries and unified governance?

 

Community Connectors are built on the foundational power of Lakeflow Connect and bridge the gap between "DIY" and “fully managed." These new AI-powered, open-source templates help you build, manage and host your own ingestion paths while inheriting many of the robust, built-in features of the Databricks Platform.

 

In this lightning talk, we’ll explore:

  • Built-in tooling: Use out-of-the-box observability, SCD Type 2 handling, automatic retries and more
  • Flexible customization: Use open-source templates to meet requirements that go beyond standard managed connectors
  • Unified governance: Centrally manage credentials, pipelines, data and lineage via Unity Catalog

 

Join us to get started and learn how they offer the flexibility of a DIY approach with the scale and reliability of a managed Databricks service.

Session Speakers

Peter Pogorski

/Staff Product Manager
Databricks