Under the Hood: Scalable Ingestion Strategies for Databases, Applications, and File Sources

Data engineers spend far too much time maintaining fragile scripts for a fragmented landscape of databases, applications, and file sources. In this technical session, we’ll explore how Lakeflow Connect allows you to build robust pipelines without the operational burden of DIY infrastructure.

We’ll dive into:

• Database ingestion: A look under the hood at efficient CDC for sources like Oracle, MySQL, and Postgres, alongside query-based ingestion for sources like Teradata, Redshift, and Synapse.
• Application and file ingestion: What’s new—and happening behind the scenes—across our newest connectors for customer 360, employee 360, financial analytics, and more.
• Cost and performance optimization: Technical best practices for networking, security, and minimizing impact on source systems—plus how to use new features to slash TCO and boost performance.

We’ll wrap up with how to create your own application and database connectors. Stop managing infrastructure and start delivering data.