Session

From Datavault to Delta Lake: Streamlining Data Sync with Lakeflow Connect

Overview

ExperienceIn Person
TypeBreakout
TrackData Lakehouse Architecture and Implementation
IndustryHealth and Life Sciences
TechnologiesDelta Lake, LakeFlow, Unity Catalog
Skill LevelIntermediate
Duration40 min
In this session, we will explore the Australian Red Cross Lifeblood's approach to synchronizing an Azure SQL Datavault 2.0 (DV2.0) implementation with Unity Catalog (UC) using Lakeflow Connect. Lifeblood's DV2.0 data warehouse, which includes raw vault (RV) and business vault (BV) tables, as well as information marts defined as views, required a multi-step process to achieve data/business logic sync with UC. This involved using Lakeflow Connect to ingest RV and BV data, followed by a custom process utilizing JDBC to ingest view definitions, and the automated/manual conversion of T-SQL to Databricks SQL views, with Lakehouse Monitoring for validation. In this talk, we will share our journey, the design decisions we made, and how the resulting solution now supports analytics workloads, analysts, and data scientists at Lifeblood.

Session Speakers

IMAGE COMING SOON

Andrew Clarke

/Senior AI/ML Engineer
Australian Red Cross Lifeblood