Session

Reducing Transaction Conflicts in Databricks—Fundamentals and Applications at Asana

Overview

ExperienceIn Person
TypeLightning Talk
TrackData Lakehouse Architecture and Implementation
IndustryEnterprise Technology
TechnologiesApache Spark, Delta Lake
Skill LevelIntermediate
Duration20 min

When using ACID-guaranteed transactions on Databricks concurrently, we can run into transaction conflicts. This talk discusses the basics of concurrent transaction functionality in Databricks—what happens when various combinations of INSERT, UPDATE and MERGE INTO happen concurrently. We discuss how table isolation level, partitioning and deletion vectors affect this.  We also mention how Asana used an intermediate blind append stage to support several hundred concurrent transaction updates into the same table.

Session Speakers

Dima Kamalov

/Software Engineer
Asana