Session
Reducing Transaction Conflicts in Databricks—Fundamentals and Applications at Asana
Overview
Experience | In Person |
---|---|
Type | Lightning Talk |
Track | Data Lakehouse Architecture and Implementation |
Industry | Enterprise Technology |
Technologies | Apache Spark, Delta Lake |
Skill Level | Intermediate |
Duration | 20 min |
When using ACID-guaranteed transactions on Databricks concurrently, we can run into transaction conflicts. This talk discusses the basics of concurrent transaction functionality in Databricks—what happens when various combinations of INSERT, UPDATE and MERGE INTO happen concurrently. We discuss how table isolation level, partitioning and deletion vectors affect this. We also mention how Asana used an intermediate blind append stage to support several hundred concurrent transaction updates into the same table.
Session Speakers
Dima Kamalov
/Software Engineer
Asana