Reducing Transaction Conflicts in Databricks—Fundamentals and Applications at Asana

When using ACID-guaranteed transactions on Databricks concurrently, we can run into transaction conflicts. This talk discusses the basics of concurrent transaction functionality in Databricks—what happens when various combinations of INSERT, UPDATE and MERGE INTO happen concurrently. We discuss how table isolation level, partitioning and deletion vectors affect this.  We also mention how Asana used an intermediate blind append stage to support several hundred concurrent transaction updates into the same table.