SESSION

Delta Merge Optimizations with Jodie Helpers

OVERVIEW

EXPERIENCEIn Person
TYPELightning Talk
TRACKData Lakehouse Architecture
INDUSTRYEnterprise Technology, Professional Services
TECHNOLOGIESDelta Lake
SKILL LEVELIntermediate
DURATION20 min

The talk will primarily revolve around Delta Merge Optimizations and the contributions we made to the Jodie repo:

  • Delta Merge Optimization Strategies: https://medium.com/@joydeep.roy/delta-merge-optimisation-strategies-b78f18066966
  • Change Data Feed implications on Delta tables: Performance Considerations and Failure Scenarios to look out for. Some content would be taken from https://medium.com/@joydeep.roy/delta-merge-optimisation-strategies-b78f18066966, but other strategies would also be covered
  • Delta Merge Data Skipping: Based on the contribution made in Jodie - https://github.com/MrPowers/jodie?tab=readme-ov-file#number-of-shuffle-files-in-merge--other-filter-condition

SESSION SPEAKERS

Joydeep Banik Roy

/Head of Data Science and ML Engineering
Zeotap