Lessons Learned from Migrating the Largest Immunization Registry

The CA Dept of Public Health recently migrated the U.S.'s largest immunization registry to Databricks. The registry manages over 50 million individuals with nearly one billion records. The migration involved implementing SCD Type 2 tables using Delta Live Tables (DLT) and change data capture from an Oracle database. Insights from the implementation include: Know Your Data - Understanding the nature of updates is crucial. Understand File Structure Impact on Performance - SCD Type 2 requires multiple operations that depend on the organization of the data. Consider Your Compute Requirements: Crafting an effective cluster strategy requires balancing cost and meeting SLAs. Decouple Unrelated Workflows: We successfully decoupled workflows, optimizing computing resources for critical functionality. We anticipate features like Liquid Clustering and serverless computing to enhance our immunization registry analytics platform's efficiency in managing vast healthcare datasets.