The California Healthcare Eligibility, Enrollment, and Retention System (CalHEERS)—one of the largest State-based health exchanges in the country—was looking to modernize their data warehouse (DWH) environment to support the vision that every decision to design, implement and evaluate their state-based health exchange portal was informed by timely and rigorous evidence about its consumers’ experience. The scope of the project was to replace the existing Oracle-based DWH with an analytics platform that could support a much broader range of requirements with ability to provide unified analytics capabilities including ML. The modernized analytics platform comprises a cloud native data lake and DWH solution using Databricks Lakehouse Platform along with other key technologies. This lakehouse oriented architecture provides significantly higher performance and elastic scalability to better handle larger/varying data volumes with much lower cost of ownership compared to the existing solution. The solution replaced a massive oracle-based infrastructure with a server-less solution using Databricks to ingest data from the source systems into an AWS S3 data lake where the data is curated prior to provisioning it to the downstream data marts and reports. The following processes/workflows will be covered as part of the Data Analytics Solution:
-Overall Solution Architecture
-Design of the data pipelines /framework that loads DWH
-Error handling process
-Various use cases supported (Reporting and Dashboards, Ad-hoc Analysis, ML, Data Extracts)
-Sandbox environment to perform data analysis and integrate data across source environments
-End User interaction with Tools (Databricks, Snowflake and Tableau)
-Security and user access protocols
-KPIs comparing before and after states