Big Data Visualization in Public Health

The need for cost-effective, responsive, and timely big data visualization for public health purposes is paramount. In this presentation, we'll highlight how the CDC applied an enhanced big data processing methodology with Databricks to produce visualizations to inform public health situational awareness and action. This process optimized a respiratory illness dashboard monitoring COVID-19, RSV, and influenza. Using a medallion architecture, smart parameterization in SQL, and reusable software engineering templates allows us to manage data in a common data model and accelerate the pace of standard visualization delivery on data sizes ranging up to petabyte scale. We moved many hard-coded variables to a self-service list, enhancing maintenance, reducing costs, and improving scalability. These advancements cut down the time from data acquisition to gaining actionable insights, achieving up to a tenfold increase in analysis and visualization processes efficiency.