From 8 Hours to 8 Minutes: Automating A/B Test Analysis on Databricks

Mobile game studios run multiple experiments simultaneously, but daily insights are often delayed by manual data preparation, inconsistent methods, and ad hoc notebooks. This presentation shows how SEGA HARDlight migrated from Tableau to Databricks and built an automated framework for experiment analysis: it ingests definitions and telemetry, applies uniform statistical models, and publishes a daily dashboard with an LLM-generated summary and KPI diagnostics. Unity Catalog manages assets and lineage, while Lakeflow streamline pipelines. MLflow ensures consistent experiment tracking, and AI/BI dashboards with Genie enable natural language exploration. The dashboard locks in results and decisions, creating an auditable record of truth. Early estimates show over eight hours saved weekly for the data team and roughly half a day saved per new query through automation, aiming to double testing capacity—while also boosting trust and consistency across experiments.