From Eight Hours to Eight Minutes: Automating A/B Test Analysis on Databricks

The team's arc is familiar: early wins with scrappy SQL and dashboards, a reckoning over statistical rigor, and finally a full workflow redesign on the Databricks Platform. For studios and data teams evaluating similar infrastructure decisions, the enterprise analytics platform evaluation guide outlines the criteria that distinguish platforms built for scale from those that merely handle volume.

The central lesson is not a single statistical trick but a system-level change. Scaling trustworthy experiments requires clean data, consistent methods, and self-serve tools, so product teams can make faster, clearer decisions while analysts focus on improving the methodology.

From ad hoc SQL to live monitoring and statistical rigor

Early efforts focused on getting the basics right, including correct player enrollment, retention and monetization tracking, and reliable readouts. Experiments were framed like clinical trials, with "time in experiment" as day zero for each player, rather than relying on calendar time. A single parent SQL job eventually fed a Tableau live monitoring dashboard that exposed engagement and monetization side by side for treatment and control.

Visibility created a new bottleneck. Directional patterns were not enough for confident decisions, especially under pressure to start and stop tests at an optimal cadence. The studio needed significance guardrails and an intuitive way to express the likelihood of improvement.

Combined frequentist and Bayesian methods for actionable results

Rather than choose a single paradigm, the team runs frequentist and Bayesian analyses in parallel. Frequentist tests, including chi-square for conversion, Kaplan–Meier survival with a log-rank test for retention, and Welch's t-test for numerical metrics, set significance boundaries, with FDR correction to handle many metric and segment combinations. Bayesian outputs translate results into probabilities that designers and product managers can act on.

Metrics are grouped into four families: conversion, retention, numerical engagement, and lifetime value. Lifetime value serves as the north star, projected with an exponential decay curve and a 95 percent confidence interval on the difference between groups. Winsorization at segment and experiment level controls heavy revenue tails typical of free-to-play games, and minimum sample size thresholds prevent premature calls. Statistical evidence informs judgment, it does not replace it.

Rebuilding the workflow around scale on the Databricks Platform

The real constraint was people, not math. Analysts were spending up to eight hours per experiment pulling data, running local Python, and interpreting outputs, which risked inconsistent conclusions. The fix was a unified system that standardizes and automates the entire path from telemetry to decision.

Player events flow into a medallion architecture, with Lakeflow Jobs orchestrating ingestion, feature preparation, and gold-layer materialized views of precomputed A/B metrics. The statistical layer, including Bayesian inference, bootstrapping, winsorization, and lifetime value modeling, runs automatically per test using the same assumptions every time. Results surface in AI/BI Dashboards that refresh nightly. Unity Catalog provides governance and lineage from raw events to insights, and MLflow versions models and assets for auditability. Genie sits alongside AI/BI Dashboards for natural-language questions.

Dashboards that let teams decide, not just observe

The dashboard is layered for different audiences. An LLM-generated summary at the top captures the outcome in plain language. Health checks validate enrollment and flag anomalies. Mid-test engagement views highlight how players interact with features. A final analysis tab summarizes outcomes by metric family, pairing frequentist significance with Bayesian probabilities. A simple experiment selector enables rapid context switching, moving ownership to live operations, designers, and product managers, with analysts stepping in to refine methods or investigate ambiguous results.

What changes when experiments run at the speed of play

Analyst time per experiment fell from roughly eight hours to about eight minutes, and throughput doubled. Standardization, automation, accessibility, and governance turned experimentation into a continuous, reliable practice. Treating scale as a workflow problem, not only a statistics problem, unlocked consistent, compounding improvements to player experience and business outcomes.