Running LLaMA at Scale: Production Inference on Databricks Model Serving

The story matters because it shows how to migrate a latency-critical, user-facing feature to a managed platform without sacrificing trust. It pairs disciplined validation and client-side observability with pragmatic scaling and incident-readiness.

Collapsing a multi-model stack into one fast, cheap LLM

Superhuman's grammar system evolved from thousands of hand-written linguistic rules to multiple small classifiers, each with its own serving footprint and aggregation logic. Maturing LLMs enabled a simpler architecture: a single model with light post-processing, provided it could match the legacy system's quality, latency, and cost.

Starting from Llama, the team leaned on optimizations such as FP8 and speculative decoding. Inference costs landed in cents per million tokens, far below frontier APIs and comparable to their previous smaller models. Users expect immediate feedback while typing, so responses target well under one second despite heavy, always-on usage.

Why managed serving won, and how parity was proven

An internal vLLM stack worked but hit limits: a single AWS region with periodic GPU scarcity, disjoint systems for training, storage, and serving, and a desire to focus engineering time upstream of inference. Databricks offered a unified path from data to model to endpoint, along with a serving team that could absorb much of the optimization work.

Parity was proven in layers. Exact output match between in-house and Databricks serving hovered near 94 percent, with differences typically representing equally valid corrections. Task-specific quality checks used M2 precision and recall on evaluation sets. A production A/B experiment then tested for no statistically significant change in business metrics, reframing success as parity, not lift. Cost was modeled as cost per million tokens using representative shadow or gated traffic, where 10 to 30 percent over roughly a week yielded stable, production-like numbers normalized by total input and output tokens.

Rollout guardrails: autoscaling, regressions, and end-to-end observability

Early latency was shaky with outliers. Client-side metrics in Superhuman's LLM proxy showed queues building when concurrency-based autoscaling pushed GPUs to their limit. Switching to RPS-based autoscaling, viable because request token lengths were similar, stabilized latency by leaving modest headroom on each GPU.

A subtle regression surfaced mid-experiment when suggestion counts drifted. A model container change had stopped prepending the beginning-of-text token that tokenizer configuration normally added. Quality checks reproduced the issue and the fix was to restore the expected token behavior. The episode underscored the value of active experimentation frameworks and periodic quality audits to catch silent failures.

Shadow traffic uncovered a non-obvious bottleneck unrelated to inference. Under load, Databricks' authentication subsystem hit limits, which did not appear on inference dashboards. Client-side telemetry identified the hop causing latency, leading to auth-side scaling and configuration fixes. The team also ran a mock support ticket to test escalation paths, which revealed a routing gap later resolved with a dedicated inference category.

Production readiness hinges on parity, telemetry, and practice

The rollout now serves roughly 80 percent of red-underline traffic through provisioned foundation model throughput on Databricks at 200-plus RPS, with availability at or above four nines. The durable lesson is not a single optimization, but a method: prove parity before lift, keep robust client-side observability, scale on reliable load proxies like RPS, shadow end-to-end paths, and rehearse incident response before it counts.