Session

Production Lakebase Patterns: OAuth Token Lifecycle, Async Pooling, and Caching for Databricks Apps

Overview

ExperienceIn Person
TrackApplication Development
IndustryEnergy & Utilities
TechnologiesUnity Catalog, Databricks Apps, Lakebase
Skill LevelAdvanced
DOWNLOAD SESSION SLIDES

Lakebase gives you PostgreSQL on Databricks, but production applications need patterns the documentation doesn't cover. From deploying a conversation persistence layer for an energy sector knowledge assistant, we share battle-tested solutions: Proactive OAuth token refresh that recreates connection pools before credentials expire, async connection pooling with psycopg3 sized for Databricks Apps concurrency, LRU caching with TTL for high-frequency thread lookups, and upsert patterns with conflict resolution for concurrent agent writes. Every pattern is production code running on Databricks Apps today.

 

This lightning talk is for developers who've tried Lakebase in a notebook and want to know what it takes to run it in a real application with real users.

Session Speakers

Irvin Umaña

/Lead Scale Solution Engineer
Databricks

Full Summary

Four production patterns for running Lakebase-backed agentic apps without the 60-minute auth cliff

Agentic apps on the Databricks Platform often run flawlessly for about an hour, then fall off a cliff with a cryptic authentication failure. The root cause is not a broken secret or a revoked service principal. Lakebase uses OAuth tokens as Postgres passwords, tokens expire at 60 minutes, and expiry is enforced at login. Existing connections keep working, but any new one past minute 60 fails.

FAQ


The managed PgBouncer only supports native Postgres password authentication. OAuth-based connections bypass it, so you must run your own pool and handle token rotation in the application.

OAuth is the default for Databricks Apps and service principals because it aligns with Databricks identity and Unity Catalog governance. Native passwords fit long-lived services and third-party tools that cannot rotate credentials hourly.

Keep min at 1 to 3 to avoid cold starts from scale to zero. Set max to your peak concurrent users, not the Postgres ceiling, since each slot is a real backend process. The talk cites roughly 1,600 max connections on 8 CU and 3,300 on 16 CU.

ON CONFLICT requires a pre-existing unique constraint on the target columns, for example thread_id and turn_index. Without it, unit tests may pass, but production writes will fail on the first concurrent update.

New provisioned instances are no longer created as of March 2026, and existing ones are being upgraded to autoscaling Postgres 17. A pg_dump and pg_restore path exists if you need to migrate, and the patterns here assume the autoscaling model.