Enabling Evolutionary Database Development: database branching with Lakebase, continued

This series revisits the methodolgy of Evolutionary Database Design, twenty years later. A key constraint to database-changes-as-code has always been around shared database resources. With copy-on-write branching in Databricks Lakebase, a one-second, zero-storage-at-creation branch of a terabyte-scale production database is now an O(1) operation, and the constraint that kept Practice #4 (everybody gets their own database instance) aspirational has lifted.  In this series, the authors describes what changes when the constraint lifts: not the methodology, that holds, but the practices that emerge for the first time, the team-scale governance that becomes automatic, the role evolution for the DBA, and the new capability that agents share with their human counterparts.

Jen is the developer character from Evolutionary Database Design. In that essay she implemented a database refactoring, splitting an inventory_code field into location_code, batch_number, and serial_number, as a routine user story, illustrating that DBAs and developers can collaborate, schemas can evolve in small increments, and migrations carry the change forward safely.

The series picks up with Jen twenty years later. The methodology she follows is the same one she followed in 2003. What's new is the technical capability underneath her workflow, enabled by the lakebase architecture: copy-on-write database branching, which makes the practices she has been reading about operationally real at production scale. Across the three parts of this series she is the same Jen at three scopes, her day (Part 1), her new playbook (Part 2), and her team (Part 3).

Part 1 walked Jen through one feature. The practices she followed were described in the 2003 Evolutionary Database Design essay, expanded in the 2006 Refactoring Databases book , and brought into the CI/CD pipeline in the 2010 Continuous Delivery book (Chapter 12).

Original Seven Practices

The 2003 essay named seven practices. Five of the seven had limitations in their application until 2026.

1. DBAs collaborate closely with developers.
2. All database artifacts are version controlled with application code.
3. All database changes are migrations.
4. Everybody gets their own database instance.
5. Developers continuously integrate database changes.
6. All database changes are database refactorings.
7. Developers can update their databases on demand.

Limitations in their application

• Practice #1 (DBA collaboration). Every schema change had production-scale consequences if it got loose, so DBA review remained synchronous and gating. Collaboration was constrained by the DBA's calendar.
• Practice #4 (Everybody gets their own database instance). Licensing costs, infrastructure costs, DBA time. Aspirational on most teams. Most teams fell back to shared development databases and accepted the contention.
• Practice #5 (Continuous integration of database changes). The 2010 Continuous Delivery wave brought migrations into the pipeline, but the pipeline ran migrations against shared target databases. Per-pipeline isolation was missing.
• Practice #6 (All database changes are refactorings). Applying each refactoring required practice spaces (test databases) that most teams did not have at PR granularity.
• Practice #7 (Developers update on demand). Developers could run migrations against shared environments on demand, but could not safely experiment, because their experiments would affect others.

What Changed

The technology introduced by Databricks Lakebase removes the roadblocks to the implementation of the above practices. Databricks Lakebase is a managed Postgres database that uses the same object storage layer (the data lake) that the rest of the Databricks lakehouse runs on. The database's data lives in shared, durable storage – effectively S3 buckets; the Postgres engine runs as a separate compute layer above it. Compute and storage scale independently. The engine can scale up under load, down when traffic drops, and to zero when idle. Lakebase is integrated with the unity catalog allowing for unified governance across multiple environments.

Copy-on-write database branching is what the decoupled architecture makes practical at scale. A branch creates a new pointer into the same shared storage with a divergence marker. Until the branch writes, it shares all pages with its parent. When the branch writes, only the modified pages diverge; the parent stays untouched. Branching is a metadata operation, no data copy required, completing in roughly one second regardless of parent size. The branches maintain data in the changed pages only.

When the technical cost of a branch is decoupled from the size of the data inside it, the constraint behind Practice #4 (everybody gets their own database instance) is unblocked. Per-developer, per-PR, per-experiment branches become routine. The compensating layer above can come out. Mocks come out of the test loop, replaced by real Postgres on a per-test branch. Shared staging stops being the only place to test schema changes. In-memory database substitutes (H2, SQLite) come out of the unit-test layer. The devops tax to create docker based containers to run local databases is not necessary, DBA ticket queues for provisioning shrink because branches are self-service.

The technology is what enables methodology optimizations and completes the goal of the practices behind the original 2003 post.

Emerging practices for 2026

Lakebase copy-on-write database branching lifts the previous limitations on the original practices and enables four additional ones for elaboration.

1. DBAs collaborate closely with developers. With the schema diff posted on every PR, the DBA reviews async, like any other code reviewer. Since the provisioning tax is negligible the DBA’s now have the bandwidth to review and maybe even work with the developers to create the solutions in the first place instead of reviewing post implementation.
2. All database artifacts are version controlled with application code. The schema diff, the database migrations and migration test results now join the artifact set.
3. All database changes are migrations. Plus a new authorship rule: idempotency. Allowing all merged migrations to be deployed to downstream environments like QA, staging and Production automatically.
4. Everybody gets their own database instance. Operational at per-developer, per-PR, and per-experiment granularity. Allowing developers to experiment on multiple solutions and not just settle for the first solution that works. This freedom will result in optimal solutions being developed for production scale.
5. Developers continuously integrate database changes. Operational at PR granularity. Every PR runs through CI on its own branch.
6. All database changes are database refactorings. The 2006 catalog still applies; branches give you a cheap rehearsal space to try the refactorings on various sized environments, migrations that worked in development databases almost always did not perform in production, now you can try migrations in larger size databases to ensure they are performant.
7. Developers can update their databases on demand. "On demand" now means one second, isolated, against production-shaped data.
8. Destructive testing as a default option. Blast radius is zero; reset costs one second. Don’t have to worry about, “Will my test pollute data for others?”, since now after every test run I can get a new branch.
9. A/B variant prototyping at the database level. Build two designs on parallel branches; work with the larger team and the DBA’s to discuss the solution and do a show and tell, then keep the winner.
10. With the integrated unity catalog, governance is designed once, inherited by all the branches. Policies follow each branch automatically, we will discuss this in detail in Part 3.
11. Agent-as-practitioner with the same branching capability. Agents get branches, not production, we will discuss this in detail in Part 3.

How the workflow runs in CI

 

The two practices that get improved the most because of the capability provided by Lakebase are #4 (everybody gets their own database instance) and #5 (developers continuously integrate database changes). Now not just everyone gets a database, every PR or every branch or multiple databases per branch can be had with negligible cost and time. The mechanism can be automated using two GitHub Actions workflow templates that can be scaffolded into every project.

Per-PR branch creation. pr.yml triggers on pull_request: [opened, synchronize] and creates ci-pr-<N> forked from the PR's base branch:

The same job applies migrations against the CI branch and runs the application's test suite against real Postgres. Full example of the workflow can be found here: pr.yml

Schema diff posted on the PR. The same pr.yml job dumps the schema of both branches, formats the diff, and posts it as a PR comment. This is what lets the DBA review async like any code reviewer (Practice #1, re-cast):

The DBA, the code reviewer, the team and any agent author of the migration see the same artifact on the PR.

Branch cleanup on merge. merge.yml destroys  the CI Pull Request Branch ci-pr-<N> and the linked feature branch's Lakebase branch the moment the PR merges:

Full example of the merge workflow can be found here: merge.yml With so many branches floating around it may be worth having a weekly cleanup script that removes orphaned and unused branches, an example workflow can be found here: cleanup-orphans.yml. Finally, if the merge is into a ‘tiered’ branch, e.g. used for staging or main/production (this concept will be further elaborated in part 3), the workflow may be orchestrated to cut a fresh branch from the tier to test migrations against it before applying the final migration to any environment where users are found live and constantly adding data.

Together these workflows enforce every PR gets its own database and branches are ephemeral as properties of the pipeline, not developer disciplines.

The new playbook for Evolutionary Database Development

Practice #1: DBAs collaborate closely with developers

Rule. The DBA collaborates with developers throughout the feature, not just at gate-review time. The collaboration is asynchronous, inline with the PR, the way other code reviewers participate.

Why is this a durable habit now? The schema diff and migration test results land on every PR automatically (see How the workflow runs in CI above). The DBA reviews a concrete artifact on their own schedule. The migration has already run against a real-data CI branch, so the DBA does not need to mentally simulate the change.

Mechanics:

• The DBA is a CODEOWNER on schema-affecting paths (migrations/, db/, schema test directories).
• The DBA reviews on the PR like any other reviewer, async.
• Review focus shifts from will this break the database to is this the right design? Has it been implemented the right way? Topics: data integrity rules, indexing strategy, design cohesion, future extensibility, long-term maintainability.

Anti-pattern. Not including the DBA in the PR flow with there being legitimate artifacts to review.

The DBA role also gains new responsibilities in policy administration and governance at team scale. Part 3 covers those.

Practice #2: All database artifacts are version controlled with application code

Rule. Every SQL file, migration script, and schema test lives in the same repository as the application code. The schema diff and migration test results join the artifact set as PR-time outputs.

Why is this a durable habit now? Branching adds two new artifacts to the set: the per-PR schema diff and the per-PR schema migration test results. Both are generated by CI from the actual branch state. Both land in the PR as concrete evidence about what was changed and how the change migration script performed.

Mechanics:

• Migration files in a versioned directory (migrations/,  db/migration/, alembic/versions/,  framework-dependent).
• Schema-affecting tests in the test tree alongside application tests.
• Schema diff generated per PR by CI, posted as a PR comment.
• Migration test results in the CI run summary and PR comment.

Anti-pattern. Generating the schema diff outside the PR flow (a separate dashboard the reviewer has to open). The artifact has to live where the review happens. Since the schema changes are tied to the code changes and breaking this dependency creates downstream problems for deployment.

Practice #3: All database changes are migrations

Rule. No manual ALTER TABLE against any environment. Every schema change is a versioned, checked-in migration script. Migrations are idempotent.

Why is this a durable habit now? The migration-as-artifact rule is unchanged from 2003. What is new is the authorship discipline of idempotency. The same migration runs against many branches over the life of a transition, so it has to behave the same way every time. A migration that fails on re-apply is a bug.

Mechanics:

• Use migration frameworks like flyway, liquibase, Knex, Alembic or others, these frameworks keep track of which migrations have been run and which have not been, this allows the team to apply a command like flyway migrate which just applies the changes that have not been applied (by keeping track of the changes in a metadata table)
• It’s better to split irreversible work across migrations. For example a migration that adds new columns and drops the source column in one shot makes rollback harder than they need to be, so using the expand first and contract later strategy, provides many more options after a deploy cycle confirms no live readers reference it.
• The 2006 database refactoring catalog names which refactorings are reversible and which are not. Use it.

Anti-pattern. A migration that depends on the schema being in a specific intermediate state because of local changes made in the branch. The migration must apply correctly against any parent branch that includes prior migrations.

Practice #4: Everybody gets their own database instance

Rule. Every developer, every PR, every experiment, every test run gets its own Lakebase branch.

Why is this a durable habit now? The extra effort to create docker containers, to install local database servers, acquire licenses, hydrate empty databases with existing schema and test data is no longer required. Just a simple create-branch Lakebase command branches a 1TB database in the same one second as a 1MB database. No data is copied at creation; only modified pages consume storage. Per-developer, per-PR, and per-experiment instances are routine.

Mechanics:

• Per-developer branches: created on demand via databricks postgres create-branch or the SCM extension's branch-create flow.
• Per-PR branches: created automatically by CI on PR open (pr.yml), destroyed on merge or close (merge.yml). See How the workflow runs in CI above for the PR and merge snippets.
• Per-experiment branches: forked off staging or production for design exploration; discarded after the experiment.

Anti-pattern. Sharing a development database across the team "for convenience." The contention-driven serialization Part 1 named comes back the moment the branches collapse.

Where Jen's example extends. Her per-developer branch was forked off staging at feature start. The CI branch was forked off staging on PR open. Her A/B exploration branches (Practice #9) were forked off staging in parallel. Four branches across one feature, all in seconds, all isolated.

Practice #5: Developers continuously integrate database changes

Rule. Every PR runs through CI against a fresh Lakebase branch, with migrations applied and tests run against real Postgres.

Why is this a durable habit now? The CI pipeline has had migration discipline since 2010. What is new is per-pipeline isolation: each PR gets its own branch, so integration runs against real-shaped data without contention.

Mechanics:

• CI creates a branch on PR open. See How the workflow runs in CI above for the PR snippet.
• Migrations applied against the branch via lakebase-migrate apply.
• Application tests run against the migrated branch through the ORM, no mocks.
• Schema diff posted on the PR.
• Branch destroyed on merge or close.

Anti-pattern. Running PR validation against shared staging. The serialization comes back; the per-PR isolation property is lost.

Practice #6: All database changes are database refactorings

Rule. Schema changes follow named refactoring patterns: Split Column, Rename Column, Move Column, Replace Type, and so on. Each has explicit transition mechanics (keep old plus new in parallel, populate from old, swap readers, drop old).

Why is this a durable habit now? The 2006 catalog at databaserefactoring.com names 70+ refactorings with worked examples. What is new in 2026 is a cheap place to rehearse the transition mechanics: a developer branch absorbs the rehearsal; the CI branch verifies; production sees only the verified result.

Mechanics:

• Identify the refactoring by name. Look it up in the catalog.
• Apply the named transition mechanics on a per-developer branch. Validate against production-shaped data.
• Open the PR. CI runs the migration on its own branch and posts the diff.
• Merge once the diff and the test results land approval.

Anti-pattern. A one-off schema change that does not map to a named refactoring. The 70+ catalog covers the common cases; if you find yourself outside it, you are likely combining multiple refactorings into one migration and should split.

Where Jen's example extends. Her V87 migration is the Split Column refactoring: splitting inventory_code into location_code, batch_number, and serial_number. The catalog page at databaserefactoring.com/SplitColumns.html names the transition mechanics. Her branch was the rehearsal space; the PR's CI run was the verification.

Practice #7: Developers can update their databases on demand

Rule. A developer can refresh their branch's database state on demand: reset to the parent's current state, fork a fresh branch off production, discard an experimental branch, share a branch with a teammate. All in seconds.

Why is this a durable habit now? "On demand" in 2026 means one second, isolated, against production-shaped data. None of these operations consult ops calendars or DBA queues.

Mechanics:

• Reset: delete and recreate the branch from its parent.
• Fork off production: databricks postgres create-branch --source production.
• Discard: databricks postgres delete-branch.
• Share: hand the branch endpoint to a teammate for a pairing session.

Anti-pattern. Treating any branch as durable beyond its purpose. The migration is the durable artifact; the branch is the workspace.

Practice #8: Destructive testing as a default option

Rule. When the blast radius of a destructive action is zero, destructive testing becomes a daily option rather than a quarterly exercise.

Why is this a durable habit now? A branch resets in one second. Anything you do to a branch can be undone by creating a fresh one off the same parent. Destructive tests stop needing ops calendars and approval gates.

Things that now fit inside a normal feature cycle:

• "What happens if my migration fails halfway through the UPDATE statement?" Run it. Kill the process at 50%. Verify the rollback works. Reset.
• "What happens if a backup-restore is mid-way through when a failover triggers?" Simulate the partial state. Verify the application's behavior. Reset.
• "What is the actual time-to-recover for our DR runbook?" Run the runbook. Measure. Reset.
• “Does this migration run successfully against production shape data or size” verify it before c

Cultural effect. When reset costs nothing, teams stop treating the test database as a precious resource. Tests can be aggressive. Cleanup can be skipped, because the next branch starts fresh.

Where Jen's example extends. Before opening her PR, she took the production-shaped data on her branch and deliberately corrupted around one percent of the inventory_code values to look like edge cases: missing digits, embedded spaces, trailing whitespace. The kinds of artifacts that historical data accumulates. She ran her migration. Two rows failed her substring math. She fixed the script and re-ran. The branch absorbed the destructive test. Production never saw it.

Practice #9: A/B variant prototyping at the database level

Rule. When two designs are in contention, build them on parallel branches, compare against production-shaped data, and keep the better solution.

Why is this a durable habit now? Per-branch cost is near zero. Exploring two schema designs no longer requires picking one in advance, and it no longer requires a provisioning process most teams will not undertake for an exploratory question.

Mechanics:

• Create two branches off the same parent.
• Apply Design A's migration to one branch, Design B's migration to the other.
• Run the application against each. Measure what matters: query latency on the common read path, migration time at production volume, index footprint, lock contention under simulated load.
• Pick the better solution. Discard the sub optimal solution. Document the decision in the PR description so the next person who has to extend the schema knows what was considered and why this design was picked.

Anti-pattern. Running A/B prototypes without writing down the decision and the reasoning. The branches are gone in a second; the design decision should be permanent.

Where Jen's example extends. She considered two designs for the location/batch/serial feature: three new columns on the existing inventory table, or a separate inventory_attributes lookup table keyed by inventory_id, anticipating that more attributes would be added later. She built both on parallel branches off staging. She ran the application's read path against each, measured query performance against production-shaped data, and looked at how each migration would scale to production volumes. The lookup-table version performed worse on the common read path because every inventory display required a join. She shipped the columns version, threw away the lookup-table branch, and left a note in the PR description: Considered lookup-table version; rejected because the common read path becomes a join. Revisit if more than five attributes accumulate.

What Jen's New Playbook Shows

We’ve named the seven practices from 2003 with the limitations that kept five of them aspirational, and re-cast them for 2026 once branching landed, and added four new practices that branching enables. Eleven practices total in the new playbook for Evolutionary Database Development, 9 of which are explained above.

In Part 1 – Jen’s story: one feature, one database we saw Jen work through one feature: she paired a code branch with a Lakebase branch, ran a real migration against production-shaped data in seconds, tested without mocks, opened a PR with the schema diff posted inline, and merged with the migration applied and the ephemeral branches cleaned up. Database change became part of normal development.

In Part 3 – Jen's Team at Scale, we look at the playbook at fifty developers, the DBA's evolved responsibilities in policy administration and governance, and the agents creating branches alongside Jen. Practices #10 and #11 get their full treatment there.

The Companion: Plugin Walkthrough covers the Lakebase SCM Extension for VS Code and Cursor.

A Lakebase App Dev Kit for agents, with a companion e-book for human practitioners, will be released as a follow-on.