Sync Tables: Unity Catalog to Lakebase (Autoscaling)

Prerequisites

This template creates a synced table that mirrors a Unity Catalog table into Lakebase Postgres. Verify these Databricks workspace features are enabled before starting.

• Databricks CLI authenticated. Run databricks auth profiles and confirm at least one profile shows Valid: YES. If none do, authenticate with databricks auth login --host <workspace-url> --profile <PROFILE>.
• Lakebase Autoscaling available. Run databricks postgres list-projects --profile <PROFILE> and confirm your Autoscaling project is listed. A not enabled error means Lakebase is not available to this identity.
• Project created via the /database/ API (not the older /postgres/ API). Programmatic synced-table creation via databricks database create-synced-database-table only works on projects created through the newer /database/ API. If your Autoscaling project was created via the older /postgres/ endpoint, the CLI path in Step 1 is not available yet and you must create synced tables through the Databricks UI (Catalog → source table → Create synced table). This gap is expected to close in a future release.
• Unity Catalog source table with a primary key. Run databricks tables get <CATALOG>.<SCHEMA>.<SOURCE_TABLE> --profile <PROFILE> and confirm at least one column is declared as the table's primary key. Synced tables reject sources without a PK.
• External-storage catalog for the source (currently required for Sync Tables). Sync Tables today requires the source UC catalog to use external storage. If your source catalog uses the metastore's default managed storage, complete the Unity Catalog Setup template first and move the source table into an external-storage catalog.
• Change Data Feed enabled on the source table (for Triggered / Continuous mode only). Skip this check if you plan to use Snapshot mode. Otherwise run the ALTER TABLE <catalog>.<schema>.<table> SET TBLPROPERTIES (delta.enableChangeDataFeed = true); statement from Step 1 against your SQL warehouse.

Sync a Unity Catalog Table to Lakebase

Serve lakehouse data through Lakebase Autoscaling Postgres so your applications can query it with sub-10ms latency. This creates a synced table, a managed copy of your Unity Catalog table in Lakebase that stays up to date automatically.

This template is for Lakebase Autoscaling (projects/branches/endpoints with scale-to-zero). For Lakebase Provisioned (manually scaled instances), see the Provisioned Sync Tables template (coming soon).

When to use this

• Your app needs fast lookup-style queries against analytics data (user profiles, feature values, risk scores)
• You want to serve gold tables, ML outputs, or enriched records through a standard Postgres connection
• You need ACID transactions and sub-10ms reads alongside your operational state

Choose a sync mode

Triggered and Continuous modes require Change Data Feed (CDF) enabled on the source table. If it's not enabled, run:

sql

ALTER TABLE <catalog>.<schema>.<table> SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

Sync throughput

Autoscaling CUs are physically 8x smaller than Provisioned CUs, so per-CU throughput differs:

A 10x speedup for large-table snapshot sync (writing Postgres pages directly, leveraging separation of storage and compute) is coming for Autoscaling only.

1. Create a synced table

databricks database create-synced-database-table \
  --json '{
    "name": "<CATALOG>.<SCHEMA>.<SYNCED_TABLE_NAME>",
    "database_instance_name": "<INSTANCE_NAME>",
    "logical_database_name": "<POSTGRES_DATABASE>",
    "spec": {
      "source_table_full_name": "<CATALOG>.<SCHEMA>.<SOURCE_TABLE>",
      "primary_key_columns": ["<PRIMARY_KEY_COLUMN>"],
      "scheduling_policy": "<SNAPSHOT|TRIGGERED|CONTINUOUS>",
      "create_database_objects_if_missing": true
    }
  }' --profile <PROFILE>

If your Lakebase database is registered as a Unity Catalog catalog, you can omit database_instance_name and logical_database_name.

Verify:

databricks database get-synced-database-table <CATALOG>.<SCHEMA>.<SYNCED_TABLE_NAME> --profile <PROFILE>

Important: If your Autoscaling project was created via the /postgres/ API (not /database/), programmatic synced table creation is not yet available via CLI. Use the Databricks UI as a fallback. In Catalog, select the source table → Create synced table, then choose your Lakebase project, branch, sync mode, and pipeline. This gap is expected to close soon.

2. Configure pipeline reuse

How you set up pipelines depends on your sync mode:

3. Schedule ongoing syncs

The initial snapshot runs automatically on creation. For Snapshot and Triggered modes, subsequent syncs need to be triggered.

Note: Table-update triggers for sync pipelines are not yet available via CLI and must be configured through the Databricks UI: Workflows → create/open a job → add a Database Table Sync pipeline task → Schedules & Triggers → add a Table update trigger pointing to your source table.

Trigger a sync update programmatically via the Databricks SDK:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

table = w.database.get_synced_database_table(
    name="<CATALOG>.<SCHEMA>.<SYNCED_TABLE_NAME>"
)
pipeline_id = table.data_synchronization_status.pipeline_id

w.pipelines.start_update(pipeline_id=pipeline_id)

4. Query the synced data in Postgres

Once synced, the table is available in Lakebase Postgres. The Unity Catalog schema becomes the Postgres schema:

SELECT * FROM "<schema>"."<synced_table_name>" WHERE "user_id" = 12345;

Connect with any standard Postgres client (psql, DBeaver, your application's Postgres driver).

What you end up with

• A synced table in Unity Catalog that tracks the sync pipeline
• A read-only Postgres table in Lakebase that your apps can query with sub-10ms latency
• A managed Lakeflow pipeline that keeps the data in sync based on your chosen mode
• Up to 16 connections per sync to your Lakebase database

Important constraints

• Primary key is mandatory. Synced tables always require a primary key. It enables efficient point lookups and incremental updates. Rows with nulls in PK columns are excluded from the sync.
• Duplicate primary keys fail the sync unless you configure a timeseries_key for deduplication (latest value wins per PK). Using a timeseries key has a performance penalty.
• Schema changes: For Triggered/Continuous mode, only additive changes (e.g., adding a column) propagate. Dropping or renaming columns requires recreating the synced table.
• FGAC tables: Direct sync of Fine-Grained Access Control tables fails. Workaround: create a view (SELECT * FROM table), then sync the view in Snapshot mode. Caveat: runs as the sync creator and only sees their visible rows.
• Connection limits: Autoscaling supports up to 4,000 concurrent connections (varies by compute size). Each sync uses up to 16 connections.
• Read-only in Postgres: Synced tables should only be read from Postgres. Writing to them interferes with the sync pipeline.

Cost guidance

Cost formula: [Rows / (Speed × CUs × 3600)] × DLT Hourly Rate

Example costs (181M rows, 1 CU, $2.80/hr DLT rate):

Troubleshooting

References

• Synced tables (Autoscaling)
• Change Data Feed
• Lakebase Autoscaling
• DevHub: Data Lakehouse overview