Reimagining Data Modeling on the Lakehouse: Introducing Vibe Data Modeling

The challenges with Data Modeling

In every analytics stack, the Silver layer is where it is made or broken. BI and dashboards read from Gold; Gold is built from Silver. The Silver-layer model is the foundation every analyst, data scientist, and BI tool depends on. If Silver is messy, ungoverned, or full of duplicates, everything above it gets harder, slower, and more expensive.

Getting there has always been the problem. Most organizations either spend six months to three years hand-building a Silver model from scratch, or they buy a generic industry template (ACORD for insurance, FHIR for healthcare, ARTS for retail, TM Forum SID for telecom) and then spend nine to twelve months trimming, renaming, and rewiring it. A template is the average of a whole sector: typically 20 to 40 percent is relevant, and it was built for no specific business. Neither path keeps up with how fast modern data products need to ship.

Today, we are announcing Vibe Data Modeling

Vibe Data Modeling is a multi-model LLM agent that turns a plain-English description of your business into a complete, governed, deployable Silver-layer data model. It ships as a single notebook: four widgets, one run, a fully deployed model in Unity Catalog. If you do not like what came out, you “vibe” it in plain English until it fits.

• Hours, not months: a deployed Minimum Viable Model in under two hours, an Expanded Coverage Model in a single afternoon.
• 100% relevant to you: it uses your terminology, divisions, and domains, not a sector average.
• Trustworthy by construction: 251 enforceable rules, two architect reviews, and an agentic loop that proves the model before it ships.
• Native Unity Catalog deployment: schemas, tables, foreign keys, classification tags, metric views, an RDFS ontology, a DBML diagram, and sample data, generated and versioned together.

User vibes are the supreme authority

One principle governs the whole agent: what you say wins. An explicit instruction in a widget, in model_vibes, or in your business description outranks every heuristic, scoring formula, gate, and LLM opinion in the pipeline. If you say “exactly 10 domains,” no tier classifier may add an eleventh.

How a vibe becomes a data model

Behind the four widgets, the agent runs a pipeline in four stages: it understands your input, designs the model top-down, connects it with relationships and metrics, then deploys. Each stage validates before the next begins, so only a clean stage advances. Underneath, it is a multi-model ensemble: a large thinker model handles reasoning and reviews, a large worker generates the high volume of products and attributes, smaller models handle domains, tagging, and sample data, and a judge scores competing proposals on one rubric. The roster self-heals, demoting a failing model and restoring it once healthy.

How the model is organized

Every model follows the same shape, top to bottom: organization, divisions, domains, subdomains, products, attributes. At the top sit the three divisions almost every organization shares: Operations (what they do), Business (who they serve), and Corporate (how they work). Operations and Business are the core; Corporate is the supporting minority. A domain is a bounded context that owns a distinct set of concepts; a product is a real business concept a domain expert would recognize (an invoice, an order), never plumbing or analytics; and every attribute has to earn its place.

A single source of truth, and a clean graph

Two structural guarantees keep the model coherent, and both are enforced. Single source of truth means one concept has exactly one owning product; a customer is defined once in customer.customer and everyone else references it by foreign key. And the relationships form a directed acyclic graph: foreign keys point child to parent, never in a cycle, no product is left siloed, and redundant columns are normalized away when a key lands.

The rules that make it trustworthy

The agent enforces 251 rules across 20 groups. The structural ones are deterministic gates that read the real model dictionary, so they cannot be talked out of a verdict, and they run as the model is built and again at the install gate against the deployed model. The quality score the run reports is computed from the model itself, not the LLM’s self-assessment.

The agentic loop: generate, validate, retry differently

A single LLM pass is never trusted as final. The loop generates one concrete attempt, validates it against the deterministic gates and static analysis, and on failure changes strategy rather than repeating. Unsatisfied requirements and structural residuals (denormalized keys, cross-domain duplicates, unlinked or cyclic foreign keys) route to a sandboxed repair step and back through validation. A monotonic guard reverts any pass that makes the model worse, so it can only improve or hold.

How a vibe is verified

When you iterate, your request is parsed into structured verification requirements (VREQs), each a discrete, checkable directive. Each is applied by a sandboxed mutator and verified independently, deterministically where possible: the gate reads the real model and the physical Unity Catalog rather than asking an LLM whether the change happened. The run reports an adherence score, and anything unverified is requeued rather than quietly dropped.

Two architect gates

Rules catch what is mechanically wrong; the architect gates catch what is structurally unwise. The Domain Architect reviews each domain in isolation; the Global Architect reviews the whole model for cross-domain duplicates, single-source-of-truth violations, and structural integrity. Findings are applied automatically, tracked as landed, regressed, or blocked, and the review reruns up to eight passes until clean.

What you get from one run

• A logical model (model.json) with every domain, product, attribute, foreign key, and classification tag.
• A physical deployment in Unity Catalog: schemas, tables, foreign keys (informational), and classification tags.
• Unity Catalog metric views: reusable KPI definitions on the products, ready for AI/BI dashboards and Genie.
• An RDFS ontology for semantic tools and AI agents, and a DBML file for dbdiagram.io.
• Synthetic sample data generated against the same model, plus a full pipeline log and a next_vibes file of suggested refinements.

model.json: one source of truth

Everything the agent produces derives from one artifact, model.json. The physical deployment, the ontology, the DBML diagram, the metric views, the sample data, the docs, and the next_vibes suggestions are all generated from it. Nothing is authored twice, so the logical model and every downstream artifact can never drift apart.

What lands in Unity Catalog

When you set a deployment catalog, domains become schemas, products become Delta tables, attributes become columns; foreign keys are applied as informational constraints; classification tags (PII, glossary, provenance) are applied as it builds; and metric views land on top.

Two scopes: MVM and ECM

Most teams do not need every domain on day one, so the agent produces two scopes from the same engine. The Minimum Viable Model is the lean core, built first; the Expanded Coverage Model is full coverage across the whole business. You can build either, shrink an ECM into an MVM, or enlarge an MVM into an ECM, and the shrink is LLM-guided so it protects the core products.

Vibe it until it fits

Refinement is where Vibe Data Modeling earns its name. v1 is the base model and it evolves forward, never sideways: no version is overwritten, and every iteration is auditable and reversible. Changes come in three intent modes: surgical (fix exactly this), holistic (apply everywhere), and generative (create something new), all under the same rules and reviews.

One agent, six operations

The same notebook does more than build a first model. The operation widget selects one of six operations, all sharing the same rules, architect gates, and agentic loop.

How to vibe a version (VOV)

To vibe an existing version, select the “vibe modeling of version” operation, point it at the version to build on, and write your changes in plain English (or paste the suggestions from next_vibes.txt). The agent parses them into VREQs, reruns the pipeline on top of that version, and writes a new numbered version; the one you started from is untouched.

One logical model, many physical layouts

The logical model is one artifact; the physical layout is a separate decision controlled by a single widget. The same model can be rendered as one catalog, a catalog per division, or a catalog per domain. If your governance reality changes, you redeploy to a different convention; the logical model is unchanged.

Industry templates are not enough

The argument for a generic template was always the head start. The reality, learned the hard way, is that the head start costs nine to twelve months of fitting and renaming. A template is the average model for a sector; by construction it is nobody’s actual business. Vibe Data Modeling produces a model in your terminology, with your divisions and domains, generated in hours and validated by the same rules every other model is.

Example models built with the agent

The same industry-agnostic agent has produced full-business Expanded Coverage Models across very different sectors, each referencing the recognized standards for its industry. The counts below are the published reference models in the open-source repository.

Available today

The reference implementation is a single Databricks notebook at agent/dbx_vibe_modelling_agent.ipynb. Fill in the four core widgets and run; everything else defaults from your industry.

A concrete starting point: here is the prompt we used to generate a manufacturing model, and the first plain-English vibe we sent to refine it.

• Reference repository (github.com/databricks-industry-solutions/lakehouse-industry-data-models): the agent notebook, the orchestrator, a test harness, 40+ open-source reference models, and guides covering design, integration, quality gates, and the rule catalog.
• The Vibe Data Modeling whitepaper: the full technical treatment of every pipeline stage, the complete rule catalog, the architect-review methodology, and the ensemble architecture.

If your team has been carrying a Silver-layer project for months without shipping, this is the shortest path we have found to actually shipping one. Describe your business in plain English, get a model, iterate until it fits, and put it into production.