We Killed ETL: Unstructured.io Foundation
Overview
| Experience | In Person |
|---|---|
| Track | Artificial Intelligence & Agents |
| Industry | Enterprise Technology |
| Technologies | Delta Sharing |
| Skill Level | Beginner |
You’ve known Unstructured as the company to power your RAG applications; first we transformed your PDFs to JSON, then we built ETL pipelines for delivering chunked, embedded JSON to Delta Tables that enabled LLMs with low-latency RAG. Now we’re changing the model entirely. Imagine if agents could take context from your whole enterprise, without having to copy every document into Databricks? Foundation builds enterprise-wide context on demand. Source documents stay where they are. Extracted data lands in ephemeral Postgres provisioned via Neon, with pointers for the agent back to the originals. All your data outside the Databricks ecosystem is now workable from inside it. This talk is for every team still maintaining a pipeline.
Session Speakers
Christopher Maddock
/Head of Product and Engineering
Unstructured
Full Summary
Killing ETL for Gen AI: Why Unstructured is betting on zero-copy foundations
Enterprises are racing to make unstructured data usable for generative AI, yet most rely on bespoke retrieval chains built from connectors, chunkers, embedders, and vector stores. That approach is fragile and costly at scale. The talk outlines a pragmatic alternative: standardize transformations where they are clearly valuable, then move to a catalog-first, zero-copy pattern that lets agents find and materialize only what they need.
FAQ
The platform handles roughly 74 to 75 file types, including PDFs, PowerPoint, Word, images, and audio. It offers about 50 connectors spanning common sources and destinations such as S3, SharePoint, Salesforce, Snowflake, Kafka, Azure storage, and a range of object, vector, graph, and SQL stores.
Traditional catalogs track locations and governance metadata. Foundation also peeks inside files to extract entities, topics, and summaries, plus custom ontology fields, so agents can query by content rather than only by path or owner.
Foundation targets cost sensitive, latency tolerant scenarios. It avoids precomputing embeddings for entire corpora, which reduces storage and token costs. Pipelines suit cases where lower latency and predictable performance justify embedding and storing content up front.
Agents call Foundation's MCP server to query the asset catalog, iteratively narrow candidates, and request only the needed documents. If the result set is large, Foundation materializes the relevant slices into an ephemeral Neon database and returns them on demand.
Foundation was announced in early access and is gated to design partners. Interested teams are invited to sign up and collaborate on features and use cases.