From Unstructured Data to Structured Insights: How YipitData Scales Data Enrichment with AI Agents
Overview
| Experience | In Person |
|---|---|
| Track | Artificial Intelligence & Agents |
| Industry | Enterprise Technology, Retail & Consumer Goods, Financial Services |
| Technologies | Databricks Apps, Databricks Agents, Lakebase |
| Skill Level | Intermediate |
| DOWNLOAD SESSION SLIDES | |
YipitData analyzes billions of unstructured data points at petabyte scale to deliver high fidelity insights to institutional investors and Fortune 500 companies. With 100s of heterogeneous data sources, regex, classic ML and NLP techniques never met our accuracy hurdle, limiting our product breadth for years.
In this session, we reframe entity resolution as an agentic problem and share our production-grade enrichment platform built on Apache Spark™, Agent Bricks, Vector Search and Lakebase. This AI-native architecture continuously discovers and tags data at 90%–95% accuracy, reliably covering 60,000+ companies—a 20x improvement.
Data/ML leaders, engineers and practitioners will learn:
- Modular, pipeline design applicable to any classification scenario
- Batch inference patterns in Spark that streamlines infrastructure
- Techniques for continuous, low maintenance entity discovery
Join us and leave with a blueprint to turn enrichment bottlenecks into self-improving, AI pipelines.
Session Speakers
Anup Segu
/Chief Architect
YipitData
Edward Goo
/Head of Data Engineering
YipitData
Full Summary
How YipitData Uses AI Agents to Turn Messy Transaction Data Into Trusted Insights
Data teams are realizing that AI quality is capped by the structure and governance of the data underneath it. YipitData, a research provider serving institutional investors and large enterprises, rebuilt its enrichment pipelines around agentic reasoning to classify messy transaction strings at petabyte scale. The approach aligns with modern patterns for AI-powered analytics on governed data, replacing brittle rules with a retrieval-grounded reasoning loop that delivers accuracy and scale. Teams exploring how this reasoning-first approach fits into broader platform strategy will find useful context in how AI is transforming data analytics across industries and use cases.
FAQ
Use it when data is genuinely ambiguous and resists deterministic rules. Regex and traditional NLP remain effective for clean, well-structured cases. Prove the agentic loop on a hard subset first, then scale only after evaluation shows clear quality gains.
Constrained selection eliminates most hallucinations because the model must choose from vetted candidates. That framing lets the LLM apply business rules and hierarchy knowledge, rather than guess from an unbounded space of entities.
Most AI tooling is optimized for single-request, real-time chat. Processing millions of rows requires orchestration, concurrency, incremental processing, and checkpointing, which teams must assemble from platform primitives like Lakeflow Jobs, UDTFs, and Delta Lake.
Prefer serverless GPUs for incremental workloads that benefit from fast startup and model flexibility. Self-hosted GPUs make sense for long-running historical backfills where shared capacity variability hurts throughput and predictability.
The ontology stores entities with rich attributes, including products, categories, tickers, competitors, and websites. Once a transaction is attributed, a simple join enriches outputs with these fields, turning classification into a broader enrichment platform.