Mo’ Agents Mo’ Problems: How We Simplified Lakehouse Threat Hunting as a Service at Scale

Security teams face an impossible ratio of millions of events, thousands of alerts, and a handful of analysts. Most hunt reactively or not at all. We built a threat hunting platform on Databricks that flips this equation. Our architecture spans the full stack: Lakeflow DLT normalizes 20+ sources into OCSF, billions of events are governed in Unity Catalog, and detection pipelines form the foundation for specialized AI agents that hunt autonomously. These agents generate hypotheses, investigate the lakehouse, enrich IOCs via MCP tools, and produce findings without prompting. The critical constraint: agents never see raw telemetry. They query pre-aggregated views, keeping billion-row datasets out of the LLM context while preserving investigative depth. You'll learn how we architected safe agent-to-lakehouse communication, orchestration patterns for autonomous hunt cycles, and detection at scale, while hearing about our service delivery at petabyte scale.