Redefining Detection Engineering With Interactive AI Agents in the SOC

Grounded in work with a hedge fund and the evolution toward Lakewatch on the Databricks Platform, the session shows how to turn agents into analyst multipliers, not auto-pilots.

The core idea is simple. Writing SQL is easy. Knowing what to detect, what normal looks like, and what thresholds are safe is hard. Agents add the most value when they expose ambiguity, standardize repeatable steps, and hand key decisions back to humans.

Why full automation misses the point in detection engineering

Detections are just queries with filters, but good filters depend on deep context: who is allowed to do what, what behavior varies by role or host, and which anomalies matter. That understanding is built iteratively. Left to a vague brief like "write a detection for internal data movement," a coding model tends to produce generic rules. In the demo, an unguided model caught 1 of 30 malicious events while generating about 16,000 alerts, underscoring why quality suffers when agents rush to closure.

Large language models are biased toward resolution. They optimize for finishing the task, not pausing to clarify requirements or validate assumptions. Effective agent design in the SOC deliberately slows the model down so the human can speed up.

Designing agents that ask before they act

Practical patterns emerged. First, co-design with the people already doing the work. Let detection engineers pilot the agent during buildout so the workflow, terminology, and outputs match how detections are actually reviewed and deployed.

Second, extract the repeatable steps into a small set of parameterized tools. Schema inspection, percentile baselining, Z-scores, and backtesting can be captured as Python tools, avoiding open-ended SQL generation while improving consistency. In the case study, roughly 90 percent of baselining needs collapsed into about six tools, which also standardized how detections were authored across teams.

Third, reframe the agent's goal through prompting. Instead of "produce a detection," direct the agent to explore the data, surface ambiguities it cannot resolve, propose segmentations, and backtest candidate thresholds. A dedicated evaluation step that simulates alert volume over historical windows helps prevent rules that either miss real activity or flood the queue.

From custom LangGraph to Lakewatch on the Databricks Platform

What took thousands of lines of LangGraph orchestration is now far lighter to build and operate. Lakewatch, an open agentic SIEM built on the Databricks Platform, brings native data ingestion, detection rule management, and security case workflows together with Genie for interactive SQL, rule population, and case creation. Behavioral guardrails that once lived in bespoke tools can be expressed as markdown skills that Genie consumes.

In the demo, a skill-driven workflow used a temporal baseline to flag successful sign-ins from never-before-seen countries, generated and ran the query on serverless compute, and then created both a detection rule and a security case for analyst review. Early adopters have reported 60 to 80 percent TCO savings, and Adobe is processing about 21 petabytes on the platform. Lakewatch also emphasizes human-in-the-loop controls at each step, aligning automation with analyst judgment rather than replacing it.

Put leverage first, not autonomy

Agents that respect analyst judgment consistently outperform those that chase end-to-end autonomy. Constraining agent behavior, standardizing repeatable steps, and requiring human sign-off at decision points produces detections that are faster to build and safer to deploy. Invest early to slow the agent down, and the team speeds up.