Normalising Chaos: Atlassian’s OCSF Security Lakehouse on Databricks

Atlassian’s security data lake, built on Delta Lake and standardized to the OCSF schema, powers 500+ production threat detection signals and enables fast, cost‑efficient investigations on open lakehouse architecture.We’ll share hard‑won practices: adopting OCSF to unify schemas of relevant security logs and accelerate development, a ruthless, value‑vs‑cost log prioritization rubric and Databricks optimizations that unlocked capabilities like “conversational incident response” and live ML threat detections.Learn how Atlassian achieved:

• 80% ingestion cost reduction moving to file notification loader
• Scheduled jobs running queries in seconds from minutes
• Day‑long searches accelerated from minutes to seconds with Z‑ORDER and OPTIMIZE
• We’ll cover PySpark patterns, and collaboration spaces, reusable libraries and notebooks that make IR handoffs repeatable. Walk away able to design OCSF‑first schemas, implement cost‑aware ingestion, and operationalize ML detections on an open lakehouse