Scaling Cybersecurity Analytics Without Scaling Costs
Overview
| Experience | In Person |
|---|---|
| Track | Cybersecurity |
| Industry | Enterprise Technology, Travel & Hospitality |
| Technologies | AI/BI, Databricks SQL, Unity Catalog |
| Skill Level | Intermediate |
Cybersecurity teams often struggle with the cost and scale of analyzing massive telemetry. JetBlue solved this by building a security-first analytics platform on the Databricks lakehouse.
By leveraging Delta Lake, Auto Loader, and Databricks SQL, JetBlue ingests high-cardinality data into low-cost cloud storage, applying compute only when necessary. This enables long-term retention and flexible analysis that legacy tools find prohibitively expensive.
Using Databricks SQL, they developed a fastly-focused security layer to:
- Detect anomalies and bot abuse
- Measure CAPTCHA effectiveness
- Analyze attack trends by IP, geography, and path
- Support real-time and historical investigations
- Decoupling storage from compute resulted in faster investigations, better visibility, and significant savings. Attendees will learn how to use Databricks to scale detection and investigation workflows cost-effectively.
Session Speakers
Greg Speranza
/Engineer IT Security
JetBlue Airways
Full Summary
How JetBlue rebuilt security analytics on the Databricks Platform for scale, speed, and cost control
Security programs are overwhelmed by rapidly growing telemetry, high cardinality data, and SIEM pricing that penalizes retention. JetBlue rearchitected its security analytics over roughly 18 months around the Databricks Platform to cut SIEM dependence, extend historical visibility, and deliver analyst-friendly tools that accelerate investigations.
FAQ
SIEM indexing and retention pricing is expensive, which limits how long data can be kept and slows historical investigations. Storing raw telemetry in ADLS and processing it with Spark enables far longer retention at lower cost while reserving the SIEM for use cases that truly benefit from its indexing.
Auto Loader incrementally discovers new files as they land, so ingestion scales with bursty or continuous sources without manual tweaks. Pipelines can run as frequently as every 5 to 15 minutes or on daily cadences, matching each source's characteristics.
No. Databricks Apps provide point-and-click interfaces with filters, regex search, and custom time ranges. Some analysts use SQL for specific tasks, but the goal is to avoid requiring KQL or SPL for day-to-day work.
About 200 terabytes of telemetry currently reside in ADLS, with the earliest datasets going back to 2024 and growing as new sources are added. Dashboards often provide a fast 90-day lookback via materialized views, and longer-term investigations are now feasible.
After enabling CAPTCHA on www.jetblue.com, analysts observed a spike in invalid tokens as a threat actor hit a flight search endpoint without valid tokens. The WAF blocked the traffic, and the app made the impact immediately visible for operations and reporting.