Skip to main content
CUSTOMER STORY

Helping businesses keep their customers safe from cyberthreats

Palo Alto Networks drives scalable security to confront online dangers head-on with Databricks

40%

Reduction in time required for data preparation and movement 

3x

Decrease in engineering development time

3x

Faster iterations on AI/ML features shipped on Databricks compared with prior solution

SOLUTION: Threat Detection
CLOUD: AWS

Thank you Krishnan Narayan, Senior Distinguished Engineer at Palo Alto Networks, and Ram Katakam, Distinguished Engineer at Palo Networks, for their contributions to this story.

Palo Alto Networks is committed to advancing cybersecurity, with a vision to create a safer world every day. Driving digital transformation through best-in-class platforms, intelligence and services, the company aims to stay ahead in the continually evolving cybersecurity industry. Because of rapid technological growth in the AI era, Palo Alto began to prioritize cloud security, AI and ML integrations and a unified platform strategy. However, data fragmentation across their Prisma Cloud platform created challenges in managing governance, enhancing analytics and fostering collaboration. Tackling these hurdles, Palo Alto turned to the Databricks Data Intelligence Platform and achieved 3x faster iterations on AI/ML features on Databricks compared with the prior solution.

Dismantling data silos and point solutions for a better product

Palo Alto Networks is a known leader in cybersecurity, focused on safeguarding digital ecosystems through advanced solutions for organizations worldwide. With their Prisma Cloud product, the company planned to double down on securing cloud environments — as businesses increasingly shifted to cloud-based infrastructure. As part of their broader objectives, Palo Alto wanted to capitalize on the advancements in AI and machine learning to enhance their threat detection and response capabilities. At the same time, the team was embarking on a “platformization” strategy to meet the industry’s shift toward comprehensive security solutions. As they moved away from isolated, single-purpose tools, it allowed them to integrate various security functions into a unified platform, making it simpler and more efficient for clients to manage their cybersecurity needs in one place. Plus, Palo Alto’s investment in this initiative aimed to fuel the growth of their next-generation security (NGS) business, which reinforced the company’s attempt to tackle the growing complexity of the sophisticated cyberthreat landscape.

However, implementing this initiative revealed substantial challenges within Palo Alto’s data infrastructure. Each Prisma Cloud module operates with its own storage mechanisms, data models and governance protocols. Inevitably, this led to data silos that hindered a cohesive security strategy, limited threat visibility across platforms and placed a heavy burden on DevOps, who were responsible for managing data access and movement. Without a shared operating system for analytics, Palo Alto’s teams faced various obstacles in developing reusable analytical tools, restricting the pace of response to new threats and slowing innovation. “Operating in silos limited our ability to see the full picture and respond quickly to threats,” Krishnan Narayan, Senior Distinguished Engineer at Palo Alto Networks, explained. “We needed a unified platform to break down barriers between modules, streamline data access and enable our teams to develop reusable tools without the bottlenecks caused by disconnected systems.” Wanting to address all these issues in one fell swoop, Palo Alto Networks adopted the Databricks Data Intelligence Platform.

Streamlining data access to elevate threat detection and more

To establish a streamlined and secure data infrastructure for their Prisma Cloud platform, Palo Alto Networks utilized Databricks’ data and AI tools, starting with Unity Catalog as a foundational component. Integrated with S3 for storage and SQL databases, Unity Catalog enabled Palo Alto to centralize and standardize data cataloging across their multiple cloud modules, a necessity for the organization’s complex data environment within Prisma Cloud. This centralization minimized the need for frequent cross-team coordination, allowing teams to access and manage data independently through Unity Catalog’s built-in access controls and guardrails. By improving data governance and simplifying access, Unity Catalog helped Palo Alto reduce security risks, enforce governance protocols consistently and contain costs for high-demand data operations by applying job and concurrency limits on resource-intensive queries.

Unity Catalog’s data governance supported Palo Alto’s initiative to provide real-time security recommendations to customers. With Unity Catalog, they could assess security issues across a customer’s cloud environment, identify areas of vulnerability and recommend prioritized action plans. Delta Lake — a storage layer supporting ACID transactions — and Delta Live Tables, which automates data pipelines, provided the essential infrastructure for managing real-time information, including system configurations, threat alerts and usage patterns. Delta Lake enabled Palo Alto to maintain up-to-date digital twins, or real-time virtual replicas, of each customer’s Prisma Cloud environment to track any new data, configuration updates or system changes. Delta Live Tables further automated data transformation, ensuring that highly dynamic data remained updated without complex engineering workflows. This robust, automated foundation supported their ability to provide timely, actionable steps for security measures. “With Delta Lake and Delta Live Tables, we could finally automate real-time data handling and maintain precise, current views of each customer’s cloud environment, which was critical for delivering proactive security responses,” Krishnan detailed.

Additionally, Databricks SQL and autoscaling clusters provided the on-demand infrastructure necessary for refining threat detection within complex cloud infrastructures. This included processing large datasets such as real-time security alerts, customer environment configurations, threat intelligence and behavioral data patterns. Databricks SQL, designed for high-performance analytics queries, helped rapidly populate data into low-latency optimized indexes for dashboards and analytics reports.

While supporting these efforts, Databricks played a crucial role in Palo Alto Networks’ generative AI (GenAI) strategy, which is part of their Precision AI initiative — fully integrated with the Prisma Cloud — aimed at elevating the company’s security capabilities to better serve customers. This initiative was focused on three key areas: Secure With AI, which leverages AI to proactively detect and counteract cyber threats across customer environments; Secure the AI, which safeguards Palo Alto’s AI models and data pipelines against potential threats; and AI Experiences, which enhances customer-facing, AI-driven applications to deliver more intuitive and effective security tools. By centralizing and organizing data, Databricks keeps data safe, well-governed and accessible across teams. The unified data environment empowered Palo Alto’s precise, AI-driven insights to their customers to take on the complex challenges common in cybersecurity.

Driving efficiency and accelerating innovation at scale

In the first year of adopting Databricks, Palo Alto Networks has already seen a substantial impact throughout their business that directly benefits both customers and internal operations. By centralizing data workflows and reducing the complexity of existing data processing workflows, Databricks has contributed to approximately 3x faster iterations on AI/ML features. Operational costs have also dropped, with Palo Alto reporting a 20% reduction in the cost of goods sold (COGS) and a 3x decrease in engineering development time. Now, their teams can deliver faster security solutions with a 50x greater scale capability than previous systems, marking a transformative shift in how quickly Palo Alto processes data and brings impactful features to market.

IT operations have similarly benefited from Databricks’ streamlined platform and reduced resource consumption and onboarding times. Over 20 engineers became productive on Databricks within just two months, accelerating deployment timelines. Training efforts cut down the usual number of iterations needed to achieve production-ready rigor, leading to quicker rollouts of new features and less time spent on monitoring and design. With this level of efficiency, multiple teams have freed up time to focus on application-specific use cases, which now require 40% less time in data preparation and movement across silos.

Looking forward, Palo Alto Networks views Databricks as an integral part of their AI and ML strategy, particularly in maintaining the Prisma Cloud platform as a unified data environment. With Databricks, Palo Alto can develop, deploy and test more AI-driven security solutions and set the stage for advanced features once growth is stabilized and data silos are fully resolved. Krishnan concluded, “With Databricks, we’ve turned data complexity into an advantage. The platform’s scalability has reshaped our approach, accelerated our security innovation and helped us to deliver impactful features to customers with unprecedented speed.”