Chris Stevens

Imagine an organization with thousands of users who want to run data analytics workloads. These users shouldn’t have to worry about provisioning instances from a cloud provider, deploying a runtime processing engine, scaling resources based on utilization, or ensuring their data is secure. Nor should the organization’s system administrators.

In this talk we will highlight some of the exciting problems we’re working on at Databricks in order to meet the demands of organizations that are analyzing data at scale. In particular, data engineers attending this session will walk away with learning how we:

• Manage a typical query lifetime through the Databricks software stack
• Dynamically allocate resources to satisfy the elastic demands of a single cluster
• Isolate the data and the generated state within a large organization with multiple clusters

Meltdown and Spectre are two security vulnerabilities disclosed in early 2018 that expose systems to cross-VM and cross-process attacks. They were the first of their kind and opened up a new class of exploits that allow one program to scan another program’s memory. The kernel and VM patches released to address these vulnerabilities have shown to degrade the performance of Apache Spark workloads in the cloud by 2-5%.

This talk will dive deep into the exploits and their patches in order to help explain the origin of this decline in performance.

Session hashtag: #DD6SAIS