Chaos Engineering in the World of Large-Scale Complex Data Flow
- Data Engineering
- Moscone South | Upper Mezzanine | 155
- 35 min
A complex data flow is a set of operations to extract information from multiple sources, copy them into multiple data targets while using extract, transformations, joins, filters, and sorts to refine the results. These are exactly the capabilities that the new open modern data stack provides us. Spark and other tools allow us to develop complex data flows on large-scale data. Chaos Engineering concepts discuss the principles of experimenting on a distributed system to build confidence in the system’s capability to withstand turbulent conditions in production. Or, how stable is your distributed system?
We tend to adopt practices that improve the flexibility of development and the velocity of code deployment, but how confident are we that the complex data system is safe once it arrives in production? We must be able to experiment in production, automate actions while minimizing customer pain and reducing damage for code and data. If your product’s value is derived from data in the shape of analytics or machine learning, losing it, or having corrupted data can easily translate into pain. In this session, you will discover how chaos engineering principles apply to distributed data systems and the tools that enable us to make our data workloads more resilient. We will also show you how to recover from deploying Spark code on a databricks environment that resulted in corrupted data, which can happen with complex distributed data systems with many moving parts.