Dean Wampler

Ray (ray.io) is an open-source, distributed framework from U.C. Berkeley's RISELab that easily scales Python applications from a laptop to a cluster. It was developed to solve the general challenges of reinforcement learning, but it is flexible for any demanding workload that requires the following:

1. Low-latency scheduling and execution of small-to-large 'tasks' that perform a wide variety of computation chores, with logical sequencing of dependent tasks.
2. Management of 'arbitrary', distributed state, with thread-safe updates and access from other Ray tasks across a cluster.
3. Near-linear scaling.
4. An intuitive API that hides complexity from the user.

Ray has been used for reinforcement learning, hyper parameter tuning, model serving, and other applications in clusters up to thousands of nodes. I'll discuss examples that illustrate how Ray can be used with Spark to build robust, scalable data applications for enterprises, when to use Ray versus alternative choices, and how to adopt it in your projects.

The Application Spotlight will highlight selected “Certified on Spark” applications that leverage Spark to help their users derive greater value from their data. For each application their will be a brief demo of key functionality followed by a fireside chat discussing the developers experience with Spark, lessons learned, and wish list for the future.