HomepageData + AI Summit 2022 Logo
Watch on demand

Efficient and Multi-Tenant Scheduling of Big Data and AI Workloads

On Demand

Type

  • Session

Format

  • Hybrid

Track

  • MLOps and DataOps

Difficulty

  • Intermediate

Room

  • Moscone South | Upper Mezzanine | 160

Duration

  • 35 min

Overview

Many ML and big data teams in the open source community are looking to run their workloads in the cloud and they invariably face a common set of challenges such as multi-tenant cluster management, resource fairness and sharing, gang scheduling and cost-effective infrastructure operations. Kubernetes is the de-facto standard platform for running containerized applications in the cloud. However, the default resource scheduler in Kubernetes leaves more to be desired for AI scenarios when running ML/DL training workloads or large-scale data processing jobs for feature engineering. In this talk, we will share how the community leverage and build upon Apache YuniKorn to address the unique resource scheduling needs for ML and big data teams.

Session Speakers

Chaoran Yu

Senior Software Engineer

Apple

Chenya Zhang

Senior Software Engineer

Apple

See the best of Data+AI Summit

Watch on demand