HomepageData + AI Summit 2022 Logo
Watch on demand

Adversarial Drifts, Model Monitoring, and Feedback Loops: Building Human-in-the-Loop Machine Learning Systems for Content Moderation

On Demand

Type

  • Session

Format

  • Hybrid

Track

  • Data Science, Machine Learning and MLOps

Difficulty

  • Intermediate

Room

  • Moscone South | Upper Mezzanine | 156

Duration

  • 35 min
Download session slides

Overview

Protecting the user community and the platform from illegal or undesirable behavior is an important problem for most large online platforms. Content moderation (aka Integrity) systems aim to define, detect and take action on bad behavior/content at scale, usually accomplished with a combination of machine learning and human review.

Building hybrid human/ML systems for content moderation presents unique challenges, some of which we will discuss in this talk:
* Human review annotation guidelines & how it impacts label quality for ML models
* Bootstrapping labels for new categories of content violation policies
* Role of adversarial drift in model performance degradation
* Best practices for monitoring model performance & ecosystem health
* Building adaptive machine learning models

The talk is a distillation of learnings from building such systems at Facebook, and from talking to other ML practitioners & researchers who’ve worked on similar systems elsewhere.

Session Speakers

Nihit Desai

Co-Founder & CTO

Refuel.AI

See the best of Data+AI Summit

Watch on demand