Adversarial Drifts, Model Monitoring, and Feedback Loops: Building Human-in-the-Loop Machine Learning Systems for Content Moderation
On Demand
Type
- Session
Format
- Hybrid
Track
- Data Science, Machine Learning and MLOps
Difficulty
- Intermediate
Room
- Moscone South | Upper Mezzanine | 156
Duration
- 35 min
Overview
Protecting the user community and the platform from illegal or undesirable behavior is an important problem for most large online platforms. Content moderation (aka Integrity) systems aim to define, detect and take action on bad behavior/content at scale, usually accomplished with a combination of machine learning and human review.
Building hybrid human/ML systems for content moderation presents unique challenges, some of which we will discuss in this talk:
* Human review annotation guidelines & how it impacts label quality for ML models
* Bootstrapping labels for new categories of content violation policies
* Role of adversarial drift in model performance degradation
* Best practices for monitoring model performance & ecosystem health
* Building adaptive machine learning models
The talk is a distillation of learnings from building such systems at Facebook, and from talking to other ML practitioners & researchers who’ve worked on similar systems elsewhere.
Building hybrid human/ML systems for content moderation presents unique challenges, some of which we will discuss in this talk:
* Human review annotation guidelines & how it impacts label quality for ML models
* Bootstrapping labels for new categories of content violation policies
* Role of adversarial drift in model performance degradation
* Best practices for monitoring model performance & ecosystem health
* Building adaptive machine learning models
The talk is a distillation of learnings from building such systems at Facebook, and from talking to other ML practitioners & researchers who’ve worked on similar systems elsewhere.
Session Speakers
See the best of Data+AI Summit
Watch on demand