Protecting the user community and the platform from illegal or undesirable behavior is an important problem for most large online platforms. Content moderation (aka Integrity) systems aim to define, detect and take action on bad behavior/content at scale, usually accomplished with a combination of machine learning and human review.

Building hybrid human/ML systems for content moderation presents unique challenges, some of which we will discuss in this talk:
* Human review annotation guidelines & how it impacts label quality for ML models
* Bootstrapping labels for new categories of content violation policies
* Role of adversarial drift in model performance degradation
* Best practices for monitoring model performance & ecosystem health
* Building adaptive machine learning models

The talk is a distillation of learnings from building such systems at Facebook, and from talking to other ML practitioners & researchers who’ve worked on similar systems elsewhere.