Maarten Sap and Xuhui Zhou from Carnegie Mellon University's School of Computer Science have earned a Dynabench Data Collection and Benchmarking Platform award from Meta Research for their work to create context-aware toxicity-detection systems for digital content. 

Two years ago, Meta Research launched its Dynabench technology, a platform for dynamic data collection and benchmarking in artificial intelligence that uses both humans and models to create new datasets that will lead to better, more flexible AI. Sap, an assistant professor in the Language Technologies Institute (LTI), will work with LTI Ph.D. student Zhou on the project "ContExTox: Context-Aware and Explainable Toxicity Detection."

Online platforms struggle to moderate the increasing amounts of toxic, biased or hateful content posted on their sites. While AI systems can help humans sift through potentially toxic content, those systems are often brittle and biased, which can hinder their fairness. One key component they lack is contextual awareness — the systems don't know who is speaking or what was said before. Sap and Zhou will study how social and conversational context can be modeled ethically and safely for toxicity-detection systems using collaborative human-machine setups. They'll also explore how to shift the AI toxicity paradigm from detection to explanation by incorporating descriptions of why and to whom content might be toxic.

Sap and Zhou are one of five research teams to receive awards from 26 proposals submitted around the theme of Rethinking Benchmarking. For more on the awards, visit the Meta AI Research website.