The rise of voice chat in online videogames has inevitably led to a growth in bad behavior between players—it is, after all, a lot easier to scream slurs and abuse at other people than it is to stop playing and type them out. Moderating that sort of toxicity is tough, and some companies are turning to AI to help crack down on the worst of it. Speaking at GDC, Roblox Senior Technical Director Kiran Bhat and Voice Safety Lead Hannes Heikinheimo said Roblox has been using machine learning voice chat moderation for a year and has had good success with it—but acknowledged that sometimes, actual human beings are better at the job.
"Moderating voice in real time sounds really daunting, because to get the voice moderated, you not only have to know what the person is saying, but you also want to capture the tone and the intensity of what is being said to decide whether something is toxic or not," Bhat said during the talk. "There's also context that is part of whether something is offensive. So it's a really hard problem."
Mitigating that challenge, Bhat explained, is that at least 85% of what is considered toxic can be covered across four major categories, and the "vast majority" of offenses are captured with a list of top-50 keywords—"so already, if you did top 50 keywords across these categories, you've done a good job."
I'm naturally skeptical of big claims about the capabilities of AI, but Heikinheimo, Roblox's voice safety lead, claims impressive results: After running the system for about year, during which time it's been expanded to cover 31 countries, he said the number of abuse reports made per "active hour of talk" has been cut in half.
One of the advantages of using machine learning for voice moderation is "consistency," Heikinheimo said: Unlike humans, machines don't have different outlooks or opinions, they don't get tired or angry or have bad days, they just do what they do. So in "very clear cases [of toxic behavior], machines are more effective and more consistent" than people. But the essential humanity that can make us difficult to pin down also enables us to do a better job than systems-driven machines in situations that aren't so clear cut: Machines may be [[link]] good at matching patterns, but when it comes to understanding intent and exercising discretion, well, "humans still surpass machines."
"Humans are still better in those types of things where you might be very close to the decision boundary," Heikinheimo said. "Or it might be a case that's very rare where we don't have that much data for the machine learning system to learn from. So in those cases, humans can still be better."
Roblox isn't the only one [[link]] pursuing AI-powered voice moderation. At GDC 2024, for instance, we spoke with two software makers working on machine learning systems to help with voice chat moderation. It's also notable that Activision has previously reported similar success with its own AI-powered voice moderation tools in Call of Duty.
Innate skepticism aside, I do acknowledge that machine learning—AI, whatever you want to call it—has many potential real-world use cases, and this is one good example. But I also appreciate [[link]] that it demonstrates that it's a tool, not a panacea, and that we human beings still have a place in this brave new world of the future.
