On October 24, 2026, a paper titled “What the Specification Did Not Say: Specification-Oracle Gaps in Test-Graded Coding-Agent Benchmarks” was published on arXiv (ID 2610.30635v1), systematically exposing the pervasive “specification-oracle gap” issue in current benchmarking of coding agents.
This gap refers to the following: tasks are described in natural language, yet scoring relies on real test cases from upstream projects; and to pass these tests, one often needs to know implementation details that are never mentioned in the description—such as the exact name of a constant, the module import path, or the error message string. Such information cannot be reasonably inferred from the specification, meaning that the evaluation effectively assesses the agent’s ability to reproduce the original code rather than its generalization‑to‑new‑tasks programming capability.
The research team formally modeled this gap and the inconsistencies it introduces in multi-stage dependent task chains, and developed a static auditing tool that supports five programming languages. The tool can automatically identify implicit dependencies at the name, path, and literal levels for each scoring test, achieving 89.2% precision and 92.9% recall on the validation set. On the SWE-Milestone benchmark—comprising 98 scoring milestones and 35 public runs—25 milestones were confirmed to contain at least one test with implicitly undeducible names or paths.
Empirical evidence shows that build units containing such implicit dependencies fail in 80.6% of runs, whereas comparable dependency-free units fail in only 48.4% of cases; merely hiding a key name in a reference implementation leads to compilation failures in 126 out of 128 cases. Although the overall ranking remains robust after removing the affected tests, the scores within the repository shift and increasingly favor agents that replicate upstream naming conventions.
Further analysis of four additional benchmark categories reveals that the prevalence of disparities is highly dependent on how task specifications are generated: when specifications are derived from release notes or large-model summaries, more than one-third of tasks are affected; by contrast, tasks that have been manually curated or specifically tailored for testing are hardly impacted. The researchers have open-sourced their auditing tools, a revised filtering list, and supplementary specification notes.