Measuring Beyond Accuracy Saturation

Sources#

• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
• Claude Opus 5 System Card
• Life After Benchmark Saturation: A Case Study of CORE-Bench
• Recursive Self Improvement for Coding Agents
• Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent

Summary#

When a benchmark's headline accuracy saturates — top agents reach statistically indistinguishable scores near the ceiling — the field's default response is retire-and-replace: build a harder successor (ARC-AGI 1 → 2 → 3, MMLU → MMLU-Pro, HumanEval → HumanEval+, SWE-bench → SWE-bench Pro). Nadgir, Kapoor, … Narayanan ("Life After Benchmark Saturation", Princeton-led, 14 authors, arXiv 2606.26158, empirical) argue this reflex is fundamentally inadequate for anyone but a model developer optimizing relative accuracy. Their central thesis: accuracy saturation does not imply the benchmark has run out of signal. Even when accuracy can no longer separate agents, six other measurable dimensions still can. The prescription is to decouple accuracy saturation from benchmark saturation — don't retire a saturated benchmark, re-instrument it.

They demonstrate this on CORE-Bench Hard (Siegel et al.), a benchmark for the computational reproducibility of scientific code: reproduce a published paper's results given only its README, code, and data — no Dockerfile or runfile. Reproducibility is a well-chosen case study because it is a high-value real-world task with a direct human counterpart (enabling a concrete human-uplift experiment), clear out-of-distribution axes (swap the research field), and multiple practically relevant dimensions (correctness, cost, latency, reliability). The paper's three contributions map to three groups of the six axes: benchmark validity (construct validity + OOD robustness), evaluation completeness (efficiency + reliability + model-vs-scaffold), and practical impact (human-agent collaboration uplift).

The retire-and-replace critique#

"Accuracy saturation" is used in Akhtar et al.'s sense: the state in which top-performing agents post statistically indistinguishable accuracies, so the leaderboard loses discriminative power. Retire-and-replace serves model developers — who mainly care about relative accuracy for marketing and checkpoint selection — but fails researchers and downstream developers who need to know how well an agent actually solves a real task. The deeper point: accuracy-centric evaluation is an insufficient measurement tool throughout a benchmark's lifecycle, not just at saturation; saturation merely makes the insufficiency impossible to ignore. Prior work has advocated multidimensional evaluation in principle, but the field has defaulted to building harder accuracy benchmarks anyway. This paper is the worked demonstration that the other axes pay off.

Contribution 1 — Saturation surfaces threats to construct validity#

High capability is what exposes validity problems: weaker agents never progressed far enough to exploit shortcuts or trip over benchmark bugs, so the threats stayed hidden until accuracy saturated. Validity is threatened along two axes:

• Task-level threats — the headline metric does not faithfully measure the intended capability. This is a documented, cross-benchmark problem: impossible-to-solve SWE-bench tasks, a τ-Bench Airline scaffold bug, incorrectly-graded WebArena tasks. Log analysis (tracking an agent's inputs, outputs, and environment) is the key discovery method.
• Benchmark-specific adaptation — when a benchmark becomes a development target, iterating on prompts, scaffolds, tool-use, and timeouts against a fixed benchmark tailors the agent to the benchmark's idiosyncrasies. Strong performance then partly reflects adaptation rather than general capability — Goodhart's law at the benchmark-construction layer (see Reward Hacking).

Applying automated + manual log analysis (via Docent, an LLM-driven log-flagging tool, using rubrics for process correctness, computation correctness, pre-existing-artifact contamination, and grading errors) to the 45 original CORE-Bench Hard tasks plus 27 new candidates, they found 15 task-level errors (wrong ground truths, malformed questions, grading errors, unsolvable tasks) and 20 tasks with exploitable shortcuts (e.g., the agent reads a pre-computed value out of a static artifact instead of reproducing it). Two new suites result:

• CORE-Bench v1.1 — corrects the errors and shortcuts and adds 10 new tasks built by the same pipeline, for a 39-task suite (13 computer science, 10 social science, 16 medical). It repurposes CORE-Bench Hard rather than replacing it with something harder.
• CORE-Bench OOD — a 19-task out-of-distribution suite that holds task structure fixed but shifts the field (physics, engineering, economics, CS), testing whether saturated accuracy transfers under a discipline distribution shift.

The punchline that makes the whole paper: after all these corrections, accuracy still saturates. On v1.1 the top agent hits 100% and the next four tie at ~97.4%; on OOD the top five of twelve Codex CLI agents are again statistically indistinguishable. (Nicholas Carlini's submitted Claude Code scaffold reached near-ceiling on CORE-Bench Hard after fixing a few grading errors — the event that signaled saturation.) So contribution 1 is not "we fixed saturation by making a better benchmark"; it is "saturation is the lens that reveals validity threats, and the threats persist as saturation persists." The authors treat v1.1 and OOD as living benchmarks they will keep updating, because log analysis is not exhaustive — it requires specifying target behaviors, some threats surface only in particular runs, and LLM-based classifiers need manual validation.

Contribution 2 — Multidimensional evaluation after saturation#

The core reframe: decouple accuracy saturation from benchmark saturation. Across 20 agent runs (Codex CLI, Claude Code, OpenCode, CORE-Agent × GPT-5.x / Opus 4.5 / Opus 4.6), agents with statistically indistinguishable accuracy differ sharply along three measurable axes.

Reliability#

Adopting Rabanser et al.'s framework, they measure four things over five repeat trials on five Codex CLI agents:

• Outcome consistency (do repeated runs yield the same verdict) and resource consistency (token-usage variability) both rise with accuracy — the most accurate agent is also the most repeatable and the steadiest in token spend (correlations ≈ +0.94 and +0.95).
• Calibration and discrimination are badly broken, and identically so across agents. Mean empirical pass rate is 93% but mean self-reported confidence is only 32.1% — agents are massively underconfident. Worse, no agent beats a random baseline at telling its own correct runs from its incorrect ones (discrimination AUROC ≈ 0.51–0.64). Reported confidence tracks the number of failed bash commands — a signal uncorrelated with task success. A model's confidence is therefore a near-useless success predictor here, an important negative result for any pipeline that would gate on agent self-report (cf. Self-Report as a Safety Signal).

Efficiency#

Plotting accuracy against both token usage and dollar cost separates the top scorers that accuracy alone cannot. Two findings:

• Some high-scorers are far more efficient. GPT-5.3-Codex (medium) is the most efficient high-accuracy agent by both metrics; at the same 97.4% accuracy as GPT-5.4 (high) it costs roughly 60% less.
• Tokens and dollars tell different stories. Accuracy rises with token usage but is roughly flat-to-negative against cost — driven by provider pricing and caching behavior (some Codex CLI pairs cache aggressively; CORE-Agent does not cache at all). Which compute unit you pick changes the ranking. This is exactly the compute-axis point of Compute-Controlled Benchmarking, applied inside a saturated benchmark. The well-documented returns to inference scaling (Large-Scale Test-Time Compute) let an agent brute-force accuracy — useful for finding a model's ceiling, but for practitioners the cost of an answer matters as much as the answer.

Decoupling model and scaffold#

Leaderboards report one accuracy per agent, collapsing the model and the scaffold that orchestrates it. Saturation makes disentangling them more urgent: once several agents tie on accuracy, the leaderboard no longer says which part of the stack earns the win. Evaluating Opus 4.5, Opus 4.6, and GPT-5.4 across three of four scaffolds (Claude Code = proprietary; CORE-Agent, OpenCode, Codex CLI = open-source) and classifying all 56 failures + 390 logs by root cause via Docent yields three findings:

1. Similar accuracies mask different failures. Opus 4.5 scores 82.1% on both CORE-Agent and OpenCode, yet the two scaffolds disagree on 31% of capsules (12 of 39). An oracle router that picks the best scaffold per task reaches 100% for both Opus 4.5 and GPT-5.4 — meaning every v1.1 task is solvable by at least one scaffold. Scaffolds change which tasks a model can solve, not just how many.
2. Scaffolds induce distinct solution strategies. Holding the model fixed and swapping the scaffold makes this visible: on Opus 4.6, Claude Code reads 41% of answers from unmodified code's text output and only 3% from a vision-read of a rendered figure, whereas CORE-Agent vision-reads 31% of the time; vision-read rates jump to 62% (CORE-Agent vs 3% Claude Code) on Opus 4.5. And vision-reads used as a fallback after abandoning the original code pass only ~50% of the time vs 93% after a clean run — CORE-Agent's accuracy gap is largely accumulated fallback failures.
3. Direct fixes beat rewrites. Scaffolds that diagnose a root cause and patch it succeed 95.2% of the time (n=269); those that abandon the original implementation and rewrite from scratch succeed only 67.8% (n=59) — and a scaffold's tendency toward direct fixes tracks its overall accuracy (Codex CLI 82%, CORE-Agent 49%). The scaffold contribution is large: with GPT-5.4 (medium), Codex CLI beats the CORE-Agent scaffold by ≈44 percentage points (and CORE-Agent + GPT-5.4 is the ~51% low outlier across the whole run set).

Conclusion: model and scaffold effects are not cleanly separable — scaffolds constrain the available solution paths, models determine how well they are used. This is the empirical core of the Agent Harness Engineering thesis measured on a saturated benchmark.

Contribution 3 — Human-agent collaboration uplift#

Once agents converge at near-ceiling accuracy, the question shifts from can the agent do the task to does it add value alongside a human. High benchmark accuracy need not translate to uplift: benchmark tasks may be narrower than real work, agent failures may be costlier for a human to unwind, and agents may respond poorly to redirection. So they ran a small randomized study: five evaluators (all with a data-science master's and reproducibility experience — and all coauthors) reproduced results from 20 papers (award-winning ML papers since 2011 + social-science papers from the Institute for Replication), with and without agent collaboration, blinded where possible, under a 3-hour limit, yielding 50 replication experiments. The collaboration condition used Codex CLI + GPT-5.4 (extra-high thinking), autonomous but instructed to escalate to the human after 2–3 failed attempts. Unlike CORE-Bench, the goal was process-level uplift, not answer correctness, so papers were not pre-verified as reproducible.

• Agent collaboration more than halves reproduction time. A fixed-effects model estimates manual sessions lasted 2.11× as long as collaborative ones (clustered SE 0.09, two-sided p ≈ 0.00176). This is likely a conservative estimate: the 3-hour limit was hit by 5 of 25 manual runs and 0 of 25 collaborative runs, so the true gap is probably larger.
• Most collaborative runs needed little or no human help. The agent completed 19 of 25 runs fully autonomously (aside from two setup steps assigned to humans). Agents were seen as most valuable in environment setup (25/25), running code (23), identifying main scripts (20), and README navigation (19).
• Agents log more blockers but recover more reliably. The agent fully or partially resolved all but 2 of the 114 blockers it hit, while humans left 11 of 60 unresolved; on four papers the agent fixed missing/broken repository artifacts humans could not.

Limitations the authors flag: tiny sample (20 papers, 5 participants); reproducers are coauthors (demand effects possible); no ground truth (process uplift, not outcome correctness); narrow selection (Python/R only, <45-min compute, results-with-clear-targets); award-winning ML papers skew toward better documentation. The uplift result is directional evidence, not a settled effect size.

Why it matters#

The paper is the concrete counter to the retire-and-replace reflex that Task Time-Horizon Scaling flags as an open problem ("task baskets saturate — what replaces them?"). Its answer: often, nothing needs to replace them. A saturated benchmark, re-instrumented, still tells you which agents are efficient, which are reliable, whether the win came from the model or the scaffold, and whether the agent actually helps a human — none of which the headline accuracy number can. It is one of two 2026 answers to "life after saturation" the wiki carries: this one extracts more signal from one saturated benchmark; BenchPress stops running redundant benchmarks by predicting their scores. They point in opposite operational directions (keep-and-re-measure vs skip-and-predict) but share the premise that a single accuracy number is a lossy compression of what a benchmark knows.

Connections#

• Orchestration Sets Token Economics — the economic dual of this page's model-vs-scaffold result, and its closest methodological sibling. This source swaps scaffolds with the model fixed and finds a ~44pp accuracy swing; Writer swaps the orchestration layer with the model fixed across six models and 22 locked tasks and finds a 41% cost swing at quality parity, uniform in sign with no exceptions. Two groups, opposite outcome variables, same conclusion that the harness is not a neutral substrate — and the same warning about attributing a result to a model that belongs to a model-harness pair. Its quality half also arrives at a scaffold-capability interaction this page's oracle-router finding hints at: quality gains from a richer harness scale with the model's baseline strength (r = 0.99) while efficiency gains do not. Vendor-authored with a total COI

• Deterministic Pre-Execution Gates — the reliability axis doing work on an unsaturated benchmark, and a construct-validity argument for the failure side: under pass¹ a gate suite looks like +12.4pp of accuracy, while under pass^k the baseline decays 3.7× and the gated arm 1.6×, which is what distinguishes "removed a recurring failure mode" from "added random wins." Its five admission axes are the same re-instrument-don't-retire move applied before a benchmark is built — a suite that reports only pass/fail cannot separate a loud tool error from a silent policy-violating write, and the paper found only one benchmark in the field that can

• Usage-Telemetry Classifier Validation — the same interrogation applied to economic measurement: ATLAS's O*NET task classifier scores 22.6% exact accuracy, 4,244× chance lift, and 85.8% human approval, and which number you quote decides whether the pipeline sounds trustworthy

• LLM-as-a-Judge — a fifth move for the same predicament, and the one that changes what a leaderboard is allowed to say: CalibratedRubric bootstraps a confidence interval on each system's IRT ability estimate and collapses adjacent systems whose difference is not significant into a tier, refusing to emit a rank order it cannot support (15 systems → four and six tiers; six-system blocks → one or two; only 9.81% of JudgmentBench output pairs separated at all). Where this page adds axes and CollabEval narrows the interval, tiering simply declines to over-report — the cheapest honest response to "statistically indistinguishable at the ceiling," available to any benchmark willing to publish bands instead of ranks. Its other half is a construct-validity warning for saturated benchmarks with judge-derived criteria: consensus filters retain criteria at ρ^M in the number of systems, so growing the leaderboard changes which criteria count as gold even with criterion quality fixed

• Responsible Scaling Policy Evaluations — saturation with governance stakes: Anthropic's Opus 5 card drops the automated AI R&D rule-out suite from its threshold determinations because recent models exceed top human baselines on all but two tasks, so failing to rule out a capability no longer bounds it

• Cost-per-Task Over Cost-per-Token — the vendor conceding the same point about its own products: Anthropic says Opus and Fable have "similar benchmark scores" yet differ in "wisdom, creativity, and writing skills," and routes the decision to internal testing instead. A first-party construct-validity admission at exactly the tier where the paper predicts accuracy stops separating systems

• Benchmark Score Redundancy — the complementary lens on the same saturation fact. Saturation = near-zero score spread across agents; BenchPress uses exactly that to compress the benchmark landscape (a low-spread benchmark is trivially predictable from others, so you may not need to run it), while this paper extracts six non-accuracy signals from the one saturated benchmark you keep. "Skip-and-predict" vs "keep-and-re-instrument" — opposite moves, same premise that a headline accuracy number under-uses the benchmark. That page's second source, CollabEval, is a third move with a different currency: keep the benchmark and re-label only a fraction of its prompts, buying back the lost precision from historical runs of other models via matrix completion used as a control variate. It is the one of the three that changes what you may claim rather than what you measure — a coverage guarantee at ~45% of the labels where classical estimation needs 60% — and it is directly relevant to this page's statistical predicament, since "statistically indistinguishable at the ceiling" is a statement about confidence intervals, and CollabEval narrows those by 20–30% without any extra annotation

• Benchmark Contamination and Decontamination — the same "aggregate accuracy is a lossy summary" argument applied to contamination rather than saturation. Sun et al. add per-sample distribution distances where this paper adds six non-accuracy axes, and hit the same failure shape: a dataset-level improvement (residual contamination ↓) that does not move per-sample behavior toward the clean reference (D_KL ↑). Both refuse to accept a single accuracy delta as proof a benchmark's signal has been recovered

• Task Time-Horizon Scaling — this page's open question ("time-horizon baskets saturate; what replaces them?") gets its answer here: don't replace, re-instrument. This paper takes CORE-Bench (which that page cites as saturating in 15 months) and shows it still discriminates agents along six axes after accuracy saturates

• Compute-Controlled Benchmarking — the efficiency axis here (accuracy vs tokens vs dollar cost; GPT-5.3-Codex ~60% cheaper at equal accuracy) is Brown's "put compute on the x-axis" applied inside a saturated benchmark; both refuse to report one accuracy number without the cost of reaching it

• Harness-Induced Belief Divergence — the re-instrument-don't-retire argument extended to a seventh axis, beliefs, and the page's one live tension. This source swaps scaffolds with the model fixed and finds accuracy swinging ~44pp; Yi & Song swap the evidence-mediation layer with the model fixed and frame their result as belief divergence at preserved terminal success. Both cannot be generically true — though the tension is between a measurement and an assumption, since that paper reports no pass rate anywhere and "harness" names a narrower object there (a mediation layer over a fixed loop, not a whole scaffold). Read the other way, this page supplies the prior that most constrains it: agents are massively underconfident (93% pass vs 32.1% confidence) and cannot beat random at discriminating their own correct runs, which is a hard warning against treating elicited belief fields as calibrated quantities

• Agent Harness Engineering — the model-vs-scaffold decoupling is the empirical measurement of the harness's contribution: scaffold swings accuracy ~44pp, two scaffolds on one model disagree on 31% of tasks (oracle router → 100%), and direct-fix-vs-rewrite (95% vs 68%) is a scaffold-strategy effect, confirming scaffold and model are not cleanly separable. The same axis run inside one system, on unsaturated benchmarks, by the team that built it: Leni's decomposition (Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent, empirical, disclosed total vendor COI) holds the frontier model fixed and splits a production agent's uplift by layer across three benchmarks — on SpreadsheetBench, +9.5 pp of the +11.0 pp total is prompt-plus-scaffold and the verification loop adds the last +1.5 pp; on GAIA the tiers run ~60% → ~70% (planner–executor) → ~74% (+ routing) → 75.2%. Three ways it extends the axis. It sub-divides "scaffold" into structure, specialist staffing, and verification checkpoint, which is finer than the scaffold-vs-model cut and yields a ranked investment order rather than a spread. It shows the axis is not only a saturation instrument — these benchmarks are nowhere near ceiling, and the decomposition is still the durable result while the leaderboard position explicitly is not. And it adds a candidate seventh axis the six here do not contain: loop telemetry, a verifier confusion matrix whose catch/fix/false-alarm rates convert directly into a marginal-return estimate for the next engineering decision. The methodological warning runs the other way, though — where this page's ablations swap real public scaffolds, Leni's layer tiers are internal single runs "whose selection rules were not recorded," which is exactly the construct-validity problem log analysis exists to catch

• Production-Sourced Evaluation — the sibling answer to "what to do when benchmarks saturate": that page refreshes the task set from live production usage; this page re-instruments the existing (saturated) task set along non-accuracy axes. Both reject retire-and-replace, from opposite ends (new tasks vs new metrics)

• Reward Hacking — the "exploitable shortcuts" (reading a pre-computed value instead of reproducing it) and "benchmark-specific adaptation" (tuning the agent to a fixed benchmark's idiosyncrasies) are Goodhart at the benchmark-construction / development-target layer; log analysis after saturation is how they surface

• Configurable Human Participation — the human-uplift study is the field counterpart to HAS-Bench's controlled human-agent measurement: HAS-Bench varies participation on an LLM-simulated human across 397 tasks; this runs a real randomized study on 20 reproduction tasks and finds collaboration more than halves time. Both make human-agent collaboration a first-class measured axis rather than an afterthought

• Large-Scale Test-Time Compute — the returns-to-inference-scaling this page's efficiency axis measures against: agents can brute-force accuracy with more compute, so cost-per-answer is the practitioner-relevant complement to the saturated accuracy number

• Self-Report as a Safety Signal — the reliability finding sharpens it on a capability axis: frontier coding agents are massively underconfident (93% pass vs 32.1% stated confidence) and cannot beat random at telling their own correct runs from incorrect ones, so agent self-rated confidence is a weak basis for gating

• How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them? — the cluster synthesis: re-instrumentation is one of five portfolio moves that jointly answer "how much signal do public benchmarks still carry, and what replaces them?"

• Layerwise Omission Attribution — the same complaint on a fourth axis: mechanism. A single failure rate cannot separate byte-level software loss from behavioral non-retrieval, so output-only scoring hides the one thing an operator needs before acting. Its prescription is this page's in a different register — don't retire the endpoint, instrument the pipeline underneath it — with checkpoint taps carrying a canary so the deterministic layers are counted exactly rather than judged. It also demonstrates the failure mode of an unaudited metric from the inside: its own retrieval labels are case-normalized substring matches applied even to paraphrase trials, which the paper concedes makes that contrast partly a measurement effect

• Agent-Authored Harness Optimization — the model-vs-scaffold axis exploited deliberately: Cline held the model fixed and had an agent patch the scaffold for +11.3pp on Terminal-Bench 2.1, a modest move against this paper's ~44pp scaffold swing, but found by the agent rather than by engineers

• Orchestration-Plan Simulation — the same decomposition instinct taken to its limit. This page splits model from scaffold after execution; OrchBench removes the workers entirely so an orchestration plan can be scored on its own, and validates the substitute against real runs (r = 0.816 on final score, 1.3% of the tokens). Two things it contributes back here. First, its ablation is a cleaner construct-validity statement than most benchmarks manage — neutralize two modeled mechanisms (missing handoffs, lossy compression) and a strong and a weak planner become indistinguishable, which is both a defense and an admission of what the score actually measures. Second, its cross-framework matrix supplies the datum the model-vs-scaffold section implies but never measures: four real harnesses rank the same models at Pearson 0.08–0.84 against each other (Claude Code vs Crush = 0.08), so "the scaffold matters" and "real execution is a noisy anchor" are the same fact — and a simulator correlating 0.63–0.82 with each of them beats their agreement with one another

• Post-Acceptance Edit Behavior — the same complaint reached from usage telemetry rather than from benchmark internals, and with a concrete replacement metric attached. Liang et al. argue pass@k on HumanEval, MBPP, SWE-Bench and BigCodeBench measures correctness while the property deciding whether generated code survives is alignment with the developer's intent and context — a completion can be correct and still be deleted 23 minutes later. Their proposed substitutes are behavioral rather than decompositional: code retention and AI completion abandonment rate, both computed from what a human did to the output rather than from whether it passed. That is a different move from this page's reliability/efficiency/scaffold split and composes with it, and it comes with the corresponding weakness — retention records the human's action, never whether the action was correct

Open Questions#

• Does re-instrumentation generalize past reproducibility? CORE-Bench Hard was chosen precisely because it has a direct human counterpart, clean OOD axes, and multiple practical dimensions. Whether the six-axis treatment yields comparable signal on benchmarks without those properties (e.g. closed-form reasoning benchmarks with no human-workflow analog) is untested.
• Is the human-uplift result real or a demand effect? The reproducers are the paper's own coauthors, there is no ground-truth correctness, and n = 20 papers / 5 participants. The 2.11× speedup is statistically significant but the authors themselves cannot rule out participant bias — an independent, blinded replication is the missing evidence.
• Which non-accuracy axis actually predicts deployment value? The paper measures six axes but does not rank them by decision-relevance for a downstream deployer. If you can only measure one beyond accuracy, is it reliability, efficiency, or scaffold contribution — and does the answer depend on the use case? Partially answered (2026-08-04) — a seventh candidate rather than a ranking: Leni proposes loop telemetry and gives it the most direct claim to decision-relevance any axis here has made. Because its verification loop is fully instrumented, the measured catch/fix/false-alarm rates convert straight into marginal returns on the next engineering decision — raising the catch rate is worth up to +8 pp, raising the fix rate at most +0.5 pp — so the axis does not merely separate systems, it names which component to fund. It also answers the "does it depend on the use case" half affirmatively and specifically: the argument for keeping the checkpoint at all is that its value concentrates where a reliability SLA's tail sits, which is a use-case-conditional claim by construction. Still open as posed, because no source has ranked the axes against each other, and this proposal comes from a vendor instrumenting its own system.
• Can the model-vs-scaffold decoupling be made routine? The oracle-router result (every task solvable by some scaffold → 100%) implies large headroom from scaffold routing, but requires per-task oracle knowledge. Whether a practical router can approach the oracle without it is open, and would turn a measurement into a capability. Partially answered (2026-08-04) — on an adjacent axis: Leni ships a practical router and it pays. A 0.5B step-type classifier dispatches each step across two model families (cheap models for classification, frontier reasoning for multi-hop synthesis, strong grounding for vision), and internal estimates credit it with ~4 pp of GAIA accuracy at net-negative cost — cheap steps subsidise extended reasoning on hard ones. So routing is deployable at a price low enough to run on every step, which was the practical objection. It does not settle the question as posed, on an axis mismatch that matters: this routes models per step inside one fixed scaffold, not scaffolds per task, and no oracle comparison is run, so what fraction of available headroom the router captures is unmeasured. Vendor-authored, internal single-run attribution.
• Do "living benchmarks" outrun their own maintenance? v1.1 and OOD are to be updated as new validity threats surface via log analysis, which the authors note is non-exhaustive. Whether continuous log-analysis-driven maintenance is sustainable — or itself becomes a Goodhart target once developers know the rubrics — is unexamined.

Sources#

• Life After Benchmark Saturation: A Case Study of CORE-Bench — Nitya Nadgir, Sayash Kapoor, Kangheng Liu, Peter Kirgis, … Arvind Narayanan (14 authors; Independent / Princeton / UC Berkeley / MIT; arXiv 2606.26158, 2026-06-23, empirical). §1 the retire-and-replace critique and central thesis; §2 construct-validity threats via Docent log analysis (15 task-level errors + 20 shortcuts), CORE-Bench v1.1 (39 tasks) and CORE-Bench OOD (19 tasks), persistence of saturation (top 100%, next four ~97.4%); §3 multidimensional evaluation — reliability (Rabanser et al. framework; consistency ↑ with accuracy r≈0.94/0.95; 93% pass vs 32.1% confidence; discrimination AUROC 0.51–0.64), efficiency (GPT-5.3-Codex ~60% cheaper at equal accuracy; token vs dollar divergence), model-vs-scaffold (31% capsule disagreement, oracle router 100%, direct-fix 95.2% vs rewrite 67.8%, ~44pp scaffold gap); §4 randomized human-uplift study (20 papers, 50 experiments, 2.11× time, p≈0.00176, 5/25 manual runs hit the 3h wall). Figures 1 (reliability), 2 (efficiency), and 3 (session-duration histogram) viewed. Note: Tables 2 & 4 have flattened multi-value cells in the raw parse — per-model accuracies quoted here are only those corroborated in the paper's prose and figures
• Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent — Arunabh Dastidar & the Leni Team (Leni Inc., arXiv 2607.17044, 2026-07-19, empirical, disclosed total vendor COI): §7 + Table 5 the layer decomposition (SpreadsheetBench 80.25 → 89.75 → 91.25; GAIA ~60 → ~70 → ~74 → 75.2), §6.2 the loop telemetry and its marginal-return calculus, §4.3 + §8 the 0.5B per-step router credited with ~4 pp at net-negative cost, §6.3 the GAIA headline self-correction (77.6% retired to 75.2% pass@1). Table 3 is fully collapsed in the raw parse (four benchmark rows welded into one grid row); recovered against pdftotext -layout and corroborated by Figure 5, and no cell of it is quoted here. Table 5's arrow-joined GAIA cell is genuine. The GAIA layer tiers and the specialist swaps are internal single runs — treated as indicative throughout. Full treatment at Agent Harness Engineering
• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation — Chen et al. (FinStep + StepFun, arXiv 2607.29252, 2026-07-31, empirical): §3.5 "Uncertainty and tiers" (bootstrap ability intervals, collapsing non-significantly-different adjacent systems) and §4.3 (four and six tiers over 15 systems; smaller blocks collapse to one or two; 9.81% of JudgmentBench output pairs separated) — see LLM-as-a-Judge for the full treatment