Compute-Controlled Benchmarking

Sources#

• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
• Gemini 3.5 Flash-Lite Model Card
• Gemma 4 Technical Report
• Inkling: Our Open-Weights Model
• Kimi K3 Model Card
• More compute, more capability: Why AI agent evaluations need to account for test-time compute
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown
• Recursive Self Improvement for Coding Agents
• Rethinking the Evaluation of Harness Evolution for Agents

Summary#

The evaluation consequence of Large-Scale Test-Time Compute: if capability is a function of inference budget, then a benchmark number reported without its budget is meaningless. Noam Brown's essay is aimed squarely at the "benchmark grid" — the standard release artifact with benchmarks on the x-axis, models on the y-axis, and a single score per cell. His fix: put compute on the x-axis. Report performance as a function of tokens, cost, or time; or fix a budget and compare within it (practitioner-opinion).

The motivating incident: when GPT-5.5 launched, the grid showed only a few-percentage-point gain over GPT-5.4, and the initial reaction was skepticism that it was meaningfully better. It was — 5.5 was simply far more compute-efficient, reaching the same or better answers with much less thinking. 5.4 at max settings thinks longer per response. Control for thinking time and 5.5 is "a substantial jump," which matched users' day-to-day experience once they played with it. The grid hid the improvement because it didn't equalize compute.

Benchmark-maxxing: gaming the score with scaffolds#

The sharper worry is that the grid is trivially inflatable. Benchmark-maxxing is Brown's term for scaffolding tricks that raise a benchmark number without a real capability gain once compute is held equal: run the model five times and take the best answer; add an LLM judge to pick the strongest of N candidates. These "look a lot better on paper but are actually not better once you control for the amount of test-time compute." It is Goodhart's law moved out of the training loop and into the eval-reporting layer: the measure (grid score) becomes the target, so it stops measuring capability and starts measuring willingness to spend inference on best-of-N.

The standing defense against optimizing to a benchmark (as opposed to for the underlying skill) is a held-out private set that isn't publicly available — Brown says OpenAI tries not to optimize for specific benchmarks, but "once you put out a benchmark it's always at risk of just being optimized for."

The bad equilibrium#

Brown frames the grid's persistence as a coordination failure, not a disagreement. Privately, researchers agree the x-axis should be cost/tokens/time — "yeah, that makes sense, we should do that." But the published answer is "people expect us to publish the grid," and people expect it because everybody publishes the grid. Everyone knows it's a bad equilibrium and nobody wants to move first. Writing the essay is an explicit attempt to give the field permission to defect: so that "next time there's a model release, a company can feel comfortable not publishing the grid, at least not at the top line." This is Goodhart named as a collective-action problem rather than an individual temptation.

Independent adoption: UK AISI reports curves, not scores (July 2026)#

Brown wrote the critique; the UK AI Security Institute operationalized it, and as a government evaluator rather than a competing lab it is the cleanest party to defect from the grid. Its July 2026 study is empirical and its conclusion is Brown's prescription almost verbatim: "Evaluations should report capability curves, especially when performance may still be rising" — and "agent capability cannot be interpreted without the compute budget used to estimate it." A capped score, AISI warns, can "make model comparisons unequal" and "quietly mislead" a decision-maker into treating an under-resourced evaluation as evidence of a low-capability model.

The prescription is now AISI's practice, not just its recommendation. It evaluates frontier models across multiple budgets (including very large ones for the hardest tasks), reports reliability and reach against budget, and is defining "minimum informative budgets" — a budget declared sufficient only once a model's reach stops rising with more compute. That last is the operational answer to "how much is enough?" that a single grid number never had.

Worked example: Gemma 4's headline table (July 2026)#

Brown published the critique in June 2026. Three weeks later Gemma 4 shipped the artifact, and it is worth naming because the paper is otherwise careful (empirical, sixteen tables).

Table 5 — the report's most-cited table, and the basis of its "leap in performance" claim — compares Gemma 4 in thinking mode against Gemma 3 27B non-thinking. AIME 2026 goes 20.8 → 89.2; Codeforces Elo goes 110 → 2150. No thinking-versus-non-thinking ablation for the same Gemma 4 model appears anywhere in the document. The generational improvement and the inference-budget increase are therefore confounded in the headline number, and the report never states the thinking budget it allowed.

What makes the case instructive rather than merely damning is that the same report does control, elsewhere, without remarking on it:

• Table 9 (long context) is run "without thinking" on both sides. It is the one clean generational comparison in the paper — and its gains are real and large (LOFT Text Retrieval @128k: 8.6 → 79.5).
• Table 6 (vision) repeats the confound: Gemma 4 thinking versus Gemma 3 27B non-thinking with Pan & Scan.

So the fix is not beyond the authors; it is applied inconsistently and never flagged. This is what Brown means by a bad equilibrium rather than a disagreement — the compute-controlled table exists inside the same PDF as the uncontrolled one, because nobody expects the top-line table to control and everybody expects the long-context table to.

There is a second-order cost specific to this paper. Gemma 4's actual contribution is inference efficiency — a 37.5% smaller KV cache, sub-gigabyte quantization, a drafter head. Those are precisely the gains a compute-uncontrolled grid cannot display, exactly as the grid hid GPT-5.5's efficiency advantage over 5.4. By publishing the standard grid, the report obscures its own best result.

A vendor half-defects: Inkling publishes its own curve (July 2026)#

Inkling's release is the first vendor artifact in this corpus to do what Brown asked for — partially. TML sweeps its controllable effort dial from 0.2 to 0.99 and publishes performance against mean generated tokens on Terminal Bench, HLE, and IFBench: capability curves, exactly AISI's prescription, and the framing is Brown's ("looking at the full cost curve allows developers to choose the best model for each use case"). The curve shows what a grid cell can't: Inkling matches Nemotron 3 Ultra on Terminal Bench at roughly a third of the tokens.

The defection is half because it is asymmetric: competing models appear at their default operating points — single dots against Inkling's curve — and the main benchmark table is still a standard grid at effort=0.99. A curve for me, points for thee is progress on the bad equilibrium, but it also flatters the model with the dial: the comparison Brown's critique actually demands (every model swept over its own budget axis) remains unpublished, now with the additional wrinkle that the effort dial is a trained capability (Inkling) rather than a serving parameter, so the sweep is cheap only for the vendor who trained it in (vendor-claim).

Harness disclosure is not compute control: Kimi K3's footnotes (July 2026)#

Kimi K3's model card (vendor-claim) is the most granular evaluation disclosure in this corpus and the sharpest illustration that granularity and control are different axes. Its 45-row grid is a standard grid. What sits underneath it is not standard at all.

What it discloses, in a class of its own:

• Every score is pinned to a named harness. Not "DeepSWE: 67.5" but "67.5 with the Kimi Code harness, 67.3 with mini-SWE-agent, on the DeepSWE v1.1 task set." Per-model harness pairings are listed benchmark by benchmark — Claude Code for the Claude models, Codex for the OpenAI models, Kimi Code for K3 — because the harness is now a first-class variable in the score, exactly as model-versus-scaffold contribution argues.
• Reasoning effort is named per model. All at max, except GPT-5.5 at "xhigh"; K3 at temperature 1.0 with top-p 0.95 for single-step and 1.0 for agentic tasks.
• Grader-side confounds are disclosed against the vendor's own interest. Fable 5 "hit fallbacks on 35% of the tasks" in the SWE-Marathon evaluation, "which may have negatively impacted its measured performance." On Kimi Code Bench 2.0, Fable 5 hit 13 fallbacks and 1 refusal of 80 tasks; 10 of 80 entered GPT-5.6 Sol's cyber guard; GPT-5.5 refused 3. On Agents' Last Exam, the leaderboard's Fable 5 entry has 40% of tasks "annotated as downgraded." A vendor volunteering that its rivals' numbers were depressed by the rivals' safety machinery is the opposite of benchmark-maxxing.
• Benchmark modifications are stated. SWE-Marathon was run on an H20-recalibrated branch of the pre-v1.1 tasks (Docker images, performance gates and reference oracles recalibrated for H20; correctness and anti-cheat validators untouched). PostTrainBench was re-run on H20 rather than the official H100.
• A self-deflating number is reported. On its own in-house Kimi Code Bench 2.0, K3 scores 73.7 under the rival Claude Code harness and 72.9 under Kimi Code — and the card prints 72.9 in the table. Likewise BrowseComp: 91.2 with a 300K-token compaction strategy, disclosed alongside 90.4 with the full 1M window and no context management.

What it does not do — and this is the point. Nowhere is there a token count, a dollar cost, a wall-clock figure, or a curve. "Reasoning effort = max" is a setting, not a budget: it says all models were told to think hard, not that they spent comparable compute doing so, and it is precisely the reading of "max" that Brown's GPT-5.5-versus-5.4 anecdote shows to be uninformative. The comparison is also structurally asymmetric — K3 is evaluated on Moonshot's own Kimi Code harness while competitors are quoted at "the best score across harnesses" or lifted from leaderboards run by third parties at unknown budgets. Where Inkling published a curve for itself and points for everyone else, Moonshot publishes provenance for everyone and a home-field harness for itself.

There is a second finding here that belongs to safety rather than evaluation: those fallback fractions are the first third-party measurements of how often Fable 5's classifier fallback fires on ordinary technical work.

A price axis inside the grid: Gemini 3.5 Flash-Lite (July 2026)#

DeepMind's Gemini 3.5 Flash-Lite card (2026-07-21, vendor-claim) does something none of the others do: the first two rows of the benchmark table are prices — input and output $/1M tokens — printed for every model in the comparison, the vendor's own and its rivals' alike. Not a footnote, not an appendix, not a separate pricing page: a cost row sitting above SWE-Bench Pro in the same grid a buyer reads.

That is the first symmetric cost disclosure in this corpus. Inkling published a curve for itself and points for everyone else; Moonshot published provenance for everyone and a home-field harness for itself; DeepMind publishes the same cost figure for all four columns and gives itself no special treatment on that axis.

And it is still not compute control. Price per token is a rate; the number a buyer needs is rate × tokens-per-task. The card reports no token counts, no wall-clock, no thinking budget, and no effort setting for any of the four models — so the grid now displays a cost column that cannot be turned into a cost. The gap is not academic: it is exactly the quantity Brown's GPT-5.5-versus-5.4 anecdote turns on. A model priced at half the rate that emits three times the tokens is more expensive, and this table would show it as cheaper. Adding price without tokens can therefore make the grid more confidently misleading than a grid with no cost information at all, because it invites an arithmetic the data does not support (Cost-per-Task Over Cost-per-Token).

A second defect is orthogonal to compute and worth separating out: comparator selection. DeepMind's July 2026 model is set against GPT-5.4 mini and Claude Haiku 4.5 while its own comparator is its immediate predecessor. This corpus has been tracking GPT-5.5, GPT-5.6 Sol and the Claude 4.8/5 generation for weeks; the card does not date its rivals, state whether newer efficiency-tier models from either lab exist, or say when the rivals' scores were collected. Compute control and comparator currency are independent axes of grid honesty, and a vendor can defect on one while quietly failing the other.

So the corpus now has four vendor half-defections with four shapes: Inkling gave the curve without the competitors, Kimi gave the competitors' conditions without the curve, DeepMind's Gemini line gave everyone's price without anyone's token count, and Gemma 4 gave none of it at the top line while controlling quietly further down. None of the four did what Brown asked. The bad equilibrium is not holding because vendors refuse to disclose — Moonshot discloses more than AISI would need, and DeepMind puts dollars in the headline table — it is holding because the axis nobody puts on the page is the one that costs money to sweep. Prices are free to publish; tokens-per-task must be measured.

Picking the axis: METR says dollars, and measures why (July 2026)#

Every artifact above answers "should there be an axis?"; METR's expenditure-horizon note (2026-07-21, empirical) is the first in this corpus to argue which axis and back the choice with a measurement. Its answer is dollars, and the reason is not accounting taste:

• On an agentic AI R&D task, tokens are the minority of the bill. Across six agent runs on the NanoGPT speedrun at up to $10,000 each, experiment compute was ~70–90% of the cost of most trajectories — GPU time spent running and validating candidate training recipes, not model inference. A tokens-only x-axis on this task would be plotting against the smaller 10–30% of what was actually spent, and would rank a token-frugal agent that runs many experiments above a token-heavy one that runs few. Dollars are the only unit that absorbs both.
• Dollars are the unit the decision is actually made in. "Returns to expenditure directly measure the economically-relevant variables for an AI R&D lab that is choosing between spending on human vs agentic labor." Tokens and wall-clock cannot be compared to a salary; dollars can, which is what lets METR put a human curve on the same axis.

That last point is the substantive extension, and it is what none of the four half-defections above attempt: METR plots a human returns-to-expenditure line on the same axis as the agent curve and reports where they cross. Brown asked for the budget to be named; AISI reports reach and reliability against budget; METR adds a second curve to compare against, so the x-axis carries a unit of human labour rather than only a unit of spend. On NanoGPT that calibration is ~$2,500 per 1% speedup, and the six agent runs land at $0–$3,300.

Two caveats this page should carry, because they are the cost of the choice:

• The dollar axis absorbs harness inefficiency. METR's agents had continuous access to 4 H100 nodes and ran experiments freely, which is where the 70–90% came from; METR calls its own harness "likely inefficient" and expects an optimized one to lower cost for the same optimization. So a dollar-denominated curve measures agent-plus-harness, and comparing two labs' curves compares their harnesses too. METR's defence — that shifting the curves horizontally "would not dramatically change the expenditure horizon" — is read off the curves' flatness at the right end, not tested against a cheaper harness.
• Denominating in money imports a price. The $2,500/1% human rate assumes a $150/hour wage, and METR's own sensitivity table shows the resulting horizon moving from $120 to $14,400 for one model across a 10× range of the human-cost assumption. The dollar axis is honest about what was spent and inherits an assumption about what labour is worth — an exposure the token axis does not have.

Routing and consensus are subject to the same budget question#

The critique generalizes to the vendors whose value proposition is a scaffold. A routing / consensus layer (send each sub-task to the right model, or aggregate several models' answers) can indeed beat any individual model on a benchmark. But Brown's principle collapses the apparent win into the same question: does the routed ensemble beat the same model simply thinking longer at equal test-time compute? Consensus-among-models is just another way to spend inference; unless it's compared at a fixed budget — and shown to hold on real use cases, not just the benchmarks it was tuned on — the improvement may be an artifact of spending more, or of over-fitting the eval.

Benchmark-maxxing, measured: harness evolution at a matched budget (July 2026)#

Brown's argument that scaffolding tricks inflate scores was an argument. Wang, Zhu, Hu et al. (arXiv 2607.12227, Ai2 / UW, 2026-07-14, empirical) run it as an experiment against a named method class, and it is the first time this critique appears in the corpus with data attached.

The target is automatic harness evolution — an agent that iteratively rewrites its own scaffold using benchmark feedback and reports the final score on that same benchmark (Agent-Authored Harness Optimization). The observation that makes it a compute-control problem: harness evolution is itself a search procedure that repeatedly evaluates and revises candidates against task feedback. So it belongs on the same axis as parallel sampling and sequential refinement, and the honest comparison holds feedback and inference budget matched across all three. Nobody had done that.

Fix K = 5 for every method on Terminal-Bench 2.1 across three frontier models, and the ordering inverts the published one: parallel sampling averages 72.3 pass@1 without unit-test feedback and 86.0 with, against harness evolution's 67.4 and 75.8 — the evolution arm finishing below the do-nothing baseline (68.2) in the first setting. Evolving a harness on held-out tasks transfers +0.6pp.

Two things generalize beyond harness evolution:

• The pass@1-versus-pass@k split is a portable diagnostic for benchmark-maxxing. Harness evolution's pass@5 (86.2) sits level with parallel sampling's flat 86.0 while its pass@1 (75.8) trails by ten points. The authors' inference: "if harness revision genuinely produced better harnesses, we would expect the improvement to be reflected in pass@1. Instead, the benefit only materializes when we can select among multiple trajectories." A gain that appears only under best-of-N selection is best-of-N, whatever the paper calls it — which is exactly Brown's charge, now with a test that separates the two.
• Search-set and evaluation-set overlap is a second, independent confound. When the tasks a method optimizes against are the tasks it reports on, gains "may reflect adaptation to task-specific patterns." That is Benchmark Contamination and Decontamination's leakage argument arriving through the scaffold rather than through training data, and the remedy is the same held-out private set Brown prescribes — applied to the optimization loop, not just to the model.

The result cuts both ways for this page's thesis, and the paper says so. Its §5.2 concedes Terminal-Bench may be the wrong instrument — agents already score highly, and "a minimal setup consisting of a shell tool and a basic prompt already suffices for most solvable tasks," so there is little for scaffolding to buy. A budget-matched comparison run on a benchmark with no headroom can be uninformative in the other direction, which is a caveat the compute-control literature has not had to state before: controlling the budget fixes one confound and does not manufacture sensitivity where the benchmark has none.

Connections#

• Open-Ended Discovery Harnesses — a comparison that is controlled against one baseline and not the other, in the same table. SwarmResearch and the multi-agent baseline CORAL both run to a $50/task cap on the same runtime and model; the evolutionary baseline EvoX runs 100 iterations at ~$23.50/task average — roughly half the spend, and the arm where the reported margins are largest. Its second experiment repeats the pattern one level down: the winning orchestrator-guided arm uses a stronger model for the orchestrator than the fixed-scaling baseline has anywhere
• Large-Scale Test-Time Compute — the root cause: capability scales with inference budget, so a score without a budget is undefined
• Reward Hacking — benchmark-maxxing is Goodhart at eval-report time, the sibling of reward hacking in the training loop
• Evaluation Awareness & Grader Gaming — the in-model version of gaming a measure; benchmark-maxxing is the same pressure applied by the evaluator rather than the model
• Latent Capability Overhang — the flip side of the same axis: if grids under-report because they under-spend, released models hold capability nobody has paid to reveal
• Benchmark Score Redundancy — the sibling cost reduction on the benchmark-count axis: where this page compresses eval cost by naming the compute budget per benchmark, that page predicts a model's held-out benchmark scores from ~5 probes (the matrix is rank-2); the catch it flags is that BenchPress runs on exactly the uncontrolled public grid this critique targets, so it inherits the grid's heterogeneity and vendor-optimism. Its second source, DeepMind's CollabEval, opens a third cost axis — the annotation budget within one benchmark, where a skipped prompt costs neither inference nor scoring — and is the arm that answers the uncontrolled-grid objection rather than inheriting it: three of its five score matrices are single-lab runs under a uniform harness, and the correlation it exploits is strongest there. Worth noting the direction of travel: it reduces eval spend without touching the compute-per-item this page asks to be reported, so the two compose cleanly (state the budget, then label fewer items at it)
• Task Time-Horizon Scaling — the compute-controlled successor metric: reliable task length at a budget, rather than accuracy at an unnamed one; both face benchmark saturation
• Expenditure Horizon — the axis question answered and then extended: dollars, because experiment compute is 70–90% of an agentic R&D trajectory's bill, and because a human returns curve can be plotted on a dollar axis and not on a token one — the crossing point is the score
• Responsible Scaling Policy Evaluations — the safety-eval instance of the same demand: a threat-model determination reported without its compute budget is as under-specified as a capability score
• Gemma 4 — the worked example: a thinking-mode model benchmarked against a non-thinking predecessor in its own headline table
• Jagged Intelligence (Ghosts, Not Animals) — the confound cuts across model sizes too: Gemma 4's reasoning wins over a 10×-larger predecessor are partly bought with inference, while its knowledge losses are not recoverable that way
• Inference Efficiency as Capability — the gain an uncontrolled grid structurally cannot show, which is why efficient models are systematically under-credited
• The Open-Weight Frontier Gap — Arena Elo inherits the same defect: human preference scored at an unnamed inference budget
• Open-Weight Elicitation Irreversibility — the stakes when the un-budgeted determination is a safety one and the weights are public
• UK AI Security Institute — the independent evaluator that adopted "report capability curves" in practice; empirical backing for the whole critique
• Noam Brown — the source and author of the essay
• Measuring Beyond Accuracy Saturation — this critique applied inside a saturated benchmark: Nadgir et al.'s efficiency axis plots accuracy against tokens and dollar cost (GPT-5.3-Codex ~60% cheaper than an equal-accuracy peer; token-cost and dollar-cost rank agents differently), which is "put compute on the x-axis" restated — and generalizes it, arguing accuracy should also be reported alongside reliability and model-vs-scaffold contribution, not only a compute budget
• Benchmark Contamination and Decontamination — a sibling benchmark-trust critique on a different confound: this page says an unnamed compute budget makes a score meaningless; that page says training-data leakage makes it meaningless (memorization inflates it). Both attack taking the headline number at face value, and both propose recovering what the single number hides (the capability curve here; the clean per-sample distribution there)
• Kimi (Moonshot AI) — the far end of the disclosure axis: per-benchmark harness pinning, named effort settings, self-deflating numbers, and rivals' fallback/refusal fractions volunteered — with no compute budget on any of it
• Capability-Gated Model Fallback — the safety mechanism whose firing rate K3's footnotes accidentally measure from outside Anthropic
• Cost-per-Task Over Cost-per-Token — why a price row is not a cost: the bill is rate × tokens-per-task, and only the rate is ever published
• How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them? — the cluster synthesis: the unnamed compute budget is the first of four corruption channels, and capability curves at stated budgets are the put-compute-on-the-x-axis move of the five-part replacement portfolio
• Agent-Authored Harness Optimization — both halves of this page's critique, on one method class. Cline's headline is a harness-uncontrolled comparison (88.8% at $49.8 on a harness tuned for 17 hours against that benchmark, set beside Fable 5's $552 and GPT-5.6 Terra's $400 from unstated configurations); and the method class as a whole was compute-uncontrolled until Wang et al. ran the matched-budget arms above, which is where benchmark-maxxing stopped being an argument and became a measurement

Open Questions#

• Can you certify "no benchmark-maxxing" — verify a reported score used a stated, reproducible compute budget rather than a hidden best-of-N scaffold?
• Compute has several units (tokens, dollars, wall-clock). They diverge (a more efficient model wins on cost but not always on tokens). Which x-axis is the honest one, and does it depend on the buyer? (AISI reports against tokens on a log axis, and notes that as cost-per-token falls, the high budgets that reveal capability become progressively cheaper to reach.) Partially answered (2026-08): METR argues dollars, and supplies the measurement that makes the argument bite rather than assert — on an agentic AI R&D task, experiment compute is ~70–90% of trajectory cost, so tokens and dollars are not proportional and a token axis omits most of the spend. It also names the buyer the axis is for (a lab choosing between human and agentic labour, which only dollars can price) and demonstrates the payoff: a second, human curve can be drawn on a dollar axis and cannot be drawn on a token one. Still open in two respects — the finding is one task class (long-horizon optimization with heavy experiment compute; a chat or single-shot benchmark inverts the ratio), and the dollar axis buys comparability at the cost of importing a wage assumption and the evaluator's own harness efficiency.
• Does a compute-controlled evaluation regime advantage frontier labs (who can afford the full curve) over academics and third-party evaluators who can't? Sharpened (2026-07): a government evaluator (AISI) does run the full curves — so it is affordable to a well-funded public body — but AISI itself flags that "the most informative evaluations may be expensive" and is researching how to forecast high-budget performance from cheap runs precisely to relieve that cost. So cost is the binding constraint even for a funded third party; it just isn't fatal to one. Partially answered on a sibling axis (2026-07): BenchPress shows the analogous cost problem on the benchmark-count axis is largely solvable — a model's full 133-benchmark scorecard is recoverable from ~5 probes to within ~3.93 points — but that reduces which benchmarks to run, not the compute-per-benchmark the full-curve question is about, so it relieves eval cost on a different axis than the one this question poses.
• Gemma 4 controls for compute in its long-context table and not in its headline table, without comment. Is partial control worse than none — does it lend the uncontrolled tables borrowed credibility?
• Is a grid with a price row but no token counts more misleading than a grid with no cost information? Falsifiable directly: run the four models in DeepMind's table on one agentic benchmark, record tokens-per-task, and check whether the price-implied cost ranking survives.
• Moonshot reports Fable 5 hitting fallbacks on 35% of SWE-Marathon tasks and 40% of Agents' Last Exam tasks "downgraded". Would re-running those with safeguards disabled change the ranking — and should a leaderboard publish the safeguarded score, the unsafeguarded one, or both?

Sources#

• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — No Priors interview (2026-06-26); the benchmark-grid critique, benchmark-maxxing, the bad-equilibrium framing, and the routing/consensus argument (practitioner-opinion)
• Rethinking the Evaluation of Harness Evolution for Agents — Yike Wang, Huaisheng Zhu, Zhengyu Hu et al. (Allen Institute for AI / University of Washington, arXiv 2607.12227, 2026-07-14, empirical): the abstract's two charges (matched feedback and inference budgets; search and evaluation sharing one benchmark), §3 the unified budget formalism, §4.2–4.3 Tables 1–2 and the pass@1-versus-pass@5 diagnostic, §4.4 Table 3 the held-out split, §5.2 the concession that Terminal-Bench may lack both the headroom and the harness sensitivity a fair test needs. Tables 1–3 verified exact against the PDF at ingest (no collapse, no shift); Figure 1 reproduces Table 1's averages. Full treatment on Agent-Authored Harness Optimization
• Gemma 4 Technical Report — Table 5 (thinking vs non-thinking across generations), Table 9 (the controlled long-context comparison), Table 6 (the confound repeated on vision) (empirical)
• More compute, more capability: Why AI agent evaluations need to account for test-time compute — UK AISI (2026-07-02, empirical): "report capability curves"; multiple-budget evaluation, reliability/reach-against-budget, and "minimum informative budgets" as adopted practice
• Inkling: Our Open-Weights Model — the effort-sweep chart: a vendor publishing its own capability curve, against competitors' default operating points (vendor-claim)
• Gemini 3.5 Flash-Lite Model Card — Google DeepMind, 2026-07-21 (vendor-claim): the July 2026 evaluation table with input/output $/1M as its first two rows across four models, no token counts or effort settings anywhere, and GPT-5.4 mini / Claude Haiku 4.5 as the cross-vendor comparators
• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT — Cunningham, Shetty, Cheng & Rush (METR, 2026-07-21, empirical): the "Calibrating to money vs calibrating to time" section and the measured 70–90% experiment-compute share of agent trajectories — the corpus's first argued-and-measured answer to which x-axis, plus the human-curve overlay that turns a budget axis into a comparison. Full treatment on Expenditure Horizon
• Kimi K3 Model Card — §3 footnotes 1–4 (2026-07-26, vendor-claim): per-benchmark harness pairings, named reasoning-effort settings, the H20-recalibrated SWE-Marathon branch, the 73.7-vs-72.9 cross-harness disclosure, BrowseComp with and without 300K compaction, and the fallback/refusal fractions for Fable 5, GPT-5.6 Sol and GPT-5.5