Large-Scale Test-Time Compute

Sources#

• Claude Opus 5 System Card
• Gemma 4 Technical Report
• More compute, more capability: Why AI agent evaluations need to account for test-time compute
• Prompting Claude Opus 5
• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown
• Rethinking the Evaluation of Harness Evolution for Agents
• Rewriting Bun in Rust
• SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery
• Verbalizable Representations Form a Global Workspace in Language Models

Summary#

Test-time compute (inference-time compute) is the compute a model spends thinking about a single query — tokens generated, dollars spent, wall-clock time. Noam Brown (OpenAI research scientist, one of the pioneers of inference-time scaling) argues in his June 2026 essay Implications of Large-Scale Test-Time Compute that this axis has become a primary determinant of capability: "the capability of the model is a function of how much money you put into it." A $10 budget does one thing; $10,000 does much more; $10 million more still. The load-bearing consequence is that the question every evaluation implicitly asks — how capable is this model? — is ill-posed until you name the budget (practitioner-opinion; Brown offers arguments and anecdotes, not measurements).

This page is the hub for the test-time-compute cluster. The root claim lives here; its three downstream breaks live elsewhere: it breaks benchmarking (single-number grids don't control for compute), it strains safety evals (dangerous capability also scales with budget), and it reshapes takeoff forecasts (compute-dependence makes time the binding constraint). The under-explored upside is a capability overhang in already-released models.

The plateau moved out#

The intuitive objection to "just spend more compute" is that performance plateaus — run the model until the benchmark curve flattens, and evaluate to that point. Brown's answer is that the plateau is now weeks of thinking away, too far to reach in practice. In "GPT-3 land" (2022) models couldn't think productively for long, so you could run them to plateau cheaply. Modern models, if "scaffolded reasonably well," keep improving for weeks on some benchmarks before flattening. Brown cites AISI cyber evaluations where models were still improving at 100 million tokens of a single run. So "evaluate to plateau" is no longer a bounded procedure — you have to impose a budget (tokens/cost/time) or plot the whole curve.

Independent empirical corroboration (UK AISI, July 2026)#

Brown's thesis is practitioner-opinion — arguments and anecdotes. The UK AI Security Institute supplied the first independent, government-institute, empirical confirmation, and it is the primary source behind the "still improving at 100M tokens" anecdote above. Its July 2026 study restates the root claim in almost the same words: "model capability is not a single score but a curve over test-time compute" — and if the curve is still rising when the evaluation stops, the reported score is a lower bound, not a ceiling.

The measured version of "the plateau moved out":

• Cyber. ~8% of AISI's narrow cyber tasks were solved only once the per-task budget reached ≥10M tokens (some up to 50M); at smaller budgets those successes were invisible. The latest models kept climbing at 100M+.
• Public benchmarks. Raising the total token budget 1M→10M lifted software-engineering scores ~25% (TerminalBench 2.0, SWE-Bench Pro) and maths/academic scores ~22% (Humanity's Last Exam, to 5M tokens). TerminalBench kept improving even at 10× the budget public evaluations typically report.

Two further AISI findings sharpen downstream pages rather than this one: compute demand scales with human task time (a power law feeding Task Time-Horizon Scaling), and newer models turn extra compute into disproportionately larger gains (reshaping the doubling rate on Task Time-Horizon Scaling and the danger surface on Responsible Scaling Policy Evaluations). The independence matters: a thesis the vault had drawn almost entirely from Noam Brown (OpenAI) is now anchored by a government evaluator's controlled sweeps.

The capability spectrum: when compute helps and when it doesn't#

More test-time compute is not uniformly useful. Brown places every task between two poles:

• Flat — factual retrieval. Ask when Abraham Lincoln was born; if the model doesn't know, a week of thinking won't help (no external lookup). More compute buys almost nothing — a little thinking helps, then it saturates fast.
• Unbounded — guess-and-check search. Sudoku: try random fills, check the constraints, retry. With enough time any puzzle falls, so capability rises without limit in test-time compute.
• Everything in between. Real benchmarks sit somewhere on this line, which is why controlling for compute matters — the same model can look flat or unbounded depending on where the task sits.

The AISI study gives this spectrum an empirical face: gains are largest where an agent can check its own work (code, cyber, maths — run the code, test the exploit), and small where feedback is weak or absent — HealthBench plateaued within every model's usual budget, the measured flat pole. The mechanism is the verifier: cheap self-checking is what converts extra tokens into the guess-and-check gains of the unbounded end.

The overthinking result is the shadow case: on ~7.7% of standard benchmark problems, more generated tokens hurt large models — a reminder that test-time compute is a resource to be allocated well, not a monotone dial. Brown's own view on user practice is that flexible thinking-time (fast when it should be fast, long when the problem warrants) beats always-maximal budgets, because a week-long wait is impractical to iterate against.

The vendor's allocation advice: default down, not up#

Brown's thesis is stated as an upward curve; the deployment advice that comes with Opus 5 is the first vendor guidance to read as an instruction to spend less by default (prompting guide, vendor-claim):

• low and medium effort "produce strong quality at a fraction of the tokens and latency of higher settings." Anthropic recommends using them liberally as the primary control for token cost and response time wherever quality holds, reserving xhigh for demanding coding and agentic work — and re-running an effort sweep on your own evals if you carried defaults over from a prior model.
• The floor of the dial beats the off switch. Thinking can be disabled only at effort high or below, and Anthropic's stated mitigation for the artifacts that appear when it is disabled is to keep it on: "for most tasks, thinking enabled at low effort performs better than thinking disabled at similar cost" (Instruction Compounding). A little thinking at low budget dominates no thinking at the same price — the flat pole of the spectrum above starts above zero, not at it.
• Per-task-class allocation, in practice. Code-review accuracy "holds at lower effort settings, which supports a fast pass at review time and a more thorough pass later" (Review as the Control Point) — an explicit recipe for varying budget by task class rather than setting one global level.
• The dial does not control output length. Effort governs thinking tokens; the visible response is a separate cost center controlled only by the prompt (Output Length Calibration). "Spend less" is two knobs, and an effort sweep will not surface a verbosity regression.

Taken with the effort inversion on the same model (Unproductive Self-Verification), the practical shape of the thesis at deployment is a hump, not a ramp: capability rises with budget until the marginal tokens go into re-checking and over-serving, and the vendor's own advice is to sit below the peak unless the task warrants otherwise.

Scaffolding unlocks the horizon#

The lever that turns "a model" into "weeks of productive thinking" is the harness. Brown: with GPT-3 there was little you could scaffold into a useful week-long run; with modern models you can scaffold "a series of experiments that can run for weeks, for months." His concrete near-future claim: give a well-scaffolded model a long-horizon goal and tell it to "go work on this for a month," and it returns a state-of-the-art result. This is the same capability METR's time-horizon curve measures from the outside — and it is why the harness is where the budget is actually spent.

But not on searching for the harness#

The first budget-matched test of where to spend the marginal token, and it favors the trajectories over the scaffold. Wang, Zhu, Hu et al. (arXiv 2607.12227, Ai2 / UW, 2026-07-14, empirical) fix K = 5 on Terminal-Bench 2.1 across Claude Opus 4.6, GPT-5.4 and GPT-5.4 mini, and give the same feedback and inference budget to four ways of spending it. Parallel sampling — draw five independent trajectories, select one — is the most consistent arm, averaging 72.3 pass@1 without unit-test feedback against a 68.2 do-nothing baseline, and 86.0 with it. Sequential refinement is second on pass@5 (91.8, the highest number in the study). Letting a meta agent rewrite the harness with that same budget averages 67.4 — below doing nothing at all — and its held-out transfer is +0.6pp.

Two things generalize past the method under test. Without an external correctness signal, depth loses to breadth: sequential refinement gains only 1.1 points on average and degrades GPT-5.4, because self-generated feedback is noisy and each revision conditions on the last, so early mistakes compound — while independent draws cannot compound anything. And a gain visible only in pass@k is a gain in attempts, not capability (Compute-Controlled Benchmarking) — the diagnostic that separates spending the budget well from spending it at all. Details and the contrary evidence on Agent-Authored Harness Optimization.

The denominator: efficiency is capability#

And the shape of the spend matters more than its size#

Wang et al. above ask what object to spend the budget on (trajectories vs the harness). SwarmResearch (Virk, Edds, Xia & Zhang, UIUC, arXiv 2607.02807, empirical) holds the object fixed — always solutions — and asks how to shape a fixed budget across parallel width and serial depth. Fixed scaling means n concurrent agents × k serial iterations over a shared git history; the sweep is (5,12), (10,6), (15,4), (20,3), (30,2) within a 60-iteration budget, 3 runs averaged, on five open-ended optimization tasks. Table 2 verified clean against the PDF.

• Wide beats deep, until it doesn't. The optimal fixed configuration is wider than deeper on 4 of 5 tasks — long serial runs "stay within the same neighborhood of solutions and miss stronger neighborhoods discovered by highly parallel runs" — but the widest setting (30, 2) is never optimal. Two iterations is not enough to build anything worth having.
• There is no transferable (n, k). Four of the five tasks have a different best configuration, which is the actual argument for handing the choice to an orchestrator rather than tuning it.
• Adaptive width-at-depth beats the best fixed setting on 4/5, at the same 60-iteration budget, with the orchestrator adding only 7.7% to total output tokens — wide initial exploration, then deep serial iteration applied selectively to what survived.

This does not contradict the parallel-sampling result above; it refines it. Wang et al. show breadth beating depth when the arms are independent and the selector is the model itself. Here breadth also wins, and the extra gain comes from reallocating between breadth and depth mid-run rather than committing to a ratio. One control is missing and matters: the winning arm's orchestrator runs Claude Sonnet-4.6 while every subagent and the entire fixed-scaling baseline run Minimax-M2.5, so this is orchestration by a capable model beating fixed hyperparameters, not orchestration beating them at equal model (Compute-Controlled Benchmarking).

Brown's thesis is stated in the numerator — spend more, get more. It has a corollary he doesn't develop: anything that lowers the cost of a token raises capability at a fixed budget. Gemma 4 (DeepMind, July 2026, empirical) is the corollary made concrete — a 37.5% smaller KV cache, quantization to sub-gigabyte checkpoints, a released speculative-decoding drafter head. Under this page's framing those are not engineering footnotes; they are capability gains denominated in dollars rather than parameters.

The connection runs the other way too. A thinking mode multiplies tokens per query, so shipping one in a model meant to run on a phone is only coherent once tokens are cheap. Gemma 4 ships both in the same release. Developed in Inference Efficiency as Capability.

Connections#

• Unproductive Self-Verification — the counter-instance: Opus 5's FrontierCode score peaks at medium effort and its best GDPval-AA/AA-Briefcase results come from xhigh rather than max, because the marginal compute is spent re-verifying and over-engineering

• Instruction Compounding — why the budget dial substitutes for the disable switch: thinking-off introduces output artifacts that thinking-on at low effort avoids at comparable cost

• Output Length Calibration — the cost the dial doesn't reach: visible-response tokens are prompt-controlled, so effort and verbosity are separate levers

• Claude Opus 5 — and the substitution result: tools beat thinking as a way to spend the budget (Chartography 29.6%→83.0%, BenchCAD 0.366→0.821 when given a container and a cropping tool)

• Automatic vs. Flexible Cognition in LLMs — a mechanistic floor under the thesis: a transformer's only route past its feedforward depth is to write intermediates into the context and read them back, so chain-of-thought is externalized workspace and test-time compute partly buys serial depth the architecture doesn't otherwise have

• Inference Efficiency as Capability — the corollary: cheaper inference is capability, and Gemma 4's efficiency stack is the worked example

• Compute-Controlled Benchmarking — the benchmarking consequence: publish performance against a cost/token/time x-axis, not a single number

• Latent Capability Overhang — the upside consequence: released models can do far more than anyone has paid to extract

• Responsible Scaling Policy Evaluations — the safety consequence: if capability scales with budget, "at what budget do you evaluate for dangerous capability?" is unanswered

• Task Time-Horizon Scaling — the external trendline (reliable task length doubling ~every 4 months) is this thesis measured as a capability curve

• Scale-Dependent Prompt Sensitivity — the overthinking counter-case: more test-time compute can reduce accuracy, so budget must be allocated, not maximized

• Intelligence Explosion Dynamics — compute-dependence is Brown's mechanism for why takeoff is time-bottlenecked rather than instantaneous

• The Verifiability Thesis — the search-heavy gains (the Sudoku pole) are largest where a cheap verifier lets the model check its own guesses

• Open-Weight Elicitation Irreversibility — the governance consequence for published weights: unbounded elicitation budget, no recall

• Single-Rollout Optimization / Asynchronous RL for LLMs — the training-side complement: the RL loop that produces the long-horizon agentic models whose capability then scales with this inference budget

• Dynamic Workflows: An Algebra for Agents — the orchestration-side extension of the axis, now with a bill attached: the Bun Zig→Rust port spent 5.9B uncached input + 690M output + 72B cached-read tokens ≈ $165k over 11 days to complete one task, and the 12:1 cached-read-to-fresh-input ratio says a long fan-out run is priced as a re-reading workload, not a generation one (case-study, first-party). Brown's "capability is a function of how much money you put into it" gets a concrete upper data point, and it is an orchestrated budget rather than a single long context

• Expenditure Horizon — the thesis's accounting corrected on an agentic AI R&D task: across six runs at up to $10,000 each, ~70–90% of trajectory cost was experiment compute, not model inference, so "how much money you put into it" and "how many tokens it generated" come apart by roughly an order of magnitude and only the money axis is complete. Also a rare measured shape for the curve itself — L-shaped, with the newer models still rising roughly log-linearly into the thousands of dollars while two older ones flatten at zero real gain

• UK AI Security Institute — the independent government evaluator that measured this thesis across benchmarks, promoting it from anecdote to empirical fact

• Noam Brown — the source; the researcher who pioneered inference-time scaling

Open Questions#

• Can high-budget performance be predicted from low-budget runs? Brown's proposed research question: forecast the $10,000-inference result using only $10–$100 runs. If the curve is regular, evaluation could project rather than pay in full. Sharpened (2026-07): AISI names this exact problem — "can high-budget performance be estimated from cheaper runs? … the most informative evaluations may be expensive" — as an explicit, unsolved research direction it is now actively pursuing (alongside defining "minimum informative budgets"). Still open, but no longer just one researcher's proposal: a government institute is working it. Sibling axis demonstrated (2026-07): a different projection — across benchmarks rather than across compute budgets — now works: BenchPress predicts a model's held-out benchmark scores from ~5 observed ones because the 84×133 public score matrix is effectively rank-2 (5-probe scorecard recovery to ~3.93 points). Strong evidence the eval landscape is low-dimensional and redundant, but it does not answer the cross-budget forecast — no one has yet shown the $10K-compute result is predictable from $10 runs; only the analogous cross-benchmark result.
• Where does each real task sit on the flat↔unbounded spectrum, and can that be predicted before spending the compute?
• Is there a task class where scaffolding cannot extend the productive-thinking horizon — a hard ceiling no budget crosses? (Brown's factual-retrieval pole says yes for some; the boundary is unmapped.)

Sources#

• Really Big Test-Time Compute in AI Changes Benchmarks, Safety and Research with OpenAI's Noam Brown — No Priors interview (2026-06-26), Noam Brown on his essay Implications of Large-Scale Test-Time Compute (practitioner-opinion)
• Gemma 4 Technical Report — the efficiency corollary: a thinking mode plus a five-lever inference-cost reduction, shipped together (empirical)
• Verbalizable Representations Form a Global Workspace in Language Models — chain-of-thought as externalized workspace: a transformer's only way past its feedforward depth is to write intermediates into the context and read them back
• Prompting Claude Opus 5 — Anthropic platform docs (retrieved 2026-07-25, vendor-claim): the deployment-side allocation advice — default to low/medium, xhigh for demanding agentic work, thinking-on-at-low over thinking-off, re-sweep effort per model
• SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery — Virk, Edds, Xia & Zhang (UIUC), arXiv 2607.02807 (2026-07-02, empirical): §3.1 and §3.4 + Table 2 — the (n, k) sweep within a 60-iteration budget, orchestrator-guided scaling winning 4/5 at +7.7% output tokens, and the unmatched orchestrator model. Table 2 verified clean; Table 1 is collapsed in the raw and is not cited here (recovered on Open-Ended Discovery Harnesses)
• Rewriting Bun in Rust — Jarred Sumner, bun.com (2026-07-08, case-study): the token accounting for one orchestrated 11-day campaign — 5.9B uncached input, 690M output, 72B cached reads, ~$165k at list API pricing
• More compute, more capability: Why AI agent evaluations need to account for test-time compute — UK AISI, More compute, more capability (2026-07-02, empirical): the independent measured confirmation — capability curves over token budget, ~8% of cyber tasks solved only at ≥10M tokens, +25% SWE / +22% maths from 1M→10M, HealthBench as the flat pole