H
Howardism
Howardism · Vol. 03Plate II · No. 02

Evals & Benchmarks, in order.

Notes15DomainEvals & BenchmarksOpen Qs58Newest4 Aug 2026Oldest14 Apr 2026

Benchmark design, contamination, LLM judges, and measuring capability.

Map of Content for the evals-and-benchmarks domain — 15 concepts. The science of measuring models: benchmark validity, contamination, saturation, LLM judges, and production-sourced evaluation. Curated entry point; see Home for all domains.

  • Benchmark Contamination and Decontamination — Sun, Zhan & Gales (Cambridge): per-sample distribution distances expose that aggregate-accuracy decontamination can worsen residual contamination, and Uncertainty-Based Decontamination (UBD) — deep LoRA ensembles exposing memorized samples as confident-but-batch-order-sensitive — debiases without a clean reference model.
  • Benchmark Score Redundancy — Zeng & Papailiopoulos: an 84-model × 133-benchmark public score matrix is effectively rank-2, so BenchPress matrix completion predicts held-out scores to ~4.6 MedAE and a 5-benchmark probe set recovers a full scorecard. DeepMind's CollabEval takes the same premise down to models × prompts and inverts its use — completion output becomes a control variate inside prediction-powered inference, so the redundancy buys unbiased estimates with valid confidence intervals whose correctness survives the matrix not being low-rank at all (and item-level matrices need ~16 components, not 2).
  • Compute-Controlled Benchmarking — Noam Brown's critique: the single-number benchmark grid is broken because it ignores test-time compute — plot performance against a cost budget instead; benchmark-maxxing, held-out private sets, the Goodhart equilibrium that keeps the grid alive, disclosure exemplars (Kimi K3's footnotes, Gemini's price rows), and the first budget-matched test of a named method class.
  • DRACO Benchmark — Perplexity's benchmark of 100 production-sourced deep-research tasks (10 domains, 40 countries) graded by 26-expert rubrics on accuracy/completeness/objectivity/citation; Perplexity Deep Research leads every domain and axis, Claude Opus 4.6 is the strongest non-Perplexity system, factual accuracy is the universal weak spot
  • Expenditure Horizon — METR's continuous generalization of time horizon: the dollar spend at which an agent's improvement on an optimization problem equals a human's at the same budget, measured by crossing an agent's returns-to-expenditure curve with the local returns to human labor — proof-of-concept on the NanoGPT speedrun, where humans cost ~$2,500 per 1% speedup and six agent runs from record #78 re-validate to horizons of $0-$3,300, of which the maintainer would merge ~70% of the ideas but only 50-60% of the speedup
  • LLM-as-a-Judge — Using one LLM to grade another's outputs against criteria/rubrics; DRACO's protocol is per-criterion binary MET/UNMET + justification, weight-aggregated into normalized score and pass rate; key properties — rankings stay stable across judge models while absolute magnitudes vary, and adaptive per-case rubrics (Google's AutoRaters) detect failures but blend them away, motivating stable custom metrics for the behavior under change; a 21-judge / ~541K-judgment audit finds raw exact-match agreement overstates chance-corrected reliability by 33–41pp (kappa deflation) and high test-retest can mask severe position bias, so judges need chance-correction, bias, and cross-benchmark validation before thresholded use; upstream, CalibratedRubric makes the rubric bank the instrument — measurability, informativeness and validity are distinct, and unanimity filters decay with leaderboard size
  • LLM-Judge Validation — UC Berkeley's 21-judge / 9-provider / ~541K-judgment audit (Norman et al., 2026): LLM-as-a-judge validation is systematically under-rigorous — exact-match agreement overstates chance-corrected κ by 33–41pp (kappa deflation, universal across every judge), judge rankings shift up to 14 positions across benchmarks, and high test-retest reliability masks severe position bias (the consistency–bias paradox); distilled into a 5-step Minimum Viable Validation Protocol. Yang et al. (2026) add the judge-version axis: upgrading the evaluator is not a reliability intervention (one robust step in 18, and scaling makes it worse on 2 of 4 datasets), and repeated-sample juries are capped by measured error correlation ρ ≈ 0.66–0.97. Chen et al. (2026) push it down to the rubric item: measurability (judge agreement), informativeness (IRT information) and validity are three different properties
  • Matched Comparisons for Memorization Claims — Cooper et al. (arXiv 2607.12649): a generation rate measured only on training data is not a memorization rate — comparable non-training sequences must be scored by the identical procedure to supply a predictability floor. A conformal test calibrates the threshold to a chosen false-positive rate for populations, a census calibrates a single document against a matched control book, and 'extractable memorization' is redefined to require both a calibrated claim and near-certain generation within a realistic query budget.
  • Measuring Beyond Accuracy Saturation — Princeton-led case study (arXiv 2606.26158): accuracy saturation is not benchmark saturation — re-instrument a saturated benchmark instead of retiring it, because statistically-indistinguishable agents still differ sharply in reliability, cost-efficiency, scaffold contribution, and human-collaboration speedup (CORE-Bench).
  • Orchestration-Plan Simulation — OrchBench (Ren et al.): score a multi-agent orchestration plan without running workers — a deterministic simulator over a fixed task DAG correlates r=0.816 with real Claude Code quality at ~1% of the tokens; transfer coverage dominates agent count, multi-agent wins only under context pressure, and the headline correlation weakens sharply once the weakest planner is dropped.
  • Production-Sourced Evaluation — Building benchmarks from de-identified real production usage rather than synthetic or hand-authored tasks; DRACO's central method — difficulty-proxied sampling, PII-stripping, augmentation, automatable refresh with a human QA gate; representativeness vs. over-specification tradeoff; production traffic as a proprietary eval asset; plus the buyer-side instance, where a customer builds the eval from its own engineering work to decide what to buy
  • Reference-Free Judge Over-Crediting — Reference answers are a first-order determinant of LLM-judge verdicts: without one, judges systematically over-credit wrong answers (up to 85% verdict flips when the reference is added — Kranti & Vajjala), and self-play against a reference-free judge inflates pass rate at flat true accuracy (Zhou); the fix is the judge committing its own answer first.
  • Scale-Dependent Prompt Sensitivity — Large models underperform small ones on 7.7% of standard benchmarks due to overthinking; brevity constraints recover 26pp and fully reverse hierarchy on GSM8K/MMLU-STEM
  • Task Time-Horizon Scaling — METR's measure of the task length AI can complete reliably on its own, doubling roughly every 4 months (up from every 7): Opus 3 ~4min (Mar 2024) → Opus 4.6 ~12hr (2026) → weeks projected for 2027; paired with benchmark saturation (SWE-bench, CORE-Bench)
  • Usage-Telemetry Classifier Validation — Google ATLAS is the first AI-usage-economics program to publish accuracy numbers for the LLM classifiers every such study rests on — and they are humbling: 22.6% exact accuracy on O*NET task assignment and 42.5% on occupation title, against 85.8% human approval of the same labels; the accuracy/approval gulf, its mitigations (presence-not-frequency, task-type aggregation to 70.4%), and what it means for every headline number in the genre

Open questions 58 open

  • Benchmark Contamination and Decontamination4 open
    • SourceCan the ensemble be derived from one released model? The whole method rests on having several checkpoints differing in batch ordering; the authors flag single-checkpoint ensemble derivation (e.g. via cheap perturbations) as the key unlock for adoption. Until then it needs provider cooperation to release a LoRA ensemble.
    • SourceDoes it extend past MCQ? UBD-Debiasing is classification-only today; whether per-decoding-step debiasing recovers the clean distribution for open-ended generation (where contamination shows as near-verbatim reproduction) is untested.
    • SourceIs batch-order sensitivity a reliable memorization tell at pretraining scale? The signal was validated on 3B models with 5 LoRA seeds and induced contamination; whether the high-confidence-high-variance signature survives full-scale pretraining and real (not synthetically injected) leakage is open. Partially answered: Matched Comparisons for Memorization Claims (Cooper et al., arXiv 2607.12649, empirical) settles the second half — real, non-injected leakage is detectable at pretraining scale, on OLMo 2 7B–32B against its published corpus and Llama 3.1 8B/70B against Books3 — but with a different tell: a matched non-member baseline rather than ensemble variance, needing no extra checkpoints. It also bounds what an uncalibrated statistic is worth at that scale (a 10-token verbatim match is ~24% false positive; at 50 tokens the floor is 0.02%). The batch-order signature itself remains untested above 3B.
    • SourceDoes correcting toward an ensemble-averaged uncontaminated reference introduce its own bias? The D_KL/D*_L1 targets are themselves an average over a 5-member uncontaminated LoRA ensemble; how much the "clean" target moves with ensemble size/composition is unexamined.
  • Benchmark Score Redundancy6 open
    • WaitDoes the rank stay 2? The geometry is snapshot-specific; whether a third latent factor emerges as the matrix grows (novel capability profiles, new benchmark families) is the paper's own named signal for when the recipe needs refreshing — and an open empirical watch item. Partially answered (2026-08-04) — on the stakes rather than the fact, by CollabEval. Nobody has re-measured the 84×133 matrix, so whether its rank holds is still open. But CollabEval shows the consequence of the geometry breaking is entirely a function of how you spend it: used as a prediction (BenchPress) a broken rank silently corrupts the answer; used as a control variate inside PPI, correctness is independent of the rank and only efficiency degrades, gracefully, back to the classical sample mean. The question stays live for BenchPress's use case and is largely defused for the inferential one. Two further data points against the geometry being permanent: at the item level rank 2 is nowhere near enough (>16 components for >50% of variance, and IterativeSVD keeps improving through rank 32), and the redundancy is weakest exactly on the largest matrix (MMLU, ~0.66 cumulative EVR at 32 components).
    • SourceCan outlier models be anchored without any scores? BenchPress fails on a model whose capability profile has no close neighbor in the matrix; the authors propose folding in external metadata (training-data composition, architecture, size) to compute model-to-model similarity before any benchmark is run, but do not build it. Partially answered (2026-08-04) by CollabEval — not by anchoring the outlier, but by bounding what its failure costs. Its anchor ablation finds CI coverage flat across anchor-set size and selection strategy, with positive variance reduction even at the smallest sets, and the control-variate construction means a target model uncorrelated with every anchor yields a wider interval, not a wrong one. It also inverts the intuitive anchor policy: Random-$k$ > Top-$k$ > Bottom-$k$, so curating anchors toward the strongest models is actively worse than sampling them. The metadata-similarity idea remains unbuilt.
    • SourceDoes the low-rank treatment carry beyond text/vision? Audio/speech, robotics/embodied agents, and scientific-simulator ecosystems are untested; whether the same rank-2 structure holds there is open. (CollabEval does not touch this — all five of its datasets are text generation.)
    • SourceDoes vendor optimism manufacture the correlation? Four-fifths of the scores are provider-reported and possibly inflated; the paper flags this could inflate apparent cross-benchmark correlation but cannot separate it — would a fully standardized re-evaluation still be rank-2, or is some of the redundancy an artifact of shared reporting bias? Partially answered (2026-08-04) by CollabEval, at the item level. Its AQA (21 systems from one paper, one AutoAIS scorer) and WMT24++ (15 systems, all scored with MetricX under one protocol) matrices are single-lab runs under a uniform harness with no vendor self-reporting anywhere — and they are the strongest arms in the whole study (+17.3% and +12.9% CI-size reduction at 50% labeled). So exploitable cross-model correlation clearly survives standardized re-evaluation; it is not an artifact of shared reporting bias. What this does not settle is BenchPress's actual question, since these are item-level matrices within one benchmark rather than a standardized redo of the 84×133 cross-benchmark grid — the correlation surviving at one granularity is evidence, not proof, at the other.
    • WaitWould a public probe set become a Goodhart target? If "run these 5 benchmarks and infer the rest" becomes practice, the compact probe set is a small, public, high-leverage surface to optimize against — the same eval-report Goodhart pressure Compute-Controlled Benchmarking names, now concentrated on five benchmarks. Unexamined here.
    • SourceIs a valid interval around an autorater's mean worth anything? CollabEval's guarantee is about sampling uncertainty in the mean of the score a benchmark already computes — and on three of its five datasets that score is an autorater or learned metric (GPT-4 Turbo win-rate, AutoAIS, MetricX), which LLM-Judge Validation shows can be systematically wrong in ways no amount of tighter sampling detects. Directly falsifiable: run CollabEval on a task with both human labels and autorater scores, take human labels as $Y$ and the autorater as an anchor row, and check whether the reported CI actually covers the human parameter — or only the autorater's.
  • Compute-Controlled Benchmarking6 open
    • SourceCan you certify "no benchmark-maxxing" — verify a reported score used a stated, reproducible compute budget rather than a hidden best-of-N scaffold?
    • SourceCompute has several units (tokens, dollars, wall-clock). They diverge (a more efficient model wins on cost but not always on tokens). Which x-axis is the honest one, and does it depend on the buyer? (AISI reports against tokens on a log axis, and notes that as cost-per-token falls, the high budgets that reveal capability become progressively cheaper to reach.) Partially answered (2026-08): METR argues dollars, and supplies the measurement that makes the argument bite rather than assert — on an agentic AI R&D task, experiment compute is ~70–90% of trajectory cost, so tokens and dollars are not proportional and a token axis omits most of the spend. It also names the buyer the axis is for (a lab choosing between human and agentic labour, which only dollars can price) and demonstrates the payoff: a second, human curve can be drawn on a dollar axis and cannot be drawn on a token one. Still open in two respects — the finding is one task class (long-horizon optimization with heavy experiment compute; a chat or single-shot benchmark inverts the ratio), and the dollar axis buys comparability at the cost of importing a wage assumption and the evaluator's own harness efficiency.
    • Does a compute-controlled evaluation regime advantage frontier labs (who can afford the full curve) over academics and third-party evaluators who can't? Sharpened (2026-07): a government evaluator (AISI) does run the full curves — so it is affordable to a well-funded public body — but AISI itself flags that "the most informative evaluations may be expensive" and is researching how to forecast high-budget performance from cheap runs precisely to relieve that cost. So cost is the binding constraint even for a funded third party; it just isn't fatal to one. Partially answered on a sibling axis (2026-07): BenchPress shows the analogous cost problem on the benchmark-count axis is largely solvable — a model's full 133-benchmark scorecard is recoverable from ~5 probes to within ~3.93 points — but that reduces which benchmarks to run, not the compute-per-benchmark the full-curve question is about, so it relieves eval cost on a different axis than the one this question poses.
    • SourceGemma 4 controls for compute in its long-context table and not in its headline table, without comment. Is partial control worse than none — does it lend the uncontrolled tables borrowed credibility?
    • SourceIs a grid with a price row but no token counts more misleading than a grid with no cost information? Falsifiable directly: run the four models in DeepMind's table on one agentic benchmark, record tokens-per-task, and check whether the price-implied cost ranking survives.
    • SourceMoonshot reports Fable 5 hitting fallbacks on 35% of SWE-Marathon tasks and 40% of Agents' Last Exam tasks "downgraded". Would re-running those with safeguards disabled change the ranking — and should a leaderboard publish the safeguarded score, the unsafeguarded one, or both?
  • DRACO Benchmark3 open
    • WaitThe benchmark is static; the construction pipeline is automatable. Will Perplexity actually refresh it, and does a vendor-built benchmark on which the vendor's own product wins stay credible over time?
    • SourceRankings are judge-stable but magnitudes aren't — how much do absolute scores move under a non-Gemini judge, and does that matter for cross-paper comparison? Partially answered (2026-08-04) by Yang et al. (2026), with a construct caveat. They hold the candidate pairs fixed and vary only the judge, which is this question's design: on adversarial LLMBar the measured quantity spans 0.463 (Qwen3-1.7B) → 0.900 (GLM-5.1) across ten judges, and no judge leads all four datasets — so the spread from judge choice alone is large, and its direction is dataset-dependent, which is the part that kills naive cross-paper comparison. Two calibrations run the other way, though. Adjacent releases of the same family (MiniMax M2 → M2.7) move accuracy by at most 0.022 and never significantly under paired McNemar, so a routine provider upgrade is a small perturbation; and a single judge held fixed still flips 14.7% of its own verdicts under pure A/B reversal, meaning some of what looks like judge-choice variance is within-judge protocol noise a position-randomization fix would remove. The caveat that keeps this open: their "absolute score" is a judge's agreement accuracy against human preference labels on pairwise items, not a system's normalized rubric score on long-form reports. Rubric-weighted grading of open-ended research output has no gold pairwise label to be accurate against, so the magnitude of DRACO-style score movement under a swapped judge is still unmeasured.
    • SourceDoes the production-sourced, expert-rubric method generalize cheaply to non-English, multimodal, and multi-turn deep research?
  • Expenditure Horizon3 open
    • SourceThe existence of a crossing is assumed, sourced to RE-bench and PaperBench rather than measured here. On which frontier optimization problems, and at what budgets, do agent returns actually stop diminishing faster than human returns — the event that both retires this metric and trips the RSP threshold?
    • SourceDoes the horizon ranking survive a compute-efficient harness? METR's agents spent 70–90% of budget on experiments with continuously-available nodes, and the claim that shifting curves left leaves horizons roughly unchanged is read off curve shape rather than tested.
    • SourceThe hybrid curve — human assisted by agent — is the quantity a lab actually buys, and it is illustrated but never measured; existing evidence points both ways (dominance if humans allocate LLM effort well, degradation if they don't — Becker et al. 2025). What would a runnable hybrid-expenditure experiment look like at a cost anyone would pay, and does it belong on the same dollar axis?
  • LLM-as-a-Judge3 open
    • How far can the judge's absolute calibration be trusted for thresholded decisions (ship/no-ship, RSP gating) as opposed to rankings? Partially answered: Norman et al. (2026) show absolute calibration is worse than the reported number implies — the metric practitioners cite (exact-match agreement) systematically overstates chance-corrected reliability by 33–41pp on balanced label sets, so an "85% agreement" judge is really at κ ≈ 0.48 (moderate), and a threshold set on raw agreement is calibrated to an inflated figure. The "rankings are safe" fallback is also bounded — stable across judge-model choice (DRACO) but fragile across benchmark choice (up to 14 rank positions). It does not close the question: it prescribes a pre-deployment checklist (the Minimum Viable Validation Protocol) rather than declaring thresholded judge decisions safe, and defers calibration proper (ECE/Brier) because most providers don't expose logprobs. Sharpened further by Kranti & Vajjala (2026): absolute scores are not just chance-inflated but reference-inflated — with no gold answer in the prompt, judges systematically over-credit incorrect answers, so a threshold set on reference-free correctness is calibrated to an inflated number, and adding the reference flips up to 85% of verdicts (worst in low-resource Telugu). A human study confirms the reference-driven stricter verdicts are more correct, so the reference-free absolute score is genuinely wrong, not merely a different opinion. Demonstrated on three judge models (open-weight Qwen3-32B / Gemma3-27B, closed Gemini-3.1-Flash-Lite) in zero-shot binary QA across English/Arabic/Telugu — magnitudes are model- and language-dependent, not universal.
    • SourceCan a fully-autonomous, well-aligned rubric+judge pipeline match expert-authored rubrics, removing the human bottleneck DRACO still relies on? Partially answered (2026-08-04) by Chen et al. (2026) — and the partition it draws is the useful part. CalibratedRubric removes the expert from filtering, weighting and sizing the bank, with no human labels and no gold judge required: measurability filtering lifts human-gold κ 0.604 → 0.743 on JudgmentBench, and IRT selection reaches the target rank correlation with 49 instead of 131 rubrics. It does not touch authoring or validating them — "we take the candidate pools as given," the method "cannot recover dimensions absent from the candidate pool," and measurability is stated outright to be "necessary but not sufficient for substantive expert endorsement." So the automatable half is the psychometric half; the half DRACO spends 26 experts on — deciding what a good report is in the first place — is untouched. Two measurements bound how close the autonomous pipeline gets. (i) Against a human reference ranking on FinResearch, the task-adaptive scorer and the plain binary baseline achieve the identical Spearman ρ = 0.8833 — so the paper's own front end buys discrimination and cost, not external validity, and 0.8833 is what this generation of automated rubric grading matches human judgment at. (ii) The judge–human gap survives the filter: LLM judges assign positive labels at 55.6–62.9% against a 47.1% human-gold rate, "a systematic judge–human mismatch that measurability filtering does not fully eliminate" — a directional generosity bias in exactly the direction Reference-Free Judge Over-Crediting measures.
    • SourceWhen does judge-lineage bias actually flip a result, versus merely shift magnitudes? Partially answered (2026-08-12), and the answer is "it flips" — on a detection task, at low evidence: Greptile (case-study) has two frontier models review two 500-PR corpora, one authored by each family, and the ranking of which reviewer is better reverses between the corpora (Opus 53.7 vs GPT 62.0 on Claude-authored PRs; Opus 60.0 vs GPT 50.5 on Codex-authored PRs) while the reviewers' pooled averages sit 0.6pp apart. So on this task lineage does not shift a magnitude — it decides the rank, and a leaderboard built on either corpus alone would report the opposite winner. Three things keep it partial: the grading task is bug detection rather than quality scoring, so the bias surfaces as recall rather than as generosity and may not transfer to rubric grading; the ground truth is vendor-built with no released artifact, no agreement statistic and no validation of the matching judge; and one arm's review prompt was tuned against the outcome metric, which cannot manufacture a crossover but does make the magnitudes soft. The clean version of the experiment — a third-family reviewer across both corpora, which would separate lineage from stylistic fit — is named on that page and has not been run.
  • LLM-Judge Validation4 open
    • SourceThe MVVP validates reliability and bias; calibration proper (ECE/Brier) is deferred for lack of provider logprobs. How far can a judge's absolute score be trusted for a threshold once confidence calibration is measurable?
    • SourceAll judges were run with thinking suppressed. Does reasoning-on flip the consistency–bias paradox, or just move the numbers?
    • SourceHosted endpoints drift silently between provider updates. How stable are these agreement/bias profiles over a longer horizon than five weeks — and should judge validation be continuous rather than one-shot? Partially answered (2026-08-04) by Yang et al. (2026), on the announced-upgrade sibling of the question rather than silent drift. Across four released MiniMax generations (M2 → M2.1 → M2.5 → M2.7) on four datasets, adjacent accuracy moves at most 0.022 and not one of the nine adjacent McNemar tests reaches even uncorrected p < 0.05 — and because those tests are paired on parse-shared examples, this is stability of the individual verdicts, not merely of the aggregate. So a deliberate version step at the top of the capability range is a small perturbation to the agreement profile. Three things keep this open. (i) These are version-labeled releases you can pin, not the unannounced same-endpoint drift the question is about — nobody has re-measured a fixed endpoint over months. (ii) Stability of accuracy is not stability of bias: position-flip rates still span 0.117–0.147 across the MiniMax releases, and no one tracked whether that band moves within a single version. (iii) The finding runs the other way on the parameter axis, where a step does move things and can move them down (Qwen3 1.7B → 32B costs 0.065 on Judge's Verdict), so "upgrades are safe" is not the lesson — "upgrades are a measurement event that must be re-validated, and the null case is the lucky case" is. That is an argument for continuous validation, from the direction of the one axis where it was cheap to check.
    • Does the paradox generalize beyond position bias — i.e., are there other biases (self-preference, lineage) that high test-retest also masks? Partially answered by Kranti & Vajjala (2026): yes — reference-presence sensitivity is a large one. Their temperature-0 judges (thus perfectly reproducible) systematically over-credit incorrect answers in no-reference settings, an invalidity invisible to any reliability metric until a gold answer is added, which flips up to 85% of verdicts and lifts human-alignment sharply (e.g. Gemma3-27B 0.34→0.85 NR→RV). It does not close the question — self-preference and lineage remain unisolated (their design deliberately overlaps judge and generator but doesn't attribute the effect), and it is a different bias on a different (multilingual QA, three specific judges) setup, not a re-run of this study's position-bias protocol. The self-preference half then largely resolves negatively (2026-08-04), via Zhou (2026): errors optimized against a self-judge transfer to judges from other families that were never in the loop (Llama 0.480 → 0.568, Gemma 0.764 → 0.918) and to same-family judges 3.5× larger (still 77%), with a three-family unanimous-accept ensemble passing 55% and judge acceptances pairwise correlated at φ = 0.29–0.38. So lineage amplifies — the self-judge is the worst single cell at 0.906 — but is not the mechanism; the bias is a shared property of the candidate-conditioned channel. This also adds a bias class no reliability metric on this page can reach, because it is not a property of the judge at all: the same judge, unchanged, is valid before optimization (discrimination 0.31) and invalid after (0.09), so any one-shot validation — including the full MVVP — certifies a judge that will be true only until something starts optimizing against it. That is a direct argument for the continuous-validation question two bullets up.
  • Matched Comparisons for Memorization Claims3 open
    • SourceDo the calibrated rates hold for instruction-tuned production models? Everything here runs on open-weight base models with known or inferable training corpora; the parrot demonstration is the only instruction-tuned experiment and it is deliberately degenerate. Whether matched controls can be constructed at all for a closed production model — where the cutoff is approximate and the corpus unpublished — is the gap between this method and the deployment setting where the copyright claims actually land.
    • SourceDoes the near-verbatim ε-ball resolution change the answer, or just the accounting? The near-verbatim test is a strictly broader instance of the same hierarchy, computed with a beam-search lower bound rather than exactly, and its floors differ from verbatim ones by a decade on at least one pair (Collins: 10⁻²⁷ verbatim vs 10⁻²⁶ near-verbatim). Whether calibrated rates move as much as thresholds do is not reported.
    • WaitWhat is the right realistic query budget? 10⁵ is picked "for illustration purposes," and the k-CBS result shows a smarter decoder shifts the frontier at fixed cost. Any threshold that determines whether text counts as extractable in a legal or policy setting needs a defensible budget, and there is no principle here for choosing one.
  • Measuring Beyond Accuracy Saturation5 open
    • SourceDoes re-instrumentation generalize past reproducibility? CORE-Bench Hard was chosen precisely because it has a direct human counterpart, clean OOD axes, and multiple practical dimensions. Whether the six-axis treatment yields comparable signal on benchmarks without those properties (e.g. closed-form reasoning benchmarks with no human-workflow analog) is untested.
    • SourceIs the human-uplift result real or a demand effect? The reproducers are the paper's own coauthors, there is no ground-truth correctness, and n = 20 papers / 5 participants. The 2.11× speedup is statistically significant but the authors themselves cannot rule out participant bias — an independent, blinded replication is the missing evidence.
    • SourceWhich non-accuracy axis actually predicts deployment value? The paper measures six axes but does not rank them by decision-relevance for a downstream deployer. If you can only measure one beyond accuracy, is it reliability, efficiency, or scaffold contribution — and does the answer depend on the use case? Partially answered (2026-08-04) — a seventh candidate rather than a ranking: Leni proposes loop telemetry and gives it the most direct claim to decision-relevance any axis here has made. Because its verification loop is fully instrumented, the measured catch/fix/false-alarm rates convert straight into marginal returns on the next engineering decision — raising the catch rate is worth up to +8 pp, raising the fix rate at most +0.5 pp — so the axis does not merely separate systems, it names which component to fund. It also answers the "does it depend on the use case" half affirmatively and specifically: the argument for keeping the checkpoint at all is that its value concentrates where a reliability SLA's tail sits, which is a use-case-conditional claim by construction. Still open as posed, because no source has ranked the axes against each other, and this proposal comes from a vendor instrumenting its own system.
    • SourceCan the model-vs-scaffold decoupling be made routine? The oracle-router result (every task solvable by some scaffold → 100%) implies large headroom from scaffold routing, but requires per-task oracle knowledge. Whether a practical router can approach the oracle without it is open, and would turn a measurement into a capability. Partially answered (2026-08-04) — on an adjacent axis: Leni ships a practical router and it pays. A 0.5B step-type classifier dispatches each step across two model families (cheap models for classification, frontier reasoning for multi-hop synthesis, strong grounding for vision), and internal estimates credit it with ~4 pp of GAIA accuracy at net-negative cost — cheap steps subsidise extended reasoning on hard ones. So routing is deployable at a price low enough to run on every step, which was the practical objection. It does not settle the question as posed, on an axis mismatch that matters: this routes models per step inside one fixed scaffold, not scaffolds per task, and no oracle comparison is run, so what fraction of available headroom the router captures is unmeasured. Vendor-authored, internal single-run attribution.
    • SourceDo "living benchmarks" outrun their own maintenance? v1.1 and OOD are to be updated as new validity threats surface via log analysis, which the authors note is non-exhaustive. Whether continuous log-analysis-driven maintenance is sustainable — or itself becomes a Goodhart target once developers know the rubrics — is unexamined.
  • Orchestration-Plan Simulation3 open
    • SourceDoes the r = 0.816 sim-to-real correlation survive on a set of comparable planners? Leave-one-out puts it at 0.421 (p = 0.500) once the weakest of six models is dropped, and only two of seven rows clear p < 0.05, so the fidelity claim may be entirely the strong-vs-weak spread. Settling it needs a run over ten or more frontier-tier planners with the weak tail excluded.
    • SourceDoes the multi-agent / single-agent crossover survive real execution? In simulation the advantage falls from +0.302 at 16k to +0.007 at 128k and reverses on 82% of model-problem pairs — but the simulated single agent suffers only compression loss, with no attention degradation, distraction, or long-context recall failure priced in, and the paper never runs the single-agent comparison for real. A real 128k single-agent-vs-multi-agent arm on the same tasks would settle whether 128k is the true crossover or an artifact of a generous single-agent model.
    • SourceIs the transfer-coverage result about orchestration or about the penalty? All measured separation between a strong and a weak planner vanishes when λ goes to 1 or omitted transfers are auto-completed, and λ = 0.5 was chosen for discriminative power rather than fitted to observed handoff loss. What would settle it: an execution study measuring how much downstream quality an actually-omitted handoff costs in a real framework.
  • Production-Sourced Evaluation3 open
    • SourceHow much does augmentation distort the distribution it claims to represent? Is there a measurable representativeness loss between raw queries and augmented tasks?
    • SourceDifficulty-by-thumbs-down biases toward current failures — does that make the benchmark a moving target that flatters the next model trained on those failures?
    • SourceCan the privacy pipeline (no human sees raw queries) be trusted/audited well enough for regulated domains (medicine, law) where the source traffic is most sensitive?
  • Reference-Free Judge Over-Crediting5 open
    • SourceDoes the two-stage pipeline transfer beyond binary QA? Calibration + sensitivity are demonstrated on binary correct/incorrect factual QA. Do the same probes diagnose reference-sensitivity for graded rubrics, long-form generation, or multi-turn agent transcripts, where "the reference" is a rubric rather than a gold answer?
    • SourceIs over-crediting a knowledge gap or a generosity prior? In low-resource Telugu the judge flips the same extracted answer once a reference appears — is the NR generosity driven by insufficient task knowledge (calibration failure) or by a default lean-toward-CORRECT that a reference overrides? The two have different fixes (better judge vs. always supply a reference). Partially answered (2026-08-04) by Zhou (2026): neither, at least on English math — it is candidate anchoring. The same Qwen3-4B judge that accepts 0.91 of wrong answers when scoring a shown candidate solves those problems itself at 0.93 accuracy and drops to 0.012 false positives once required to commit its own answer first, with the candidate still fully visible. So the knowledge is present and a lean-toward-CORRECT is not the mechanism either: conditioning on the candidate is. Corollary 1 turns this into a test any deployment can run — a measured FPR above 1 − solve-acc certifies the verdicts as anchored, and Corollary 2 prices the excess in bits (0.719 against a 0.07 ceiling ⇒ ≥ 1.2 bits of candidate leakage). It does not close the question for this page's setting: Zhou's judges are mid-size open-weight models on exact-matchable grade-school math, where a knowledge gap is implausible by construction, and the low-resource Telugu case — where the judge may genuinely not know the answer — is exactly where anchoring and ignorance are hardest to separate.
    • SourceHow much does self-/same-family overlap contribute? The design deliberately overlaps generator and judge (Qwen3-32B self-judging; Gemini/Gemma family) and Qwen self-judging is the most reference-sensitive, but the paper does not isolate a self-preference effect from a low-resource effect. When does judge–generator lineage amplify reference-free over-crediting? Partially answered (2026-08-04) by Zhou (2026): lineage amplifies it but does not cause it. The self-judge is the worst cell (FPR 0.906 post-self-play) but errors optimized against it transfer to judges from other families that were never in the loop — Llama-3.1-8B 0.480 → 0.568, Gemma-3-12B 0.764 → 0.918 — and a three-family ensemble still accepts 55%, with acceptances pairwise correlated at φ = 0.29–0.38 (581 unanimous accepts where independence predicts ≈ 497). The residual: this measures transfer of self-play-manufactured errors, not a controlled self-preference ablation, and Kranti's question is about reference-sensitivity on organic responses.
    • SourceDoes the effect shrink with stronger or thinking-enabled judges? All judges are ≤ mid-tier at temperature 0 with no reasoning channel. Would a frontier reasoning judge over-credit less in NR, or just flip at different rates? Partially answered (2026-08-04): scale alone does not fix it, and the reasoning half is confounded. Zhou sweeps judge size to 14B (3.5× the policy) and every judge's discrimination collapses under optimization — 14B still accepts 77% of the hacked errors, and the strictest base judge is driven to the highest post-hoc false-positive rate. Reasoning is not cleanly separated: the recompute prompt (solve it yourself, reject when uncertain) leaves FPR at 0.719, but the blind-solve verifier that reaches 0.012 runs with reasoning enabled, so thinking-on and de-anchoring co-vary in the arm that works. A frontier reasoning judge scoring a shown candidate remains untested.
    • SourceDoes commit-first survive the loss of an exact-matchable answer? The de-anchoring fix accepts only when the judge's independently committed answer exactly matches the candidate's — which is what makes its false-positive rate provably bounded by 1 − solve-acc. Zhou names extending commitment to open-ended outputs ("committed rubrics, executable tests") as future work, and that is the regime the vault actually cares about: a taste, quality or report-grading reward has no exact match to accept on, and a committed rubric compared for partial agreement reintroduces exactly the graded, plausibility-shaped judgment the fix removes. Does a commitment survive contact with an output that can only be scored by degree, or is the whole result a property of tasks with a checkable final token?
  • Scale-Dependent Prompt Sensitivity5 open
    • SourceDoes the RLHF length-bias hypothesis replicate when tested against base (non-instruct) model variants directly? If verbose generation were primarily pretrained, base-model verbosity differences should match instruct-model differences.
    • SourceWhat problem characteristics predict prompt sensitivity? An automated classifier would make scale-specific prompting deployable.
    • SourceHow does the overthinking effect interact with tool-using agents? If brevity helps large models but tools require structured reasoning, the optimal prompt is not uniformly brief.
    • SourceDo reasoning models (o1, DeepSeek-R1 style) exhibit different overthinking dynamics than instruct models? Their trained behavior is explicitly to generate long CoT — does brevity intervention hurt them?
    • SourceIs BoolQ's functional-elaboration exception a clean taxonomy boundary, or does every task type have a context-dependent optimal length?
  • Task Time-Horizon Scaling2 open
    • WaitIs the 4-month doubling a stable regime or a local steepening? The trend's shape (exponential vs S-curve) is undetermined. Sharpened (2026-07): AISI adds that the doubling rate itself is budget-dependent — the same cyber suite doubles ~60% faster measured at 50M than at 2.5M tokens/task — so the headline rate is undefined without naming the eval budget. The stability question is now entangled with a budget question, not just an exponential-vs-S-curve one.
    • Time horizon is measured on task baskets that themselves saturate; what replaces them once weeks-long tasks become measurable — and who builds those tasks? Partially answered / reframed (2026-07): Nadgir et al. argue don't replace — re-instrument: they take CORE-Bench (cited above as saturating in 15 months) and show it still discriminates agents along six non-accuracy axes after accuracy saturates, so "what replaces a saturated basket" can be "keep it and measure differently" rather than "build a harder one." This addresses accuracy saturation, not the length-metric saturation this page's basket faces, and does not answer who builds the next weeks-long tasks — so it reframes the retire reflex without closing the question. Second partial answer, from METR itself (2026-08): Expenditure Horizon is the metric's own authors proposing a third option — neither "build a harder basket" nor "re-instrument the old one," but change the object measured: drop the basket entirely for a single frontier optimization problem humans are still actively improving, and read capability off where the agent's dollar curve crosses theirs. It answers "who builds the tasks" by not needing tasks to be built — the speedrun leaderboard is the instrument and it refreshes itself. But it buys that with two costs the basket doesn't have: it needs continuously-scored, smooth-returns problems (a narrower class than a general task basket), and it needs a per-problem estimate of the returns to human labour, which on NanoGPT took two contributor interviews, an LLM judge over 82 PRs and a bootstrapped correction factor to produce — and still lands on a number METR calls "highly uncertain."
  • Usage-Telemetry Classifier Validation3 open
    • SourceThe synthetic ground truth is Gemini-generated and Gemini-classified. How much of the 22.6% is real capability vs same-family cues, and how much is the classifier being penalized for picking a better fit than the seed? Google says it cannot measure this without inspecting private logs.
    • SourceWould running ATLAS's validation battery against the AEI and OpenAI classifiers explain the 2–4× cross-study disagreements, or are the gaps driven by sampling and product mix instead?
    • NowHuman raters disagree with each other on 42–48% of 3-digit occupations. Is there a principled way to establish the ceiling a classifier could reach, so accuracy can be reported relative to it rather than to 100%?