Benchmark Score Redundancy

Sources#

• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
• CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion
• You Don't Need to Run Every Eval

Summary#

A modern model release reports 40+ benchmark rows, and the same suites are run and re-run far more often behind the scenes to track training, compare design choices, and select checkpoints — thousands of dollars and days of wall-clock per full run. Zeng & Papailiopoulos ("You Don't Need to Run Every Eval", Microsoft Research AI Frontiers, arXiv 2606.24020, empirical) ask whether you have to: they compile a public score matrix of 84 frontier models × 133 benchmarks (2,604 observed cells, 23.3% filled) and find it is effectively rank-2 — a model's scores across all 133 benchmarks are largely determined by just two numbers. Building on that, BenchPress predicts any missing cell, and a handful of probe benchmarks recovers most of a model's scorecard for free. The operational claim: run a small set of probes and infer the rest, instead of running every evaluation independently.

This is the eval-cost story on a different axis from the Large-Scale Test-Time Compute cluster: that cluster compresses cost along the compute-budget axis (name the budget, don't run to plateau); this paper compresses along the benchmark-count axis (run a few benchmarks, predict the rest). They are complementary, and — as the paper itself flags — the second inherits the heterogeneity the first warns about, because BenchPress runs on top of the same uncontrolled public grid.

The page carries a second source with the same premise and a different purpose: CollabEval (DeepMind, July 2026) drops a granularity to the models × prompts matrix and uses completion not to predict a score but to bound one — trading BenchPress's point estimate for an unbiased estimate with a valid confidence interval. Its architecture also supplies the answer to this page's biggest liability: it is built so that a broken low-rank geometry costs efficiency, not correctness.

The matrix is effectively rank-2#

The score matrix is heterogeneous and sparse by construction. A crawl of release blogs, system cards, technical reports, HuggingFace cards, and six primary leaderboards (MathArena, ARC-Prize, Terminal-Bench, LMArena, Epoch AI, LiveBench) yields a raw May-2026 pool of 188 models × 316 benchmarks at only 7.6% fill; canonicalizing near-duplicate model variants and benchmark configurations, then filtering to ≥15 observations/model and ≥8/benchmark, produces the analysis matrix (84 × 133, 23.3% fill). Roughly four in five scores come from the model provider's own materials.

Two independent diagnostics converge on rank 2:

• Held-out completion bottoms at rank 2. Sweeping the rank of Soft-Impute matrix completion, held-out Median Absolute Percentage Error is minimized at rank 2 and rises for higher ranks — the same curve in raw-score and logit-transformed space. This is the operational reason to use rank 2: it is the rank that best predicts unseen scores, not merely the rank that best compresses seen ones.
• Fully-observed submatrices show matching geometry. Mean-centering each benchmark column (equivalent to PCA) and taking SVD of the largest complete submatrices, the top two components explain >90% of cross-model variance at every shape tested (4 bench × 42 models: 97.6%; 7×11: 91.2%; 10×7: 95.0%; 13×6: 95.9%), with stable rank ~1.1–1.3 — one direction dominates, a second cleans up most of the rest.

Finding 1. The 84 × 133 model-benchmark score matrix behaves as an effectively rank-2 prediction problem.

This sharpens, on a heterogeneous frontier-era matrix, a line of prior low-rank findings the paper builds on: Burnell et al. (2023) — three latent factors (reasoning, comprehension, core language modeling) explain most variance across 27 HELM tasks; Ilić & Gignac (2024) — a psychometric g-factor accounts for 85% of variance across 12 benchmarks / 591 Open-LLM-Leaderboard models; and Burnham/Epoch (2025) — an independent PCA decomposition of the Epoch Capabilities Index into "general capability + provider-specific residual" (informally "claudiness"), a rank-2 structure matching this one. The redundancy is not new; a prediction system built on it, with explicit failure analysis, is.

BenchPress: the predictor#

The authors evaluate all combinations of 7 feature transforms × 12 prediction methods (identity/log/logit/arcsinh/probit/sqrt/quantile transforms against benchmark-mean, model-mean, KNN, per-benchmark and per-model regression, Soft-Impute, NMF, PMF, nuclear-norm, ALS, and an MLP), grid-searching hyperparameters, under a 50%-per-model holdout (3 folds × 10 seeds, ~20K test predictions per pair). Per-benchmark and per-model regressions score marginally best on raw error but leave some cells unpredicted (<100% coverage). Among full-coverage methods the winner is logit-space, bias-decomposed alternating least squares (Koren et al. 2009), rank 2, λ=0.1 — reaching 4.6 median absolute error (score points) on held-out cells at 100% coverage.

Finding 2. Logit-transformed, bias-decomposed rank-2 ALS matrix completion gives near-best score-prediction accuracy while predicting every missing model-benchmark score.

The recipe: (1) logit-transform percentage scores (leave non-percentage scores like Codeforces rating / Arena Elo native); (2) standardize each benchmark column; (3) fit bias ALS = global level + model offset + benchmark offset + a rank-2 residual interaction (the biases absorb row/column level so the low-rank term only models residual model×benchmark structure); (4) invert the transforms. One fit predicts the whole matrix deterministically.

Versus an LLM predictor. Prompting GPT-5.5 to predict a target cell from five nearest-peer examples beats BenchPress when real model and benchmark names are visible (3.50 vs 4.63 MedAE) — but that edge is leaderboard memory, not structure: blind the names and the LLM (4.70) is no better than BenchPress (4.63) and far less scalable, since it pays per cell while BenchPress fits once (Finding 3).

What it enables#

• Budgeted scorecard recovery (Finding 4). Greedily selecting probe benchmarks, five probes recover a model's full 133-benchmark scorecard to a pooled 3.93 MedAE points: {GPQA-Diamond, HLE, Codeforces, MMLU-Pro, ARC-AGI-1}. Restricting to a low-inference-cost allowlist barely hurts — {GPQA-Diamond, MMLU-Pro, Aider Polyglot, MATH-500, AIME 2026} reaches 4.55 — because the highest-signal probes are already cheap. All selected sets lean heavily on reasoning/math benchmarks: reasoning is the dominant axis of variation, so those probes triangulate the rest. Two blunt lessons: which benchmarks you run matters far more than how many (greedy sets clear a random baseline at every budget), and greedy selection matches near-exhaustive search.
• Rankings survive (Finding 5). Completed scores preserve 92.1% of same-benchmark pairwise model orderings when the true gap is ≥5 points (88.0% at ≥2 points; 83.8% including near-ties), across hundreds of thousands of comparable pairs — prediction error rarely flips a meaningful ranking decision.
• Brand-new models (Finding 6). Under a strict temporal cutoff — train BenchPress only on models released before the target, so it has zero prior information about the new release — revealing 5 seed scores on the new model brings predictions within 4.83 points; 10 seeds tighten to 2.57 (vs 9.20 with a single seed, across 27 targets). A small seed evaluation on the new model buys more than additional historical models.

When to trust a prediction#

Per-model error varies ~40× across the 84 models, so a trust layer matters. Hypothesis tests (Spearman / paired Wilcoxon, reject at p<0.01) identify what drives prediction quality:

• Benchmark-side: wider score spread across models is harder to predict; more observed model scores and the presence of at least one strongly-correlated neighbor benchmark make it easier. Same-category metadata does not help — the predictor exploits observed correlations, not benchmark labels.
• Model-side: reasoning models are easier than non-reasoning; higher-scoring models easier than lower; a strongly-correlated peer model, more observed scores, and recent training anchors all help. Provider identity does not matter — BenchPress uses capability-profile similarity, not "all the GPT rows," to predict a GPT model (Finding 7).

A hybrid reliability estimator combines two signals — ensemble spread (how much plausible alternative predictors disagree on the cell) and matrix support (the coverage/neighbor/peer features above) — into a per-cell risk score. Keeping only the most-trusted 20% of predictions drops selective MedAE to 1.83 points (2.51 at 40%, 3.10 at 60%), beating either signal alone (Finding 8). The confidence layer is what makes "skip this eval" a defensible decision rather than a guess.

Scope: inferable, not unnecessary#

The paper is unusually careful to bound its own claim, and the bounds are the load-bearing nuance for the wiki:

• Score inferability ≠ benchmark redundancy. The result identifies benchmark scores that are currently inferable from others, not benchmarks whose existence is unnecessary. Benchmarks still do work BenchPress cannot: failure-mode discovery, contamination and distribution-shift monitoring, and incentive-shaping for model developers. A perfectly predictable benchmark can still be the one that catches the next regression.
• Snapshot dependence. Rank-2 is a property of this 84 × 133 May-2026 snapshot. A future release with a capability profile unlike anything in the matrix can break the geometry — the authors name tracking "does a third latent factor emerge?" as the signal to refresh the recipe.
• Public-score heterogeneity. The matrix mixes vendor-reported and third-party scores under varying harnesses, prompts, reasoning budgets, and dates; the same model can shift 1–3 points across runs and 5+ points across harnesses. Vendor scores (four-fifths of the matrix) may be optimistically biased, which could inflate the apparent cross-benchmark correlation the whole method rests on. BenchPress therefore predicts what this public matrix would extrapolate to, not what a controlled re-evaluation would yield, and its errors are an upper bound conflating prediction error with measurement noise.
• Probe-set specificity. The 5-benchmark sets are selected on the current matrix and must be re-derived as it grows; they are a snapshot recommendation, not a permanent eval suite.

The dataset (HuggingFace), BenchPress code (github.com/microsoft/benchpress), and an interactive per-cell predictor are released as a living, community-extensible artifact.

CollabEval: the same redundancy, spent on a confidence interval instead of a prediction#

Fisch, Deutsch, Maynez, Agarwal, Berant, Cohen, Globerson & Eisenstein ("CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion", Google DeepMind + Google Research, arXiv 2607.05046, 2026-07-06, empirical) take the premise one granularity down and change what it is for. BenchPress completes a models × benchmarks matrix to predict a missing score; CollabEval completes a models × prompts matrix to put an interval around one. The paper cites this page's source directly (alongside Ye et al. and Zhang et al. for the benchmark level, and Zhou et al. for the item level) as the low-rank evidence it builds on — so the two are explicitly the same literature at two scales.

The construction#

Partition the $M$ models into anchors (densely evaluated historical runs) and targets (new models, of which only a fraction $p$ of the $N$ prompts are scored). Complete the matrix, then — and this is the move — do not use the imputed values as scores. Use them as control variates inside cross-prediction-powered inference (CrossPPI; Zrnic & Candès 2024), which forms a rectified mean: the prediction mean plus an explicit bias-rectifier term measured on the labeled subset. Because the completion depends on the observed entries, the imputation is run $K$ times under cross-fold masks (default $K=10$), each holding out one fold of labeled prompts across all target models, so every observed cell is scored by a predictor that never saw it. The result estimates both individual model means $\theta_i$ and pairwise differences $\Delta_{ij}$, with plug-in covariance giving normal-quantile intervals.

Two properties come out of this that BenchPress does not have:

• Unbiasedness by construction. The rectifier cancels the completion's bias in expectation regardless of how bad the completion is. The paper spends a section on why this matters: the naive estimator that drops imputed values in as labels does reduce variance, but its MSE picks up a $\left(\frac{N}{N+n}\right)^2(\mathbb{E}[\hat Y]-\mathbb{E}[Y])^2$ bias term that no amount of data removes. Empirically the naive arm fails coverage while CollabEval holds it.
• Skipped prompts cost nearly zero. The PPI literature CollabEval extends (Boyeau et al.; Fisch et al. 2024) cuts labeling cost by substituting a cheap autorater — but still pays to generate the output being rated. Here a skipped prompt needs neither inference nor scoring: the target model is never run on it. On WMT24++ the authors price what that avoids — a full evaluation of one new model is ~2M tokens consumed and generated (55 languages × 998 paragraphs × ~35 tokens).

Validity does not depend on the matrix being low-rank#

This is the load-bearing difference from BenchPress and the reason the two pages' risks are not the same risk. CollabEval "maintains valid coverage even if the score matrix lacks low-rank structure and the attempted imputation is noisy." Low rank buys efficiency only. Combined with power tuning (PPI++; a learned scalar $\lambda_i$ on the control variate), Proposition A.9 guarantees asymptotic variance ≤ classical estimation, with the reduction scaling in the squared covariance between true and imputed scores — so an imputation uncorrelated with truth degrades gracefully back to the plain sample mean rather than to a confident wrong answer. The only requirement on the completion algorithm is a mild stability condition (Assumption A.1: cross-fold predictions converge to a fold-independent population limit).

Finding 9. Using matrix completion as a control variate rather than as a substitute score makes the estimator's correctness independent of whether the low-rank structure is real. Redundancy becomes a source of statistical power, not a source of assumptions.

Item-level matrices are not rank-2#

The strongest correction this source makes to the rest of the page. Rank-2 is a property of the benchmark-aggregate matrix; it does not carry down to items. Across all five of CollabEval's datasets, the first 16 singular vectors (or fewer) explain over 50% of observed-score variance — against BenchPress's top-two components explaining >90% at the benchmark level. Figure 3 (viewed) plots singular-value decay and cumulative explained variance for the top 32 components of each matrix; reading the curves, the small dense matrices concentrate fastest (SWE-bench, $M{=}12 \times N{=}500$, and AQA, $21 \times 3610$, are the only two to cross 90% inside 32 components) while the large ones do not come close — WMT ($825 \times 998$) plateaus near 0.78, AlpacaEval ($221 \times 790$) near 0.73, and MMLU ($108 \times 14042$) reaches only ~0.66 by component 32, starting from ~0.20 at the first. The rank ablation corroborates from the other direction: IterativeSVD's CI-size reduction rises monotonically with max rank and only plateaus around 16 (average across tasks: +7.6% at rank 1, +9.2% at 2, +12.5% at 16, +13.0% at 32).

So the two results are compatible but the headline does not transfer: aggregate benchmark scores are a rank-2 summary of a substantially higher-rank item-level reality. Averaging thousands of items into one benchmark number is itself most of the compression. This does not weaken BenchPress — it bounds what "the eval landscape is 2-dimensional" is a claim about.

What it buys#

All metrics over 10k bootstrap trials, with ground truth defined as the average over each dataset's full $N$ base examples.

• Coverage holds. CollabEval tracks the nominal 90% rate (Figure 2, viewed: all arms sit in an 89–90% band across the labeled-fraction sweep), including at $p<0.2$ where the guarantees are only asymptotic. Naive imputation does not.
• CI width. Over 20% average reduction for pointwise $\theta_i$ at small $p$ (Figure 2's averaged curve peaks ~22–23% with the cross-dataset band reaching ~31%); over 30% on AQA. For target-vs-target differences the prose claims over 30% average — the averaged curve reads closer to 28% at the lowest $p$ with the band extending past 30%, so read the headline as the top of the range rather than the mean. At a moderate $p=0.5$, ~10–20% remains.
• Effective sample fraction. The curves stay above the $y=x$ parity line throughout: where the classical estimator needs 60% of the labels for a given precision on $\theta$, CollabEval reaches it at roughly 45%.
• Gains concentrate where labels are scarce ($p \approx 0.1$–$0.2$) — enough data to learn the latent structure, not enough for the classical estimator to be precise — and vanish as $p \to 1$.

Ablations worth keeping#

• Paired sampling matters only for target-vs-target comparisons. Observing the same prompt subset for every target model is indistinguishable from IID for individual means, decisively better for differences between target models (the classical paired estimator is already competitive there, and CollabEval on top of it is the best arm), and irrelevant for target-vs-anchor differences, where every target observation is automatically paired against a dense anchor row.
• Anchor choice moves efficiency, never validity. Coverage is flat across anchor-set size and composition, exactly as the theory says. Efficiency grows with the anchor set; Random-$k$ beats Top-$k$ beats Bottom-$k$ — unbiased anchor selection wins, and cherry-picking the strongest historical models is worse than sampling them. Even tiny anchor sets still yield positive variance reduction.
• Matrix completion beats IRT by ~3× on the datasets where IRT applies. At a fixed 50% labeled fraction, average CI-size reduction: NN +13.5%, IterativeSVD +12.5%, nuclear norm +10.2%, 2PL IRT +3.8% (MMLU +1.7%, SWE-bench +5.9%; N/A elsewhere, since 2PL needs binary items). This is the empirical form of the paper's structural argument — a valid IRT model implies a very low-rank log-odds matrix but not conversely, so completion can capture model×item interactions a single per-model "ability" parameter cannot. Relevant because the psychometric line (the g-factor and tinyBenchmarks work this page's own history section cites) is the tradition CollabEval measures itself against and beats. The NN's larger average comes with higher across-task variance at low $p$; IterativeSVD is chosen for the speed/simplicity/stability tradeoff, and the validity result holds for any of them.
• Coresets lose on both counts. Anchor Points (select a representative prompt subset, evaluate only those) gives a point estimate with no valid interval, can be biased, and is empirically less stable across tasks.

Scope#

Five text-generation benchmarks: AlpacaEval 2.0 (17 recent leaderboard submissions as targets, the 204 earlier ones as anchors; GPT-4 Turbo autorater win-rate), MMLU (6 Open-LLM-Leaderboard targets — Amber-6.7B, OLMo 1/2-7B, Pythia-2.8B/6.9B, K2-65B — against 102 anchors), AQA (21 systems, leave-two-out, NLI-based AutoAIS attribution score), WMT24++ (15 systems × 55 languages flattened to 825 × 998, MetricX, comparisons restricted within-language), and SWE-bench (12 systems with public item-level scores under the mini-SWE-agent V2 harness). Nothing outside text; no multimodal, agentic-trajectory, or audio matrices tested.

One caveat the paper does not state, and it is the seam with LLM-Judge Validation: the estimand is the mean of whatever the rater produces. Three of the five datasets score items with an autorater or learned metric (GPT-4 Turbo win-rate, AutoAIS, MetricX). A statistically valid interval around an autorater's mean is a statement about sampling uncertainty, not about construct validity — CollabEval tightens the interval around the number the benchmark already computes, and inherits every bias in that number unchanged. The PPI ancestry it builds on (Fisch et al. 2024's stratified PPI, Boyeau et al.) exists precisely to relate cheap ratings to expensive human ones, so composing the two is natural, but this paper does not do it.

Connections#

• Compute-Controlled Benchmarking — the sibling eval-cost reduction on the other axis: that page compresses cost by naming the compute budget per benchmark; this page compresses by predicting across benchmarks so you run only a few. The tension the paper itself raises: BenchPress operates on exactly the uncontrolled public grid Brown/AISI critique — heterogeneous configs, vendor-optimistic scores — so its predictions inherit that grid's noise; combining "control compute per eval" with "predict across evals" is unaddressed. CollabEval sits on a third axis, the annotation budget within one benchmark (label $p$ of the prompts, skip the rest at near-zero cost), and is the one member of the trio that survives the uncontrolled-grid objection by construction: three of its five matrices are single-lab runs scored under one harness, so its correlations are not artifacts of vendor self-reporting
• Large-Scale Test-Time Compute — the hub whose open question ("can high-budget performance be predicted from cheap runs?") this answers a sibling of: cross-benchmark prediction works, which is strong evidence the eval landscape is low-dimensional, but it does not show the cross-compute-budget forecast works — that axis is still open
• Scale-Dependent Prompt Sensitivity — the item-level counterpart to this matrix-level result: that paper found ~27% of benchmark problems are non-discriminative (ceiling/floor); this finds the whole 133-benchmark matrix is rank-2. Both quantify how much less independent signal aggregate benchmarks carry than their count implies
• Task Time-Horizon Scaling — benchmark saturation is redundancy seen from this page's angle: a saturated benchmark has near-zero score spread across models, and low spread is exactly what makes a benchmark trivially predictable here (H3) — so saturation both flattens the horizon metric's usefulness and makes the score inferable
• UK AI Security Institute — the government evaluator pursuing the compute-axis version of this idea (forecast high-budget performance from cheap runs); BenchPress is the benchmark-axis counterpart — both are bets that expensive evaluation can be projected rather than paid in full
• Measuring Beyond Accuracy Saturation — the complementary answer to saturation. Saturation = near-zero score spread, which is exactly what makes a benchmark trivially predictable here (H3) — so this page's move is skip-and-predict (a low-spread benchmark's score is inferable, don't run it). Nadgir et al.'s move is the opposite: keep-and-re-instrument (extract efficiency, reliability, and model-vs-scaffold signal from the one saturated benchmark you keep). Opposite operational directions, same premise that a headline accuracy number under-uses the benchmark. CollabEval is a third move on the same premise — keep the benchmark, label less of it — and is the only one of the three that changes what you can claim rather than what you measure, since it returns a coverage guarantee instead of a number
• LLM-Judge Validation — the seam under CollabEval's guarantee, and the reason a valid interval is not a valid measurement. Three of its five matrices are filled with autorater or learned-metric scores (GPT-4 Turbo win-rate, AutoAIS, MetricX), and the estimand is the mean of whatever the rater produces — so the CI quantifies sampling uncertainty about a possibly-invalid quantity, and inherits kappa deflation, position bias and every other defect of the grader unchanged. Read together the two pages bound the eval-cost trade from both ends: this page says you can label far fewer items and still say something rigorous; that page says the rigor is about the number, not about the capability
• LLM-as-a-Judge — the psychometric line CollabEval measures itself against, running in the other direction. CalibratedRubric applies the same 2PL machinery to a systems × rubric-items matrix but uses it for selection rather than imputation — keep the items whose fitted difficulty sits inside the observed capability range, drop the rest — and reaches a target rank correlation with 49 items instead of 131. The two compressions compose rather than compete (select fewer items, then label fewer model×item cells), and both land on the same verdict about 2PL on small leaderboards: CollabEval measures it completing at +3.8% against IterativeSVD's +12.5%, while CalibratedRubric's own AIC and BIC select 1PL in five of six blocks and it reframes its regularized 2PL as an operational difficulty-targeting device rather than an identified model. It also supplies the generative reason an evaluation matrix is redundant, which BenchPress only observes: consensus-derived item sets are retained at ρ^M in the number of systems, so the grid is shaped by the panel that built it
• Benchmark Contamination and Decontamination — the correction-side counterpart to this page's own scope caveat. A key reason "scores are inferable, not benchmarks unnecessary" is that benchmarks still do work BenchPress can't — contamination monitoring among them; Sun et al. take a benchmark whose scores are already inflated by leakage and recover the clean per-sample distribution without a reference model. Their correction leans on the same ground-truth-probability signal contamination distorts, so it operates inside the integrity problem the uncontrolled public grid embodies
• How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them? — the cluster synthesis this page anchors: rank-2 redundancy is the matrix-level member of a three-granularity signal-shrinkage result, and this page's "inferable ≠ unnecessary" caveat supplies the keystone of what benchmarks alone still do

Open Questions#

• Does the rank stay 2? The geometry is snapshot-specific; whether a third latent factor emerges as the matrix grows (novel capability profiles, new benchmark families) is the paper's own named signal for when the recipe needs refreshing — and an open empirical watch item. Partially answered (2026-08-04) — on the stakes rather than the fact, by CollabEval. Nobody has re-measured the 84×133 matrix, so whether its rank holds is still open. But CollabEval shows the consequence of the geometry breaking is entirely a function of how you spend it: used as a prediction (BenchPress) a broken rank silently corrupts the answer; used as a control variate inside PPI, correctness is independent of the rank and only efficiency degrades, gracefully, back to the classical sample mean. The question stays live for BenchPress's use case and is largely defused for the inferential one. Two further data points against the geometry being permanent: at the item level rank 2 is nowhere near enough (>16 components for >50% of variance, and IterativeSVD keeps improving through rank 32), and the redundancy is weakest exactly on the largest matrix (MMLU, ~0.66 cumulative EVR at 32 components).
• Can outlier models be anchored without any scores? BenchPress fails on a model whose capability profile has no close neighbor in the matrix; the authors propose folding in external metadata (training-data composition, architecture, size) to compute model-to-model similarity before any benchmark is run, but do not build it. Partially answered (2026-08-04) by CollabEval — not by anchoring the outlier, but by bounding what its failure costs. Its anchor ablation finds CI coverage flat across anchor-set size and selection strategy, with positive variance reduction even at the smallest sets, and the control-variate construction means a target model uncorrelated with every anchor yields a wider interval, not a wrong one. It also inverts the intuitive anchor policy: Random-$k$ > Top-$k$ > Bottom-$k$, so curating anchors toward the strongest models is actively worse than sampling them. The metadata-similarity idea remains unbuilt.
• Does the low-rank treatment carry beyond text/vision? Audio/speech, robotics/embodied agents, and scientific-simulator ecosystems are untested; whether the same rank-2 structure holds there is open. (CollabEval does not touch this — all five of its datasets are text generation.)
• Does vendor optimism manufacture the correlation? Four-fifths of the scores are provider-reported and possibly inflated; the paper flags this could inflate apparent cross-benchmark correlation but cannot separate it — would a fully standardized re-evaluation still be rank-2, or is some of the redundancy an artifact of shared reporting bias? Partially answered (2026-08-04) by CollabEval, at the item level. Its AQA (21 systems from one paper, one AutoAIS scorer) and WMT24++ (15 systems, all scored with MetricX under one protocol) matrices are single-lab runs under a uniform harness with no vendor self-reporting anywhere — and they are the strongest arms in the whole study (+17.3% and +12.9% CI-size reduction at 50% labeled). So exploitable cross-model correlation clearly survives standardized re-evaluation; it is not an artifact of shared reporting bias. What this does not settle is BenchPress's actual question, since these are item-level matrices within one benchmark rather than a standardized redo of the 84×133 cross-benchmark grid — the correlation surviving at one granularity is evidence, not proof, at the other.
• Would a public probe set become a Goodhart target? If "run these 5 benchmarks and infer the rest" becomes practice, the compact probe set is a small, public, high-leverage surface to optimize against — the same eval-report Goodhart pressure Compute-Controlled Benchmarking names, now concentrated on five benchmarks. Unexamined here.
• Is a valid interval around an autorater's mean worth anything? CollabEval's guarantee is about sampling uncertainty in the mean of the score a benchmark already computes — and on three of its five datasets that score is an autorater or learned metric (GPT-4 Turbo win-rate, AutoAIS, MetricX), which LLM-Judge Validation shows can be systematically wrong in ways no amount of tighter sampling detects. Directly falsifiable: run CollabEval on a task with both human labels and autorater scores, take human labels as $Y$ and the autorater as an anchor row, and check whether the reported CI actually covers the human parameter — or only the autorater's.

Sources#

• You Don't Need to Run Every Eval — Yuchen Zeng & Dimitris Papailiopoulos, You Don't Need to Run Every Eval (Microsoft Research AI Frontiers, arXiv 2606.24020, 2026-06-22, empirical): the 84×133 rank-2 score matrix (two diagnostics — Soft-Impute rank sweep + submatrix SVD >90% top-2 variance), the Logit Bias ALS BenchPress predictor (4.6 MedAE at 100% coverage), 5-probe scorecard recovery (3.93 / 4.55 low-cost), 92.1% ranking preservation at ≥5pt margin, new-model prediction (4.83 at 5 seeds under a temporal cutoff), the LLM-predictor comparison, the reliability/trust layer, and the four scope caveats (inferability≠redundancy, snapshot dependence, public-score heterogeneity, probe-set specificity)
• CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion — Adam Fisch, Daniel Deutsch, Joshua Maynez, Alekh Agarwal, Jonathan Berant, William Cohen, Amir Globerson & Jacob Eisenstein (Google DeepMind + Google Research, arXiv 2607.05046, 2026-07-06, empirical): §3 the PPI/CrossPPI background and the naive-estimator bias argument; §4.3 cross-fold matrix completion with IterativeSVD; §4.4 + Appendix A the covariance estimator, Theorem A.5 coverage, and power tuning (Proposition A.9's variance-never-worse-than-classical guarantee); §5 the five datasets and four metrics over 10k bootstrap trials; §5.5 coverage at nominal 90%, >20% pointwise CI reduction at small $p$ (>30% on AQA), the 60%→45% effective-sample result, paired-vs-IID sampling, and the anchor-set ablation (Random-$k$ > Top-$k$ > Bottom-$k$); Appendix C.1 footnote 3 + Figure 3's effective-rank analysis (≤16 singular vectors for >50% variance); Appendix D Tables 1–3 (rank, fold, and completion-algorithm ablations, including 2PL IRT at +3.8% against IterativeSVD's +12.5%). Figures 2 and 3 viewed per the two-pass rule. Parse note: ingest's table-collapse check flagged 24 cells, all of them Table 2's, and all false positives — that table's cells legitimately hold a coverage, CI-reduction pair per its own caption; all three appendix tables were reconciled cell-for-cell against pdftotext -f 27-28 -layout and are exact (see wiki/sources.md)
• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation — Chen et al. (FinStep + StepFun, arXiv 2607.29252, 2026-07-31, empirical): IRT used for item selection rather than imputation (§3.4–3.5, §4.3 — 49 vs 131 rubrics at target rank correlation; AIC/BIC select 1PL in five of six blocks) and Prop. 1's ρ^M attrition as a generative account of why an evaluation grid is shaped by the panel that built it — see LLM-as-a-Judge for the full treatment