How Much Signal Do Public Benchmarks Still Carry — and What Replaces Them?

Synthesis of the 2026 eval-science cluster: public benchmark suites carry far less independent signal than their count implies (133 benchmarks ≈ rank-2; accuracy saturates even after validity fixes) and the headline number is corrupted through four distinct channels (unnamed compute budget, contamination, vendor optimism, unvalidated judges) — but ordinal comparisons survive under verified invariances, and nothing replaces benchmarks wholesale: the field's answer is a five-part portfolio (predict-don't-run, re-instrument saturated suites, compute-controlled curves, production-sourced refresh, judge validation), with failure-mode discovery, contamination monitoring, and incentive shaping as the jobs only benchmarks still do