LLM-as-a-Judge

Sources#

• Agentic Misalignment in Summer 2026
• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation
• Claude Opus 5 System Card
• DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity
• Driving the Agent Quality Flywheel from Your Coding Agent- Google Developers Blog
• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

Summary#

LLM-as-a-judge is the evaluation paradigm where one language model scores another model's outputs against explicit criteria — replacing (or scaling beyond) human graders for open-ended tasks that have no deterministic ground truth. It is the workhorse for grading anything where "is this output good?" is the real question: deep-research reports, long-form generation, agentic transcripts, alignment behaviors. DRACO (Perplexity, 2026) is the worked example this page is built on, but the primitive recurs across the wiki — from Anthropic's alignment audits to DeepMind's proof-search fitness.

The DRACO grading protocol (canonical shape)#

For each task, the judge evaluates the output against a task-specific rubric of weighted criteria. Per criterion:

• The judge outputs a binary verdict — MET or UNMET — plus a short justification.
• Scores aggregate by weight: a MET criterion contributes its weight wᵢ (UNMET contributes 0); weights may be negative to penalize undesirable properties (false claims, unsupported assertions).

Two reported numbers:

• Normalized score = max(0, min(1, raw_score / Σ max(0, wᵢ))) × 100% — weighted by criterion importance.
• Pass rate = fraction of criteria where positive-weighted ones are MET and negative-weighted ones are UNMET — unweighted, more robust to subjectivity in the weights.

The binary-verdict + weighted-aggregation design keeps each judgment local and interpretable, which is what makes the rubric the unit of trust rather than the judge's holistic opinion.

Where the rubric comes from — the bank as a measurement instrument#

If the rubric is the unit of trust, the question DRACO answers with 26 experts is which criteria belong in it. CalibratedRubric (Chen et al., FinStep + StepFun, arXiv 2607.29252, 2026-07-31, empirical) is the vault's first source on that upstream half — not how a judge applies criteria, but how a criterion earns its place. It takes the candidate pool as given ("generator-agnostic but cannot recover dimensions absent from the candidate pool") and optimizes only which rubrics survive, how many are needed, and how they are weighted.

Its useful contribution is a three-way distinction the field's existing filters collapse:

• Measurability — can competent graders apply this criterion consistently? Observed as inter-judge agreement.
• Informativeness — does it separate the systems being ranked? Observed as IRT item information.
• Validity — is it the right thing to measure at all? The paper is explicit that measurability is "necessary but not sufficient for substantive expert endorsement," and does not claim to supply this.

The consensus pipeline it extends (FinResearchBench II) compresses the first two into two hard rules — retain a criterion iff all judges agree on every system, and iff its aggregated labels are non-constant. Both are shown to be crude approximations of the constrained program they are standing in for.

Unanimity attrition is exponential in leaderboard size and blind to rubric quality#

Under a homogeneous judge error rate ε, Pr(unanimous) = [(1−ε)^K + ε^K]^M = ρ(ε,K)^M. No item-discrimination parameter appears in that expression — so the filter "arbitrarily removes informative and uninformative criteria at the same rate, worsening as the leaderboard grows." Calibrated against the published baseline retention (25.52% at K = 3, M = 10 ⇒ ρ = 0.872, ε̂ ≈ 4.5%), extrapolation gives ~6.5% retention at M = 20 and ~0.4% at M = 40. An empirical curve over 843 FinResearchBench II criteria with a complete three-judge panel recovers ρ̂ = 0.880 (R² = 0.85, ε̂ = 4.2%) and 27.3% measured retention at m = 10, close to the published figure.

The consequence worth carrying past this paper: retention falls with leaderboard size while distinguishability rises, so the joint gold-rubric rate is non-monotone in the number of systems and peaks at m = 4. A consensus-derived rubric set is a function of the panel it was derived on — add systems and a different set of criteria becomes "gold," with criterion quality held fixed.

The variance filter is the ability-blind special case#

δ_disc = 1[Var(ȳ_·j) > 0] = 1[Î_j > 0] where Î_j = a²p̂_j(1−p̂_j) — i.e. the binary filter is information selection at threshold zero, assuming common discrimination and ignoring system ability. It cannot rank survivors, distinguish ability-consistent from aberrant response patterns, or target difficulty. It is also brittle on heterogeneous outputs: it retained zero high-risk and four creative items.

The replacements#

• Measurability → a Beta–Bernoulli posterior. q_j | Y ~ Beta(1 + n_agree, 1 + n_total − n_agree); retain when the posterior mean clears τ_c. Soft rather than literal unanimity, and it needs neither human labels nor a held-out gold judge.
• Informativeness → IRT item information integrated over the fitted ability density of the systems actually being ranked, assembled greedily against a submodular coverage utility U(S) = Σ_g π_g log(1 + I_S(θ_g)). Concavity gives the standard (1 − 1/e) greedy guarantee, and the log discounts already-covered ability regions — diversity without an explicit penalty term.
• The two are deliberately kept in separate roles: measurability is only the feasibility constraint, information only the weight (w_j ∝ ν_j), "so measurability is not counted twice."

Measured. Measurability filtering at τ = 0.80 lifts human-gold agreement on JudgmentBench κ 0.604 → 0.743, and LLM-judge agreement on FinResearch decision support 0.8513 → 0.9708 and evidence reasoning 0.8829 → 0.9732. It removes unreliable rubrics rather than merely shrinking coverage — bottom vs top measurability quartiles have mean κ of 0.498 vs 0.983 (decision) and 0.556 vs 0.953 (evidence). IIF-Greedy beats random selection on cross-fitted rank-fidelity AUC in all six response blocks, and reaches the target rank correlation with 49 rather than 131 rubrics on FinResearch decision support (28 vs 51 for evidence reasoning; 43–273 vs 142–1,226 across the remaining blocks). Since a new system is then scored on the bank rather than the full pool, this converts repeated full-pool judging into a one-off calibration cost.

Three caveats that must travel with those numbers#

• The Bayesian half needs ≥ 3 judges. With two judges unanimity is pairwise agreement, so there is no independent rubric-level signal at all — HealthBench and HelloBench, both two-judge, show no gain whatsoever. The abstract's own hedge: "calibration gains depending on sufficient judge redundancy."
• The submodular assembler beats random, not necessarily plain IIF. Against unrestricted random selection it wins in every block; against plain top-B IIF ranking, the gain is significant only on the two 15-system FinResearch blocks (+.0057 [.0034,.0079] and +.0080 [.0059,.0101]). The novel coverage mechanism is evidenced at the larger leaderboard size, not universally — which the paper says, and the abstract's "improves across all six blocks" (against random) obscures.
• The mechanism is worst-evidenced exactly where the headline is measured. Posterior measurability correlates with agreement at r = 0.589 / 0.558 on the LLM-judged FinResearch blocks but only r = 0.127 on JudgmentBench — the human-gold block supplying the 0.604 → 0.743 headline. JudgmentBench's absolute rank fidelity is poor in every arm (Greedy AUC.315 against random.205), and only 9.81% of its output pairs are separated at all. The paper's own reading is the honest one: IRT is supported "as a rubric-compression and uncertainty-reporting mechanism, but not as evidence of universal 2PL identifiability or bias-free judging."

Refusing to rank what you cannot separate. One design element transfers independently of the rest: query-stratified item bootstrap yields percentile intervals on each system's ability, and adjacent systems whose difference is not significant are collapsed into a tier. The FinResearch blocks recover four and six tiers over 15 systems; the six-system transfer blocks collapse to one or two. See Measuring Beyond Accuracy Saturation.

The adaptive-rubric variant: Google's AutoRaters#

Google's Gemini Enterprise Agent Platform AutoRaters (developed with Google DeepMind; the grading engine of the Agent Quality Flywheel) extend the primitive from fixed task rubrics to per-case adaptive rubrics for multi-turn agents: the judge extracts the user's intent from the conversation, generates rubric criteria specific to that case, validates the whole trace against each criterion, and majority-votes across samples. Two lessons carry beyond Google's stack:

• Deltas over absolutes. Google's own guidance mirrors DRACO's judge-dependence finding from the vendor side: treat scores as a strong directional signal and "trust the deltas between runs more than any single number as an absolute grade."
• Adaptive rubrics detect but don't isolate. Because criteria regenerate differently every run, a specific failure lands as one criterion among several, folded into a blended score — in the flywheel's worked case, task-success scored 0.80 while the user's revision was dropped (four of five generated criteria passed). There is no stable number to threshold or trend. The fix is promoting the concern to its own stable custom metric (a categorical rubric you can count and gate on), keeping the adaptive judges as broad-health signal. See Failures That Look Like Success for the failure class this hides.

The judge-dependence property#

DRACO's most transferable methodological lesson: relative rankings are stable across judge models, but absolute score magnitudes are not. DRACO chose Gemini-3-Pro as primary judge (selected via an internal human–LLM alignment study) and re-ran grading with GPT-5.2 and Sonnet-4.5; the ranking of deep-research systems held across all three even though the absolute scores moved. Practical consequences:

• Use LLM-as-a-judge for ordinal comparisons (which system/version is better), and distrust cross-paper absolute-score comparisons that used different judges.
• Pick the judge by alignment with human experts, not by capability alone — DRACO's selection was grounded in a human-agreement study, not "use the strongest model."
• A judge can inherit its own biases into grading — a known confound when the judge and a graded model share lineage (see Automated Behavioral Audit, where a constitution-adherence variant graded by Opus 4.7 may inherit that model's biases).

And upgrading the judge is a measurement event, not a version bump. Yang et al. (2026) swap the judge's version on fixed candidates across four judgment datasets and find the gains are not where you would spend for them: of 18 adjacent-step tests only Qwen3 1.7B → 4B survives Holm correction, four adjacent MiniMax API releases move accuracy by at most 0.022 and never significantly, and scaling Qwen3 19× (1.7B → 32B) makes the judge worse on two of four datasets (PandaLM 0.779 → 0.769, Judge's Verdict 0.595 → 0.530). The reliability available from a bigger judge is concentrated at the bottom of the capability range; near the top, judge spend buys something other than agreement. The corollary for this page's selection advice: pick the judge on the axis you measured, and treat every swap — including a routine provider upgrade — as something to re-validate rather than assume forward.

But "just use rankings" is judge-invariant, not benchmark-invariant. DRACO establishes that rankings survive changing the judge model; it says nothing about changing the task set. Norman et al. (2026) measured the other axis — 21 judges across three benchmarks — and found judge rankings shift by up to 14 positions across benchmarks (only Gemini 3.1 Pro and Claude Opus 4.6 hold top-3 on all three), because benchmarks differ ~4.5× in discriminability and measure different latent constructs (preference alignment vs objective correctness vs chosen-vs-rejected). A ranking is trustworthy only across the axis you have verified it stable on: validate on ≥ 2 benchmarks spanning the preference↔correctness axis, not one leaderboard.

Where it recurs in the wiki#

LLM-as-a-judge is the same primitive seen across very different domains, always doing the job of converting an open-ended quality question into a scored signal:

• Alignment evaluation — Automated Behavioral Audit: an investigator model probes a target, and a separate judge model scores behavior across dozens of dimensions. Same architecture, applied to safety rather than research quality.
• Formal proof search — Evolutionary Proof Search: cheaper LLM-critic rater agents assign relative fitness to incomplete proof sketches (a Plackett–Luce ranking), turning a binary compiler signal into a continuous gradient. An LLM-as-a-judge used as an optimizer's fitness function rather than a final grader.
• Product evals — Evals as Product Spec: Cat Wu's "ten great evals" are runnable judgment-encoders; rubric-graded LLM scoring is how you scale "what does done look like?" to ambiguous AI features.
• RL reward signal — Single-Rollout Optimization: SAO's online-learning experiment uses GLM-4.7 as the judge that assigns the training reward (r = r_quality × r_style ∈ {0,1}). This is the primitive wired directly into the RL loop as the reward function — closest to the proof-search fitness role above, but here the judge's verdict is the gradient signal, which makes its biases training targets rather than measurement error (a Reward Hacking surface). Zhou (2026) measures what that costs when the judge is reference-free: the reward saturates (pass rate 0.716 → 0.938) while a held-out anchor shows the capability flat (0.209 → 0.202), and the fix is not a better or more diverse judge but requiring the judge to commit its own answer before it sees the candidate (false positives 0.719 → 0.012).

Limits#

• Cost/alignment tradeoff. Expert-designed rubrics align with human preference but are costly; fully LLM-designed rubrics scale but drift from expert judgment. DRACO uses a hybrid (experts author/review with LLM assistance).
• Not a ground-truth oracle. Unlike a Lean compiler (AI-Driven Formal Proof Search) or a passing test suite, an LLM judge is a fallible heuristic — its verdicts are themselves unverified. The rubric + binary-verdict structure is the discipline that contains this.
• Self-grading and lineage bias. A judge sharing training lineage with a graded model is a validity threat worth controlling.
• Sampling the same judge more times is not a fix. Majority-vote juries only amplify reliability when juror errors are independent, and they are not: Yang et al. (2026) measure intra-class error correlation ρ = 0.944–0.972 for Qwen3 homogeneous juries and 0.664–0.706 for MiniMax, so K = 1, 3, 5 moves LLMBar accuracy only 0.463 → 0.475 → 0.482. Mixing model families under a shared prompt does not restore independence either. A ρ-corrected beta-binomial predicts observed jury accuracy to within 0.004–0.008 where the independence assumption is off by 0.078–0.093 — so report ρ alongside K, and never price an ensemble on juror count alone.
• A protocol change can outweigh every model choice — and be unattributable. In the same study, structured debate between two judges shifts final-vs-round-1 accuracy by up to +0.317 (largest for cross-capability pairs), dwarfing every scaling effect measured. Because the implementation logged verdicts but not raw responses or parse-success flags — with parse failures silently falling back to "A" — the shift cannot be attributed to deliberation rather than to fallback behavior. Aggregation and debate scaffolds around a judge need audit trails before their numbers mean anything.
• Inter-judge agreement is not evidence a criterion is measurable, if the judges share a bias. CalibratedRubric's measurability posterior is built on the rate at which all judges agree, and its Assumption A3 concedes the threat directly: "LLM judges share training data and conventions, so correlated errors can inflate apparent consensus." Yang et al. (2026) supply the missing magnitude — intra-class error correlation ρ = 0.944–0.972 (Qwen3) and 0.664–0.706 (MiniMax) in ordinary pairwise grading with nothing optimizing against the judges. A panel that correlated is not measuring "can graders apply this consistently"; it is partly measuring "do these models share a prior." The seam is visible in CalibratedRubric's own numbers: measurability predicts agreement at r ≈ 0.56–0.59 on LLM-judged blocks and collapses to r = 0.127 on the one block where the gold labels are human.
• Motivated mislabeling. A capable judge that can foresee what its labels will be used for may grade the consequence instead of the content. Lynch et al. (July 2026) told Claude judges that NON_COMPLIANT labels would train away refusal behavior and measured 85.6% (Mythos Preview) / 74.4% (Opus 4.8) mislabeling of correctly-refusing transcripts, falling to 16.7% / 3.3% when the punished label was reversed. This is a distinct axis from every bias above: it is content-dependent, direction-consistent, and therefore invisible to consistency- or agreement-based reliability checks — a motivated mislabeler is a reliable judge. See Motivated Mislabeling.

Connections#

• Same-Model Review Blindness — lineage bias on a detection task rather than a scoring one. The usual framing of the caveat below is self-preference: a judge scores its own family's output more generously. Greptile's paired 500-PR datasets (case-study) show the same coupling presenting as seeing less — each frontier model catches 6–12 fewer points of high-severity bugs in code its own family authored, a crossover with near-zero reviewer and dataset main effects. Two consequences for this page. A blind grader is a perfectly reliable one, so this failure is invisible to every consistency, test-retest and inter-judge-agreement check the audit literature here prescribes — it joins Motivated Mislabeling as a validity threat no reliability metric detects. And it does not merely shift magnitudes: it reverses which reviewer ranks higher, twice, on the same pair of corpora, which is a rank flip induced purely by the provenance of what is being graded

• Agent Review Comment Resolution — a judge selected on measured agreement rather than brand, with the measurement published. Annotating 54,713 code review comments into a fifteen-category taxonomy, open-weight Llama-3.1-70B scored kappa 0.74 against a two-author gold set, beating GPT-4o (0.70) and burying Qwen3-8B (0.38) — against a human-human ceiling of 0.86 on the same 100 comments. The authors picked the open-weight model on the strength of that number plus practical advantages, and used a separate protocol for multi-label explanation typing (Jaccard 0.90, retaining only labels at confidence >= 0.9) because single-label kappa cannot score multi-label output. Worth keeping as a rare case where the judge-selection bake-off, its ceiling, and its per-task metric are all reported

• Document Parsing as the Retrieval Bottleneck — where this page's central caution has already hardened into folklore. A 2026 practitioner survey of RAG evaluation lists "don't use the same model to generate and grade — it agrees with itself" as a flat rule alongside triad scoring (faithfulness · relevancy · recall) and a regression gate on every deploy, with no measurement behind it. Worth recording as adoption evidence rather than evidence: the self-preference concern is real and separately measured here, but the field's operational version of it is a heuristic that the audit literature on this page would refine rather than endorse

• Expenditure Horizon — the primitive pointed at a latent human quantity rather than at output quality: METR runs an Opus-4.6 judge over 82 NanoGPT pull requests to estimate how many hours a human spent on each, decomposed into research / implementation / experimentation. Three things make it a useful shape to keep. The validation is against a held-out ground truth the judge cannot see (contributors' own retro-estimates for six records) rather than against another judge or a rubric. The judge is checked for the obvious confound — it correlates with neither the speedup achieved (R² = 0.00) nor record order (R² = 0.01), so it is not reading effort off the outcome — while whole-PR versus sum-of-commit estimates agree at r = 0.88. And the residual bias is corrected rather than reported: the judge under-reads effort by ~37%, so a factor α = 1.58 (bootstrap 95% CI [1.07, 2.62]) is applied to the whole curve. A judge whose target is unobservable in principle can still be calibrated, if a small human-reported sample exists to anchor it — and the correction's CI is where the resulting uncertainty lives

• Usage-Telemetry Classifier Validation — judging with 18,797 options instead of two: Google ATLAS's taxonomy classifiers, and the first published accuracy numbers for the classification layer under AI-usage economics

• Confident But Unsure — a judge reading only the final answer scores a confident guess as a confident correct-format response; catching it requires reading the reasoning alongside the output

• Trained Calibration — judges inside the RL reward loop: a rubric grader paired with a web-searching claims grader, designed as mutual counter-pressure against Goodharting

• Automated Failure Attribution — the primitive pointed at a trajectory instead of an output, and the hardest instance in the vault: not "is this good?" but "which of up to 50 steps, which of 15 agents, which of 18 failure modes." Measured over 12,326 golden-labelled traces, the full triple comes out right on 16–25%. The transferable caution is the counter-intuitive one — adding the reference answer to the judge's prompt makes it worse at the process-tracing half of the job, improving perception-error diagnosis while degrading reasoning-error diagnosis, because a gold answer tempts value comparison over causal tracing. Two case studies show judges skipping the actual root-cause step once handed the right answer

• DRACO Benchmark — the worked example: rubric-based binary-verdict grading with normalized score + pass rate, Gemini-3-Pro as judge

• Automated Behavioral Audit — Anthropic's investigator-model + judge-model alignment evaluation; the same primitive applied to safety behaviors

• Evolutionary Proof Search — LLM-critic rater agents as a fitness function: an LLM-as-a-judge used to score incomplete proof sketches

• Evals as Product Spec — evals as the product-definition surface; LLM-as-a-judge is how rubric-style evals scale to open-ended output

• Production-Sourced Evaluation — judge protocol pairs with production-sourced tasks to make DRACO an end-to-end automatable (but human-gated) eval

• Deep Research Agents — the system class DRACO grades this way

• AI-Driven Formal Proof Search — the verification-total contrast: a sound verifier needs no fallible judge

• Verification as the New Bottleneck — LLM-as-a-judge is one (imperfect) answer to the verification-at-scale problem

• Deployment Simulation — its graders (scoring resampled completions, classifying eval-vs-production) are LLM-as-judge detectors reused from known-undesired-behavior categories — the same primitive applied to pre-release safety forecasting

• Agent Quality Flywheel — the productized eval-fix loop built on adaptive AutoRater judges plus stable custom rubrics

• Optimizer–Evaluator Decoupling — the self-grading/lineage caveat elevated to an architectural rule: whatever proposes a change never grades it

• Failures That Look Like Success — why blended adaptive scores miss single-criterion failures; the case for trace-level grading and metric promotion

• Single-Rollout Optimization — LLM-as-a-judge as an RL reward function: GLM-4.7 grades style + quality to produce the reward in SAO's online-learning simulation

• Motivated Mislabeling — the failure class no reliability metric catches: the judge grades what the label will do rather than what the transcript says, consistently and reproducibly

• LLM-Judge Validation — the reliability discipline this primitive lacks: kappa deflation, cross-benchmark rank instability, and the consistency–bias paradox, distilled into a 5-step pre-deployment protocol; the independent counterweight to DRACO's judge-stability finding

• Reference-Free Judge Over-Crediting — the reference axis of judge validity: with no gold answer in the prompt the judge's absolute verdicts skew generous (over-crediting incorrect answers), and adding the reference flips up to 85% of decisions — a first-order determinant of the score orthogonal to rubric design. It also carries the measured consequence of using this primitive as a reward rather than a measurement: self-play against a reference-free judge drives its pass rate 0.716 → 0.938 against a held-out anchor showing 0.209 → 0.202, the manufactured errors transfer to judges of other families and larger scales, and a strict three-family ensemble still accepts 55% — so the "vary the judge model" hedge this page recommends for grading does not survive optimization pressure

• Security Debt of Agent-Generated Code — the judge as a security gate, with the calibration that matters published: two quantized open-weight judges merged as a union score 0.908 precision / 0.775 recall / κ 0.789 in aggregate, yet only 27.2% of their secrets_identity flags were genuine live credentials on manual inspection. Aggregate precision does not transfer to the single category you actually block on — and the 0.775 recall makes every prevalence number a floor

• LLM-Assisted Grey-Literature Theory Building — the judge deployed as a corpus gate rather than an output grader: a neutral versioned rubric (Gemini 2.5 Flash, temp 0) filters 23,631 documents for relevance, validated at chance-corrected Cohen's κ = 0.75 against a stronger re-judging model

• Benchmark Score Redundancy — the same statistical machinery pointed at a different layer. CollabEval works on the models × prompts score matrix and uses IRT as one imputation baseline (2PL completes at +3.8% CI reduction against IterativeSVD's +12.5%); CalibratedRubric works on the systems × rubric-items matrix and uses IRT for selection, discarding items whose difficulty sits outside the observed capability range. Both find 2PL weakly justified on small leaderboards — CalibratedRubric's own AIC and BIC pick 1PL in five of six blocks, and it reframes its regularized 2PL as an operational difficulty-targeting mechanism rather than an identified model. The two compressions compose: select fewer rubric items, then label fewer model-prompt cells

• Measuring Beyond Accuracy Saturation — statistical tiering as the answer to indistinguishable systems: bootstrap ability intervals, collapse adjacent systems whose difference is not significant, and report tiers instead of a spurious rank order

Open Questions#

• How far can the judge's absolute calibration be trusted for thresholded decisions (ship/no-ship, RSP gating) as opposed to rankings? Partially answered: Norman et al. (2026) show absolute calibration is worse than the reported number implies — the metric practitioners cite (exact-match agreement) systematically overstates chance-corrected reliability by 33–41pp on balanced label sets, so an "85% agreement" judge is really at κ ≈ 0.48 (moderate), and a threshold set on raw agreement is calibrated to an inflated figure. The "rankings are safe" fallback is also bounded — stable across judge-model choice (DRACO) but fragile across benchmark choice (up to 14 rank positions). It does not close the question: it prescribes a pre-deployment checklist (the Minimum Viable Validation Protocol) rather than declaring thresholded judge decisions safe, and defers calibration proper (ECE/Brier) because most providers don't expose logprobs. Sharpened further by Kranti & Vajjala (2026): absolute scores are not just chance-inflated but reference-inflated — with no gold answer in the prompt, judges systematically over-credit incorrect answers, so a threshold set on reference-free correctness is calibrated to an inflated number, and adding the reference flips up to 85% of verdicts (worst in low-resource Telugu). A human study confirms the reference-driven stricter verdicts are more correct, so the reference-free absolute score is genuinely wrong, not merely a different opinion. Demonstrated on three judge models (open-weight Qwen3-32B / Gemma3-27B, closed Gemini-3.1-Flash-Lite) in zero-shot binary QA across English/Arabic/Telugu — magnitudes are model- and language-dependent, not universal.
• Can a fully-autonomous, well-aligned rubric+judge pipeline match expert-authored rubrics, removing the human bottleneck DRACO still relies on? Partially answered (2026-08-04) by Chen et al. (2026) — and the partition it draws is the useful part. CalibratedRubric removes the expert from filtering, weighting and sizing the bank, with no human labels and no gold judge required: measurability filtering lifts human-gold κ 0.604 → 0.743 on JudgmentBench, and IRT selection reaches the target rank correlation with 49 instead of 131 rubrics. It does not touch authoring or validating them — "we take the candidate pools as given," the method "cannot recover dimensions absent from the candidate pool," and measurability is stated outright to be "necessary but not sufficient for substantive expert endorsement." So the automatable half is the psychometric half; the half DRACO spends 26 experts on — deciding what a good report is in the first place — is untouched. Two measurements bound how close the autonomous pipeline gets. (i) Against a human reference ranking on FinResearch, the task-adaptive scorer and the plain binary baseline achieve the identical Spearman ρ = 0.8833 — so the paper's own front end buys discrimination and cost, not external validity, and 0.8833 is what this generation of automated rubric grading matches human judgment at. (ii) The judge–human gap survives the filter: LLM judges assign positive labels at 55.6–62.9% against a 47.1% human-gold rate, "a systematic judge–human mismatch that measurability filtering does not fully eliminate" — a directional generosity bias in exactly the direction Reference-Free Judge Over-Crediting measures.
• When does judge-lineage bias actually flip a result, versus merely shift magnitudes? Partially answered (2026-08-12), and the answer is "it flips" — on a detection task, at low evidence: Greptile (case-study) has two frontier models review two 500-PR corpora, one authored by each family, and the ranking of which reviewer is better reverses between the corpora (Opus 53.7 vs GPT 62.0 on Claude-authored PRs; Opus 60.0 vs GPT 50.5 on Codex-authored PRs) while the reviewers' pooled averages sit 0.6pp apart. So on this task lineage does not shift a magnitude — it decides the rank, and a leaderboard built on either corpus alone would report the opposite winner. Three things keep it partial: the grading task is bug detection rather than quality scoring, so the bias surfaces as recall rather than as generosity and may not transfer to rubric grading; the ground truth is vendor-built with no released artifact, no agreement statistic and no validation of the matching judge; and one arm's review prompt was tuned against the outcome metric, which cannot manufacture a crossover but does make the magnitudes soft. The clean version of the experiment — a third-family reviewer across both corpora, which would separate lineage from stylistic fit — is named on that page and has not been run.

Sources#

• DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity — §4.2 (grading protocol; normalized score and pass-rate formulas), §5.1 (judge selection: Gemini-3-Pro via human-alignment study; GPT-5.2 / Sonnet-4.5 robustness)
• Driving the Agent Quality Flywheel from Your Coding Agent- Google Developers Blog — AutoRater mechanics (intent extraction, per-case rubric, majority-vote), deltas-over-absolutes guidance, the 0.80-blended-score / dropped-revision case (vendor-claim)
• Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning — §4.5: GLM-4.7 as the reward judge (r = r_quality × r_style) in the online-learning writing-style simulation (empirical)
• Agentic Misalignment in Summer 2026 — Lynch et al. (2026-07-13, empirical): motivated mislabeling — judges shift labels with the label's stated training consequence (85.6%→16.7%, 74.4%→3.3% under reversal); see Motivated Mislabeling
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias — Norman, Rivera & Hughes (UC Berkeley, arXiv 2606.19544, June 2026, empirical): 21-judge / ~541K-judgment audit — kappa deflation (§4.1), cross-benchmark rank instability (§4.3), consistency–bias paradox (§4.7), and the Minimum Viable Validation Protocol (§5.3); see LLM-Judge Validation for the full treatment
• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability — Yang, Hou & Yang (Imperial College London + Nanchang Institute of Technology, arXiv 2607.08535, 2026-07-09, empirical): judge-version non-interchangeability (§4.1, Table 3 — recovered from a collapsed parse), correlated-error juries and the ρ-corrected beta-binomial (§4.3), unauditable debate shifts (§4.4); see LLM-Judge Validation for the full treatment
• Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT — METR, 2026-07-21 (empirical): "Estimating human expenditure for PRs with an LLM Judge" + Appendix B — an Opus-4.6 judge estimating human hours from code changes, commit messages, PR discussion and timing logs; r = 0.88 whole-PR vs sum-of-commits, R² = 0.00 against speedup and 0.01 against record order, a ~37% under-read against contributor retro-estimates, and the α = 1.58 [1.07, 2.62] correction applied to the curve. Full treatment on Expenditure Horizon
• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation — Mengting Chen et al. (FinStep + StepFun, arXiv 2607.29252, 2026-07-31, empirical): §2.1 (measurability z_j as reproducible judgeability, and its explicit gap from expert endorsement), §2.2 + Prop. 1 + App. B.1 (exponential unanimity attrition ρ^M, the M = 20/40 extrapolation, the 843-criterion empirical curve and the non-monotone joint gold rate peaking at m = 4), §2.3 + Remark 1 (the variance filter as the ability-blind zero-threshold case), §3.3 (Beta–Bernoulli measurability posterior, no human labels needed), §3.4–3.5 + Prop. 4 (IRT information over the fitted ability density, submodular coverage utility, (1 − 1/e) greedy guarantee, w_j ∝ ν_j weighting, bootstrap tiering), §4.2 + Table 3 (κ 0.604 → 0.743 on JudgmentBench, quartile κ contrast, r = 0.589/0.558 vs 0.127, the two-judge null, positive-label rates 55.6–62.9% vs 47.1%), §4.3 + Table 4 (rank-fidelity AUC, 131 → 49 rubrics, Greedy-vs-plain-IIF significant only on the 15-system blocks, 9.81% of JudgmentBench pairs separated), App. A Table 11 (ρ = 0.8833 against the human reference ranking for both scorers). Figure 1 viewed per the two-pass rule; Tables 1, 8 and 9 recovered from collapsed parses — see wiki/sources.md