LLM-Judge Validation

UC Berkeley's 21-judge / 9-provider / ~541K-judgment audit (Norman et al., 2026): LLM-as-a-judge validation is systematically under-rigorous — exact-match agreement overstates chance-corrected κ by 33–41pp (kappa deflation, universal across every judge), judge rankings shift up to 14 positions across benchmarks, and high test-retest reliability masks severe position bias (the consistency–bias paradox); distilled into a 5-step Minimum Viable Validation Protocol. Yang et al. (2026) add the judge-*version* axis: upgrading the evaluator is not a reliability intervention (one robust step in 18, and scaling makes it worse on 2 of 4 datasets), and repeated-sample juries are capped by measured error correlation ρ ≈ 0.66–0.97. Chen et al. (2026) push it down to the rubric *item*: measurability (judge agreement), informativeness (IRT information) and validity are three different properties