LLM-as-a-Judge

Using one LLM to grade another's outputs against criteria/rubrics; DRACO's protocol is per-criterion binary MET/UNMET + justification, weight-aggregated into normalized score and pass rate; key properties — rankings stay stable across judge models while absolute magnitudes vary, and adaptive per-case rubrics (Google's AutoRaters) detect failures but blend them away, motivating stable custom metrics for the behavior under change; a 21-judge / ~541K-judgment audit finds raw exact-match agreement overstates chance-corrected reliability by 33–41pp (kappa deflation) and high test-retest can mask severe position bias, so judges need chance-correction, bias, and cross-benchmark validation before thresholded use; upstream, CalibratedRubric makes the rubric bank the instrument — measurability, informativeness and validity are distinct, and unanimity filters decay with leaderboard size