Judge Datasheet v1.0¶

Generators answer “what comes next?” Verifiers answer “is this correct/good?” Reward models learn a scalar field of preference or quality; judges may produce a score, ranking, critique, or decision.

1. Four families that should not be conflated¶

1.1 Scalar Reward Model¶

\[ r_\theta(x,y)\in\mathbb R \]

Input: prompt/context \(x\) + response/action \(y\).
Output: scalar.

1.2 Pairwise Preference Model¶

\[ P(y_A\succ y_B|x) \]

A latent score can induce preference:

\[ P(A>B)=\sigma(r_A-r_B) \]

1.3 Verifier¶

Decides a verifiable property:

\[ V(x,y)\rightarrow\{0,1\} \]

or a probability:

\[ P(correct|x,y) \]

It may be:

rule-based;
executable;
learned;
generative.

1.4 Judge¶

More general system:

\[ J(x,y,criteria)\rightarrow score/rank/critique \]

It may be a generative LLM, not necessarily a specialized reward head.

2. Bradley–Terry preference model¶

If \(r_A,r_B\) are utilities:

\[ P(A>B)= \frac{e^{r_A}}{e^{r_A}+e^{r_B}} = \sigma(r_A-r_B) \]

Loss for chosen/rejected pair:

\[ \mathcal L = -\log\sigma(r_{chosen}-r_{rejected}) \]

Training learns relative differences, not necessarily a calibrated absolute scale.

3. Score ≠ probability¶

A reward of 7.4 does not mean a 74% chance of correctness.

To turn a score into a reliable probability, we need calibration:

\[ P(correct|r) \]

estimated/validated separately.

Possible techniques:

Platt/logistic scaling;
isotonic regression;
temperature scaling;
binning/calibration curves.

4. Outcome Reward Model — ORM¶

Assigns an outcome-level score, supervised by the final result. A common case evaluates only the final answer:

\[ R_{outcome}=R(x,y_{final}) \]

An ORM can also consume the complete solution or trajectory. Outcome supervision describes the source of the target signal, not a mandatory restriction on the model's input.

Advantages:

cheap;
simpler labeling;
strong when the outcome is verifiable.

Limitations:

cannot identify where the trajectory went wrong;
may reward a correct result produced through spurious reasoning.

5. Process Reward Model — PRM¶

Evaluates intermediate steps:

\[ r_t=R(x,y_{1:t}) \]

or transitions:

\[ r_t=R(s_t,a_t,s_{t+1}) \]

This enables:

early error detection;
search pruning;
agent supervision;
dense reward.

Cost¶

If every step requires evaluation:

\[ C_{PRM}\propto N_{steps}\times C_{verifier} \]

Process supervision improves observability but costs far more inference/annotation.

6. Bidirectional / lookahead process evaluation¶

Evaluating a step from the prefix alone can be ambiguous:

\[ P(correct\ step|prefix) \]

Continuation/outcome information can be incorporated:

\[ P(step\ valid|prefix,suffix/outcome) \]

This reduces myopia, but creates leakage if used incorrectly in an online setting.

7. Generative verifier¶

Instead of a scalar head:

\[ (x,y) \rightarrow LLM \rightarrow analysis/critique \rightarrow verdict \]

It may use tools/environment:

\[ Verifier \rightarrow Tool \rightarrow Evidence \rightarrow Verdict \]

In 2026, environment-aware generative PRMs are appearing specifically to catch silent failures that do not surface as exceptions.

Trade-off¶

A qualitative trade-off to assess empirically; the arrows do not assert a universal increase in latency, cost, or variance:

\[ VerifierCapability\uparrow \leftrightarrow Latency/Cost/Variance\uparrow \]

8. Rule-based verifier¶

When a deterministic checker exists:

\[ V(y)=1[y\in\mathcal S_{valid}] \]

Examples:

compilation;
unit tests;
symbolic equivalence;
schema validation;
game outcome;
constraint solver.

This class provides high-precision reward within the covered domain, but that domain may be narrow.

9. RL with Verifiable Rewards — RLVR¶

Policy:

\[ y\sim\pi_\theta(y|x) \]

Verifier:

\[ r=V(x,y) \]

RL optimizes:

\[ \max_\theta E_{y\sim\pi_\theta}[r] \]

The bottleneck becomes verifier coverage.

Generative verifiers broaden the domain, but trade deterministic certainty for model error/calibration.

10. LLM-as-a-Judge¶

Evaluation prompt:

\[ J(prompt,response,rubric)\rightarrow judgment \]

Outputs may include:

scalar 1–10;
binary pass/fail;
pairwise preference;
categorical rubric;
structured JSON;
textual critique.

Important¶

The judge is another model with its own priors.

\[ JudgeError\neq0 \]

11. Pairwise judging¶

Given A/B:

\[ J(x,A,B)\rightarrow A>B\text{ or }B>A \]

Advantage: relative comparison is often cognitively easier than absolute scoring.

Problem: position bias.

Swap test:

\[ J(x,A,B)\stackrel{?}{=}inverse(J(x,B,A)) \]

If not, the judge is positionally unstable.

12. Judge biases¶

Position bias¶

Preference for the first/second option.

Verbosity bias¶

A judge exhibits verbosity bias when it systematically favors the longer response even when answer quality is held equivalent. Measure this with controlled pairs, randomized positions, and an explicit tie policy; a higher win rate for longer answers without controlling quality does not establish this bias.

Style bias¶

Markdown, assertiveness, equations, or tone may change the score without changing correctness.

Self-preference¶

A model may favor outputs that resemble its own style/distribution.

Reference anchoring¶

A flawed reference can pull the judge toward the wrong conclusion.

13. Calibration¶

If the judge returns confidence \(p_i\):

Ideal calibration:

\[ P(correct|p=0.8)\approx0.8 \]

Brier score¶

\[ BS=\frac1N\sum_i(p_i-y_i)^2 \]

Expected Calibration Error¶

\[ ECE=\sum_b\frac{|B_b|}{N} |acc(B_b)-conf(B_b)| \]

High accuracy does not imply good calibration.

14. Bias-corrected evaluation¶

If a judge has imperfect sensitivity/specificity, its raw judgment rate may be a biased estimator of true performance.

Conceptually:

\[ ObservedScore =F(TrueQuality,JudgeQuality,Calibration) \]

When judge quality shifts across models/domains, direct score comparisons can reverse conclusions.

So the SOTA++ question is not merely:

“which judge is better?”

but:

“how stable is this judge's error/calibration function in this domain?”

15. Best-of-N¶

The generator produces:

\[ Y=\{y_1,\ldots,y_N\} \]

The verifier selects:

\[ i^*=\arg\max_{i\in\{1,\ldots,N\}}R(x,y_i), \qquad y^*=y_{i^*} \]

Use a fixed tie-breaking rule when several candidates share the maximum score. Increasing \(N\) can raise the probability that a good candidate is available, but the verifier must identify it to turn that coverage into better selected quality.

For nested candidate sets and a fixed scoring function, the maximum proxy score cannot decrease; the oracle-best true quality also cannot decrease over the same nested sets. The true quality of the verifier-selected response can decrease because of ranking errors or reward overoptimization. Separately resampled candidate sets need not show either maximum increasing on every run.

For any given candidate set:

\[ Quality_{selected} \le Quality_{oracle-best} \]

The gap depends on verifier error; this upper bound is not a guarantee of improvement with more candidates.

16. Selection economics¶

Approximate cost:

\[ C_{total} =N\cdot C_{generation} +N\cdot C_{verification} \]

If the verifier is itself a large generative model, selection may cost as much as generation.

Evaluate marginal gain:

\[ \frac{\Delta Quality}{\Delta Compute} \]

17. Majority voting / self-consistency¶

For discrete answers:

\[ \hat y=mode(y_1,\ldots,y_N) \]

Canonicalize equivalent answers before taking the mode. Independence alone is insufficient: for iid binary votes, majority accuracy tends to 1 as the number of votes grows only when each vote is correct with probability \(p>1/2\); if \(p<1/2\), it tends to 0. For iid votes over a fixed finite set of answer categories, the correct answer must be the unique most probable category for the sample mode to converge to it.

At comparable individual accuracy, dependence between errors can reduce voting gains. The following is a heuristic, not a universal monotonic law:

\[ Corr(error_i,error_j)\uparrow \Rightarrow benefit\downarrow \]

Ensemble diversity matters.

18. Hidden-state verification¶

Verification does not need to inspect only final surface text.

If a reasoning trace has hidden states:

\[ h_{start},h_{end} \]

we can define:

\[ \Delta h=h_{end}-h_{start} \]

and classify trajectories using internal geometry.

ACL 2026 work shows that correct/incorrect trajectories can exhibit useful geometric differences for selection without training an additional reward model.

This expands “verifier” beyond surface text.

19. Rubric-augmented judging¶

Rubric:

\[ R=\{criterion_1,\ldots,criterion_m\} \]

A judge may decompose:

\[ score(y)=\sum_iw_i score_i(y) \]

Advantage: auditability.

Risk: a poor rubric creates structured misguidance — a wrong criterion consistently steers the model toward the wrong decision.

20. Reward hacking¶

The policy optimizes a proxy:

\[ \max R_{proxy} \]

but we want:

\[ \max U_{true} \]

If:

\[ R_{proxy}\neq U_{true} \]

the policy may exploit flaws.

Goodhart:

when a measure becomes a target, its relationship with the true objective can degrade.

21. Overoptimization¶

Even a reasonable reward model can fail out of distribution when the policy is pushed toward extremes:

The relevant shift is toward responses poorly covered by the reward model's training distribution. These are qualitative risk relationships, not a pointwise ordering of policy probabilities.

The harder you optimize against an imperfect RM, the more likely you are to exploit uncalibrated regions.

22. Verifier-guided search¶

Reasoning tree:

\[ s_0\rightarrow\{s_1^1,\ldots,s_1^k\} \]

PRM/verifier scores:

\[ R(s_t^i) \]

Beam/search keeps the top candidates.

Trade-off:

\[ BranchingFactor\uparrow \Rightarrow SearchCoverage\uparrow, Compute\uparrow \]

23. Code: executor as verifier¶

In code:

\[ code\rightarrow compiler/tests\rightarrow result \]

This is an excellent external verifier for the properties it covers.

But:

\[ TestsPass\not\Rightarrow ProgramCorrect\ universally \]

Test coverage limits the evidence.

See 03_CODE_GENERATION_AND_AGENTIC_CODING_DATASHEET.

24. Agents: process reward + environment¶

For trajectory:

\[ \tau=(s_0,a_0,s_1,a_1,\ldots) \]

An environment-aware PRM can verify:

whether the action actually observed evidence;
whether the state changed as expected;
whether an error was recoverable;
whether the tool was used correctly.

This moves evaluation closer to control systems rather than text alone.

25. Metrics¶

Pairwise accuracy¶

\[ Acc=\frac{correct\ preferences}{total} \]

Kendall \(\tau\)¶

Ordinal correlation.

Spearman \(\rho\)¶

Rank correlation.

Calibration¶

ECE/Brier/reliability diagram.

Agreement¶

Cohen's kappa or equivalent when appropriate.

Do not use one metric for everything.

26. Failure surfaces¶

reward-model domain shift;
preference inconsistency;
label noise;
verbosity/style bias;
position bias;
reference error;
calibration drift;
reward hacking;
overoptimization;
correlated verifier/generator error;
hidden rubric assumptions;
process reward penalizing legitimate exploration.

27. SOTA Snapshot 2026¶

Relevant trends:

PRMs are moving beyond math-step scoring and beginning to interact with environments;
generative verifiers broaden RLVR into domains without trivial checkers;
hidden-state trajectories are starting to be used as correctness signals;
judge calibration/bias is being treated as an explicit statistical problem;
rubric-augmented reward modeling tries to decompose “quality” into more auditable criteria.

28. Checklist for any evaluator¶

Is the output a score, probability, rank, or critique?
Is the scale calibrated?
Was it trained pairwise or pointwise?
Does it evaluate outcome or process?
Does it use external evidence?
Is a deterministic checker available?
What is the domain shift?
Is there position/verbosity/style bias?
Was a swap test performed?
What is the cost per candidate?
Do generator and judge share model-family bias?
Can the reward be exploited?
Which metric validates agreement/calibration?
Does Best-of-N improve because of the generator or the selector?

Snapshot references¶

DataPRM / Process-Level Reward Modeling for Agentic Data Analysis — https://arxiv.org/abs/2604.24198
Your Reasoning Model is Secretly a Reward Model — https://aclanthology.org/2026.acl-long.788/
Bias and Uncertainty in LLM-as-a-Judge Estimation — https://arxiv.org/abs/2605.06939
C2: Rubric-Augmented Reward Modeling — https://aclanthology.org/2026.acl-long.523/
The Bidirectional Process Reward Model — https://aclanthology.org/2026.acl-long.572/
Crossing the Reward Bridge / generative verifier RLVR — https://aclanthology.org/2026.acl-long.178/