Judge Independence: Options and Measured Evidence

Historical record — status checked 2026-09-06. The concluded judge-calibration and
training apparatus had been removed in #1019
(commit dated 2026-09-05). References to its commands, labels and reports describe that
historical experiment, not runnable files in the current checkout. Recover the original
apparatus from git history.
Recorded findings and the decision to keep the production default unchanged still apply.

Status: historical experiment log (begun 2026-08-02). Protocol:
EVR-2 human-validation protocol (Amendment 1:
AI-annotator gold). All rows measured on the SAME 96 real agent trajectories
(run 12 pack, qwen3-coder-tools:30b agent) against verifier gold (validated
κ=1.0 vs independent annotation) — reproducible via
--replay-pack + --human-labels on the committed labels
(benchmarks/judge_calibration/labels/run12/).

The option ladder

Tier	Independence from the agent	α (real pack)	Verdict
enhanced (in-loop self-assessment)	none — reads the live trace	−0.837	anti-correlated; run 12
Fresh-context same model (self-judge, current rubric wiring)	context only	0.469 (qa −0.03, refactor 0.10)	FAILS — run 14
Cross-model gemma4:31b	context + weights	0.694 (refactor 0.30)	FAILS at n=96 — run 12; demoted
Cross-model llama3.3:70b	context + weights	0.878 (refactor 1.00)	PASSES — run 13; the pinned judge
Linear classifier on verifier gold (E2 arm A)	weights + grounded training	−0.266 (dev-scripted 0.930)	FAILS — lexical features don't transfer
ModernBERT encoder (E2 arm B)	weights + grounded + semantic	pending	training
~1.5B QLoRA (E3)	as above + generative capacity	not started	pre-registered escalation only

What the ladder establishes

1. Context isolation is necessary but insufficient: fresh context recovers
   1.3 α over in-loop self-assessment, yet weight-level correlation still costs
   ~0.41 α vs an independent judge on identical trajectories. Session-model
   self-judging cannot enter the calibrated set.
2. Weight independence is what graduates on the calibration corpus:
   llama3.3:70b passes every corpus gate (scripted 1.000 at n=96, real 0.878,
   refactor 1.000) against both verifier and annotator gold. But this did NOT
   transfer — see point 4.
3. Grounded training beats nothing yet: the lexical baseline collapses on
   real trajectories (−0.266 despite 0.930 scripted-dev) — the semantic
   encoder arm was the open question, now answered in point 4.
4. The calibration corpus was not the shipping distribution (2026-08-05,
   DECISIVE): on in-container-verified SWE-bench-lite (real GitHub issues,
   17% solve) the corpus gate-passers over-credit (gemma4:31b α=−0.52,
   llama3.3:70b α=0.26) and a ModernBERT classifier trained on those real
   trajectories under-credits (α=−0.05, majority-class collapse). All three
   fail; only the in-container verifier discriminates. Do not graduate an
   LLM/classifier completion judge as a default — keep completion_strategy= "enhanced" + the verifier oracle. Revisit only with a higher-solve-rate
   agent (positives are scarce, not the judge's capacity).

Committed reports: benchmarks/judge_calibration/labels/run12/reports-*/;
SWE-bench re-gate: labels/swe-bench-lite-30/ and labels/swe-bench-lite-60/.
FINDINGS rows: runs 12–14 + the two SWE-bench re-gate sections in
FINDINGS.