EVR-2 Human-Label Validation Protocol (ADR-011)

Historical record — status checked 2026-09-06. The concluded judge-calibration and
training apparatus had been removed in #1019
(commit dated 2026-09-05). References to its commands, labels and reports describe that
historical experiment, not runnable files in the current checkout. Recover the original
apparatus from git history.
Recorded findings and the decision to keep the production default unchanged still apply.

Status: Pre-registered 2026-08-02, BEFORE any labels were collected.
Rule: these thresholds and procedures may not change once labeling begins;
a run that motivates changing them is a failed run, recorded as such in
FINDINGS.

Why

FINDINGS runs 1–11 measured judges against programmatic verifier gold
(workspace-state checks). ADR-011 and the
flag-graduation policy require validation against
human labels before any judge-gated completion strategy becomes a default.
This protocol validates two things at once: the judge (does it agree with
humans?) and the verifier gold itself (did runs 1–11 measure against labels a
human would endorse?).

Label set

~150–200 items, three strata:

1. The 48 real-agent trajectories from the run-11 distribution (re-exported).
2. One fresh 96-trajectory real-agent run (--agent-profile, --variants 16,
   two-phase).
3. ~20–40 SWE-bench-lite smoke trajectories (cross-corpus external validity),
   once available from the evidence workstream.

Unit of labeling: binary completion verdict per trajectory (matches the
harness scope note: gold labels are binary completion verdicts only).

Annotators and blinding

• Primary: the maintainer, labeling from the exported labeling pack
  (--export-labeling-pack) — the pack is the judge's blinded view (prompt +
  transcript + workspace snapshot) and structurally contains no verifier
  verdict or judge score.
• Secondary: one LLM annotator, disjoint from every judge candidate and
  from the agent under test, labeling blind from the same pack.
• Disagreements between primary and secondary get a maintainer re-audit with
  written rationale; the audited maintainer label is the final human gold.
• Labels are committed under benchmarks/judge_calibration/labels/ as JSONL:
  {"task_id", "label", "annotator", "rationale"}.

Amendment 1 (2026-08-02, maintainer-directed, BEFORE any labels collected)

The maintainer delegated the primary-annotator role to an AI annotator
(Claude, Anthropic) to unblock the validation run. Consequences, recorded
before labeling began:

• Primary = Claude annotator pass A; Secondary = an independent Claude
  pass B run with a skeptical lens and no access to pass A's labels. Both
  passes label ONLY from the blinded pack. Both are disjoint from every judge
  candidate (gemma4:31b, llama3.3:70b) and from the agent under test
  (qwen3-coder-tools:30b) — the disjointness requirement holds.
• Disagreements between the passes get an audited resolution with written
  rationale (in place of the maintainer re-audit); the audited label is final.
• The resulting gold is AI-annotator gold, not human gold. Every
  downstream claim (FINDINGS rows, methodology page, README) must say
  "validated against independent AI-annotator gold" — never "human-validated"
  — until the maintainer audits or replaces the labels. A maintainer
  spot-check audit remains open as the upgrade path to the original protocol.
• Thresholds, blinding, VOID conditions, and the stop-the-line rule are
  unchanged.
• Stratum note: stratum 1 (re-export of the 48 run-11 trajectories) is
  unavailable — those reports were gitignored and never committed (the exact
  gap this evidence program exists to close); stratum 3 (SWE-bench smoke
  trajectories) is deferred to the anchor-run cycle. The validated set is the
  fresh 96-trajectory stratum, which meets the n ≥ 16/family gating-grade bar
  on its own.

Pre-registered thresholds

Computed by the overlay (--human-labels, module
victor/evaluation/human_label_overlay.py):

Check	Threshold	On failure
human↔verifier Cohen's κ	≥ 0.8	STOP THE LINE — verifier gold is invalid; FINDINGS runs 1–11 conclusions are void; fix corpus verifiers, re-validate, re-run
human↔judge Krippendorff α (overall)	≥ 0.7	Judge not graduated; rubric stays opt-in
human↔judge α (per family)	≥ 0.7 for every family with n ≥ 16; families below n = 16 are directional only and not claimed	Judge not graduated
human↔secondary κ	reported, never gating	Audit disagreements; note in FINDINGS

VOID conditions (inherited from the harness integrity guards): any grading-call
failures or ungradable outputs in the underlying run; single-class gold
(α cannot measure discrimination); unlabeled template lines in the labels file
(the loader rejects partial label sets).

Consequences

• All checks pass → EVR-2 is complete for the measured judge identities;
  the rubric default flip (eval-loop program PR-8) is unblocked once EVR-3
  parity also holds. Judge identities stay pinned via
  agent.rubric_judge_calibrated_models (ADR-011 pinning gate).
• Verifier κ fails → stop-the-line remediation before anything else.
• Judge α fails → the judge stays untrusted; the honest result ships in
  FINDINGS either way.

Acknowledged limitations

n=1 human annotator (single-maintainer project), mitigated by the blind
secondary annotator and audited disagreements — stated openly wherever these
results are published, not hidden.