FEP-0025 n=60 — a promotable candidate that was a regression

Historical experiment record. Dates, candidate outcomes, commands, and local artifacts
below belong to this experiment. Preserve its verdict; use the Roadmap
for current execution status, and do not treat local database paths as distributed fixtures.

Date: 2026-07-28
Status: resolved — do not promote. Supersedes the n=24 checkpoint
(docs/analysis/2026-07-27-fep-0025-checkpoint.md, #709), whose "nothing has
earned promotion" was correct at n=24 but for the wrong reason: at n=60 a
candidate did clear the gate — and then turned out to be a regression on the
target branch.

This is a measurement-first case study. Three things had to be true at once for
the right answer to land: the gate had to be comparative, the result had to be
recordable, and the measurement had to be on the branch it would ship to. All
three failed in a different way, and each failure would have produced a wrong
promotion decision on its own.

The arc

run	tree	baseline	best candidate	verdict
n=24 (#709)	pre-fix	—	all ±2, p≥0.75	refused (noise)
n=60	stale (c8edc8d74)	31/60 (51.7%)	b54762402bda +9/60 (disc 14/5, p=0.06)	cleared the gate — but discarded (see below)
n=60 develop	bbe2bd655	46/60 (76.7%)	b54762402bda −10/60 (disc 7/17, p=0.06)	refused — regression

The defect the n=60 exposed

At n=60 the ±2 at n=24 resolved into a real +9/60 for both candidates; the
winner b54762402bda cleared the corrected gate (effect 9 ≥ √19 = 4.36). The
suite reported "no candidate met the benchmark approval threshold" anyway. The
gate was not the problem — the result was never recorded.

Candidates are stored under the provider they were evolved under
(provider=moonshot). They are measured under whichever provider has quota
(deepseek). Serving resolves that mismatch — optimization_injector
falls back to find_candidate_any_provider, so the candidate's text was
injected correctly and the +9/60 measurement is valid. Recording did not —
record_benchmark_result used _find_candidate with no fallback, found
nothing under ::deepseek, returned None, and decision.passed = passed AND recorded collapsed to false however strongly the candidate won.

Every cross-provider benchmark is silently un-recordable under that code, and
every benchmark is cross-provider in practice. Fixed in #718:
record_benchmark_result now falls back to find_candidate_any_provider,
matching serving. Production-validated by the develop run below —
benchmark_runs incremented for the first time on a deepseek measurement.

This is the same defect class as the self-inflicted prompt-injection work: the
framework classifies its own artifacts (messages by role, candidates by
provider) and then cannot carry a measurement across the boundary it drew.

The decision the develop run reversed

With recording fixed, the same suite was re-run on develop (bbe2bd655,
which includes the prompt-injection fixes #708/#713/#714/#715). The seed jumped
51.7% → 76.7% — the shipped prompt and agent got dramatically better. The
candidate that was +9/60 against the old, weak seed is −10/60 against the
improved one: b54762402bda is the worst arm on develop.

b54762402bda is the "naming rule" candidate ("the tests are the specification;
match their identifiers, do not edit the tests"). At n=24 it cut its target
failure class 9→6. It was evolved against a baseline that no longer exists. The
fixes that made the seed better also addressed the weakness the candidate
targeted — so the candidate now adds noise and misdirection on top of a prompt
that no longer needs it.

Decision: do not promote. Neither candidate clears the gate on develop.

Why the within-develop verdict is solid

The develop venv lacks the victor-coding vertical (0.8.0 is not on PyPI; only
0.7.2, a version-mismatch risk), so code search is off — a different tool
environment from the stale-tree n=60. This confounds the stale-vs-develop
comparison (tree and code-search changed). It does not confound the
promotion decision: both arms ran the same 60 tasks in the same develop
environment, so the −4/−10 paired contrasts are environment-controlled. On
develop, the candidates do not beat the seed. The missing vertical affects
absolute rates, not the contrast.

(The default shipped victor does not bundle victor-coding either, so the
no-code-search environment is the representative one.)

Lessons

1. A candidate that clears the gate on one tree can be a regression on a
   later tree whose seed has improved. Always re-confirm on the target branch
   before promoting. The stale-tree +9/60 would have shipped a −10/60
   regression.
2. Serving and recording must agree. A candidate the framework will serve,
   it must be able to record — or measurements vanish silently and read as
   refusals. #718 made the two paths use one resolution.
3. A gate refusal is not self-evidently statistical. "No candidate met the
   threshold" can mean the candidate lost or that the result was never
   recorded. Verify benchmark_runs actually incremented before treating a
   refusal as evidence about the prompt.
4. The seed can outgrow its candidates. The prompt-injection fixes improved
   the shipped prompt faster than evolution had. The candidates here targeted a
   weakness that was since fixed at the source; they should be retired and
   re-evolved against the new 76.7% baseline.

Resume

The candidate store holds 785071738065 and b54762402bda under
GROUNDING_RULES::moonshot, both now measured on develop. Neither is
promotable. Re-evolution should start from the current develop seed, not the
pre-fix one these candidates descend from.