Skip to content

U10 — Evaluation, observability, experiments, benchmark & contrib

210 modules / 79,702 LOC / 1,512 churn commits (12m). Reviewer: parallel unit review, 2026-09-03.

Overview

evaluation/ (39k) and observability/ (24k) carry the weight; contrib/ is actually only 4.9k LOC of vertical base classes (null-object impls consumed by core/bootstrap.py). The judge-independence program left ~2.5k LOC of concluded research scaffolding plus ~1.4k LOC of negative-A/B effect-gate aftermath. The strongest live designs are the runtime_feedback trust model, run_kind tagging, and the acceptance-oracle/flag-graduation policy; the weakest spots are a parallel agent-creation path in the churn-leader adapter, synchronous per-event disk writes on the chat hot path, and three overlapping benchmark execution stacks.

Findings

ID Sev Location Finding Why it matters Fix sketch Effort
F1 P1 victor/evaluation/agent_adapter.py:1808-1821 from_profile hand-builds the stack (load_settings → mutate → ProviderRegistry.create → direct AgentOrchestrator(...)) while create_from_session_config (1845-1853) uses Agent.create. Two creation paths in the churn=71 leader module Violates the AgentFactory single-authority rule; eval arms silently miss middleware/verticals/services defaults production gets; #820 was a bug only this path could grow Route from_profile through Agent.create/profile bootstrap; keep ProviderRegistry out of the adapter M
F2 P1 victor/observability/analytics/enhanced_logger.py:504-505, 533-535 log_event does .stat() (rotation check) + open/append/close per event, inline on the hot path — called per tool result (tool_service.py:427), per turn with full prompt/response content (chat_service.py:1445-1447), per stream (metrics_collector.py:338) Observer effect: multiple syscalls + regex PII scrub + deep sanitize per turn/tool event on the streaming path; the bus already has a batching design (BufferedExporter, exporters.py:557) that nothing uses Make the usage logger an ObservabilityBus exporter behind BufferedExporter, or copy chat_service's persist_synchronously=False executor pattern M
F3 P2 victor/evaluation/harness.py:1572-1578, 1182-1193 Checkpoint rewrites ALL accumulated results (incl. generated_code full source) after every task completion, inside the shared async lock O(N²) write amplification over a run; parallel completions serialize behind the lock for the dump Append-only JSONL checkpoint + compact manifest; full snapshot at run end S/M
F4 P2 victor/evaluation/harness.py:1123-1133 Checkpoint id hashes only benchmark/model/max_tasks/provider/prompt; agent flags, temperature, tool allowlist, framework version absent Resuming after a flag flip or code change silently mixes arms/results — poisons exactly the A/B comparisons this harness exists for Add an agent-config hash (AgentSettings snapshot + flags + version) to _checkpoint_path S
F5 P2 victor/evaluation/judge_calibration*.py, calibration_{rubric,classifier,corpus,agent_executor}.py, judge_training_data.py (~2.0-2.5k LOC) Concluded judge program's machinery: rubric/classifier judges, training-data generator, calibration corpus have zero non-eval non-test consumers; only calibration_enhanced_judge.py (119 LOC, the #884 keeper adapter) re-wraps the in-container verifier 2k LOC of churn surface for a program whose decision is codified negative; future readers can't tell evidence tooling from live eval paths Keep enhanced adapter + stratum; archive the judge arms and training-data generator next to benchmarks/judge_training/ or an eval-research extra M
F6 P2 victor/evaluation/completion_strategy_ab.py (601), effect_gate_corpus.py (282), victor/framework/effect_gate.py (533) effect_gated_completion A/B verdict is HOLD (on 0.847 vs off 0.864, snapshots at repo root 2026-08-04); flag stays default-off (agent_config.py:92) but the corpus, dedicated A/B runner, and 533-line framework gate persist unmarked Dead-flag aftermath: 1.4k LOC + config surface that will never graduate; flag_ab.py FLAG_ENV still lists it first, implying it's an active candidate Remove flag + gate + corpus + dedicated runner (record the negative A/B in the PR), or park behind an explicit eval-research marker; keep generic flag_ab/flag_graduation infra M
F7 P2 victor/observability/resilience.py (868 LOC) Circuit breaker/retry/bulkhead/timeout re-implemented "subclassing canonical from providers"; sole consumer is its own test; agent/resilience.py:135,305 and providers/resilience.py:280 reference it only in docstrings Third resilience implementation that can drift; 868 LOC for no caller Delete module + test; repoint docstring references S
F8 P2 victor/evaluation/{harness,agentic_harness,code_generation_harness}.py; victor/benchmark/ (agent.py, workflows/*.yaml, harness_integration.py) Three-plus benchmark execution stacks: three Python harnesses, a YAML-workflow benchmark vertical, BenchmarkAgent, and VictorAgentAdapter bridged by harness_integration.create_agent_callback. The irreducible model (task→run→trace→score) exists once in protocol.py (fan_in=27) but five entry points wrap it Duplication of parallelism/checkpoint/scoring logic; drift between what YAML benchmarks measure and what Python harnesses measure Declare protocol.py's BenchmarkRunner/EvaluationHarness the single harness; make the YAML vertical compile to the same protocol; fold code_generation_harness into it L
F9 P3 victor/evaluation/harness.py:96-165 vs victor/experiments/ab_testing/statistics.py:84+ McNemar/paired-contrast stats hand-rolled in harness while a scipy-backed StatisticalAnalyzer lives in experiments/ab_testing Two statistical toolkits can disagree on CIs — the exact quantity flag graduation reads One shared stats module S
F10 P3 victor/observability/query_service.py:288-387; enhanced_logger.py:558-587 JSONL query path scans the full file then slices offset:limit; both the TUI dashboard (ui/commands/dashboard.py:134) and HTTP surface (observability_routes.py:32) poll it Dashboard refresh cost grows with log size; every request is O(file) Tail-cursor or sqlite event sink (project.db already exists) with incremental read M
F11 P3 victor/observability/analytics/streaming_metrics.py:68-70, 756-788 StreamMetrics deprecated-in-place "in 0.10.0" with fan_in=10 (metrics_collector.py:341 still imports it); MetricsStreamWrapper.__anext__ runs hasattr-chains + regex token estimation per chunk Deprecation without a named replacement blocks cleanup; per-chunk string work is pure overhead when provider usage accounting already emits stream_completed totals Declare the replacement (usage-based accounting), migrate metrics_collector, delete S/M
F12 P3 victor/observability/emitters/ (2,056 LOC) Five protocol + five implementation emitter modules have exactly one consumer: bridge.py:42, itself used only by orchestrator_creation.py:105 ISP layer that never grew consumers; 2k LOC of interface surface Fold into bridge.py or freeze with a stability note S
F13 P3 victor/observability/metrics.py:278 Histogram.observe calls sorted(self._bucket_counts.keys()) on every observation plus increments every bucket ≤ value Per-observation sort on a hot in-memory path Precompute sorted bucket tuple at construction; bisect S
F14 P3 chat_service.py:1445-1447 → ~/.victor/logs/usage.jsonl Full user prompt + assistant response content persisted per turn, unencrypted by default (encrypt=False; PII scrubber is regex-level) Conversation retention at rest in a world-readable-home default path Make emission run_kind-aware; document retention; default-on encryption for content-bearing events M

Co-design opportunities

  1. One batched observability sink: EnhancedUsageLogger as an ObservabilityBus exporter behind the existing BufferedExporter — metrics collector, RL trace-miner, and dashboard share one I/O cadence (F2/F10).
  2. (task, agent-config) hash as the single cache/resume key across harness checkpoints, flag_ab arms, and prompt-candidate runs (F4).
  3. Converge the YAML benchmark vertical onto the BenchmarkRunner protocol so both stacks exercise Agent.create via AgentFactory identically (F1/F8).
  4. run_kind-gated payload detail: the contextvar already distinguishes evaluation/delegate/interactive; extend from a label to a policy on content volume (F14).
  5. Verifier-vs-X agreement tool: replace the calibration-judge family with one parameterized agreement report (judge injected, verifier as gold) so the negative-evidence dataset stays reusable (F5).
  6. Joint schema ownership: add the acceptance oracle as a streaming consumer of per-task events so eval progress is observable without reading checkpoint files (F3).

Keep intact

  1. runtime_feedback source-trust + freshness half-lives (runtime_feedback.py:26-45) — consumed by agent/services/runtime_intelligence.py:31; the joint producer/consumer schema, load-bearing for RL attribution.
  2. run_kind contextvar tagging (harness.py:1275 run_kind_scope(RunKind.EVALUATION)) — fixed a ~2x eval-share misclassification; every new runner must keep tagging.
  3. flag_graduation.assess_graduation (flag_graduation.py:81-140) — safety AND benefit policy (#861); generic infra per_turn_auditor graduation still needs; do not delete with the effect-gate cleanup.
  4. EvalContainer persistent per-task lifecycle (container_eval.py:255-261) + stale-sandbox sweep (#877) + deterministic image names (#868) — container reuse at the correct granularity.
  5. ObservabilityBus lossy contract — AT_MOST_ONCE, get_delivery_pressure_stats, register_drop_alert_handler (backends.py:1095-1129); dropped-spans-not-blocking is the right behavior; sandhi_meter's default-off optional bridge likewise.

Quick metrics

  • Unit: evaluation 78 files/38,927 LOC; observability 66/23,852; experiments 16/6,206; benchmark 12/5,794; contrib 38/4,923 (contrib is a vertical base-class/null-object library, not 80k).
  • Churn leaders: agent_adapter 71, harness 55, swe_bench 41, protocol fan_in=27.
  • Judge-era: 9 modules ≈2.0-2.5k LOC, 0 external non-test consumers; keeper adapter 119 LOC. Effect-gate aftermath: 1,416 LOC + flag wiring; A/B verdict HOLD.
  • Dead-on-arrival code: resilience.py 868 LOC (1 consumer = own test); emitters/ 2,056 LOC (1 consumer chain); BufferedExporter exported but unused.
  • Hot-path write counts per turn: 1 stat + 1 open/append/close × (2 content events + 1 stream_completed + n tool_results) from 3 separate call sites.
  • Two benchmark agents, three Python harnesses, one YAML benchmark vertical, four YAML workflows.