Skip to content

U5 — Storage, core, state & victor-codegraph

313 modules / 111,994 LOC / 2,579 churn commits (12m). Reviewer: parallel unit review, 2026-09-03. Metrics gathered during a live reindex (consistent-snapshot where noted).

Overview

Layered cleanly (Infrastructure) and the recent graph-backend work (#873–#915) shows real co-design discipline. The dominant weaknesses are on the default SQLite path: (1) the incremental machinery does not actually amortize parsing — one changed file triggers a full single-threaded parse_repo pass; (2) project.db is a derived, rebuildable artifact that grows to its churn high-water mark and never reclaims (2.64 GB measured, no auto-vacuum in the file header, manual-only maintain()); (3) per-node/per-file round trips survive on SQLite that were already batched on Proxima. Protocol/backend capability drift is the main structural debt.

Findings

ID Sev Location Finding Why it matters Fix sketch Effort
U5-01 P1 victor/core/database.py:1637-1646, 641-669; victor/ui/commands/db.py:225-253 project.db grows to churn high-water mark and never shrinks: auto_vacuum=INCREMENTAL set per-connection but the file header shows auto_vacuum=0 (takes effect only after a VACUUM); maintain() reachable only via manual victor db maintain. Measured 424,453 free pages (1.74 GB, 66% of file) mid-churn; WAL hit 281 MB despite wal_autocheckpoint=10000 (40 MB) — checkpoint starvation Derived artifact is the largest file in the repo dir; disk + reader-open cost grow monotonically Auto PRAGMA incremental_vacuum + wal_checkpoint(TRUNCATE) after force rebuilds and periodically in GraphManager; one-time migration VACUUM M
U5-02 P1 victor/core/graph_rag/indexing.py:527-532; codegraph_repository.py:24-38; victor-codegraph/repository_index.py:156-250 Any incremental run with ≥1 changed file calls parse_repo(root) — a serial full-repo read+parse+resolve of every source file — then filters to allowed files after parsing; the mtime+hash manifest only saves legacy extraction, CCG and DB writes, not this pass Dominant CPU and latency term of every watcher-triggered refresh; ParsedRepository holds all file texts + symbols in RAM Thread a previous manifest into parse_repo: parse only added/changed, reuse prior ParsedCode; generation_id already hashes the manifest L
U5-03 P2 victor/storage/graph/sqlite_store.py:1136-1152, 1115-1134; indexing.py:1289 get_stale_files runs one SELECT MAX(mtime)…WHERE file IN (…) per file (~4k queries per incremental run), each preceded by Path.resolve() syscalls; get_file_hashes repeats the shape Pure planning overhead on the hot reindex-on-save path Single SELECT file, mtime, content_hash FROM graph_file_mtime + dict comparison in Python S
U5-04 P2 indexing.py:3054-3068; sqlite_store.py:1064-1099 On SQLite, embedding bookkeeping writes one transaction per node (lock + SELECT + UPDATE + commit); the batched path (#905) landed only for upsert_node_records (Proxima) First-time embed of ~25k symbol nodes = ~25k fsynced WAL transactions Protocol-level batched metadata update (executemany) or one write_batch around the mark-embedded loop S
U5-05 P2 sqlite_store.py:697-743, 1225-1233; indexing.py:3259-3265, 3311-3328 All SQLite I/O runs synchronously on the event loop inside async def store methods (no to_thread); _flush Phase B commits 20 files of CCG nodes+edges (10⁴–10⁵ rows × node+FTS-trigger writes) while holding self._lock; comment at indexing.py:3259 admits loop blocking Chat process with background reindex stalls per mini-batch write; open cursors pin read snapshots (contributes to WAL starvation) Dedicated writer thread (or asyncio.to_thread per flush); keep the asyncio lock as admission control M
U5-06 P2 indexing.py:711-733, 2940; sqlite_store.py:1038-1045 _persisted_symbols() materializes get_all_nodes() — all 1.18 M nodes including 1.10 M statement nodes the passes never target — into GraphNode objects per indexing run Hundreds of MB RAM spike and multi-second load per incremental run on the default backend; Proxima returns Tier-A only, so default backend is the worst case Store-level get_symbol_index() / iter_nodes(type IN (…)) with the filter pushed into SQL M
U5-07 P2 victor/storage/graph/protocol.py:132-380; cpg_fragments.py:1-16 Protocol/backend drift: ~10 protocol methods unimplemented on SQLite (get_subgraph, multi_hop_traverse, …); update_node_metadata/write_batch exist on both real backends but not the protocol; the Tier-A hot/Tier-B cold CPG split exists only behind proxima_store — SQLite keeps 93% of graph_node as statement rows in hot tables Same logical write produces structurally different graphs per backend; getattr feature-probing everywhere; protocol is aspirational, not a contract Split protocol into core + optional capability Protocols; port cpg_fragments tiering to SQLite L
U5-08 P2 victor/agent/conversation/store.py:1135-1160, ~1186; conversation/migrations.py:83-98; victor/core/database.py:1679-1730 ConversationStore owns a second connection stack + migration system for the same project.db: own thread-local pool (busy_timeout 60 s) AND raw sqlite3.connect() per call in update_session_token_usage, bypassing ProjectDatabaseManager (busy_timeout 5 s, FK ON); version tracking split between _project_metadata.schema_version=9 (int) and a schema_version table with string rows Three lock disciplines and two migration chains in one SQLite file; FK enforcement differs by writer Route conversation writes through ProjectDatabaseManager (or one shared provider); single version table M
U5-09 P3 sqlite_store.py:615-665; indexing.py:1294-1301 Per-stale-file delete loop: each delete_by_file constructs a fresh embedding provider via EmbeddingRegistry.create(_build_embedding_config()) where _build_embedding_config calls load_settings() per file; each delete its own transaction A 200-file branch switch = 200 settings loads + provider constructions + 200 transactions before parsing starts Hoist provider/settings resolution to once per run; batch deletes in one write_batch S
U5-10 P3 sqlite_store.py:382-397 vs 599-610 write_batch opens BEGIN IMMEDIATE, but _delete_by_repo_conn uses executescript, which implicitly COMMITs pending transactions A future caller wrapping delete_by_repo in write_batch silently degrades to autocommit; crash mid-batch leaves partial rebuild Replace executescript with individual conn.execute calls S
U5-11 P3 victor/core/database.py:795-905 vs 1736-1906, 385-485 vs 1020-1111, 1432-1547 DatabaseManager and ProjectDatabaseManager duplicate execute/query/query_one/transaction/table_exists/get_stats and near-identical legacy migration copy (~250 lines); write-batching API (~115 lines) has zero callers 2k-line file where twin drift is the churn driver (churn=35, fan_in=24); dead API misleads Shared _DatabaseManagerBase; delete the dead batching API S
U5-12 P3 sqlite_store.py:566-581 vs 432-470 _delete_by_file_conn builds unchunked IN (…) placeholder lists while _drop_dangling_edge_rows chunks at 900 >999/>32,766 symbols per file breaks incremental delete mid-run Reuse the 900-chunk helper S
U5-13 P3 sqlite_store.py:951-972 stats() exception handler references node_count/edge_count unbound if the first COUNT fails → UnboundLocalError masks the original error Robustness path that itself fails; omits indexed_files in the degraded shape Bind defaults before the try S
U5-14 P3 victor/storage/graph/incremental_indexing_simple.py; victor/core/schema.py:291-296 incremental_indexing_simple imported only by one integration test — superseded by the graph_rag pipeline + manifest contract; Schema.SYS_SCHEMA_VERSION table created but never read Parallel "simple" indexer rotting vs the real one Delete both; keep the test on the real pipeline S

Co-design opportunities

  1. Manifest-aware parse_repo (victor-codegraph × Victor indexer × ProximaDB SDK): accept a previous manifest, parse only added/changed, reuse prior ParsedCode — fixes U5-02 for all three consumers at the ADR-014 boundary.
  2. One staleness primitive on the protocol: get_staleness(files) → {file: (mtime, hash)} batched per backend — removes the N+1 and the getattr probing.
  3. Tier-A/B parity on SQLite: port cpg_fragments tiering; 1.10 M statement nodes leave graph_node, its 5 indexes and FTS (~700+ MB), aligning hot-graph size across backends.
  4. Derived-state lifecycle ownership: rebuild completion implies reclamation (auto incremental_vacuum + WAL truncate) instead of a manual CLI.
  5. Batched node-metadata persistence seam: protocol update_nodes_metadata(pairs) per backend; _generate_embeddings stops feature-probing; SQLite stops paying a transaction per node.
  6. One connection owner per DB file: conversation persistence through ProjectDatabaseManager so busy_timeout, WAL pragmas, FK enforcement, migrations are uniform.

Keep intact

  1. Legacy calls fallback (_build_calls_edges_legacy, indexing.py:2664; provider fallback 1680-1740) — explicitly load-bearing; only path with symbols when no enhanced TSA provider registers.
  2. Self-heal schema re-ensure at index start (indexing.py:460-470) — compensates for the intentionally non-atomic DROP/CREATE truncate; do not "fix" the truncate into a slow DELETE.
  3. Provider resolution before the parallel parse burst (indexing.py:481-490) — reordering reintroduces the documented race where 51/92 files silently lost IMPORTS edges.
  4. Edge endpoint integrity drop (_drop_dangling_edge_rows, sqlite_store.py:432-470) — keeps SQLite and ORION producing identical graphs.
  5. Deliberate busy_timeout=5000 divergence (database.py:1642-1646) and the separate undo.db — intentional drop-not-fail policy for non-critical writers.

Quick metrics

  • Live DB (mid-rebuild snapshot): graph_node 1,184,230 (statement 93%; function 23.8k; class 4.4k; module 1.8k), graph_edge 2,716,131, graph_file_mtime 3,864; FTS rows == node total (in sync).
  • project.db 2.64 GB; freelist 85,213 pages (350 MB) after rebuild reuse, was 424,453 (1.74 GB) during the delete phase; PRAGMA auto_vacuum = 0 in header.
  • dbstat live pages: graph_node 660 MB, graph_edge 655 MB, edge PK 169 MB, idx_graph_edge_file 155 MB, src/dst-type indexes 226 MB, FTS 109 MB (~2.1 GB live vs 2.64 GB file).
  • WAL 127 MB at rest → 281 MB during rebuild vs ~40 MB target — checkpoint starvation confirmed empirically.
  • Unused-in-practice tables: compaction_events 0 rows, compaction_history 0, graph_query_cache 0, context_sizes 4; messages 5,658 / sessions 41 (~11 MB) — conversation data is not the size driver; graph churn is.
  • Churn leaders: graph_rag/indexing.py 3,354 LOC (churn 47), database.py 2,006 (35), sqlite_store.py 1,509 (30), schema.py 1,420 (32); victor/state/ small (1,643 LOC) and clean; victor-codegraph 3,108 LOC / 14 modules, 3-consumer manifest contract (manifest.py:1-21).