cstat/archive/overcomplicated/roadmap.md

Roadmap: Asymptotically Optimal Codebase Simplifier

Notes May 19 26

• cstat mostly useful as code combing tool for an agent loop with extant verifiable behavior tests. Making a neruo-symbolic driver is not something well scoped enough, requires too much resources that you don't have. Instead just use the tool as-is, and find simple ways to add on to it as you find actual flaws when using it.

Framing

The end goal is a tool that takes an existing codebase and reduces its complexity along a vector of objective structural metrics, subject to the constraint that observable behavior is preserved (tests pass, conformance specs still hold). The tool should be useful with a frontier LLM acting as the agent, and more useful when paired with a non-LLM optimizer — ideally neuro-symbolic, possibly co-trained, but that architecture is a downstream decision, not an upstream one.

cstat (this repo) is the starting point for the oracle side. It already emits a deterministic JSON state describing modularity, complexity, maintainability, redundancy, dead code, coupling, cohesion, and a diagnostic list with locations. That is the seed of the state vector. The improvements doc (improvements-discussion.md) expands the seed to Halstead, instability/abstractness, information surface area, scale-free exponents, self-similarity, and symbol/IR metrics. The current code operates at AST level only; the future stack will be multi-level.

This document is a strategic roadmap. It defines milestones and ordering, not implementation. Every milestone before #7 is reversible.

---

Operating Principles

These are constraints we have already committed to. They shape every milestone below.

1. Objective metrics only. No calibration against human taste. The target is the structural vector itself. Refactors invisible to the vector (renames, doc improvements, better error types) are out of scope for the optimizer — they live on the LLM-only frontier and are not measured.

2. Multi-level representation. The oracle must be able to consume information at any IR level (source, AST, HIR/MIR, LLVM IR, symbol table, DWARF). Moves are tagged by the level(s) at which they operate. Some metrics and some moves are only visible/legal at specific levels.

3. Vector, not scalar. Scores stay decomposed. The current HealthScores triple is a human display; the optimizer operates on the raw component bundle. No weighted sums imposed upstream of the optimizer.

4. Pareto, not minimum. Metrics trade against each other (collapse classes → fewer objects, more rigidity). The optimizer searches the frontier; the operator weights. Degenerate minima (delete everything, one God-class) are prevented by this plus the behavioral constraint.

5. Tests are constraint, not objective. Behavior preservation gates acceptance of a move. Folding correctness into the objective produces reward hacking.

6. Minimize reliance on human refactor corpora inside the loop. LLMs already encode that prior. Reusing it as training data confounds the experiment we are actually running, which is: what does objective-metric-driven optimization of code structure look like, when the actor is not trying to imitate humans? Human-derived data is acceptable during bootstrap, not in the steady-state loop.

7. General actor interface. The system should accept any actor that maps (state, move_space) → ranked_moves: a frontier LLM, a small distilled model, a symbolic search, or a co-trained neural policy. The harness is the same.

---

Milestone 1 — Define the Representation Stack

Strategic role: Fixes the coordinate system. Every later milestone is parameterized by this choice. Cost of getting it wrong is highest because it propagates.

Decisions to make:

• Which IR levels are ingested in v1 vs. deferred. Source and AST are given (cstat already does this). MIR and LLVM IR require either rustc internals (unstable) or a stable bridge (e.g. cargo rustc -- --emit, llvm-tools-preview, or compiler-as-library via stable JSON dumps). Symbol table and DWARF are accessible via nm / objdump / gimli on debug builds.
• The oracle interface: every level contributes named scalars to a single flat state vector. Levels are independent ingestion modules; missing levels degrade the vector gracefully (NaN or omitted, not zero).
• The move-tagging convention: every move declares a set of representation levels it reads from (preconditions) and writes to (effects), even if the actual edit is source-text. This is what later lets a planner reason about which moves are cheap to predict vs. which require re-running the oracle.

Done when: A spec document and an empty-but-typed oracle interface exist. Concrete metric implementations can land later; the shape is fixed.

Foundational dependency: Nothing precedes this.

---

Milestone 2 — Inventory the Move Space, Per Level

Strategic role: Establishes what actually acts on the state vector. Determines whether a symbolic DSL is feasible or whether moves are too ad-hoc.

Approach: Discover, do not enumerate by intuition. Apply candidate moves on real targets, observe what the vector does, keep what produces predictable, well-typed deltas.

Per-level expectations (anticipated, to be verified):

• AST-level: extract function, inline function, merge functions, delete unreachable function, move function across modules, split module, replace nested match with early returns, parameterize a struct.
• HIR/MIR-level: dead basic-block elimination after lowering, collapse redundant match arms, simplify control flow that's only redundant post-desugar.
• LLVM IR / codegen-level: devirtualization opportunities, monomorph pressure (replace generic with dyn, or the reverse, based on source-to-symbol ratio).
• Cross-level: a single source edit (generic → dyn) can produce small AST delta and large symbol-table delta. These moves carry the most information per edit.

Move record format: name, level tags, preconditions (queryable from oracle), predicted effect on state vector (which components, predicted sign, predicted magnitude class), and confidence (deterministic vs. empirical).

Done when: A typed catalog of ~10–20 moves exists, with at least one empirical trajectory per move showing the vector delta it produces on a real target.

Depends on: Milestone 1.

---

Milestone 3 — Profile LLM Cleanup Behavior

Strategic role: Tells us where the LLM is reliable and where the eventual non-LLM optimizer must cover. Without this, the symbolic side is designed against an imagined LLM, not a real one.

Method: Controlled cleanup sessions on messy targets, varying scope:

• Single function (local complexity reduction).
• Single file (intra-module restructuring).
• Cross-module (move/split, dependency detangling).
• Multi-step coordinated (3+ moves where intermediate states look neutral or worse).

Recorded per session: moves proposed, moves applied cleanly, moves that broke build/tests, vector deltas, where the LLM plateaued or hallucinated. Multiple actors, multiple seeds, multiple targets.

Output: A capability profile — per move type and per scope, how reliably the LLM produces a clean, correct, vector-improving edit. This profile is what the optimizer in #7 is designed against (covers gaps, delegates strengths).

Depends on: Milestone 2 (so we know what moves to evaluate). Can run in parallel with #2 once #2 has an initial move set.

---

Milestone 4 — Design the Test Methodology

Strategic role: Gates everything downstream that requires measurement. Bad methodology produces unfalsifiable results.

Requirements:

• Reproducible. Same target, same starting code, same metric, same result.
• Non-human-judged. Success is defined by behavioral conformance, not taste.
• Behavioral spec is independent of structural metric. Tests evaluate what the code does, the vector evaluates how it is built. The two must not collapse into each other.
• Scope-graduated. Tiny targets (toy VM) for fast iteration; medium targets (Chip-8, NES subset, JSON parser, regex engine) for trajectory-shape evidence; larger targets (something approaching but below GBA emulator scope) noted as future once smaller scopes are reliable.

Methodology design produced now, before any large-scope target is attempted. Specifically: scoring protocol, trajectory definition (sequence of (state, move, new_state, behavior_test_result) tuples), cross-run comparison method, and what counts as a "completed" cleanup run.

Strategic note on the emulator idea: This is the eventual stress test. "Build XYZ from a spec" is reliably accomplished by current LLMs at sub-GBA scope. Pairing the cleanup tool against an LLM-built emulator gives us a target where (a) behavioral correctness is mechanically checkable via a conformance suite, (b) the initial code quality is representative of agent-generated mess, and (c) the scope is large enough that non-trivial restructuring matters. Note for future; the methodology design happens now, the large-target runs happen post-#6.

Depends on: Milestone 1 (state vector defined), #2 (moves defined for trajectory representation).

---

Milestone 5 — Design Data Collection

Strategic role: Captures the substrate that everything after #6 is built on. Insufficient logging is unrecoverable; over-logging is harmless.

Constraint: Minimize human-corpus reliance inside the loop (principle #6 above). Bootstrap from existing LLM behavior is acceptable; ongoing training data should come from the loop's own operation on the test targets from #4.

Logged per step:

• Full state vector before and after.
• Representation-level snapshots referenced by the oracle (cheaply: which IR levels contributed, what their summary scalars were).
• Proposed move (or moves, if multiple were ranked).
• Applied move's diff.
• Level tags on the applied move.
• Behavioral test result.
• Actor identity (which LLM, which prompt strategy, which seed) and optionally chain-of-thought / explanation when the actor produces one.
• Wall-clock and oracle-recompute cost.

Not logged: Pre-stored human refactor exemplars used as targets. We want trajectories shaped by the metric, not by mimicry of canonical human-authored diffs. (If a target happens to be a human codebase, fine — but the actor's training signal should be the oracle, not the human's historical edits.)

Depends on: Milestones 1 (vector shape) and 2 (move shape). Runs in parallel with #4.

---

Milestone 6 — Bootstrap Loop, End-to-End

Strategic role: Produces the empirical substrate for every subsequent design decision. Not intended to produce clean codebases at this stage — intended to produce trajectories.

Configuration:

• Actor: LLM (frontier, well-instructed).
• Oracle: full representation stack from #1.
• Move space: catalog from #2.
• Targets: graduated set from #4.
• Logging: per #5.

Runs: Multiple targets × multiple seeds × multiple prompt strategies. Both successful and failed trajectories are useful.

Outputs:

• A trajectory dataset.
• An empirical answer to "how often does a single move improve the vector vs. requires a coordinated sequence." This is the locality question that drives algorithm choice in #7.
• An empirical answer to "where does the LLM plateau, and which axes does it never touch." This shapes the optimizer's coverage requirements.
• Goodhart-resistance evidence: run at least one degenerate prompt ("minimize the vector, ignore behavior except to keep tests passing") and inventory the exploits found. This is methodology stress-testing, not a separate milestone.

Done when: Sufficient trajectory data exists to make the design choices in #7 evidence-based rather than guesswork.

Depends on: All of 1–5.

---

Milestone 7 — Optimizer Design Discussion

Strategic role: Commits to a mechanism. First non-reversible point in the roadmap.

Inputs from earlier milestones:

• Move catalog with level tags, preconditions, predicted effects (#2).
• LLM capability profile, with strengths and gaps (#3).
• Trajectory dataset showing locality / non-locality (#6).
• Goodhart exploit inventory (#6).

Decisions to make:

• Algorithm class. Candidates include greedy descent, beam search, simulated annealing on the Pareto frontier, MCTS over move sequences, neuro-symbolic policy (symbolic move enumeration + learned move-prior net). Choice is informed by the locality evidence from #6, not by intuition.
• Where the LLM sits in the final system: primary actor with optimizer as guardrail, primary optimizer with LLM as move-proposer for moves outside the catalog, or full delegation to a co-trained policy with LLM available only as fallback.
• Training regime if a policy net is included. The trajectory dataset from #6 is the seed; whether on-policy data collection continues in steady state is a sub-decision.

Done when: An architecture is selected and a v1 build is specified. Build itself is post-roadmap.

Depends on: All previous milestones.

---

Dependency Graph

            #1 (representation stack)
              │
        ┌─────┴─────┐
        ▼           ▼
       #2          #3
   (moves)    (LLM profile)
        │           │
        └─────┬─────┘
              ▼
        ┌─────┴─────┐
        ▼           ▼
       #4          #5
  (test method)  (data collection)
        │           │
        └─────┬─────┘
              ▼
              #6
       (bootstrap loop)
              │
              ▼
              #7
       (optimizer design)

#1 strictly first. #2 and #3 in parallel. #4 and #5 in parallel after #2/#3. #6 after all of 1–5. #7 after #6.

---

Parked / Future

These are noted because they belong on the long horizon but should not distract from the milestone sequence above.

• Co-training a neural policy with the symbolic optimizer. Possible only after #7's architecture is built and #6's data collection is in steady state. Treat as a v2 concern.
• Generalization across languages. cstat is Rust-only. The representation stack from #1 should not preclude other languages, but v1 is Rust.
• Temporal / git-history metrics. Deferred per improvements-discussion.md. May re-enter as a data source for the bootstrap phase only (per principle #6) if useful.
• Agent-vs-human style classification. Out of scope for the optimizer, in scope for cstat as a measurement tool.
• Emulator-class targets. Designed for in #4, executed against in a post-#6 follow-up. The methodology is built early; the large runs come later.
• Human-codebase Pareto comparison. Compare optimizer-cleaned codebases against the same codebases as cleaned by human maintainers, on the structural vector. Not for validation (we explicitly do not calibrate against human taste) but for characterization of where the two diverge.

---

Status

Document drafted; no milestones started. Milestone #1 is the next deliverable.