# PLAN — Capability vector pipeline for Qwen3.5-4B

## Stage A — Trace harvesting (CPU-only, fast)

`scripts/collect_traces.py`
- Inputs:
  - `--positive`: list of (run_dir, task, model_label) where reward.txt == 1
  - `--negative`: list of (run_dir, task, model_label) where reward.txt == 0
                  OR where trace contains `"type": "parse_fail"`
- Walk each task dir: read `trace.jsonl`, extract `assistant_raw` records in order,
  drop the meta header. Stitch into a list of `{role: "assistant", content: ...}`
  alternating with the user / shell_result turns reconstructed from the trace.
- Re-build the **chat prompt the SFT model originally saw** + the **completion it
  produced**, using Qwen3.5-4B chat template.
- Output: `traces_pos.jsonl`, `traces_neg.jsonl` — each line:
  ```
  {"task": str, "model": str, "messages": [...], "asst_token_spans": [(start, end), ...]}
  ```
  where `asst_token_spans` are character-level offsets of every `assistant_raw`
  block inside the rendered prompt.

Hard-coded source buckets (verified to exist):

POSITIVES (5 traces):
- ~/runs/.../sft_full_tbench/20260514_122545/{cobol-modernization,git-leak-recovery,sqlite-with-gcov}
- ~/runs/.../qwen_sglang_eval/20260514_103006/sft-results/{log-summary-date-ranges,modernize-scientific-stack}

NEGATIVES (15 traces):
- qwen_sglang_eval/20260514_103006/{Qwen_Qwen3.5-4B-results,cp600-results,dpo-results}/*  (15 task-runs total, all reward=0)

Exclude sqlite-with-gcov from base-pass (would contaminate negatives).
Final: 5 pos vs ~15 neg.

## Stage B — Activation capture (needs GPU)

`scripts/capture_activations.py`
- Load `Qwen/Qwen3.5-4B` bf16 on cuda.
- For each trace:
  1. Tokenize full prompt+completion.
  2. Find token indices that fall inside any `asst_token_spans`.
  3. Forward in eval mode, `output_hidden_states=True`.
     `hidden_states` is a tuple of (n_layers+1) tensors, shape [1, T, D].
  4. For each layer L (1..n_layers), index `hidden_states[L][:, asst_tok_idx, :]`,
     mean over those positions, → vector shape [D].
  5. Save to disk: `activations/{bucket}/{task}__{model}.npz` containing array
     `act` of shape [n_layers, D].
- Memory bound: hidden_states[L] for T=4096 is 4096·2560·2 bytes = ~21 MB per layer
  × 37 layers ≈ 0.8 GB — fits. If a trace exceeds context (8k), truncate from the left
  keeping the **last** assistant block (model state most relevant at the end of trace).
- Throughput: ~5 s/trace on A6000 → 20 traces ≈ 2 min.

## Stage C — Direction computation

`scripts/compute_directions.py`
- Load all `.npz`, stack to `pos: [n_pos, n_layers, D]` and `neg: [n_neg, n_layers, D]`.
- Per layer L:
  - `mu_pos = pos[:, L].mean(0)`; `mu_neg = neg[:, L].mean(0)`
  - `dir_L = (mu_pos - mu_neg) / ||mu_pos - mu_neg||`
  - score: cosine separation `1 - cos(mu_pos, mu_neg)` (higher = better),
    and projection separation `(dir_L · mu_pos − dir_L · mu_neg)` (margin).
  - sanity: report `pca_explained_variance_ratio[0]` of `pos - neg` to verify the
    signal is concentrated in one direction (capability is "single direction" — yes,
    we are testing this hypothesis).
- Save `vectors/dir.pt`: dict with
  `{layer_idx: {"dir": tensor[D], "mu_pos": tensor[D], "mu_neg": tensor[D], "score": float, "margin": float}}`.
- Print a table of (layer, score, margin) sorted by score and save to `vectors/ranking.csv`.

## Stage D — Steered inference + sweep

`scripts/serve_steered.py`
- A thin wrapper that loads base Qwen3.5-4B + registers a `pre_forward_hook` on
  `model.model.layers[L]` that does `hidden_state[..., :] += α · dir_L`.
  Hook is toggleable (env var `STEER_LAYER`, `STEER_ALPHA`).
- Reuses the existing serve_model_v2.py pattern from `~/runs/.../serve_model_v2.py`
  if available — we'll copy and patch.

`scripts/sweep_steering.sh`
- For (layer, α) in top-5-layers × {0.5, 1, 2, 4, 8}, plus the (no-op) baseline:
  - Start `serve_steered` on port 8001.
  - Run terminus_runner over the 5 sprint tasks.
  - Collect reward + parse_fail rate.
  - Stop the server.
- Output: `results/sweep.csv` with (layer, alpha, n_pass, n_parse_fail, mean_turns).

## Stage E — VERIFY + ship

`scripts/verify.py`
- Pass conditions:
  1. At least one (layer, α) ≠ baseline strictly improves either pass-rate
     or parse-fail rate.
  2. Generation samples from steered model on a held-out prompt are
     still coherent English (not gibberish — steering can wreck the model).
  3. No NaN in vectors.
- Write `VERIFY.md`.

`scripts/hf_push.sh` (use the skill helper)
- Push to `AlexWortega/qwen3.5-4b-capability-vector-{date}`:
  - `vectors/dir.pt`, `vectors/ranking.csv`
  - all `scripts/*.py`
  - `TASK.md` `RESEARCH.md` `PLAN.md` `RESULTS.md` `VERIFY.md`
  - `results/sweep.csv`
  - `README.md` model card

## Stop / blocker conditions

- GPU not free → write BLOCKER.md, ask user.
- Doom-loop guard: if eval crashes 3× same way, stop.
- If after sweep nothing beats baseline → still ship the vectors as a **dataset**
  repo (negative result is valuable), don't pretend it worked.