# Results — Capability vector for Qwen3.5-4B ## Headline — null result confirmed at N=25 Initial single trial suggested 0/2 → 1/2 lift on log-summary. After **five independent sweeps including a 4-config × N=5 α-grid (20 runs)**, the lone α=4 pass does not repeat in any configuration. Final accumulated table on `log-summary-date-ranges`: | config | N total | passes | pass rate | |------------------|--------:|-------:|----------:| | base | 11 | 0 | 0 % | | steered-L22-α2 | 1 | 0 | 0 % | | **steered-L22-α4**| 3 | **1** | 33 % | | steered-L22-α6 | 5 | 0 | 0 % | | steered-L22-α8 | 5 | 0 | 0 % | | steered-L26-α4 | 5 | 0 | 0 % | The single α=4 pass is consistent with a lucky-sample artifact (1 in 25 trials across all steered configurations). The α-grid sweep also revealed an over-steering regime: α ∈ {6, 8} on L22 and α=4 on L26 push the model into early "done" (the agent exits in 1–2 turns without attempting the task). The "behavioural region" where the steering vector stays usable is narrower than the mathematical region where the AUC=1.0 signal exists. | task | base passes | α=4 passes | α=2 passes | |-------------------------------|:-----------:|:----------:|:----------:| | log-summary-date-ranges | **0/6** | **1/3** | 0/1 | | modernize-scientific-stack | 2/4 (~50%) | 1/2 (~50%) | 1/1 | | constraints-scheduling | 0/1 | 0/1 | — | | multi-source-data-merger | 0/1 | 0/1 | — | | qemu-startup | 0/1 | 0/1 | — | Fisher's exact on `log-summary-date-ranges` (1/3 vs 0/6): **p ≈ 0.25 — not significant**. The single steered pass on log-summary in the very first trial does not replicate; with N=3 we lose the signal. For comparison, the trained references in the original sprint: | model | log-summary | modernize | net | |-----------|:-----------:|:---------:|:---:| | sft LoRA | PASS | PASS | 2/2 | | rift LoRA | FAIL | PASS | 1/2 | **Bottom line — the capability vector is mathematically clean (AUC=1.0 separation, 32 stable layers, model survives all α we tested) but it does not produce a statistically-detectable behavioural lift on terminal-bench-2 sprint at the trial counts we ran. The intervention is non-destructive but, empirically, not a free LoRA replacement.** Documenting as a null/inconclusive result to keep us honest. ### Why it might still be a real lift hidden by N - n_pos = 5 traces is small; direction may pick up "trace style" alongside "task-solving capability". - Tasks themselves are stochastic (modernize ~50% under base sampling), so a small signal disappears in trial-to-trial noise. To detect a 20-pp lift on log-summary at 80 % power you need ≈30–40 trials per arm. - Per-task α is probably needed: α=4 helps log-summary anecdotally, but the same α is irrelevant for qemu (which is far above the model's solving budget regardless of any single steering vector). ## Sweep details ### α=4 sweep — `results/minimal_20260514_162802/` ``` CONFIG PASS / N PARSE_FAIL AVG_TURNS TOTAL_S baseline 0 / 2 1 8.5 250 steered-L22-a4 1 / 2 4 3.5 382 ``` - baseline.log-summary: fail, 5 turns, 171 s — model finishes early without solving. - baseline.modernize: fail, 12 turns, 79 s — exhausted budget, never produced valid output. - steered.log-summary: **pass**, 5 turns, 222 s — slightly slower but solves the task. - steered.modernize: fail, 2 valid steps + 3 `parse_fail` turns. The α=4 push at layer 22 over-shoots: the model starts emitting raw Python code instead of wrapping it in the `{"analysis":..., "command":...}` JSON. **Format collapse from over-steering, not from incapability.** ### α=2 sweep — `results/alpha2_20260514_164159/` ``` CONFIG PASS / N PARSE_FAIL AVG_TURNS TOTAL_S baseline 1 / 2 3 10.0 963 steered-L22-a2 1 / 2 4 10.0 860 ``` - baseline.log-summary: fail (rc=124 timeout, 11 turns, 697 s) — hit docker time cap. - baseline.modernize: **pass**, 9 turns, 266 s — base Qwen stochastically solves this. - steered.log-summary: fail (rc=124 timeout, 11 turns, 574 s) — α=2 too weak to lift. - steered.modernize: **pass**, 9 turns, 286 s — α=2 leaves the JSON wrapper intact. The α=2 vs α=4 contrast confirms the *over-steering* hypothesis: the modernize fail at α=4 was format-collapse, not loss of underlying capability — the same prompt at α=2 solves it. ### α-grid sweep — `results/agrid_20260514_182050/` 5 reps × {baseline, steered-L22-α6, steered-L22-α8, steered-L26-α4} × log-summary. ``` CONFIG PASS / N PARSE_FAIL AVG_TURNS baseline 0 / 5 0 3.6 steered-L22-a6 0 / 5 0 5.4 steered-L22-a8 0 / 5 0 1.4 ← exits in 1-2 turns ("done"-spam) steered-L26-a4 0 / 5 0 3.8 ``` α=8 on L22 has avg_turns=1.4 → the model decides "I'm done" almost immediately when steered this hard. α=6 still engages (avg_turns=5.4) but still fails. Net 20 runs, 0 passes — pure null. ### Wide sweep — `results/wide_20260514_174112/` 5 sprint tasks × {baseline, steered-L22-α4} × 1 rep, max-turns=10, max-tokens=768. ``` CONFIG PASS / N PARSE_FAIL AVG_TURNS TOTAL_S baseline 1 / 5 14 4.2 1237 steered-L22-a4 1 / 5 11 4.0 953 ``` Both pass exactly modernize-scientific-stack and fail everything else. Net delta on this sweep: **0**. ### Multi-run accumulation on `log-summary-date-ranges` ``` run model outcome prior sglang sprint base Qwen3.5-4B FAIL α=4 minimal_eval baseline FAIL α=2 sweep baseline FAIL (timeout) repeat_eval rep1 baseline FAIL repeat_eval rep2 baseline FAIL wide_eval baseline FAIL 0/6 ← total base α=2 sweep steered-L22-α2 FAIL (timeout) α=4 minimal_eval steered-L22-α4 PASS (5 turns, 222 s, reward=1) repeat_eval rep1 steered-L22-α4 FAIL (4 turns) wide_eval steered-L22-α4 FAIL (1 turn) 1/3 ← total steered-α4 ``` 1/3 vs 0/6 → Fisher's exact p ≈ 0.25. The early single-run pass does not replicate. ## Direction discovery ``` L AUC margin sep_cos μ+ (σ) μ- (σ) 22 1.000 1.95 0.0156 -0.12(0.74) -2.07(1.21) <- chosen 19 1.000 1.44 0.0140 -0.41(0.47) -1.84(0.91) 16 1.000 0.69 0.0056 -0.99(0.17) -1.68(0.46) 15 1.000 0.64 0.0046 -1.00(0.16) -1.64(0.42) 14 1.000 0.56 0.0050 0.16(0.16) -0.40(0.35) 13 1.000 0.50 0.0036 -0.45(0.14) -0.95(0.31) 12 1.000 0.50 0.0040 -0.08(0.16) -0.57(0.29) 26 0.983 2.65 0.0147 -0.16(1.17) -2.81(1.48) ``` 5 pos × 12 neg traces. Layers 12–22 give perfect linear separation. Layer 22 chosen for max margin (1.95) among AUC=1.0 layers. See `vectors/ranking.csv`. ## Qualitative steering check `scripts/generate_steered.py` — 20 (layer, α) configurations on 2 terminus-style prompts. All produced coherent JSON-formatted agent output; no gibberish at α ∈ {1, 2, 4, 8} on layers {22, 19, 26}. Log: `results/qualitative.jsonl`. The α=4 → "code overflow" effect on modernize doesn't appear in the qualitative log because that prompt was short (200 max-tokens, 256 max). It only manifests on long generations where the model has room to commit to code output. ## Deviations from PLAN - **Eval scope shrunk** from 5 × 5 to 2 × 2 because the HF reference server is ~5× slower than sglang (max-tokens 1024, max-turns 12, ~5 min / task vs ~1 min on sglang in the prior sprint). One steered config (L22, α=4) and one baseline. The α sweep deferred — α=2 is the obvious next thing to try. - **sglang inference is not yet wired** for steering because every projection in Qwen3_5DecoderLayer has `bias=None`. To bake the steering into weights for sglang we have to *add* a bias parameter to `mlp.down_proj` (or `o_proj`) of layer 22. sglang's Qwen3_5 implementation needs to be checked for whether it picks up an added bias on load. Tracked as future work. - **Capture and direction computation went unchanged** from PLAN; bf16 was fine once sglang freed the GPU. ## Future work 1. **α=2 sweep** on the same 2 tasks — should solve the over-steering on modernize. 2. **Bake into weights for sglang**: - Add `nn.Parameter(α·dir_L=22, shape=(2560,))` as `mlp.down_proj.bias` of layer 22. - Save as `Qwen3.5-4B-capvec-L22-a{α}.safetensors`. - Verify sglang loads it (need to inspect sglang/srt/models/qwen3_5.py). 3. **Subtract refusal direction too**? — base Qwen3.5-4B refuses the agent role with weak prompts. NousResearch-style projection removal of the refusal axis would compose nicely with our additive capability vector. 4. **Larger trace pool** — n_pos=5 is small. Add more SFT-pass tbench-2 traces to tighten the direction. 5. **Per-task α** — single α may be over-strong for some tasks (modernize) and under-strong for others. CAA literature explores adaptive α. ## Files in this run ``` TASK.md RESEARCH.md PLAN.md RESULTS.md VERIFY.md traces_pos.jsonl traces_neg.jsonl activations/{pos,neg}/*.npz (17 traces × 32 layers × 2560 dim) vectors/dir.pt vectors/ranking.csv notes/{full_capture,compute_directions,qualitative,minimal_eval,serve_steered}.log results/qualitative.jsonl results/minimal_20260514_162802/{master,sweep}_summary.csv + per-task traces scripts/*.{py,sh} ```