# Research — capability steering for agent LLMs

## Source papers / repos

- **NousResearch/llm-abliteration** — https://github.com/NousResearch/llm-abliteration
  - Pipeline: `measure.py` (compute per-layer direction) → `analyze.py` (rank
    layers) → `sharded_ablate.py` (apply to weights, YAML-configured).
  - Direction = `mean(harmful) − mean(harmless)`, L2-normalized.
  - Operates on residual activations (post-layer, equiv. of `resid_post`).
  - Supports norm-preserving biprojected ablation (--normpreserve).
- **failspy/llama-3-70B-Instruct-abliterated/ortho_cookbook.ipynb**
  - Contrast sets: AdvBench harmful (~520) × Alpaca harmless filtered (~6k → equalized),
    80/20 split.
  - Hooks: `resid_pre`, `resid_mid`, `resid_post` × every intermediate layer
    (`range(1, n_layers)`), pos = **last token** of the prompt.
  - Formula: `d_l = (harmful_mean_l − harmless_mean_l).normalize()`.
  - Selection: brute-force eval top-K by `|mean(d)|`, **greedy decode**, manual
    visual grade of completions. Apply hook to **all 3 act types × all layers**.
  - Inference hook = orthogonal projection removal: `a' = a − (a·r̂)r̂`.
  - Notebook is inference-only; sharded_ablate.py does weight-time orthogonalization
    of W_proj / W_down (we don't need it — we want to *add*, not subtract).
- **Arditi et al. "Refusal in LLMs is mediated by a single direction"** (arXiv 2406.11717)
  - Picks a single late-middle layer (~40–60 % depth).
- **Panickssery et al. "Steering Llama 2 via Contrastive Activation Addition" (CAA)**
  - Per-prompt **pairs** (same prompt, contrasting completions A vs B), residual at
    end-of-answer token, average difference = steering vector.
  - Add `α · v` to residual at chosen layer during generation. Greedy or sampled.
  - For Llama-2-7B (32 layers), best layer typically 13–16 (~40–50 % depth).
- **CAA / activation-addition family** is the closest framing to *our* task:
  same prompt, contrasting completions, signed addition (not orthogonal removal).

## Mapping to our problem

| concept in literature   | our analogue                                            |
|-------------------------|----------------------------------------------------------|
| harmful prompt set      | failed agent rollouts (DPO format-broken, base fail)     |
| harmless prompt set     | SFT-success agent rollouts                              |
| pos = last token        | pos = mean over assistant-raw token positions per turn   |
| direction               | capability+ direction (we ADD it, scale α > 0)           |
| layer selection         | brute-force α-sweep across {α: 5 values} × {best 5 layers} on the 5-task eval |
| application             | inference-time pre-forward hook on `model.layers[L]`     |

## Open differences (we'll deviate)

1. **Same prompt available** for sprint tasks (5 tasks × 5 models all hit same tbench env).
   This lets us compute **paired** vectors per task → average → much cleaner signal
   than unpaired AdvBench/Alpaca. We'll use this CAA-style pairing for the sprint set,
   plus an unpaired pool from full tbench-2 (different tasks per bucket).
2. **Run the SFT-produced text through the BASE model.**
   The steering vector lives in the base model's residual space, so to push *base*
   toward *SFT-like* behavior we want base-model representations of SFT outputs vs
   base-model representations of failed outputs. (Running the SFT LoRA itself would
   give us SFT's residual space, which is what we want to *target* but not what we
   inject *into*.)
3. **Hook only at `transformers.LlamaDecoderLayer.output`** = HF native `resid_post`.
   The other two activation types (resid_pre/mid) require monkey-patching the block
   internals; resid_post alone is the canonical CAA choice and is enough.

## Risks

- **Confound: identity of generator.** SFT outputs are longer / more structured than
  DPO failures. The capability direction may just be "well-formed JSON" direction,
  not "solve the task" direction. **Mitigation**: include format-correct-but-wrong
  rollouts (base Qwen failing kv-store-grpc *with* good format) in the negative pool.
- **GPU contention.** sglang is using 40 GB. Need user OK before paus​ing it.
- **Token-budget**: capturing 36 layers × 2560 dims × per-token for thousands of tokens
  is gigabytes. Solution: aggregate to per-trace mean on the fly, don't store per-token.