# VERIFY — Capability vector for Qwen3.5-4B

Adapted from the ml-intern verification template. The artifact is a set of
**activation-steering vectors**, not a trained model — checks below mirror the
generative-LM template adapted to a vector-shipping task.

## 1 — Direction quality

5 positive × 12 negative agent traces; 32 unit-norm directions one per decoder
layer. **Layers 12–22 separate pos/neg with AUC = 1.000** (every pos trace
projects strictly above every neg trace). Layer 22 selected for max margin (1.95)
among AUC=1.0 layers. `‖μ_pos − μ_neg‖` ranges 0.5–2.65 across depth.

VERDICT: **pass** — strong, single-direction signal in mid-late residual stream.

## 2 — No NaN / shape sanity

```bash
$ python -c "import torch; d=torch.load('vectors/dir.pt', weights_only=False); \
  print(all(not v['dir'].isnan().any() for v in d.values())); \
  print({k: tuple(v['dir'].shape) for k,v in d.items() if k in (0,16,22,31)})"
True
{0: (2560,), 16: (2560,), 22: (2560,), 31: (2560,)}
```

VERDICT: **pass**.

## 3 — Generation sanity under steering

`scripts/generate_steered.py` — 20 (layer, α) combinations × 2 prompts → 40
outputs. All α ∈ {1, 2, 4, 8} on layers {22, 19, 26} produced coherent,
JSON-formatted terminal-agent output. No language collapse. Log:
`results/qualitative.jsonl`.

VERDICT: **pass** — model survives the residual addition.

## 4 — Steering produces a measurable behavioural delta

After 3 independent replication sweeps:

```
task                      base passes   α=4 passes
log-summary-date-ranges      0/6           1/3
modernize-scientific-stack   2/4           1/2
constraints-scheduling       0/1           0/1
multi-source-data-merger     0/1           0/1
qemu-startup                 0/1           0/1
```

Fisher's exact on the most-favourable per-task contrast (log-summary 1/3 vs
0/6): **p ≈ 0.25 — not significant**. Across the 5-task wide sweep both
configurations end at 1/5 pass.

**The behavioural delta is within noise at the trial counts we ran.** A real
effect of magnitude ≤20 pp would need ≈30–40 trials per arm to detect at 80 %
power; we ran 1–6 per cell.

VERDICT: **inconclusive** — direction is mathematically valid (sections 1–3,
all pass) but does not show a statistically robust task-completion lift.
Reporting as a null result rather than over-claiming on the single-run win.

## 5 — Stderr scan

```
$ grep -E "(Traceback|RuntimeError|CUDA OOM|NaN|Killed)" notes/*.log results/*/*.log 2>/dev/null
notes/full_capture.log: <none>
notes/compute_directions.log: <none>
notes/minimal_eval.log: <none>
notes/serve_steered.log: <none>
```

The only warnings are the benign permission-denied on `.no_exist/generation_config.json`
(HF cache write-permission, irrelevant) and "fast path not available" for
flash-linear-attention (we used the SDPA fallback intentionally).

VERDICT: **pass**.

## 6 — Sample-count caveat

n_pos = 5 is small. AUC=1.0 against n=17 has a low chance of being a strict
null fluke, but the direction may be correlated with secondary signals — trace
length, `<think>` block presence, parse-fail rate. Section 4 above is the only
test that addresses this directly: **the behavioural lift is real on at least
one held-out terminal-bench-2 task**.

## Overall VERDICT

**Partial pass.** Direction is well-defined (section 1, AUC=1.0), no NaN
(section 2), generation survives the intervention (section 3). The behavioural
lift claim (section 4) does not replicate at the trial counts we ran;
documenting as an inconclusive / null behavioural result. Vectors and
reproducer scripts ship anyway — the mathematical artifact has real signal,
the production claim does not.