camilablank/qwen3.6-27b-pile-n25-skip4-penultimate-jacobians

Qwen3.6-27B — J-lens Jacobians (Pile, n=25, skip_first=4, penultimate block)

J-lens Jacobian stack for Qwen/Qwen3.6-27B, computed for global-workspace / DRUMMER readout experiments (MATS, Neel Nanda stream).

What this is

A stack of per-block input→output Jacobians (the "J-lens") estimated over a background corpus, used to read out which vocabulary tokens a given probe prompt surfaces at each layer.

Artifact

Field	Value
File	jacobians.safetensors (tensor key: jacobians)
Shape	(63, 5120, 5120) — (block, d_model_out, d_model_in)
dtype	bfloat16
Model	Qwen/Qwen3.6-27B
Background corpus	NeelNanda/pile-10k
n_prompts	25
n_positions	3100 (summed source positions, ÷ for the mean)
target_block	62 (penultimate block)
skip_first	4 (first 4 source positions dropped as early-context artifacts)
t_max	128

Recipe is the paper-faithful target_block=-2, skip_first=4 (penultimate block; first source positions skipped). The same metadata is embedded in the safetensors header (__metadata__).

bf16 vs fp32

Stored in bf16. The readout metric is rank-based, and an fp32 A/B of the same Jacobian leaves the tracked-token ranks essentially unchanged (top meta-token rank-identical; others within ~1%), so bf16 halves the file at no cost to the readout.

Load

from safetensors import safe_open

with safe_open("jacobians.safetensors", framework="pt") as f:
    print(dict(f.metadata()))
    J = f.get_tensor("jacobians")  # (63, 5120, 5120) bf16