agu18dec/qwen3.6-27b-pile-n10-skip4-penultimate-jacobians

Qwen3.6-27B — J-lens Jacobian (pile, n=10, skip4, penultimate)

Averaged Jacobian J_ℓ for the J-lens readout on Qwen/Qwen3.6-27B. For each layer ℓ, J_ℓ is the mean (over a pretraining corpus) of ∂(penultimate-block residual)/∂(hₗ), transporting any layer's residual stream into penultimate-layer geometry so the logit lens can be read off it: lens(hₗ) = softmax(W_U · norm(J_ℓ · hₗ)).

Spec / provenance (from the safetensors metadata)

field	value
model_id	Qwen/Qwen3.6-27B
dataset_id	NeelNanda/pile-10k
target_block	62 (penultimate residual; CLI target_block=-2)
skip_first	4 (early-context source positions dropped)
t_max	128
n_prompts	10
docs_consumed	11 (1 doc shorter than t_max was skipped)
n_positions	1240 (= n_prompts × (t_max − skip_first); the two-stage-mean divisor)
git_commit	eb5e1e47adea16f8599ca83da6c27cd0c2b694e2
dtype	bfloat16
tensor	jacobians [63, 5120, 5120] (layers 0..62, d_model 5120)

How it was built

One forward pass per prompt (bf16, SDPA) capturing every layer's residual stream, then reverse-mode autograd to the penultimate block (d_model=5120 cotangents, batched in chunks of 64). Rows are summed over the 124 kept source positions per prompt and across the 10 prompts, then divided by n_positions (the paper's two-stage mean). By construction J[62] = I (identity anchor; verified at save time).

Use

from global_workspace.jlens.jacobian import load_jacobians
jacobians, meta = load_jacobians("jacobians.safetensors")
# read with global_workspace.jlens.lens.jacobian_lens_with_fallback(resid, jacobians, W_U, final_norm)

Research artifact; no warranty. Built with the global-workspace J-lens pipeline.