stanleytheli/qwen3.6-35B-A3B-av-RL1200

qwen3.6-35B-A3B-av-1200

RL-tuned Activation Verbalizer (AV) for Qwen/Qwen3.6-35B-A3B — the SFT model qwen3.6-35B-A3B-av-sft after 1200 steps of GRPO against a frozen reconstruction reward. Companion reward/ reconstruction model (frozen throughout RL): qwen3.6-35B-A3B-ar-sft.

RL setup

• GRPO: group size 8, group-normalized advantages, single-update policy gradient, k3 KL penalty (β=0.02) against the SFT policy, temperature 1.0 rollouts
• Reward: cosine similarity between the frozen AR's reconstruction of the generated explanation and the original activation — i.e. the roundtrip metric itself
• LoRA r=32 (attention/GDN/shared-expert), merged into plain HF weights
• Prompts: held-out layer-29 activation vectors, injected at the marker token
• 1200 steps total (200 + 1000 continuation), batch 24 prompts × 8 samples/step

Held-out roundtrip (AV → text → frozen AR → vector, n=128)

checkpoint	FVE (norm)	cos mean / median	FVE (raw)
SFT (av-sft)	0.418	0.807 / 0.835	−0.26
GRPO 200	0.583	0.862 / 0.887	+0.17
GRPO 1200 (this)	0.644	0.882 / 0.907	+0.32

Usage notes

Same injection contract as the SFT model: nla_meta.yaml + injection_scale.txt included; scan for the marker token id at inference time (never precompute positions). See the NLA repo for injection utilities.