veri-safe/SWD-Qwen2.5-3B

SWD Checkpoints for Qwen2.5-3B

This repository contains the factor-only Sparse Weight Decomposition (SWD) checkpoints used for the Qwen2.5-3B layer 18 mlp.down_proj experiments. It does not redistribute the Qwen2.5-3B base model. Load the base model first, then apply one checkpoint with swd_loader.py.

Included Checkpoints

Replacement	Setting	Data used	CE delta vs dense
Layer 18 mlp.down_proj	s=0.5	2,048 tokens	0.000733
Layer 18 mlp.down_proj	s=0.75	1,048,576 tokens	0.000611

Each replaced matrix is represented as

output = input @ read @ write + bias

The intermediate coordinates are the SWD bottleneck units used for circuit scoring and ablation. s=0.5 and s=0.75 mean that 50% and 75% of all entries across the two factors are zero, respectively.

Usage

pip install torch safetensors transformers huggingface_hub

from transformers import AutoModelForCausalLM
from swd_loader import apply_swd_checkpoint

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-3B")
apply_swd_checkpoint(
    model,
    "checkpoints/qwen2.5-3b/layer18-down-proj/s0p5-tokens2048",
    mode="factorized",
)

Use mode="folded" to write read @ write into the original dense module for conventional inference. Every checkpoint directory contains model.safetensors, config.json, and provenance.json.

Links

• Main SWD checkpoints: https://huggingface.co/veri-safe/SWD
• Code: https://github.com/Veri-Safe/SWD

License

The Qwen2.5-3B-derived factor checkpoints are distributed under the Qwen Research License and are limited to non-commercial research and evaluation. See LICENSE and NOTICE. The SWD loader code is available under Apache-2.0; see LICENSES/APACHE-2.0.txt.