Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: gemma
|
| 3 |
+
base_model: google/gemma-2-2b
|
| 4 |
+
library_name: peft
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
# sae-icm final checkpoint (gemma-2-2b, lambda = 1e-05)
|
| 8 |
+
|
| 9 |
+
Stage-2 artifacts for the paper *Towards Isolated Interventions via Almost
|
| 10 |
+
Orthogonal Features in Language Models* (arXiv:2602.04718): the LoRA adapter
|
| 11 |
+
trained around a fixed sparse autoencoder, plus the fine-tuned SAE state.
|
| 12 |
+
|
| 13 |
+
- base model: `google/gemma-2-2b`
|
| 14 |
+
- orthogonality penalty lambda: `1e-05`
|
| 15 |
+
- SAE: TopK (K = 20), d_sae = 65536, spliced into the residual stream after block 12 (0-indexed)
|
| 16 |
+
- files: `adapter_model.safetensors`, `adapter_config.json`, `sae_state.safetensors` (keys `W_enc.weight`, `W_enc.bias`, `W_dec.weight`, `W_dec.bias`)
|
| 17 |
+
|
| 18 |
+
Load with `src/poet/load_hub.py` from https://github.com/mrtzmllr/sae-icm.
|