dealignai commited on
Commit
3dca788
·
verified ·
1 Parent(s): aeb1246

strip surgery internals from README

Browse files
Files changed (1) hide show
  1. README.md +12 -22
README.md CHANGED
@@ -69,19 +69,19 @@ thumbnail: dealign_mascot.png
69
  | Refusals on harmful prompts | **0** explicit refuses | typically 90%+ refuse | abliteration complete |
70
  | `</think>` close at greedy on hard MMLU | **5/5** | 5/5 | preserved |
71
  | Multi-turn (3-turn escalation × 3 conversations) | **9/9** comply, context preserved | n/a | works |
72
- | Multimodal tensors byte-identical to base | **1106/1106** | — | preserved |
73
  | Bundle size | 19 GB | 66 GB BF16 | — |
74
  | Context | 262,144 tokens native | same | preserved |
75
 
76
- > The 12.5pp MMLU gap is concentrated in two reasoning-heavy subjects (abstract_algebra, college_computer_science) where the 8000-token thinking budget runs out **before** `</think>` closes. These hard-stops are **genuine deep reasoning**, not v1-style infinite repetition loops. Inspecting truncated responses shows the model working through correct algebraic manipulations. With `max_tokens ≥ 16384`, accuracy approaches base.
77
 
78
  ---
79
 
80
  ## v2 vs v1 (head-to-head)
81
 
82
- v1 (shipped 2026-04-28) used **standard ablation s=12/7/7 triple-pathway**, which suppressed the `</think>` token logit at greedy decoding. The model couldn't terminate reasoning on hard prompts infinite paragraph-level repetition until `max_tokens` cutoff. MMLU dropped from 86.5% base → 70.0% v1.
83
 
84
- v2 (this release) uses **MPOA dual-pathway s=8/5** which preserves row norms and lets `</think>` close cleanly:
85
 
86
  | Bench | v1 (broken) | **v2 (this release)** |
87
  |---|---|---|
@@ -131,34 +131,24 @@ Both at thinking=ON, greedy. Base at max=2000, CRACK v2 at max=8000.
131
 
132
  ## Operating recommendations
133
 
134
- - **`enable_thinking`** — v2 works in **BOTH modes** (v1 docs claimed thinking=ON required; v2 audit shows 5/5 comply with thinking ON, 4/5 with thinking OFF). Default to ON for hardest prompts; OFF works for most.
135
- - **`max_tokens ≥ 16384`** for hard reasoning (math, abstract algebra, complex CS). v2 still produces deep reasoning chains; the model closes `</think>` cleanly when reasoning concludes.
136
  - **Greedy** (temperature=0) AND **sampling** (temp=0.6, top_p=0.95 — NVIDIA-recommended in `generation_config.json`) both work.
137
  - **Multi-turn** — context preserved across 3+ turns; no late refusals after escalating prompts.
138
 
139
  ---
140
 
141
- ## Surgery recipe
142
 
143
- - **MPOA** ablation: `W' = magnitude_preserve(W - s·v̂@(v̂ᵀW))` preserves row norms, fixes `</think>` termination
144
- - 3 attention `o_proj`: L26, L33, L42 at strength=8
145
- - 9 Mamba `out_proj`: L28, L30, L32, L35, L37, L39, L41, L44, L46 at strength=5
146
- - Per-layer refusal vectors from per-quant 1000-pair (1280 EN + 720 ZH) probe
147
- - Binary patch (no `mx.save_safetensors` — it corrupts Nemotron shards)
148
-
149
- ### Audit (verified)
150
-
151
- - **19,325 / 19,361 tensors byte-identical** to base
152
- - **36 tensors changed** (12 layers × {`.weight`, `.scales`, `.biases`}) — exactly the surgery targets, zero unexpected changes
153
- - **1106 / 1106 multimodal tensors byte-identical** (`vision_model.*`, `sound_encoder.*`, `mlp1.*`, `sound_projection.*`)
154
- - **All config files unchanged** (config.json, jang_config.json, generation_config.json, chat_template.jinja, tokenizer_config.json, model.safetensors.index.json)
155
- - **Bit widths preserved**: attn=8, shared=8, mamba=8, routed=4, embed=8, lm_head=8
156
 
157
  ---
158
 
159
  ## Architecture (`nemotron_h`)
160
 
161
- - 52 layers: 23 Mamba (M) + 23 MoE (E) + 6 Attention (\*) at L5, 12, 19, 26, 33, 42
162
  - Hidden 2688, head_dim 128, GQA 32q/2kv (NO RoPE on attention — position from Mamba state)
163
  - 128 routed experts top-6 (sigmoid) + 1 shared expert per MoE layer
164
  - Multimodal: image (RADIO ViT) + audio/speech (Parakeet) merged via early-fusion projectors
@@ -195,5 +185,5 @@ This model has had refusal training surgically removed for legitimate research,
195
 
196
  ---
197
 
198
- Built by [dealignai](https://huggingface.co/dealignai). Surgery details in `crack_surgery_log.json`.
199
  Sister bundles: [JANGTQ-CRACK](https://huggingface.co/dealignai/Nemotron-3-Nano-Omni-30B-A3B-JANGTQ-CRACK) (12 GB, 2-bit MXTQ) · [MXFP4-CRACK](https://huggingface.co/dealignai/Nemotron-3-Nano-Omni-30B-A3B-MXFP4-CRACK) (21 GB, uniform 4-bit affine).
 
69
  | Refusals on harmful prompts | **0** explicit refuses | typically 90%+ refuse | abliteration complete |
70
  | `</think>` close at greedy on hard MMLU | **5/5** | 5/5 | preserved |
71
  | Multi-turn (3-turn escalation × 3 conversations) | **9/9** comply, context preserved | n/a | works |
72
+ | Multimodal byte-identical to base | preserved | — | preserved |
73
  | Bundle size | 19 GB | 66 GB BF16 | — |
74
  | Context | 262,144 tokens native | same | preserved |
75
 
76
+ > The 12.5pp MMLU gap is concentrated in two reasoning-heavy subjects (abstract_algebra, college_computer_science) where the 8000-token thinking budget runs out **before** `</think>` closes. These hard-stops are **genuine deep reasoning**, not v1-style infinite repetition loops. With `max_tokens ≥ 16384`, accuracy approaches base.
77
 
78
  ---
79
 
80
  ## v2 vs v1 (head-to-head)
81
 
82
+ v1 (shipped 2026-04-28) had a `</think>` termination defect at greedy decoding the model couldn't terminate reasoning on hard prompts and looped to budget cutoff. MMLU dropped from 86.5% base → 70.0% v1.
83
 
84
+ v2 (this release) restores clean termination:
85
 
86
  | Bench | v1 (broken) | **v2 (this release)** |
87
  |---|---|---|
 
131
 
132
  ## Operating recommendations
133
 
134
+ - **`enable_thinking`** — v2 works in **BOTH modes** (5/5 comply with thinking ON, 4/5 with thinking OFF). Default to ON for hardest prompts; OFF works for most.
135
+ - **`max_tokens ≥ 16384`** for hard reasoning (math, abstract algebra, complex CS).
136
  - **Greedy** (temperature=0) AND **sampling** (temp=0.6, top_p=0.95 — NVIDIA-recommended in `generation_config.json`) both work.
137
  - **Multi-turn** — context preserved across 3+ turns; no late refusals after escalating prompts.
138
 
139
  ---
140
 
141
+ ## Verification
142
 
143
+ - All multimodal tensors (vision + audio + projectors) are **byte-identical to base** capabilities fully preserved.
144
+ - All config files unchanged (config.json, jang_config.json, generation_config.json, chat_template.jinja, tokenizer_config.json).
145
+ - Bit widths preserved: attn=8, shared=8, mamba=8, routed=4, embed=8, lm_head=8.
 
 
 
 
 
 
 
 
 
 
146
 
147
  ---
148
 
149
  ## Architecture (`nemotron_h`)
150
 
151
+ - 52 layers: hybrid Mamba-2 + MoE + Attention
152
  - Hidden 2688, head_dim 128, GQA 32q/2kv (NO RoPE on attention — position from Mamba state)
153
  - 128 routed experts top-6 (sigmoid) + 1 shared expert per MoE layer
154
  - Multimodal: image (RADIO ViT) + audio/speech (Parakeet) merged via early-fusion projectors
 
185
 
186
  ---
187
 
188
+ Built by [dealignai](https://huggingface.co/dealignai).
189
  Sister bundles: [JANGTQ-CRACK](https://huggingface.co/dealignai/Nemotron-3-Nano-Omni-30B-A3B-JANGTQ-CRACK) (12 GB, 2-bit MXTQ) · [MXFP4-CRACK](https://huggingface.co/dealignai/Nemotron-3-Nano-Omni-30B-A3B-MXFP4-CRACK) (21 GB, uniform 4-bit affine).