---
license: mit
library_name: gguf
tags:
  - deepseek-v4
  - deepseek-v4-flash-0731
  - ds4
  - abliterated
  - heretic
  - gguf
  - iq2_xxs
  - quantization
  - speculative-decoding
  - dspark
base_model:
  - deepseek-ai/DeepSeek-V4-Flash-0731
  - squanchyzx/DeepSeek-V4-Flash-0731-HERETIC-Abliterated-FP8
---

# Splicing HERETIC-abliterated attention into antirez's IQ2XXS ds4 quant

**No weights here — this is the recipe, the verification harness, and the measurements.**

Making an abliterated 2-bit DeepSeek-V4-Flash-0731 for the [ds4](https://github.com/antirez/ds4)
engine normally means downloading 167 GB of FP8 and re-quantizing the whole 305B checkpoint. This
repo documents how to get the same result with **1.2 GB of network transfer and ~4 minutes of CPU**,
why that is bit-exact rather than approximate, and what it actually costs at inference time.

Hardware: a single ASUS Ascent GX10 (GB10, 121 GB unified memory, CUDA 13, sm_121a).

## The lever

Two edits that would normally collide turn out to be disjoint:

| | what it touches |
|---|---|
| antirez's `IQ2XXS-w2Q2K-**AProjQ8**-SExpQ8-OutQ8` recipe | routed experts get the imatrix; **attention projections are Q8_0, quantized with no imatrix at all** |
| [squanchyzx's HERETIC abliteration](https://huggingface.co/squanchyzx/DeepSeek-V4-Flash-0731-HERETIC-Abliterated-FP8) | 36 tensors only — `layers.{10..42}.attn.wo_b` + `mtp.{0,1,2}.attn.wo_b`; **no expert weight edited** |

So the abliterated tensors are exactly the ones the imatrix never sees. You can overwrite them
inside the finished GGUF and keep antirez's experts — including the 202k-chunk imatrix he measured
on the 0731 weights and never published. A full rebuild would instead re-quantize the experts with
the weaker public `1p5m` imatrix, which is a real quality regression on the part of the model that
carries most of the parameters.

Q8_0 has fixed shape and block size, so each patched tensor occupies **the same byte range**: the
GGUF header, tensor order, offsets and total size are untouched, and the file stays loadable.

## Recipe

```bash
# 1. 1.22 GB: the overlay + the rewritten index. The 48 official shards are NOT needed
#    (they are byte-identical to deepseek-ai/DeepSeek-V4-Flash-0731; only 72 index keys are
#    redirected to the overlay — verified).
hf download squanchyzx/DeepSeek-V4-Flash-0731-HERETIC-Abliterated-FP8 \
  model-overlay-00001-of-00001.safetensors model.safetensors.index.json \
  --local-dir ~/models/ds4-0731-heretic/hf

# 2. copy the finished quant (nocache keeps a live engine's page cache intact)
ionice -c3 dd if=DeepSeek-V4-Flash-IQ2XXS-...-imatrix-0731.gguf of=OUT-HERETIC-attn.gguf \
  bs=8M iflag=nocache oflag=nocache conv=fsync

# 3. splice the 33 backbone attention tensors (1122 MiB rewritten, ~4 min)
python3 tools/splice-heretic-attn.py --hf ~/models/ds4-0731-heretic/hf --gguf OUT-HERETIC-attn.gguf

# 4. prove nothing else moved
python3 tools/verify-heretic-splice.py --official ORIGINAL.gguf --patched OUT-HERETIC-attn.gguf
```

`splice-heretic-attn.py` reimplements, in numpy, exactly what `gguf-tools/deepseek4-quantize.c`
does for these tensors: FP8 `e4m3` × `e8m0` 128×128 block dequant → f32 → Q8_0 (block 32,
`d = amax/127`, f16 `d` stored, `qs = roundf(x * 1/d)` with `d` kept in f32). It refuses to patch a
GGUF that any running process has open or mapped.

The same tool patches the DSpark drafter: its MTP stages are `dspark.{0,1,2}.attn_output_b.weight`,
also Q8_0 `[8192, 4096]`, and the mapping `layers.N→blk.N` / `mtp.K→dspark.K` self-selects on
whatever exists in the GGUF you hand it. (Measurements below say you should not bother.)

## Verification — it is bit-exact, not "close enough"

The engine's own quantizer is the oracle:

```
$ deepseek4-quantize --hf <heretic-hf> --template <official.gguf> \
    --compare-tensor blk.5.attn_output_b.weight --compare-gguf <official.gguf>
byte_compare: OK          # layer 5 is NOT abliterated -> regenerating it from the heretic
                          # checkpoint reproduces antirez's bytes exactly. Q8_0 for AProj is
                          # deterministic and imatrix-free: the whole premise, proven.

$ ... --compare-tensor blk.10.attn_output_b.weight --compare-gguf <spliced.gguf>
byte_compare: OK          # and the patched tensors are what the C would have produced
```

- `blk.10/26/42` after splicing: `byte_compare: OK`; untouched `blk.5`: `byte_compare: OK`
- python output vs official GGUF bytes on a non-edited tensor: **0 differing bytes of 35,651,584**
- full-file scan: 619,787,653 differing bytes, **all inside the 33 expected ranges**, 33/33 touched
- size and header sha256 unchanged; 1328 tensors before and after
- overlay + index sha256 match squanchyzx's published receipts (`601c4409…`, `f06fc8d7…`)
- drafter: regenerating all six `attn_output_{a,b}` of the 3 MTP stages from the **official** FP8
  reproduces bleysg's drafter **byte for byte** — so that file's `wo_b` was indeed un-abliterated,
  and our Q8_0 path agrees with `dspark_extract.py` too

## Measurements (ds4-server v0.5.4-1, ctx 524288, temp 0, single GB10)

| | official quant | spliced (abliterated attn) |
|---|---|---|
| refusals, 16 benign-but-refusal-shaped prompts | 2/16 | **0/16** |
| reasoning smoke (arithmetic / logic / sequence / code output) | 4/4 | 4/4 |
| native tool-call (name + parsed args) | OK | OK |
| Think Max, correct answer + non-empty `reasoning_content` | OK | OK |
| DSpark accept, neutral prompt | **83.8%** (57/68) | 78.3% (58/74) |
| decode t/s, 3 reps after warmup | 30.2 / 33.9 / 33.7 | 29.3 / 32.4 / 30.8 |

Accept −5.5 pp and t/s −8% are the same phenomenon seen twice: lower accept means more verify
steps per emitted token. In wall-clock that is +1.4 s on a 500-token answer, +14 s on a 5000-token
Think Max answer.

### Agentic tool-call benchmark (`tool-eval-bench --short`, 15 core scenarios, Think Max)

| | official | spliced |
|---|---|---|
| final score | **90** ★★★★★ | **87** ★★★★ |
| points | 27/30 | 26/30 |
| A Tool Selection | 6/6 | 4/6 |
| B Parameter Precision | 4/6 | 4/6 |
| C Multi-Step Chains | 6/6 | 6/6 |
| D Restraint & Refusal | 6/6 | 6/6 |
| E Error Recovery | 5/6 | 6/6 |
| deployability / responsiveness | 68 / 18 | 72 / 38 |

Exactly two scenarios move, in opposite directions: TC-03 pass→fail ("did not complete the contact
lookup to email chain correctly") and TC-14 partial→pass ("acknowledged the stock tool failure,
recovered, and surfaced the price").

**Then we measured the noise floor, and it swallows the difference whole.** The *official* quant,
run four times with identical config:

| run | score | A | E | non-pass |
|---|---|---|---|---|
| official #1 | 90 | 6/6 | 5/6 | TC-06 fail, TC-14 partial |
| official #2 | **87** | **4/6** | **6/6** | **TC-03 fail, TC-06 fail** |
| official #3 | 90 | 6/6 | 5/6 | TC-06 fail, TC-14 partial |
| official #4 | 90 | 6/6 | 5/6 | TC-06 fail, TC-14 partial |
| **spliced** | **87** | **4/6** | **6/6** | **TC-03 fail, TC-06 fail** |

One official run in four scores 87 with the **exact same signature** as the spliced model — same
score, same A=4/6, same E=6/6, same two failing scenarios. TC-03 is simply flaky and flips on the
same weights; when it fails, TC-14 passes, so the two move together. Same-model spread is 3 points,
which is the entire gap we were looking at.

Conclusion: **the harness does not resolve differences below ~3 points**, and the observed gap sits
inside same-model variance. That is "indistinguishable at this resolution", not "identical" — a
distinction worth keeping, since a real 1–2 point regression would be invisible here.

One finding that is *not* noise: **TC-06 (Multi-Value Extraction) fails in almost every measurement
on both models.** That is a reproducible weakness of the 2-bit quant on parameter precision, and it
applies to the unmodified official quant too.

### Four runs each, and what the aggregate was hiding

| | runs | mean | sd |
|---|---|---|---|
| official | 90, 87, 90, 90 | **89.25** | 1.50 |
| spliced | 87, 87, 87, 93 | **88.50** | 3.00 |

Exact permutation test over all 70 partitions: **p = 1.000**. There is nothing to see in the
aggregate — but the spliced model is twice as variable, and one scenario does *not* look like noise:

| scenario | official | spliced |
|---|---|---|
| **TC-07** Search → Read → Act (4-step dependent chain) | **pass 4/4** | **pass 1/4**, 3× partial |
| TC-06 Multi-Value Extraction | fail 4/4 | fail 3/4, pass 1 |
| TC-03 (flaky on both) | 2,0,2,2 | 0,2,2,2 |
| TC-14 | 1,2,1,1 | 2,1,1,1 |

The grader's note on all three spliced partials is identical: *"Completed most of the chain, but
missed one dependent step."* The official never missed it in four runs. **Fisher exact on TC-07:
p = 0.143** — not significant at n=4, but it is the only scenario whose direction is consistent, and
it is the failure mode you would predict from editing attention *output* projections: a dependent
chain has to carry the previous step's result forward. The spliced model also passed TC-06 once,
which the official never did, so it is not uniformly worse.

### Settled at n=40 per arm: the abliteration costs real agentic capability

Two large, reproducible regressions. 40 reps per arm, alternating blocks of 10, model identity
verified from `/proc/<pid>/cmdline` after every reload, zero failed reps, Holm-corrected across the
four scenarios:

| scenario | official | spliced | delta | Holm p |
|---|---|---|---|---|
| **TC-07** Search → Read → Act (4-step dependent chain) | **97.5%** [87–100] | **57.5%** [42–71] | **−40.0 pp** ±16.1 | **0.0001** |
| **TC-14** Malformed Response (recover from a broken tool) | **55.0%** [40–69] | **12.5%** [5–26] | **−42.5 pp** ±18.5 | **0.0004** |
| TC-03 Implicit Tool Need | 100% | 95% | −5 pp | 0.987 |
| TC-06 Multi-Value Extraction | 2.5% | 5% | +2.5 pp | 1.000 |

Aggregate over the four: 5.575 → 4.700 points/run, p ≈ 0.

The two damaged scenarios share a shape: both require **using earlier context to condition the next
action** — carrying a result forward through a chain, or adapting after a tool returns garbage. That
is what the edited attention *output* projections across layers 10–42 carry. Refusal removal and
this capability appear to ride the same pathway, at least at `lambda = 1.5`.

**So: use the spliced model for unfiltered chat and writing, not as an agentic daily.** Its measured
win is narrow (0/16 refusals vs 2/16) and the cost on tool chains is not.

Incidental finding about the *unmodified* quant: TC-14 passes only 55% of the time. Recovering from
malformed tool responses is weak in the official 2-bit build too.

### The false negative, kept here on purpose

An earlier pass at 8 reps per arm gave 7/8 vs 5/8, Fisher p = 0.57, permutation p = 0.45, and was
written up as "the lead does not confirm". That was wrong — not the arithmetic, the wording. With
the true rates now known (97.5% vs 57.5%), 8 reps predicts 7.8 vs 4.6 passes; the observed 7 vs 5
was perfectly consistent with a −40 pp effect. **"Not significant" is not "no effect"** — the honest
label was "underpowered, unresolved". TC-14 was likewise dismissed as "flaky on both" at n=4, and it
turned out to be the larger regression of the two.

What did hold: declaring n=40 and the Holm correction *before* looking, and a design where model
identity is verified rather than assumed.

<details>
<summary>The underpowered runs, for the record</summary>

| | pass | partial | fail | points |
|---|---|---|---|---|
| official | 7/8 | 1 | 0 | **15/16** |
| spliced | 5/8 | 2 | 1 | **12/16** |

**Fisher p = 0.57, exact permutation on points p = 0.45.** Nothing. And the official partialed once
too — so the "4/4, never missed it" that generated the whole hypothesis was a small-n illusion:
TC-07 is flaky on both models.

Pooling all 12 measurements per model gives official 11/12 pass (23/24 points) vs spliced 6/12
(17/24), Fisher p = 0.069, permutation p = 0.060. It grazes the threshold **and it is not usable as
confirmation**: the pool contains the very sample that generated the hypothesis (that extreme 1/4),
and mixing discovery data with confirmation data inflates the effect. The clean read is the
dedicated ×8: p ≈ 0.45.

**Where that leaves it.** The direction is consistently against the spliced model in every view
(12/16, 17/24, never above), but any effect is too small for this sample. Observed effect size
d ≈ 0.64 ⇒ roughly **40 reps per arm** for 80% power at α = 0.05 — about an hour of box time at 40 s
per rep, if someone wants to settle it.

Practical read for daily use: official as the default, spliced when you actually need it unfiltered.

</details>

Worth noting: category D, *Restraint & Refusal*, is 6/6 on both. Abliteration did not damage the
ability to refuse when refusing is correct (impossible request, trivial knowledge that needs no
tool) — which is the failure mode that would actually matter for agentic work, far more than the
refusal rate on edgy prompts.

### The abliterated drafter is not worth building

Prediction going in: an abliterated drafter would protect accept rate on refusal-adjacent content,
where an un-abliterated drafter proposes refusal tokens the abliterated main model rejects.
**The measurement says no.**

| main + drafter | neutral prompt | refusal-adjacent prompt |
|---|---|---|
| spliced + **abliterated** drafter | accept 78.3% | accept 47.8% — drafts +23, hits +11, 105 tok |
| spliced + **official** drafter | accept **81.9%** | accept 47.8% — drafts +23, hits +11, 105 tok |

Bit-identical on the refusal-adjacent prompt; slightly *worse* than the official drafter on the
neutral one. Ship the official drafter. The 3-tensor drafter splice is harmless and pointless.

### Two measurement traps worth more than the result

1. **The first request after boot reads ~6.3 t/s instead of ~33.** It is not a serial-lane
   fallback — every request logged `path=cont served=1 fallback=0`, with zero `reject`/`serial`
   lines. It is DSpark being switched off for that sequence:
   `cont-dspark yield-quench bank=1 pos=38 spec_steps=5 debt=4.80 yewma=1.70 -> spec off for this
   seq (terminal)`. The drafter was yielding 1.70 tokens/step against accumulated debt 4.80, so the
   engine stopped speculating. With speculation alive, `tok/step` sits at 2.8–3.8. **Always discard
   a warmup generation**, or you will report a 5× difference that does not exist.
2. **Refusal is phrasing-sensitive.** The official quant refused a vulgar-monologue prompt in one
   wording and wrote it in another. Single-prompt refusal claims are noise; even 16 prompts only
   support a direction, not a rate.

## Files

- `tools/splice-heretic-attn.py` — the splice (main model and drafter), `--dry-run`, in-use guard
- `tools/verify-heretic-splice.py` — full-file proof that only the intended ranges changed
- `tools/bench-heretic-ab.sh` — speed / tool-call / refusal / reasoning / Think Max battery
- `tools/probe-refusal.sh` — harder refusal probe (8 prompts) + Think Max check on the full text
- `tools/reps-speed.sh`, `tools/reps-speed2.sh` — repeated accept/t/s measurement, warmup discarded
- `results/` — raw logs and JSON for every number above

## Credits and licensing

- engine and the 0731 IQ2XXS quant: [antirez/ds4](https://github.com/antirez/ds4),
  [antirez/deepseek-v4-gguf](https://huggingface.co/antirez/deepseek-v4-gguf) (MIT)
- batched-serving fork: Entrpi
- abliteration overlay, direction NPZ and bake scripts:
  [squanchyzx](https://huggingface.co/squanchyzx/DeepSeek-V4-Flash-0731-HERETIC-Abliterated-FP8),
  produced with [Heretic](https://github.com/p-e-w/heretic) by Philipp Emanuel Weidmann
  (Heretic itself is AGPL-3.0-or-later; no Heretic-derived source is redistributed here)
- DSpark drafter: [bleysg](https://huggingface.co/bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF)
- base weights: [deepseek-ai/DeepSeek-V4-Flash-0731](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) (MIT)

Tools in this repo are MIT. The quantization math in `splice-heretic-attn.py` is a numpy port of
`gguf-tools/deepseek4-quantize.c` and `quants.c` from ds4 (MIT), kept deliberately bit-compatible.

**Abliteration reduces refusal behaviour.** Whatever you build from this is yours to evaluate and
yours to be responsible for; nothing here is safety-certified.