cognitivers
/

Qwen3.8-27B-OBLITERATED-12GB-GGUF

@@ -1,82 +1,109 @@
----
-license: apache-2.0
-base_model: OBLITERATUS/Qwen3.8-27B-OBLITERATED
-library_name: gguf
-tags:
-  - gguf
-  - imatrix
-  - qwen3_5
-  - qwen3.8
-  - abliterated
-  - uncensored
-  - low-vram
-  - iq2
-quantized_by: cognitivers
----
-![New weights incoming](assets/notice-banner.png)
-# Qwen3.8-27B-OBLITERATED — low-bit imatrix GGUFs (runs on 12 GB VRAM)
-> ### ⚠️ Rebuild in progress (2026-08-23)
->
-> **The files currently in this repo were quantized from the base repo's `Q8_0` as it stood on 2026-08-22, before the author fixed his GGUF conversion pipeline.**
->
-> On 2026-08-23 the author of the base model [stated publicly](https://x.com/elder_plinius/status/2091339694303068467) that the GGUFs published before that date "weren't being properly converted" (the bf16 safetensors were always fine), and re-uploaded a corrected `Q8_0` (commit `V3: fixed merge (abliterated weights, not stock)`, 2026-08-23 00:10 UTC). Two further upstream fixes, to `config.json` (a pad token inside `eos_token_id` caused early stopping) and to `chat_template.jinja`, also postdate every file here.
->
-> We are rebuilding the entire ladder from the corrected V3 `Q8_0`, with a fresh importance matrix and the current template/eos metadata patched into each GGUF. **The files will be replaced in this same repo, so these URLs and the `-hf` / Ollama commands below keep working.** Until that lands, read the numbers in the table as measured against the old baseline, and expect the abliteration to be weaker than the author intends.
-First working IQ2/IQ3_XXS quantizations of [OBLITERATUS/Qwen3.8-27B-OBLITERATED](https://huggingface.co/OBLITERATUS/Qwen3.8-27B-OBLITERATED), sized so **12 GB (and even 10 GB) GPUs run this 27B fully offloaded**. Until now the smallest GGUF anywhere for this model was Q2_K at 10.9 GB — with no VRAM headroom for context on a 12 GB card, and no IQ-class quants at all.
-**Why no one had published these levels:** the public imatrix for this model is missing entries for several tensors, so `llama-quantize` refuses to produce any IQ2/IQ3-class quant from it ("the result will be garbage"). Two separate causes, both solved here:
-1. imatrix collection for hybrid SSM+attention models was buggy before llama.cpp [#14994](https://github.com/ggml-org/llama.cpp/pull/14994) — we computed a **fresh imatrix** (included in this repo) on current master with the single-sequence batching workaround, giving full coverage of all executed tensors.
-2. `blk.64` is an **MTP (multi-token-prediction) layer** that never runs during normal inference, so no imatrix data can exist for it; its 8 weight tensors are pinned to `q4_K` via `--tensor-type` (+~120 MB, zero quality impact on normal decoding).
-![A 27B model on a 12 GB GPU](assets/bench-vram-12gb.png)
-## Benchmarks (measured, not estimated)
-Perplexity on wikitext-2 (120 chunks, ctx 512), against the official Q8_0 as baseline. VRAM is the **measured peak** (`nvidia-smi`, 1 s sampling) with `-ngl 999 -c 8192 -fa on -ctk q8_0 -ctv q8_0` on an RTX A4000/A40.
-| File | Size | PPL | ΔPPL vs Q8_0 | Peak VRAM @ 8k ctx | tg128 / pp512 (RTX A4000) | Target cards |
-|---|---|---|---|---|---|---|
-| IQ3_XXS | 11.4 GB | 7.99 ± 0.12 | **+6.2 %** | 11.7 GiB | 25.7 / 690 t/s | 12 GB (headless, or ctx ≤4k with desktop) |
-| IQ2_M | 10.2 GB | 8.42 ± 0.12 | **+12.0 %** | 10.8 GiB | 26.1 / 609 t/s | 12 GB, comfortable |
-| IQ2_S | 9.6 GB | 8.82 ± 0.13 | **+17.3 %** | 10.0 GiB | 26.6 / 600 t/s | 10–12 GB |
-| IQ2_XXS | 8.7 GB | 9.70 ± 0.15 | **+29.0 %** | 9.2 GiB | 28.1 / 694 t/s | 8–10 GB — usable, clearly degraded |
-| Q8_0 (reference) | 29.0 GB | 7.52 ± 0.11 | — | — | — | not in this repo |
-![Quality cost vs decode speed](assets/bench-quality-vs-speed.png)
-Speeds via `llama-bench -fa 1 -r 3` on an RTX A4000 (448 GB/s — comparable bandwidth to an RTX 3060 12GB/4070 class). Every file was additionally smoke-tested end-to-end: downloaded from this repo, loaded fully on GPU, generated correct output — in both llama.cpp and **Ollama** (hybrid `qwen3_5` arch confirmed working in Ollama's engine).
-**Pick:** on a 12 GB card, `IQ3_XXS` for best quality or `IQ2_M` for context headroom / desktop use. `IQ2_XXS` exists so 8–10 GB cards can run the model at all — expect visibly degraded long-form coherence, code and math.
-## How to run
-```bash
-# llama.cpp
-llama-server -hf cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF:IQ2_M \
-  -ngl 999 -c 8192 -fa on -ctk q8_0 -ctv q8_0
-# Ollama
-ollama run hf.co/cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF:IQ2_M
-```
-Also works in LM Studio, Jan and koboldcpp. The base model is multimodal; these files are **text-only** — for vision add `mmproj-model-bf16.gguf` from the [original repo](https://huggingface.co/OBLITERATUS/Qwen3.8-27B-OBLITERATED/tree/main) (~1 GB extra VRAM).
-## Provenance
-- Source: official `Q8_0` GGUF from the base repo (Q8_0 ≈ lossless vs bf16; requantized with `--allow-requantize`).
-- Importance matrix: `imatrix.gguf` in this repo — computed by us on the abliterated weights (llama.cpp master ≥ #14994, `-c 512 -b 512 -ub 512`, bartowski's calibration_datav3, 100 % coverage of executed 2-D tensors, verified programmatically).
-- Quantized with `llama-quantize` b10567: `--imatrix` + `--tensor-type blk.64.*=q4_K` (MTP layer, see above).
-- Reproducible: same inputs + same build → same bytes.
-At 2-bit levels quantization loss is real (see table). These quants exist to make the model *runnable* on mainstream GPUs, not to replace Q4+ if your hardware fits them.
-## Credits
-- [OBLITERATUS](https://huggingface.co/OBLITERATUS) — original abliterated model.
-- [mradermacher](https://huggingface.co/mradermacher) — first GGUF/imatrix work on this model; their repo documents the missing-tensor problem these files solve.
-- [llama.cpp](https://github.com/ggml-org/llama.cpp) — and compilade's hybrid-model imatrix fix (#14994).

+---
+license: apache-2.0
+base_model: OBLITERATUS/Qwen3.8-27B-OBLITERATED
+library_name: gguf
+tags:
+  - gguf
+  - imatrix
+  - qwen3_5
+  - qwen3.8
+  - abliterated
+  - uncensored
+  - low-vram
+  - iq2
+quantized_by: cognitivers
+---
+![New weights published](assets/notice-banner.png)
+# Qwen3.8-27B-OBLITERATED — low-bit imatrix GGUFs (runs on 12 GB VRAM)
+> ### Rebuilt 2026-08-23 — re-download if you pulled these before that date
+>
+> The author of the base model [stated publicly](https://x.com/elder_plinius/status/2091339694303068467) on 2026-08-23 that every GGUF he had published until then "weren't being properly converted", and his repo carries the commit `V3: fixed merge (abliterated weights, not stock)`. The files that were in this repo were quantized on 2026-08-22, before both.
+>
+> **Every file here has been rebuilt from the corrected `Q8_0`**, with a fresh importance matrix computed on the corrected weights and the current upstream chat template baked in. Filenames and URLs are unchanged, so `-hf` and Ollama commands keep working. The previous files remain reachable in this repo's git history at revision `ad034226` — their sha256 are listed in the changelog at the bottom.
+IQ2/IQ3_XXS quantizations of [OBLITERATUS/Qwen3.8-27B-OBLITERATED](https://huggingface.co/OBLITERATUS/Qwen3.8-27B-OBLITERATED), sized so **12 GB (and even 10 GB) GPUs run this 27B fully offloaded**. The smallest GGUF in the base repo is Q2_K at 10.9 GB, with no headroom for context on a 12 GB card and no IQ-class quants at all.
+![Sizes against a 12 GB budget](assets/bench-size.png)
+## Why these levels need work that the base repo does not do
+1. The public imatrix for this model is missing entries for several tensors, so `llama-quantize` refuses to produce any IQ2/IQ3-class quant from it. imatrix collection for hybrid SSM+attention models was buggy before llama.cpp [#14994](https://github.com/ggml-org/llama.cpp/pull/14994); we compute a **fresh imatrix** (shipped here as `imatrix.gguf`) on current master with the single-sequence batching workaround.
+2. `blk.64` is an **MTP (multi-token-prediction) layer** that never runs during normal decoding, so no imatrix data can exist for it and the IQ quantizers abort. Its 8 weight tensors are pinned to `q4_K` via `--tensor-type`. That costs roughly 120 MB of file size for weights a plain llama.cpp decode never executes; it is kept so speculative-decoding front-ends still find the head.
+## Benchmarks
+| File | Size | PPL (wikitext-2) | vs corrected Q8_0 | Fits a 12 GB card |
+|---|---|---|---|---|
+| **IQ3_XXS** | 11.43 GB | 8.2767 ± 0.124 | **+6.6 %** | headless, or ctx ≤4k with a desktop |
+| **IQ2_M** | 10.24 GB | 8.8054 ± 0.133 | **+13.4 %** | yes, comfortably |
+| **IQ2_S** | 9.60 GB | 9.2450 ± 0.140 | **+19.1 %** | yes, with context headroom |
+| **IQ2_XXS** | 8.67 GB | 10.2938 ± 0.159 | **+32.6 %** | 8-10 GB cards, visibly degraded |
+| Q8_0 (base repo, our source) | 29.05 GB | 7.7630 ± 0.116 | — | no |
+**Pick:** `IQ3_XXS` for best quality on a 12 GB card, `IQ2_M` if you want context headroom or run a desktop on the same GPU. `IQ2_XXS` exists so 8-10 GB cards can run the model at all; expect visibly weaker long-form coherence, code and maths.
+![Quality cost per level](assets/bench-quality.png)
+### The rebuild, measured
+Perplexity alone cannot tell you the old files came from a different model, so here is the direct test: how often each model picks a **different top token** than the corrected `Q8_0`, over the same text.
+![Provenance](assets/bench-provenance.png)
+| Comparison | Same top token | Median KLD | RMS Δp |
+|---|---|---|---|
+| Old `Q8_0` vs corrected `Q8_0` | **89.26 ± 0.43 %** | 0.0328 | 6.94 % |
+| Our rebuilt IQ2_M vs its own `Q8_0` | 81.86 ± 0.54 % | 0.0911 | 13.28 % |
+Two `Q8_0` quantizations of the *same* weights agree on the top token about 99.9 % of the time. These two disagree **10.7 %** of the time, so they are not the same model. For scale, quantizing all the way from 8-bit to 2-bit costs 18.1 % disagreement: **the upstream merge fix is worth 59 % of that**.
+One counter-intuitive detail worth stating plainly: the **old** `Q8_0` scores a *better* perplexity (7.3158 ± 0.108 vs 7.7630 ± 0.116). That is consistent with the upstream commit message `V3: fixed merge (abliterated weights, not stock)` — the old file sat closer to the stock model, which reads better on wikitext and is worse at the one thing this model exists for. The same applies to the IQ2_M we previously shipped (8.2428 ± 0.122).
+We are not claiming the old files were broken in use. We did not measure a refusal difference and we are not publishing one. The case for the rebuild is the author's own statement, his commit history, and the 10.7 % top-token divergence above.
+## How to run
+```bash
+# llama.cpp
+llama-server -hf cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF:IQ2_M \
+  -ngl 999 -c 8192 -fa on -ctk q8_0 -ctv q8_0
+# Ollama
+ollama run hf.co/cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF:IQ2_M
+```
+Also works in LM Studio, Jan and koboldcpp. The base model is multimodal; these files are **text-only** — for vision add `mmproj-model-bf16.gguf` from the [original repo](https://huggingface.co/OBLITERATUS/Qwen3.8-27B-OBLITERATED/tree/main) (~1 GB extra VRAM).
+The author's recommended sampling for this model: `temp 0`, `repetition_penalty 1.15`, no system prompt, thinking off for coding.
+## Provenance & reproducibility
+- Source: `Qwen3.8-27B-OBLITERATED-Q8_0.gguf` from the base repo, **as of 2026-08-23** (the corrected merge). Requantized with `--allow-requantize`.
+- Importance matrix: `imatrix.gguf` here, computed by us on the corrected weights with bartowski's `calibration_datav3` (129 chunks, `-c 512 -b 512 -ub 512 --parse-special`). The calibration corpus is deliberately **not** the evaluation corpus.
+- `llama-quantize` from llama.cpp commit `c060ca9`, with `--imatrix` and `--tensor-type "blk\.64\..*=q4_K"`.
+- Chat template: patched with `gguf_new_metadata` from the upstream `chat_template.jinja` of 2026-08-23 17:42, verified byte-identical inside every shipped file (sha256 `58b76d2f9875fb3e9359a686567d2e30`). `eos_token_id=248046`, `padding_token_id=248044`.
+- Evaluation: wikitext-2 raw test, 120 chunks, `-c 512 -fa on`, same binary for every row of the table. Perplexity is reported with its standard error; differences smaller than the error bars are not meaningful.
+## Changelog
+**2026-08-23 — all four files rebuilt.**
+Source changed from the base repo's `Q8_0` as of 2026-08-22 to the corrected one of 2026-08-23. New importance matrix, upstream chat template patched in. Filenames unchanged.
+The previous files are still reachable at revision [`ad034226`](https://huggingface.co/cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF/tree/ad034226d179088ccdd04b226961a6f802d69d65) of this repo, so anything you pinned still resolves:
+| File | sha256 before | sha256 now |
+|---|---|---|
+| IQ3_XXS | `3b7ac301cba8bc56…` | `83cef0f72449b8c0…` |
+| IQ2_M | `e4978299d3d97900…` | `2b4552dbb7f6d410…` |
+| IQ2_S | `b3f540d3786e666d…` | `9f00561bd9d9a706…` |
+| IQ2_XXS | `7efc31c3b158ab2d…` | `3edb134e49153c27…` |
+## Credits
+- [OBLITERATUS](https://huggingface.co/OBLITERATUS) — original abliterated model, and for publicly flagging the conversion bug that prompted this rebuild.
+- [mradermacher](https://huggingface.co/mradermacher) — first GGUF/imatrix work on this model.
+- [llama.cpp](https://github.com/ggml-org/llama.cpp) — and compilade's hybrid-model imatrix fix (#14994).
+- [bartowski](https://huggingface.co/bartowski) — the `calibration_datav3` corpus.