---
license: apache-2.0
base_model: OBLITERATUS/Qwen3.8-27B-OBLITERATED
library_name: gguf
tags:
  - gguf
  - imatrix
  - qwen3_5
  - qwen3.8
  - abliterated
  - uncensored
  - low-vram
  - iq2
quantized_by: cognitivers
---

![New weights published](assets/notice-banner.png)

# Qwen3.8-27B-OBLITERATED — low-bit imatrix GGUFs (runs on 12 GB VRAM)

> ### Rebuilt 2026-08-23 — re-download if you pulled these before that date
>
> The author of the base model [stated publicly](https://x.com/elder_plinius/status/2091339694303068467) on 2026-08-23 that every GGUF he had published until then "weren't being properly converted", and his repo carries the commit `V3: fixed merge (abliterated weights, not stock)`. The files that were in this repo were quantized on 2026-08-22, before both.
>
> **Every file here has been rebuilt from the corrected `Q8_0`**, with a fresh importance matrix computed on the corrected weights and the current upstream chat template baked in. Filenames and URLs are unchanged, so `-hf` and Ollama commands keep working. The previous files remain reachable in this repo's git history at revision `ad034226` — their sha256 are listed in the changelog at the bottom.

IQ2/IQ3_XXS quantizations of [OBLITERATUS/Qwen3.8-27B-OBLITERATED](https://huggingface.co/OBLITERATUS/Qwen3.8-27B-OBLITERATED), sized so **12 GB (and even 10 GB) GPUs run this 27B fully offloaded**. The smallest GGUF in the base repo is Q2_K at 10.9 GB, with no headroom for context on a 12 GB card and no IQ-class quants at all.

![Sizes against a 12 GB budget](assets/bench-size.png)

## Why these levels need work that the base repo does not do

1. The public imatrix for this model is missing entries for several tensors, so `llama-quantize` refuses to produce any IQ2/IQ3-class quant from it. imatrix collection for hybrid SSM+attention models was buggy before llama.cpp [#14994](https://github.com/ggml-org/llama.cpp/pull/14994); we compute a **fresh imatrix** (shipped here as `imatrix.gguf`) on current master with the single-sequence batching workaround.
2. `blk.64` is an **MTP (multi-token-prediction) layer** that never runs during normal decoding, so no imatrix data can exist for it and the IQ quantizers abort. Its 8 weight tensors are pinned to `q4_K` via `--tensor-type`. That costs roughly 120 MB of file size for weights a plain llama.cpp decode never executes; it is kept so speculative-decoding front-ends still find the head.

## Benchmarks

| File | Size | PPL (wikitext-2) | vs corrected Q8_0 | Fits a 12 GB card |
|---|---|---|---|---|
| **IQ3_XXS** | 11.43 GB | 8.2767 ± 0.124 | **+6.6 %** | headless, or ctx ≤4k with a desktop |
| **IQ2_M** | 10.24 GB | 8.8054 ± 0.133 | **+13.4 %** | yes, comfortably |
| **IQ2_S** | 9.60 GB | 9.2450 ± 0.140 | **+19.1 %** | yes, with context headroom |
| **IQ2_XXS** | 8.67 GB | 10.2938 ± 0.159 | **+32.6 %** | 8-10 GB cards, visibly degraded |
| Q8_0 (base repo, our source) | 29.05 GB | 7.7630 ± 0.116 | — | no |

**Pick:** `IQ3_XXS` for best quality on a 12 GB card, `IQ2_M` if you want context headroom or run a desktop on the same GPU. `IQ2_XXS` exists so 8-10 GB cards can run the model at all; expect visibly weaker long-form coherence, code and maths.

![Quality cost per level](assets/bench-quality.png)

### The rebuild, measured

Perplexity alone cannot tell you the old files came from a different model, so here is the direct test: how often each model picks a **different top token** than the corrected `Q8_0`, over the same text.

![Provenance](assets/bench-provenance.png)

| Comparison | Same top token | Median KLD | RMS Δp |
|---|---|---|---|
| Old `Q8_0` vs corrected `Q8_0` | **89.26 ± 0.43 %** | 0.0328 | 6.94 % |
| Our rebuilt IQ2_M vs its own `Q8_0` | 81.86 ± 0.54 % | 0.0911 | 13.28 % |

Two `Q8_0` quantizations of the *same* weights agree on the top token about 99.9 % of the time. These two disagree **10.7 %** of the time, so they are not the same model. For scale, quantizing all the way from 8-bit to 2-bit costs 18.1 % disagreement: **the upstream merge fix is worth 59 % of that**.

One counter-intuitive detail worth stating plainly: the **old** `Q8_0` scores a *better* perplexity (7.3158 ± 0.108 vs 7.7630 ± 0.116). That is consistent with the upstream commit message `V3: fixed merge (abliterated weights, not stock)` — the old file sat closer to the stock model, which reads better on wikitext and is worse at the one thing this model exists for. The same applies to the IQ2_M we previously shipped (8.2428 ± 0.122).

We are not claiming the old files were broken in use. We did not measure a refusal difference and we are not publishing one. The case for the rebuild is the author's own statement, his commit history, and the 10.7 % top-token divergence above.

## How to run

```bash
# llama.cpp
llama-server -hf cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF:IQ2_M \
  -ngl 999 -c 8192 -fa on -ctk q8_0 -ctv q8_0

# Ollama
ollama run hf.co/cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF:IQ2_M
```

Also works in LM Studio, Jan and koboldcpp. The base model is multimodal; these files are **text-only** — for vision add `mmproj-model-bf16.gguf` from the [original repo](https://huggingface.co/OBLITERATUS/Qwen3.8-27B-OBLITERATED/tree/main) (~1 GB extra VRAM).

The author's recommended sampling for this model: `temp 0`, `repetition_penalty 1.15`, no system prompt, thinking off for coding.

## Provenance & reproducibility

- Source: `Qwen3.8-27B-OBLITERATED-Q8_0.gguf` from the base repo, **as of 2026-08-23** (the corrected merge). Requantized with `--allow-requantize`.
- Importance matrix: `imatrix.gguf` here, computed by us on the corrected weights with bartowski's `calibration_datav3` (129 chunks, `-c 512 -b 512 -ub 512 --parse-special`). The calibration corpus is deliberately **not** the evaluation corpus.
- `llama-quantize` from llama.cpp commit `c060ca9`, with `--imatrix` and `--tensor-type "blk\.64\..*=q4_K"`.
- Chat template: patched with `gguf_new_metadata` from the upstream `chat_template.jinja` of 2026-08-23 17:42, verified byte-identical inside every shipped file (sha256 `58b76d2f9875fb3e9359a686567d2e30`). `eos_token_id=248046`, `padding_token_id=248044`.
- Evaluation: wikitext-2 raw test, 120 chunks, `-c 512 -fa on`, same binary for every row of the table. Perplexity is reported with its standard error; differences smaller than the error bars are not meaningful.

## Changelog

**2026-08-23 — all four files rebuilt.**
Source changed from the base repo's `Q8_0` as of 2026-08-22 to the corrected one of 2026-08-23. New importance matrix, upstream chat template patched in. Filenames unchanged.

The previous files are still reachable at revision [`ad034226`](https://huggingface.co/cognitivers/Qwen3.8-27B-OBLITERATED-12GB-GGUF/tree/ad034226d179088ccdd04b226961a6f802d69d65) of this repo, so anything you pinned still resolves:

| File | sha256 before | sha256 now |
|---|---|---|
| IQ3_XXS | `3b7ac301cba8bc56…` | `83cef0f72449b8c0…` |
| IQ2_M | `e4978299d3d97900…` | `2b4552dbb7f6d410…` |
| IQ2_S | `b3f540d3786e666d…` | `9f00561bd9d9a706…` |
| IQ2_XXS | `7efc31c3b158ab2d…` | `3edb134e49153c27…` |

## Credits

- [OBLITERATUS](https://huggingface.co/OBLITERATUS) — original abliterated model, and for publicly flagging the conversion bug that prompted this rebuild.
- [mradermacher](https://huggingface.co/mradermacher) — first GGUF/imatrix work on this model.
- [llama.cpp](https://github.com/ggml-org/llama.cpp) — and compilade's hybrid-model imatrix fix (#14994).
- [bartowski](https://huggingface.co/bartowski) — the `calibration_datav3` corpus.