---
library_name: transformers
license: apache-2.0
base_model: orcarouter/Qwen3.8-27B-Uncensored
base_model_relation: quantized
pipeline_tag: image-text-to-text
tags:
- qwen3_5
- int8
- autoround
- w8a16
---

# Qwen3.8-27B-Uncensored INT8 AutoRound (W8A16, linear attention BF16, group size 128)

INT8 weight-only quantization of [orcarouter/Qwen3.8-27B-Uncensored](https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored)
with [AutoRound](https://github.com/intel/auto-round) (SignRound), following the recipe of
[Minachist/Qwen3.8-27B-INT8-AutoRound](https://huggingface.co/Minachist/Qwen3.8-27B-INT8-AutoRound) branch `linear-attn-bf16-gs128`,
with two changes: linear attention is **excluded from tuning** (not swapped back to BF16 after the fact), and **500 iters** instead of 250.

| Tensors | Precision |
|---|---|
| `self_attn.{q,k,v,o}_proj` (16 full-attention layers), `mlp.{gate,up,down}_proj` (64 layers), MTP block projections | INT8 symmetric, group_size 128 |
| `linear_attn.{in_proj_qkv,in_proj_z,out_proj,in_proj_a,in_proj_b}` (48 GDN layers), `embed_tokens`, `lm_head`, `mtp.fc`, norms, vision tower | BF16 |

263 INT8 linears / 354 BF16 linears. Format: `auto_round:auto_gptq` packing (vLLM loads it via GPTQ-Marlin with BF16 activations).

## Recipe

AutoRound `main` @ `b9f3d0079d014c73a1ff009800c597b9bc3f2a36` (version string 0.15.0), transformers 5.15.1, torch 2.13.0+cu130, one RTX PRO 6000 Blackwell.
`scheme="W8A16"` (bits 8, group_size 128, sym), `iters=500`, `nsamples=1024`, `seqlen=2048`, `batch_size=4`, `gradient_accumulate_steps=2`, `low_gpu_mem_usage=False`, `seed=42`.
Calibration: 256 samples built from `NeelNanda/pile-10k` + 768 from `codeparrot/github-code-clean` (documents concatenated so every sample is >= 2048 tokens, then truncated to 2048).
Every layer is named in full in `layer_config` (avoids AutoRound's shared-dict regex aliasing bug). Tuning took 1.26 h.

## KL divergence vs the BF16 source

Teacher-forced top-24 logprobs on one 128,000-token wikitext-103 stream (rows 100k+ of the train split), one sequence, **BF16 KV cache**, vLLM 0.27.1, `KL(P_bf16 || Q_int8)` in nats over the truncated top-24.
**These numbers are only comparable to other models scored with the same script, stream and teacher.**

| depth | n | KL mean | KL p50 | KL p99 | top-1 agreement | ΔNLL |
|---|---|---|---|---|---|---|
| 0k-4k | 3,999 | 0.00189 | 0.00056 | 0.0239 | 97.67% | +0.0038 |
| 4k-16k | 12,000 | 0.00363 | 0.00078 | 0.0338 | 97.51% | +0.0011 |
| 16k-48k | 32,000 | 0.00264 | 0.00085 | 0.0293 | 97.22% | +0.0019 |
| 48k-128k | 80,000 | 0.00320 | 0.00088 | 0.0338 | 97.28% | +0.0022 |

Own NLL: BF16 1.8244, INT8 1.8265. For reference, the same script on `Qwen/Qwen3.8-27B` gives FP8 (`Qwen/Qwen3.8-27B-FP8`) KL 0.0048 / top-1 96.5% and Minachist's INT8 0.0029 / 97.2%.

## Serving

```bash
vllm serve <this-repo> --tensor-parallel-size 2 --max-model-len 262144 --trust-remote-code
```
Tested with vLLM 0.27.1 (`Using MarlinLinearKernel for AutoGPTQLinearMethod`). MTP speculative decoding: `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'`.