---
license: apache-2.0
base_model: orcarouter/Qwen3.8-27B-Uncensored
base_model_relation: quantized
language:
- en
- zh
pipeline_tag: image-text-to-text
tags:
- qwen3.8
- qwen3_5
- nvfp4
- fp4
- compressed-tensors
- vllm
- blackwell
- rtx-5090
- uncensored
- abliterated
- quantized
- vision-language
- function-calling
- reasoning
- coding-agent
---

# Qwen3.8-27B-Uncensored-NVFP4

NVFP4 (W4A4) quantization of
[orcarouter/Qwen3.8-27B-Uncensored](https://huggingface.co/orcarouter/Qwen3.8-27B-Uncensored),
an abliterated fine-tune of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B).
**~19GB** (down from ~56GB BF16) — built for single consumer Blackwell GPUs.
On an RTX 5090 (32GB) it serves with **160k context** and room to spare.

- FP4 weights **and** FP4 activations on the transformer linear layers →
  native speedups on Blackwell (SM120: RTX 50-series, RTX PRO; SM100: B200)
- Kept in BF16: `lm_head`, the vision tower, and all gated-DeltaNet
  `linear_attn` layers (the hybrid architecture's linear-attention blocks)
- Vision, tool calling, and reasoning (`<think>`) all functional

## Serving with vLLM

Verified config for a single RTX 5090 (32GB), tested on vLLM 0.27.1:

```bash
vllm serve preetpatel/Qwen3.8-27B-Uncensored-NVFP4 \
    --max-model-len 163840 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.87 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 8192 \
    --limit-mm-per-prompt '{"image": 2, "video": 0}' \
    --mm-processor-kwargs '{"max_pixels": 2097152}' \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3
```

Notes from real-world testing on the 5090:

- **Leave headroom.** A tighter config (BF16 KV, 131k context,
  higher utilization) boots fine but OOMs mid-request: the gated-DeltaNet
  prefill kernel makes ~200MB transient allocations that the startup
  profiler does not fully account for. `--gpu-memory-utilization 0.87`
  leaves ~1.5GB of slack; `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`
  helps against fragmentation.
- **FP8 KV cache** halves KV to ~32KiB/token. The hybrid architecture only
  keeps KV for 16 of 64 layers, so 160k context needs only ~5GB of KV.
- **Tool calls** use the Qwen3-Coder XML format
  (`<function=...><parameter=...>`) — `--tool-call-parser qwen3_coder`
  is required for structured `tool_calls` output; `--reasoning-parser qwen3`
  moves `<think>` content into `reasoning_content`.
- The model's `generation_config.json` supplies the recommended sampling
  defaults (temperature 1.0, top_p 0.95, top_k 20).

## Measured results (RTX 5090, vLLM 0.27.1)

| Test | Result |
|---|---|
| Needle retrieval, 155k-token prompt, needle at 85% depth | exact |
| Needle retrieval, 155k-token prompt, needle at 10% depth | exact |
| 2 × 60k-token prompts, concurrent | both exact, 16s wall |
| Structured tool calling (`tools` API) | correct `tool_calls` + `finish_reason` |
| Prefill throughput | ~6-7k tok/s (≤93k ctx), ~4k tok/s at 155k |
| VRAM | ~19GB weights + ~7GB KV pool + headroom |

Also verified end-to-end as the backend of a coding agent
([pi](https://github.com/badlogic/pi-mono)): multi-turn tool loop, file
writes, and shell execution all behave correctly.

## Known caveat

vLLM logs at load time:

> In NVFP4 linear, the weight global scale is different for parallel layers
> (e.g. q_proj, k_proj, v_proj).

This checkpoint stores an independent NVFP4 global scale per linear layer;
vLLM reconciles them when fusing q/k/v (and gate/up) GEMMs, which adds a
small one-time rounding cost on those layers. The functional testing above
(tool calling, deep-context retrieval, agent use) shows no observable
degradation, but exact-benchmark users should be aware. If future
llmcompressor releases expose fused-layer scale sharing, a v2 revision may
be published.

## How it was made

One-shot RTN quantization with [llmcompressor](https://github.com/vllm-project/llm-compressor)
0.13.0 (`compressed-tensors` 0.17.0):

```python
recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["lm_head", "re:.*visual.*", "re:.*mtp.*"],
)
```

Calibration: 20 samples from `ultrachat_200k` (2048 max sequence length),
used only to set activation global scales — no training involved. The
exact recipe ships in this repo as `recipe.yaml`; the full quantization
and serving setup lives at
[preetpatel/qwen3.8_27B-nvidia5090](https://github.com/preetpatel/qwen3.8_27B-nvidia5090).

## Attribution & license

Apache 2.0, inherited from the base model. All credit for the model
weights to [Qwen](https://huggingface.co/Qwen) (base model) and
[orcarouter](https://huggingface.co/orcarouter) (abliterated fine-tune);
this repo only changes the numeric format. The base model is an
abliterated/uncensored variant intended by its authors for AI red-teaming
and research — apply your own judgment and safety measures downstream.