Qwen3.8-27B-Uncensored-NVFP4

NVFP4 (W4A4) quantization of orcarouter/Qwen3.8-27B-Uncensored, an abliterated fine-tune of Qwen/Qwen3.8-27B. ~19GB (down from ~56GB BF16) — built for single consumer Blackwell GPUs. On an RTX 5090 (32GB) it serves with 160k context and room to spare.

• FP4 weights and FP4 activations on the transformer linear layers → native speedups on Blackwell (SM120: RTX 50-series, RTX PRO; SM100: B200)
• Kept in BF16: lm_head, the vision tower, and all gated-DeltaNet linear_attn layers (the hybrid architecture's linear-attention blocks)
• Vision, tool calling, and reasoning (<think>) all functional

Serving with vLLM

Verified config for a single RTX 5090 (32GB), tested on vLLM 0.27.1:

vllm serve preetpatel/Qwen3.8-27B-Uncensored-NVFP4 \
    --max-model-len 163840 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.87 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 8192 \
    --limit-mm-per-prompt '{"image": 2, "video": 0}' \
    --mm-processor-kwargs '{"max_pixels": 2097152}' \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3

Notes from real-world testing on the 5090:

• Leave headroom. A tighter config (BF16 KV, 131k context, higher utilization) boots fine but OOMs mid-request: the gated-DeltaNet prefill kernel makes ~200MB transient allocations that the startup profiler does not fully account for. --gpu-memory-utilization 0.87 leaves ~1.5GB of slack; PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True helps against fragmentation.
• FP8 KV cache halves KV to ~32KiB/token. The hybrid architecture only keeps KV for 16 of 64 layers, so 160k context needs only ~5GB of KV.
• Tool calls use the Qwen3-Coder XML format (<function=...><parameter=...>) — --tool-call-parser qwen3_coder is required for structured tool_calls output; --reasoning-parser qwen3 moves <think> content into reasoning_content.
• The model's generation_config.json supplies the recommended sampling defaults (temperature 1.0, top_p 0.95, top_k 20).

Measured results (RTX 5090, vLLM 0.27.1)

Test	Result
Needle retrieval, 155k-token prompt, needle at 85% depth	exact
Needle retrieval, 155k-token prompt, needle at 10% depth	exact
2 × 60k-token prompts, concurrent	both exact, 16s wall
Structured tool calling (tools API)	correct tool_calls + finish_reason
Prefill throughput	~6-7k tok/s (≤93k ctx), ~4k tok/s at 155k
VRAM	~19GB weights + ~7GB KV pool + headroom

Also verified end-to-end as the backend of a coding agent (pi): multi-turn tool loop, file writes, and shell execution all behave correctly.

Known caveat

vLLM logs at load time:

In NVFP4 linear, the weight global scale is different for parallel layers (e.g. q_proj, k_proj, v_proj).

This checkpoint stores an independent NVFP4 global scale per linear layer; vLLM reconciles them when fusing q/k/v (and gate/up) GEMMs, which adds a small one-time rounding cost on those layers. The functional testing above (tool calling, deep-context retrieval, agent use) shows no observable degradation, but exact-benchmark users should be aware. If future llmcompressor releases expose fused-layer scale sharing, a v2 revision may be published.

How it was made

One-shot RTN quantization with llmcompressor 0.13.0 (compressed-tensors 0.17.0):

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["lm_head", "re:.*visual.*", "re:.*mtp.*"],
)

Calibration: 20 samples from ultrachat_200k (2048 max sequence length), used only to set activation global scales — no training involved. The exact recipe ships in this repo as recipe.yaml; the full quantization and serving setup lives at preetpatel/qwen3.8_27B-nvidia5090.

Attribution & license

Apache 2.0, inherited from the base model. All credit for the model weights to Qwen (base model) and orcarouter (abliterated fine-tune); this repo only changes the numeric format. The base model is an abliterated/uncensored variant intended by its authors for AI red-teaming and research — apply your own judgment and safety measures downstream.