Instructions to use inferRouter/Qwen3.6-27B-FP8-lmhead-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inferRouter/Qwen3.6-27B-FP8-lmhead-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inferRouter/Qwen3.6-27B-FP8-lmhead-fp8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inferRouter/Qwen3.6-27B-FP8-lmhead-fp8")
model = AutoModelForMultimodalLM.from_pretrained("inferRouter/Qwen3.6-27B-FP8-lmhead-fp8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inferRouter/Qwen3.6-27B-FP8-lmhead-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inferRouter/Qwen3.6-27B-FP8-lmhead-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inferRouter/Qwen3.6-27B-FP8-lmhead-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/inferRouter/Qwen3.6-27B-FP8-lmhead-fp8

SGLang

How to use inferRouter/Qwen3.6-27B-FP8-lmhead-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inferRouter/Qwen3.6-27B-FP8-lmhead-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inferRouter/Qwen3.6-27B-FP8-lmhead-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inferRouter/Qwen3.6-27B-FP8-lmhead-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inferRouter/Qwen3.6-27B-FP8-lmhead-fp8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use inferRouter/Qwen3.6-27B-FP8-lmhead-fp8 with Docker Model Runner:
```
docker model run hf.co/inferRouter/Qwen3.6-27B-FP8-lmhead-fp8
```

Qwen3.6-27B-FP8 with FP8-quantized `lm_head` (vLLM PR repro artifact)

🚀 RTX 5090 update: we got the official FP8-weight path running on a single 32 GB RTX 5090 by combining vLLM hybrid TurboQuant with FP8 quantization of the large lm_head tensor.

This checkpoint is the lm_head-FP8 half of that experiment. It is loadable today with the C4 overlay image documented below. Stock vLLM ≤ 0.20 cannot load it without source patches; once vllm-project/vllm#41000 merges and lands in a release image, vLLM will load it as-is.

Quality has not been re-validated under the final builder-built C4 image yet. Treat this as an RTX 5090 FP8 deployment candidate / PR repro artifact, not as a general production recommendation.

ℹ️ Not recommended: the NVFP4 alternative. There is also an inferRouter/Qwen3.6-27B-NVFP4 build that fits a single RTX 5090 32 GB, but its quality is materially worse than the upstream FP8 — same regression we observed across every NVFP4 27B variant we evaluated. The C4 path here (official FP8 weights + hybrid TurboQuant + FP8 lm_head) is the route that preserves upstream FP8 quality on the 32 GB card.

Summary

Derivative of Qwen/Qwen3.6-27B-FP8 with one specific change: the lm_head projection has been block-FP8-quantized ([128, 128] blocks, e4m3fn, BF16 scales, DeepSeek-V3-style weight_scale_inv) to free up GPU memory on tight cards (RTX 5090 32 GB).

Tensor	Source (BF16)	This repo (FP8)
`lm_head.weight`	shape `[248320, 5120]`, BF16, 2.368 GiB	shape `[248320, 5120]`, F8_E4M3, 1.184 GiB
`lm_head.weight_scale_inv`	(does not exist)	shape `[1940, 40]`, BF16, ~150 KiB

Net VRAM saving on GPU at load time: ~1.18 GiB (BF16 → FP8 of one tensor).

This matches what is observed on the C4 stack: BF16-lm_head model load reads as 27.66–27.69 GiB; FP8-lm_head model load reads as 26.5 GiB on RTX 5090. The earlier "~0.32 GiB real saving" measurement (vLLM 0.19.1 + RTX 5090) was a Triton-autotune-budget artefact — autotune scratch was absorbing most of the weight-side delta. Under vLLM 0.20 + the hybrid TurboQuant runtime (PR #39931) the autotune scratch is paid out of a different bucket and the full FP8 delta falls through to the "Model loading took" line.

All other layers and config are byte-identical to the upstream Qwen/Qwen3.6-27B-FP8 checkpoint. The only modified files are:

outside.safetensors — lm_head.weight re-quantized + lm_head.weight_scale_inv added
model.safetensors.index.json — added lm_head.weight_scale_inv entry
config.json — quantization_config.modules_to_not_convert no longer lists lm_head; quantization_config.lm_head: true added as opt-in flag

Why this exists

Qwen3.6-27B-FP8 BF16-lm_head weights load as ~27.66 GiB on a 32 GB GPU, leaving only ~1 GiB headroom. That is not enough for Triton autotuner scratch (GDN/Mamba solve_tril kernels) + KV cache blocks, so the engine OOMs at startup or during inference even with conservative gpu_memory_utilization and max_model_len.

Quantizing lm_head (the largest BF16 tensor remaining outside the FP8 quant scope of the upstream variant) frees ~1.18 GiB. Combined with hybrid TurboQuant KV (PR #39931), this is enough to pass the previously failing profiling/startup path in the C4 experiments on RTX 5090 (which previously failed with "Tried to allocate 272 MiB"). The recommended production envelope remains max_num_seqs=3 until the full builder-built image gate and stability test are recorded.

Loadable today

This checkpoint loads end-to-end on RTX 5090 32 GB using the InferRouter C4 overlay image, validated 2026-04-29:

inferrouter/vllm-openai-v020-pr39931-lmheadfp8-turboquant:latest
base:    vllm/vllm-openai:v0.20.0
overlay: PR #39931 (hybrid TurboQuant on Mamba/GDN models)
         + PR #41000 (FP8 ParallelLMHead opt-in) ported in-image
opt-in:  ENABLE_LM_HEAD_FP8=1 in c4_prepare_vllm020_pr39931_overlay.sh

Build script (InferRouter quantization workspace): qwen-36-27b/c4_turboquant/c4_prepare_vllm020_pr39931_overlay.sh.

Single-GPU TP=1 only. Multi-GPU TP>1 raises NotImplementedError from the loader shim — the proper vocab-shard-aware FP8 scale loader is a follow-up tracked behind PR #41000.

Once PR #41000 merges into vLLM main and lands in a release image, the overlay step becomes unnecessary; this checkpoint will load on stock vLLM directly.

Recommended runtime profile (RTX 5090, 32 GB)

--language-model-only
--quantization fp8
--kv-cache-dtype turboquant_k8v4
--gpu-memory-utilization 0.96
--max-model-len 5120
--max-num-seqs 3
--max-num-batched-tokens 15360
--enforce-eager
--no-enable-chunked-prefill
--reasoning-parser qwen3

Conservative fallback if scratch headroom matters more than concurrency: turboquant_k8v4 / max_model_len=4096 / max_num_seqs=3. Optional long-context profile with more KV-quality risk: turboquant_4bit_nc / max_model_len=6144 / max_num_seqs=2. Do not pair turboquant_4bit_nc with max_num_seqs=4 for near-full-context traffic; measured 4352 / 4608 × 4 profiles failed warmup OOM despite the better theoretical KV compression.

Status — why stock vLLM ≤ 0.20 doesn't load this

Three layered gaps in stock vLLM prevent loading this checkpoint out of the box:

Fp8Config.get_quant_method does not opt into ParallelLMHead. Returns None for any non-LinearBase/FusedMoE/Attention layer → falls through to UnquantizedEmbeddingMethod, which only registers weight (no weight_scale_inv).
Qwen3.5/3.6 model classes (vllm/model_executor/models/qwen3_5.py) build ParallelLMHead(...) without quant_config=, so the dispatcher above is never even called for lm_head.
VocabParallelEmbedding.weight_loader asserts loaded_weight.shape[output_dim] == self.org_vocab_size for every parameter routed through it. Companion FP8 params like weight_scale_inv (shape [1940, 40]) trip this assertion.

Tracking issue: vllm-project/vllm#40999 — "[Feature][FP8] Opt-in ParallelLMHead quantization in legacy Fp8Config". Filed 2026-04-27.

PR: vllm-project/vllm#41000 — "[FP8] Add opt-in ParallelLMHead dispatch to Fp8Config". Open since 2026-04-27. Implements all three gaps (Gap 1: dispatcher; Gap 2: qwen3_5 quant_config plumbing; Gap 3: companion-param loader). Mechanically and functionally validated end-to-end on RTX 6000 Pro 96 GB (token emission, deterministic + Czech sanity) and on RTX 5090 32 GB via the C4 overlay (model load + math + Czech sanity).

Related prior art: vllm-project/vllm#35696 ("[Model] Optional FP8 lm_head compression for Llama and Mistral") — different shape (per-model post-load dtype cast, env-var gated), stalled on maintainer review since 2026-03.

Reproducing the failure on stock vLLM

# Pull this checkpoint
huggingface-cli download inferrouter/Qwen3.6-27B-FP8-lmhead-fp8 \
  --local-dir ./qwen36-27b-lmhead-fp8

# Try to serve with stock vLLM (≤ 0.20) — will crash at load_weights:
vllm serve ./qwen36-27b-lmhead-fp8 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 4096 \
  --max-num-seqs 4 \
  --enforce-eager \
  --kv-cache-dtype fp8_e4m3

Expected error:

ValueError: There is no module or parameter named 'lm_head.weight_scale_inv'
in Qwen3_5ForCausalLM. The available parameters belonging to lm_head
(ParallelLMHead) are: {'lm_head.weight'}

(after applying patches #1 + #2 from PR #41000:)

File "vllm/model_executor/layers/vocab_parallel_embedding.py", line 463
    assert loaded_weight.shape[output_dim] == self.org_vocab_size
AssertionError

(with the full PR #41000 applied, or the C4 overlay image, the load succeeds and the engine reaches Application startup complete.)

How the lm_head was quantized

DeepSeek-V3 block-FP8 scheme, matching the existing weight format used by the upstream FP8 layers in this checkpoint:

BLOCK = 128
FP8_E4M3_MAX = 448.0

# For each [128, 128] block of the [248320, 5120] weight:
scale_inv = max(abs(block)) / FP8_E4M3_MAX     # per-block scale, BF16
quantized = clamp(block / scale_inv, -FP8_E4M3_MAX, FP8_E4M3_MAX).to(F8_E4M3)

No calibration data is needed (FP8 dynamic activation scheme). Both dimensions of lm_head.weight are exact multiples of 128 — no padding.

Reproducer script: quantize_lm_head.py in the InferRouter quantization workspace (linked from PR #41000).

License

Inherited from upstream Qwen license (Apache-2.0). All weights are mathematically derived from the upstream checkpoint by deterministic quantization; no retraining or fine-tuning was performed.

Acknowledgements

Built while debugging vLLM FP8 deployment for InferRouter on RTX 5090 32 GB hardware. The three-layer finding above came out of that work and is the basis for PR #41000.

Downloads last month: 47

Safetensors

Model size

28B params

Tensor type

BF16

F8_E4M3

Model tree for inferRouter/Qwen3.6-27B-FP8-lmhead-fp8

Base model

Qwen/Qwen3.6-27B

Quantized

Qwen/Qwen3.6-27B-FP8