--- license: apache-2.0 base_model: gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 pretty_name: Qwen3.8-27B-NVFP4-RTX5090-LMHead4 library_name: transformers pipeline_tag: image-text-to-text tags: - nvfp4 - modelopt - nvidia - rtx-5090 - blackwell - quantization - qwen3.8 - sglang - vllm --- > ## ⚠️ Superseded > > The main checkpoint now has the NVFP4 `lm_head` **and** the MTP head removed — it is strictly better than this variant (17.92 GB, 2 shards). > > **Use [`Qwen3.8-27B-NVFP4-RTX5090`](https://huggingface.co/gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090) instead.** This repo stays up so existing pins keep working. --- # Qwen3.8-27B-NVFP4-RTX5090-LMHead4 [`Qwen3.8-27B-NVFP4-RTX5090`](https://huggingface.co/gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090) with **`lm_head` quantized to NVFP4** instead of left in BF16. Every other tensor is bit-identical to the parent. **Smaller and measurably faster, with no accuracy or acceptance cost.** | | Parent | **This variant** | | | --- | --- | --- | --- | | Checkpoint | 20.59 GB | **18.77 GB** | −8.8% | | Weights in VRAM | 18.80 GB | **17.10 GB** | −9.0% | | Decode, no speculation | 81.6 tok/s | **88.45 tok/s** | **+8.4%** | | Decode + [DSpark drafter](https://huggingface.co/gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4) | 158.0 tok/s | **181.7 tok/s** | **+15.0%** | | Accuracy smoke (60 items) | 38/60 | **42/60** | no degradation | | DSpark acceptance (held-out) | 2.886 | **2.904** | +0.6% | | DSpark acceptance (agentic) | 3.766 | **3.891** | +3.3% | **2.23× the decode of the unquantized-lm_head parent without speculation**, on one RTX 5090. --- ## Why this is free speed At concurrency 1 this model is **weight-bandwidth bound, not compute bound**. Measured on the parent: 18.80 GiB of resident weights at 81.6 tok/s implies a **1.65 TB/s** read rate against the RTX 5090's 1.79 TB/s spec — about **92% of peak**. Decode is streaming the whole model once per token. `lm_head` is a full-vocabulary (248,320 × 5,120) GEMM evaluated on **every** token, so it is 2.54 GB of that per-token read. Quantizing it to NVFP4 cuts it to 0.72 GB. The bandwidth model predicted **89.7 tok/s**; measured **88.45** — within **1.4%**. On this hardware, bytes removed from the weight read path convert almost linearly into tokens per second. Note the distinction: embeddings are the same 2.54 GB but are a *gather* (~10 KB/token), so quantizing them would save capacity, not decode speed. `lm_head` is the only large BF16 block on the per-token critical path. --- ## Accuracy Same 20 items per task, seed `20260815`, thinking on, `temperature=1.0`, `top_p=0.95`, 24k generation cap — both arms run through the same harness on SGLang. | Task | Parent | This variant | | --- | --- | --- | | GPQA Diamond | 10/20 | 13/20 | | AIME 2025 | 11/20 | 12/20 | | MMLU-Pro | 17/20 | 17/20 | | **Overall** | **38/60 (63%)** | **42/60 (70%)** | **Read this as "no degradation", not "an improvement."** At n=20 per task with `temperature=1.0`, a 4-item difference is well inside noise, and 6–9 items per task hit the 24k truncation cap on both arms. The claim supported by this data is that 4-bit logits did not measurably hurt quality — not that they helped. These numbers are also **not** comparable to the parent card's published smoke, which was measured on vLLM 0.27.1; this harness runs on SGLang. Both arms here share one harness, so the comparison between them is valid. ## Speculative decoding The v2 DSpark drafter was trained against the parent's logits, so changing `lm_head` could have degraded acceptance. It did not — acceptance is flat to slightly up: | Domain | Parent | This variant | | --- | --- | --- | | Math | 4.388 | 4.496 | | Coding | 3.804 | 3.864 | | Long-context | 2.334 | 2.474 | | Chat | 2.450 | 2.363 | | **Held-out overall** | **2.886** | **2.904** | | **Agentic tool calling** | **3.766** | **3.891** | Acceptance measured per request with the prefix cache flushed, from cumulative verify counters. --- ## Serve Identical to the parent — no flag changes: ```bash sglang serve --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090-LMHead4 \ --trust-remote-code --tp-size 1 \ --context-length 65536 --kv-cache-dtype fp8_e4m3 \ --attention-backend flashinfer --chunked-prefill-size 2048 \ --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \ --mem-fraction-static 0.90 --max-running-requests 2 \ --speculative-algorithm DSPARK \ --speculative-draft-model-path gittensor-model-hub/Qwen3.8-27B-DSpark-NVFP4 \ --speculative-dspark-block-size 7 --speculative-draft-model-quantization modelopt_fp4 \ --reasoning-parser qwen3 --tool-call-parser qwen3_coder ``` The native MTP head is retained, so vLLM speculation still works. --- ## What changed `lm_head.weight` (BF16, 248320 × 5120) replaced by four NVFP4 tensors: ``` lm_head.weight U8 (248320, 2560) packed E2M1 lm_head.weight_scale F8_E4M3 (248320, 320) per-group-16 scales lm_head.weight_scale_2 F32 () global weight scale lm_head.input_scale F32 () activation scale, amax 49.75 ``` The activation scale comes from measuring real `lm_head` inputs on the served parent (amax **49.75**, p50 40.0 — a clean distribution, max/p50 = 1.24, no outlier tail). ModelOpt 0.45 performed the quantization so the E2M1 packing and UE4M3 scale layout match what the runtime expects. `lm_head` was also removed from **both** exclude lists — `hf_quant_config.json → quantization.exclude_modules` **and** `config.json → quantization_config.ignore`. Missing the second one causes a silent fallback to an unquantized layer, surfacing as `Parameter lm_head.input_scale not found in params_dict` and then a 5120-vs-2560 shape error. ## License Apache 2.0, same as the parent and the Qwen3.8-27B base model.