---
license: apache-2.0
base_model: Qwen/Qwen3.8-Flash-Next
tags:
- qwen3.8-flash-next
- ple
- quantization
---

# Qwen3.8-Flash-Next quantized PLE tables

The 51.2B-parameter n-gram (PLE) table is the reason this model wants ~100 GB of free host RAM:
the CPU-offload worker holds it in BF16 (95.4 GB). This repo ships the same table quantized —
**FP8 per-row (49 GB)** and **INT4 group-16 (32 GB)** — plus a two-file overlay for the
`vllm/vllm-openai:qwen38-flash-next` image that serves them memory-mapped straight from disk.
Host RAM cost becomes page cache only, reclaimable under pressure.

Built from the original BF16 tables, so it works with any checkpoint of this model that keeps
them: [our mixed NVFP4/FP8 build](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8),
[our plain NVFP4](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4), or the original
model. It does not apply to checkpoints that re-quantized the tables themselves.

## Serve

```bash
hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \
  worker_image_quant.py ple_layer_quant.py --local-dir .
hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant --include "ples_int4/*" --local-dir .
# (or ples_fp8/* for the FP8 table)

docker run --gpus all --ipc=host -p 8000:8000 \
  -v $PWD/worker_image_quant.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py:ro \
  -v $PWD/ple_layer_quant.py:/usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py:ro \
  -v $PWD/ples_int4:/ples_int4 -e VLLM_PLE_QUANT_DIR=/ples_int4 \
  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600 \
  -e VLLM_GDN_DECODE_KERNEL=triton \
  vllm/vllm-openai:qwen38-flash-next \
  --model primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 \
  --distributed-executor-backend mp \
  --gpu-memory-utilization 0.92 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3
```

Drop `-e VLLM_GDN_DECODE_KERNEL=triton` when serving the plain NVFP4 build. No container
memory cap needed: unlike the BF16 disk path, the quantized tables fit the page cache next to
checkpoint streaming.

## Measured

One RTX PRO 6000 Blackwell (96 GB), 176 GB host, local NVMe, mixed NVFP4/FP8 checkpoint.
Throughput: 8K in / 512 out, prefix-cache-free, two seeds (shown a / b). Accuracy: the same
pinned 1,170-item knowledge + 200-item tool-calling protocol as the model cards, thinking on.

| table | size | host RSS | boot | tok/s @ 1 | tok/s @ 32 | TTFT @ 1 | knowledge | call acc |
|---|---|---|---|---|---|---|---|---|
| BF16, in RAM (baseline) | 95.4 GB | ~95 GB | 302 s | 84.5 / 84.4 | 516.8 / 523.6 | 569 / 573 ms | 92.2 | 84.8 (n=3) |
| FP8 per-row, mmapped | 49 GB | 52.6 GB° | 364 s | 80.3 / 80.1 | 489.7 / 500.7 | 759 / 768 ms | 92.2 | 77.5 (n=1) |
| INT4 group-16, mmapped | 32 GB | 32.9 GB° | 333 s | 80.2 / 80.1 | 483.6 / 487.9 | 663 / 671 ms | 92.9 | 79.5 (n=1) |

° mapped file pages, reclaimable under memory pressure — not anonymous RAM. Tool-calling
single runs sit inside the suite's ±1.5 repeat spread (BF16 three-run pool: 78.0–80.5); the
BF16 column shows the published three-run mean. Generation-sanity gates passed on every
configuration.

## Format

Sidecars are 128 shard files (`shard_N.safetensors`, 2,500,012 rows each, concatenated in
shard order) plus `META.json`. Row width 160.

| variant | tensors per shard | dequant |
|---|---|---|
| `ples_fp8` | `weight_fp8` [rows, 160] e4m3fn; `weight_scale` [rows] fp32 | `row = fp8 * scale[row]` |
| `ples_int4` | `weight_i4` [rows, 80] uint8, two nibbles, low first; `weight_scale` [rows, 10] fp16 | `row[c] = (nibble - 8) * scale[row, c // 16]` |

The overlay maps every shard with safetensors' native mmap and dequantizes only the gathered
rows (~100–200 KB per decoded token), so cold-start cost and steady-state RAM both scale with
the working set, not the table.

## Notes

- The overlay targets this exact image; the gather hook lives in a vendored model file
  (`vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py`), which is why this ships as an overlay
  rather than a vLLM PR. The BF16 disk path, which needs no model-file hook, is PR
  [vllm-project/vllm#54070](https://github.com/vllm-project/vllm/pull/54070).
- MTP speculative decoding (`num_speculative_tokens: 3`) composes well with the quantized
  tables: 129.6 tok/s single-stream on the real-prompt eval with the INT4 table, vs 142.6 with
  the BF16 table in RAM and 77.5–82.3 with the BF16 table on NVMe. Speculation multiplies
  gather traffic; the INT4 working set still fits the page cache where the BF16 one does not,
  so quantization is what makes MTP + low-RAM hosts viable together.
- The image's stock worker cannot load quantized tables at all (it rejects
  `ngram_embedding.weight_scale`), which also rules out CPU offload for checkpoints that ship
  FP8 tables with a global scale.

---

<p align="center">
  <br>
  <img src="https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8/resolve/main/assets/primitive-logo.png" alt="Primitive" width="34"><br>
  <sub>
    <a href="https://primitive.com"><b>primitive</b></a> ·
    <a href="https://huggingface.co/primitive-ai">more models</a> ·
    inference economics for production LLM systems
  </sub>
</p>