Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,105 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3.8-Flash-Next
+tags:
+- qwen3.8-flash-next
+- ple
+- quantization
+---
+
+# Qwen3.8-Flash-Next quantized PLE tables
+
+The 51.2B-parameter n-gram (PLE) table is the reason this model wants ~100 GB of free host RAM:
+the CPU-offload worker holds it in BF16 (95.4 GB). This repo ships the same table quantized —
+**FP8 per-row (49 GB)** and **INT4 group-16 (32 GB)** — plus a two-file overlay for the
+`vllm/vllm-openai:qwen38-flash-next` image that serves them memory-mapped straight from disk.
+Host RAM cost becomes page cache only, reclaimable under pressure.
+
+Built from the original BF16 tables, so it works with any checkpoint of this model that keeps
+them: [our mixed NVFP4/FP8 build](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8),
+[our plain NVFP4](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4), or the original
+model. It does not apply to checkpoints that re-quantized the tables themselves.
+
+## Serve
+
+```bash
+hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant \
+  worker_image_quant.py ple_layer_quant.py --local-dir .
+hf download primitive-ai/Qwen3.8-Flash-Next-PLE-quant --include "ples_int4/*" --local-dir .
+# (or ples_fp8/* for the FP8 table)
+
+docker run --gpus all --ipc=host -p 8000:8000 \
+  -v $PWD/worker_image_quant.py:/usr/local/lib/python3.12/dist-packages/vllm/v1/ple_offload/worker.py:ro \
+  -v $PWD/ple_layer_quant.py:/usr/local/lib/python3.12/dist-packages/vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py:ro \
+  -v $PWD/ples_int4:/ples_int4 -e VLLM_PLE_QUANT_DIR=/ples_int4 \
+  -e VLLM_PLE_CPU_OFFLOAD=1 -e VLLM_PLE_OFFLOAD_READY_TIMEOUT=3600 \
+  -e VLLM_GDN_DECODE_KERNEL=triton \
+  vllm/vllm-openai:qwen38-flash-next \
+  --model primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8 \
+  --distributed-executor-backend mp \
+  --gpu-memory-utilization 0.92 \
+  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
+  --reasoning-parser qwen3
+```
+
+Drop `-e VLLM_GDN_DECODE_KERNEL=triton` when serving the plain NVFP4 build. No container
+memory cap needed: unlike the BF16 disk path, the quantized tables fit the page cache next to
+checkpoint streaming.
+
+## Measured
+
+One RTX PRO 6000 Blackwell (96 GB), 176 GB host, local NVMe, mixed NVFP4/FP8 checkpoint.
+Throughput: 8K in / 512 out, prefix-cache-free, two seeds (shown a / b). Accuracy: the same
+pinned 1,170-item knowledge + 200-item tool-calling protocol as the model cards, thinking on.
+
+| table | size | host RSS | boot | tok/s @ 1 | tok/s @ 32 | TTFT @ 1 | knowledge | call acc |
+|---|---|---|---|---|---|---|---|---|
+| BF16, in RAM (baseline) | 95.4 GB | ~95 GB | 302 s | 84.5 / 84.4 | 516.8 / 523.6 | 569 / 573 ms | 92.2 | 84.8 (n=3) |
+| FP8 per-row, mmapped | 49 GB | 52.6 GB° | 364 s | 80.3 / 80.1 | 489.7 / 500.7 | 759 / 768 ms | 92.2 | 77.5 (n=1) |
+| INT4 group-16, mmapped | 32 GB | 32.9 GB° | 333 s | 80.2 / 80.1 | 483.6 / 487.9 | 663 / 671 ms | 92.9 | 79.5 (n=1) |
+
+° mapped file pages, reclaimable under memory pressure — not anonymous RAM. Tool-calling
+single runs sit inside the suite's ±1.5 repeat spread (BF16 three-run pool: 78.0–80.5); the
+BF16 column shows the published three-run mean. Generation-sanity gates passed on every
+configuration.
+
+## Format
+
+Sidecars are 128 shard files (`shard_N.safetensors`, 2,500,012 rows each, concatenated in
+shard order) plus `META.json`. Row width 160.
+
+| variant | tensors per shard | dequant |
+|---|---|---|
+| `ples_fp8` | `weight_fp8` [rows, 160] e4m3fn; `weight_scale` [rows] fp32 | `row = fp8 * scale[row]` |
+| `ples_int4` | `weight_i4` [rows, 80] uint8, two nibbles, low first; `weight_scale` [rows, 10] fp16 | `row[c] = (nibble - 8) * scale[row, c // 16]` |
+
+The overlay maps every shard with safetensors' native mmap and dequantizes only the gathered
+rows (~100–200 KB per decoded token), so cold-start cost and steady-state RAM both scale with
+the working set, not the table.
+
+## Notes
+
+- The overlay targets this exact image; the gather hook lives in a vendored model file
+  (`vllm/models/qwen3_8_flash_next/nvidia/ple_layer.py`), which is why this ships as an overlay
+  rather than a vLLM PR. The BF16 disk path, which needs no model-file hook, is PR
+  [vllm-project/vllm#54070](https://github.com/vllm-project/vllm/pull/54070).
+- MTP speculative decoding (`num_speculative_tokens: 3`) composes well with the quantized
+  tables: 129.6 tok/s single-stream on the real-prompt eval with the INT4 table, vs 142.6 with
+  the BF16 table in RAM and 77.5–82.3 with the BF16 table on NVMe. Speculation multiplies
+  gather traffic; the INT4 working set still fits the page cache where the BF16 one does not,
+  so quantization is what makes MTP + low-RAM hosts viable together.
+- The image's stock worker cannot load quantized tables at all (it rejects
+  `ngram_embedding.weight_scale`), which also rules out CPU offload for checkpoints that ship
+  FP8 tables with a global scale.
+
+---
+
+<p align="center">
+  <br>
+  <img src="https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8/resolve/main/assets/primitive-logo.png" alt="Primitive" width="34"><br>
+  <sub>
+    <a href="https://primitive.com"><b>primitive</b></a> ·
+    <a href="https://huggingface.co/primitive-ai">more models</a> ·
+    inference economics for production LLM systems
+  </sub>
+</p>