Card: the sidecar now also serves FP8-table checkpoints; nvidia's NVFP4 booted on one card with it
Browse files
README.md
CHANGED
|
@@ -16,10 +16,16 @@ the CPU-offload worker holds it in BF16 (95.4 GB). This repo ships the same tabl
|
|
| 16 |
serves them memory-mapped straight from disk. Host RAM cost becomes page cache only,
|
| 17 |
reclaimable under pressure.
|
| 18 |
|
| 19 |
-
Built from the original BF16 tables, so it works with any checkpoint of this model
|
| 20 |
-
|
| 21 |
-
[our plain NVFP4](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4),
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
## Serve
|
| 25 |
|
|
|
|
| 16 |
serves them memory-mapped straight from disk. Host RAM cost becomes page cache only,
|
| 17 |
reclaimable under pressure.
|
| 18 |
|
| 19 |
+
Built from the original BF16 tables, so it works with any checkpoint of this model:
|
| 20 |
+
[our mixed NVFP4/FP8 build](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8),
|
| 21 |
+
[our plain NVFP4](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-NVFP4), the original model,
|
| 22 |
+
and, since 2026-09-06, checkpoints that ship their own table in FP8. The overlay stubs the table
|
| 23 |
+
parameter and serves gathers from the sidecar, so the checkpoint's own table tensors are dropped
|
| 24 |
+
before they load. That turns the table format into a non-issue:
|
| 25 |
+
[nvidia's NVFP4 build](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-NVFP4), whose card documents
|
| 26 |
+
an 8-GPU deployment because its FP8 table cannot use vLLM's BF16-only host offload, boots on one
|
| 27 |
+
96 GB card this way (234 s, 88,828 MiB, 348k-token KV pool) and scores 92.1 knowledge / 78.3
|
| 28 |
+
tool-calling under our protocol. Its 53.7 GB table-and-MTP shard can be skipped at download time.
|
| 29 |
|
| 30 |
## Serve
|
| 31 |
|