---
license: apache-2.0
base_model: nex-agi/Nex-N2.5-mini
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
- vllm
- compressed-tensors
- fp8
- quantized
- moe
- nex-n2.5
- agentic
- tool-calling
- single-gpu
- blackwell
---
The fastest of the four single-stream, and the first FP8 of this model.
FP8 quantization of nex-agi/Nex-N2.5-mini to
38.13 GB, on one 96 GB card.
All 40 layers of routed experts at 8 bits, per output channel; attention, linear attention, shared experts,
embeddings and the vision tower stay BF16.
> **Three builds, one measurement.** We quantized this model's experts three ways and measured all of
> them against the BF16 original under one protocol: this one, an
> [NVFP4 build](https://huggingface.co/primitive-ai/Nex-N2.5-mini-NVFP4) at 23.91 GB, and an
> [mixed build](https://huggingface.co/primitive-ai/Nex-N2.5-mini-mixed-NVFP4-FP8) at 26.04 GB. On
> accuracy the four are one band. They separate on size and on speed, and which one is fastest
> depends on your concurrency. This is the fastest single-stream, and the largest.
---
## Why this quant
* ⚡ **164 tok/s single-stream, the fastest of the four builds**, ahead of the 4-bit build's 145 and
the BF16 original's 141. Eight-bit experts skip the activation quantization and group-scale work
that 4-bit pays on every token, and at batch 1 that cost does not amortize.
* 🗜️ **38.13 GB against 70.24 GB in BF16.** The routed experts are 91.8% of this model's weight, so
quantizing them alone gets most of the reduction.
* 🎯 **Nothing measurable is lost.** 90.9 on the 1,170-item knowledge suite and 75.5 on the 200-item
tool-calling suite, against the BF16 original's 89.6 and 75.3 on the same box.
* 🥇 **The first FP8 build of this model.** At the time of writing the field has GGUF, MLX, two NVFP4
builds and one INT4/INT8 mix, and no FP8.
* 🧩 **Stock vLLM, no patches.** `compressed-tensors`, W8A8 with per-channel weight scales and
dynamic per-token activation scales, served by the released 0.28.0 image.
---
## Serve it
```bash
hf download primitive-ai/Nex-N2.5-mini-FP8 --local-dir ./Nex-N2.5-mini-FP8
docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
vllm/vllm-openai:v0.28.0 \
--model /models/Nex-N2.5-mini-FP8 \
--max-model-len 32768 --gpu-memory-utilization 0.92 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--reasoning-parser qwen3
```
`Qwen3_5MoeForConditionalGeneration` is in the 0.28.0 release, so no nightly is needed. The numbers
on this page were measured on `0.28.1rc1.dev472`, and 0.28.0 serves the same weights with the same
kernel (the Triton FP8 MoE path). Boot takes about 310 s on an idle card, longer than the 4-bit
build because the FP8 path compiles more kernels.
Two things about this model's chat template are worth knowing before you wire it up. It emits an
empty thinking block by default, so replies arrive with two leading newlines and the `reasoning`
field holds only whitespace; a short correct answer looks like `"\n\n4"`. And `reasoning_effort`
(`low` / `medium` / `xhigh`) is a per-request field here, unlike some other reasoning models where
it is fixed at server start.
---
## Measured
One RTX PRO 6000 Blackwell, 96 GB, one card. The 1,170-item knowledge suite and the 200-item
tool-calling suite, `temperature 0.6` / `top_p 0.95` / `top_k 20`, thinking on, a 16,384-token
budget, concurrency 32, auto-scored with no LLM judge. Throughput is 8K in / 512 out, prefix-cache
free, two seeds per cell. Every row below was run on the same box in the same sitting.
| build | size | knowledge | tool-calling | call | abstain | finished | tok/s @1 | tok/s @32 |
|---|---|---|---|---|---|---|---|---|
| BF16 original | 70.24 GB | 89.6 | 75.3 | 79.7 | 58.7 | 99.7% | 141.0 | 664 |
| **this repo, FP8 experts** | **38.13 GB** | 90.9 | 75.5 | 79.4 | 60.0 | 99.5% | **163.8** | 901 |
| [mixed](https://huggingface.co/primitive-ai/Nex-N2.5-mini-mixed-NVFP4-FP8) | 26.04 GB | 90.6 | 76.5 | 80.9 | 58.7 | 99.2% | 147.2 | 990 |
| [NVFP4](https://huggingface.co/primitive-ai/Nex-N2.5-mini-NVFP4) | 23.91 GB | 90.5 | 77.0 | 81.9 | 57.5 | 99.3% | 145.3 | **1019** |
Tool-calling is the mean of two runs per build (three for BF16). Pooled within-build spread is
0.60 standard deviations over those nine runs, and the four build means span 1.7, so the column is
one band. Read the ordering with that in mind: it runs opposite to precision, which no mechanism
would predict, and is the clearest evidence that this suite cannot resolve the difference between
4-bit and 16-bit experts on this model.
Speed does separate, and it crosses over. FP8 experts decode 13% faster single-stream because the
4-bit path pays for activation quantization and group scales on every token, a cost that does not
amortize at batch 1. At concurrency 32 the ranking inverts and NVFP4 wins by 13%, because there
the bottleneck is weight bandwidth and 4-bit moves half the bytes. Pick by the concurrency you
actually run.
---
## Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different `temperature`, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from the one fixed protocol described above, the
same 1,370 items, auto-scored, no LLM judge.
| model | shape | size | overall | knowledge | call | abstain | finished | out/answer |
|---|---|---|---|---|---|---|---|---|
| [Laguna-XS-2.1](https://huggingface.co/primitive-ai/Laguna-XS-2.1-mixed-NVFP4-MXFP8) | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 |
| [Nemotron-3.5-Lightning-30B-A3B](https://huggingface.co/primitive-ai/Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8) | 30 B MoE+Mamba | 19.2 GiB | 87.1 | 87.9 | 85.4 | 70.5 | 97.9% | 1429 |
| [Ornith-1.5-35B-A3B](https://huggingface.co/primitive-ai/Ornith-1.5-35B-A3B-mixed-NVFP4-FP8) | 35 B MoE | 22.6 GiB | 88.7 | 91.7 | 74.4 | 60.0 | 99.3% | 760 |
| [Muse-Glimmer-30B](https://huggingface.co/primitive-ai/Muse-Glimmer-30B-mixed-NVFP4) | 30 B MoE | 20.4 GiB | 86.6 | 88.8 | 78.6 | 54.5 | 99.7% | 800 |
| [Qwen3.8-27B](https://huggingface.co/primitive-ai/Qwen3.8-27B-mixed-NVFP4-FP8) | 27 B dense | 20.7 GiB | 88.8 | 90.4 | 85.5 | 54.5 | 99.7% | 651 |
| [Granite-4.2-30B](https://huggingface.co/primitive-ai/granite-4.2-30b-mixed-NVFP4-FP8) | 30 B dense | 18.1 GB | 85.5 | 86.2 | 85.8 | 60.8 | 98.5% | 1502 |
| [Nex-N2.5-mini NVFP4](https://huggingface.co/primitive-ai/Nex-N2.5-mini-NVFP4) | 35 B MoE, 3 B active | 23.91 GB | 88.5 | 90.5 | 81.9 | 57.5 | 99.3% | 524 |
| [Nex-N2.5-mini mixed](https://huggingface.co/primitive-ai/Nex-N2.5-mini-mixed-NVFP4-FP8) | 35 B MoE, 3 B active | 26.04 GB | 88.5 | 90.6 | 80.9 | 58.7 | 99.2% | 504 |
| **Nex-N2.5-mini FP8** (this repo) | 35 B MoE, 3 B active | **38.13 GB** | **88.7** | 90.9 | 79.4 | 60.0 | 99.5% | 545 |
| [K2-Horizon-MoVA-36B-A4B NVFP4](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4) | 37 B MoE+MoVA, 4 B active | 36.7 GB | 84.1 | 86.5 | 71.8 | 60.4 | 95.6% | 1234 |
| [K2-Horizon-MoVA-36B-A4B mixed](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8) | 37 B MoE+MoVA, 4 B active | 44.5 GB | 84.9 | 87.3 | 73.5 | 58.9 | 96.3% | 1118 |
| [Laguna-S-2.1](https://huggingface.co/primitive-ai/Laguna-S-2.1-mixed-NVFP4-MXFP8) | 110 B MoE | 64.0 GiB | 84.3 | 87.1 | 64.6 | 81.0 | 97.3% | 995 |
| [Qwen3.8-Flash-Next](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8) | 180 B MoE, 6 B active | 183.7 GB | 90.3 | 92.2 | 84.8 | 56.7 | 99.5% | 686 |
`overall` pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by
item count. **Read it with `finished`**: `overall` scores an answer that overran the token budget as
wrong, and cannot say whether the model needed the room or failed to stop. A gap under **1.0** in
`overall` is a tie. Sizes are as each card reports them, which mixes GB and GiB.
---
## What's quantized to what
Nex-N2.5-mini is 35.11 B parameters, and they are distributed very unevenly:
| | params | share |
|---|---|---|
| routed experts, 40 layers × 256 × (gate+up+down) | 32.21 B | 91.8% |
| linear attention (Gated DeltaNet), 30 layers | 1.01 B | 2.9% |
| `embed_tokens` and `lm_head`, untied, vocab 248320 | 1.02 B | 2.9% |
| vision tower, 27 blocks | 0.45 B | 1.3% |
| full attention, 10 layers | 0.27 B | 0.8% |
| shared experts, routers, norms | 0.15 B | 0.4% |
| tensors | count | format |
|---|---|---|
| routed experts on all 40 layers | 30,720 modules | FP8 E4M3, per output channel |
| everything else | 946 tensors | BF16, byte-identical to the source |
`compressed-tensors`, format `mixed-precision`, one config group, W8A8: a per-output-channel weight
scale in fp32 and dynamic per-token activation scales. The source ships experts as two batched 3-D
tensors per layer; they are unfolded into per-expert modules, which is the layout vLLM's loader
consumes.
Weights-only round-to-nearest, no calibration. Nothing in the quantization is fitted to data.
---
primitive ·
more models ·
inference economics for production LLM systems