| --- |
| language: |
| - en |
| - zh |
| library_name: vllm |
| pipeline_tag: text-generation |
| tags: |
| - nvfp4 |
| - moe |
| - quantized |
| - vllm |
| - blackwell |
| - nvidia |
| - modelopt |
| - base_model:nex-agi/Nex-N2-mini |
| - base_model:quantized:nex-agi/Nex-N2-mini |
| license: apache-2.0 |
| base_model: nex-agi/Nex-N2-mini |
| --- |
| |
| # Nex-N2-mini NVFP4 |
|
|
| NVFP4-quantized [Nex-N2-mini](https://huggingface.co/Nex-AGI/Nex-N2-mini) (Qwen3.5-MoE-35B fine-tune) optimized for NVIDIA Blackwell (SM 12.1) serving via vLLM with FlashInfer CUTLASS kernels. |
|
|
| **3.2× compression** (70 GB BF16 → 22.1 GiB NVFP4) with zero quality loss on capability tests. |
|
|
| ## Quick Start (Docker) |
|
|
| The easiest way to serve this model — auto-downloads on first run: |
|
|
| ```bash |
| docker run -d --name nex-n2-mini-nvfp4 \ |
| --gpus all \ |
| --shm-size=8g \ |
| -e HF_TOKEN=hf_xxxxx \ |
| -v nex-n2-model:/mnt/model \ |
| -p 8000:8000 \ |
| ghcr.io/r0b0tlab/nex-n2-mini-nvfp4:latest |
| ``` |
|
|
| Then query: |
| ```bash |
| curl http://localhost:8000/v1/chat/completions \ |
| -H "Content-Type: application/json" \ |
| -d '{"model":"r0b0tlab/nex-n2-mini-nvfp4","messages":[{"role":"user","content":"Hello!"}],"max_tokens":100}' |
| ``` |
|
|
| **Docker Compose:** |
| ```bash |
| echo "HF_TOKEN=hf_xxxxx" > .env |
| docker compose up -d |
| ``` |
|
|
| See the [GitHub repo](https://github.com/r0b0tlab/nex-n2-mini-nvfp4) for full documentation, environment variables, and AGENTS.md. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | Base model | [Nex-AGI/Nex-N2-mini](https://huggingface.co/Nex-AGI/Nex-N2-mini) | |
| | Architecture | Qwen3_5MoeForConditionalGeneration | |
| | Parameters | 35B total / 3B active (MoE, 256 experts, top-8 routing) | |
| | Layers | 40 (30 linear attention + 10 full attention) | |
| | Vocabulary | 248,320 tokens | |
| | Vision encoder | ViT (27 blocks, 1152 hidden) — kept BF16 | |
| | Original size | ~70 GB (BF16) | |
| | NVFP4 size | ~22.1 GiB | |
| |
| ## Quantization |
| |
| - **Method**: NVFP4 via [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.44.0 |
| - **Scope**: MLP-only (expert projections + shared expert) |
| - **Group size**: 16 |
| - **Calibration**: 128 samples from CNN/DailyMail |
| - **KV cache**: FP8 e4m3 with per-layer calibrated scales |
| - **Expert calibration**: 10,240/10,240 (100%) |
| |
| **Included in checkpoint:** |
| - NVFP4 quantized MoE weights (256 experts × 40 layers) |
| - BF16 attention weights (self_attn, linear_attn) |
| - BF16 vision encoder weights |
| - BF16 lm_head.weight |
| - Calibrated FP8 KV cache scales (k: 0.016–0.038, v: 0.010–0.040, q: 0.043) |
|
|
| ## Benchmarks |
|
|
| Tested on NVIDIA GB10 (Blackwell SM 12.1), vLLM v0.22.0, FlashInfer CUTLASS NVFP4. |
|
|
| ### Throughput (llama-benchy, 3 runs per test) |
|
|
| | Test | Throughput | Peak t/s | |
| |---|---|---| |
| | pp2048 | 1,974 t/s | — | |
| | tg128 | **33.35 t/s** | 38.33 | |
| | pp2048 @ d4096 | 4,007 t/s | — | |
| | pp2048 @ d8192 | 4,793 t/s | — | |
| | pp2048 @ d16384 | 5,017 t/s | — | |
|
|
| Decode is rock-stable at 32–33 t/s across all context depths (0–16K). Only 2.8% degradation. |
|
|
| ### Concurrency Scaling |
|
|
| | Concurrency | Aggregate t/s | Per-request t/s | Power | Temp | |
| |---|---|---|---|---| |
| | C1 | 28.5 | 28.6 | 20.4 W | 45°C | |
| | C2 | 51.6 | 25.8 | 18.4 W | 46°C | |
| | C4 | 105.3 | 26.3 | 20.2 W | 47°C | |
| | C8 | **185.5** | 23.2 | 22.1 W | 48°C | |
|
|
| 6.5× scaling at C8. 8.42 t/s/W efficiency. Peak 23.3W at 48°C. |
|
|
| ### Capability Tests: 13/13 Passed |
|
|
| Math (3/3), Reasoning (3/3), Coding (3/3), Knowledge (2/2), Instruction (2/2). |
|
|
| ## Serving Stack |
|
|
| - **vLLM** v0.22.0 (Docker, aarch64) |
| - **FlashInfer CUTLASS** NVFP4 GEMM kernel + MoE backend |
| - **FP8 KV cache** with per-layer calibrated scales (~10–15% throughput improvement) |
|
|
| ## Known Issues |
|
|
| - `max_num_batched_tokens` must be >= 2096 (Mamba block alignment) |
| - NVFP4 KV cache unavailable (requires `torch.nvfp4`, NVIDIA-internal only) |
| - Vision encoder profiling takes ~4–5 min on first startup |
| - Requires NVIDIA Blackwell GPU (SM 12.1) for NVFP4 kernels |
|
|
| ## Links |
|
|
| - **Container**: [ghcr.io/r0b0tlab/nex-n2-mini-nvfp4](https://github.com/r0b0tlab/nex-n2-mini-nvfp4/pkgs/container/nex-n2-mini-nvfp4) |
| - **Source code**: [github.com/r0b0tlab/nex-n2-mini-nvfp4](https://github.com/r0b0tlab/nex-n2-mini-nvfp4) |
| - **Base model**: [Nex-AGI/Nex-N2-mini](https://huggingface.co/Nex-AGI/Nex-N2-mini) |
| - **Quantization**: [NVIDIA ModelOpt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) |
| - **Serving**: [vLLM](https://github.com/vllm-project/vllm) |
|
|
| ## License |
|
|
| Apache 2.0. |
|
|
