--- license: gemma base_model: google/gemma-4-31B-it base_model_relation: quantized library_name: vllm pipeline_tag: text-generation tags: - gemma - gemma-4 - 31b - quantized - nvfp4 - vllm - reasoning - thinking - modelopt language: - en --- # gemma-4-31B-it-NVFP4A16 NVFP4-A16 (4-bit FP4 weights, BF16 activations) quantization of [`google/gemma-4-31B-it`](https://huggingface.co/google/gemma-4-31B-it) — the dense 31B Gemma 4 instruction-tuned model. This is the **31B dense reference baseline** for the Gemma 4 26B-A4B vs 31B comparison cohort. Numbers below align with [`gemma-4-A4B-98e-v5-it`](https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v5-it) (same canonical eval recipe). ## Quant recipe - **Quantizer:** [`omnimergekit/scripts/quantize_any.py --method nvfp4a16`](https://github.com/mann1x/omnimergekit/blob/main/scripts/quantize_any.py) - **Engine:** `nvidia-modelopt==0.43.0` with `NVFP4_DEFAULT_CFG` - **Calibration:** 128 samples from `tatsu-lab/alpaca`, max_length=512 - **Excluded modules:** `*vision_tower*`, `*embed_vision*`, `*embed_audio*` (kept BF16 for vLLM's `gemma4_mm` loader) - **Hardware:** vast.ai RTX 6000 Ada 48GB pod (Türkiye), stack-pinned to vLLM 0.20.2 stock - **Producer stack:** transformers 5.5.0, torch 2.10.0+cu128, safetensors 0.7.0 - **Date:** 2026-05-17 ## Canonical 9-bench comparison — Gemma 4 cohort (NVFP4A16) All scores: greedy sampler (`temperature=0`, `top_p=1`, `top_k=0`, `do_sample=False`), `thinking_token_budget=12288`, `max_gen_toks=16384`, vLLM 0.20.2 stock with `--reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking": true}'`. Bench definitions: [`omnimergekit/eval/templates/`](https://github.com/mann1x/omnimergekit/tree/main/eval/templates). 128e and v4 columns are the published canonical numbers from the [v5-it card](https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v5-it). | Bench (n) | 128e ref | 98e v4 | 98e v5 | 98e v5-coder | **31B-it (this model)** | 31B-he1 | |---|---:|---:|---:|---:|---:|---:| | GPQA Diamond (198) | 73.23 % | 69.19 % | 68.69 % | 68.69 % | **81.31 %** | _coming soon_ | | GSM8K-100 | 91.00 % | 86.00 % | _in flight_ | 86.00 % | **93.00 %** | _coming soon_ | | MATH-500-100 | 89.00 % | 89.00 % | _in flight_ | 92.00 % | **97.00 %** | _coming soon_ | | AIME 2024 (30) | 36.67 % | 36.67 % | _in flight_ | 36.67 % | **76.67 %** | _coming soon_ | | IFEval-100 (prompt_strict) | 95.00 % | 93.00 % | _in flight_ | 94.00 % | **96.00 %** | _coming soon_ | | HumanEval-164 chat | 96.95 % | 96.95 % | _in flight_ | **98.17 %** | 97.56 % | _coming soon_ | | HumanEval+ chat (164) | 92.07 % | 91.46 % | 89.63 % | **92.68 %** | 92.07 % | _coming soon_ | | LCB-medium-55 (v4 split) | 87.27 % | 78.18 % | _in flight_ | 85.45 % | **96.36 %** | _coming soon_ | | ARC-Challenge chat (1172) | 95.99 % | 95.99 % | 96.59 % | 95.31 % | **98.04 %** | _coming soon_ | **Bold** = best in row. Both v5-coder and 31B-it ARC rows were rescored 2026-05-18 on stack-pinned hosts (stock vLLM 0.20.2 + Fix-A patched lm-eval): v5-coder = **95.31 %**, 31B-it = **98.04 %** (now the cohort leader on ARC). The original ⚠ came from an unpatched lm-eval `local-chat-completions` that discarded vLLM's `reasoning_content`; the cohort-wide [Fix-A reasoning_content fallback](https://github.com/mann1x/omnimergekit/blob/main/eval/EVAL_PROTOCOL.md#fix-a) is now applied across the cohort. ### Key takeaways - **31B-it dominates the reasoning frontier:** GPQA +8.08 pp vs 128e, +12.62 pp vs v5-coder; AIME +40.00 pp vs 128e, +40.00 pp vs v5-coder; MATH +8.00 pp vs 128e; LCB-medium +9.09 pp vs 128e, +10.91 pp vs v5-coder. - **128e leads HumanEval-164 marginally**, but **v5-coder takes HumanEval-164 and HumanEval+** — the code-specialist 26B-A4B variant beats both 31B and 128e on code generation at ~8× smaller active footprint (~4B active params vs 31B / ~10B for 128e MoE active). - **AIME shows the architectural gap:** 31B-it at 76.67 % vs all 26B-A4B variants at 36.67 % — long-form multi-step math is where the dense 31B's depth pays off. - **GSM8K and IFEval are saturated** across the cohort (>90 % for everyone); the 31B-it edge here is small (+2 pp on each). - **Per-FLOP frontier:** 98e-v5-coder remains the right pick for code workloads (HumanEval+, LCB) when VRAM or throughput matters; 31B-it is the right pick when reasoning / math / GPQA-style hard benches dominate. ## Usage with vLLM ```bash python -m vllm.entrypoints.openai.api_server \ --model ManniX-ITA/Gemma-4-31B-it-NVFP4A16 \ --port 8099 --tensor-parallel-size 1 \ --gpu-memory-utilization 0.70 \ --max-model-len 32768 --dtype bfloat16 --trust-remote-code \ --reasoning-parser gemma4 \ --default-chat-template-kwargs '{"enable_thinking": true}' ``` Per-request: ```json {"thinking_token_budget": 12288, "temperature": 0.0, "top_p": 1.0, "top_k": 0} ``` ## VRAM | GPU | Runs? | Memory (peak) | Notes | |---|---|---|---| | RTX 3090 24 GB | ❌ | OOM | Single-GPU at 32k ctx exceeds 24 GB; use TP=2 | | RTX 4090 24 GB | ❌ | OOM | Same — TP=2 required | | RTX 6000 Ada 48 GB | ✅ | ~32 GB | Single-GPU, 32k ctx | | L40 / L40S 48 GB | ✅ | ~32 GB | Single-GPU, 32k ctx | | A100 / H100 80 GB | ✅ | ~32 GB | Single-GPU, comfortable headroom for 65k ctx | | 2× RTX 3090 / 4090 | ✅ | ~16 GB / GPU | TP=2, 32k+ ctx | ## Related models | Model | Description | |---|---| | [gemma-4-31b-he1-it](https://huggingface.co/ManniX-ITA/gemma-4-31b-he1-it) | Partial head-prune of 31B-it (L0–L3 Q-prune + lstsq heal). Same NVFP4A16 cohort eval is queued. | | [gemma-4-31b-he1-it-NVFP4A16](https://huggingface.co/ManniX-ITA/gemma-4-31b-he1-it-NVFP4A16) | NVFP4A16 quant of the head-pruned variant | | [gemma-4-A4B-98e-v5-it](https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v5-it) | 26B-A4B MoE variant — v4 + shared FFN α=1.2 | | [gemma-4-A4B-98e-v5-coder-it](https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v5-coder-it) | 26B-A4B MoE — code-specialist drop map (C6 v4-floor breadth=50) + shared α | | [gemma-4-A4B-98e-v4-it](https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v4-it) | 26B-A4B MoE — multi-class CD-map drop, baseline for v5 | ## Source - **BF16 base:** [`google/gemma-4-31B-it`](https://huggingface.co/google/gemma-4-31B-it) - **Quant script:** [`omnimergekit/scripts/quantize_any.py`](https://github.com/mann1x/omnimergekit/blob/main/scripts/quantize_any.py) - **Eval orchestrator:** [`omnimergekit/scripts/eval_suite_vllm.sh`](https://github.com/mann1x/omnimergekit/blob/main/scripts/eval_suite_vllm.sh) + [`eval/templates/`](https://github.com/mann1x/omnimergekit/tree/main/eval/templates) - **Eval protocol:** [`omnimergekit/eval/EVAL_PROTOCOL.md`](https://github.com/mann1x/omnimergekit/blob/main/eval/EVAL_PROTOCOL.md) (v2, vLLM-first, sampler-pinned greedy, stack-pinned per cohort) ## Coming soon - `gemma-4-31b-he1-it` column — eval queued on pod 36949547 after v5-it GGUF imatrix completes. - v5 remaining 6 benches (GSM8K, MATH, AIME, IFEval, HE-164, LCB) — running on stack-pinned L40 pod. - ARC-Challenge re-run for 31B-it and 98e-v5-coder with [Fix-A reasoning_content patch](https://github.com/mann1x/omnimergekit/blob/main/eval/EVAL_PROTOCOL.md#fix-a).