Gemma-4 Dense
Collection
Gemma 4 dense (non-MoE) variants — 31B head-prune + quants. • 7 items • Updated
gemma-4-31B-it-NVFP4A16
NVFP4-A16 (4-bit FP4 weights, BF16 activations) quantization of google/gemma-4-31B-it — the dense 31B Gemma 4 instruction-tuned model.
This is the 31B dense reference baseline for the Gemma 4 26B-A4B vs 31B comparison cohort. Numbers below align with gemma-4-A4B-98e-v5-it (same canonical eval recipe).
Quant recipe
- Quantizer:
omnimergekit/scripts/quantize_any.py --method nvfp4a16 - Engine:
nvidia-modelopt==0.43.0withNVFP4_DEFAULT_CFG - Calibration: 128 samples from
tatsu-lab/alpaca, max_length=512 - Excluded modules:
*vision_tower*,*embed_vision*,*embed_audio*(kept BF16 for vLLM'sgemma4_mmloader) - Hardware: vast.ai RTX 6000 Ada 48GB pod (Türkiye), stack-pinned to vLLM 0.20.2 stock
- Producer stack: transformers 5.5.0, torch 2.10.0+cu128, safetensors 0.7.0
- Date: 2026-05-17
Canonical 9-bench comparison — Gemma 4 cohort (NVFP4A16)
All scores: greedy sampler (temperature=0, top_p=1, top_k=0, do_sample=False), thinking_token_budget=12288, max_gen_toks=16384, vLLM 0.20.2 stock with --reasoning-parser gemma4 --default-chat-template-kwargs '{"enable_thinking": true}'. Bench definitions: omnimergekit/eval/templates/. 128e and v4 columns are the published canonical numbers from the v5-it card.
| Bench (n) | 128e ref | 98e v4 | 98e v5 | 98e v5-coder | 31B-it (this model) | 31B-he1 |
|---|---|---|---|---|---|---|
| GPQA Diamond (198) | 73.23 % | 69.19 % | 68.69 % | 68.69 % | 81.31 % | coming soon |
| GSM8K-100 | 91.00 % | 86.00 % | in flight | 86.00 % | 93.00 % | coming soon |
| MATH-500-100 | 89.00 % | 89.00 % | in flight | 92.00 % | 97.00 % | coming soon |
| AIME 2024 (30) | 36.67 % | 36.67 % | in flight | 36.67 % | 76.67 % | coming soon |
| IFEval-100 (prompt_strict) | 95.00 % | 93.00 % | in flight | 94.00 % | 96.00 % | coming soon |
| HumanEval-164 chat | 96.95 % | 96.95 % | in flight | 98.17 % | 97.56 % | coming soon |
| HumanEval+ chat (164) | 92.07 % | 91.46 % | 89.63 % | 92.68 % | 92.07 % | coming soon |
| LCB-medium-55 (v4 split) | 87.27 % | 78.18 % | in flight | 85.45 % | 96.36 % | coming soon |
| ARC-Challenge chat (1172) | 95.99 % | 95.99 % | 96.59 % | 95.31 % | 98.04 % | coming soon |
Bold = best in row. Both v5-coder and 31B-it ARC rows were rescored 2026-05-18 on stack-pinned hosts (stock vLLM 0.20.2 + Fix-A patched lm-eval): v5-coder = 95.31 %, 31B-it = 98.04 % (now the cohort leader on ARC). The original ⚠ came from an unpatched lm-eval local-chat-completions that discarded vLLM's reasoning_content; the cohort-wide Fix-A reasoning_content fallback is now applied across the cohort.
Key takeaways
- 31B-it dominates the reasoning frontier: GPQA +8.08 pp vs 128e, +12.62 pp vs v5-coder; AIME +40.00 pp vs 128e, +40.00 pp vs v5-coder; MATH +8.00 pp vs 128e; LCB-medium +9.09 pp vs 128e, +10.91 pp vs v5-coder.
- 128e leads HumanEval-164 marginally, but v5-coder takes HumanEval-164 and HumanEval+ — the code-specialist 26B-A4B variant beats both 31B and 128e on code generation at
8× smaller active footprint (4B active params vs 31B / ~10B for 128e MoE active). - AIME shows the architectural gap: 31B-it at 76.67 % vs all 26B-A4B variants at 36.67 % — long-form multi-step math is where the dense 31B's depth pays off.
- GSM8K and IFEval are saturated across the cohort (>90 % for everyone); the 31B-it edge here is small (+2 pp on each).
- Per-FLOP frontier: 98e-v5-coder remains the right pick for code workloads (HumanEval+, LCB) when VRAM or throughput matters; 31B-it is the right pick when reasoning / math / GPQA-style hard benches dominate.
Usage with vLLM
python -m vllm.entrypoints.openai.api_server \
--model ManniX-ITA/Gemma-4-31B-it-NVFP4A16 \
--port 8099 --tensor-parallel-size 1 \
--gpu-memory-utilization 0.70 \
--max-model-len 32768 --dtype bfloat16 --trust-remote-code \
--reasoning-parser gemma4 \
--default-chat-template-kwargs '{"enable_thinking": true}'
Per-request:
{"thinking_token_budget": 12288, "temperature": 0.0, "top_p": 1.0, "top_k": 0}
VRAM
| GPU | Runs? | Memory (peak) | Notes |
|---|---|---|---|
| RTX 3090 24 GB | ❌ | OOM | Single-GPU at 32k ctx exceeds 24 GB; use TP=2 |
| RTX 4090 24 GB | ❌ | OOM | Same — TP=2 required |
| RTX 6000 Ada 48 GB | ✅ | ~32 GB | Single-GPU, 32k ctx |
| L40 / L40S 48 GB | ✅ | ~32 GB | Single-GPU, 32k ctx |
| A100 / H100 80 GB | ✅ | ~32 GB | Single-GPU, comfortable headroom for 65k ctx |
| 2× RTX 3090 / 4090 | ✅ | ~16 GB / GPU | TP=2, 32k+ ctx |
Related models
| Model | Description |
|---|---|
| gemma-4-31b-he1-it | Partial head-prune of 31B-it (L0–L3 Q-prune + lstsq heal). Same NVFP4A16 cohort eval is queued. |
| gemma-4-31b-he1-it-NVFP4A16 | NVFP4A16 quant of the head-pruned variant |
| gemma-4-A4B-98e-v5-it | 26B-A4B MoE variant — v4 + shared FFN α=1.2 |
| gemma-4-A4B-98e-v5-coder-it | 26B-A4B MoE — code-specialist drop map (C6 v4-floor breadth=50) + shared α |
| gemma-4-A4B-98e-v4-it | 26B-A4B MoE — multi-class CD-map drop, baseline for v5 |
Source
- BF16 base:
google/gemma-4-31B-it - Quant script:
omnimergekit/scripts/quantize_any.py - Eval orchestrator:
omnimergekit/scripts/eval_suite_vllm.sh+eval/templates/ - Eval protocol:
omnimergekit/eval/EVAL_PROTOCOL.md(v2, vLLM-first, sampler-pinned greedy, stack-pinned per cohort)
Coming soon
gemma-4-31b-he1-itcolumn — eval queued on pod 36949547 after v5-it GGUF imatrix completes.- v5 remaining 6 benches (GSM8K, MATH, AIME, IFEval, HE-164, LCB) — running on stack-pinned L40 pod.
- ARC-Challenge re-run for 31B-it and 98e-v5-coder with Fix-A reasoning_content patch.
- Downloads last month
- 12