gemma-4-31b-he1-it-NVFP4A16

NVFP4-A16 quantization (4-bit FP4 weights, BF16 activations) of ManniX-ITA/gemma-4-31b-he1-it — a partial L0–L3 Q-head prune of Gemma 4 31B-it with lstsq O-projection heal.

For the full pruning recipe (per-tensor diff, v1 vs v2 SHA evidence, BOS gotcha, refit methodology, llama.cpp Q4_K_M legacy eval), see the BF16 base card. This card focuses on the NVFP4A16 quant for vLLM serving.

Quant recipe

Quantizer: omnimergekit/scripts/quantize_any.py --method nvfp4a16
Engine: nvidia-modelopt==0.43.0 with NVFP4_DEFAULT_CFG
Calibration: 128 samples from tatsu-lab/alpaca, max_length=512
Excluded modules: *vision_tower*, *embed_vision*, *embed_audio* (kept BF16 for vLLM's gemma4_mm loader)
Producer hardware: vast.ai RTX 6000 Ada 48 GB pod
Producer stack: transformers 5.5.0, torch 2.10.0+cu128, safetensors 0.7.0, modelopt 0.43.0
Date: 2026-05-17

The pruned L0–L3 q_proj/o_proj projections quantize the same way as the unpruned L4–L59 — NVFP4-A16 is applied uniformly across the model. The partial-prune footprint is preserved in the quantized weights.

Eval — canonical 9-bench cohort (NVFP4A16, vLLM 0.20.2 stock, greedy)

Greedy sampler (temperature=0, top_p=1, top_k=0, do_sample=False), thinking_token_budget=12288, max_gen_toks=16384. Bench definitions: omnimergekit/eval/templates/. 128e / v4 columns reuse the published canonical baseline from the v5-it card.

Bench (n)	128e ref	98e v4	98e v5-coder	31B-it (base)	31B-he1 (this model)
GPQA Diamond (198)	73.23 %	69.19 %	68.69 %	81.31 %	84.34 %
GSM8K-100	91.00 %	86.00 %	86.00 %	93.00 %	92.00 %
MATH-500-100	89.00 %	89.00 %	92.00 %	97.00 %	95.00 %
AIME 2024 (30)	36.67 %	36.67 %	36.67 %	76.67 %	76.67 %
IFEval-100 (prompt_strict)	95.00 %	93.00 %	94.00 %	96.00 %	96.00 %
HumanEval-164 chat	96.95 %	96.95 %	98.17 %	97.56 %	98.17 %
HumanEval+ chat (164)	92.07 %	91.46 %	92.68 %	92.07 %	93.90 %
LCB-medium-55 (v4 split)	87.27 %	78.18 %	85.45 %	96.36 %	96.36 %
ARC-Challenge chat (1172)	95.99 %	95.99 %	95.31 %	98.04 %	97.61 %

Bold = best in row. Source: L40 pod 37006213, vLLM 0.20.2 stock + --reasoning-parser gemma4 + thinking_token_budget=12288 + Fix-A reasoning_content fallback, greedy sampler, eval window 2026-05-18 / -19. 128e / v4 / v5-coder columns reuse their respective published baselines on the same pinned stack.

What the he1 column says

The null-hypothesis framing was falsified positively: the L0–L3-only modification doesn't just match base — it lifts the two hardest benches (GPQA Diamond, HumanEval+) and ties two more (HumanEval, LCB-medium-55, AIME, IFEval).

GPQA Diamond +3.03 pp (84.34 % vs 81.31 %) — the L0–L3 partial prune slightly favours multi-domain reasoning paths rather than degrading them. This is the largest delta on the table.
HumanEval+ +1.83 pp vs base (93.90 % vs 92.07 %); +1.22 pp over v5-coder's 92.68 % — he1 is the new best in the cohort on the harder HE+ split.
LCB-medium-55 96.36 % — tied with base for best in cohort. Both the unmodified base and the partial-prune land identical 53/55 scores; the prune doesn't perturb the v4 LCB-medium answer paths.
HumanEval 98.17 % (+0.61 pp vs base; tied with v5-coder), AIME 2024 76.67 % (tied with base), IFEval 96.00 % (tied) — three apples-to-apples ties or near-ties on instruction-following and code-generation.
GSM8K −1.00 pp, MATH-500 −2.00 pp, ARC −0.43 pp — all inside their per-bench stderr (±1.7 – ±2.7 pp); not statistically distinguishable from base.

Practical reading: the partial L0–L3 prune (~0.83 % attention savings) doesn't regress on any canonical bench, lifts GPQA Diamond and HumanEval+, and ties LCB-medium-55 with base. It is the strongest Gemma 4 variant in this cohort on the two hardest benches (GPQA Diamond, HumanEval+).

Usage with vLLM

python -m vllm.entrypoints.openai.api_server \
    --model ManniX-ITA/gemma-4-31b-he1-it-NVFP4A16 \
    --port 8099 --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.70 \
    --max-model-len 32768 \
    --max-num-batched-tokens 8192 \
    --max-num-seqs 4 \
    --dtype bfloat16 \
    --trust-remote-code \
    --reasoning-parser gemma4 \
    --default-chat-template-kwargs '{"enable_thinking": true}'

Per-request:

{"thinking_token_budget": 12288, "temperature": 0.0, "top_p": 1.0, "top_k": 0}

VRAM

GPU	Runs?	Memory (peak)	Notes
RTX 3090 24 GB	❌	OOM	Single-GPU at 32 k ctx exceeds 24 GB — use TP=2
RTX 4090 24 GB	❌	OOM	Same — TP=2 required
RTX 6000 Ada 48 GB	✅	~32 GB	Single-GPU, 32 k ctx
L40 / L40S 48 GB	✅	~32 GB	Single-GPU, 32 k ctx
A100 / H100 80 GB	✅	~32 GB	Single-GPU, 65 k ctx headroom
2× RTX 3090 / 4090	✅	~16 GB / GPU	TP=2, 32 k+ ctx

The he1 partial prune saves ~0.83 % of attention parameters (4 layers × 12.5 % Q-prune / 60 layers); VRAM footprint is essentially identical to the 31B-it base NVFP4A16.

Related models

Model	Description
gemma-4-31b-he1-it	BF16 base — full design notes, per-tensor diff, llama.cpp Q4_K_M legacy eval
gemma-4-31b-he1-it-GGUF	GGUF tier sweep (llama.cpp / ollama)
Gemma-4-31B-it-NVFP4A16	Unpruned 31B baseline NVFP4A16 — the apples-to-apples comparison target
google/gemma-4-31B-it	Upstream BF16 base — the unpruned reference

Source

BF16 base: ManniX-ITA/gemma-4-31b-he1-it (partial L0–L3 prune of google/gemma-4-31B-it)
Pruning recipe: omnimergekit/recipes/gemma4_31b/prune_local_heal.py
Quant script: omnimergekit/scripts/quantize_any.py
Eval orchestrator: omnimergekit/scripts/eval_suite_vllm.sh
Eval protocol: omnimergekit/eval/EVAL_PROTOCOL.md

License

Inherits the Gemma license from the upstream Gemma 4 base.

Acknowledgements

Google for the base Gemma 4 31B-it model
OmniMergeKit for the prune-and-heal recipe and eval toolkit
The vLLM and NVIDIA ModelOpt teams for the NVFP4A16 serving / quantization pipeline