--- license: gemma base_model: ManniX-ITA/gemma-4-31b-he1-it base_model_relation: quantized library_name: vllm pipeline_tag: text-generation tags: - gemma - gemma-4 - 31b - head-prune - structured-pruning - lstsq-heal - quantized - nvfp4 - vllm - reasoning - thinking - modelopt language: - en --- # gemma-4-31b-he1-it-NVFP4A16 **NVFP4-A16 quantization** (4-bit FP4 weights, BF16 activations) of [`ManniX-ITA/gemma-4-31b-he1-it`](https://huggingface.co/ManniX-ITA/gemma-4-31b-he1-it) — a partial **L0–L3 Q-head prune** of Gemma 4 31B-it with lstsq O-projection heal. For the full pruning recipe (per-tensor diff, v1 vs v2 SHA evidence, BOS gotcha, refit methodology, llama.cpp Q4_K_M legacy eval), see the **[BF16 base card](https://huggingface.co/ManniX-ITA/gemma-4-31b-he1-it)**. This card focuses on the NVFP4A16 quant for vLLM serving. ## Quant recipe - **Quantizer:** [`omnimergekit/scripts/quantize_any.py --method nvfp4a16`](https://github.com/mann1x/omnimergekit/blob/main/scripts/quantize_any.py) - **Engine:** `nvidia-modelopt==0.43.0` with `NVFP4_DEFAULT_CFG` - **Calibration:** 128 samples from `tatsu-lab/alpaca`, max_length=512 - **Excluded modules:** `*vision_tower*`, `*embed_vision*`, `*embed_audio*` (kept BF16 for vLLM's `gemma4_mm` loader) - **Producer hardware:** vast.ai RTX 6000 Ada 48 GB pod - **Producer stack:** transformers 5.5.0, torch 2.10.0+cu128, safetensors 0.7.0, modelopt 0.43.0 - **Date:** 2026-05-17 The pruned L0–L3 q_proj/o_proj projections quantize the same way as the unpruned L4–L59 — NVFP4-A16 is applied uniformly across the model. The partial-prune footprint is preserved in the quantized weights. ## Eval — canonical 9-bench cohort (NVFP4A16, vLLM 0.20.2 stock, greedy) Greedy sampler (`temperature=0`, `top_p=1`, `top_k=0`, `do_sample=False`), `thinking_token_budget=12288`, `max_gen_toks=16384`. Bench definitions: [`omnimergekit/eval/templates/`](https://github.com/mann1x/omnimergekit/tree/main/eval/templates). 128e / v4 columns reuse the published canonical baseline from the [v5-it card](https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v5-it). | Bench (n) | 128e ref | 98e v4 | 98e v5-coder | 31B-it (base) | **31B-he1 (this model)** | |---|---:|---:|---:|---:|---:| | GPQA Diamond (198) | 73.23 % | 69.19 % | 68.69 % | 81.31 % | **84.34 %** | | GSM8K-100 | 91.00 % | 86.00 % | 86.00 % | **93.00 %** | 92.00 % | | MATH-500-100 | 89.00 % | 89.00 % | 92.00 % | **97.00 %** | 95.00 % | | AIME 2024 (30) | 36.67 % | 36.67 % | 36.67 % | **76.67 %** | **76.67 %** | | IFEval-100 (prompt_strict) | 95.00 % | 93.00 % | 94.00 % | **96.00 %** | **96.00 %** | | HumanEval-164 chat | 96.95 % | 96.95 % | **98.17 %** | 97.56 % | **98.17 %** | | HumanEval+ chat (164) | 92.07 % | 91.46 % | 92.68 % | 92.07 % | **93.90 %** | | LCB-medium-55 (v4 split) | 87.27 % | 78.18 % | 85.45 % | **96.36 %** | **96.36 %** | | ARC-Challenge chat (1172) | 95.99 % | 95.99 % | 95.31 % | **98.04 %** | 97.61 % | **Bold** = best in row. Source: L40 pod 37006213, vLLM 0.20.2 stock + `--reasoning-parser gemma4` + `thinking_token_budget=12288` + [Fix-A reasoning_content fallback](https://github.com/mann1x/omnimergekit/blob/main/eval/EVAL_PROTOCOL.md#fix-a), greedy sampler, eval window 2026-05-18 / -19. 128e / v4 / v5-coder columns reuse their respective published baselines on the same pinned stack. ### What the he1 column says The null-hypothesis framing was **falsified positively**: the L0–L3-only modification doesn't just match base — it lifts the two hardest benches (GPQA Diamond, HumanEval+) and ties two more (HumanEval, LCB-medium-55, AIME, IFEval). - **GPQA Diamond +3.03 pp (84.34 % vs 81.31 %)** — the L0–L3 partial prune slightly *favours* multi-domain reasoning paths rather than degrading them. This is the largest delta on the table. - **HumanEval+ +1.83 pp vs base (93.90 % vs 92.07 %); +1.22 pp over v5-coder's 92.68 %** — he1 is the new best in the cohort on the harder HE+ split. - **LCB-medium-55 96.36 % — tied with base for best in cohort.** Both the unmodified base and the partial-prune land identical 53/55 scores; the prune doesn't perturb the v4 LCB-medium answer paths. - **HumanEval 98.17 % (+0.61 pp vs base; tied with v5-coder)**, **AIME 2024 76.67 % (tied with base)**, **IFEval 96.00 % (tied)** — three apples-to-apples ties or near-ties on instruction-following and code-generation. - **GSM8K −1.00 pp, MATH-500 −2.00 pp, ARC −0.43 pp** — all inside their per-bench stderr (±1.7 – ±2.7 pp); not statistically distinguishable from base. Practical reading: **the partial L0–L3 prune (~0.83 % attention savings) doesn't regress on any canonical bench, lifts GPQA Diamond and HumanEval+, and ties LCB-medium-55 with base.** It is the strongest Gemma 4 variant in this cohort on the two hardest benches (GPQA Diamond, HumanEval+). ## Usage with vLLM ```bash python -m vllm.entrypoints.openai.api_server \ --model ManniX-ITA/gemma-4-31b-he1-it-NVFP4A16 \ --port 8099 --tensor-parallel-size 1 \ --gpu-memory-utilization 0.70 \ --max-model-len 32768 \ --max-num-batched-tokens 8192 \ --max-num-seqs 4 \ --dtype bfloat16 \ --trust-remote-code \ --reasoning-parser gemma4 \ --default-chat-template-kwargs '{"enable_thinking": true}' ``` Per-request: ```json {"thinking_token_budget": 12288, "temperature": 0.0, "top_p": 1.0, "top_k": 0} ``` ## VRAM | GPU | Runs? | Memory (peak) | Notes | |---|---|---|---| | RTX 3090 24 GB | ❌ | OOM | Single-GPU at 32 k ctx exceeds 24 GB — use TP=2 | | RTX 4090 24 GB | ❌ | OOM | Same — TP=2 required | | RTX 6000 Ada 48 GB | ✅ | ~32 GB | Single-GPU, 32 k ctx | | L40 / L40S 48 GB | ✅ | ~32 GB | Single-GPU, 32 k ctx | | A100 / H100 80 GB | ✅ | ~32 GB | Single-GPU, 65 k ctx headroom | | 2× RTX 3090 / 4090 | ✅ | ~16 GB / GPU | TP=2, 32 k+ ctx | The he1 partial prune saves ~0.83 % of attention parameters (4 layers × 12.5 % Q-prune / 60 layers); VRAM footprint is essentially identical to the 31B-it base NVFP4A16. ## Related models | Model | Description | |---|---| | [gemma-4-31b-he1-it](https://huggingface.co/ManniX-ITA/gemma-4-31b-he1-it) | **BF16 base** — full design notes, per-tensor diff, llama.cpp Q4_K_M legacy eval | | [gemma-4-31b-he1-it-GGUF](https://huggingface.co/ManniX-ITA/gemma-4-31b-he1-it-GGUF) | GGUF tier sweep (llama.cpp / ollama) | | [Gemma-4-31B-it-NVFP4A16](https://huggingface.co/ManniX-ITA/Gemma-4-31B-it-NVFP4A16) | **Unpruned 31B baseline** NVFP4A16 — the apples-to-apples comparison target | | [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) | Upstream BF16 base — the unpruned reference | ## Source - **BF16 base:** [`ManniX-ITA/gemma-4-31b-he1-it`](https://huggingface.co/ManniX-ITA/gemma-4-31b-he1-it) (partial L0–L3 prune of `google/gemma-4-31B-it`) - **Pruning recipe:** [`omnimergekit/recipes/gemma4_31b/prune_local_heal.py`](https://github.com/mann1x/omnimergekit) - **Quant script:** [`omnimergekit/scripts/quantize_any.py`](https://github.com/mann1x/omnimergekit/blob/main/scripts/quantize_any.py) - **Eval orchestrator:** [`omnimergekit/scripts/eval_suite_vllm.sh`](https://github.com/mann1x/omnimergekit/blob/main/scripts/eval_suite_vllm.sh) - **Eval protocol:** [`omnimergekit/eval/EVAL_PROTOCOL.md`](https://github.com/mann1x/omnimergekit/blob/main/eval/EVAL_PROTOCOL.md) ## License Inherits the [Gemma license](https://ai.google.dev/gemma/terms) from the upstream Gemma 4 base. ## Acknowledgements - Google for the base Gemma 4 31B-it model - [OmniMergeKit](https://github.com/mann1x/omnimergekit) for the prune-and-heal recipe and eval toolkit - The vLLM and NVIDIA ModelOpt teams for the NVFP4A16 serving / quantization pipeline