Brooooooklyn/Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx

Gemma-4-31B-IT — UD-NVFP4_K_XL (mlx-node)

NVFP4 (NVIDIA Blackwell FP4) quantization of google/gemma-4-31b-it for Apple Silicon, using the Unsloth Dynamic quantization strategy via mlx-node.

	Original (BF16)	UD-NVFP4_K_XL (this model)
Size	~62 GB	24 GB
Format	SafeTensors	SafeTensors
Precision	BF16 uniform	NVFP4 (E4M3 scales) + mixed affine + BF16
FFN group size	—	16
Biases	—	no (MLP); yes (affine layers)

What is NVFP4?

NVFP4 is NVIDIA's Blackwell FP4 micro-scaling format. Each group of 16 elements shares a single 8-bit E4M3 scale (a full FP8 number with mantissa, not just an exponent), and elements themselves are stored as E2M1 FP4 values. Compared to MXFP4 (OCP):

• Higher fidelity scale: E4M3 scale has 3 mantissa bits, MXFP4's E8M0 only encodes a power-of-two
• Smaller group: 16 vs. 32 — fewer outliers per scale, better dynamic range tracking
• Higher scale density: 2× the scales per weight (16 elements vs 32 per scale), so per-byte overhead is roughly the same as MXFP4 despite the richer scale type
• No biases: zero-point implicit (FP4 covers ±range)

For dense LLM weights, NVFP4 typically beats MXFP4 on perplexity at the same nominal bit budget, with the trade-off of slightly more metadata per weight.

All Variants

Repo	Bit budget	Size	Decode (tok/s)
Brooooooklyn/Gemma-4-31B-IT-UD-Q2_K_XL-mlx	2-bit base	18 GB	12.1
Brooooooklyn/Gemma-4-31B-IT-UD-Q3_K_XL-mlx	3-bit base	21 GB	10.4
Brooooooklyn/Gemma-4-31B-IT-UD-Q4_K_XL-mlx	4-bit base	24 GB	9.2
Brooooooklyn/Gemma-4-31B-IT-UD-MXFP4_K_XL-mlx	mxfp4	24 GB	10.6
Brooooooklyn/Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx (this model)	nvfp4	24 GB	9.9
Brooooooklyn/Gemma-4-31B-IT-UD-Q5_K_XL-mlx	5-bit base	28 GB	6.8
Brooooooklyn/Gemma-4-31B-IT-UD-Q6_K_XL-mlx	6-bit base	31 GB	7.2
Brooooooklyn/Gemma-4-31B-IT-UD-MXFP8_K_XL-mlx	mxfp8	34 GB	7.8
Brooooooklyn/Gemma-4-31B-IT-UD-Q8_K_XL-mlx	8-bit base	35 GB	7.2

Benchmarked on Apple M3 Max 128GB via examples/lm.ts (best decode tok/s across turns 2–4, steady-state, capitals chat with reasoningEffort: 'low').

Performance

Steady-state decode: 9.9 tok/s on Apple M3 Max 128GB (best of turns 2–4, examples/lm.ts capitals chat with reasoningEffort: 'low'). Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. Gemma-4-31B is fully dense (all 31B parameters active per token), so each decoded token must stream the entire quantized weight footprint from unified memory.

Per-Tensor Bit Assignments (N=4)

Weight	Mode	Bits	Group	Rationale
embed_tokens	6-bit affine	6	64	Tied with lm_head (Gemma4 shares weights); affine-only loader
self_attn.q_proj	6-bit affine	6	64	AWQ-corrected via input_layernorm
self_attn.k_proj	6-bit affine	6	64	AWQ-corrected via input_layernorm
self_attn.v_proj	6-bit affine	6	64	AWQ-corrected via input_layernorm
mlp.gate_proj	nvfp4	4	16	Dense MLP (top-level default) (top-level default)
mlp.up_proj	nvfp4	4	16	Dense MLP (top-level default) (top-level default)
mlp.down_proj	5-bit affine	5	64	Dense MLP; "slightly more sensitive" (unsloth base+1)
embed_vision.embedding_projection	8-bit affine	8	64	Vision adapter; affine-only loader
self_attn.o_proj	bf16	—	—	NOT AWQ-correctable; kept full-precision

Quantization Strategy

Built on Unsloth Dynamic 2.0 per-tensor KLD analysis. At --q-bits 4 (implicit under --q-mode nvfp4) the unsloth recipe assigns 4-bit to MLP gate/up projections, 5-bit affine to down_proj, 6-bit affine + AWQ pre-scaling to attention q/k/v, and keeps self_attn.o_proj as bf16. Then --q-mode nvfp4 orthogonally promotes only the 4-bit affine decisions to NVFP4 (mode="nvfp4", bits=4, group_size=16) — non-4-bit decisions stay affine, and affine-only loaders (embed_tokens, lm_head, embed_vision.embedding_projection) keep their explicit affine overrides.

imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead).

Architecture

Parameter	Value
Total parameters	~31B (fully dense — all parameters active per token)
Hidden size	5,376
Layers	60 (sliding-window attention)
Attention heads	32 (16 KV heads, GQA 2:1)
Head dimension	256
MLP intermediate size	21,504
Vocab size	262,144
Max context	131,072 tokens
Vision	yes (Gemma4ForConditionalGeneration)

Usage

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx');

for await (const event of session.sendStream('Explain the sliding-window attention design in Gemma-4.', {
  config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
  if (!event.done) process.stdout.write(event.text);
}

How It Was Made

mlx convert \
  -i gemma-4-31b-it \
  -o Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx \
  -q --q-mode nvfp4 --q-recipe unsloth \
  --imatrix-path imatrix_unsloth.gguf

Acknowledgments

• Unsloth — Quantization strategy based on their per-layer KLD benchmarks and Dynamic 2.0 methodology
• Google DeepMind — For the Gemma-4 model family
• NVIDIA NVFP4 — For the NVFP4 specification
• Apple MLX — For the Metal-accelerated ML framework

License

Gemma Terms of Use (inherited from base model).