gemma-4-12B-it-6bit-XL-mlx

MLX BaseQuant_XL 6/8-bit mixed quantization of google/gemma-4-12B-it for Apple Silicon.

Architecture	Gemma4UnifiedForConditionalGeneration (dense, no MoE)
Bits per weight	7.118
Total size	~9.9 GB (2 shards)
Framework	mlx_vlm
Multimodal	Vision + audio + text (full)
tie_word_embeddings	True

BaseQuant_XL Strategy

This is the dense XL recipe — precision allocated by layer criticality, not by parameter count.

Bit Depth	Layers	Rationale
bf16 (unquantized)	vision_embedder, embed_vision, embed_audio, norms, scalars	Multimodal bridges — small, quality-critical
8-bit	embed_tokens (tied w/ lm_head), self_attn (q/k/v/o_proj)	Quantization-sensitive, moderate param footprint (~27% of total)
6-bit	mlp (gate/up/down_proj)	Bulk of params (~69%) — sole compression lever in a dense model

Why no bf16 MLP?

In the MoE XL recipe, the shared/dense MLP stays bf16 because routed experts (6-bit) carry the bulk and are sparsely activated. In a dense model, the MLP is the bulk — keeping it at bf16 would negate the purpose of quantization. Instead, we elevate attention from 6-bit to 8-bit (vs the older _68 recipe that quantized both at 6-bit), preserving the component most sensitive to quantization noise.

Quantization Details

Layer	Bits	Group Size
vision_embedder, embed_vision, embed_audio	bf16	—
embed_tokens	8	64
self_attn (q/k/v/o_proj)	8	64
mlp (gate/up/down_proj)	6	64
Default fallback	8	64

• Quantization type: BaseQuant_XL mixed (multimodal, vision+audio preserved)
• Method: Custom quant_predicate via mlx_vlm
• Group size: 64

About XL Quantization

BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — critical layers get higher precision, bulk parameters get lower precision. The result is a transparent, faithful capture of the source model.

Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.

Recommended Inference Parameters

Parameter	Value
temperature	1.0
top_p	1.0
top_k	64
min_p	0.01
repeat_penalty	1.1

Reasoning Parsing in LM Studio and Others

• Start string: <|channel|>thought
• End string: <|channel|>

Usage

pip install -U mlx-vlm

python -m mlx_vlm.generate --model leonsarmiento/gemma-4-12B-it-6bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"

Intelligence Benchmarks (Instruct Mode, n=50)

Benchmark	Samples	Gemma-4 12B 6bit-XL (this)	Gemma-4 12B OptiQ 4-bit	Gemma-4 26B 6bit-XL
MMLU	50/14042	68%	64%	76%
MMLU_PRO	50/12032	76%	74%	82%
ARC_CHALLENGE	50/1172	88%	86%	90%
HUMANEVAL	50/164	94%	90%	98%
MBPP	50/500	72%	72%	82%
Average		79.6%	77.2%	85.6%
bpw		7.118	~4.5	7.061
Size		9.9 GB	~7 GB	21 GB

XL beats data-aware OptiQ 4-bit across the board (+4 MMLU, +2 MMLU_PRO, +2 ARC, +4 HUMANEVAL, +2.4 avg), while the 26B MoE big sister retains the crown as expected (+6 points avg, but 2.1× the size).

⚠️ n=50 sampling means wide confidence intervals (±~13% at 95% CI). Differences under ~6 points may not be statistically significant. Data-aware methods (OptiQ) may be calibrated on benchmark-like data — their scores carry a benchmaxxing caveat. BaseQuant_XL (data-agnostic) provides the most honest generalization estimate.

---

Local SOTA Comparison (for context)

The 26B XL is the current champion for 48GB Macs. The dense 12B offers nearly identical architecture at half the size:

Benchmark	Gemma-4 26B 6bit-XL	Qwen3.6-35B 6bit-XL	Gemma-4 12B 6bit-XL (this)
MMLU	76%	64%	68%
MMLU_PRO	82%	64%	76%
ARC_CHALLENGE	90%	90%	88%
HUMANEVAL	98%	78%	94%
MBPP	82%	78%	72%
Average	85.6%	74.8%	79.6%
Size	21 GB	28 GB	9.9 GB

Chat Template

Ships with Google's canonical chat_template.jinja (18,681 chars) in both chat_template.jinja and tokenizer_config.json. Includes preserve_thinking support for tool-calling workflows.