pearsonkyle/gemma-4-E4B-imatrix-awq-GGUF

Gemma-4 E4B — Imatrix vs AWQ Quantization Study

A 12-quant matrix comparing imatrix and AWQ calibration methods across three quantization types (IQ2_M, IQ3_M, IQ4_XS) on two same-caliber models:

• google/gemma-4-E4B-it — base E4B (instruction-tuned)
• google/gemma-4-E4B-it-qat-q4_0-unquantized — QAT (quantization-aware trained)

All quants were calibrated on 15M tokens of pearsonkyle/llmtk-sft-corpus-v2 (32k context, seed-42 shuffle, 9,597 train sessions) and benchmarked on pearsonkyle/broad-domain-supplement (general-domain + instruct/tools splits).

Results

Model	Quant	Method	Size (GiB)	BPW	KLD (general)	KLD (instruct)	Top-P (general)	Top-P (instruct)	Decode (tok/s)
gemma-4-E4B-it	IQ2_M	imatrix	3.550	4.056	1.461	0.978	50.0	61.8	50.3
gemma-4-E4B-it	IQ2_M	AWQ	3.527	4.030	12.266	13.830	0.1	0.1	69.9
gemma-4-E4B-it	IQ3_M	imatrix	4.391	5.017	0.342	0.222	74.9	81.1	68.8
gemma-4-E4B-it	IQ3_M	AWQ	4.365	4.988	16.393	17.650	0.3	0.3	63.5
gemma-4-E4B-it	IQ4_XS	imatrix	4.723	5.396	0.149	0.094	82.8	87.6	59.5
gemma-4-E4B-it	IQ4_XS	AWQ	4.691	5.360	16.443	17.562	0.03	0.02	40.8
gemma-4-E4B-it-qat	IQ2_M	imatrix	3.527	4.060	4.888	4.940	21.9	23.8	76.2
gemma-4-E4B-it-qat	IQ2_M	AWQ	3.527	4.060	18.363	20.090	0.1	0.0	65.1
gemma-4-E4B-it-qat	IQ3_M	imatrix	4.365	5.025	0.255	0.205	76.6	80.1	56.6
gemma-4-E4B-it-qat	IQ3_M	AWQ	4.365	5.025	15.942	17.118	0.2	0.1	59.5
gemma-4-E4B-it-qat	IQ4_XS	imatrix	4.691	5.400	0.057	0.049	88.0	89.8	86.1
gemma-4-E4B-it-qat	IQ4_XS	AWQ	4.691	5.400	14.067	15.494	0.9	0.7	68.4

Key findings

1. Imatrix dominates AWQ for gemma-4 E4B across every quant type. KLD is 5–120× lower with imatrix; top-p agreement is 100× higher.
2. IQ4_XS imatrix is the sweet spot: near-FP16 KLD (0.057–0.149) at 4.7 GiB, 86–88% top-p agreement.
3. IQ3_M imatrix is the best size/quality tradeoff: KLD 0.22–0.34 at 4.4 GiB, 75–81% top-p.
4. IQ2_M imatrix is viable at 3.5 GiB (KLD 1.0–1.5, 50–62% top-p) but AWQ at 2-bit is unusable.
5. QAT model benefits more from imatrix at higher bit rates (IQ3_M, IQ4_XS) — the QAT IQ4_XS imatrix achieves the best KLD of the entire matrix (0.057).
6. AWQ is unusable for this model — logit drift is too large (F16 reference KLD > 12 on all AWQ quants), likely due to the model's GQA architecture (4G GQA, 24 KV-grouped layers) which makes the RMSNorm fold less effective.

Files

File	Type	Method	Size
gemma-4-E4B-it-IQ2_M-imatrix.gguf	IQ2_M	imatrix	3.55 GiB
gemma-4-E4B-it-IQ2_M-awq.gguf	IQ2_M	AWQ	3.53 GiB
gemma-4-E4B-it-IQ3_M-imatrix.gguf	IQ3_M	imatrix	4.39 GiB
gemma-4-E4B-it-IQ3_M-awq.gguf	IQ3_M	AWQ	4.37 GiB
gemma-4-E4B-it-IQ4_XS-imatrix.gguf	IQ4_XS	imatrix	4.72 GiB
gemma-4-E4B-it-IQ4_XS-awq.gguf	IQ4_XS	AWQ	4.69 GiB
gemma-4-E4B-it-qat-IQ2_M-imatrix.gguf	IQ2_M	imatrix	3.53 GiB
gemma-4-E4B-it-qat-IQ2_M-awq.gguf	IQ2_M	AWQ	3.53 GiB
gemma-4-E4B-it-qat-IQ3_M-imatrix.gguf	IQ3_M	imatrix	4.37 GiB
gemma-4-E4B-it-qat-IQ3_M-awq.gguf	IQ3_M	AWQ	4.37 GiB
gemma-4-E4B-it-qat-IQ4_XS-imatrix.gguf	IQ4_XS	imatrix	4.69 GiB
gemma-4-E4B-it-qat-IQ4_XS-awq.gguf	IQ4_XS	AWQ	4.69 GiB

Methodology

• Calibration: 15M tokens from pearsonkyle/llmtk-sft-corpus-v2 32k split (seed-42 shuffle, 9,597 sessions, 32k context). Imatrix collected via llama-imatrix -c 32768 --parse-special; hybrid_custom variant re-weights per-tensor. AWQ α-search via proxy quantizer, imatrix collected on folded F16.
• GQA fix: gemma-4 has 4G GQA (8 Q / 2 KV heads, 42 layers, 24 KV-grouped layers). llama-imatrix only collects K/V stats for layers 0–23; a _backfill_missing_kv_layers patch copies per-channel mean of collected K/V vectors onto the 18 missing layers.
• Evaluation: KLD + perplexity + top-p agreement via llama-perplexity on pearsonkyle/broad-domain-supplement (general-domain: 30,710 tokens; instruct/tools: 30,853 tokens). Speed via llama-bench.
• Machine: Apple Silicon (MPS), llama.cpp vendored build.