Gemma-4-12B-MXFP4A16-Text-Only

This model is an experimental MXFP4A16 compressed checkpoint derived directly from google/gemma-4-12B-it.

It was quantized from the original BF16 base model, not from the FP8 model. Likely image/audio special-token embeddings were zeroed, and the model is intended to be served as text-only.

Model Details

Field	Value
Base model	google/gemma-4-12B-it
Quantization	MXFP4A16
Quantization tool	llmcompressor
Serving tested with	vLLM
Approx. model folder size	~7.4 GB

Text-Only Changes

This checkpoint disables likely image/audio special-token embeddings by zeroing their embedding rows. The tokenizer entries and architecture remain present so the model can still load with Gemma 4 tooling.

Serve with multimodal inputs blocked:

limit_mm_per_prompt={"image": 0, "audio": 0}

For vLLM, ensure the checkpoint includes:

processor_config.json

Quantization Recipe

Default scheme:

MXFP4A16

Targeted layers:

Linear

Ignored layers/patterns:

lm_head
re:.*embed.*
re:.*router.*
re:.*vision.*
re:.*audio.*
re:.*multimodal.*
re:.*image.*

Intended Use

This checkpoint is intended for:

• text-only generation
• context-grounded QA
• small-footprint serving experiments
• comparison against the FP8 checkpoint

It is not intended for image/audio input.

Usage With vLLM

vllm serve /path/to/model \
  --served-model-name fp4-gemma \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --max-model-len 2048 \
  --gpu-memory-utilization 0.45 \
  --enforce-eager

HotpotQA Benchmark

Benchmark:

Dataset: HotpotQA Distractor
Slice: validation[0:459]
Model name: fp4-gemma
Successful requests: 459/500

System prompt:

Answer questions using only the provided context. Some questions require combining facts from multiple passages. Return only the shortest correct answer. When possible, copy the exact name, date, place, number, or phrase from the context. Do not explain your reasoning or add extra text.

Metric	Value
Questions	459
Successful	459
Exact Correct	294/459
Exact Match	64.05%
Token F1	78.69%
Exact + Partial	401/459 = 87.36%
Avg Latency	0.54s
P50 Latency	0.48s
Max Latency	1.94s
Total Output Tokens	2341
Avg Output Tokens	5.10
Output Tokens/sec	9.46

Limitations

• Experimental text-only quantized checkpoint.
• Serve with multimodal inputs blocked.
• Reported benchmark is HotpotQA Distractor validation[0:459].

License

This checkpoint is derived from google/gemma-4-12B-it. Follow the applicable Gemma 4 terms and Apache 2.0 license information from the base model.