---
library_name: mlx
license: apache-2.0
license_link: https://ai.google.dev/gemma/docs/gemma_4_license
pipeline_tag: image-text-to-text
tags:
- mlx
base_model: google/gemma-4-31B-it
---

[Gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) optimized for MLX.
This quant supports image input and requires a vision-enabled MLX server.

# Usage

```sh
# Start server at http://localhost:8080/chat/completions
uvx --from mlx-vlm --with torchvision \
  mlx_vlm.server \
  --host 127.0.0.1 \
  --port 8080 \
  --model spicyneuron/Gemma-4-31B-MLX-4.9bit-vision
```

# Methodology

Quantized using a custom script inspired by Unsloth/AesSedai/ubergarm style mixed-precision GGUFs.
MLX quantization options differ than llama.cpp, but the principles are the same:

- Sensitive layers like MoE routing, attention, and output embeddings get higher precision
- More tolerant layers like MoE experts get lower precision

# Benchmarks

metric | unsloth_gemma-4-31b-it-UD-MLX-4bit | 4.9 bit (this model)
--- | --- | ---
bpw | 5.765 | 4.904
prompt processing (1024) | 354.269 | 355.741
token gen (512) | 24.842 | 28.376
peak mem | 23.700 | 20.441
perplexity | 35.710 ± 0.201 | 31.903 ± 0.249
hellaswag | 0.53 ± 0.011 | 0.534 ± 0.011
piqa | 0.736 ± 0.01 | 0.748 ± 0.01
winogrande | 0.664 ± 0.013 | 0.665 ± 0.013

- Bits per weight calculated against only the `language_model` weights.
- Perplexity in Gemma 4 was surprisingly high but seemed consistent across my trials.
  Could be a side effect of using `allenai/tulu-3-sft-mixture`.
  Best to interpret it as weaker signal than the other benchmark results.

Tested with:

```sh
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.perplexity --sequence-length 2048 --seed 123
mlx_lm.evaluate --tasks hellaswag --seed 123 --num-shots 0 --limit 2000
mlx_lm.evaluate --tasks piqa --seed 123 --num-shots 0 --limit 2000
mlx_lm.evaluate --tasks winogrande --seed 123 --num-shots 0 --limit 2000
```