jokernifty/gemma-4-12B-it-mlx-4bit-multimodal

gemma-4-12B-it — MLX 4-bit, multimodal weights preserved

The first MLX conversion of google/gemma-4-12B-it that preserves the encoder-free vision_embedder and audio_embedder weights, not just the text backbone.

Status: Text inference works today. Vision + audio inference requires the companion package mlx-gemma4-unified (preview — image/audio preprocessors are next).

Why this exists

google/gemma-4-12B-it is Google's unique encoder-free Gemma 4 model: text, image, and audio inputs all flow directly into the LLM via lightweight embedder modules (no SigLIP or Conformer tower). When this model dropped, mlx-community published every other Gemma 4 size (e2b/e4b/26b-a4b/31b) within 48h, but skipped 12B — because mlx-lm and mlx-vlm didn't know about the gemma4_unified architecture, and the encoder-free embedder weights got stripped by every available converter.

This checkpoint keeps them all.

What's in the box

Component	Format	Size
Token embedding (262k × 3840)	bf16	2.0 GB
48 transformer layers (KV-shared, sliding/full alternating)	4-bit, group_size 64	~5.7 GB
embed_vision (patch_ln + patch_dense + pos_embedding + pos_norm + projection)	bf16	~30 MB
embed_audio (scaleless RMSNorm + Linear 640→3840)	bf16	~5 MB
Tokenizer + chat template + processor config	—	<10 MB
Total	—	~8.2 GB across 5 shards

The token embedding stays in bf16 because quantizing 262k × 3840 in one Metal command exceeded the macOS GPU watchdog (5s). The multimodal embedders are tiny and also kept bf16 for fidelity.

Quick start — text inference (works today)

pip install mlx-lm --break-system-packages

Then either:

Option A — Use the mlx-gemma4-unified package (recommended; loads multimodal weights without warnings):

import json, mlx.core as mx, mlx.nn as nn
from mlx_gemma4_unified.model import Gemma4UnifiedModel, ModelArgs

src = "jokernifty/gemma-4-12B-it-mlx-4bit-multimodal"  # or local path after hf download
# (full loader code in the repo)

Option B — Use mlx-lm with a 4-line alias (text-only; ignores the multimodal weights):

# Drop this at mlx_lm/models/gemma4_unified.py
from . import gemma4
ModelArgs = gemma4.ModelArgs
_SKIP = ("vision_embedder.", "embed_vision.", "embed_audio.")

class Model(gemma4.Model):
    def sanitize(self, weights):
        f = {k: v for k, v in weights.items()
             if not any((k[6:] if k.startswith("model.") else k).startswith(p) for p in _SKIP)}
        return super().sanitize(f)

Then:

python -m mlx_lm chat --model jokernifty/gemma-4-12B-it-mlx-4bit-multimodal

Vision + audio inference (work in progress)

The model weights are all here, but you need the image/audio preprocessors to actually use them. Those live in mlx-gemma4-unified:

• ✅ VisionEmbedder MLX module — implemented, tested
• ✅ AudioEmbedder MLX module — implemented, tested
• ✅ Token-splicing forward pass — implemented, tested
• 🚧 Image preprocessor (resize → bucket to 70/140/280/560/1120 tokens → 48×48 patches → factorized 2D position IDs) — next
• 🚧 Audio preprocessor (16kHz waveform → 640-sample frames) — next
• 🚧 Chat-template integration for <boi>...<eoi> and <boa>...<eoa> blocks — next

Track progress on the GitHub repo.

Recommended sampling (from Google's model card)

• temperature = 1.0, top_p = 0.95, top_k = 64
• Enable "thinking" mode by prepending <|think|> to the system prompt.

Architecture notes

• model_type: gemma4_unified (1 of 5 Gemma 4 sizes; the only "Unified" encoder-free one)
• 48 layers, hidden_size 3840, 16 heads, head_dim 256 (sliding) / 512 (full), context 256k
• Tied word embeddings (no separate lm_head)
• Sliding/full attention pattern from explicit layer_types (every 6th layer is full)
• attention_k_eq_v=true, final_logit_softcapping=30.0

License

Apache 2.0 (inherited from upstream). Use is also subject to Google's Gemma Terms and Prohibited Use Policy. All credit for the model itself goes to Google DeepMind. This repository only contains the MLX conversion.