---
pipeline_tag: image-text-to-text
library_name: mlx-vlm
license: other
base_model: kai-os/Grug-12B
base_model_relation: quantized
tags:
- mlx
- mlx-vlm
- gemma4_unified
- gemma-4
- vision-language
- image-text-to-text
- quantized
- 4-bit
- 6-bit
- 8-bit
- apple-silicon
datasets:
- hotdogs/uka-glm-5.2
- Scale-or-Reason/general-reasoning-ift-pairs
- samcheng0/lumia-reasoning-sft-v1
- HSH-Intelligence/verified-math-reasoning-3k
- kd13/CodeDebug-Instruct-v2-Reasoning
- Madarabr/cortex-adaptive-thinking
- CL-From-Nothing/code_rose_initial_1_7B_SFT_10K_rollouts_Qwen3-4B-Thinking-2507_k12_t0.7_maxtok12288
---

# Grug-12B VLM MLX

This repository contains MLX VLM quantizations of
[`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B), packaged in one
Hugging Face repo with separate folders for each quantization level.

`Grug-12B` is a compact-reasoning fine-tune of
[`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it). The
source model was released as merged Transformers/safetensors weights after
QLoRA training. This repo only provides MLX quantized derivatives for Apple
Silicon inference and keeps the original vision-language model structure.

## Available variants

| Folder | Quantization | Local size | Notes |
| --- | --- | ---: | --- |
| `mlx-8bit/` | MLX affine 8-bit, group size 64 | 12 GB | Highest quality local MLX variant. |
| `mlx-6bit/` | MLX affine 6-bit, group size 64 | 9.1 GB | Balanced size and quality. |
| `mlx-4bit/` | MLX affine 4-bit, group size 64 | 6.3 GB | Smallest and easiest to run. |

These are not GGUF files and are not llama.cpp quants. They are MLX safetensors
folders intended for `mlx-vlm`.

## Usage

Download only the variant you want:

```python
from pathlib import Path
from huggingface_hub import snapshot_download

repo_id = "chanderbalaji/Grug-12B-VLM-MLX"
variant = "mlx-4bit"

snapshot = snapshot_download(
    repo_id,
    allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)
```

Run with `mlx-vlm`:

```bash
python -m mlx_vlm.generate \
  --model /path/to/downloaded/snapshot/mlx-4bit \
  --prompt "Describe this image." \
  --image /path/to/image.jpg \
  --max-tokens 256
```

For text-only prompts, omit the `--image` argument.

## Provenance and attribution

- Source model: [`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B)
- Base model: [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it)
- Relationship: MLX quantized derivatives of the source model
- Source revision used locally: `ad3feab42542e3361dcaf0ebe795d55009765918`
- Conversion target: Gemma 4 unified VLM with `vision_config` preserved

The source model card describes the original training recipe, datasets, local
evaluation, limitations, and acknowledgements. Please refer to that card for
the full model provenance and license context.

## Limitations

Quantization can change output quality, numerical behavior, and edge-case
performance. These files are intended for local MLX inference on Apple Silicon.
Use the source model repo for the original BF16 Transformers weights.