File size: 3,100 Bytes
e36a69c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
pipeline_tag: image-text-to-text
library_name: mlx-vlm
license: other
base_model: kai-os/Grug-12B
base_model_relation: quantized
tags:
- mlx
- mlx-vlm
- gemma4_unified
- gemma-4
- vision-language
- image-text-to-text
- quantized
- 4-bit
- 6-bit
- 8-bit
- apple-silicon
datasets:
- hotdogs/uka-glm-5.2
- Scale-or-Reason/general-reasoning-ift-pairs
- samcheng0/lumia-reasoning-sft-v1
- HSH-Intelligence/verified-math-reasoning-3k
- kd13/CodeDebug-Instruct-v2-Reasoning
- Madarabr/cortex-adaptive-thinking
- CL-From-Nothing/code_rose_initial_1_7B_SFT_10K_rollouts_Qwen3-4B-Thinking-2507_k12_t0.7_maxtok12288
---
# Grug-12B VLM MLX
This repository contains MLX VLM quantizations of
[`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B), packaged in one
Hugging Face repo with separate folders for each quantization level.
`Grug-12B` is a compact-reasoning fine-tune of
[`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it). The
source model was released as merged Transformers/safetensors weights after
QLoRA training. This repo only provides MLX quantized derivatives for Apple
Silicon inference and keeps the original vision-language model structure.
## Available variants
| Folder | Quantization | Local size | Notes |
| --- | --- | ---: | --- |
| `mlx-8bit/` | MLX affine 8-bit, group size 64 | 12 GB | Highest quality local MLX variant. |
| `mlx-6bit/` | MLX affine 6-bit, group size 64 | 9.1 GB | Balanced size and quality. |
| `mlx-4bit/` | MLX affine 4-bit, group size 64 | 6.3 GB | Smallest and easiest to run. |
These are not GGUF files and are not llama.cpp quants. They are MLX safetensors
folders intended for `mlx-vlm`.
## Usage
Download only the variant you want:
```python
from pathlib import Path
from huggingface_hub import snapshot_download
repo_id = "chanderbalaji/Grug-12B-VLM-MLX"
variant = "mlx-4bit"
snapshot = snapshot_download(
repo_id,
allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)
```
Run with `mlx-vlm`:
```bash
python -m mlx_vlm.generate \
--model /path/to/downloaded/snapshot/mlx-4bit \
--prompt "Describe this image." \
--image /path/to/image.jpg \
--max-tokens 256
```
For text-only prompts, omit the `--image` argument.
## Provenance and attribution
- Source model: [`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B)
- Base model: [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it)
- Relationship: MLX quantized derivatives of the source model
- Source revision used locally: `ad3feab42542e3361dcaf0ebe795d55009765918`
- Conversion target: Gemma 4 unified VLM with `vision_config` preserved
The source model card describes the original training recipe, datasets, local
evaluation, limitations, and acknowledgements. Please refer to that card for
the full model provenance and license context.
## Limitations
Quantization can change output quality, numerical behavior, and edge-case
performance. These files are intended for local MLX inference on Apple Silicon.
Use the source model repo for the original BF16 Transformers weights.
|