| --- |
| pipeline_tag: image-text-to-text |
| library_name: mlx-vlm |
| license: other |
| base_model: kai-os/Grug-12B |
| base_model_relation: quantized |
| tags: |
| - mlx |
| - mlx-vlm |
| - gemma4_unified |
| - gemma-4 |
| - vision-language |
| - image-text-to-text |
| - quantized |
| - 4-bit |
| - 6-bit |
| - 8-bit |
| - apple-silicon |
| datasets: |
| - hotdogs/uka-glm-5.2 |
| - Scale-or-Reason/general-reasoning-ift-pairs |
| - samcheng0/lumia-reasoning-sft-v1 |
| - HSH-Intelligence/verified-math-reasoning-3k |
| - kd13/CodeDebug-Instruct-v2-Reasoning |
| - Madarabr/cortex-adaptive-thinking |
| - CL-From-Nothing/code_rose_initial_1_7B_SFT_10K_rollouts_Qwen3-4B-Thinking-2507_k12_t0.7_maxtok12288 |
| --- |
| |
| # Grug-12B VLM MLX |
|
|
| This repository contains MLX VLM quantizations of |
| [`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B), packaged in one |
| Hugging Face repo with separate folders for each quantization level. |
|
|
| `Grug-12B` is a compact-reasoning fine-tune of |
| [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it). The |
| source model was released as merged Transformers/safetensors weights after |
| QLoRA training. This repo only provides MLX quantized derivatives for Apple |
| Silicon inference and keeps the original vision-language model structure. |
|
|
| ## Available variants |
|
|
| | Folder | Quantization | Local size | Notes | |
| | --- | --- | ---: | --- | |
| | `mlx-8bit/` | MLX affine 8-bit, group size 64 | 12 GB | Highest quality local MLX variant. | |
| | `mlx-6bit/` | MLX affine 6-bit, group size 64 | 9.1 GB | Balanced size and quality. | |
| | `mlx-4bit/` | MLX affine 4-bit, group size 64 | 6.3 GB | Smallest and easiest to run. | |
|
|
| These are not GGUF files and are not llama.cpp quants. They are MLX safetensors |
| folders intended for `mlx-vlm`. |
|
|
| ## Usage |
|
|
| Download only the variant you want: |
|
|
| ```python |
| from pathlib import Path |
| from huggingface_hub import snapshot_download |
| |
| repo_id = "chanderbalaji/Grug-12B-VLM-MLX" |
| variant = "mlx-4bit" |
| |
| snapshot = snapshot_download( |
| repo_id, |
| allow_patterns=[f"{variant}/*"], |
| ) |
| model_path = Path(snapshot) / variant |
| print(model_path) |
| ``` |
|
|
| Run with `mlx-vlm`: |
|
|
| ```bash |
| python -m mlx_vlm.generate \ |
| --model /path/to/downloaded/snapshot/mlx-4bit \ |
| --prompt "Describe this image." \ |
| --image /path/to/image.jpg \ |
| --max-tokens 256 |
| ``` |
|
|
| For text-only prompts, omit the `--image` argument. |
|
|
| ## Provenance and attribution |
|
|
| - Source model: [`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B) |
| - Base model: [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it) |
| - Relationship: MLX quantized derivatives of the source model |
| - Source revision used locally: `ad3feab42542e3361dcaf0ebe795d55009765918` |
| - Conversion target: Gemma 4 unified VLM with `vision_config` preserved |
|
|
| The source model card describes the original training recipe, datasets, local |
| evaluation, limitations, and acknowledgements. Please refer to that card for |
| the full model provenance and license context. |
|
|
| ## Limitations |
|
|
| Quantization can change output quality, numerical behavior, and edge-case |
| performance. These files are intended for local MLX inference on Apple Silicon. |
| Use the source model repo for the original BF16 Transformers weights. |
|
|
