Add Grug-12B MLX model card

e36a69c verified 5 days ago

3.1 kB

pipeline_tag: image-text-to-text
library_name: mlx-vlm
license: other
base_model: kai-os/Grug-12B
base_model_relation: quantized
tags:
  - mlx
  - mlx-vlm
  - gemma4_unified
  - gemma-4
  - vision-language
  - image-text-to-text
  - quantized
  - 4-bit
  - 6-bit
  - 8-bit
  - apple-silicon
datasets:
  - hotdogs/uka-glm-5.2
  - Scale-or-Reason/general-reasoning-ift-pairs
  - samcheng0/lumia-reasoning-sft-v1
  - HSH-Intelligence/verified-math-reasoning-3k
  - kd13/CodeDebug-Instruct-v2-Reasoning
  - Madarabr/cortex-adaptive-thinking
  - >-
    CL-From-Nothing/code_rose_initial_1_7B_SFT_10K_rollouts_Qwen3-4B-Thinking-2507_k12_t0.7_maxtok12288

Grug-12B VLM MLX

This repository contains MLX VLM quantizations of kai-os/Grug-12B, packaged in one Hugging Face repo with separate folders for each quantization level.

Grug-12B is a compact-reasoning fine-tune of google/gemma-4-12B-it. The source model was released as merged Transformers/safetensors weights after QLoRA training. This repo only provides MLX quantized derivatives for Apple Silicon inference and keeps the original vision-language model structure.

Available variants

Folder	Quantization	Local size	Notes
`mlx-8bit/`	MLX affine 8-bit, group size 64	12 GB	Highest quality local MLX variant.
`mlx-6bit/`	MLX affine 6-bit, group size 64	9.1 GB	Balanced size and quality.
`mlx-4bit/`	MLX affine 4-bit, group size 64	6.3 GB	Smallest and easiest to run.

These are not GGUF files and are not llama.cpp quants. They are MLX safetensors folders intended for mlx-vlm.

Usage

Download only the variant you want:

from pathlib import Path
from huggingface_hub import snapshot_download

repo_id = "chanderbalaji/Grug-12B-VLM-MLX"
variant = "mlx-4bit"

snapshot = snapshot_download(
    repo_id,
    allow_patterns=[f"{variant}/*"],
)
model_path = Path(snapshot) / variant
print(model_path)

Run with mlx-vlm:

python -m mlx_vlm.generate \
  --model /path/to/downloaded/snapshot/mlx-4bit \
  --prompt "Describe this image." \
  --image /path/to/image.jpg \
  --max-tokens 256

For text-only prompts, omit the --image argument.

Provenance and attribution

Source model: kai-os/Grug-12B
Base model: google/gemma-4-12B-it
Relationship: MLX quantized derivatives of the source model
Source revision used locally: ad3feab42542e3361dcaf0ebe795d55009765918
Conversion target: Gemma 4 unified VLM with vision_config preserved

The source model card describes the original training recipe, datasets, local evaluation, limitations, and acknowledgements. Please refer to that card for the full model provenance and license context.

Limitations

Quantization can change output quality, numerical behavior, and edge-case performance. These files are intended for local MLX inference on Apple Silicon. Use the source model repo for the original BF16 Transformers weights.