---
base_model: kai-os/Grug-12B
library_name: gguf
pipeline_tag: text-generation
license: other
tags:
- gguf
- llama.cpp
- q4_k_m
- quantized
- reasoning
- compact-reasoning
- grugthink
- gemma-4
---

![Grug 12B banner](banner.png)

# Grug 12B GGUF

This is the adjacent GGUF release for [`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B).
The main repo contains the merged Transformers/safetensors fine-tune; this repo contains a llama.cpp quantized file for local inference.

## Files

- `Grug-12B-Q4_K_M.gguf` - practical mixed-precision 4-bit GGUF quant, about 7.0 GB.
- `banner.png` - model banner.

## Conversion

Converted from the merged full model release, then quantized with llama.cpp.

- Source model: `kai-os/Grug-12B`
- llama.cpp commit: `4fc4ec5`
- Conversion outtype: `BF16`
- Uploaded quant: `Q4_K_M`
- Quantized size: `7024.34 MiB`
- Bits per weight: `4.95 BPW`
- SHA256: `3928e9af604369c111ec7098660781f26e3dc350080e3786ef9dd69881967348`

## Usage

Use a recent llama.cpp build with Gemma 4 / Gemma4 Unified GGUF support.

```bash
hf download kai-os/Grug-12B-GGUF Grug-12B-Q4_K_M.gguf
llama-cli -m Grug-12B-Q4_K_M.gguf -p "What is 2+2? Answer briefly." -n 64
```

## Training Summary

Grug 12B is a compact-reasoning fine-tune of `google/gemma-4-12B-it` trained with QLoRA, then merged into the base model for release.
The training target is terse, high-density reasoning that preserves constraints, branching decisions, invariants, edge cases, and final-answer checks while reducing unnecessary reasoning-token verbosity.

See the full model card at [`kai-os/Grug-12B`](https://huggingface.co/kai-os/Grug-12B) for the technique details, dataset provenance, filtering notes, and benchmark notes.

Thanks to [Lambda](https://lambda.com/) for the compute credits used for this work.