README.md

---
license: apache-2.0
base_model: JetBrains/Mellum2-12B-A2.5B-Thinking
library_name: gguf
pipeline_tag: text-generation
tags: [mellum, mellum2, code, coding, reasoning, thinking, moe, gguf, llama.cpp, local-llm]
---

# ⚙️ Mellum2-12B-A2.5B-Reasoning-Distill (GGUF) ⚙️
### 🧑‍💻 A fast little coding brain — local AI for *everyone*

> **12B total params, only 2.5B active per token.** This is a **Mixture-of-Experts** model, so it *runs
> like a ~2.5B model* but *thinks like a 12B* one. 🚀
> Built on **JetBrains' Mellum 2** (a from-scratch software-engineering model) and tuned on **Claude Opus
> 4.6 / 4.7 / 4.8** reasoning traces — it reasons step-by-step in `<think>` blocks, then answers. 🧠💻
> All local, all yours, no API, no cloud. And it's **seriously fast.**

---

## ⚡ Blazing fast — *measured*, not marketing 🏎️💨
**~440 tokens/sec on a single RTX 5090** at Q4_K_M (`--n-gpu-layers 99 -fa on`) — and **generation quality
holds up**: correct, coherent code and clean step-by-step reasoning. 🎯

You get **big-model answers at small-model speed.** Why so quick? It's a **Mixture-of-Experts** (only 2.5B of
the 12B params fire per token) with a **compact 98K vocab** — so it generates several times faster than a
dense model its size, **with no draft / speculative model needed.** 💚

| Hardware | Quant | Generation speed (measured) |
|---|---|---|
| RTX 5090 (32 GB) | Q4_K_M | **~440 tok/s** ⚡ |

---

## 📦 Pick your size (GGUF quants)

| Quant | Size | Vibe |
|------|------|------|
| 🟢 **Q2_K** | **5.0 GB** | tiniest — runs almost anywhere |
| 🔵 **Q4_K_M** | **8.1 GB** | the sweet spot 👌 (recommended) |
| 🟣 **Q6_K** | **10.9 GB** | near-lossless |
| ⚪ **Q8_0** | **12.9 GB** | basically full quality |

> 💡 It's a **Mixture-of-Experts**: all 64 experts live on disk/VRAM (so size is for the *whole* 12B), but
> only 8 fire per token — that's why it's so quick.

---

## 🧮 "Will it fit?" — rough VRAM guide

Mellum2 has a **tiny KV cache** (GQA with just 4 KV heads, and sliding-window attention on 3 of every 4
layers) — so **context is rarely the limiter.** Pick the quant that fits your VRAM and you'll have plenty
of room for long context (max is **131K**). Rough numbers 🤓 (weights + ~2 GB overhead):

| Your VRAM / unified mem | Best quant that fits | Context headroom |
|---|---|---|
| **8 GB**  | 🟢 Q2_K | comfy (long ctx still fits) |
| **12 GB** | 🔵 Q4_K_M | lots |
| **16 GB** | 🟣 Q6_K / ⚪ Q8_0 | lots |
| **24 GB+**| ⚪ Q8_0 | up to 131K 🎉 |

> 💡 Apple Silicon / iGPUs with **unified memory** count too — same idea, just slower than a dGPU.
> 💡 Tight on room? Drop a quant or use a `q4_0` KV cache for even more context.

---

## 🚀 How to run it (super easy)

### Option A — llama.cpp (recommended) 🦙
1. Grab a quant above (e.g. `…-Q4_K_M.gguf`) and `llama-server` from [llama.cpp](https://github.com/ggml-org/llama.cpp).
   > ⚠️ Needs a **recent llama.cpp** that supports the **`mellum2`** architecture (a mid-2026 build or newer).
   > Older builds fail with `unknown architecture: 'mellum2'`.
2. Run a server (Windows `.bat` shown — tweak `--port`, `--ctx-size` to taste):

```bat
@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\mellum2-claude-Q4_K_M.gguf ^
  --ctx-size 16384 ^
  --n-gpu-layers 99 ^
  --no-mmap ^
  -fa on ^
  --jinja --reasoning-format deepseek ^
  --temp 0.6 --top-p 0.95 --top-k 20 ^
  --host 0.0.0.0 --port 18080
pause
```
3. Open `http://localhost:18080` and chat. 🎉  (Tip: bump `--ctx-size` — the KV cache is small, so go big.)

### Option B — one-click apps 🖱️
Works in **LM Studio**, **Jan**, **Ollama**, etc. — just import the GGUF, pick your quant, go. 🐾
*(Make sure the app ships a recent llama.cpp that knows `mellum2`.)*

### 🧠 Thinking mode
This model thinks natively in `<think> … </think>` blocks. The chat template handles it automatically;
the `--reasoning-format deepseek` flag tells llama.cpp to surface the reasoning cleanly.
**Recommended sampling: `temp 0.6, top_p 0.95, top_k 20`** (JetBrains' official settings for the Thinking model).

<details><summary>🐍 Prefer raw transformers? (click)</summary>

> This repo ships **GGUF** (for llama.cpp). To run in raw transformers you need non-GGUF weights — point `mid`
> at the original JetBrains checkpoint (or your own merged fp16). Needs **`transformers >= 5.8`** (the `mellum`
> architecture is built in — no `trust_remote_code`). It's a plain text **CausalLM** (not multimodal).

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "JetBrains/Mellum2-12B-A2.5B-Thinking"   # GGUF won't load here — use non-GGUF weights
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.bfloat16,
                                             device_map="auto", attn_implementation="sdpa")

msgs = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
inputs = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True,
                                 enable_thinking=True)
inputs = inputs.to(model.device)
out = model.generate(inputs, max_new_tokens=512, temperature=0.6, top_p=0.95, top_k=20)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=False))
```
</details>

---

## ⚡ Why no MTP / draft model?
JetBrains' Mellum 2 uses Multi-Token Prediction as a **training-time** objective only — it isn't exported to
the released weights, so there's no draft to ship. **You don't need one:** with just 2.5B active params and a
compact vocab, it already generates extremely fast (~440 tok/s on a 5090 at Q4_K_M). 🏎️

---

## 🧩 What is this, exactly?
- **Base:** [`JetBrains/Mellum2-12B-A2.5B-Thinking`](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)
  — a **from-scratch**, software-engineering-focused **Mixture-of-Experts** model (12.15B total / **2.5B active**,
  28 layers, 64 experts with 8 active, 131K context). JetBrains already did SFT + RLVR on it; this is a light
  extra **LoRA distillation** pass on top.
- **This fine-tune:** a low-intensity QLoRA-style pass over the attention projections, distilling **Claude Opus
  reasoning style** into the model. It keeps Mellum 2's coding/agent strengths while nudging the reasoning
  voice toward Opus. 💡

---

## ⚠️ Good to know
- **Coding-first:** Mellum 2 is built for code generation, editing, debugging, tool-calls and agents. Great at
  programming + structured reasoning; it's **not** a general-knowledge encyclopedia.
- **Reduced refusals:** the distillation data omits safety hedging, so it refuses less than a typical aligned
  chat model. It is **not** safety-aligned — add your own guardrails for production. Use responsibly. 🙏
- The reasoning is *stylistic* synthetic CoT — great for structure, but double-check facts and numbers.
- English-centric (handles other languages, but English is strongest).

---

## 📚 Data & License
- **Base model:** [`JetBrains/Mellum2-12B-A2.5B-Thinking`](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking),
  released under **Apache-2.0**.
- **Training data:** built on the public, **Apache-2.0** dataset
  [`angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k`](https://huggingface.co/datasets/angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k),
  **augmented with additional Opus 4.8-generated reasoning samples** I curated and mixed in.
- Personal/hobby project — shared as-is, no warranty. Have fun! 🐾✨