--- license: apache-2.0 base_model: JetBrains/Mellum2-12B-A2.5B-Thinking library_name: gguf pipeline_tag: text-generation tags: [mellum, mellum2, code, coding, reasoning, thinking, moe, gguf, llama.cpp, local-llm] --- # ⚙️ Mellum2-12B-A2.5B-Reasoning-Distill (GGUF) ⚙️ ### 🧑‍💻 A fast little coding brain — local AI for *everyone* > **12B total params, only 2.5B active per token.** This is a **Mixture-of-Experts** model, so it *runs > like a ~2.5B model* but *thinks like a 12B* one. 🚀 > Built on **JetBrains' Mellum 2** (a from-scratch software-engineering model) and tuned on **Claude Opus > 4.6 / 4.7 / 4.8** reasoning traces — it reasons step-by-step in `` blocks, then answers. 🧠💻 > All local, all yours, no API, no cloud. And it's **seriously fast.** --- ## ⚡ Blazing fast — *measured*, not marketing 🏎️💨 **~440 tokens/sec on a single RTX 5090** at Q4_K_M (`--n-gpu-layers 99 -fa on`) — and **generation quality holds up**: correct, coherent code and clean step-by-step reasoning. 🎯 You get **big-model answers at small-model speed.** Why so quick? It's a **Mixture-of-Experts** (only 2.5B of the 12B params fire per token) with a **compact 98K vocab** — so it generates several times faster than a dense model its size, **with no draft / speculative model needed.** 💚 | Hardware | Quant | Generation speed (measured) | |---|---|---| | RTX 5090 (32 GB) | Q4_K_M | **~440 tok/s** ⚡ | --- ## 📦 Pick your size (GGUF quants) | Quant | Size | Vibe | |------|------|------| | 🟢 **Q2_K** | **5.0 GB** | tiniest — runs almost anywhere | | 🔵 **Q4_K_M** | **8.1 GB** | the sweet spot 👌 (recommended) | | 🟣 **Q6_K** | **10.9 GB** | near-lossless | | ⚪ **Q8_0** | **12.9 GB** | basically full quality | > 💡 It's a **Mixture-of-Experts**: all 64 experts live on disk/VRAM (so size is for the *whole* 12B), but > only 8 fire per token — that's why it's so quick. --- ## 🧮 "Will it fit?" — rough VRAM guide Mellum2 has a **tiny KV cache** (GQA with just 4 KV heads, and sliding-window attention on 3 of every 4 layers) — so **context is rarely the limiter.** Pick the quant that fits your VRAM and you'll have plenty of room for long context (max is **131K**). Rough numbers 🤓 (weights + ~2 GB overhead): | Your VRAM / unified mem | Best quant that fits | Context headroom | |---|---|---| | **8 GB** | 🟢 Q2_K | comfy (long ctx still fits) | | **12 GB** | 🔵 Q4_K_M | lots | | **16 GB** | 🟣 Q6_K / ⚪ Q8_0 | lots | | **24 GB+**| ⚪ Q8_0 | up to 131K 🎉 | > 💡 Apple Silicon / iGPUs with **unified memory** count too — same idea, just slower than a dGPU. > 💡 Tight on room? Drop a quant or use a `q4_0` KV cache for even more context. --- ## 🚀 How to run it (super easy) ### Option A — llama.cpp (recommended) 🦙 1. Grab a quant above (e.g. `…-Q4_K_M.gguf`) and `llama-server` from [llama.cpp](https://github.com/ggml-org/llama.cpp). > ⚠️ Needs a **recent llama.cpp** that supports the **`mellum2`** architecture (a mid-2026 build or newer). > Older builds fail with `unknown architecture: 'mellum2'`. 2. Run a server (Windows `.bat` shown — tweak `--port`, `--ctx-size` to taste): ```bat @echo off cd /d C:\llama.cpp llama-server.exe ^ -m C:\models\mellum2-claude-Q4_K_M.gguf ^ --ctx-size 16384 ^ --n-gpu-layers 99 ^ --no-mmap ^ -fa on ^ --jinja --reasoning-format deepseek ^ --temp 0.6 --top-p 0.95 --top-k 20 ^ --host 0.0.0.0 --port 18080 pause ``` 3. Open `http://localhost:18080` and chat. 🎉 (Tip: bump `--ctx-size` — the KV cache is small, so go big.) ### Option B — one-click apps 🖱️ Works in **LM Studio**, **Jan**, **Ollama**, etc. — just import the GGUF, pick your quant, go. 🐾 *(Make sure the app ships a recent llama.cpp that knows `mellum2`.)* ### 🧠 Thinking mode This model thinks natively in ` … ` blocks. The chat template handles it automatically; the `--reasoning-format deepseek` flag tells llama.cpp to surface the reasoning cleanly. **Recommended sampling: `temp 0.6, top_p 0.95, top_k 20`** (JetBrains' official settings for the Thinking model).
🐍 Prefer raw transformers? (click) > This repo ships **GGUF** (for llama.cpp). To run in raw transformers you need non-GGUF weights — point `mid` > at the original JetBrains checkpoint (or your own merged fp16). Needs **`transformers >= 5.8`** (the `mellum` > architecture is built in — no `trust_remote_code`). It's a plain text **CausalLM** (not multimodal). ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer mid = "JetBrains/Mellum2-12B-A2.5B-Thinking" # GGUF won't load here — use non-GGUF weights tok = AutoTokenizer.from_pretrained(mid) model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.bfloat16, device_map="auto", attn_implementation="sdpa") msgs = [{"role": "user", "content": "Write a Python function to check if a number is prime."}] inputs = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True, enable_thinking=True) inputs = inputs.to(model.device) out = model.generate(inputs, max_new_tokens=512, temperature=0.6, top_p=0.95, top_k=20) print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=False)) ```
--- ## ⚡ Why no MTP / draft model? JetBrains' Mellum 2 uses Multi-Token Prediction as a **training-time** objective only — it isn't exported to the released weights, so there's no draft to ship. **You don't need one:** with just 2.5B active params and a compact vocab, it already generates extremely fast (~440 tok/s on a 5090 at Q4_K_M). 🏎️ --- ## 🧩 What is this, exactly? - **Base:** [`JetBrains/Mellum2-12B-A2.5B-Thinking`](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) — a **from-scratch**, software-engineering-focused **Mixture-of-Experts** model (12.15B total / **2.5B active**, 28 layers, 64 experts with 8 active, 131K context). JetBrains already did SFT + RLVR on it; this is a light extra **LoRA distillation** pass on top. - **This fine-tune:** a low-intensity QLoRA-style pass over the attention projections, distilling **Claude Opus reasoning style** into the model. It keeps Mellum 2's coding/agent strengths while nudging the reasoning voice toward Opus. 💡 --- ## ⚠️ Good to know - **Coding-first:** Mellum 2 is built for code generation, editing, debugging, tool-calls and agents. Great at programming + structured reasoning; it's **not** a general-knowledge encyclopedia. - **Reduced refusals:** the distillation data omits safety hedging, so it refuses less than a typical aligned chat model. It is **not** safety-aligned — add your own guardrails for production. Use responsibly. 🙏 - The reasoning is *stylistic* synthetic CoT — great for structure, but double-check facts and numbers. - English-centric (handles other languages, but English is strongest). --- ## 📚 Data & License - **Base model:** [`JetBrains/Mellum2-12B-A2.5B-Thinking`](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking), released under **Apache-2.0**. - **Training data:** built on the public, **Apache-2.0** dataset [`angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k`](https://huggingface.co/datasets/angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k), **augmented with additional Opus 4.8-generated reasoning samples** I curated and mixed in. - Personal/hobby project — shared as-is, no warranty. Have fun! 🐾✨