Add model card

README.md

@@ -1,3 +1,160 @@
 ---
-license: mit
+license: apache-2.0
+base_model: JetBrains/Mellum2-12B-A2.5B-Thinking
+library_name: gguf
+pipeline_tag: text-generation
+tags: [mellum, mellum2, code, coding, reasoning, thinking, moe, gguf, llama.cpp, local-llm]
 ---
+
+# ⚙️ Mellum2-12B-A2.5B-Reasoning-Distill (GGUF) ⚙️
+### 🧑‍💻 A fast little coding brain — local AI for *everyone*
+
+> **12B total params, only 2.5B active per token.** This is a **Mixture-of-Experts** model, so it *runs
+> like a ~2.5B model* but *thinks like a 12B* one. 🚀
+> Built on **JetBrains' Mellum 2** (a from-scratch software-engineering model) and tuned on **Claude Opus
+> 4.6 / 4.7 / 4.8** reasoning traces — it reasons step-by-step in `<think>` blocks, then answers. 🧠💻
+> All local, all yours, no API, no cloud. And it's **seriously fast.**
+
+---
+
+## ⚡ Blazing fast — *measured*, not marketing 🏎️💨
+**~440 tokens/sec on a single RTX 5090** at Q4_K_M (`--n-gpu-layers 99 -fa on`) — and **generation quality
+holds up**: correct, coherent code and clean step-by-step reasoning. 🎯
+
+You get **big-model answers at small-model speed.** Why so quick? It's a **Mixture-of-Experts** (only 2.5B of
+the 12B params fire per token) with a **compact 98K vocab** — so it generates several times faster than a
+dense model its size, **with no draft / speculative model needed.** 💚
+
+| Hardware | Quant | Generation speed (measured) |
+|---|---|---|
+| RTX 5090 (32 GB) | Q4_K_M | **~440 tok/s** ⚡ |
+
+---
+
+## 📦 Pick your size (GGUF quants)
+
+| Quant | Size | Vibe |
+|------|------|------|
+| 🟢 **Q2_K** | **5.0 GB** | tiniest — runs almost anywhere |
+| 🔵 **Q4_K_M** | **8.1 GB** | the sweet spot 👌 (recommended) |
+| 🟣 **Q6_K** | **10.9 GB** | near-lossless |
+| ⚪ **Q8_0** | **12.9 GB** | basically full quality |
+| (f16) | 24.3 GB | full precision (overkill for most) |
+
+> 💡 It's a **Mixture-of-Experts**: all 64 experts live on disk/VRAM (so size is for the *whole* 12B), but
+> only 8 fire per token — that's why it's so quick.
+
+---
+
+## 🧮 "Will it fit?" — rough VRAM guide
+
+Mellum2 has a **tiny KV cache** (GQA with just 4 KV heads, and sliding-window attention on 3 of every 4
+layers) — so **context is rarely the limiter.** Pick the quant that fits your VRAM and you'll have plenty
+of room for long context (max is **131K**). Rough numbers 🤓 (weights + ~2 GB overhead):
+
+| Your VRAM / unified mem | Best quant that fits | Context headroom |
+|---|---|---|
+| **8 GB**  | 🟢 Q2_K | comfy (long ctx still fits) |
+| **12 GB** | 🔵 Q4_K_M | lots |
+| **16 GB** | 🟣 Q6_K / ⚪ Q8_0 | lots |
+| **24 GB+**| ⚪ Q8_0 (or f16) | up to 131K 🎉 |
+
+> 💡 Apple Silicon / iGPUs with **unified memory** count too — same idea, just slower than a dGPU.
+> 💡 Tight on room? Drop a quant or use a `q4_0` KV cache for even more context.
+
+---
+
+## 🚀 How to run it (super easy)
+
+### Option A — llama.cpp (recommended) 🦙
+1. Grab a quant above (e.g. `…-Q4_K_M.gguf`) and `llama-server` from [llama.cpp](https://github.com/ggml-org/llama.cpp).
+   > ⚠️ Needs a **recent llama.cpp** that supports the **`mellum2`** architecture (a mid-2026 build or newer).
+   > Older builds fail with `unknown architecture: 'mellum2'`.
+2. Run a server (Windows `.bat` shown — tweak `--port`, `--ctx-size` to taste):
+
+```bat
+@echo off
+cd /d C:\llama.cpp
+llama-server.exe ^
+  -m C:\models\mellum2-claude-Q4_K_M.gguf ^
+  --ctx-size 16384 ^
+  --n-gpu-layers 99 ^
+  --no-mmap ^
+  -fa on ^
+  --jinja --reasoning-format deepseek ^
+  --temp 0.6 --top-p 0.95 --top-k 20 ^
+  --host 0.0.0.0 --port 18080
+pause
+```
+3. Open `http://localhost:18080` and chat. 🎉  (Tip: bump `--ctx-size` — the KV cache is small, so go big.)
+
+### Option B — one-click apps 🖱️
+Works in **LM Studio**, **Jan**, **Ollama**, etc. — just import the GGUF, pick your quant, go. 🐾
+*(Make sure the app ships a recent llama.cpp that knows `mellum2`.)*
+
+### 🧠 Thinking mode
+This model thinks natively in `<think> … </think>` blocks. The chat template handles it automatically;
+the `--reasoning-format deepseek` flag tells llama.cpp to surface the reasoning cleanly.
+**Recommended sampling: `temp 0.6, top_p 0.95, top_k 20`** (JetBrains' official settings for the Thinking model).
+
+<details><summary>🐍 Prefer raw transformers? (click)</summary>
+
+> This repo ships **GGUF** (for llama.cpp). To run in raw transformers you need non-GGUF weights — point `mid`
+> at the original JetBrains checkpoint (or your own merged fp16). Needs **`transformers >= 5.8`** (the `mellum`
+> architecture is built in — no `trust_remote_code`). It's a plain text **CausalLM** (not multimodal).
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+mid = "JetBrains/Mellum2-12B-A2.5B-Thinking"   # GGUF won't load here — use non-GGUF weights
+tok = AutoTokenizer.from_pretrained(mid)
+model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.bfloat16,
+                                             device_map="auto", attn_implementation="sdpa")
+
+msgs = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
+inputs = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True,
+                                 enable_thinking=True)
+inputs = inputs.to(model.device)
+out = model.generate(inputs, max_new_tokens=512, temperature=0.6, top_p=0.95, top_k=20)
+print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=False))
+```
+</details>
+
+---
+
+## ⚡ Why no MTP / draft model?
+JetBrains' Mellum 2 uses Multi-Token Prediction as a **training-time** objective only — it isn't exported to
+the released weights, so there's no draft to ship. **You don't need one:** with just 2.5B active params and a
+compact vocab, it already generates extremely fast (~440 tok/s on a 5090 at Q4_K_M). 🏎️
+
+---
+
+## 🧩 What is this, exactly?
+- **Base:** [`JetBrains/Mellum2-12B-A2.5B-Thinking`](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)
+  — a **from-scratch**, software-engineering-focused **Mixture-of-Experts** model (12.15B total / **2.5B active**,
+  28 layers, 64 experts with 8 active, 131K context). JetBrains already did SFT + RLVR on it; this is a light
+  extra **LoRA distillation** pass on top.
+- **This fine-tune:** a low-intensity QLoRA-style pass over the attention projections, distilling **Claude Opus
+  reasoning style** into the model. It keeps Mellum 2's coding/agent strengths while nudging the reasoning
+  voice toward Opus. 💡
+
+---
+
+## ⚠️ Good to know
+- **Coding-first:** Mellum 2 is built for code generation, editing, debugging, tool-calls and agents. Great at
+  programming + structured reasoning; it's **not** a general-knowledge encyclopedia.
+- **Reduced refusals:** the distillation data omits safety hedging, so it refuses less than a typical aligned
+  chat model. It is **not** safety-aligned — add your own guardrails for production. Use responsibly. 🙏
+- The reasoning is *stylistic* synthetic CoT — great for structure, but double-check facts and numbers.
+- English-centric (handles other languages, but English is strongest).
+
+---
+
+## 📚 Data & License
+- **Base model:** [`JetBrains/Mellum2-12B-A2.5B-Thinking`](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking),
+  released under **Apache-2.0**.
+- **Training data:** built on the public, **Apache-2.0** dataset
+  [`angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k`](https://huggingface.co/datasets/angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k),
+  **augmented with additional Opus 4.8-generated reasoning samples** I curated and mixed in.
+- Personal/hobby project — shared as-is, no warranty. Have fun! 🐾✨