README.md

---
license: apache-2.0
base_model: google/gemma-4-12B-it
library_name: transformers
pipeline_tag: text-generation
tags: [gemma4, coding, agentic, terminal, tool-use, reasoning, thinking, safetensors, transformers]
---

# 💻🤖 Gemma4-12B **v2** — **safetensors master (full precision)** ✨
### Coding + Agentic Edition · Composer 2.5 × Fable 5 · v2

> **This is the full-precision `safetensors` master** for my Gemma 4 12B **coding + agentic** fine-tune — the same
> model many of you have been running as GGUF, now in its original weights. 🧠🛠️ v2 is the big **agentic** upgrade:
> it reads, reasons, *uses tools*, and works through multi-step technical tasks before it acts. This repo is for
> *builders* — roll your own quants, fine-tune further, or run it in `transformers`.

---

## 🎉 Surprise!

A huge thank-you for all the attention this project has gotten — really, thank you. 🙏 I only managed to get out
**tonight** to upload the **full-precision original (safetensors master)** of this model, so sorry for the wait — I'd
planned to put it up last week. But the delay comes with **two big surprises** I've been dying to share:

**1. v3 is coming soon.** 🔮 The next version is on its way and will fix several of the known issues you've reported.

**2. I'm now working with a top-tier AI lab to give back to the open-source community.** 🤝 Many of you have already
noticed the side effects in v1 and v2 — and honestly they come down to just two things: **(1) not enough compute, and
(2) one person with limited expertise** behind the whole thing. This collaboration **solves both of those completely.**
And the **benchmarks you care about will absolutely be addressed** — the things I simply couldn't fully pull off before
because of time and compute limits. The people working on this with me are **PhDs from top universities, with seriously
strong papers and citation records.** Just think about that for a second: the people who *actually build large models*
are now contributing to the open-source community *together with me* — that is genuinely **wild**. 🤯 We're in active
discussions right now, and the project is still in the **R&D phase**, so I can't share specifics yet — but the **moment**
I have news, **you'll be the first to know.** 🚀

---

## 🎯 What this repo is for

This repo holds the **un-quantized master weights** (`model.safetensors`, bf16). Use it to:

- 🔧 **Roll your own quants** — make custom GGUF / **MLX** / AWQ / GPTQ builds from full precision.
- 🧪 **Fine-tune further** — it's a clean base for your own LoRA / continued training.
- 🤗 **Run it in `transformers`** (needs a recent build with `gemma4_unified` support).

> 🏃 **Just want to run it?** You don't need this repo — grab a ready-made quant from the
> **[GGUF repo →](https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF)** (runs in
> ~4.5 GB of VRAM / unified memory in LM Studio, Ollama, llama.cpp, Jan…). This master is for *builders*. 💚

---

## 📊 The headline — it works as an agent (tau2-bench)

v2 is built for **coding + agentic** work — writing code, running commands, using tools, debugging, multi-step
technical tasks. The clearest signal is **tau2-bench `telecom`**, an agentic tool-use benchmark whose
*diagnose → fix → verify* loop mirrors real terminal/debugging work:

| tau2-bench **telecom** · 20 tasks · local, same harness, **all Q8_0** | score |
|---|---|
| official `gemma-4-12B-it` (base) | **~15%** |
| 🟢 **Gemma4-12B v2 (this model)** | **~55%** |

→ Roughly **3.5× higher** than the base model on technical-agentic tasks. 🎯

> 🔬 *Honest methodology:* these are **local, same-harness, relative** numbers (**all models tested at Q8_0**, greedy
> decoding, self-simulated user, 20 tasks). They are **not** directly comparable to published tau2-bench leaderboard
> figures (different user-simulator, full task sets, full precision) — local self-eval runs *systematically lower* than
> published scores. Read them as **"v2 vs the base model under identical conditions"**, which is the comparison that
> actually matters here.

**Grounded, not made-up.** A coding/terminal *fabrication probe* (tasks that deliberately tempt the model to invent
file paths / function signatures / values) found v2 **grounds before it acts** just like the base — it `grep`/`read`/`ls`
first, and **doesn't make things up** (0% fabrication, on par with the base).

**The trade-off — no free lunch.** On a general-knowledge benchmark (**MMLU-Pro**), v2 lands a little **below** the base —
completely normal for a focused fine-tune: you trade a sliver of broad-knowledge breadth for coding + agentic strength.
Need a generalist? Try my general-purpose
**[Claude Opus 4.6/4.8 distillation](https://huggingface.co/yuxinlu1/gemma-4-12B-it-Claude-4.6-4.8-Opus-GGUF)** or the
base `google/gemma-4-12B-it`. Need a **local coding/agentic** worker? That's what v2 is tuned for. 💚

---

## 🤗 Run it in transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

msgs = [{"role": "user", "content": "Write a Python function to check if a string is a valid IPv4 address."}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=1024)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```

> 🧠 **Thinking mode:** it thinks in Gemma's native thought channel before answering (keep `enable_thinking=true`, the
> default chat template handles it). Recommended sampling: `temp 1.0, top_p 0.95, top_k 64`; for coding you can also go
> greedy (`temp 0`). Needs a **recent `transformers`** that knows the `gemma4_unified` architecture.
>
> 🛠️ **Agentic / tool use:** v2 emits structured tool-calls in Gemma 4's **native** protocol. The smoothest agent
> setup is a GGUF quant served with llama.cpp `--jinja` (pass your tools via the OpenAI `tools` field) — see the GGUF
> repo for the full command.

---

## 📦 Ready-made GGUF quants

All from the **[GGUF repo](https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF)**:

| Quant | Size | Vibe |
|------|------|------|
| 🟡 [**Q3_K_M**](https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/blob/main/gemma4-v2-Q3_K_M.gguf) | **5.7 GB** | great for 8 GB VRAM |
| 🔵 [**Q4_K_M**](https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/blob/main/gemma4-v2-Q4_K_M.gguf) | **6.87 GB** | the sweet spot 👌 (recommended) |
| 🟣 [**Q6_K**](https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/blob/main/gemma4-v2-Q6_K.gguf) | **9.11 GB** | near-lossless |
| ⚪ [**Q8_0**](https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/blob/main/gemma4-v2-Q8_0.gguf) | **11.8 GB** | basically full quality |

> ⚠️ GGUF needs a **recent llama.cpp** — this is the `gemma4_unified` architecture, older builds won't load it.
> ℹ️ **No Q2_K this release** — it didn't pass real stress-testing (2-bit is too lossy for 12B coding). Smallest
> reliable quant = **Q3_K_M**.

---

## 📚 What's new in v2 (training)

v2 continues from the v1 coder and adds a big **agentic** push — the piece v1 was missing:

- **🛠️ Agentic / terminal** — real **multi-step tool-use** trajectories (*read → reason → act → verify*), in Gemma 4's
  native tool protocol. This is what drove the tau2-bench telecom jump, and it fixes v1's "stops after the first step"
  behavior.
- **💻 Coding** — verified chain-of-thought over Python tasks (**real CoT, gated on passing tests**) plus the
  Fable-5-redo set for the hard cases.
- **📚 General** — a curated slice of reasoning/instruction data to keep broad competence.

All reasoning is **distilled CoT**. A bittersweet note: none of us saw it coming that **Fable 5 would be retired**, and
only my own dataset holds Fable 5's genuine, self-authored traces — so for the community-contributed data I **rebuilt the
missing reasoning from scratch with Opus 4.8 (xhigh)**. It may diverge from the original Fable 5 traces, but it was the
only workable path — and the improvement turned out **really huge**. 💚

---

## ⚡ Speculative decoding (MTP draft) — verified build

The GGUF repo's `MTP/` folder ships the Gemma 4 multi-token-prediction draft (unsloth's GGUF conversion of Google's
official `gemma-4-12B-it-assistant`) for speculative decoding. Gemma 4 MTP is in **llama.cpp mainline** (PR #23398) — no
fork needed — but the `gemma4-assistant` loader is **build-sensitive right now**, so use the exact build below:

- ✅ **Verified working: llama.cpp `b9553` (commit `9e3b928fd`).** Reproduced with `gemma4-v2-Q8_0` + the `MTP-Q8_0`
  draft: loads cleanly and accelerates generation (~88 → ~180 tok/s on a simple deterministic prompt; expect ~1.2–1.3×
  on real coding/thinking). **Lossless** either way.
- ⚠️ **Newer builds (e.g. b9702 / b9717) currently crash** while loading the draft with `invalid vector subscript` — an
  **upstream regression** in the `gemma4-assistant` loader path, *not* a problem with the GGUFs. Stick with **b9553**
  until it's fixed upstream.

```bat
llama-server -m gemma4-v2-Q8_0.gguf ^
  --model-draft MTP\gemma-4-12B-it-MTP-Q8_0.gguf ^
  --spec-type draft-mtp --spec-draft-n-max 4 ^
  -ngl 99 -ngld 99 -fa on --jinja
```

> ℹ️ The draft is the generic Gemma 4 assistant (not retrained for v2), so acceptance is a touch lower than a
> model-specific draft would give — still 100% lossless.

---

## ⚠️ Good to know
- **Specialized for coding / terminal / agentic.** General-knowledge facts/numbers should still be double-checked.
- **Reduced refusals:** task-focused training, not safety-aligned — add your own guardrails for production. Use
  responsibly. 🙏
- English-centric.

---

## 📚 Base & License
- **License: Apache 2.0.** Gemma 4 is released by Google under
  **[Apache 2.0](https://ai.google.dev/gemma/apache_2)** (unlike the older Gemma 1/2/3 terms), so this fine-tune is
  **Apache 2.0** too — free to use, modify, and redistribute. 🎉
- **Base model:** [`google/gemma-4-12B-it`](https://huggingface.co/google/gemma-4-12B-it).
- Personal/hobby project — shared as-is, no warranty. Built with time, care, and a lot of coffee. Have fun, and happy
  hacking! 🐾✨