Instructions to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M
# Run inference directly in the terminal:
./llama-cli -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Use Docker

docker model run hf.co/deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

LM Studio
Jan

vLLM

How to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Ollama
How to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with Ollama:
```
ollama run hf.co/deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M
```
Unsloth Desktop

How to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with Docker Model Runner:
```
docker model run hf.co/deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M
```

Lemonade

How to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Run and chat with the model

lemonade run user.Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF-Q3_K_M

List all available models

lemonade list

Hermes Agent

How to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "deucebucket/Gemma-4-26B-A4B-it-Cerebellum-v6-GGUF:Q3_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

deucebucket commited on Jun 22

Commit

c0b8ca7

verified ·

1 Parent(s): c5b8844

docs: consolidate card — single Evaluation section, drop dated/append blocks

Browse files

Files changed (1) hide show

README.md +44 -163

README.md CHANGED Viewed

@@ -74,187 +74,68 @@ model-index:
 # Gemma 4 26B-A4B-it Cerebellum GGUF
-This repository contains GGUF builds derived from
-`google/gemma-4-26B-A4B-it`.
-## 2026-05-22 Update
-Added:
-```text
-gemma-4-26B-A4B-it-cerebellum-v6.1-templatefix-Q3_K_M.gguf
-sha256: d24229facdef8360a7ffa8b37a50e1de636b9139a5eba0efe899828e45ae7989
-gemma-4-26b-a4b-it.mmproj.gguf
-sha256: b762c43119ebdc3e3c36d929d958e827fac35b03278dda9203f87131aee1f185
-```
-The v6.1 file keeps the v6 tensor allocation and updates GGUF/runtime-facing
-metadata for Gemma 4 chat-template use. The update was tested with
-`llama-server --jinja --reasoning auto` and request-level no-thinking controls.
-Older files in this repository are retained for reproducibility.
-## Measured launch (RTX 3090, llama.cpp)
-Measured 2026-06-13 on a single RTX 3090 (24 GB), one `llama-server`, KV cache `q8_0`:
-| metric | measured |
-|---|---|
-| decode speed | 123 tok/s |
-| peak VRAM (4-slot serving) | 15.1 GB |
-| max measured context (q8_0 KV) | 131,072 |
-```bash
-llama-server -m gemma-4-26B-A4B-it-cerebellum-v6.1-templatefix-Q3_K_M.gguf \
-  -ngl 99 --parallel 4 -c 24576 --jinja --reasoning-budget 0
-```
-_This rig's measurements; no quality claims beyond them._
-## Tested Runtime
-Runtime used for the 2026-05-22 templatefix checks:
-```text
-llama.cpp fork: https://github.com/deucebucket/llama.cpp
-branch: cerebellum/gemma4-runtime-fixes
-fork commit: ded491334 fix: harden Gemma 4 server budgets
-base build: b8930-59fa0b455
-```
-Server shape used locally:
 ```bash
 llama-server \
   --model gemma-4-26B-A4B-it-cerebellum-v6.1-templatefix-Q3_K_M.gguf \
   --mmproj gemma-4-26b-a4b-it.mmproj.gguf \
-  --n-gpu-layers 99 \
-  --ctx-size 65536 \
-  --parallel 1 \
-  --flash-attn on \
-  --cache-type-k q8_0 \
-  --cache-type-v q8_0 \
-  --jinja \
-  --reasoning auto \
-  --media-path /tmp/
-```
-Normal no-thinking requests used:
-```json
-{
-  "chat_template_kwargs": {"enable_thinking": false},
-  "thinking_budget_tokens": 0
-}
 ```
-Bounded-thinking smoke requests used `thinking_budget_tokens: 128`.
-## 2026-05-22 Templatefix Test Artifacts
-Creative-writing smoke files:
-```text
-creative_eval_20260522/regular_v6_1_templatefix_creative_summary.json
-creative_eval_20260522/regular_v6_1_templatefix_creative_rerun_longcaps_summary.json
-```
-Non-coding tool-use files:
-```text
-agentic_eval_20260522/README.md
-agentic_eval_20260522/regular_v6_1_noncoding_agentic_tools_strict_summary.json
-```
-Observed 2026-05-22 results from those artifacts:
-| Area | Harness | Observed result |
-|---|---|---|
-| No-thinking output channel | six creative prompts | `reasoning_len=0` in recorded outputs |
-| Template leakage markers | six creative prompts | no `<think>` marker or template marker recorded by checker |
-| Creative long-cap rerun | four prompts rerun after initial length caps | four stop finishes in rerun summary |
-| Non-coding tool workflow | three strict OpenAI-style tool tasks | `schedule_strict`, `release_notes_strict`, `creative_brief_strict` listed in `pass_cases` |
-The non-coding tool harness used mock tools named `list_calendar`,
-`create_calendar_hold`, `search_notes`, `save_note`, and `add_task`. It did not
-test code editing.
-## Evaluation
-Benchmark results for the Cerebellum v6 tensor allocation, measured directly
-on the GGUF with llama.cpp `llama-server` on an RTX 3090. The v6.1
-templatefix file keeps the v6 tensor allocation with zero tensor changes
-(metadata-only update), so these measurements describe the same weights.
-Summary JSONs are in `benchmark_results/` in this repository.
-| Benchmark | Cerebellum v6 (11 GB) | Local Q3_K_M baseline |
-|-----------|:---:|:---:|
-| ARC-Challenge | **95.56%** (1172 q) | 95.22% |
-| HellaSwag | 84.55% (10042 q) | 86.57% |
-| MMLU-Redux | 71.33% (2400 q) | 73.67% |
-Protocol: multiple-choice benchmarks run against a local `llama-server` with
-the project benchmark harness at temperature 0. HumanEval is not listed in
-the metadata because the retained v6 HumanEval artifacts are marked for audit
-in local notes. For Gemma 4, the current HumanEval/EvalPlus protocol uses the
-chat-completions harness (`scripts/benchmark_evalplus_chat.py`) with
-`enable_thinking: false`, `thinking_budget_tokens: 0`, and `BENCH_WORKERS=1`,
-not raw completions.
-## Historical Same-Repo Benchmark Artifacts
-The following benchmark artifacts are from the earlier v6 line and the local
-Q3_K_M baseline. They are included as historical same-project measurements, not
-as new v6.1 measurements.
-| Artifact set | ARC-Challenge | HellaSwag | MMLU-Redux | HumanEval note |
-|---|---:|---:|---:|---|
-| `q3km_baseline_*` | 95.2218 | 86.5664 | 73.6667 | `q3km_baseline_humaneval_results.json`: 62.2 pass@1 |
-| `cerebellum_v6_*` | 95.5631 | 84.55 | 71.3333 | v6 HumanEval artifacts are retained but marked for audit in local notes |
-For Gemma 4 HumanEval/EvalPlus, the local protocol now uses chat completions,
-not raw completions:
-```text
-llama-server --jinja --reasoning auto
-chat_template_kwargs: {"enable_thinking": false}
-thinking_budget_tokens: 0
-BENCH_WORKERS=1
-```
-## Files and Provenance
-Main v6.1 GGUF:
-```text
-source base: google/gemma-4-26B-A4B-it
-quantization family: mixed-precision GGUF
-recipe lineage: Cerebellum v6 tensor allocation
-base quant lineage: Q3_K_M with bartowski imatrix
-```
-Matching mmproj:
-```text
-gemma-4-26b-a4b-it.mmproj.gguf
-```
-## Notes
-- The 2026-05-22 tests were run on local `llama-server`.
-- The opencode coding-agent test is not used as a model-card result. In one
-  internal White and Black project run, the model connected through the harness
-  and ran a Godot test, then produced malformed edit-tool calls.
-- The creative-writing checks are smoke tests plus mechanical checks, not a
-  human preference benchmark.
-- The non-coding tool checks use mocked tools and fixed task definitions.
 ## Credits
 - Base model: Google Gemma Team, `google/gemma-4-26B-A4B-it`
-- Imatrix source used in the v6 lineage: bartowski,
-  `bartowski/google_gemma-4-26B-A4B-it-GGUF`
-- GGUF/runtime: llama.cpp
-- Method and quantization workflow: deucebucket/cerebellum Cerebellum pipeline
-- Local test artifacts: deucebucket Cerebellum workflow

 # Gemma 4 26B-A4B-it Cerebellum GGUF
+Sensitivity-guided mixed-precision GGUF of [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it):
+a Q3_K_M base with the Cerebellum v6 tensor allocation. The shipped file carries
+the v6 weights plus Google's updated Gemma 4 chat-template metadata (the 2026-05-18
+template state) with zero tensor changes versus v6. Newer versions appear in
+filenames, not the repo name.
+## Files
+| File | Description |
+|------|-------------|
+| `gemma-4-26B-A4B-it-cerebellum-v6.1-templatefix-Q3_K_M.gguf` | ~11 GB; v6 allocation + updated chat-template metadata |
+| `gemma-4-26b-a4b-it.mmproj.gguf` | vision projector (required for image/video) |
+## Evaluation
+Measured directly on the GGUF with llama.cpp `llama-server` on an RTX 3090,
+temperature 0, project benchmark harness. v6.1 is metadata-only over v6, so these
+describe the same weights. The comparison column is our own same-size uniform
+Q3_K_M build measured on the same harness. Summary JSONs are in `benchmark_results/`.
+| Benchmark | Cerebellum v6 (11 GB) | Uniform Q3_K_M (11 GB) |
+|-----------|:---:|:---:|
+| ARC-Challenge (1172 q) | 95.56% | 95.22% |
+| HellaSwag (10042 q) | 84.55% | 86.57% |
+| MMLU-Redux (2400 q) | 71.33% | 73.67% |
+| HumanEval (raw-completions, legacy) | pending re-audit | 62.2% pass@1 |
+HumanEval for Gemma 4 must use the chat-completions harness
+(`scripts/benchmark_evalplus_chat.py`, `enable_thinking: false`,
+`thinking_budget_tokens: 0`, `BENCH_WORKERS=1`). The retained v6 HumanEval
+artifacts were raw-completions and are marked for re-audit, so no v6 HumanEval
+number is published here.
+## Usage
+Gemma 4 requires `--jinja`. For non-thinking output, pass request-level
+`chat_template_kwargs: {"enable_thinking": false}` and `thinking_budget_tokens: 0`;
+do not set a fixed server `--reasoning-budget` (it can burn output into hidden
+reasoning until the length cap, which looks like a repetition loop).
 ```bash
 llama-server \
   --model gemma-4-26B-A4B-it-cerebellum-v6.1-templatefix-Q3_K_M.gguf \
   --mmproj gemma-4-26b-a4b-it.mmproj.gguf \
+  -ngl 99 --ctx-size 65536 --parallel 1 --flash-attn on \
+  --cache-type-k q8_0 --cache-type-v q8_0 --jinja --reasoning auto
 ```
+Measured on one RTX 3090 (24 GB), KV q8_0: ~123 tok/s decode, 15.1 GB peak VRAM
+(4-slot serving), context to 131,072. This rig's measurements; no quality claims
+beyond them.
+## Provenance
+- Base: [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it) — Google Gemma Team
+- Base quant lineage: Q3_K_M with the bartowski imatrix (`bartowski/google_gemma-4-26B-A4B-it-GGUF`)
+- Recipe: Cerebellum v6 tensor allocation; v6.1 is a chat-template metadata refresh
+  (Google 2026-05-18 template), zero tensor changes
 ## Credits
 - Base model: Google Gemma Team, `google/gemma-4-26B-A4B-it`
+- Imatrix: bartowski, `bartowski/google_gemma-4-26B-A4B-it-GGUF`
+- GGUF runtime: [llama.cpp](https://github.com/ggml-org/llama.cpp)
+- Quantization method: [Cerebellum](https://github.com/deucebucket/cerebellum) — deucebucket