Instructions to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
# Run inference directly in the terminal:
./llama-cli -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Use Docker

docker model run hf.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

LM Studio
Jan

vLLM

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Ollama
How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Ollama:
```
ollama run hf.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
```
Unsloth Desktop

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Docker Model Runner:
```
docker model run hf.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
```

Lemonade

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Run and chat with the model

lemonade run user.Gemma-4-E4B-it-Cerebellum-v2-GGUF-Q3_K_M

List all available models

lemonade list

Hermes Agent

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Cerebellum

Gemma 4 E4B — Cerebellum v2 GGUF (4.2 GB)

Ablation-informed mixed-precision quantization of google/gemma-4-e4b-it. 4.2 GB file size, 52.20 perplexity — smaller and better than Cerebellum v1 (4.3 GB, PPL 55.10).

Cerebellum v1 fixed Gemma 4's broken small quants by protecting PLE tensors. v2 goes further — a 26-tensor ablation sweep identified which non-PLE tensors are sacred (need more precision) and which actively improve when crushed to Q2_K. The result: 2 tensors promoted to Q6_K, 3 demoted to Q2_K, everything else unchanged.

Benchmarks

Benchmark	v2 (4.2 GB)	v1 (4.3 GB)	BF16 (15 GB)
Perplexity (WikiText-2, 2048 ctx)	52.20	55.10	54.58
HumanEval pass@1	68.3%	65.9%	—
ARC-Challenge	85.7%	85.1%	—
HellaSwag	75.3%	75.5%	—
MMLU-Redux	58.4%	58.0%	—

All benchmarks measured directly on this file.

How Cerebellum v2 Works

Step 1: PLE Protection (same as v1)

Gemma 4's 174 Per-Layer Embedding tensors have a hard precision cliff between Q4_K and Q3_K. Pinning them to Q5_K eliminates the cliff. See Cerebellum v1 for the full PLE analysis.

Step 2: Ablation Sweep (new in v2)

We crushed each of 26 non-PLE tensors individually to Q2_K while keeping everything else at baseline (Q3_K_M + PLE Q5_K), then measured the PPL delta for each.

Sacred tensors — promoted to Q6_K:

Tensor	PPL Delta	Action
`blk.41.attn_q`	+5.44	Q3_K_M → Q6_K
`blk.20.ffn_up`	+1.58	Q3_K_M → Q6_K

blk.41.attn_q is the single most important non-PLE tensor in the model — crushing it to Q2_K adds 5.44 PPL. The final layer's Q projection carries disproportionate weight in output quality.

Demotable tensors — crushed to Q2_K (PPL improved):

Tensor	PPL Delta	Action
`blk.15.ffn_gate`	-0.64	Q3_K_M → Q2_K
`blk.40.attn_q`	-0.64	Q3_K_M → Q2_K
`blk.41.ffn_up`	-0.63	Q3_K_M → Q2_K

These tensors are actively worse at Q3_K_M than Q2_K — the quantization noise at Q3_K hurts more than the precision loss at Q2_K.

Full Ablation Results

Tensor	PPL	Delta	Verdict
blk.41.attn_q	60.54	+5.44	sacred — promoted
blk.20.ffn_up	56.68	+1.58	sacred — promoted
blk.35.ffn_gate	55.53	+0.43	leave at Q3_K_M
blk.25.ffn_gate	55.44	+0.34	leave at Q3_K_M
blk.20.attn_q	55.45	+0.35	leave at Q3_K_M
blk.1.ffn_up	55.32	+0.22	leave at Q3_K_M
blk.10.ffn_up	55.26	+0.16	leave at Q3_K_M
blk.30.attn_q	55.24	+0.14	leave at Q3_K_M
blk.0.ffn_gate	55.14	+0.04	noise
blk.0.attn_k	55.13	+0.03	noise
blk.0.attn_q	55.12	+0.02	noise
blk.10.attn_q	55.11	+0.01	noise
blk.41.attn_v	55.11	+0.01	noise
blk.0.ffn_up	55.07	-0.03	demotable
blk.1.ffn_gate	55.07	-0.03	demotable
blk.5.ffn_gate	55.04	-0.06	demotable
blk.10.ffn_gate	55.04	-0.06	demotable
blk.40.ffn_up	55.04	-0.06	demotable
blk.41.ffn_gate	54.94	-0.16	demotable
blk.20.ffn_gate	54.84	-0.26	demotable
blk.40.ffn_gate	54.77	-0.33	demotable
blk.30.ffn_up	54.71	-0.39	demotable
blk.30.ffn_gate	54.68	-0.42	demotable
blk.15.ffn_gate	54.46	-0.64	crushed in v2
blk.40.attn_q	54.46	-0.64	crushed in v2
blk.41.ffn_up	54.47	-0.63	crushed in v2

Why PPL Is Below BF16

v2's PPL (52.20) is lower than BF16 (54.58). This happens because certain tensors at Q3_K_M carry quantization noise that actively degrades prediction. Crushing them to Q2_K removes that noise pattern, and promoting sacred tensors to Q6_K preserves the precision that matters most. The net effect is better than keeping everything at full precision — targeted precision allocation beats uniform precision.

VRAM Requirements

Context	VRAM
2K	~4.5 GB
8K	~5.5 GB
16K	~6.5 GB

Fits comfortably on a 6 GB GPU at moderate context.

Measured launch (RTX 3090, llama.cpp)

Measured 2026-06-13 on a single RTX 3090 (24 GB), one llama-server, KV cache q8_0:

metric	measured
decode speed	116 tok/s
peak VRAM (4-slot serving)	5.3 GB
max measured context (q8_0 KV)	65,536

llama-server -m Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf \
  -ngl 99 --parallel 4 -c 24576 --jinja --reasoning-budget 0

This rig's measurements; no quality claims beyond them.

Usage

# llama.cpp
llama-server \
  --model Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 8192

# Ollama
echo 'FROM ./Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf' > Modelfile
ollama create gemma4-e4b-v2 -f Modelfile
ollama run gemma4-e4b-v2

Reproducing This Quant

pip install git+https://github.com/deucebucket/cerebellum.git

# 1. Generate imatrix from weight statistics (~45 seconds on CPU)
python -m osmosis.imatrix_stream \
  --model google/gemma-4-e4b-it \
  --output imatrix.dat -v

# 2. Convert to BF16 GGUF (using llama.cpp)
python convert_hf_to_gguf.py gemma-4-e4b-it --outfile gemma4-e4b-bf16.gguf --outtype bf16

# 3. Quantize with Cerebellum v2 overrides
llama-quantize \
  --imatrix imatrix.dat \
  --tensor-type-file cerebellum_v2_overrides.txt \
  gemma4-e4b-bf16.gguf Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf Q3_K_M

The imatrix.dat, cerebellum_v2_overrides.txt, and ablation_results.json are included in this repo.

Files

File	Size	Description
`Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf`	4.2 GB	The quantized model
`imatrix.dat`	4.6 MB	Weight-sensitivity importance matrix
`cerebellum_v2_overrides.txt`	4.6 KB	179 tensor type overrides (174 PLE Q5_K + 2 Q6_K + 3 Q2_K)
`ablation_results.json`	3 KB	Full ablation sweep data (26 tensors)

Model Details

Base model: google/gemma-4-e4b-it
Architecture: Dense transformer with PLE, 42 layers, 720 tensors
Quantization: Q3_K_M base + 174 PLE at Q5_K + 2 sacred at Q6_K + 3 demoted to Q2_K
Imatrix: Weight-sensitivity (L2_norm × max_abs × variance), no calibration data
Vocabulary: 262,144 tokens (text + vision + audio)
File format: GGUF v3
Cerebellum v1: deucebucket/Gemma-4-E4B-it-Cerebellum-v1-GGUF

Test Hardware

Component	Spec
GPU	NVIDIA RTX 3090 (24 GB)
CPU	AMD Ryzen 7 5800XT
RAM	64 GB DDR4
OS	Fedora Linux 43 (Atomic)

Attribution

Google DeepMind — Gemma 4 base model
llama.cpp — quantization and tensor type override support
AWQ — channel-level weight sensitivity insights

License

Gemma Terms of Use

Downloads last month: 45

GGUF

Model size

8B params

Architecture

gemma4

Hardware compatibility

3-bit

Model tree for deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF

Base model

google/gemma-4-E4B

Finetuned

google/gemma-4-E4B-it

Quantized

(331)

this model

Paper for deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Paper • 2306.00978 • Published Jun 1, 2023 • 15

Evaluation results

normalized accuracy on AI2 Reasoning Challenge
test set Local benchmark run (RTX 3090, llama.cpp)

0.857
accuracy on HellaSwag
validation set Local benchmark run (RTX 3090, llama.cpp)

0.753
accuracy on MMLU-Redux
test set Local benchmark run (RTX 3090, llama.cpp)

0.584
pass@1 on HumanEval (pass@1)
test set Local benchmark run (RTX 3090, llama.cpp)

0.683
perplexity on WikiText-2 Perplexity
test set Local benchmark run (RTX 3090, llama.cpp)

52.200