Instructions to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
# Run inference directly in the terminal:
./llama-cli -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Use Docker

docker model run hf.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

LM Studio
Jan

vLLM

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Ollama
How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Ollama:
```
ollama run hf.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
```
Unsloth Desktop

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Docker Model Runner:
```
docker model run hf.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M
```

Lemonade

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Run and chat with the model

lemonade run user.Gemma-4-E4B-it-Cerebellum-v2-GGUF-Q3_K_M

List all available models

lemonade list

Hermes Agent

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF:Q3_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

chanti

by chantidas - opened May 18

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

-106

Files changed (4) hide show

.gitattributes +0 -2
Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf → Gemma-4-E4B-it-Cerebellum-v2.gguf +0 -0
README.md +4 -101
cerebellum_banner.png +0 -3

.gitattributes CHANGED Viewed

@@ -35,5 +35,3 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imatrix.dat filter=lfs diff=lfs merge=lfs -text
 Gemma-4-E4B-it-Cerebellum-v2.gguf filter=lfs diff=lfs merge=lfs -text
-Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
-cerebellum_banner.png filter=lfs diff=lfs merge=lfs -text

 *tfevents* filter=lfs diff=lfs merge=lfs -text
 imatrix.dat filter=lfs diff=lfs merge=lfs -text
 Gemma-4-E4B-it-Cerebellum-v2.gguf filter=lfs diff=lfs merge=lfs -text

Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf → Gemma-4-E4B-it-Cerebellum-v2.gguf RENAMED Viewed

File without changes

README.md CHANGED Viewed

@@ -20,88 +20,8 @@ tags:
   - ablation
   - 3-bit
   - conversational
-model-index:
-- name: Gemma-4-E4B-it-Cerebellum-v2-GGUF
-  results:
-  - task:
-      name: Text Generation
-      type: text-generation
-    dataset:
-      name: AI2 Reasoning Challenge
-      type: ai2_arc
-      config: ARC-Challenge
-      split: test
-    metrics:
-    - name: normalized accuracy
-      type: acc_norm
-      value: 0.857
-    source:
-      name: Local benchmark run (RTX 3090, llama.cpp)
-      url: https://huggingface.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF/tree/main/benchmark_results
-  - task:
-      name: Text Generation
-      type: text-generation
-    dataset:
-      name: HellaSwag
-      type: hellaswag
-      split: validation
-    metrics:
-    - name: accuracy
-      type: acc
-      value: 0.753
-    source:
-      name: Local benchmark run (RTX 3090, llama.cpp)
-      url: https://huggingface.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF/tree/main/benchmark_results
-  - task:
-      name: Text Generation
-      type: text-generation
-    dataset:
-      name: MMLU-Redux
-      type: cais/mmlu
-      config: all
-      split: test
-    metrics:
-    - name: accuracy
-      type: acc
-      value: 0.584
-    source:
-      name: Local benchmark run (RTX 3090, llama.cpp)
-      url: https://huggingface.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF/tree/main/benchmark_results
-  - task:
-      name: Text Generation
-      type: text-generation
-    dataset:
-      name: HumanEval (pass@1)
-      type: openai_humaneval
-      split: test
-    metrics:
-    - name: pass@1
-      type: pass@1
-      value: 0.683
-    source:
-      name: Local benchmark run (RTX 3090, llama.cpp)
-      url: https://huggingface.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF/tree/main/benchmark_results
-  - task:
-      name: Text Generation
-      type: text-generation
-    dataset:
-      name: WikiText-2 Perplexity
-      type: wikitext
-      config: wikitext-2-raw-v1
-      split: test
-    metrics:
-    - name: perplexity
-      type: perplexity
-      value: 52.20
-    source:
-      name: Local benchmark run (RTX 3090, llama.cpp)
-      url: https://huggingface.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v2-GGUF/tree/main/benchmark_results
 ---
-<p align="center">
-  <img src="cerebellum_banner.png" alt="Cerebellum" width="640">
-</p>
 # Gemma 4 E4B — Cerebellum v2 GGUF (4.2 GB)
 Ablation-informed mixed-precision quantization of [google/gemma-4-e4b-it](https://huggingface.co/google/gemma-4-e4b-it). **4.2 GB** file size, **52.20 perplexity** — smaller and better than [Cerebellum v1](https://huggingface.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v1-GGUF) (4.3 GB, PPL 55.10).
@@ -194,34 +114,17 @@ v2's PPL (52.20) is lower than BF16 (54.58). This happens because certain tensor
 Fits comfortably on a 6 GB GPU at moderate context.
-## Measured launch (RTX 3090, llama.cpp)
-Measured 2026-06-13 on a single RTX 3090 (24 GB), one `llama-server`, KV cache `q8_0`:
-| metric | measured |
-|---|---|
-| decode speed | 116 tok/s |
-| peak VRAM (4-slot serving) | 5.3 GB |
-| max measured context (q8_0 KV) | 65,536 |
-```bash
-llama-server -m Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf \
-  -ngl 99 --parallel 4 -c 24576 --jinja --reasoning-budget 0
-```
-_This rig's measurements; no quality claims beyond them._
 ## Usage
 ```bash
 # llama.cpp
 llama-server \
-  --model Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf \
   --n-gpu-layers 99 \
   --ctx-size 8192
 # Ollama
-echo 'FROM ./Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf' > Modelfile
 ollama create gemma4-e4b-v2 -f Modelfile
 ollama run gemma4-e4b-v2
 ```
@@ -243,7 +146,7 @@ python convert_hf_to_gguf.py gemma-4-e4b-it --outfile gemma4-e4b-bf16.gguf --out
 llama-quantize \
   --imatrix imatrix.dat \
   --tensor-type-file cerebellum_v2_overrides.txt \
-  gemma4-e4b-bf16.gguf Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf Q3_K_M
 ```
 The `imatrix.dat`, `cerebellum_v2_overrides.txt`, and `ablation_results.json` are included in this repo.
@@ -252,7 +155,7 @@ The `imatrix.dat`, `cerebellum_v2_overrides.txt`, and `ablation_results.json` ar
 | File | Size | Description |
 |------|------|-------------|
-| `Gemma-4-E4B-it-Cerebellum-v2-Q3_K_M.gguf` | 4.2 GB | The quantized model |
 | `imatrix.dat` | 4.6 MB | Weight-sensitivity importance matrix |
 | `cerebellum_v2_overrides.txt` | 4.6 KB | 179 tensor type overrides (174 PLE Q5_K + 2 Q6_K + 3 Q2_K) |
 | `ablation_results.json` | 3 KB | Full ablation sweep data (26 tensors) |

   - ablation
   - 3-bit
   - conversational
 ---
 # Gemma 4 E4B — Cerebellum v2 GGUF (4.2 GB)
 Ablation-informed mixed-precision quantization of [google/gemma-4-e4b-it](https://huggingface.co/google/gemma-4-e4b-it). **4.2 GB** file size, **52.20 perplexity** — smaller and better than [Cerebellum v1](https://huggingface.co/deucebucket/Gemma-4-E4B-it-Cerebellum-v1-GGUF) (4.3 GB, PPL 55.10).
 Fits comfortably on a 6 GB GPU at moderate context.
 ## Usage
 ```bash
 # llama.cpp
 llama-server \
+  --model Gemma-4-E4B-it-Cerebellum-v2.gguf \
   --n-gpu-layers 99 \
   --ctx-size 8192
 # Ollama
+echo 'FROM ./Gemma-4-E4B-it-Cerebellum-v2.gguf' > Modelfile
 ollama create gemma4-e4b-v2 -f Modelfile
 ollama run gemma4-e4b-v2
 ```
 llama-quantize \
   --imatrix imatrix.dat \
   --tensor-type-file cerebellum_v2_overrides.txt \
+  gemma4-e4b-bf16.gguf Gemma-4-E4B-it-Cerebellum-v2.gguf Q3_K_M
 ```
 The `imatrix.dat`, `cerebellum_v2_overrides.txt`, and `ablation_results.json` are included in this repo.
 | File | Size | Description |
 |------|------|-------------|
+| `Gemma-4-E4B-it-Cerebellum-v2.gguf` | 4.2 GB | The quantized model |
 | `imatrix.dat` | 4.6 MB | Weight-sensitivity importance matrix |
 | `cerebellum_v2_overrides.txt` | 4.6 KB | 179 tensor type overrides (174 PLE Q5_K + 2 Q6_K + 3 Q2_K) |
 | `ablation_results.json` | 3 KB | Full ablation sweep data (26 tensors) |

cerebellum_banner.png DELETED Viewed

Git LFS Details

SHA256: 78e521f01919cfd30f3f0c1352baa298a0f4108811266e4cc9f9918f175afaa5
Pointer size: 131 Bytes
Size of remote file: 230 kB