Instructions to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
# Run inference directly in the terminal:
./llama-cli -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Use Docker

docker model run hf.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

LM Studio
Jan

vLLM

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Ollama
How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Ollama:
```
ollama run hf.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
```
Unsloth Desktop

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Docker Model Runner:
```
docker model run hf.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
```

Lemonade

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-Cerebellum-GGUF-Q3_K_M

List all available models

lemonade list

Hermes Agent

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

deucebucket commited on Jun 15

Commit

67dd37f

verified ·

1 Parent(s): 5a405d1

docs: rewrite card around 14GB recommended build; benches in Evaluations; trim legacy v1/v3 clutter

Browse files

Files changed (1) hide show

README.md +30 -82

README.md CHANGED Viewed

@@ -76,118 +76,66 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value: 0.652
     source:
       name: Local benchmark run (RTX 3090, llama.cpp)
       url: https://huggingface.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF/tree/main/benchmark_results
 ---
-## 14 GB variant — recommended (the sweet spot)
-`Qwen3.6-35B-A3B-Cerebellum-14GB.gguf` — 14.0 GB, 3.34 bpw. Q3_K_M base + Q4_K on the routed `ffn_down_exps` (ablation-informed).
-Size ladder (1x RTX 3090, llama.cpp; HumanEval/+ via upstream EvalPlus, greedy temp 0, n=164):
-| build | size | HumanEval / HumanEval+ |
-|---|---|---|
-| micro | 11.96 GB | 90.9 / 87.2 |
-| **14 GB** | **14.0 GB** | **93.3 / 90.2** |
-| uniform Q3_K_M | 16.0 GB | 91.5 / 89.0 |
-| Base | 17.3 GB | 92.7 / 89.0 |
-The 14 GB build has the highest HumanEval of the family — above the 16 GB uniform Q3_K_M (-2 GB) and matching the 17.3 GB Base (-3.3 GB). Long-context needle recall passes to 90K+. Decode ~168 tok/s (3B-active MoE); fits 160K+ context at ~19 GB on a 24 GB card.
-Launch (single card): `llama-server -m Qwen3.6-35B-A3B-Cerebellum-14GB.gguf -ngl 99 -fa on --reasoning off`. Evidence: `benchmark_results/14gb/`.
----
 <p align="center">
   <img src="cerebellum_banner.png" alt="Cerebellum" width="640">
 </p>
 # Qwen 3.6 35B-A3B — Cerebellum GGUF
-Sensitivity-guided mixed-precision quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). Two variants available:
-| Variant | File | Size | BPW |
-|---------|------|------|-----|
-| **Cerebellum v3** (recommended) | `Qwen3.6-35B-A3B-Cerebellum-v3-Q3_K_M.gguf` | **11 GB** | 2.76 |
-| Cerebellum v1 (legacy) | `Qwen3.6-35B-A3B-Cerebellum-Q3_K_M.gguf` | 12 GB | 2.73 |
-Cerebellum measures which weight groups survive extreme compression and which don't, then writes a single GGUF with per-tensor precision assignments. v3 uses 360 tensor-level overrides guided by group ablation and reverse ablation analysis.
-## Benchmarks
-All benchmarks measured directly on these GGUF files using llama.cpp inference with cleaned evaluation harness.
-The model-index metadata in this card's frontmatter mirrors the recommended v3 build. Protocol: local llama.cpp chat harness on RTX 3090, temperature 0, no thinking mode. Full per-question artifacts are in `benchmark_results/v3/`.
-| Benchmark | **v3 (11 GB)** | v1 (12 GB) | Q3_K_M (15.6 GB) |
-|-----------|:---:|:---:|:---:|
-| **ARC-Challenge** | **95.8%** | 94.8% | 96.1% |
-| **HellaSwag** | **92.3%** | 91.5% | 91.5% |
-| **MMLU-Redux** | **75.0%** | 73.9% | 74.1% |
-| **HumanEval base** | **70.7%** | — | 64.0% |
-| **HumanEval+** | **65.2%** | — | 56.7% |
-| Vision smoke (36 images) | **100%** | 100% | — |
-v3 at 11 GB is **29% smaller** than stock Q3_K_M (15.6 GB) while outperforming it on 4 of the 5 measured benchmarks (ARC is the one it loses; the vision check has no Q3_K_M baseline to compare). The Q2_K regularization effect on gate/mixing weights actively improves downstream task performance despite reducing perplexity.
-## v3 Allocation
-| Group | Precision | Rationale |
-|-------|-----------|-----------|
-| `attn_qkv` | Q3_K_M | Critical for vision and attention routing |
-| `ssm_out` | Q3_K_M | Most sensitive tensor per ablation (+0.24 PPL) |
-| `ffn_gate_exps` | Q2_K | Q2_K regularization outperforms Q3_K_M |
-| `ffn_up_exps` | Q2_K | Q2_K regularization outperforms Q3_K_M |
-| `ffn_down_exps` | Q2_K | Acceptable loss for size savings |
-| `ffn_gate_shexp` | Q2_K | Q2_K regularization outperforms Q3_K_M |
-| `ffn_up_shexp` | Q2_K | Q2_K regularization outperforms Q3_K_M |
-| `ffn_down_shexp` | Q2_K | Q2_K regularization outperforms Q3_K_M |
-| `attn_gate` | Q2_K | Q2_K regularization outperforms Q3_K_M |
-| `ssm_alpha`, `ssm_beta` | Q2_K | Q2_K regularization outperforms Q3_K_M |
-Protected: all norms (F32), SSM state params (F32), router tensors (default).
-## Ablation Data
-Full ablation methodology and results are in the `ablation/` directory:
-- `group_ablation_results.log` — Forward ablation: demote each group to Q2_K, measure PPL
-- `reverse_ablation_results.log` — Reverse ablation: from fully-demoted v1, restore each group
-- `cerebellum_v3_overrides.txt` — The 360-line tensor type override file used for v3
-Key finding from reverse ablation: **7 of 10 groups perform better at Q2_K than Q3_K_M** — imatrix-guided Q2_K acts as beneficial regularization on gate, mixing, and shared expert weights.
 ## Usage
 ```bash
-# v3 (recommended, 11 GB)
-llama-server --model Qwen3.6-35B-A3B-Cerebellum-v3-Q3_K_M.gguf \
-  --mmproj mmproj-F16.gguf --n-gpu-layers 99 --ctx-size 8192
-# v1 (legacy, 12 GB)
-llama-server --model Qwen3.6-35B-A3B-Cerebellum-Q3_K_M.gguf \
-  --mmproj mmproj-F16.gguf --n-gpu-layers 99 --ctx-size 8192
 ```
 ## Files
-| File | Size | Description |
-|------|------|-------------|
-| `Qwen3.6-35B-A3B-Cerebellum-v3-Q3_K_M.gguf` | 11 GB | v3 — recommended, 29% smaller than Q3_K_M |
-| `Qwen3.6-35B-A3B-Cerebellum-Q3_K_M.gguf` | 12 GB | v1 — legacy |
-| `mmproj-F16.gguf` | 858 MB | Vision projection (F16) |
-| `benchmark_results/v3/` | — | Full benchmark JSON artifacts for v3 |
-| `ablation/` | — | Ablation logs and override files |
 ## Methodology
-Built with [Cerebellum](https://github.com/deucebucket/cerebellum) — sensitivity-guided mixed-precision quantization. v3 uses unsloth coder imatrix for importance-weighted quantization within each precision level.
-Quantized by [@deucebucket](https://huggingface.co/deucebucket).
 ## Independent records
-This build has a recorded data point in [club-3090's BENCHMARKS](https://github.com/noonghunna/club-3090/blob/master/BENCHMARKS.md) (author-rig numbers from a full `report.sh --full` chain: bench n=5, verify-full pass, soak-continuous pass). The same report led to a correction of their engine support table for this model ([issue #390](https://github.com/noonghunna/club-3090/issues/390), [PR #393](https://github.com/noonghunna/club-3090/pull/393)). The numbers there are author-reported, not club-validated.

     metrics:
     - name: pass@1
       type: pass@1
+      value: 0.902
     source:
       name: Local benchmark run (RTX 3090, llama.cpp)
       url: https://huggingface.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF/tree/main/benchmark_results
 ---
 <p align="center">
   <img src="cerebellum_banner.png" alt="Cerebellum" width="640">
 </p>
 # Qwen 3.6 35B-A3B — Cerebellum GGUF
+Sensitivity-guided mixed-precision quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B). Cerebellum measures which weight groups survive extreme compression and which don't, then writes a single GGUF with per-tensor precision assignments — a standard GGUF that runs on stock `llama.cpp`, no fork.
+| Variant | File | Size | BPW | Best for |
+|---------|------|------|-----|----------|
+| **14 GB** (recommended) | `Qwen3.6-35B-A3B-Cerebellum-14GB.gguf` | **14.0 GB** | 3.34 | best coding, 160K+ context |
+| v3 (smallest) | `Qwen3.6-35B-A3B-Cerebellum-v3-Q3_K_M.gguf` | 11 GB | 2.76 | tightest VRAM, vision |
+## Evaluations
+Coding — upstream EvalPlus (`evalplus.codegen` against `llama-server`, greedy / temp 0, n=164), same protocol across the size ladder:
+| build | size | HumanEval | HumanEval+ |
+|-------|------|:---:|:---:|
+| micro | 11.96 GB | 90.9 | 87.2 |
+| **14 GB (recommended)** | **14.0 GB** | **93.3** | **90.2** |
+| uniform Q3_K_M | 16.0 GB | 91.5 | 89.0 |
+| Base | 17.3 GB | 92.7 | 89.0 |
+Long-context: needle recall passes to 90K+ (`verify-stress`). Throughput: ~168 tok/s decode (3B-active MoE); fits 160K+ context at ~19 GB on a 24 GB card. Per-question artifacts in `benchmark_results/14gb/`.
+## Why the 14 GB over v3
+v3 (11 GB) is the tightest-VRAM build. The 14 GB spends ~3 GB more to promote the routed `ffn_down_exps` to Q4_K — the group the ablation identifies as where the quality lives — and that gives it the family's **best coding** plus 160K+ context headroom. It posts above the 16 GB uniform Q3_K_M (−2 GB) and matches the 17.3 GB Base (−3.3 GB): the Base's extra promotions buy ~0 coding, so 14 GB is the efficient point. Pick v3 only when VRAM is tight or you need the vision projector.
 ## Usage
 ```bash
+# 14 GB (recommended)
+llama-server -m Qwen3.6-35B-A3B-Cerebellum-14GB.gguf -ngl 99 -fa on --reasoning off
+# v3 (smallest, with vision)
+llama-server -m Qwen3.6-35B-A3B-Cerebellum-v3-Q3_K_M.gguf --mmproj mmproj-F16.gguf -ngl 99 -c 8192
 ```
 ## Files
+| File | Size | Notes |
+|------|------|-------|
+| `Qwen3.6-35B-A3B-Cerebellum-14GB.gguf` | 14 GB | recommended — best coding, 160K+ ctx |
+| `Qwen3.6-35B-A3B-Cerebellum-v3-Q3_K_M.gguf` | 11 GB | smallest; vision (with mmproj) |
+| `mmproj-F16.gguf` | 858 MB | vision projector (F16) |
+| `benchmark_results/` | — | per-question evaluation artifacts |
+| `ablation/` | — | ablation logs + tensor override maps |
 ## Methodology
+Built with [Cerebellum](https://github.com/deucebucket/cerebellum) — sensitivity-guided mixed-precision quantization: crush each tensor group, measure the impact, allocate precision under a size budget, output a plain GGUF. imatrix-calibrated. Quantized by [@deucebucket](https://huggingface.co/deucebucket).
 ## Independent records
+This line has a recorded data point in [club-3090's BENCHMARKS](https://github.com/noonghunna/club-3090/blob/master/BENCHMARKS.md) (author-rig numbers from a full `report.sh --full` chain). The same report corrected their engine-support table for this model ([issue #390](https://github.com/noonghunna/club-3090/issues/390), [PR #393](https://github.com/noonghunna/club-3090/pull/393)). Numbers there are author-reported, not club-validated.