Instructions to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
# Run inference directly in the terminal:
llama cli -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
# Run inference directly in the terminal:
./llama-cli -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Use Docker

docker model run hf.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

LM Studio
Jan

vLLM

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Ollama
How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Ollama:
```
ollama run hf.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
```
Unsloth Desktop

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Docker Model Runner:
```
docker model run hf.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M
```

Lemonade

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-Cerebellum-GGUF-Q3_K_M

List all available models

lemonade list

Hermes Agent

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF:Q3_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Amazing quality for the size!

by TheodoreH - opened May 31

Discussion

TheodoreH

May 31

•

edited May 31

Hi. Cerebellum quantization is the best among not-usual ('highly specific' quantizations(like APEX, or REAP, MAgic quants, etc)). It works well on almost only CPU and it is not hallucinating(mostly) and does not make mistakes(almost). Also used Gemma-4 26b heretic version - VERY capable one, and again, size is unbelivable for the quality it outputs.(using also IQ4-XS version by HauHau - not only it weights 2 GB's more, but it also seem same quality, or Cerebellum is sometimes winning(hard to tell really, keeping both for now), but one thing is important, that Cerebellum supports Vision, while IQ4-XS does not, but if it would, there would't enough space for that).
Wondering if similar Heretic version would be for Qwen 35b too?

deucebucket

Owner Jun 2

Hi. Cerebellum quantization is the best among not-usual ('highly specific' quantizations(like APEX, or REAP, MAgic quants, etc)). It works well on almost only CPU and it is not hallucinating(mostly) and does not make mistakes(almost). Also used Gemma-4 26b heretic version - VERY capable one, and again, size is unbelivable for the quality it outputs.(using also IQ4-XS version by HauHau - not only it weights 2 GB's more, but it also seem same quality, or Cerebellum is sometimes winning(hard to tell really, keeping both for now), but one thing is important, that Cerebellum supports Vision, while IQ4-XS does not, but if it would, there would't enough space for that).

Thank you! I'm glad you're having a good time with the models, when I started doing this, I had no clue if it would be better or a waste of time. After benchmarking them in loops in opencode on random idea projects for a day or 2 each, I felt they were good enough to let others have, along with the process. So I'm really glad others experiences are reflecting mine.

Wondering if similar Heretic version would be for Qwen 35b too?

Yeah I'm looking into it now actually. I didn't find any other Qwen 35b heretics out there, so there's no base to start from which is the easy road as the recipe for the quants is pretty literally drag and drop, when used on the same exact model. So I'll do my best to reply back if I get a working heretic i'm happy with, or if you see one first, i can try to quant it and see if it does anything. Let me know if you have another other suggestions too, I'm definitely open to trying it.

TheodoreH

Jun 2

Hi. Cerebellum quantization is the best among not-usual ('highly specific' quantizations(like APEX, or REAP, MAgic quants, etc)). It works well on almost only CPU and it is not hallucinating(mostly) and does not make mistakes(almost). Also used Gemma-4 26b heretic version - VERY capable one, and again, size is unbelivable for the quality it outputs.(using also IQ4-XS version by HauHau - not only it weights 2 GB's more, but it also seem same quality, or Cerebellum is sometimes winning(hard to tell really, keeping both for now), but one thing is important, that Cerebellum supports Vision, while IQ4-XS does not, but if it would, there would't enough space for that).

Thank you! I'm glad you're having a good time with the models, when I started doing this, I had no clue if it would be better or a waste of time. After benchmarking them in loops in opencode on random idea projects for a day or 2 each, I felt they were good enough to let others have, along with the process. So I'm really glad others experiences are reflecting mine.

Wondering if similar Heretic version would be for Qwen 35b too?

Yeah I'm looking into it now actually. I didn't find any other Qwen 35b heretics out there, so there's no base to start from which is the easy road as the recipe for the quants is pretty literally drag and drop, when used on the same exact model. So I'll do my best to reply back if I get a working heretic i'm happy with, or if you see one first, i can try to quant it and see if it does anything. Let me know if you have another other suggestions too, I'm definitely open to trying it.

Does it need to be specific one? There are plenty without MTP and in 3.6 version:
https://huggingface.co/TopherAU/Qwen3.6-35B-A3B-uncensored-heretic-GGUF
https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF
https://huggingface.co/mradermacher/Qwen3.6-35B-A3B-Abliterated-Heretic-BF16-GGUF
https://huggingface.co/Youssofal/Qwen3.6-35B-A3B-Abliterated-Heretic-GGUF

Or just this, if it dont really matter if it is heretic one, but it is abliterated still:
https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive

TheodoreH

Jun 2

•

edited Jun 2

Also, it is greatly impressive that other languages is mostly untouched with your Quant metod! Tried a lot of Reaps and dont really know how they could be usable even on 4km. Apex quants is somewhat good, but yours is probably the best, and space required even less, really impressive. Totally waiting for unrestricted version!

deucebucket

Owner Jun 2

•

edited Jun 2

Does it need to be specific one?

No it definitely doesn't, i just heard heretic models tend to be a bit crazy, so i just didn't know where one to go with, the last heretic base was suggested as pretty normal, and it was. So if you know which of these is probably the least off base that'd help, but i do wonder if some how cerebellum could rip a model back from the jaws of madness, it already seems to do odd stuff at smaller sizes so who knows. im always down to experiment, i just pick and choose as i run the jobs while im at work on my gaming rig, and im gaming or using the models when im home, so there may be some lag, but i do work a full 40 so it gives me a good chunk of hours to run stuff. I do wish i had more vram, im curious what it could do to a super large model like deepseek or similar, as on my 3090 i was trying to run heretic on the q36 35b and could not get it to fit no matter what i did and was looking at a multi week endeavor lol im saving for another 3090 eventually so hopefully it'll grow the abilities.

TheodoreH

Jun 2

•

edited Jun 2 by

deucebucket

How did you even end up with that quantization, and why nobody else did that?

Hah, I think a lot of people would hate this answer, but I honestly feel my lack of knowledge on the subject as a whole also takes away what's impossible. I had it explained to me like I was 5 how quantz work, and I explained back like I was 5 (lol) how I think it should work. In my head I guess I can conceptualize even foreign subjects as long as I can break them down to base principals. So after projects kept turning into other projects, and morphing into new projects, I landed on cerebellum.

it was originally a project to try to make a MoA of Bonsai 1bit 9b models, then turned into a way to generate I matrix on cpu with osmosis. Then while thinking about quantz, I was just thinking about how much of the brain you'd need just to think and have thoughts. Well you could trim away a large amount, which is funny because in movies the brain in a jar is always a full brain, but it's probably because a jar with a small piece of a brain wouldn't be recognized. So I just wondered how deep into models we can go when quanting. So I just ran ppl tests on quanted layers to find the perfect zone. And there's actually layers that perform better at lower quants, the way I look at it is: if you know a lot about cooking, but are bad at math, what happens to your math knowledge if you were to silence the noise of cooking in a way that the numbers spoke louder, in a way where there wasn't so much useless noise surrounding the thoughts.

Also, I think I'm on the spectrum 🤣

TheodoreH

Jun 2

which is funny because in movies the brain in a jar is always a full brain, but it's probably because a jar with a small piece of a brain wouldn't be recognized. So I just wondered how deep into models we can go when quanting.

that is interesting thought. But during deletion of the centers that rule the movement or other bodily processes - it is possible to harm those that is important for thinking alone. and so on and so on...

But, If you take away coocking, would it really be better at math? But you didnt delete anything from model like a surgeon would. But yet, you made the model weight less, and produce not that big amount of noise

TheodoreH

Jun 4

Now, I talk about Cerebellum quants where it is possible, because this one is really capable and usable in such small size, truly impressive! But really, could it been done a bit better, so some quants are 3K, and even 4K to fit in 14 gbRAM? But anyway, this is more than enough, because everywhere and everybody says that for low VRam rigs 2b-4b is the way to go, but 8-10t/s with 35 and 26b models on 4Vram and 16 ram, is super usable, 4b dont stand a chance with it. Waiting for unrestricted version, because again, APEX-mini, while fitting on 16gb, is quite broken(seems like calibration is not as good as Cerebellum`s!)

deucebucket

Owner Jun 5

Now, I talk about Cerebellum quants where it is possible, because this one is really capable and usable in such small size, truly impressive! But really, could it been done a bit better, so some quants are 3K, and even 4K to fit in 14 gbRAM? But anyway, this is more than enough, because everywhere and everybody says that for low VRam rigs 2b-4b is the way to go, but 8-10t/s with 35 and 26b models on 4Vram and 16 ram, is super usable, 4b dont stand a chance with it. Waiting for unrestricted version, because again, APEX-mini, while fitting on 16gb, is quite broken(seems like calibration is not as good as Cerebellum`s!)

thats awesome to hear! currently working on the heretic gguf for qwen 3.6 its just taking for ever on my system. But one day lol

eta current 5.0s avg/tensor 31m34s total 168h58m │done Thu 2026-06-11 22:18 CDT

its a slog, and then i gotta hope its good enough to let anyone see, or delete it in shame and learn from the data lol

TheodoreH

Jun 5

Now, I talk about Cerebellum quants where it is possible, because this one is really capable and usable in such small size, truly impressive! But really, could it been done a bit better, so some quants are 3K, and even 4K to fit in 14 gbRAM? But anyway, this is more than enough, because everywhere and everybody says that for low VRam rigs 2b-4b is the way to go, but 8-10t/s with 35 and 26b models on 4Vram and 16 ram, is super usable, 4b dont stand a chance with it. Waiting for unrestricted version, because again, APEX-mini, while fitting on 16gb, is quite broken(seems like calibration is not as good as Cerebellum`s!)

thats awesome to hear! currently working on the heretic gguf for qwen 3.6 its just taking for ever on my system. But one day lol

eta current 5.0s avg/tensor 31m34s total 168h58m │done Thu 2026-06-11 22:18 CDT

its a slog, and then i gotta hope its good enough to let anyone see, or delete it in shame and learn from the data lol

really waiting for it! Tried again some other Qwen 35b quants, with per weight and other, few variants and all they are really broken, broken compared to your Cerebellum Qwen 3.6.

TheodoreH

Jun 10

Hi, seems like No Qwen Heretic coming?

deucebucket

Owner Jun 11

Hi, seems like No Qwen Heretic coming?

haha, sorry i got off track with other projects after one pretty bad failure with a heretic, the hauhau one. My brain jumps around quite a bit, but its still in the list. Thank you for reminding me, i have a few other candidates i have been wanting to try as well. Just gotta get some free time away from work and not wanting to game lol. And i refuse to release something subpar, so if it doesn't pass i am not gonna push it out, not saying it'll be perfect by any means.

These are the other things i fell into, all model related if you wanna check them out, and like how my brain tends to work :)
https://github.com/deucebucket/clanker
https://github.com/deucebucket/cerebellum-brainloop

deucebucket

Owner Jun 11

Hi, seems like No Qwen Heretic coming?

https://huggingface.co/deucebucket/Qwen3.6-35B-A3B-Heretic-Cerebellum-GGUF - keep an eye here,

deucebucket

Owner Jun 12

both qwen heretics are live now:
https://huggingface.co/deucebucket/Qwen3.6-35B-A3B-Heretic-Cerebellum-GGUF
https://huggingface.co/deucebucket/Qwen3.6-27B-Heretic-Cerebellum-GGUF

both passed the full gate suite before upload, the numbers and per-question results are in each repo. fun one: the heretic 35b at 11.96gb beat the uniform q3_k_m of the same weights, which is 16.87gb, on ppl, mmlu and humaneval+. same harness, same weights, just different bit placement.

on your 3k/4k sizes for 14gb ram question, thats actually in the works. the whole point of the budget tool is i give it a size and the ablation data and it figures out the rest, so a proper size ladder per model is coming instead of one take-it-or-leave-it file. no eta because i work a full 40 and refuse to push anything that doesnt pass the gates, but its on the board.

TheodoreH

Jun 12

•

edited Jun 12

both qwen heretics are live now:
https://huggingface.co/deucebucket/Qwen3.6-35B-A3B-Heretic-Cerebellum-GGUF
https://huggingface.co/deucebucket/Qwen3.6-27B-Heretic-Cerebellum-GGUF

both passed the full gate suite before upload, the numbers and per-question results are in each repo. fun one: the heretic 35b at 11.96gb beat the uniform q3_k_m of the same weights, which is 16.87gb, on ppl, mmlu and humaneval+. same harness, same weights, just different bit placement.

on your 3k/4k sizes for 14gb ram question, thats actually in the works. the whole point of the budget tool is i give it a size and the ablation data and it figures out the rest, so a proper size ladder per model is coming instead of one take-it-or-leave-it file. no eta because i work a full 40 and refuse to push anything that doesnt pass the gates, but its on the board.

incredible, will test right away! On benchmarks it seems better, corious to see it in the field. Also, recently(after 3rd time redownloading, because it was producing bad outputs(truncating answers, heavily hallucinating)) installed Apex qwen 35b 3.6 Mini - and it seems surpassing Gemma IQ4XS in some moments(was even thinking that Gemma is even worse), but after more testing and comparing, find out that Gemma is actually more creative, but only if you add right commands, Qwen is good at default creativity, but lacks depth in bigger tasks even with same requests as for Gemma, but since those are the best models, right now, and they surprisingly run well(8-9 t/s) it is great to have them both. It is mindblowing, how today`s 20-30b models can fit so much in them and even competing with biggest models from recent past, or even beating them totally

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment