Instructions to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Use Docker

docker model run hf.co/rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Ollama
How to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with Ollama:
```
ollama run hf.co/rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M
```
Unsloth Desktop

How to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with Docker Model Runner:
```
docker model run hf.co/rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M
```

Lemonade

How to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.qwen36-a3b-claude-coder-llama.cpp-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "rafw007/qwen36-a3b-claude-coder-llama.cpp-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

What version of the llama.cpp is this GGUF for? Error on latest.

by randomly-generated-username - opened Jul 21

Discussion

randomly-generated-username

Jul 21

Hello! I have an error "llama_model_load: error loading model: missing tensor 'blk.0.ssm_dt.bias'" on the latest llama.cpp (b10075, for MS Windows) and older (b8981 and b9360, for MS Windows) too. On what versions of llama.cpp this GGUF was tested and works, or it's for ik_llama only?

Slevin5

Aug 13

Same blk.0.ssm_dt.bias error here — I reproduced it and dug into it. Two layers to
the problem; the first is fixable, the second is the real blocker.

1. ssm_dt.bias is a tensor-NAME issue (fixable, but not the whole story).
This GGUF stores the SSM delta-time bias as blk.N.ssm_dt (no .bias suffix), while
every loader looks for blk.N.ssm_dt.bias. It's the same tensor — shape [32] F32,
identical to the stock Qwen3.6-35B-A3B-Q4_K_M (lmstudio-community), which loads fine.
I renamed the 30 blk.*.ssm_dt → blk.*.ssm_dt.bias in the header (tensor data copied
verbatim, no re-quantization). That clears the ssm_dt.bias error — but the model
still won't load. Next error:

check_tensor_dims: tensor 'blk.3.attn_k.weight' has wrong shape;
expected 2048, 0, got 2048, 512, 1, 1

2. The real blocker: this is a non-mainline conversion.
Beyond the tensor name, this GGUF carries metadata the published loaders don't
understand: a full qwen35moe.vision.* block, rope.mrope_interleaved=1,
ssm.v_head_reordered=1, feed_forward_length=0. The core arch params
(head_count=16, head_count_kv=2, key_length=256, block_count=40, expert_count=256, …)
are byte-for-byte identical to the stock base model — but these extra keys route the
loader down a multimodal/reordered path that computes attn_k's expected shape with
a 0 dimension → fatal mismatch.

I tested the renamed GGUF on:

ghcr.io/ggml-org/llama.cpp:full-cuda — both the ~2-week-old build and latest
(which includes PR #19468, the merged Qwen3.5 support that loads the stock base fine)
ghcr.io/ikawrakow/ik-llama-cpp:cu12-full

All three fail with the same attn_k shape error.

The README points to "the Qwen3.5 loader from PR #19435", but that PR ("Qwen3.5 dense
and MoE support, no vision") was merged Feb 8 2026 and reverted the next day
(PR #19453), superseded by #19468. #19435 explicitly has no vision and doesn't mention
mrope_interleaved/v_head_reordered — so it isn't the converter that produced this
file. This GGUF was converted by some other fork, whose matching loader is not
published as an image.

The crucial missing part: the exact runtime this GGUF was tested on — repo +
commit, or a prebuilt image/tag. Without it, the file is unusable on any published
build. @rafw007 — could you share the build you validated with?

Alternatively, a re-export with standard tensor names (ssm_dt.bias) and standard
qwen35moe metadata (no vision block / reorder flags) would load on current mainline
llama.cpp — and since the README states the weights are stock Qwen3.6-35B-A3B, that
re-export would be functionally identical to the base model + the documented no-think
sampling preset.

(For anyone who just wants the coder behavior: the README is upfront that it's
stock weights + a system prompt + sampling config, no fine-tune. So stock
Qwen3.6-35B-A3B + no-think + --jinja + the documented sampling gets you there on
any working build — no need for this specific GGUF.)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment