Instructions to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Use Docker

docker model run hf.co/geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

LM Studio
Jan

vLLM

How to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Ollama
How to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with Ollama:
```
ollama run hf.co/geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M
```
Unsloth Desktop

How to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with Docker Model Runner:
```
docker model run hf.co/geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M
```

Lemonade

How to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3-Coder-30B-A3B-Instruct-f16-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f16:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

F16-F32?

by Trilogix1 - opened Nov 6, 2025

Discussion

Trilogix1

Nov 6, 2025

Hi, this model is particularly interesting, (good job :)
I was wondering if you can proceed with f16 and f32 and maybe try to use f32 also in the weights when doing quantization.

Thanks in advance.

geoffmunn

Owner Nov 7, 2025

•

edited Nov 7, 2025

Here you go 😀
https://huggingface.co/geoffmunn/Qwen3-Coder-30B-A3B-Instruct-f32

The base gguf model is definitely f32 - it was twice the size of the f16 version, but all the quantised model are exactly the same size so I don't think you'll notice any difference except for more memory usage (perhaps).

I haven't been able to test them yet, so let me know how it goes.

Trilogix1

Nov 7, 2025

The F32 will aways be of superior quality. Even just 0.2% margin error is enough to make a result unreliable. What is missing now to maximize or push further the accuracy is the Q8_K_XL quantization and maybe Thinking mode (so the Qwen/Qwen3-30B-A3B-Thinking-2507 in f32 Q8_k_xl). Right now your quantized models produce the best results with Qwen coder :)
Can you share the script you used to convert them with f16/f32?

Thank you very much for your work, I am adding your F32 GGUF in my top 10.

geoffmunn

Owner Nov 7, 2025

I'm using the llama-quantize script to generate these - and there's no such thing as a K_XL quant option unfortunately. I have actually done quite a bit of comparison work across all the standard Qwen3 models and the mid-range quant options are definitely the best across all the temperature ranges - you don't get any benefit from a higher precision model like Q8_0. I'm not saying they produce bad results, it's just not worth the resource investment.

I can generate the f32 versions of the Thinking-2507 model if that would help?

Trilogix1

Nov 8, 2025

The K_XL quants are made with the Unsloth method, but you are using the traditional LLama-Quantize. (I would appreciate the script anyway, so if possible just include it in the repo or paste it here).
Then According to my tests in math, coding, creative writing, general knowledge etc, the Q4 and Q6 perform quite good compared to the others (there is something that breaks the Q5, making it loop or stop in large ctx, Tested with many versions of llama.cpp, which surprisingly do not happen with the Q4-Q6-Q8_k_m ). I notice also that the Q_K_xl work really good, with a bit more accuracy, keeping the speed.
I couldn´t establish a test verdict between F16 and F32 yet (I can´t notice the difference yet or maybe the F32 is behaving worse, especially in Q5) but mathematically speaking, it should matter.
When I say that your GGUF models are the highest quality available for download, I mean it, so again great job :)

If you could generate the F16 or F32 of the Thinking 2507, that would be great. Another model that has potential and deserve a try with same treatment is: https://huggingface.co/armand0e/gpt-oss-20b-glm-4.6-distill and https://huggingface.co/TeichAI/gpt-oss-20b-claude-4.5-sonnet-high-reasoning-distill.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment