Instructions to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

LM Studio
Jan

vLLM

How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

SGLang

How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with Ollama:
```
ollama run hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL
```
Unsloth Desktop

How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.Qwen3-Coder-30B-A3B-Instruct-GGUF-UD-Q4_K_XL

List all available models

lemonade list

Hermes Agent

How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

danielhanchen commited on Jan 30

Commit

b17cb02

verified ·

1 Parent(s): ea401b3

Add files using upload-large-folder tool

Browse files

Files changed (28) hide show

.gitattributes +0 -1
BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-IQ4_NL.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-IQ4_XS.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q2_K.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q2_K_L.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q3_K_M.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q3_K_S.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q4_0.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q4_1.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q4_K_S.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q5_K_S.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q6_K.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-Q8_0.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-IQ1_M.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-IQ1_S.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-IQ2_M.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-IQ2_XXS.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-IQ3_XXS.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-Q2_K_XL.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-Q3_K_XL.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-Q5_K_XL.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-Q6_K_XL.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-Q8_K_XL.gguf +2 -2
Qwen3-Coder-30B-A3B-Instruct-UD-TQ1_0.gguf +2 -2

@@ -63,4 +63,3 @@ Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
 Qwen3-Coder-30B-A3B-Instruct-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
 Qwen3-Coder-30B-A3B-Instruct-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
 imatrix_unsloth.gguf filter=lfs diff=lfs merge=lfs -text
-imatrix_unsloth.gguf_file filter=lfs diff=lfs merge=lfs -text

 Qwen3-Coder-30B-A3B-Instruct-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
 Qwen3-Coder-30B-A3B-Instruct-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
 imatrix_unsloth.gguf filter=lfs diff=lfs merge=lfs -text

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4dda6b698742befc293936f9a7c9e7bfdd8504b5b730456ea901b6eb76f17e8d
-size 49655153376

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:00128df93af07aea9b9313708cf77de7e99c77c36fc4bcf6bf6f7cc7986d2ed7
-size 17310784064

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9946272497b3556c5f8b438272075fc1e77c67ba76393e45c174d3c17f117124
-size 16378075712

 version https://git-lfs.github.com/spec/v1
+oid sha256:c432d5b64272ae30f0df02326619557a1994b034c40139c6e8d7dce928142845
+size 49655154016

 version https://git-lfs.github.com/spec/v1
+oid sha256:27f17516dacc1576ddc8a5e31a33d9c7829fb15dfa502fb208e50b1e70612b56
+size 17310784672

 version https://git-lfs.github.com/spec/v1
+oid sha256:26cd4fef3ada9de84c6c59c4b93f9bc5df4d69a71e18cfe8a17369c2f8374ce7
+size 16378076320

 version https://git-lfs.github.com/spec/v1
+oid sha256:6db9853d31fdb928a731666c9d44e8cdebf52f62f4810cb0908c6c677e6c84b5
+size 11258612896

 version https://git-lfs.github.com/spec/v1
+oid sha256:7add73b0607b498f79157a5f4e4ccddc14ad7afd61d76655e064e1e92476267e
+size 11331542176

 version https://git-lfs.github.com/spec/v1
+oid sha256:30c83da425db2324444b6a6cecaf4c410038a2ec73a78de2436879dc0316a371
+size 14711850144

 version https://git-lfs.github.com/spec/v1
+oid sha256:17d51f5310e9a598e5ac914d30f401fb2d1bc3b6a06a846919099eac09364ae1
+size 13292471456

 version https://git-lfs.github.com/spec/v1
+oid sha256:cf9ea0572e8759348ef7f8cd899fbfecba5110929fc1aa620d18bb6d91e62700
+size 17379990688

 version https://git-lfs.github.com/spec/v1
+oid sha256:7e3ca03775ad2e3ddffe3380a391c34cf675590a54ef27bcaf94a38787bb06a8
+size 19192503456

 version https://git-lfs.github.com/spec/v1
+oid sha256:fadc3e5f8d42bf7e894a785b05082e47daee4df26680389817e2093056f088ad
+size 18556689568

 version https://git-lfs.github.com/spec/v1
+oid sha256:56a7d00783419bcb0ae566253c371bcb3678261bb79881a553539f5679864db4
+size 17456012448

 version https://git-lfs.github.com/spec/v1
+oid sha256:4b78837bbec5ee248e4a5642bf608b6793721af41b92589e40c8da0bce58b907
+size 21725584544

 version https://git-lfs.github.com/spec/v1
+oid sha256:a1a93585e669f77ac0b5f3aaf18241e051028a6c467c4eac376fd7d1c7a1ebca
+size 21080513696

 version https://git-lfs.github.com/spec/v1
+oid sha256:100b5121d09553fb1af3b873b21fb3ec3da5c306fc5cb09bd338c48e21b10875
+size 25092535456

 version https://git-lfs.github.com/spec/v1
+oid sha256:4ff1cff607804037bf6d2168249c570baa4e1621292b159c0e06591e0d7c3066
+size 32483935392

 version https://git-lfs.github.com/spec/v1
+oid sha256:87ddf47ea7be2311968f49c4585f1c12cb9a4400f1112b849449ce27e0732d0f
+size 9627540640

 version https://git-lfs.github.com/spec/v1
+oid sha256:2dc573a4748d9d9cd45482e03dc786b4151f9174d66f5f982fda64a5b346dc52
+size 8914328736

 version https://git-lfs.github.com/spec/v1
+oid sha256:7055a02d4d974fd68b26b095a9c676f099507aed1171f89a47275025df3f521d
+size 10837007520

 version https://git-lfs.github.com/spec/v1
+oid sha256:51d0a284dc26f11102032b7aa8eca28619b8c399478afc5a0ab7180cd711bf13
+size 10333691040

 version https://git-lfs.github.com/spec/v1
+oid sha256:d6b85d2b6633c55be3751848acc74fab40d28d93b27dabacd96ceabe10962057
+size 12848766112

 version https://git-lfs.github.com/spec/v1
+oid sha256:3dee4686d756e8842aec638c5725666c76584edce76d71088732b52d2c5d7a45
+size 11788590240

 version https://git-lfs.github.com/spec/v1
+oid sha256:69cd7578d77dffc0b17e34bad9ef998d08ae0e20ccceef21bad4e7eb3d8c553b
+size 13806312608

 version https://git-lfs.github.com/spec/v1
+oid sha256:2841aa314d916434860cfb8990347528dcdfe5c350dbcb9d1461dbee88ff2533
+size 17665334432

 version https://git-lfs.github.com/spec/v1
+oid sha256:eb331a4eee8eb6b5a8eb25f44f96f45c71b8d10f553c0a456190dd590a7ef77d
+size 21740305568

 version https://git-lfs.github.com/spec/v1
+oid sha256:b09fcc58fd27cd43dacdf6840c5829125c8ed9463b2a386ce2667395b1292f63
+size 26340328608

 version https://git-lfs.github.com/spec/v1
+oid sha256:c867adc2a5f823ce64f5aa0717af3c4b660959c3f16c23c5f70f7bf8ba795e28
+size 35989947552

 version https://git-lfs.github.com/spec/v1
+oid sha256:fcb403c1d7a98bc0ffbe98ec7fbfafe1cdd0050af808ba4af8cd6dcaa807f8d9
+size 8005213344