Instructions to use agentionai/Qwen3.8-27B-AP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use agentionai/Qwen3.8-27B-AP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Use Docker

docker model run hf.co/agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use agentionai/Qwen3.8-27B-AP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "agentionai/Qwen3.8-27B-AP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "agentionai/Qwen3.8-27B-AP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Ollama
How to use agentionai/Qwen3.8-27B-AP-GGUF with Ollama:
```
ollama run hf.co/agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M
```
Unsloth Desktop

How to use agentionai/Qwen3.8-27B-AP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use agentionai/Qwen3.8-27B-AP-GGUF with Docker Model Runner:
```
docker model run hf.co/agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M
```

Lemonade

How to use agentionai/Qwen3.8-27B-AP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.8-27B-AP-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use agentionai/Qwen3.8-27B-AP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use agentionai/Qwen3.8-27B-AP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "agentionai/Qwen3.8-27B-AP-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-27B · Agention Precision GGUF

Same size. Same speed. Closer to the model Qwen trained than any other quant

Agention Precision quants are the highest precision quants of Qwen3.8 27B byte-for-byte. Each file has a non-uniform assignment and low-loss error correction built with a custom encoder.

This is a drop-in GGUF pack for Qwen3.8-27B: standard llama.cpp types, no fork, no flags. Vision projector and MTP draft head included.

Every 27B quant gives something up. These give up less. We measured the leading public GGUFs against the full BF16 model on one protocol. At every size we ship, ours is the closest to Qwen’s next-token distribution.

Every quant is tested against three corpora: unseen technical prose, general web text and Wikipedia.

Swap cost is zero. VRAM and tokens/sec stay the same. The file behaves more like the weights Qwen released.

Start here	File	Size	32k VRAM	Fits	Gain vs leading same-size quant
Most headroom	`AP-Q4_K_XL`	16.35 GiB	~18 GiB	24 GB	4% closer to BF16; hardest 1% of tokens 7% closer
24 GB, smaller	`AP-Q4_K_M`	15.33 GiB	~17 GiB	24 GB	5% closer on unseen technical text; hardest 1% of tokens 9% closer
Default	`AP-IQ4_XS`	13.27 GiB	~15 GiB	16 GB	8% closer to BF16; hardest 1% of tokens 9% closer
Need headroom	`AP-Q3_K_XL`	12.24 GiB	~14 GiB	16 GB + longer ctx	10% closer on unseen technical text
12 GB, more quality	`AP-IQ3_S`	11.21 GiB	~13 GiB	16 GB at 8–16k	10% closer on unseen technical text, 8% on web; hardest 1% of tokens 13% closer
12 GB, balanced	`AP-IQ3_XS`	10.70 GiB	~12.5 GiB	12 GB at 8–16k	6–8% closer than the best research quant at this size, 19% on wikitext-2
12 GB / multi-model	`AP-IQ3_XXS`	10.00 GiB	~12 GiB	12 GB at 8–16k	12–25% closer than the best ~10 GiB research quant
12 GB, smallest	`AP-IQ2_S`	8.95 GiB	~10.9 GiB	12 GB at 8–16k	7% closer on unseen technical text, 13% on web, 18% on wikitext-2 than the best same-size research quant
Vision	`mmproj-BF16.gguf`	0.87 GiB	+0.9 GiB	any tier	Qwen’s own encoder at BF16

VRAM = weights + q8_0 KV + llama.cpp buffers. Only a quarter of layers use full attention, so 32k context is about 1 GiB at q8_0.

Take AP-IQ4_XS if it fits. Step down only for memory.

Measured

Fidelity = KL divergence of the next-token distribution vs Qwen3.8-27B BF16. Lower is closer to the original.

60 × 2048 tokens, three corpora, same build, same BF16 logits:

held-out — 1.85 MB of our own technical prose, never part of any calibration set. It is internal engineering documentation, so it is not published: this column is the one you cannot re-run yourself.
neutral web — mixedweb-v1, 800,789 chars, a seeded FineWeb slice, md5 51e0045e8cabf37922aa82766a25b7b4, published with its per-document manifest and builder in agentionai/quant-fidelity-corpora
wikitext-2 — the standard wiki.test.raw, 1,288,556 chars, md5 7c0137fc034ddbc56a296bce31b4f7fb (llama.cpp/scripts/get-wikitext-2.sh)

Reproducing these numbers

The reference is Qwen/Qwen3.8-27B at revision 1d4bf0f2 (the weights uploaded 2026-08-13; later commits are README only), converted with llama.cpp's own converter — no changes, no fork:

python convert_hf_to_gguf.py --outtype bf16 Qwen3.8-27B/ --outfile Qwen3.8-27B-BF16.gguf   # 54,657,733,888 bytes

Then one base of BF16 logits per corpus, and every quant scored against it. The KLD code is upstream and unmodified; ours was built at llama.cpp 26bc85e42:

llama-perplexity -m Qwen3.8-27B-BF16.gguf -f mixedweb-v1.txt \
  --kl-divergence-base base-mixedweb.bin -c 2048 --chunks 60 -ngl 99

llama-perplexity -m Qwen3.8-27B-AP-IQ4_XS.gguf --kl-divergence \
  --kl-divergence-base base-mixedweb.bin -c 2048 --chunks 60 -ngl 99

Mean KLD and Same top p in that output are the two numbers in the tables above; 99.0% KLD is the worst-1 % column. Every file on this page — ours, Unsloth's, ISTA-DASLab's, AtomicChat's — was measured with those exact commands, same build, same base files, same 60 × 2048 tokens.

Head to head with Unsloth, identical bytes

Same tensor types, same file size, same speed, same memory. What differs is how the weights inside each block were chosen — a different calibration and a different encoder.

	held-out	neutral web	wikitext-2	worst 1% tokens (held-out)
`UD-Q4_K_XL`	0.0117	0.0087	0.0122	0.088
`AP-Q4_K_XL`	0.0111 (−4.4%)	0.0083 (−4.5%)	0.0118	0.082 (−7.2%)
`UD-Q4_K_M`	0.0153	0.0108	0.0139	0.122
`AP-Q4_K_M`	0.0146 (−4.9%)	0.0104 (−3.7%)	0.0150	0.111 (−8.8%)
`UD-IQ4_XS`	0.0276	0.0186	0.0252	0.233
`AP-IQ4_XS`	0.0255 (−7.6%)	0.0181 (−2.4%)	0.0243 (−3.8%)	0.211 (−9.3%)
`UD-Q3_K_XL`	0.0421	0.0270	0.0337	0.366
`AP-Q3_K_XL`	0.0380 (−9.9%)	0.0250 (−7.6%)	0.0359	0.327 (−10.6%)
`UD-IQ3_S`	0.0617	0.0404	0.0470	0.553
`AP-IQ3_S`	0.0553 (−10.4%)	0.0371 (−8.1%)	0.0493	0.483 (−12.7%)

Held-out gains: 3.2σ, 3.2σ, 5.7σ, 8.1σ and 8.8σ (Q4_K_XL / Q4_K_M / IQ4_XS / Q3_K_XL / IQ3_S). Neutral-web gains: 3.7σ at Q4_K_XL, 5.0σ at Q3_K_XL, 5.2σ at IQ3_S; 2.4σ at Q4_K_M. Wikitext-2 is a statistical tie (under 1.5σ) at every tier: read that column as even.

Top-1 match with BF16 on held-out text: 92.8% · 92.2% · 90.4% · 89.2% · 87.9% · 86.6% · 86.9% (Q4_K_XL / Q4_K_M / IQ4_XS / Q3_K_XL / IQ3_S / IQ3_XS / IQ3_XXS).

The field near these sizes

file	size	held-out	neutral web	wikitext-2
`AP-Q4_K_XL`	16.35 GiB	0.0111	0.0083	0.0118
unsloth `UD-Q4_K_XL`	16.35 GiB	0.0117	0.0087	0.0122
`AP-Q4_K_M`	15.33 GiB	0.0146	0.0104	0.0150
unsloth `UD-Q4_K_M`	15.33 GiB	0.0153	0.0108	0.0139
AtomicChat `AD-IQ4_XS-IQ3_S`	13.45 GiB	0.0335	0.0234	0.0384
`AP-IQ4_XS`	13.27 GiB	0.0255	0.0181	0.0243
unsloth `UD-IQ4_XS`	13.27 GiB	0.0276	0.0186	0.0252
AtomicChat `AD-IQ3_S`	12.89 GiB	0.0441	0.0303	0.0441
`AP-Q3_K_XL`	12.24 GiB	0.0380	0.0250	0.0359
unsloth `UD-Q3_K_XL`	12.24 GiB	0.0421	0.0270	0.0337
ISTA-DASLab `GSQ-RCO-IQ3_S`	11.29 GiB	0.0594	0.0432	0.0665
`AP-IQ3_S`	11.21 GiB	0.0553	0.0371	0.0493
unsloth `UD-IQ3_S`	11.21 GiB	0.0617	0.0404	0.0470
`AP-IQ3_XS`	10.70 GiB	0.0714	0.0508	0.0642
AtomicChat `AD-IQ2_S`	10.38 GiB	0.1187	0.0858	0.1062
`AP-IQ3_XXS`	10.00 GiB	0.0985	0.0658	0.0801
ISTA-DASLab `GSQ-RCO-IQ3_XXS`	9.73 GiB	0.1123	0.0824	0.1063
ISTA-DASLab `GSQ-RCO-IQ2_S`	8.95 GiB	0.1541	0.1143	0.1470
`AP-IQ2_S`	8.95 GiB	0.1428	0.0996	0.1203
ISTA-DASLab `GSQ-RCO-IQ2_XS`	8.17 GiB	0.2263	0.1724	0.2036

ISTA rows are their -mtp builds (draft head included, same as ours).

KL is fidelity to Qwen’s predictions, not a task leaderboard. Downstream evals are next. Until then the claim is narrow and checkable: at every size we ship, you are closer to the original than the same-size alternative.

Running

Use the sampling settings from the Qwen3.8-27B model card. Thinking is on by default. To turn it off per request, send "chat_template_kwargs": {"enable_thinking": false}.

llama-server -hf agentionai/Qwen3.8-27B-AP-GGUF:IQ4_XS \
  --jinja -ngl 999 -fa on -c 32768 -ctk q8_0 -ctv q8_0

Keep the KV cache at q8_0 or f16. A 4-bit value cache makes long reasoning traces degenerate into repetition on this model family.

LM Studio: search for agentionai/Qwen3.8-27B-AP-GGUF and pick a tier. Ollama: ollama run hf.co/agentionai/Qwen3.8-27B-AP-GGUF:IQ4_XS

llama-server -hf agentionai/Qwen3.8-27B-AP-GGUF:IQ4_XS \
  --mmproj mmproj-BF16.gguf --jinja -ngl 999 -fa on -c 32768 -ctk q8_0 -ctv q8_0

Built with

The tiers are built and verified with our own Rust tooling agention-infer. Every tier is measured against BF16 on all three corpora, and shipped only if it beats the same-size alternative. unsloth's dynamic type maps underpin AP-Q4_K_XL, AP-Q4_K_M, AP-IQ4_XS, AP-Q3_K_XL and AP-IQ3_S, and we thank them for that work. AP-IQ3_XS and AP-IQ3_XXS sit at sizes no one else ships and use our own per-tensor allocation, so they are compared against the nearest published files above and below them. AP-IQ2_S is built on ISTA-DASLab's GSQ-RCO type map for that size, re-encoded with our calibration and tooling, and we thank them for that work. imatrix-mixed-v2.gguf, the importance matrix from our calibration pass, is included for anyone building their own quants.

Calibration is ordinary public text: the Bartowski and Thireus imatrix corpora, plus a one-third share of whole articles from the wikitext-2 train split — never the test split, and the 12-gram overlap with each of the three evaluation corpora above is 0, 0 and 6 (section headings and unit conversions). 6,012 documents, 4.67 M characters, md5 b8e3269f085f500ab307b0cc977126b0, 1,200 × 512 tokens through the BF16 model. The calibration is not the advantage here; anyone can build on the same text.

More tiers follow as they clear the same bar.

Support AgentionAI

These quants are released freely. If they save you VRAM or make Qwen more useful, you can buy me a coffee or some GPU time and sponsor continued quantization and benchmarking on GitHub. AgentionAi is a one person team and can use your help.

Downloads last month: 7,877

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

2-bit

3-bit

4-bit

View +1 variant

Model tree for agentionai/Qwen3.8-27B-AP-GGUF

Base model

Qwen/Qwen3.8-27B

Quantized

(1226)

this model