Instructions to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="artichoke42/Qwen3.6-27B-KR-MTP-GGUF",
	filename="Qwen3.6-27B-KR-MTP-UD-Q5_K_XL.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL
# Run inference directly in the terminal:
llama cli -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL
# Run inference directly in the terminal:
llama cli -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL
# Run inference directly in the terminal:
./llama-cli -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Use Docker

docker model run hf.co/artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

LM Studio
Jan

vLLM

How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "artichoke42/Qwen3.6-27B-KR-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "artichoke42/Qwen3.6-27B-KR-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Ollama
How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with Ollama:
```
ollama run hf.co/artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL
```

Unsloth Studio

How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for artichoke42/Qwen3.6-27B-KR-MTP-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for artichoke42/Qwen3.6-27B-KR-MTP-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for artichoke42/Qwen3.6-27B-KR-MTP-GGUF to start chatting

How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL
```

Lemonade

How to use artichoke42/Qwen3.6-27B-KR-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull artichoke42/Qwen3.6-27B-KR-MTP-GGUF:UD-Q5_K_XL

Run and chat with the model

lemonade run user.Qwen3.6-27B-KR-MTP-GGUF-UD-Q5_K_XL

List all available models

lemonade list

Qwen3.6-27B-KR-MTP-GGUF

A bilingual (English + Korean) agentic fine-tune of Qwen3.6-27B, distilled from Qwen3.5-122B-A10B, with dedicated transliteration and calibration training, and then RLVR-tuned for language purity. Built for agentic workloads — coding, tool-use / web-search, bilingual chat with reasoning, and paper summarization — and packaged as imatrix-quantized GGUF for single-GPU deployment (fits a 32 GB card such as an RTX 5090).

This is the MTP repo: the multi-token-prediction (MTP) variant + the full-precision bf16 original. The non-MTP GGUF lives in the companion repo Qwen3.6-27B-KR-GGUF.

Note on naming: the UD-Q5_K_XL tag denotes a stock llama.cpp recipe — Q5_K_M base with Q6_K token-embedding/output tensors and an imatrix — not Unsloth's published dynamic UD-Q5_K_XL. The difference is under ~1%.

Files — which one to download

The two variants are the same model, differing only by the MTP head (capability/benchmarks identical).

File	Size	Use case
`Qwen3.6-27B-KR-MTP-UD-Q5_K_XL.gguf`	~19.7 GB	Recommended default (this repo). Carries the MTP head for faster single-stream decoding via speculative decoding. Best for solo / small-team agentic use. Also serves fine without spec-decode.
`Qwen3.6-27B-KR-UD-Q5_K_XL.gguf`	~19.4 GB	Non-MTP (companion repo). Simpler; fine for high-concurrency batched serving where spec-decode helps little.

Also in this repo: the bf16 full-precision original (*.safetensors) — the model these GGUFs are quantized from, for re-quantization or further training.

Model summary

Base: Qwen/Qwen3.6-27B — 27B dense, hybrid linear-attention/Mamba architecture with a built-in MTP head, Apache 2.0.
Teacher: Qwen/Qwen3.5-122B-A10B — generated the distillation corpus, Apache 2.0.
Languages: English and Korean.
Quantization: imatrix Q5_K_M + Q6_K embed/output.

Benchmarks

Measured on the quantized Q5 GGUF (the deployment artifact), with deploy-faithful sampling (temp 1.0, no repeat-penalty):

Coding: HumanEval 79.9% (deploy-cap mt=4096; the model reasons verbosely, so budget matters).
Math (KO): HRM8K (official HAERAE-HUB Korean GSM8K, n=300) 93.0%.
Transliteration (EN↔KO proper nouns): 79% on a frozen 420-item held-out set — a dedicated training focus.
Calibration: fabrication 20.8% / appropriate-hedge 79.2% — on genuinely-unknowable questions the model hedges ("미상") instead of confabulating.
Tool-use: BFCL simple 94.0 / live-simple 84.1 / relevance 93.75; full opencode agentic round-trip live-verified.
CJK purity: Korean-answer CJK leak ~2% (temp 1.0) — RLVR shifts the base model's residual Chinese/kana bleed out of Korean answers at the distribution level.
MTP speculative decoding: 2.47 mean accepted tokens/step, ~117 tok/s single-stream — quality-equivalent to non-spec.

Serving

The validated deployment path is llama-server with --jinja, which parses the Hermes tool-call format (including the model's <think> prefix) into OpenAI-format tool_calls. The hybrid linear-attention architecture makes KV cache very cheap (~44 KB/token at q8_0), so a single 256k-token context fits on a 32 GB card. Primary example (MTP variant):

llama-server -m Qwen3.6-27B-KR-MTP-UD-Q5_K_XL.gguf \
  -ngl 99 -c 262144 --parallel 1 -fa on --jinja \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --n-predict 4096 \
  --host 127.0.0.1 --port 8080

-c 262144 --parallel 1 = the full 256k context in a single stream. For concurrency, raise --parallel: -c 262144 --parallel 4 gives 64k tokens/slot across 4 slots.
--spec-type draft-mtp --spec-draft-n-max 2 enables MTP speculative decoding (2.47 mean accepted tokens/step, ~117 tok/s single-stream; quality-equivalent to non-spec).
--cache-type-k/v q8_0 halves KV memory at no measurable quality cost (requires -fa on).
Do not set --repeat-penalty: at temp 1.0 the model does not loop, and a repeat-penalty roughly doubles fabrication by suppressing Korean hedge phrases ("미상") — so leave it off. --n-predict 4096 caps output length (also cap your agent's per-request max_tokens ≈ 4096). Avoid the DRY sampler — it corrupts verbatim strings (hashes / paths / IDs).

Recommended system prompt — identity + Korean output

Use this exact system prompt. It is also baked into the GGUF as the default system message:

당신은 Qwen3.6-27B-KR입니다. 한국어 사용자를 돕는 이중언어 AI 어시스턴트입니다. 항상 한국어로 답하며, 자신을 소개할 때 通义千问이나 Qwen 같은 중국어 이름이 아니라 Qwen3.6-27B-KR로 소개합니다. 한국어로 사고하세요

It does two things:

Identity: the model introduces itself as Qwen3.6-27B-KR — the Korean-specialized version of Qwen3.6 — instead of the base model's Chinese name 通义千问 / "Qwen". In live opencode testing this drove self-identification as 通义千问 from 42.9% → 0%.
Korean output + reasoning: it answers in Korean and moves <think> reasoning into Korean for Korean prompts, with no loss of correctness.

This is embedded in the GGUF chat_template as the default, so it applies automatically when the client sends no system message. But a client-supplied system message overrides it, and most agentic frameworks (e.g. opencode) send their own. So for agentic use, include this anchor in your client's system prompt (prepend it to your own instructions) — this is the fix that stops the model self-identifying as 通义千问. For English-only use you may omit it.

Known characteristics

Verbose reasoning: long <think> traces (inherited from the base) — a latency/length trait, not a capability gap. Give it a generous token budget.
Vision: the base is multimodal, but these are text-only GGUFs (no vision encoder/mmproj).

License & attribution

Released under Apache 2.0, matching the base and teacher.

Base model: Qwen/Qwen3.6-27B (Apache 2.0)
Teacher model: Qwen/Qwen3.5-122B-A10B (Apache 2.0)

The training corpus used prompts, grounding text, and tests from third-party datasets (never their answers — all answers are teacher-generated). The corpus itself is not distributed; only the model is. Sources, with thanks:

Coding: BAAI/TACO (Apache 2.0), deepmind/code_contests (Apache 2.0), codeparrot/apps (MIT), google-research-datasets/mbpp (CC-BY-4.0)
Tool-use / grounding: Wikipedia (CC-BY-SA-4.0), Wikidata (CC0 + CC-BY-SA), Python library documentation (BSD/MIT/Apache/PSF)
Chat prompts: allenai/tulu-3-sft-mixture (ODC-BY) — prompts only
Paper summarization: arXiv paper text (per-paper licenses; used as grounding, summarized transformatively — full text not reproduced)

Share-alike terms (e.g. Wikipedia CC-BY-SA) govern derivative works of the text; they do not attach copyleft obligations to transformatively-trained model weights. RLVR (language-purity) tuning used the same teacher/corpus lineage — no new external data.

Acknowledgements

Built on Qwen3.6-27B and Qwen3.5-122B-A10B by the Qwen team (Alibaba). Quantization and MTP speculative decoding via llama.cpp.

Downloads last month: 555

Safetensors

Model size

3.05M params

Tensor type

BF16

Model tree for artichoke42/Qwen3.6-27B-KR-MTP-GGUF

Base model

Qwen/Qwen3.6-27B

Quantized

(615)

this model