--- base_model: bottlecapai/ThinkingCap-Qwen3.6-27B license: apache-2.0 library_name: gguf tags: - gguf - rocmfp4 - qwen3.6 - mtp - speculative-decoding - efficient-thinking - strix-halo - amd - rocm - vulkan base_model_relation: quantized ---

PLUNDERSTRUCK // ROCmFP4 QUANTIZED MODEL // STRIX HALO · gfx1151
        ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
      ▟███████████████████████▙
   ▄▟█████████████████████████████▙▄
 ▟███████████████████████████████████▙
▐███████████████▛▀▀▀▀▀▜███████████████▌
▝████████████████       ████████████▛▘
       ▐████▛▘             ▝▜████▌
         ▐█▌       ▟▙        ▐█▌
         ▐█▌      ▟███▙       ▐█▌
          ▀▀     ▟█████▙      ▀▀
                ▟███████▙
THINKINGCAP-QWEN3.6-27B-MTP
4-BIT ROCmFP4 · EFFICIENT-THINKING · MTP SELF-SPECULATIVE DECODE · VISION-CAPABLE · SINGLE AMD APU

    
      FORMAT
ROCmFP4 4-BIT

      PRECISION
4.94 BPW

      SIZE
16.9 GB

      CONTEXT
262 K

    

      DRAFT
MTP n-max 5

      VISION
QWEN3-VL

      BACKEND
VULKAN0

      CALIBRATION
non-imatrix

    

ThinkingCap-Qwen3.6-27B is bottlecapai's efficient-reasoning finetune of Qwen3.6-27B — trained to cut thinking-token consumption roughly in half (↓45.8% macro average) while holding answer quality, across knowledge, math, code, long-context, and agentic tasks. It scores GPQA-Diamond 83.8, MMLU-Pro 85.4, GSM8K 96.5 (bottlecapai reported). This repo is the ROCmFP4 quantization tuned for a single AMD Strix Halo APU, with the MTP head preserved for self-speculative decode and vision carried through from the base model.

⚠ REQUIRES THE ROCmFPX FORK

The custom q4_0_rocmfp4 / q4_0_rocmfp4_fast tensor types will not load in stock llama.cpp, LM Studio, Ollama, Jan, or koboldcpp. Build/run with charlie12345/ROCmFPX · branch experimental-rocmfpx-branch:

git clone https://github.com/charlie12345/ROCmFPX

cd ROCmFPX && git checkout experimental-rocmfpx-branch

env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh

NOTE // Ignore HuggingFace's auto-detected "F16" badge — its parser can't read ROCmFP4 and mislabels by the genuinely-f16 token embeddings. This is a ~4.9 bpw 4-bit file; pick by filename in Files and versions.

01 · FILES

File	Size	Output head	Pick if
`…-STRIX-embF16-headQ6.gguf` ★	16.9 GB	Q6_K	the one build — best speed/quality balance: f16 embeddings + Q6 output head on the fast single-scale body

One file — the **best speed/quality balance** in ROCmFP4 for Strix Halo. It keeps the two quality levers that are actually *felt* — genuine **f16 token embeddings** (from F16 source) and a **Q6_K output head** — on the fast single-scale `q4_0_rocmfp4_fast` body, plus the **MTP head**, with **no imatrix** (this recipe's daily-driver default; see §05). Repo bundles the **[`mmproj-F32.gguf`](./mmproj-F32.gguf)** Qwen3-VL vision projector and **[`chat_template.jinja`](./chat_template.jinja)** (froggeric's unified Qwen3.6 template — tool calls + inline think-toggle + vision).

token_embd	F16 (full precision — a lookup, ~zero decode cost)
attention K/V (+ fused QKV)	`q4_0_rocmfp4` (dual-scale)
FFN, lm-head, rest	`q4_0_rocmfp4_fast` (single-scale)
MTP head	preserved (`nextn_predict_layers=1`)

02 · QUICK START

Run from the folder holding the `.gguf` + `chat_template.jinja`: ```bash env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \ llama-server \ -m ThinkingCap-Qwen3.6-27B-ROCmFP4-STRIX-embF16-headQ6.gguf \ --alias thinkingcap-27b \ --host 0.0.0.0 \ --port 8080 \ -dev Vulkan0 \ -ngl 999 \ -fa on \ -c 262144 \ -b 2048 \ -ub 256 \ -t 16 \ -tb 16 \ -ctk f16 \ -ctv f16 \ -cpent 256 \ -ctxcp 32 \ --cache-reuse 256 \ --cache-ram 65536 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --spec-type draft-mtp \ --spec-draft-device Vulkan0 \ --spec-draft-ngl all \ --spec-draft-type-k f16 \ --spec-draft-type-v f16 \ --spec-draft-n-max 5 \ --spec-draft-n-min 2 \ --spec-draft-p-min 0.0 \ --spec-draft-p-split 0.10 \ --chat-template-file chat_template.jinja \ --reasoning on \ --reasoning-format deepseek \ --chat-template-kwargs '{"preserve_thinking": true}' \ --jinja \ --parallel 1 \ --metrics \ --no-mmap \ --mmproj mmproj-F32.gguf \ --image-min-tokens 1024 ``` The **last two lines enable vision** — the `mmproj-F32.gguf` Qwen3-VL projector is **bundled in this repo**; omit them for text-only. **`--image-min-tokens 1024` is required whenever `--mmproj` is set.**

NOTE // bottlecapai's own recommended sampling for general use is temp=1.0, top_p=0.95, top_k=20, min_p=0.0. We serve at temp 0.6 (Qwen3.6 "precise coding" preset) by default — raise to 1.0 for open-ended/creative tasks.

Flag	Function
`HSA_OVERRIDE_GFX_VERSION=11.5.1`	treat the APU as gfx1151 (Strix Halo)
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`	allow use of the full 128 GB unified memory
`-dev Vulkan0`	run on Vulkan (KHR_coopmat) — beats ROCm/HIP here for ROCmFP4 on Strix Halo
`-ngl 999 · -fa on`	offload all layers · flash attention
`-c 262144`	context length (256K)
`-b 2048 · -ub 256 · -t/-tb 16`	prefill batch / micro-batch (256 = prefill optimum) · CPU threads
`-ctk f16 · -ctv f16`	f16 KV cache — how we run it; drop to `q8_0`/`q4_0` to use less memory
`-cpent · -ctxcp · --cache-reuse · --cache-ram 65536`	cross-turn KV checkpointing + 64 GB resident reuse cache
`--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.0`	precise-coding sampling (bottlecapai recommends 1.0 for general use)
`--spec-type draft-mtp · --spec-draft-n-max 5 · n-min 2`	built-in MTP head, self-speculative; draft depth up to 5, at least 2
`--spec-draft-device Vulkan0 · -ngl all · type-k/v f16`	draft head on Vulkan, fully offloaded, f16 draft KV
`--chat-template-file chat_template.jinja`	bundled froggeric template (tool calls + think-toggle + vision)
`--reasoning on --reasoning-format deepseek + kwargs {preserve_thinking:true}`	clean `content`+`reasoning_content`; keep `<think>` across turns so cross-turn cache survives
`--jinja --parallel 1 --metrics --no-mmap`	apply template · single slot · metrics · weights in RAM

**OpenAI-compatible client (e.g. OpenCode).** In single-model mode llama-server ignores the request's `model` field, so the client's model name is just a label. - Base URL: `http://:8080/v1` · API key: any non-empty string (e.g. `sk-local`) - Model id this server reports: `thinkingcap-27b` A patched OpenCode that compacts conversation history **without** invalidating the prompt cache is at [`PlunderStruck/opencode`](https://github.com/PlunderStruck/opencode) — pair it with the checkpoint flags to keep long sessions fast.

03 · VISION

Qwen3-VL lineage — vision works via the **bundled `mmproj-F32.gguf`** projector at launch with `--mmproj` (no different LLM GGUF needed). ```bash # add to your llama-server launch: --mmproj mmproj-F32.gguf \ --image-min-tokens 1024 # REQUIRED — Qwen-VL needs >=1024 image tokens or it misreads fine detail ```

NOTE // this is a thinking model — for one-shot image Q&A, use the bundled template's think-toggle or allow enough tokens to finish <think>, else the visible answer can come back empty. With --mmproj loaded the server disables the --cache-reuse feature (multimodal caching isn't supported).

04 · PERFORMANCE & QUALITY

**This is the best speed/quality balance in ROCmFP4 — by design, not the absolute fastest.** It keeps the two quality levers that are actually *felt* — genuine **f16 token embeddings** and a **Q6_K output head** — on the fast single-scale body, with **no imatrix** calibration (this recipe's default for the dense Qwen3.6-27B line — see the [27B card](https://huggingface.co/plunderstruck/Qwen3.6-27B-MTP-ROCmFP4-GGUF) for the full lever sweep and rationale).

WANT MAXIMUM FIDELITY INSTEAD OF SPEED? bottlecapai's own GGUF repo ships standard K-quants — Q4_K_M / Q8_0 — more faithful to the source at a slower decode. We optimize for throughput in ROCmFP4 on Strix Halo; grab one of those for the last bit of fidelity.

Hands-on, on a Framework Desktop / AMD Ryzen AI Max+ 395 (gfx1151, 128 GB unified):

DECODE	~30 t/s (Vulkan / Strix Halo, thinking on)
MTP DRAFT ACCEPTANCE	~4.15 mean accepted length (n-max 5 / n-min 2)
CONTEXT @ LOAD	full 262144, f16 KV
QUANTIZATION	non-imatrix

NOTE // built on a fork snapshot carrying an upstream fix for a self-speculative-decode bug affecting M-RoPE architectures (Qwen3.6 included) — MTP was silently degrading to plain decode under certain batch shapes on some builds. This quant's measured acceptance (~4.15) reflects the fixed path; older ROCmFPX builds may show materially lower MTP throughput on this model family.

05 · BUILD (REPRODUCIBLE)

Build the fork: ```bash git clone https://github.com/charlie12345/ROCmFPX cd ROCmFPX && git checkout experimental-rocmfpx-branch env JOBS=16 scripts/build-strix-rocmfp4-mtp.sh ``` Quantize from the bottlecapai F16 GGUF — ROCmFP4 body, genuine f16 embeddings, Q6_K head, no imatrix: ```bash # the one build: STRIX preset + f16 embeddings + Q6_K output head llama-quantize \ --token-embedding-type f16 \ --output-tensor-type q6_K \ ThinkingCap-Qwen3.6-27B-f16.gguf \ ThinkingCap-Qwen3.6-27B-ROCmFP4-STRIX-embF16-headQ6.gguf \ Q4_0_ROCMFP4_STRIX ``` **Architecture (`qwen35`):** 64 blocks, 5120 hidden, dense (not MoE), **`nextn_predict_layers=1`** MTP head — self-speculative **draft-MTP** survives quantization. **Format:** ROCmFP4 is a 4-bit weight format for AMD using an FP4-derived value codebook plus one (FAST) or two (dual) UE4M3/FP8 scale bytes per 32-weight block; tensor-aware. **This build** (`STRIX-embF16-headQ6`): quality-biased STRIX preset + f16 token embeddings (full precision; a lookup, so ~zero decode cost) + a Q6_K output head. Attention K/V (+ fused QKV) run `q4_0_rocmfp4` (dual-scale); FFN/rest run `q4_0_rocmfp4_fast` (single-scale). > Experimental research build for AMD Strix Halo — hardware-, driver-, model-, and prompt-sensitive, may not reproduce on other GPUs. **Not** native FP4 tensor-core execution. Do not treat these numbers as upstream llama.cpp claims.

06 · LINEAGE & CREDITS

BASE MODEL	bottlecapai/ThinkingCap-Qwen3.6-27B — efficient-thinking finetune of Qwen/Qwen3.6-27B (Apache 2.0). This is a derivative quantization that inherits the Apache 2.0 license.
F16 GGUF SOURCE	bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF · `ThinkingCap-Qwen3.6-27B-f16.gguf`
FORMAT + RUNTIME	charlie12345/ROCmFPX (based on llama.cpp, MIT)
CHAT TEMPLATE	froggeric/Qwen-Fixed-Chat-Templates
BENCHMARKS (BASE)	GPQA-Diamond 83.8±1.9 · MMLU-Pro 85.4±0.2 · GSM8K 96.5±0.3 · thinking-token reduction ↓45.8% macro avg (bottlecapai reported)

*Derivative quantization — Apache 2.0, same as the base model.*

FORMAT ROCmFP4 4-BIT	PRECISION 4.94 BPW	SIZE 16.9 GB	CONTEXT 262 K
DRAFT MTP n-max 5	VISION QWEN3-VL	BACKEND VULKAN0	CALIBRATION non-imatrix