kingjones777/Hemmingway-1-ROCmFP4-GGUF

Hemmingway-1 — ROCmFP4 for AMD Strix Halo (gfx1151)

Altworld/Hemmingway-1 quantized to ROCmFP4 for AMD Strix Halo, with its multi-token-prediction head carried through into the GGUF so speculative decoding works out of the box.

Everything below was measured on the shipped files on one Ryzen AI Max+ 395 (Radeon 8060S, 128 GiB unified), on an otherwise idle box. Numbers come from the built GGUFs and from server logs, never from estimates.

Which file should I use?

File	Bytes	ftype	decode (MTP n-max 2)	decode (no spec)
…-Q4_0_ROCMFP4_STRIX_LEAN.gguf ⭐	15,212,619,328	106	20.72	13.62
…-Q4_0_ROCMFP4_FAST.gguf	14,562,235,968	103	20.67	13.77
mtp-Hemmingway-1-Q4_0.gguf	1,706,486,336	2	optional draft head	—

Take STRIX_LEAN. It carries a q6_K-protected output head at 0.65 GB more than FAST, and the two are speed-equivalent — 20.72 vs 20.67 tok/s, inside run-to-run noise.

Both main files already contain the MTP block, so --spec-type draft-mtp works with no sidecar. The separate mtp-…-Q4_0.gguf draft head is bundled for runtimes that want an explicit --model-draft; it is not required.

Quick start

llama-server -m Hemmingway-1-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -dev ROCm0 -ngl 999 -fa on -dio --jinja -fit off --parallel 1 \
  -c 8192 \
  --spec-type draft-mtp --spec-draft-ngl 99 --spec-draft-device ROCm0 \
  --spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0.0

Requires a llama.cpp built with ROCmFP4 support (ggml types 100–106). Stock llama.cpp rejects these tensor types.

Speed: the MTP depth curve

Use --spec-draft-n-max 2. Measured on STRIX_LEAN, ROCm0, 8K context, 400-token generations, identical flags per arm, warm-up call before each measurement, idle box:

--spec-draft-n-max	decode tok/s	vs off	prefill tok/s	acceptance
off	13.62	—	161.07	—
1	17.33	1.27×	135.13	0.496
2	20.72	1.52×	127.98	0.495
3	17.97	1.32×	120.82	0.302
4	16.28	1.20×	118.16	0.231
6	14.78	1.09×	113.24	0.205

Classic rise / knee / decline. Acceptance holds flat through depth 2 (0.496 → 0.495) and then collapses — 0.302 at 3, 0.231 at 4, 0.205 at 6 — so past the knee the draft head spends its time generating tokens that get thrown away and the cost is still paid.

⚠️ The knee is model-specific. Other models on this architecture peak at deeper settings. Do not carry a --spec-draft-n-max over from a sibling model; measure it.

MTP costs prefill. 161.07 → 127.98 tok/s (−21%) at the recommended depth, because the draft head runs over the prompt as well. The trade favours chat and long generations over short-prompt batch work.

Reasoning controls

⛔ Thinking is ON by default, and this model is verbose about it. With a small max_tokens the entire budget is consumed inside the reasoning channel and content comes back empty — 1200 tokens was still not enough in testing (5,223 characters of reasoning_content, empty content, finish_reason: length).

Either give it real room, or switch thinking off:

# returns empty content at 400 and at 1200 tokens with thinking on
curl … -d '{"messages":[…],"max_tokens":200,"chat_template_kwargs":{"enable_thinking":false}}'

With enable_thinking: false it answers normally and stops cleanly (finish_reason: stop).

Tool calling

3/3 clean calls (get_weather {"city":"Paris"}) with --jinja on both tiers, no shim, no argument repair, finish_reason: tool_calls.

Memory

STRIX_LEAN occupies ~18 GiB of GTT at -c 8192 including the MTP draft cache. Comfortable alongside other models on a 128 GiB Strix Halo.

⚠️ The MTP path allocates its own KV cache, so a large declared context costs roughly double. The files declare 262144; size -c to what you actually need.

Quantization methodology

# 1. convert — the MTP block is folded in automatically as block 64
python convert_hf_to_gguf.py Hemmingway-1/ --outtype bf16 --outfile Hemmingway-1-BF16.gguf

# 2. quantize from BF16 only
llama-quantize --output-tensor-type q6_K  Hemmingway-1-BF16.gguf OUT Q4_0_ROCMFP4_STRIX_LEAN 16
llama-quantize                            Hemmingway-1-BF16.gguf OUT Q4_0_ROCMFP4_FAST       16

# 3. optional standalone draft head: extract blk.64.* + token_embd + output + output_norm,
#    then quantize with BOTH heads forced to q4_0 (the default protects output.weight at q6_K,
#    which makes the draft step's 248k-vocab projection needlessly expensive)
llama-quantize --output-tensor-type q4_0 --token-embedding-type q4_0 \
    mtp-Hemmingway-1-BF16.gguf mtp-Hemmingway-1-Q4_0.gguf Q4_0 16

Receipts read back from the built files by exact tensor name, never a substring match:

File	ftype	output.weight	token_embd.weight	tensors	blocks	blk.64.*
…-STRIX_LEAN.gguf	106	Q6_K	Q5_K	866	65	15
…-FAST.gguf	103	Q4_0_ROCMFP4_FAST	Q4_0_ROCMFP4_FAST	866	65	15
mtp-…-Q4_0.gguf	2	Q4_0	Q4_0	18	65	15

The MTP block survives quantization intact on every tier — 15 of 15 blk.64.* tensors present, with blk.64.nextn.eh_proj kept at Q8_0 and all norms at F32.

Model facts (read from the GGUF headers)

general.architecture qwen35 · context 262144 · n_embd 5120 · 65 blocks (64 + 1 MTP) · 24 attention heads / 4 KV heads · head_dim 256 · partial_rotary_factor 0.25 · vocab 248320 · nextn_predict_layers 1 · hybrid linear/full attention, full-attention interval 4 · attn_output_gate true, gate type swish.

Reproduction

Built with a ROCmFP4-capable llama.cpp:

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON \
  -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build -j32 --target llama-quantize llama-server

Speed runs used the environment prefix below; without it a box that also has a Vulkan llama.cpp build can hit a soname clash:

env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 llama-server …

Files

File	Bytes	sha256
Hemmingway-1-Q4_0_ROCMFP4_STRIX_LEAN.gguf	15,212,619,328	a2ffb86c0872c2bbb13ef7c0d8374dd3f777216451f254ff235ee7a04c1b31bf
Hemmingway-1-Q4_0_ROCMFP4_FAST.gguf	14,562,235,968	0a0e28791b148107dba7511fa942b6aaf95e68c8c81a36ad439326b3cf45b07a
mtp-Hemmingway-1-Q4_0.gguf	1,706,486,336	b1b67bcfc2cd0d261b92451d1b803e06c2de4098b9139813f3bf501e1c4c5792

Known issues and limits

• Thinking on by default → empty content at small max_tokens; 1200 was still not enough.
• --spec-draft-n-max above 2 makes decode slower, not faster, on this model.
• MTP costs ~21% of prefill at the recommended depth.
• No perplexity is published. It was not measured, and a number taken under a different protocol would be worse than none.
• The two tiers are speed-equivalent; pick STRIX_LEAN for the protected head.

License and attribution

Apache-2.0, inherited from Altworld/Hemmingway-1. Quantized and measured on AMD Strix Halo (gfx1151). All credit for the model itself to Altworld.