kingjones777/Step-3.7-Flash-Q4_0_ROCMFP4_STRIX_LEAN-GGUF

Step-3.7-Flash — Q4_0_ROCMFP4_STRIX_LEAN GGUF

This is a ROCmFP4 "Strix Lean" quant of stepfun-ai/Step-3.7-Flash, built and tested on a Ryzen AI Max+ 395 (Strix Halo, gfx1151, 128 GB unified memory, ROCm 7.2.4).

I run a lot of big models on Strix Halo hardware, and the ROCmFP4 format has been the best decode-speed lever I've found for it. Every ROCmFP4 build I could find was Qwen, Ornith, Laguna or Tess class, so I made one for Step-3.7-Flash. As far as I can tell this is the first one.

Short version: it's faster than IQ4_XS at both ends — about 18–20% more decode throughput and 8–12% more prefill — with equivalent output quality. It costs about 10% more disk.

Requires the ROCmFP4 fork

Q4_0_ROCMFP4_STRIX_LEAN is not a stock llama.cpp quant type. These files will not load on upstream llama.cpp, Ollama, LM Studio, or anything else that hasn't got the ROCmFP4 kernels. You need a build of the fork:

git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
HIPCXX=$(hipconfig -l)/clang HIP_PATH=$(hipconfig -R) cmake -B build \
  -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_HIP_ROCWMMA_FATTN=ON \
  -DGGML_HIP_NO_VMM=ON -DGGML_HIP_MMQ_MFMA=ON -DCMAKE_BUILD_TYPE=Release \
  -DGGML_VULKAN=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build -j 4

Keep the job count low. On a 128 GB unified-memory box the HIP flash-attention kernels spike RAM hard at -j 8 or higher and will OOM the machine.

Files

File	Size
Step-3.7-Flash-Q4_0_ROCMFP4_STRIX_LEAN-00001-of-00003.gguf	41.7 GiB
Step-3.7-Flash-Q4_0_ROCMFP4_STRIX_LEAN-00002-of-00003.gguf	41.5 GiB
Step-3.7-Flash-Q4_0_ROCMFP4_STRIX_LEAN-00003-of-00003.gguf	14.6 GiB
mmproj-Step-3.7-Flash-f16.gguf	3.7 GiB

97.7 GiB total for the model, 4.26 BPW. Point llama.cpp at the first shard and it picks up the rest automatically. The mmproj file is the original f16 vision projector from StepFun — you only need it if you want image input.

Running it

env LD_LIBRARY_PATH=/path/to/ROCmFPX/build/bin:/opt/rocm/lib \
    HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
  /path/to/ROCmFPX/build/bin/llama-server \
  --host 127.0.0.1 --port 8080 \
  --model Step-3.7-Flash-Q4_0_ROCMFP4_STRIX_LEAN-00001-of-00003.gguf \
  --mmproj mmproj-Step-3.7-Flash-f16.gguf \
  --n-gpu-layers 999 --flash-attn on -dio --no-warmup --jinja \
  --ctx-size 32768 --cache-type-k q8_0 --cache-type-v q8_0 \
  --parallel 1 --temp 0.6 --top-p 0.95 --reasoning-budget 512

Three things in there are not optional, and each one cost me time before I worked it out:

-dio. Without direct I/O the mmap path crawls up to around 120 GB RSS and sits there looking hung. It isn't hung and the file isn't corrupt, it's just the loader. With -dio it cold-loads in about 60 seconds.

The env block. HSA_OVERRIDE_GFX_VERSION=11.5.1 and unified memory are needed for gfx1151, and putting the fork's build/bin first on LD_LIBRARY_PATH avoids a soname clash if you also have a Vulkan llama.cpp build on the box.

--reasoning-budget 512. Step-3.7-Flash thinks a lot. Left alone it will spend your entire max_tokens on chain-of-thought and hand you back an empty content field with finish_reason: length, which looks exactly like a broken quant. It isn't. Cap the thinking and the answer lands. 512 works well for chat; raise it for hard reasoning work.

Benchmarks

Both quants cold-loaded one at a time on the same machine, same binary, same environment, same sampling. Every run generated exactly 256 tokens so decode numbers are actually comparable, and every prompt started with a unique nonce so nothing was served out of the prefix cache. Three runs each, medians below.

	This quant	UD-IQ4_XS	Difference
Prefill @ 8K	331 tok/s	296 tok/s	+12%
Decode @ 8K	18.8 tok/s	15.9 tok/s	+18%
Prefill @ 32K	283 tok/s	261 tok/s	+8%
Decode @ 32K	14.9 tok/s	12.4 tok/s	+20%
Size	97.7 GiB	88.8 GiB	+10%

Compared against unsloth/Step-3.7-Flash-GGUF UD-IQ4_XS, tested at 32K context with q8_0 KV cache.

Be fair about the size column. This is a ~4.26 BPW quant and UD-IQ4_XS is roughly 3.9 BPW, so it's a higher-bit quant winning on speed while costing more disk. If you're comparing against Q4_K_M instead, ROCmFP4 normally comes out around 12–18% smaller. Against a sub-4-bit IQ quant it does not, and I'd rather say that up front than let the table imply a free lunch.

Quality spot checks came out equivalent, not identical — same correct answers on a palindrome-function task and a reasoning riddle, with different wording. Worth noting that IQ4_XS was noticeably more verbose in its chain-of-thought and needed a higher token budget before its final answer appeared, which is a practical point in this quant's favour if you're serving it.

How it was made

llama-quantize --allow-requantize \
  Step-3.7-Flash-Q8_0-00001-of-000NN.gguf \
  Step-3.7-Flash-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  Q4_0_ROCMFP4_STRIX_LEAN 8

Quantized from the official Q8_0 GGUF, not from BF16. The BF16 weights are 394 GB and there was no way to hold them alongside the output on this machine, so this is a requantization and --allow-requantize was required. I'd rather disclose that than have someone assume an F16 source. It measured and benched clean, but if you want a from-BF16 build, this isn't it.

Verified after quantizing: coherent output on a load proof before anything got deleted, the full benchmark run above, and image input confirmed working through the mmproj.

Model

Step-3.7-Flash is StepFun's 198B MoE — 196B language model plus a 1.8B vision encoder, roughly 11B active parameters, 256K native context. Everything about the model's behaviour, prompt format and capabilities comes from upstream; see the base model card. All I've changed is the numeric format.

Apache 2.0, same as the base model.

Credit

The ROCmFP4 format and kernels are not my work — that's the ROCmFPX fork. I built the quant, ran the benchmarks and wrote them down.