File size: 2,952 Bytes
b69d9d8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | From: opencoti
Subject: [PATCH 0086] F16 MoE prefill — route large-batch f16 mul_mat_id to MMF (#555)
ggml_cuda_mul_mat_id had no fast CUDA path for f16/bf16/f32 experts at prefill batch:
MMQ is quantized-only, and ggml_cuda_should_use_mmf rejected mul_mat_id whenever
src1_ncols (= n_tokens) exceeded 512 (or 128 when the expert dim > 1024). At prefill
(~32k tokens) that gate always failed, so f16 MoE fell into the host-side sorted
fallback in ggml_cuda_mul_mat_id: a per-call cudaStreamSynchronize (which also disables
CUDA graphs) + an O(n_expert*n_tokens*n_used) host sort + a per-expert cuBLAS GEMM loop.
Quantized MoE dodges all of it via the single fused MMQ-id kernel. Net: f16 128e prefill
ran ~12x slower than Q6_K (690 vs 8857 tok/s on Gemma-4-A4B 128e).
The MMF kernel's `ncols_dst > 16` branch (ggml_cuda_mul_mat_f, this file) already performs
GPU-side expert compaction (ggml_cuda_launch_mm_ids_helper, mmid.cuh) entirely on-stream
and handles arbitrary n_tokens -- the gate was a conservative tuning heuristic, not a
capability limit. This compiles out that mul_mat_id gate so large-batch f16/bf16/f32 MoE
rides the fast GPU-compacted MMF path. Measured on a real 128e-F16 (bs2 RTX PRO 6000):
prefill 690 -> 3081 tok/s (4.47x); RULER-vt@32k retrieval 100.0 (== Q6_K, correctness-clean).
The residual gap to Q6_K is just the legitimate f16-vs-q6 weight-bandwidth ratio (~2.2x).
Quantized MoE (MMQ path) and dense f16 mul_mat are untouched -- both are rejected/handled
before this gate. The upstream gate is preserved, compiled out, behind
-DOPENCOTI_MMF_LEGACY_GATE.
diff --git a/llama.cpp/ggml/src/ggml-cuda/mmf.cu b/llama.cpp/ggml/src/ggml-cuda/mmf.cu
--- a/llama.cpp/ggml/src/ggml-cuda/mmf.cu
+++ b/llama.cpp/ggml/src/ggml-cuda/mmf.cu
@@ -160,11 +160,22 @@
}
if (mul_mat_id) {
+ // opencoti-hook: f16-moe-prefill (#555) — the upstream heuristic below rejected
+ // large-batch (prefill) f16/bf16/f32 mul_mat_id, sending it to the host-sync sorted
+ // fallback in ggml_cuda_mul_mat_id (per-call cudaStreamSynchronize + per-expert cuBLAS
+ // loop + disabled CUDA graphs => measured 12x slower than the quantized MMQ-id path:
+ // 690 vs 8857 tok/s prefill on Gemma-4-A4B 128e). The MMF kernel's `ncols_dst > 16`
+ // branch already performs GPU-side expert compaction (ggml_cuda_launch_mm_ids_helper,
+ // mmid.cuh) entirely on-stream and handles arbitrary n_tokens, so keep f16 MoE on the
+ // fast fused MMF path for any batch size. Upstream gate preserved (disabled) for ref;
+ // re-enable with -DOPENCOTI_MMF_LEGACY_GATE.
+#ifdef OPENCOTI_MMF_LEGACY_GATE
if (src0_ne[1] <= 1024 && src1_ncols > 512) {
return false;
} else if(src0_ne[1] > 1024 && src1_ncols > 128) {
return false;
}
+#endif
} else {
if (GGML_CUDA_CC_IS_RDNA3_0(cc) && src1_ncols > 8) {
return false;
|