patches/0042-fused-moe.patch

opencoti F5-opt W3 (#292) — op-level fused MoE up+gate+GLU (GGML_OP_MOE_FUSED_UP_GATE)

Adds a new ggml op, GGML_OP_MOE_FUSED_UP_GATE, that collapses a MoE FFN's two
separate per-expert projections (up = up_exps @ cur, gate = gate_exps @ cur)
plus the GLU activation into ONE op at decode (n_tokens == 1). The CUDA backend
dispatches it straight to the existing fused mmvq kernel (ggml_cuda_mul_mat_vec_q
with a fusion arg carrying the gate weight + glu_op), eliminating the gate_up
intermediate's HBM round-trip and the standalone GLU kernel launch per MoE layer
per decode token.

Off by default. Engaged with --fused-moe-up-gate on (server) / cparams
.fused_moe_up_gate / adapter fusedMoeUpGate. The build_moe_ffn graph hook only
emits the op when the model presents SEPARATE up/gate expert weights
(gate_exps && !gate_up_exps), same type, SILU|GELU activation, at decode — i.e.
Qwen2-MoE / Qwen3-MoE / OLMoE / Mixtral-style layouts. Models that fuse gate+up
at the weight level (a single ffn_gate_up_exps — Gemma-4 A4B, Qwen3.5-MoE)
never match the hook and run unchanged. Flag-OFF is byte-identical to the pre-W3
binary (op never emitted); flag-ON on a matching model is byte-identical to the
unfused two-matmul + GLU path. Validated on OLMoE-1B-7B-0924-Instruct-Q4_K_M:
graph-dump engagement confirmed (34 MOE_FUSED_UP_GATE nodes on CUDA0), greedy
decode byte-identical OFF vs ON, +2.4% decode tok/s (265.9 -> 272.2).

  opencoti-hook: f5-opt-fmoe — see docs/features/advanced_kv.md

UPSTREAM PROVENANCE (for future re-sync)
  Idea source : ik_llama.cpp op-level MoE fusion
                https://github.com/ikawrakow/ik_llama.cpp
                PR #229 (fused MoE mul_mat_id path), PR #520 (OOAE expert eval).
                Commit context HEAD 8960c5ba5ee9db30ba838304373aa4dbec9f7cbd.
                License MIT, Author Iwan Kawrakow.
  Code vendored: NONE. This op is original opencoti code. ik_llama's
                ggml_cuda_moe_up_gate_unary dispatches a fused mmvq-id kernel;
                our tree already carries the equivalent primitive
                (ggml_cuda_mul_mat_vec_q with ggml_cuda_mm_fusion_args.gate +
                glu_op — ggml-cuda/mmvq.cu:555-582). W3 takes only the
                op-level-fusion IDEA and routes our existing kernel through a
                new ggml op + a build_moe_ffn graph hook. No ik_llama source is
                copied into this patch.
  ABI note    : GGML_OP_COUNT 96 -> 97. The cosmocc binary AND the ggml-cuda.so
                DSO must be rebuilt together and paired (asymmetric DSO cache
                hygiene). The CPU backend deliberately aborts the op (CUDA-only;
                CPU MoE fusion is out of scope — only the CPU embedder matters
                for CPU speed, and that is the separate W2 iqk FA engine).
  Op convention (locked from ggml-cuda/mmvq.cu):
    out = up_proj(src0=up_exps @ src1=cur) (.) act(gate_proj=fusion.gate @ cur)
        == ggml_geglu_split(gate, up); the kernel reads the GLU variant from
    fusion.glu_op (NOT dst op_params). New op srcs: src0=up_exps, src1=gate_exps,
    src2=cur, src3=ids; op_params[0]=glu_op. CUDA dispatch passes src0=up_exps,
    src1=cur(=src2), ids=src3, fusion.gate=gate_exps(=src1), glu_op=op_params[0].

Files touched (all surgical; no new vendored files):
  ggml/include/ggml.h            enum GGML_OP_MOE_FUSED_UP_GATE + ggml_moe_up_gate decl
  ggml/src/ggml.c                NAME/SYMBOL entries, COUNT 96->97 asserts, creation fn
  ggml/src/ggml-cpu/ggml-cpu.c   forward-dispatch + n_tasks: CUDA-only ABORT stub
  ggml/src/ggml-cuda/ggml-cuda.cu eval dispatch (reuse fused mmvq) + supports_op gate
  src/llama-cparams.h            bool fused_moe_up_gate
  include/llama.h                bool fused_moe_up_gate (llama_context_params)
  src/llama-context.cpp          param copy + default-params initializer
  common/common.h                bool fused_moe_up_gate = false
  common/common.cpp              cparams copy
  common/arg.cpp                 --fused-moe-up-gate on|off (LLAMA_ARG_FUSED_MOE_UP_GATE)
  src/llama-graph.cpp            build_moe_ffn hook: emit op for separate-up/gate decode

---
opencoti note (2026-05-31, bug-250): hunks regenerated via snapshot-diff against the post-0040 chain so the patch strict git-applies (3 common/* hunks had whitespace context drift). Content unchanged.
diff --git a/llama.cpp/common/arg.cpp b/llama.cpp/common/arg.cpp
--- a/llama.cpp/common/arg.cpp
+++ b/llama.cpp/common/arg.cpp
@@ -1375,6 +1375,91 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
             params.slot_initial_ctx = value;
         }
     ).set_env("LLAMA_ARG_SLOT_INITIAL_CTX").set_examples({LLAMA_EXAMPLE_SERVER, LLAMA_EXAMPLE_EMBEDDING, LLAMA_EXAMPLE_PARALLEL}));
+    // opencoti F5 M1 rest-kv-eviction — see docs/features/advanced_kv.md
+    add_opt(common_arg(
+        {"--rest-kv-eviction"},
+        {"--no-rest-kv-eviction"},
+        string_format("when context shift fires, evict the lowest-retention (KeyDiff key-similarity) contiguous window instead of the positional middle (default: %s)", params.rest_kv_eviction ? "enabled" : "disabled"),
+        [](common_params & params, bool value) {
+            params.rest_kv_eviction = value;
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_REST_KV_EVICTION"));
+    add_opt(common_arg(
+        {"--rest-kv-recent"}, "N",
+        string_format("rest-kv: protect the most-recent N tokens from eviction (default: %d)", params.rest_kv_recent),
+        [](common_params & params, int value) {
+            params.rest_kv_recent = value;
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_REST_KV_RECENT"));
+    add_opt(common_arg(
+        {"--rest-kv-layer"}, "N",
+        string_format("rest-kv: model layer whose keys score retention, -1 = auto/middle (default: %d)", params.rest_kv_layer),
+        [](common_params & params, int value) {
+            params.rest_kv_layer = value;
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_REST_KV_LAYER"));
+    // opencoti F5 M2 headinfer — see docs/features/advanced_kv.md
+    add_opt(common_arg(
+        {"--headinfer-gpu-heads-frac"}, "F",
+        string_format("headinfer: fraction of KV heads kept GPU-resident; the rest are offloaded to host memory and streamed back per step (default: %.2f, 1.0 = off). Only effective for GPU-offloaded layers with flash-attention.", (double) params.headinfer_gpu_heads_frac),
+        [](common_params & params, const std::string & value) {
+            params.headinfer_gpu_heads_frac = std::stof(value);
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_HEADINFER_GPU_HEADS_FRAC"));
+    // opencoti F5 M3 neo — see docs/features/advanced_kv.md
+    add_opt(common_arg(
+        {"--neo-pipeline"}, "MODE",
+        "neo (asymmetric GPU/CPU attention pipelining): off (default), on (engage when headinfer split is active for the layer), auto (reserved, same as on). Requires --headinfer-gpu-heads-frac < 1.0 to do anything.",
+        [](common_params & params, const std::string & value) {
+            if      (value == "off")  params.neo_pipeline_mode = 0;
+            else if (value == "on")   params.neo_pipeline_mode = 1;
+            else if (value == "auto") params.neo_pipeline_mode = 2;
+            else throw std::invalid_argument("--neo-pipeline must be off|on|auto");
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_NEO_PIPELINE"));
+    // opencoti F5-opt W2 (#290) iqk CPU flash-attention — see docs/features/advanced_kv.md
+    add_opt(common_arg(
+        {"--iqk-flash-attn"}, "MODE",
+        "iqk CPU flash-attention (vendored ik_llama.cpp engine): off (default), on. "
+        "Speeds up the M2 CPU-half attention on the token-generation path; falls back "
+        "to the mainline kernel for unsupported (type, shape) cases. x86_64-only.",
+        [](common_params & params, const std::string & value) {
+            if      (value == "off" || value == "false" || value == "0") params.iqk_flash_attn = false;
+            else if (value == "on"  || value == "true"  || value == "1") params.iqk_flash_attn = true;
+            else throw std::invalid_argument("--iqk-flash-attn must be on|off");
+            ggml_iqk_flash_attn_set_enabled(params.iqk_flash_attn);
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_IQK_FLASH_ATTN"));
+    // opencoti F5-opt W3 (#292) fused-MoE — see docs/features/advanced_kv.md
+    add_opt(common_arg(
+        {"--fused-moe-up-gate"}, "MODE",
+        "fused MoE up+gate+GLU (CUDA decode): off (default), on. Collapses the "
+        "separate-up/gate expert matmuls + GLU into one op at decode so the CUDA "
+        "backend runs the fused mmvq kernel without graph-pattern fusion. Only "
+        "engages for separate-up/gate MoE models (e.g. Gemma-4 A4B) at n_tokens==1.",
+        [](common_params & params, const std::string & value) {
+            if      (value == "off" || value == "false" || value == "0") params.fused_moe_up_gate = false;
+            else if (value == "on"  || value == "true"  || value == "1") params.fused_moe_up_gate = true;
+            else throw std::invalid_argument("--fused-moe-up-gate must be on|off");
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_FUSED_MOE_UP_GATE"));
+    // opencoti F5-opt W1 (#293) pcie profile — see docs/features/rolling_kv.md
+    add_opt(common_arg(
+        {"--pcie-autodetect"}, "MODE",
+        "pcie profile: auto-detect host<->device bandwidth + PCIe link at boot (on (default), off). Reads the rebar-probe JSON / nvidia-smi; the result feeds M7 Rolling KV tile sizing.",
+        [](common_params & params, const std::string & value) {
+            if      (value == "on"  || value == "true"  || value == "1") params.pcie_autodetect = true;
+            else if (value == "off" || value == "false" || value == "0") params.pcie_autodetect = false;
+            else throw std::invalid_argument("--pcie-autodetect must be on|off");
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_PCIE_AUTODETECT"));
+    add_opt(common_arg(
+        {"--pcie-bw-gbps"}, "F",
+        string_format("pcie profile: manual override of effective host->device bandwidth in GB/s (default: %.1f = autodetect). > 0 skips detection.", (double) params.pcie_bw_gbps),
+        [](common_params & params, const std::string & value) {
+            params.pcie_bw_gbps = std::stof(value);
+        }
+    ).set_examples({LLAMA_EXAMPLE_SERVER}).set_env("LLAMA_ARG_PCIE_BW_GBPS"));
     add_opt(common_arg(
         // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
         {"--slot-shrink-idle-ms"}, "N",
diff --git a/llama.cpp/common/common.cpp b/llama.cpp/common/common.cpp
--- a/llama.cpp/common/common.cpp
+++ b/llama.cpp/common/common.cpp
@@ -1636,6 +1636,8 @@ struct llama_context_params common_context_params_to_llama(const common_params &
     cparams.headinfer_gpu_heads_frac = params.headinfer_gpu_heads_frac;
     // opencoti F5 M3 neo — see docs/features/advanced_kv.md
     cparams.neo_pipeline_mode = params.neo_pipeline_mode;
+    // opencoti F5 W3 #292 fused-MoE — see docs/features/advanced_kv.md
+    cparams.fused_moe_up_gate = params.fused_moe_up_gate;
     cparams.n_rs_seq          = params.speculative.need_n_rs_seq();
     cparams.n_batch           = params.n_batch;
     cparams.n_ubatch          = params.n_ubatch;
diff --git a/llama.cpp/common/common.h b/llama.cpp/common/common.h
--- a/llama.cpp/common/common.h
+++ b/llama.cpp/common/common.h
@@ -445,6 +445,41 @@ struct common_params {
     // active for the layer), 2 = auto (reserved for future load-based
     // engagement; same as `on` today).
     uint32_t neo_pipeline_mode    =     0;
+    // opencoti F5 W3 #292 fused-MoE — see docs/features/advanced_kv.md
+    // false = off (default; unfused 3-node path). true = fuse decode MoE
+    // up+gate+GLU into one GGML_OP_MOE_FUSED_UP_GATE op for the CUDA backend.
+    bool     fused_moe_up_gate    = false;
+    // opencoti F5-opt W1 (#293) pcie profile — see docs/features/rolling_kv.md
+    // Boot-time host<->device bandwidth + PCIe link detection, consumed by M7
+    // Rolling KV (#296) tile sizing. autodetect reads the rebar-probe JSON /
+    // nvidia-smi; bw_gbps > 0 forces a manual override (0 = autodetect/default).
+    bool    pcie_autodetect       =  true;
+    // opencoti F5-opt W2 (#290) iqk CPU flash-attention — see docs/features/advanced_kv.md
+    // Engage the vendored ik_llama.cpp CPU Flash-Attention engine on the M2
+    // CPU-half (token-generation path). Off by default; flag-off is byte-identical.
+    // x86_64-only at runtime (the engine compiles only on the AVX2 x86_64 slice).
+    bool    iqk_flash_attn        = false;
+    float   pcie_bw_gbps          =  0.0f;
+    // opencoti F4 M3 Phase 2 — see docs/decisions/0001-lazy-slot-context.md
+    // Initial KV-cell allocation per sequence. 0 = full n_ctx_seq allocation
+    // (Phase 1 behavior — only zero-fill is deferred). Positive values cap
+    // the initial allocation and enable grow-on-demand up to n_ctx_seq.
+    int32_t slot_initial_ctx      =     0;
+    // opencoti F4 M3 Phase 3 — see docs/decisions/0001-lazy-slot-context.md
+    // Idle threshold (ms) after which a grown soft cap is shrunk back to
+    // slot_initial_ctx and the over-cap pages are decommitted via
+    // posix_madvise. 0 disables shrink-on-idle (Phase 2 behavior).
+    int32_t slot_shrink_idle_ms   =     0;
+    // opencoti F5 M2 headinfer — see docs/features/advanced_kv.md
+    // Fraction of KV heads kept GPU-resident; the rest are offloaded to host
+    // memory and streamed back per step. 1.0 = off (no split). Only active for
+    // GPU-offloaded layers with flash-attention (non-transposed V cache).
+    float   headinfer_gpu_heads_frac = 1.0f;
+    // opencoti F5 M3 neo — see docs/features/advanced_kv.md
+    // 0 = off (default; M2 concat path runs), 1 = on (engage when split
+    // active for the layer), 2 = auto (reserved for future load-based
+    // engagement; same as `on` today).
+    uint32_t neo_pipeline_mode    =     0;
     // opencoti F5-opt W1 (#293) pcie profile — see docs/features/rolling_kv.md
     // Boot-time host<->device bandwidth + PCIe link detection, consumed by M7
     // Rolling KV (#296) tile sizing. autodetect reads the rebar-probe JSON /
diff --git a/llama.cpp/ggml/include/ggml.h b/llama.cpp/ggml/include/ggml.h
--- a/llama.cpp/ggml/include/ggml.h
+++ b/llama.cpp/ggml/include/ggml.h
@@ -583,6 +583,8 @@ extern "C" {
 
         GGML_OP_GLU,
 
+        GGML_OP_MOE_FUSED_UP_GATE, // opencoti-hook: f5-opt-fmoe — W3 #292
+
         GGML_OP_COUNT,
     };
 
@@ -1437,6 +1439,22 @@ extern "C" {
             struct ggml_tensor  * b,
             struct ggml_tensor  * ids);
 
+    // opencoti-hook: f5-opt-fmoe — W3 #292: op-level fused MoE up+gate+GLU.
+    // build_moe_ffn otherwise emits {mul_mat_id(up), mul_mat_id(gate), glu}
+    // as 3 nodes that the CUDA backend can only fuse via the fragile
+    // ggml_can_fuse_subgraph graph-pattern path (which does not engage at
+    // decode for Gemma-4). This single op lets the backend run the already
+    // existing fused mmvq kernel directly. Output is GLU(gate@cur) (.) (up@cur),
+    // matching ggml_geglu_split(gate, up). up_exps/gate_exps: expert weight
+    // stacks (same type); cur: input activations; ids: expert selection.
+    GGML_API struct ggml_tensor * ggml_moe_up_gate(
+            struct ggml_context * ctx,
+            struct ggml_tensor  * up_exps,
+            struct ggml_tensor  * gate_exps,
+            struct ggml_tensor  * cur,
+            struct ggml_tensor  * ids,
+            enum   ggml_glu_op    glu_op);
+
     // A: m columns, n rows,
     // B: p columns, n rows,
     // result is m columns, p rows
diff --git a/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c b/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c
--- a/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c
+++ b/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c
@@ -2066,6 +2066,13 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
             {
                 ggml_compute_forward_glu(params, tensor);
             } break;
+        case GGML_OP_MOE_FUSED_UP_GATE:
+            {
+                // opencoti-hook: f5-opt-fmoe — W3 #292. CUDA-only op. The
+                // build_moe_ffn graph hook emits it only when the CUDA backend
+                // owns the FFN nodes, so the CPU scheduler never reaches here.
+                GGML_ABORT("MOE_FUSED_UP_GATE has no CPU implementation (CUDA-only op)");
+            } break;
         case GGML_OP_GET_REL_POS:
             {
                 ggml_compute_forward_get_rel_pos(params, tensor);
@@ -2333,6 +2340,12 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
                     GGML_ABORT("fatal error");
             }
             break;
+        case GGML_OP_MOE_FUSED_UP_GATE:
+            {
+                // opencoti-hook: f5-opt-fmoe — W3 #292. CUDA-only; never
+                // scheduled on CPU. Single task keeps the planner well-formed.
+                n_tasks = 1;
+            } break;
         case GGML_OP_SILU_BACK:
         case GGML_OP_MUL:
         case GGML_OP_DIV:
diff --git a/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu b/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu
--- a/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu
+++ b/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu
@@ -3005,6 +3005,25 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
         case GGML_OP_MUL_MAT_ID:
             ggml_cuda_mul_mat_id(ctx, dst);
             break;
+        case GGML_OP_MOE_FUSED_UP_GATE:
+            {
+                // opencoti-hook: f5-opt-fmoe — W3 #292. Drive the existing fused
+                // mmvq-id kernel directly: up = src0 @ src1, gate = fusion.gate @
+                // src1, out = act(gate) (.) up. Byte-identical to the graph-pattern
+                // fusion path (see the { MUL_MAT_ID, MUL_MAT_ID, GLU } branch in
+                // the eval fusion loop). Decode-only (ne[2]==1): the build_moe_ffn
+                // hook only emits this at n_tokens==1 and supports_op enforces it.
+                const ggml_tensor * up_exps   = dst->src[0];
+                const ggml_tensor * gate_exps = dst->src[1];
+                const ggml_tensor * cur       = dst->src[2];
+                const ggml_tensor * ids       = dst->src[3];
+                GGML_ASSERT(dst->ne[2] == 1 && "MOE_FUSED_UP_GATE is decode-only (n_tokens==1)");
+                ggml_cuda_mm_fusion_args_host fusion_data{};
+                fusion_data.gate   = gate_exps;
+                fusion_data.glu_op = (ggml_glu_op) dst->op_params[0];
+                ggml_cuda_mul_mat_vec_q(ctx, up_exps, cur, ids, dst, &fusion_data);
+            }
+            break;
         case GGML_OP_OUT_PROD:
             ggml_cuda_out_prod(ctx, dst);
             break;
@@ -5210,6 +5229,44 @@ static bool GGML_CALL ggml_backend_cuda_device_supports_op(ggml_backend_dev_t de
                     return false;
             }
             break;
+        case GGML_OP_MOE_FUSED_UP_GATE:
+            {
+                // opencoti-hook: f5-opt-fmoe — W3 #292. Supported only in the
+                // exact shape the fused mmvq-id decode kernel handles: quantized
+                // up/gate expert stacks of identical type, F32 activations, I32
+                // ids, single-token (decode) output, non-split weights, and a GLU
+                // op the kernel implements explicitly (GEGLU/SWIGLU/SWIGLU_OAI —
+                // others silently degrade to a plain product). build_moe_ffn only
+                // emits the op when all of these hold, so supports_op and the hook
+                // stay consistent (the scheduler never strands it on the CPU).
+                const ggml_tensor * up_exps   = op->src[0];
+                const ggml_tensor * gate_exps = op->src[1];
+                const ggml_tensor * cur       = op->src[2];
+                const ggml_tensor * ids       = op->src[3];
+                if (!up_exps || !gate_exps || !cur || !ids) {
+                    return false;
+                }
+                if (up_exps->buffer && ggml_backend_buft_is_cuda_split(up_exps->buffer->buft)) {
+                    return false;
+                }
+                if (!ggml_is_quantized(up_exps->type) || up_exps->type != gate_exps->type) {
+                    return false;
+                }
+                if (cur->type != GGML_TYPE_F32 || ids->type != GGML_TYPE_I32) {
+                    return false;
+                }
+                if (op->ne[2] != 1) {
+                    return false;
+                }
+                switch ((ggml_glu_op) op->op_params[0]) {
+                    case GGML_GLU_OP_GEGLU:
+                    case GGML_GLU_OP_SWIGLU:
+                    case GGML_GLU_OP_SWIGLU_OAI:
+                        return true;
+                    default:
+                        return false;
+                }
+            }
         case GGML_OP_MUL_MAT:
         case GGML_OP_MUL_MAT_ID:
             {
diff --git a/llama.cpp/ggml/src/ggml.c b/llama.cpp/ggml/src/ggml.c
--- a/llama.cpp/ggml/src/ggml.c
+++ b/llama.cpp/ggml/src/ggml.c
@@ -1078,9 +1078,10 @@ static const char * GGML_OP_NAME[GGML_OP_COUNT] = {
     "OPT_STEP_SGD",
 
     "GLU",
+    "MOE_FUSED_UP_GATE",
 };
 
-static_assert(GGML_OP_COUNT == 96, "GGML_OP_COUNT != 96");
+static_assert(GGML_OP_COUNT == 97, "GGML_OP_COUNT != 97");
 
 static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
     "none",
@@ -1188,9 +1189,10 @@ static const char * GGML_OP_SYMBOL[GGML_OP_COUNT] = {
     "sgd(x)",
 
     "glu(x)",
+    "moe_up_gate(up,gate,x)",
 };
 
-static_assert(GGML_OP_COUNT == 96, "GGML_OP_COUNT != 96");
+static_assert(GGML_OP_COUNT == 97, "GGML_OP_COUNT != 97");
 
 static_assert(GGML_OP_POOL_COUNT == 2, "GGML_OP_POOL_COUNT != 2");
 
@@ -3314,6 +3316,53 @@ struct ggml_tensor * ggml_mul_mat_id(
     return result;
 }
 
+// ggml_moe_up_gate
+// opencoti-hook: f5-opt-fmoe — W3 #292. Op-level fusion of the two parallel
+// expert matmuls (up = up_exps @ cur, gate = gate_exps @ cur) plus the GLU
+// activation that build_moe_ffn otherwise emits as 3 separate graph nodes.
+// Output == ggml_geglu_split(gate, up) == act(gate) (.) up, where the CUDA
+// backend runs the existing fused mmvq kernel (act on the gate projection,
+// multiply by the up projection — see ggml-cuda/mmvq.cu). Mirrors
+// ggml_mul_mat_id's shape contract: up_exps is the primary "as" weight, cur
+// the "b" activations shared by both projections, ids the expert selection.
+
+struct ggml_tensor * ggml_moe_up_gate(
+        struct ggml_context * ctx,
+        struct ggml_tensor  * up_exps,
+        struct ggml_tensor  * gate_exps,
+        struct ggml_tensor  * cur,
+        struct ggml_tensor  * ids,
+        enum   ggml_glu_op    glu_op) {
+    GGML_ASSERT(!ggml_is_transposed(up_exps));
+    GGML_ASSERT(!ggml_is_transposed(gate_exps));
+    GGML_ASSERT(ids->type == GGML_TYPE_I32);
+
+    // up and gate are parallel expert stacks: identical shape + type so one
+    // shared-activation fused mmvq can drive both weight sets.
+    GGML_ASSERT(ggml_are_same_shape(up_exps, gate_exps));
+    GGML_ASSERT(up_exps->type == gate_exps->type);
+
+    GGML_ASSERT(up_exps->ne[3] == 1);                 // 3d (one matrix per expert)
+    GGML_ASSERT(cur->ne[3] == 1);                     // cur is 3d
+    GGML_ASSERT(ids->ne[2] == 1 && ids->ne[3] == 1);  // ids is 2d
+    GGML_ASSERT(ids->ne[1] == cur->ne[2]);            // one expert list per cur row
+    GGML_ASSERT(up_exps->ne[0] == cur->ne[0]);        // can_mul_mat
+    GGML_ASSERT(ids->ne[0] % cur->ne[1] == 0);        // can broadcast
+
+    const int64_t ne[4] = { up_exps->ne[1], ids->ne[0], cur->ne[2], 1 };
+    struct ggml_tensor * result = ggml_new_tensor(ctx, GGML_TYPE_F32, 4, ne);
+
+    ggml_set_op_params_i32(result, 0, (int32_t) glu_op);
+
+    result->op     = GGML_OP_MOE_FUSED_UP_GATE;
+    result->src[0] = up_exps;
+    result->src[1] = gate_exps;
+    result->src[2] = cur;
+    result->src[3] = ids;
+
+    return result;
+}
+
 // ggml_out_prod
 
 static inline bool ggml_can_out_prod(const struct ggml_tensor * t0, const struct ggml_tensor * t1) {
diff --git a/llama.cpp/include/llama.h b/llama.cpp/include/llama.h
--- a/llama.cpp/include/llama.h
+++ b/llama.cpp/include/llama.h
@@ -362,6 +362,9 @@ extern "C" {
         // 0 = off (default), 1 = on, 2 = auto. Effective only when the
         // M2 head split is active for a given layer.
         uint32_t neo_pipeline_mode;
+        // opencoti F5 W3 #292 fused-MoE — see docs/features/advanced_kv.md
+        // true = fuse decode MoE up+gate+GLU into one op. false = off (default).
+        bool     fused_moe_up_gate;
         uint32_t n_rs_seq;          // number of recurrent-state snapshots per seq for rollback (0 = no rollback) [EXPERIMENTAL]
         int32_t  n_threads;         // number of threads to use for generation
         int32_t  n_threads_batch;   // number of threads to use for batch processing
diff --git a/llama.cpp/src/llama-context.cpp b/llama.cpp/src/llama-context.cpp
--- a/llama.cpp/src/llama-context.cpp
+++ b/llama.cpp/src/llama-context.cpp
@@ -66,6 +66,8 @@ llama_context::llama_context(
     cparams.headinfer_gpu_heads_frac = params.headinfer_gpu_heads_frac;
     // opencoti F5 M3 neo — see docs/features/advanced_kv.md
     cparams.neo_pipeline_mode = params.neo_pipeline_mode;
+    // opencoti F5 W3 #292 fused-MoE — see docs/features/advanced_kv.md
+    cparams.fused_moe_up_gate = params.fused_moe_up_gate;
 
     cparams.n_threads        = params.n_threads;
     cparams.n_threads_batch  = params.n_threads_batch;
@@ -3359,6 +3361,7 @@ llama_context_params llama_context_default_params() {
         /*.slot_shrink_idle_ms         =*/ 0,
         /*.headinfer_gpu_heads_frac    =*/ 1.0f,
         /*.neo_pipeline_mode            =*/ 0,  // opencoti F5 M3 neo — off by default
+        /*.fused_moe_up_gate           =*/ false, // opencoti F5 W3 #292 — off by default
         /*.n_rs_seq                    =*/ 0,
         /*.n_threads                   =*/ GGML_DEFAULT_N_THREADS, // TODO: better default
         /*.n_threads_batch             =*/ GGML_DEFAULT_N_THREADS,
diff --git a/llama.cpp/src/llama-cparams.h b/llama.cpp/src/llama-cparams.h
--- a/llama.cpp/src/llama-cparams.h
+++ b/llama.cpp/src/llama-cparams.h
@@ -32,6 +32,10 @@ struct llama_cparams {
     // 2 = auto (same engagement as `on` for now — reserved for future
     //         load-based engagement heuristics).
     uint32_t neo_pipeline_mode;
+    // opencoti F5 W3 #292 fused-MoE — see docs/features/advanced_kv.md.
+    // true = build_moe_ffn collapses the up/gate expert matmuls + GLU into one
+    // GGML_OP_MOE_FUSED_UP_GATE op at decode (n_tokens==1). false = unfused path.
+    bool     fused_moe_up_gate;
     uint32_t n_rs_seq;        // number of recurrent-state snapshots per seq for rollback
     int32_t  n_threads;       // number of threads to use for generation
     int32_t  n_threads_batch; // number of threads to use for batch processing
diff --git a/llama.cpp/src/llama-graph.cpp b/llama.cpp/src/llama-graph.cpp
--- a/llama.cpp/src/llama-graph.cpp
+++ b/llama.cpp/src/llama-graph.cpp
@@ -1534,7 +1534,28 @@ ggml_tensor * llm_graph_context::build_moe_ffn(
     ggml_tensor * up = nullptr;
     ggml_tensor * experts = nullptr;
 
-    if (gate_up_exps) {
+    // opencoti-hook: f5-opt-fmoe — W3 #292: collapse the separate-up/gate
+    // expert matmuls + the GLU activation into ONE GGML_OP_MOE_FUSED_UP_GATE
+    // op at decode, so the CUDA backend runs the existing fused mmvq kernel
+    // without relying on graph-pattern fusion (ggml_can_fuse_subgraph, which
+    // does not engage for Gemma-4 at decode). Guards mirror supports_op:
+    // decode only (n_tokens==1), separate up/gate of identical type, no
+    // bias/scale, and GELU→GEGLU or SILU→SWIGLU. Off path is byte-identical.
+    bool fused_up_gate = false;
+    if (cparams.fused_moe_up_gate && gate_exps && !gate_up_exps &&
+            !up_exps_b && !gate_exps_b && !up_exps_s && !gate_exps_s &&
+            up_exps->type == gate_exps->type && n_tokens == 1 &&
+            (type_op == LLM_FFN_GELU || type_op == LLM_FFN_SILU)) {
+        const enum ggml_glu_op glu_op = (type_op == LLM_FFN_GELU)
+            ? GGML_GLU_OP_GEGLU : GGML_GLU_OP_SWIGLU;
+        cur = ggml_moe_up_gate(ctx0, up_exps, gate_exps, cur, selected_experts, glu_op);
+        cb(cur, "ffn_moe_fused_up_gate", il);
+        fused_up_gate = true;
+    }
+
+    if (fused_up_gate) {
+        // cur already holds act(gate) (.) up from the fused op
+    } else if (gate_up_exps) {
         // merged gate_up path: one mul_mat_id, then split into gate and up views
         ggml_tensor * gate_up = build_lora_mm_id(gate_up_exps, cur, selected_experts); // [n_ff*2, n_expert_used, n_tokens]
         cb(gate_up, "ffn_moe_gate_up", il);
@@ -1601,6 +1622,9 @@ ggml_tensor * llm_graph_context::build_moe_ffn(
 
     const bool has_gate = gate_exps || gate_up_exps;
 
+    // opencoti-hook: f5-opt-fmoe — W3 #292: the fused op already applied the
+    // GLU activation; skip the separate activation pass.
+    if (!fused_up_gate)
     switch (type_op) {
         case LLM_FFN_SILU:
             if (gate_exps) {