tiyuvta/gemma-4-31B-it-assistant-MTP-GGUF

gemma-4-31B-it-assistant — MTP drafter GGUF (Q8_0 + NVFP4)

GGUF builds of google/gemma-4-31B-it-assistant, the official 4-block assistant (MTP) drafter for google/gemma-4-31B-it, quantized for speculative decoding of gemma-4-31B-it GGUF trunks.

The assistant is a small draft head (4 layers, hidden 1024, tied embeddings, 262k vocab) that computes only its own queries and reads the target model's KV cache (attention_k_eq_v, all 4 layers KV-shared), so its per-draft-token cost is a small fraction of a target decode step. The target model verifies every drafted token; the emitted stream is the target's own output.

Files

file	quant	bytes	sha256
gemma-4-31B-it-official-Q8_0-MTP.gguf	Q8_0	514,666,944	f5a8758752b1195623a7e22f185b4445d61f19086eaaf9125b47c631ef7a9b66
gemma-4-31B-it-official-NVFP4-MTP.gguf	NVFP4 (6.79 BPW; norms/embeddings kept high-precision)	414,134,720	f3374652c1d302cbcd2d72480042965233dea493b72e97d8c14a1dd223844d24

Both files carry the full 17,336-byte gemma-4 chat template in their GGUF metadata, byte-identical to the source checkpoint's (verified on the exact published bytes — quantization tooling can silently drop chat templates, so this was gated, not assumed).

How they were made

• The official google/gemma-4-31B-it-assistant bf16 checkpoint was converted to an F16 GGUF with a byte-parity gate against the source weights (per-layer scalars byte-equal; output norm exact on all 1024 rows).
• llama-quantize produced the Q8_0 and NVFP4 files from that F16 conversion (no imatrix). NVFP4 is a Blackwell-native 4-bit float format; it was measured on RTX PRO 6000 Blackwell (sm_120). On other hardware, use the Q8_0 file.
• The F16 conversion itself is deliberately not published as a drafter: a float MTP head does not arm memra's gemma speculative route (float 2D matmul weights ride a different kernel path); the quantized builds are the servable artifacts.

Measured results

All numbers are our own measurements of these exact files. Acceptance is protocol-dependent — trunk quantization, sampling, workload shape and verification policy all move it — so every number below states its protocol.

Acceptance / identity / throughput A/B (fixed workload)

Protocol: memra engine (v0.95.0/v0.96.0 candidates), greedy decoding, single stream, 1× RTX PRO 6000 Blackwell 96 GB; trunk = a Q6_K-class GGUF build of gemma-4-31B-it; draft depth K=5; 12 held-out agent-shaped prompts; interleaved ×5 with one fresh boot per arm per repetition; correctness gate outranks speed — the speculative stream must reproduce plain greedy decode byte-exactly.

arm	acceptance (accepted/drafted)	tok/s median	byte-identity vs plain
Q8_0	67/135 = 0.496, bit-reproducible every repetition	118.0	30/30
NVFP4	67/135 = 0.496, numerically identical to Q8_0 every repetition	118.3	30/30
plain reference (same cell, drafter attached, spec route off)	—	58.2	—

The two quantizations are interchangeable on this protocol: acceptance is identical to the count, outputs are byte-identical to plain decode in both arms, and throughput is a wash (~2.03× plain on this cell). What NVFP4 buys is ~100.5 MB on disk/transfer and ~96 MiB resident VRAM.

Hosted-endpoint A/B (production serving config, measured 2026-08-20)

Four fixed greedy probes (256-token budget) through our then-hosted gemma-4-31B-it endpoint, minutes apart, Q8_0 vs NVFP4 under the identical serving configuration: outputs byte-identical on 4/4 probes (content and reasoning channels); probe-set acceptance 549/817 = 0.672 (Q8_0) vs 547/823 = 0.665 (NVFP4) — a wash.

Both files served that endpoint in production: the Q8_0 build until 2026-08-20, then the NVFP4 build until the hosted gemma-4-31B-it endpoint was retired on 2026-08-21. The measurements above are dated production receipts, not claims about a live service.

Lineage pairing matters

This is the official-lineage head (minted from the bf16 gemma-4-31B-it-assistant checkpoint). In our A/Bs it measured 0.58–0.60 prose acceptance on a trunk built from the official gemma-4-31B-it weights, and 0.28–0.34 when cross-paired with a QAT-lineage trunk. Pair this drafter with trunks derived from the official weights; QAT trunks want the QAT assistant head.

Run it with memra

memra serves gemma-4-31B-it with this drafter — attach it with MEMRA_DRAFT and the gemma speculative route arms automatically (K=5; engages on greedy, unconstrained, text-only sessions; speculative output is gated byte-identical to plain decode):

MEMRA_MODELS="google/gemma-4-31b-it=/path/to/gemma-4-31B-it-<trunk>.gguf" \
MEMRA_DRAFT=/path/to/gemma-4-31B-it-official-NVFP4-MTP.gguf \
memra-server

Hosted inference

The hosted gemma-4-31B-it endpoint this drafter served was retired on 2026-08-21 (dated receipts above). The same lab runs a production inference API at api.tiyuvta.ai — currently serving Qwen3.8 27B at native 262,144-token context (OpenAI Chat Completions, Responses, and Anthropic Messages on one endpoint, tool calling included), on the same memra engine and exactness gates used for every number on this card. Docs: inference.tiyuvta.ai/docs.

License

apache-2.0, inherited from google/gemma-4-31B-it-assistant; see the Gemma 4 license note linked from the base model card.