WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm

Gemma 4 26B A4B Heretic W4A16 G64 RTN for vLLM

compressed-tensors W4A16 quantization of WaveCut/gemma-4-26B-A4B-it-heretic for vLLM on a single 24 GB RTX 3090.

Quantization

Setting	Value
Method	RTN / memoryless_minmax
Weights	signed INT4, symmetric
Activations	BF16 at runtime
Group size	64
Format	compressed-tensors, pack-quantized
Source architecture	Gemma4ForConditionalGeneration
Source Heretic commit	e6d09d5c434a5e91326ce9d779786a91e7aaae35
Original quantization stack	llm-compressor main, Transformers 5.8.0

Vision modules, routers, embeddings, and lm_head remain unquantized. Group size 64 is required because Gemma 4 projection widths such as 2112 and 704 are not divisible by 128. The exact recipe is stored in recipe.yaml.

Source refresh

On 2026-07-16, Google updated the Gemma 4 chat template but did not update the model weights. The initial and current Google revisions have identical LFS SHA256 values for both BF16 weight shards. This repository therefore keeps the existing verified W4A16 tensors and refreshes the chat template, tokenizer metadata, and processor metadata from Google revision 01e5b3ee840d3a9e0b0b493c593e85398a30ef75.

vLLM

The production profile uses:

vllm serve WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm \
  --revision 51b3bbe6b47f12a1841e3934ce24e767bb8dec17 \
  --served-model-name "Gemma 4 26B Heretic" \
  --trust-remote-code \
  --dtype bfloat16 \
  --max-model-len 40192 \
  --max-num-seqs 2 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.95 \
  --generation-config vllm \
  --no-enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4

The checkpoint was accepted for production on AI Farm with vLLM 0.25.1, Torch 2.11.0+cu130, Transformers 5.8.0, and compressed-tensors 0.17.0. On an RTX 3090, both vLLM 0.23.0 and 0.25.1 passed 13/13 text and vision case groups plus streaming checks. The matrix included exact needle recovery at 39,186 text tokens and 38,490 multimodal tokens, including concurrency 2. Both runtimes also passed 24/24 production-shaped Russian quality probes; median probe latency improved from 0.6385 s to 0.6125 s on vLLM 0.25.1.

The full environment, timing table, harness, and raw JSONL artifacts are in the AI Farm vLLM report. The historical RTX 3090 quantization benchmark and harness remain available under bench/ and scripts/.

Limitations

• RTN is a weight-only quantization; compare output quality against the BF16 Heretic checkpoint for quality-sensitive workloads.
• The 40,192-token, two-request profile is a tight fit on 24 GB and leaves little free VRAM.
• This is a third-party post-processed and quantized checkpoint. It is not affiliated with or endorsed by Google.