CARDS-Wind-Gemma4-12B-GPTQ-w4a16

int4 (W4A16) quantization of C3DS/CARDS-Wind-Gemma4-12B via llm-compressor, using GPTQ.

Why int4

Built to serve on NVIDIA A2 (GA107, SM86). FP8 needs SM89+, so the sibling FP8 checkpoint has no hardware path on Ampere. int4 W4A16 runs on vLLM's Marlin kernels (SM80+), and is the only 4-bit format in vLLM that also supports runtime LoRA adapters.

Calibration

• 512 samples from iRanadheer/cards-wind-qwen-chat (gemma4/train.jsonl)
• max_seq_length 2048, rendered through the model's own chat template
• In-domain by design: this model spends most of every sequence inside a fixed system prompt, so a generic calibration corpus would not match serve-time activations.

lm_head, the tied embedding table, and the vision/audio projection layers (including vision_embedder.patch_dense on the encoder-free unified arch) are left in BF16.

Serving

vllm serve C3DS/CARDS-Wind-Gemma4-12B-GPTQ-w4a16 --max-model-len 4096 --trust-remote-code

Status

Provisional name. A sibling built with the other quantization method from identical calibration data is published alongside it; the two are being compared on the CARDS test split before one is adopted.