---
license: other
license_name: health-ai-developer-foundations
base_model: google/medgemma-27b-text-it
tags: [fp8, compressed-tensors, sglang, vllm, medical]
---

# medgemma-auto-fp8

`google/medgemma-27b-text-it` quantized to **FP8 (W8A8 dynamic)** with llm-compressor, in
**compressed-tensors** format.

## Why this exists

Serving the BF16 checkpoint with `--quantization fp8` on a 48 GB card **OOMs**: SGLang/vLLM
allocate `params_dtype` (BF16, ~51 GiB for 27B) in `create_weights` and only compress *after*
loading. A pre-quantized checkpoint makes `create_weights` allocate FP8 directly (~26 GiB).
Measured peak while producing this artifact: 51.0 GiB VRAM.

## Recipe

```python
QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=['lm_head', 're:.*embed_tokens.*', 're:model\\.layers\\.0\\..*'])
```

- weights: 8-bit float, **per-channel**, symmetric
- activations: 8-bit float, **dynamic per-token** (data-free, no calibration set)
- **left in BF16:** `lm_head` (tied to `embed_tokens` in Gemma 3), `embed_tokens`,
  and **decoder layer 0** (the most quantization-sensitive block)

## Serving with SGLang

```bash
python -m sglang.launch_server --model-path fhai50032/medgemma-auto-fp8 --kv-cache-dtype fp8_e4m3 --attention-backend triton --context-length 65536
```

**Do not pass `--quantization fp8`** - it is auto-detected from `quantization_config`, and
passing it explicitly can select the wrong quant method for a compressed-tensors checkpoint.

Use `--attention-backend triton`: FlashInfer silently disables Gemma 3 sliding window (vLLM #20865).
SGLang also disables hybrid SWA memory for **all** Gemma 2/3/3n architectures, so every layer
caches the full context - budget KV cache accordingly.

## Accuracy

Not formally evaluated. No recovery figure is claimed for this artifact.