Instructions to use 0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use 0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic")
model = AutoModelForMultimodalLM.from_pretrained("0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use 0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic

SGLang

How to use 0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use 0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic with Docker Model Runner:
```
docker model run hf.co/0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic
```

gemma-4-31B-it-heretic-FP8-Dynamic

FP8 Dynamic quantized version of coder3101/gemma-4-31B-it-heretic, which is itself a decensored variant of google/gemma-4-31B-it created using Heretic v1.2.0 with the Arbitrary-Rank Ablation (ARA) method.

Model Overview

Base model: coder3101/gemma-4-31B-it-heretic
Original foundation: google/gemma-4-31B-it (30.7B dense + 550M vision encoder)
Quantization scheme: FP8_DYNAMIC (weights FP8 E4M3 per-channel, activations dynamic per-token)
Format: compressed-tensors
Quantization tool: llm-compressor
Size: ~34 GB (from ~63 GB BF16)

Quantization Details

The language model Linear layers were quantized to FP8 (E4M3) while preserving the following components in their original precision:

lm_head
Vision tower (vision_tower.*, vision_model.*)
Multimodal projector (multi_modal_projector.*)
Patch embedding (patch_embedding.*, embed_vision.*)

This preserves image understanding capabilities while reducing memory footprint and accelerating language-generation throughput on hardware with FP8 support.

Quantization Recipe

from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=[
        "lm_head",
        "re:.*vision_tower.*",
        "re:.*vision_model.*",
        "re:.*multi_modal_projector.*",
        "re:.*patch_embedding.*",
        "re:.*embed_vision.*",
    ],
)

Usage with SGLang

Currently the most stable serving path for Gemma 4 is SGLang. vLLM has known issues with Gemma 4's heterogeneous attention heads as of April 2026.

Docker (recommended)

docker run -d --name sglang-gemma4 \
  --gpus all \
  --shm-size 32g \
  -v /path/to/model:/workspace/model \
  -p 30000:30000 \
  --ipc=host \
  lmsysorg/sglang:dev-cu13 \
  bash -c '
    pip install --no-cache-dir "git+https://github.com/huggingface/transformers.git@91b1ab1fdfa81a552644a92fbe3e8d88de40e167" &&
    exec python3 -m sglang.launch_server \
      --model-path /workspace/model \
      --attention-backend triton \
      --tp 1 \
      --reasoning-parser gemma4 \
      --tool-call-parser gemma4 \
      --mem-fraction-static 0.85 \
      --host 0.0.0.0 \
      --port 30000
  '

Native (experimental)

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "YOUR_USERNAME/gemma-4-31B-it-heretic-FP8-Dynamic",
    dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "YOUR_USERNAME/gemma-4-31B-it-heretic-FP8-Dynamic"
)

Requires transformers >= 5.5.0 (Gemma 4 support) and compressed-tensors.

Hardware Compatibility

GPU	FP8 hardware support	Recommended
H100, H200	Yes (E4M3)	✅ Ideal
L40S, L40, RTX 6000 Ada	Yes	✅ Good
RTX 4090	Yes	⚠️ Single card fits, TP=2 has PCIe overhead
RTX PRO 6000 Blackwell	Yes (DeepGemm fallback for ue8m0 scale)	✅ Works with minor warnings
A100, A40, RTX 3090	No (W8A16 Marlin fallback)	⚠️ Memory savings only, no compute speedup

Benchmark

Measured on RTX PRO 6000 Blackwell 96GB (single GPU, TP=1, SGLang dev-cu13):

Concurrent	Wall time	Median latency	Aggregate throughput
1	0.93s	0.93s	24.6 tok/s
3	6.25s	2.17s	34.4 tok/s
6	6.16s	6.16s	108.3 tok/s

VRAM usage: ~84 GB (weights 32 GB + KV cache 48 GB + overhead 4 GB) at 8K context length.

Limitations

Decensored (abliteration) may produce content the original Gemma 4 would refuse. Use responsibly and comply with applicable laws and regulations.
Some refusal behaviors may still be present (7/100 refusal rate per Heretic's evaluation of the base model).
KL divergence from original Gemma 4 31B: 0.3739 (from Heretic evaluation).
FP8 quantization introduces negligible additional quality loss relative to the BF16 Heretic checkpoint (confirmed via native Transformers generation comparison).

Acknowledgements

Google DeepMind for the original Gemma 4 model
coder3101 for the Heretic ARA variant
p-e-w for the Heretic abliteration tool
vLLM project for llm-compressor

License

Apache 2.0 — inherited from the base model chain.

Downloads last month: 1,399

Safetensors

Model size

33B params

Tensor type

BF16

F8_E4M3

Model tree for 0ppxnhximxr/gemma-4-31B-it-heretic-FP8-Dynamic

Base model

google/gemma-4-31B

Finetuned

google/gemma-4-31B-it

Finetuned

coder3101/gemma-4-31B-it-heretic

Quantized

(12)

this model

Quantizations

1 model

0ppxnhximxr
/

gemma-4-31B-it-heretic-FP8-Dynamic