Instructions to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF",
	filename="gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": [
				{
					"type": "text",
					"text": "Describe this image in one sentence."
				},
				{
					"type": "image_url",
					"image_url": {
						"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
					}
				}
			]
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE
# Run inference directly in the terminal:
llama cli -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE
# Run inference directly in the terminal:
llama cli -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE
# Run inference directly in the terminal:
./llama-cli -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Use Docker

docker model run hf.co/FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

LM Studio
Jan

vLLM

How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Ollama
How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with Ollama:
```
ollama run hf.co/FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE
```

Unsloth Studio

How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF to start chatting

How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with Docker Model Runner:
```
docker model run hf.co/FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE
```

Lemonade

How to use FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF:Q4_0_PURE

Run and chat with the model

lemonade run user.gemma-4-12B-it-qat-q4_0-heretic-GGUF-Q4_0_PURE

List all available models

lemonade list

🤖 gemma-4-12B-it-qat-q4_0-heretic — GGUF

This repository hosts GGUF weights and the associated vision/audio projection matrix for gemma-4-12B-it-qat-q4_0-heretic, quantized from the source floating-point tensors provided by coder3101/gemma-4-12B-it-qat-q4_0-unquantized-heretic.

🎯 QAT Architecture & Constraints

This model was produced using Quantization Aware Training (QAT). Because the network parameters were exposed to 4-bit constraints natively during fine-tuning, standard post-training formats such as Q4_K_M or Q5_K_M are not compatible and would degrade output quality.

This repository instead provides a single Q4_0 build, produced with the llama.cpp --pure flag, which matches the numerical profile the model was trained on.

ℹ️ Model Profile & Core Features

Gemma 4 12B is a Google DeepMind model from the Gemma 4 family, a multimodal (text, image, audio) architecture featuring hybrid sliding-window/global attention, configurable thinking mode, native function calling, and system-role support. This is the Quantization-Aware Training (QAT) variant, whose weights were trained to retain near-bfloat16 quality once quantized to 4-bit — unlike a naively post-training-quantized model, QAT checkpoints are specifically calibrated during training for low-bit deployment.

The heretic suffix denotes post-processing via Heretic v1.2.0 with the Arbitrary-Rank Ablation (ARA) method (row-norm preservation) performed by coder3101, which suppresses refusal behavior while preserving the model's reasoning and multimodal capabilities.

📋 Technical Specifications

Property	Value
Base Architecture	Gemma 4 12B (hybrid sliding-window/global attention)
Developed by	Google DeepMind
Primary Use	Multimodal reasoning, coding, agentic tasks
Context Window	262,144 tokens
Supported Modalities	Text, Image, Audio
Vision/Audio Projector	Integrated (see repository files below)
Quantization-Aware Training	Yes (Q4_0-calibrated)
Abliteration Tool	Heretic v1.2.0
Abliteration Method	Arbitrary-Rank Ablation (ARA) with row-norm preservation
Prompt Format	Gemma chat template (system/user/assistant roles)

🛠️ Heretic Overrides (ARA)

Property	Value
start_layer_index	24
end_layer_index	48
preserve_good_behavior_weight	0.3707
steer_bad_behavior_weight	0.0010
overcorrect_relative_weight	0.6177
neighbor_count	15

📊 Refusal Bypass Metrics

The metrics below are self-reported by the original model author (coder3101) and have not been independently reproduced.

Metric	This model	Original (google/gemma-4-12B-it-qat-q4_0-unquantized)
KL divergence	0.0575	0 (by definition)
Refusals	8/100	99/100

🧮 Numerical & Tensor Formats

Property	Value
Text Tensors	Q4_0 (`--pure`, uniform quantization across all tensor types)
Vision/Audio Tensors	Q8_0, BF16

Unlike standard Q4_0 builds that mix in higher-precision tensors for embeddings and select layers, this quant was produced with the --pure flag, forcing all eligible tensors to Q4_0. This most closely mirrors the QAT calibration target and is the recommended pairing for this checkpoint.

📦 Available Model Files

Main model weights

Filename	Quantization	llama.cpp Build	Size	Download
`gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf`	`Q4_0_PURE`	`b9851`	6.26 GB	📥 Download

mmproj — vision/audio projector files

Filename	Quantization	Size	Download
`mmproj-gemma-4-12B-it-qat-q4_0-heretic-Q8_0.gguf`	`Q8_0`	152 MB	📥 Download
`mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf`	`BF16`	167 MB	📥 Download

🎛️ Component Pairing Guide

Q4_0_PURE: The only main-weights format provided; required to match the QAT training profile.

mmproj files (optional): multimodal vision projectors. Pass one via the --mmproj flag in llama.cpp to enable image input.

BF16 (Recommended): Highest possible image processing accuracy. While older projectors were small, modern vision towers can hover around 1GB. If you are tight on VRAM, it is completely viable to run this on system RAM (CPU) with a minimal performance penalty, saving your precious GPU space for the main model layers.
Q8_0: Cuts the projector file size and memory footprint in half (~500MB for larger 1GB files). Use this if you prefer to keep the vision tower hosted entirely on your GPU but need to claw back some VRAM to avoid Out-Of-Memory (OOM) crashes.

⚡ Deployment & Execution Commands

The vision projector (--mmproj) must be supplied at runtime whenever image inputs are used. Omitting it disables multimodal capability entirely.

Google recommends the following sampling configuration for best results: temperature=1.0, top_p=0.95, top_k=64.

Thinking is enabled by including the <|think|> token at the start of the system prompt; remove it to disable reasoning output.

llama.cpp CLI (with image)

./llama-cli \
  -m gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf \
  --mmproj mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf \
  -c 8192 \
  -ngl 99 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --image "path/to/image.jpg" \
  -p "<start_of_turn>user\nDescribe what you see in this image.<end_of_turn>\n<start_of_turn>model\n"

OpenAI-Compatible API Server

./llama-server \
  --host 0.0.0.0 \
  --port 8080 \
  -m gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf \
  --mmproj mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf \
  -c 16384 \
  -ngl 99 \
  --flash-attn

💬 Chat Templates & Prompt Design (Gemma)

<start_of_turn>system
You are a helpful assistant.<end_of_turn>
<start_of_turn>user
Your question, image, or audio payload here.<end_of_turn>
<start_of_turn>model

⚠️ Safety & Operational Notes

This model is abliterated and will generate content that standard aligned models refuse. Use responsibly and in compliance with applicable laws.
Audio input is limited to 30 seconds; video is processed as frames at up to 60 seconds (1 fps).
Place image content before text and audio content after text in the prompt for best results.
In multi-turn conversations, do not carry prior turns' thinking content forward — only the final response should appear in history.
Vision tensors are kept at BF16 or Q8_0 depending on the mmproj variant chosen, to preserve spatial feature quality.
The --pure Q4_0 quantization applies uniform 4-bit precision across all eligible tensors rather than mixing precisions, which most closely matches this checkpoint's QAT calibration target.