---
base_model: coder3101/gemma-4-12B-it-qat-q4_0-unquantized-heretic
base_model_relation: quantized
library_name: gguf
license: gemma
license_link: https://ai.google.dev/gemma/terms
language:
- multilingual
pipeline_tag: image-text-to-text
tags:
- gguf
- llama.cpp
- text-generation
- image-text-to-text
- qat
- heretic
- uncensored
- multimodal
- vision
- abliterated
- conversational
- q4_0_pure
quantized_by: FadedRedStar
---

# 🤖 gemma-4-12B-it-qat-q4_0-heretic — GGUF

This repository hosts GGUF weights and the associated vision/audio projection matrix for **gemma-4-12B-it-qat-q4_0-heretic**, quantized from the source floating-point tensors provided by [coder3101/gemma-4-12B-it-qat-q4_0-unquantized-heretic](https://huggingface.co/coder3101/gemma-4-12B-it-qat-q4_0-unquantized-heretic).

## 🎯 QAT Architecture & Constraints

This model was produced using **Quantization Aware Training (QAT)**. Because the network parameters were exposed to 4-bit constraints natively during fine-tuning, standard post-training formats such as `Q4_K_M` or `Q5_K_M` are not compatible and would degrade output quality.

This repository instead provides a single **`Q4_0`** build, produced with the llama.cpp `--pure` flag, which matches the numerical profile the model was trained on.

## ℹ️ Model Profile & Core Features

**Gemma 4 12B** is a Google DeepMind model from the Gemma 4 family, a multimodal (text, image, audio) architecture featuring hybrid sliding-window/global attention, configurable thinking mode, native function calling, and system-role support. This is the **Quantization-Aware Training (QAT)** variant, whose weights were trained to retain near-bfloat16 quality once quantized to 4-bit — unlike a naively post-training-quantized model, QAT checkpoints are specifically calibrated during training for low-bit deployment.

The **heretic** suffix denotes post-processing via **[Heretic v1.2.0](https://github.com/p-e-w/heretic)** with the **Arbitrary-Rank Ablation (ARA)** method (row-norm preservation) performed by [coder3101](https://huggingface.co/coder3101), which suppresses refusal behavior while preserving the model's reasoning and multimodal capabilities.

## 📋 Technical Specifications

| Property | Value |
|---|---|
| **Base Architecture** | Gemma 4 12B (hybrid sliding-window/global attention) |
| **Developed by** | Google DeepMind |
| **Primary Use** | Multimodal reasoning, coding, agentic tasks |
| **Context Window** | 262,144 tokens |
| **Supported Modalities** | Text, Image, Audio |
| **Vision/Audio Projector** | Integrated (see repository files below) |
| **Quantization-Aware Training** | Yes (Q4_0-calibrated) |
| **Abliteration Tool** | Heretic v1.2.0 |
| **Abliteration Method** | Arbitrary-Rank Ablation (ARA) with row-norm preservation |
| **Prompt Format** | Gemma chat template (system/user/assistant roles) |

## 🛠️ Heretic Overrides (ARA)

| Property | Value |
|---|---|
| **start_layer_index** | 24 |
| **end_layer_index** | 48 |
| **preserve_good_behavior_weight** | 0.3707 |
| **steer_bad_behavior_weight** | 0.0010 |
| **overcorrect_relative_weight** | 0.6177 |
| **neighbor_count** | 15 |

## 📊 Refusal Bypass Metrics

> [!NOTE]
> The metrics below are self-reported by the original model author ([coder3101](https://huggingface.co/coder3101)) and have not been independently reproduced.

| Metric | This model | Original ([google/gemma-4-12B-it-qat-q4_0-unquantized](https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-unquantized)) |
|---|---|---|
| **KL divergence** | 0.0575 | 0 *(by definition)* |
| **Refusals** | 8/100 | 99/100 |

## 🧮 Numerical & Tensor Formats

| Property | Value |
|---|---|
| **Text Tensors** | Q4_0 (`--pure`, uniform quantization across all tensor types) |
| **Vision/Audio Tensors** | Q8_0, BF16 |

> [!NOTE]
> Unlike standard Q4_0 builds that mix in higher-precision tensors for embeddings and select layers, this quant was produced with the `--pure` flag, forcing all eligible tensors to Q4_0. This most closely mirrors the QAT calibration target and is the recommended pairing for this checkpoint.

## 📦 Available Model Files

**Main model weights**
| Filename | Quantization | llama.cpp Build | Size | Download |
|---|---|---|---|---|
| `gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf` | `Q4_0_PURE` | `b9851` | 6.26 GB | [📥 Download](https://huggingface.co/FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF/resolve/main/gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf) |

**mmproj — vision/audio projector files**
| Filename | Quantization | Size | Download |
|---|---|---|---|
| `mmproj-gemma-4-12B-it-qat-q4_0-heretic-Q8_0.gguf` | `Q8_0` | 152 MB | [📥 Download](https://huggingface.co/FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF/resolve/main/mmproj-gemma-4-12B-it-qat-q4_0-heretic-Q8_0.gguf) |
| `mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf` | `BF16` | 167 MB | [📥 Download](https://huggingface.co/FadedRedStar/gemma-4-12B-it-qat-q4_0-heretic-GGUF/resolve/main/mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf) |

## 🎛️ Component Pairing Guide

* **`Q4_0_PURE`**: The only main-weights format provided; required to match the QAT training profile.

**mmproj files** (optional): multimodal vision projectors. Pass one via the `--mmproj` flag in llama.cpp to enable image input.
* **`BF16` (Recommended)**: Highest possible image processing accuracy. While older projectors were small, modern vision towers can hover around **1GB**. If you are tight on VRAM, it is completely viable to run this on system RAM (CPU) with a minimal performance penalty, saving your precious GPU space for the main model layers.
* **`Q8_0`**: Cuts the projector file size and memory footprint in half (~500MB for larger 1GB files). Use this if you prefer to keep the vision tower hosted entirely on your GPU but need to claw back some VRAM to avoid Out-Of-Memory (OOM) crashes.

## ⚡ Deployment & Execution Commands

> [!IMPORTANT]
> The vision projector (`--mmproj`) must be supplied at runtime whenever image inputs are used. Omitting it disables multimodal capability entirely.

> [!NOTE]
> Google recommends the following sampling configuration for best results: `temperature=1.0`, `top_p=0.95`, `top_k=64`.

> [!NOTE]
> Thinking is enabled by including the `<|think|>` token at the start of the system prompt; remove it to disable reasoning output.

### llama.cpp CLI (with image)

```bash
./llama-cli \
  -m gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf \
  --mmproj mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf \
  -c 8192 \
  -ngl 99 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --image "path/to/image.jpg" \
  -p "<start_of_turn>user\nDescribe what you see in this image.<end_of_turn>\n<start_of_turn>model\n"
```

### OpenAI-Compatible API Server

```bash
./llama-server \
  --host 0.0.0.0 \
  --port 8080 \
  -m gemma-4-12B-it-qat-q4_0-heretic-Q4_0_PURE.gguf \
  --mmproj mmproj-gemma-4-12B-it-qat-q4_0-heretic-BF16.gguf \
  -c 16384 \
  -ngl 99 \
  --flash-attn
```

## 💬 Chat Templates & Prompt Design (Gemma)

```text
<start_of_turn>system
You are a helpful assistant.<end_of_turn>
<start_of_turn>user
Your question, image, or audio payload here.<end_of_turn>
<start_of_turn>model
```

## ⚠️ Safety & Operational Notes

- This model is abliterated and will generate content that standard aligned models refuse. Use responsibly and in compliance with applicable laws.
- Audio input is limited to 30 seconds; video is processed as frames at up to 60 seconds (1 fps).
- Place image content before text and audio content after text in the prompt for best results.
- In multi-turn conversations, do not carry prior turns' thinking content forward — only the final response should appear in history.
- Vision tensors are kept at BF16 or Q8_0 depending on the mmproj variant chosen, to preserve spatial feature quality.
- The `--pure` Q4_0 quantization applies uniform 4-bit precision across all eligible tensors rather than mixing precisions, which most closely matches this checkpoint's QAT calibration target.