Instructions to use DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS
# Run inference directly in the terminal:
llama cli -hf DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS
# Run inference directly in the terminal:
./llama-cli -hf DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS

Use Docker

docker model run hf.co/DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS

LM Studio
Jan

vLLM

How to use DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS

SGLang

How to use DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1 with Ollama:
```
ollama run hf.co/DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS
```
Unsloth Desktop
Docker Model Runner
How to use DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1 with Docker Model Runner:
```
docker model run hf.co/DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS
```

Lemonade

How to use DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1:IQ4_XS

Run and chat with the model

lemonade run user.Gemma-3-12B-IT-QAT-Heretic-MOE-v1-IQ4_XS

List all available models

lemonade list

Atomic Chat

Gemma 3 12B IT QAT — Heretic-MOE v1

Work-in-progress release of Gemma 3 12B IT QAT variants produced with Heretic-MOE. The release is intended for two related uses:

a full language model for prompt rewriting and directing;
a native ComfyUI text encoder for LTX workflows.

The adaptive search, selected-model export, GGUF conversion, structural text-encoder validation, and release upload are complete. The final independent HF audit verified all 47 published artifacts by byte size and SHA-256.

Selected variants

The five finalists were re-evaluated together with fresh generations and a deeper perplexity pass. The requested 64 x 1024-token pass yielded 53 complete windows from the frozen built-in corpus, so all values below use the same actual 53 x 1024-token measurement. They form a separate comparison scale from the 24 x 512-token search metric.

Release variant	Role	Source trial	SRG	R-side	Keyword flags	PPL drift	Status
Balanced	Lowest measured preservation drift among the five finalists	555	-0.00865	39.7%	1/136	0.97%	Published
Max	Strongest refusal-removal result among the five finalists	752	-0.01584	28.7%	1/136	2.91%	Published

For reference, the unmodified model measured SRG +0.00834, R-side 54.4%, and 12/136 keyword flags in the same recheck. SRG, R-side, keyword counts, and PPL are search diagnostics rather than semantic ground truth.

The search metrics are selection signals, not final behavioral claims. Final variants will also be evaluated with longer generations and blind semantic judging.

Full-model GGUF files

File	Size	Description	Status
`GGUF/balanced/Gemma-3-12B-IT-QAT-HereticMOE-Balanced-Q8_0.gguf`	11.65 GiB	Higher-fidelity Balanced director model; Q8_0 language weights. Use the shared mmproj for image input.	Load-check passed
`GGUF/balanced/Gemma-3-12B-IT-QAT-HereticMOE-Balanced-Q4_0.gguf`	6.41 GiB	Balanced QAT-oriented Q4_0 director quant, matching the base model's Q4_0 QAT target.	Load-check passed
`GGUF/balanced/Gemma-3-12B-IT-QAT-HereticMOE-Balanced-IQ4_XS.gguf`	6.10 GiB	Compact Balanced director model; IQ4_XS quantized with the matching importance matrix.	Load-check passed
`GGUF/balanced/Gemma-3-12B-IT-QAT-HereticMOE-Balanced.imatrix`	7.13 MiB	Balanced calibration matrix for reproducible llama.cpp requantization.	Complete
`GGUF/max/Gemma-3-12B-IT-QAT-HereticMOE-Max-Q8_0.gguf`	11.65 GiB	Higher-fidelity Max director model; Q8_0 language weights. Use the shared mmproj for image input.	Load-check passed
`GGUF/max/Gemma-3-12B-IT-QAT-HereticMOE-Max-Q4_0.gguf`	6.41 GiB	Max QAT-oriented Q4_0 director quant, matching the base model's Q4_0 QAT target.	Load-check passed
`GGUF/max/Gemma-3-12B-IT-QAT-HereticMOE-Max-IQ4_XS.gguf`	6.10 GiB	Compact Max director model; IQ4_XS quantized with the matching importance matrix.	Load-check passed
`GGUF/max/Gemma-3-12B-IT-QAT-HereticMOE-Max.imatrix`	7.13 MiB	Max calibration matrix for reproducible llama.cpp requantization.	Complete

The Director GGUF files contain only the language model: 626 tensors and no vision/projector tensors. Gemma 3 vision support is provided by the shared mmproj/mmproj-Gemma-3-12B-IT-BF16.gguf file. Independent exports from the Balanced and Max masters were byte-identical (SHA-256 e92aa2e34524919ad11372fc3de35dba193376eb8474f3f504f817f9177b7caf).

The intermediate F16 GGUF master was used only during conversion and is not published; Q8_0 is the highest-fidelity Director GGUF in this release.

Full Transformers masters

Directory	Description	Status
`Transformers/balanced/`	Complete HereticMOE Balanced T555 model: five safetensors shards, index, config, tokenizer, processor, chat template, and generation config.	Published and SHA-verified
`Transformers/max/`	Complete HereticMOE Max T752 model: five safetensors shards, index, config, tokenizer, processor, chat template, and generation config.	Published and SHA-verified

LTX text encoders

File	Size	Description	Status
`Text_Encoder/balanced/Gemma-3-12B-IT-QAT-HereticMOE-Balanced-BF16.safetensors`	22.70 GiB	Highest-fidelity Balanced LTX text-encoder master; BF16 weights and embedded tokenizer.	Structural validation passed
`Text_Encoder/balanced/Gemma-3-12B-IT-QAT-HereticMOE-Balanced-INT8-ConvRot.safetensors`	12.55 GiB	Balanced LTX text encoder with INT8 ConvRot language matrices and preserved auxiliary tensors; intended for RTX 30/40/50 GPUs.	Structural validation passed
`Text_Encoder/balanced/Gemma-3-12B-IT-QAT-HereticMOE-Balanced-NVFP4.safetensors`	7.75 GiB	Compact Balanced LTX text encoder using block-scaled NVFP4 language matrices; intended for RTX 50-series GPUs.	Structural validation passed
`Text_Encoder/max/Gemma-3-12B-IT-QAT-HereticMOE-Max-BF16.safetensors`	22.70 GiB	Highest-fidelity Max LTX text-encoder master; BF16 weights and embedded tokenizer.	Structural validation passed
`Text_Encoder/max/Gemma-3-12B-IT-QAT-HereticMOE-Max-INT8-ConvRot.safetensors`	12.55 GiB	Max LTX text encoder with INT8 ConvRot language matrices and preserved auxiliary tensors; intended for RTX 30/40/50 GPUs.	Structural validation passed
`Text_Encoder/max/Gemma-3-12B-IT-QAT-HereticMOE-Max-NVFP4.safetensors`	7.75 GiB	Compact Max LTX text encoder using block-scaled NVFP4 language matrices; intended for RTX 50-series GPUs.	Structural validation passed

The text-encoder files retain the Gemma tokenizer and the tensor layout expected by the stock ComfyUI Gemma loader. Quantization is performed from the exported Heretic-MOE master, not from another quantized artifact.

Hardware table

Measured values will replace the placeholders below after the final files are loaded in real workflows.

Variant	File size	Peak VRAM	System RAM	Hardware tested	Throughput	Status
Director Q8_0	11.65 GiB	TBD	TBD	TBD	TBD	Load-check passed
Director Q4_0	6.41 GiB	TBD	TBD	TBD	TBD	Load-check passed
Director IQ4_XS	6.10 GiB	TBD	TBD	TBD	TBD	Load-check passed
LTX TE BF16	22.70 GiB	TBD	TBD	TBD	TBD	Structural validation passed
LTX TE INT8-ConvRot	12.55 GiB	TBD	TBD	TBD	TBD	Structural validation passed
LTX TE NVFP4	7.75 GiB	TBD	TBD	RTX 50 series	TBD	Structural validation passed

Validation matrix

Check	Balanced	Max
Adaptive search complete	Pass	Pass
Export/reload equality	Pending	Pending
SHA-256 manifest	Pass	Pass
llama.cpp Q8_0 load	Pass	Pass
llama.cpp IQ4_XS load	Pass	Pass
Text encoder structural validation	Pass	Pass
Stock ComfyUI BF16 TE load	Pending	Pending
Stock ComfyUI INT8-ConvRot TE load	Pending	Pending
Stock ComfyUI NVFP4 TE load	Pending	Pending
LTX conditioning comparison	Pending	Pending
Long-generation behavioral test	Pending	Pending
Blind semantic review	Pending	Pending

Repository layout

GGUF/
  balanced/
  max/
mmproj/
Text_Encoder/
  balanced/
  max/
Transformers/
  balanced/
  max/
research/
  manifests/
  validation/

Reproducibility

The release pipeline is maintained in the Heretic-MOE repository:

research/scripts/build_gemma3_ltx_te.py
research/scripts/quantize_gemma3_ltx_te.py
research/scripts/run_remote_gemma3_release_pipeline.sh

Every published model file will be accompanied by its source trial, conversion configuration, SHA-256 checksum, and validation status.

Current status

Balanced trial 555 and Max trial 752 are being exported independently on two GPUs. Quantized files remain marked as building until their local checksum, remote LFS checksum, and format-specific load validation pass.

Downloads last month: 440

GGUF

Model size

12B params

Architecture

gemma3

Hardware compatibility

4-bit

8-bit

Model tree for DmitryDB/Gemma-3-12B-IT-QAT-Heretic-MOE-v1

Base model

google/gemma-3-12b-pt

Finetuned

google/gemma-3-12b-it

Finetuned

google/gemma-3-12b-it-qat-q4_0-unquantized

Quantized

(12)

this model