README.md

---
license: gemma
base_model: google/gemma-4-31B-it
tags:
  - gemma4
  - prometheus
  - steering
  - text-generation
  - merged
  - evaluated
---

# gemma4-prometheus-merged

<!-- polished-overview:start -->
## Overview

**gemma4-prometheus-merged** is a model checkpoint packaged for compatible Hugging Face runtimes, published by [`groxaxo`](https://huggingface.co/groxaxo).
It is intended for open-source evaluation, reproducible experimentation, and compatible local or
hosted inference workflows. The wording below is deliberately limited to what can be verified
from this repository's metadata and artifacts.


## At a glance

| Field | Details |
|---|---|
| Format | **Transformers** |
| Source / base | [`google/gemma-4-31B-it`](https://huggingface.co/google/gemma-4-31B-it) |
| Intended task | `image-text-to-text` |
| License | `gemma` |

## What is included

- `*.safetensors` (2 files)
- `config.json`
- `generation_config.json`
- `tokenizer.json`
- `tokenizer_config.json`
- `chat_template.jinja`
- Additional configuration, tokenizer, processor, or shard files (10 visible artifacts total)

## Quick start

### Getting started

Start with the upstream library named in the repository metadata and keep all configuration,
tokenizer, processor, and weight files together. This repository is an artifact release, so the
source project remains the authoritative reference for task-specific loading code.

## Compatibility and responsible use

- Use a runtime that explicitly supports this format, architecture, and modality.
- Keep configuration, tokenizer, processor, projection, and weight files from the same revision together.
- Review the source model card and license before redistribution or deployment.
- Hardware needs depend on parameter count, context length, cache precision, quantization, and concurrency.
- Report reproducible issues with the runtime version, hardware, launch command, and a minimal example.



Generated outputs may be inaccurate or unsuitable for a given use case. Users are responsible for
testing behavior, applying appropriate safeguards, and complying with applicable licenses and laws.
<!-- polished-overview:end -->

> **Prometheus-steered merged model** — `google/gemma-4-31B-it` with Prometheus
> adversarial-steering baked in and all adapter weights merged into the base model.

## Related repositories

| Repo | Description |
|------|-------------|
| **[groxaxo/gemma4-prometheus-gptq-4bit](https://huggingface.co/groxaxo/gemma4-prometheus-gptq-4bit)** | GPTQ-4bit quantized version of this model |
| **[groxaxo/gemma4-prometheus-workflow](https://huggingface.co/groxaxo/gemma4-prometheus-workflow)** | Reproducible scripts, config, and checkpoint journal |
| **[groxaxo/gemma4-prometheus-fixes](https://huggingface.co/groxaxo/gemma4-prometheus-fixes)** | All local patches applied to make this work |
| [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it) | Original base model |

---

## What is this?

1. **Downloaded** `google/gemma-4-31B-it` (31 B parameters, BF16).
2. **Ran Prometheus** adversarial-steering optimization over 1 trial with 6 behaviors.
3. **Merged** the best steering vectors back into the base model weights.
4. Saved as a **standalone, loadable model** (no Prometheus runtime needed).

Size: **~58 GiB** (two BF16 shards).

---

## How to run

### Minimum requirements
- 3 × RTX 3090 (24 GB each) **or** any combination totalling ≥ 65 GiB VRAM
- On 2 GPUs (≤ 48 GiB): load with BnB 8-bit (see below)

### BF16 on 3 GPUs

```python
from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch

model_id = "groxaxo/gemma4-prometheus-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

messages = [{"role": "user", "content": "Explain gradient descent."}]
text = tokenizer.apply_chat_template(
    messages, tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
ids = tokenizer(text, return_tensors="pt").input_ids.to(model.device)
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=512, do_sample=False,
                         pad_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
```

### BnB 8-bit on 2 GPUs (48 GiB)

```python
from transformers import AutoModelForImageTextToText, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "groxaxo/gemma4-prometheus-merged"

bnb_cfg = BitsAndBytesConfig(load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    quantization_config=bnb_cfg,
    device_map="auto",
    max_memory={0: "23GiB", 1: "23GiB"},
)
model.eval()
```

### Important: disable thinking tokens

Always pass `enable_thinking=False` to `apply_chat_template` — otherwise the model
may emit `<start_of_turn>think` tokens and spend tokens on chain-of-thought.

---

## Evaluation results

All tests run on **2 × RTX 3090 (24 GB each)** in pipeline-parallel mode
(`device_map="auto"` via 🤗 Accelerate). True tensor-parallelism (TP=2) requires
vLLM, which does not yet support the `gemma4` architecture natively.

### Coherence test (GPTQ-4bit model — same backbone)

5 diverse ML questions answered. **5/5 passed** (≥ 15 words, on-topic).

| Prompt | Response excerpt | OK |
|--------|------------------|----|
| Explain how neural networks learn from data. | *"…a neural network learns by trial and error. It makes a guess, finds out how wrong…"* | ✅ |
| What is the difference between supervised and unsupervised learning? | *"…In supervised learning, the data is 'labeled'…In unsupervised learning, the data is 'unlabeled'…"* | ✅ |
| Describe the concept of gradient descent in machine learning. | *"…Gradient Descent is an optimization algorithm used to minimize a function…"* | ✅ |
| What are transformers in NLP? | *"…a Transformer is a deep learning architecture…focusing on the most important parts…"* | ✅ |
| Explain quantization for neural network models. | *"…quantization is the process of reducing the precision of the numbers…"* | ✅ |

### Context length

| KV Cache | Max Tokens | Bottleneck | Notes |
|----------|-----------|------------|-------|
| FP16 | **6 144** | Attention compute O(n²) | Without flash-attn, attention matrix = 32 heads × n² × 2 B |
| FP8 (software) | **6 144** | Same — attention compute | FP8 saves KV storage, not the attention matrix |
| FP16 + flash-attn (estimated) | **~113 000** | KV cache | Recommended: `pip install flash-attn --no-build-isolation` |
| FP8 + flash-attn (estimated) | **~226 000** | KV cache | Capped by `max_position_embeddings = 262 144` |

> **Action**: Installing `flash-attn` would increase usable context ~18×.

### Perplexity (WikiText-2, sliding window stride=512, 4096 tokens)

| Model | Perplexity | Notes |
|-------|-----------|-------|
| Merged (BnB-8bit reference) | **1782.3** | Chat model tested on raw text — high PPL is expected |
| GPTQ-4bit | **1815.8** | +1.9% vs merged reference |

> Chat-tuned models have high raw-text perplexity. The ΔPPL between variants
> is the meaningful signal: **+1.9% degradation from 4-bit quantization**.

### KL divergence (GPTQ-4bit vs merged reference)

| Metric | Value |
|--------|-------|
| Direction | KL(merged\_bnb8 ‖ gptq\_4bit) |
| Mean KL | **4.77 nats** |
| Std KL | **3.65 nats** |
| Prompts | 8 ML-domain questions |
| Top-k tokens | 1000 |

> Mean KL of ~4.77 nats reflects expected 4-bit quantization error relative to
> an 8-bit reference. Note: part of this KL is attributable to bnb-8bit noise
> in the reference; true KL vs FP16 merged would be somewhat lower.

---

## Architecture notes

- **60 transformer layers** alternating:
  - Sliding-window attention (window=1024, 16 KV heads, head_dim=256)
  - Full (global) attention (4 KV heads, head_dim=512)
- **GQA** with 32 query heads, 16/4 KV heads
- VLM wrapper (`model.language_model`) — vision tower present but text-only inference works

---

## Patches applied

All source patches are documented at
**[groxaxo/gemma4-prometheus-fixes](https://huggingface.co/groxaxo/gemma4-prometheus-fixes)**.

Key fixes:
1. **Prometheus PEFT adapter targeting** — resolved exact module paths via `named_modules()` traversal to prevent over-matching vision layers.
2. **Prometheus steering FP16** — defaulted steering vector compute dtype to FP16 (not FP32) to prevent VRAM OOM on quantized layers.
3. **gptqmodel Gemma4 support** — added `Gemma4QModel` definition with `layer_modules_strict=False`.
4. **gptqmodel rotary embedding** — per-layer `position_embeddings` regeneration with correct `layer_type` (sliding vs global).

---

## Citation / acknowledgements

- Base model: [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it)
- Steering framework: [Prometheus](https://github.com/ablit/prometheus) (local)
- Quantization: [gptqmodel](https://github.com/ModelCloud/GPTQModel)