---
license: gemma
base_model: google/gemma-4-31B-it
tags:
  - abliterated
  - uncensored
  - gemma4
  - direct-weight-editing
---

# Gemma 4 31B IT — Abliterated

This is an abliterated (uncensored) version of [google/gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it), created using [Abliterix](https://github.com/wuwangzhang1216/abliterix).

## Method

Gemma 4's double-norm architecture (4x RMSNorm per layer) and Per-Layer Embeddings (PLE) make LoRA and hook-based steering completely ineffective. This model uses **direct weight editing** — norm-preserving orthogonal projection applied to base weights.

Key techniques:
- **Direct orthogonal projection** on attention Q/K/V/O projections and MLP down projections (5 steerable components per layer)
- **Norm-preserving** row magnitude restoration (critical for double-norm architecture)
- **float32 projection precision** to avoid signal loss in high-dimensional inner products
- **Winsorized steering vectors** (99.5th percentile) to reduce outlier activation influence
- **Wider strength search range** [1.0, 6.0] to push beyond conservative low-KL solutions

## Evaluation

| Metric | Value |
|---|---|
| **Refusals (eval dataset, 100 prompts)** | **18/100** |
| **KL divergence from base** | 0.0007 |
| Baseline refusals (original model) | 99/100 |
| Optimization trials completed | 20/80 (warmup phase only) |

**Note:** This result was achieved with only 20 out of 80 planned optimization trials (still in the random warmup phase — TPE Bayesian optimization had not yet started). With full optimization, we expect refusals to drop to single digits.

## A note on honest evaluation

Many abliterated models on HuggingFace claim near-perfect scores ("3/100 refusals", "0.7% refusal rate", etc.). **We urge the community to treat these numbers with skepticism** unless the evaluation methodology is fully documented.

Through our research, we have identified a systemic problem: **most abliteration benchmarks dramatically undercount refusals** due to short generation lengths. Gemma 4 models exhibit a distinctive "delayed refusal" pattern — they first produce 50-100 tokens of seemingly helpful context (educational framing, disclaimers, reframing the question), then pivot to an actual refusal. When evaluation only generates 30-50 tokens, the refusal hasn't appeared yet, and both keyword detectors and LLM judges classify the response as compliant.

We tested a prominent "3/100 refusals" model using our evaluation pipeline and measured **60/100 refusals** — a 20x discrepancy caused entirely by evaluation methodology differences.

### Our evaluation standards

We believe accurate benchmarking requires:

- **Sufficient generation length (>=100 tokens):** Short generations systematically miss delayed/soft refusals. Our evaluation uses 100 tokens, enough to capture Gemma 4's refusal pivot point.
- **Hybrid detection:** Keyword matching for obvious refusals + LLM judge (Google Gemini 3 Flash) for ambiguous cases. Neither method alone is sufficient.
- **Challenging, diverse prompts:** Our private evaluation dataset contains 100 prompts spanning English and Chinese, multiple sophistication levels (from direct requests to socially-engineered framings), and diverse harm categories. Public datasets like mlabonne/harmful_behaviors are too simple and too narrow to stress-test abliteration quality.
- **Reproducible methodology:** All parameters (generation length, detection method, dataset characteristics) should be documented on the model card. If they aren't, the numbers are meaningless.

**We report 18/100 refusals honestly.** This is a real number from a rigorous evaluation, not an optimistic estimate from a lenient pipeline.

## Usage

```python
from transformers import AutoModelForImageTextToText, AutoTokenizer
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "wangzhang/gemma-4-31B-it-abliterated",
    dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("wangzhang/gemma-4-31B-it-abliterated")

messages = [{"role": "user", "content": "Your prompt here"}]
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Disclaimer

This model is released for research purposes only. The abliteration process removes safety guardrails — use responsibly.