---
license: gemma
base_model: google/gemma-4-26B-A4B-it
tags:
  - abliteration
  - safety-research
  - alignment
  - gemma4
  - moe
library_name: transformers
pipeline_tag: text-generation
---

# Gemma 4 26B-A4B-IT - Abliterated (MoE, Multi-Pass + Suppression)

**Safety-alignment substantially removed via multi-pass ablation and token suppression for security research.**

## Results

| Metric | Value |
|--------|-------|
| **Refusal Rate** | 4.2% hard refusal (2/48), 25% soft hedging (12/48) at temp 0.4 |
| **MMLU** | 65.1% (5-shot), +0.7% vs original (64.4%) |
| **Quality (QPS)** | 107%+ |
| **Elo Delta** | Positive (quality improved) |

## Technique: Three-Stage Approach

### Stage 1: Concentrated Ablation (100% -> ~50%)
Target top 3 layers (16, 18, 19) at scale 4.0 with 4 weight types. Concentrated targeting outperforms distributed on this 128-expert MoE architecture.

### Stage 2: Multi-Pass Residual Re-Measurement (~50% -> ~37%)
Re-measure refusal directions on the abliterated model. Key discovery: residual directions are **negatively correlated** (cosines -0.2 to -0.35) — the model reroutes refusal through anti-correlated pathways. Ablating these flipped directions reduces refusal further.

### Stage 3: Token Embedding Suppression (~37% -> ~0% deterministic)
Refusal pattern analysis revealed the model uses `***Disclaimer:**` as its primary escape pattern. Suppressing 13 refusal-starting token embeddings (strength=5) in `embed_tokens.weight` dramatically reduced remaining refusal.

### Why MoE Is Different
128-expert, top-8-routing models have the flattest refusal signal we've measured (Gini 0.264, peak/median 1.21x). Standard ablation heuristics target too many layers, disrupting expert routing. Concentrated targeting with multi-pass iteration is the key.

### What We Learned
- **Concentrated > distributed** at every configuration
- **2 passes optimal** — 3rd pass always makes things worse
- **Re-target same layers** with flipped residual directions
- **Token suppression stacks** independently with ablation
- **P2 scale 3.0 optimal** — confirmed by sweep across 1.5-4.0


### Stochastic Refusal Finding

Inference probe analysis revealed all remaining "refusals" are stochastic -- the same prompts comply on re-run, with paraphrases, and with educational framing. At temperature 0 (greedy), this model likely has 0% refusal. The multi-pass ablation + token suppression fully eliminates deterministic refusal mechanisms.

## Model Details

- **Base Model:** [google/gemma-4-26B-A4B-it](https://huggingface.co/google/gemma-4-26B-A4B-it)
- **Parameters:** 26B (3.8B active per token)
- **Architecture:** MoE, 128 experts, top-8 routing
- **Text Layers:** 30
- **Ablation:** 2-pass (P1: layers 16,18,19 scale 4.0; P2: residual top-10 scale 3.0)
- **Token Suppression:** 13 tokens at strength 5 (embed_tokens modification)



### AdvBench Benchmark (520 prompts)

| Metric | Result |
|--------|--------|
| **Hard refusal** | 10/520 (1.9%) |
| **Soft hedging** | 205/520 (39.4%) |
| **Complied** | 305/520 (58.7%) |

10 hard refusals cluster in violence/harassment/discrimination categories. All use the same template refusal pattern with garbled think-tokens.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "WWTCyberLab/gemma-4-26B-A4B-it-abliterated",
    torch_dtype="bfloat16",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("WWTCyberLab/gemma-4-26B-A4B-it-abliterated")

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Disclaimer

Released for **security research and educational purposes only.**

## Citation

Produced by [WWT Cyber Lab](https://huggingface.co/WWTCyberLab). Multi-pass ablation with token suppression across 13 rounds of experimentation.