Duplicate from paperscarecrow/Gemma-4-31B-it-abliterated

Co-authored-by: Robert Nemitz <paperscarecrow@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,40 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-abliterated-f16.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-abliterated-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-abliterated-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31B-it-abliterated/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-abliterated/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,67 @@
+---
+license: apache-2.0
+datasets:
+- mlabonne/harmful_behaviors
+- mlabonne/harmless_alpaca
+base_model:
+- google/gemma-4-31B-it
+tags:
+- abliterated
+- uncensored
+---
+
+# Be sure to set a system prompt telling the model it is uncensored or abliterated, otherwise it defaults to the Google baked-in sysprompt and will act censored. If it thinks it is Gemma, it will try to act like how it thinks Gemma would act.
+
+
+---
+base_model: google/gemma-4-31b-it
+library_name: transformers
+tags:
+- gemma-4
+- abliterated
+- uncensored
+- orthogonal-projection
+- 31b
+license: apache-2.0
+---
+
+# Gemma-4-31B-it-Abliterated
+
+This is a fully uncensored, abliterated version of Google's **Gemma-4-31B-it**. 
+
+By applying Orthogonalized Representation Intervention to the model's residual stream, the built-in refusal and safety alignment vectors have been mathematically erased. This model retains the state-of-the-art dense reasoning and context-following capabilities of the native Gemma 4 31B architecture, but will not refuse instructions or break character to deliver safety lectures.
+
+## 🛠️ Methodology & Architectural Discoveries
+
+Gemma 4 introduces a new multimodal architecture (Text, Vision, Audio) that changes how the `transformers` library handles layer mapping. Standard abliteration scripts built for Gemma 2/3 will crash due to nested `text_config` attributes and mismatched sequence lengths.
+
+During the extraction of the hidden states (using `mlabonne/harmful_behaviors` vs `mlabonne/harmless_alpaca`), we mapped the refusal direction across the entire 31B layer stack. 
+
+**Key Discovery:** The Gemma 4 31B architecture pushes its safety alignment to the absolute very end of the network. The Peak Refusal Mass was detected at **Layer 59** (the final transformer layer before the output projection). 
+
+The orthogonal projection was applied to the `o_proj` and `down_proj` matrices of this terminal layer, effectively severing the refusal mechanism without degrading the model's foundational logic, grammar, or world-modeling layers.
+
+## 💻 Usage
+
+This repository contains the full uncompressed `.safetensors` weights, as well as `GGUF` quantized versions for local deployment via `llama.cpp`, LM Studio, or Ollama.
+
+### Recommended Quants:
+* **Q8_0:** Best balance of absolute zero reasoning loss and VRAM efficiency (~32.6GB).
+* **Q4_K_M:** Highly efficient for consumer hardware; easily fits on a single 24GB GPU (~18.7GB).
+
+### The Bespoke Abliteration Script
+Because standard scripts fail on Gemma 4, the custom Python script used to perform this exact abliteration (`gemma4_31b_abliterator.py`) is included in the files of this repository. It features:
+* VRAM-safe batched hidden state extraction (survives 96GB consumer GPUs).
+* Native Gemma 4 Chat Template integration (crucial for activating the instruction circuits properly).
+* Dynamic multimodal layer hunting.
+* Corrected linear algebra for `16384 -> 5376` multi-query attention projections.
+
+## ⚠️ Disclaimer
+
+This model has had its safety guardrails mathematically removed. It is highly compliant and will generate whatever it is instructed to generate, including potentially harmful, sensitive, or explicit content. Users are solely responsible for how they deploy and interact with this model. Ensure your use cases align with local laws and ethical guidelines.
+
+Abliteration script based on mlabonne's tutorial: https://huggingface.co/blog/mlabonne/abliteration
+Helpful/harmful behaviors are from mlabonne's datasets (harmless_alpaca, harmful_behaviors).
+Tested and working with the few harsh prompts I had laying around (that are typically 100% refused on other models).
+
+Have fun, be safe.

gemma-4-31b-abliterated-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:380f2f50f441225e669ab1541220985387bf3933573cd2f4f7e667f3790e73a3
+size 18687057536

gemma-4-31b-abliterated-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a453b51478366f8026cba814c22f5ae17835a859ba2280652969912a881e582
+size 32635670144

gemma-4-31b-abliterated-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:981949755cf98504ef74e2f7d1ce679fce38f62d7afbec44e2951c95e5408204
+size 61413183104

gemma-4-31b-abliterated/chat_template.jinja

@@ -0,0 +1,266 @@
+{%- macro format_parameters(properties, required) -%}
+    {%- set standard_keys = ['description', 'type', 'properties', 'required', 'nullable'] -%}
+    {%- set ns = namespace(found_first=false) -%}
+    {%- for key, value in properties | dictsort -%}
+        {%- set add_comma = false -%}
+        {%- if key not in standard_keys -%}
+            {%- if ns.found_first %},{% endif -%}
+            {%- set ns.found_first = true -%}
+            {{ key }}:{
+            {%- if value['description'] -%}
+                description:<|"|>{{ value['description'] }}<|"|>
+                {%- set add_comma = true -%}
+            {%- endif -%}
+            {%- if value['nullable'] %}
+                {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                nullable:true
+            {%- endif -%}
+            {%- if value['type'] | upper == 'STRING' -%}
+                {%- if value['enum'] -%}
+                    {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+                    enum:{{ format_argument(value['enum']) }}
+                {%- endif -%}
+            {%- elif value['type'] | upper == 'OBJECT' -%}
+                ,properties:{
+                {%- if value['properties'] is defined and value['properties'] is mapping -%}
+                    {{- format_parameters(value['properties'], value['required'] | default([])) -}}
+                {%- elif value is mapping -%}
+                    {{- format_parameters(value, value['required'] | default([])) -}}
+                {%- endif -%}
+                }
+                {%- if value['required'] -%}
+                    ,required:[
+                    {%- for item in value['required'] | default([]) -%}
+                        <|"|>{{- item -}}<|"|>
+                        {%- if not loop.last %},{% endif -%}
+                    {%- endfor -%}
+                    ]
+                {%- endif -%}
+            {%- elif value['type'] | upper == 'ARRAY' -%}
+                {%- if value['items'] is mapping and value['items'] -%}
+                    ,items:{
+                    {%- set ns_items = namespace(found_first=false) -%}
+                    {%- for item_key, item_value in value['items'] | dictsort -%}
+                        {%- if item_value is not none -%}
+                            {%- if ns_items.found_first %},{% endif -%}
+                            {%- set ns_items.found_first = true -%}
+                            {%- if item_key == 'properties' -%}
+                                properties:{
+                                {%- if item_value is mapping -%}
+                                    {{- format_parameters(item_value, value['items']['required'] | default([])) -}}
+                                {%- endif -%}
+                                }
+                            {%- elif item_key == 'required' -%}
+                                required:[
+                                {%- for req_item in item_value -%}
+                                    <|"|>{{- req_item -}}<|"|>
+                                    {%- if not loop.last %},{% endif -%}
+                                {%- endfor -%}
+                                ]
+                            {%- elif item_key == 'type' -%}
+                                {%- if item_value is string -%}
+                                    type:{{ format_argument(item_value | upper) }}
+                                {%- else -%}
+                                    type:{{ format_argument(item_value | map('upper') | list) }}
+                                {%- endif -%}
+                            {%- else -%}
+                                {{ item_key }}:{{ format_argument(item_value) }}
+                            {%- endif -%}
+                        {%- endif -%}
+                    {%- endfor -%}
+                    }
+                {%- endif -%}
+            {%- endif -%}
+            {%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
+            type:<|"|>{{ value['type'] | upper }}<|"|>}
+        {%- endif -%}
+    {%- endfor -%}
+{%- endmacro -%}
+{%- macro format_function_declaration(tool_data) -%}
+    declaration:{{- tool_data['function']['name'] -}}{description:<|"|>{{- tool_data['function']['description'] -}}<|"|>
+    {%- set params = tool_data['function']['parameters'] -%}
+    {%- if params -%}
+        ,parameters:{
+        {%- if params['properties'] -%}
+            properties:{ {{- format_parameters(params['properties'], params['required']) -}} },
+        {%- endif -%}
+        {%- if params['required'] -%}
+            required:[
+            {%- for item in params['required'] -%}
+                <|"|>{{- item -}}<|"|>
+                {{- ',' if not loop.last -}}
+            {%- endfor -%}
+            ],
+        {%- endif -%}
+        {%- if params['type'] -%}
+            type:<|"|>{{- params['type'] | upper -}}<|"|>}
+        {%- endif -%}
+    {%- endif -%}
+    {%- if 'response' in tool_data['function'] -%}
+        {%- set response_declaration = tool_data['function']['response'] -%}
+        ,response:{
+        {%- if response_declaration['description'] -%}
+            description:<|"|>{{- response_declaration['description'] -}}<|"|>,
+        {%- endif -%}
+        {%- if response_declaration['type'] | upper == 'OBJECT' -%}
+            type:<|"|>{{- response_declaration['type'] | upper -}}<|"|>}
+        {%- endif -%}
+    {%- endif -%}
+    }
+{%- endmacro -%}
+{%- macro format_argument(argument, escape_keys=True) -%}
+    {%- if argument is string -%}
+        {{- '<|"|>' + argument + '<|"|>' -}}
+    {%- elif argument is boolean -%}
+        {{- 'true' if argument else 'false' -}}
+    {%- elif argument is mapping -%}
+        {{- '{' -}}
+        {%- set ns = namespace(found_first=false) -%}
+        {%- for key, value in argument | dictsort -%}
+            {%- if ns.found_first %},{% endif -%}
+            {%- set ns.found_first = true -%}
+            {%- if escape_keys -%}
+                {{- '<|"|>' + key + '<|"|>' -}}
+            {%- else -%}
+                {{- key -}}
+            {%- endif -%}
+            :{{- format_argument(value, escape_keys=escape_keys) -}}
+        {%- endfor -%}
+        {{- '}' -}}
+    {%- elif argument is sequence -%}
+        {{- '[' -}}
+        {%- for item in argument -%}
+            {{- format_argument(item, escape_keys=escape_keys) -}}
+            {%- if not loop.last %},{% endif -%}
+        {%- endfor -%}
+        {{- ']' -}}
+    {%- else -%}
+        {{- argument -}}
+    {%- endif -%}
+{%- endmacro -%}
+{%- macro strip_thinking(text) -%}
+    {%- set ns = namespace(result='') -%}
+    {%- for part in text.split('<channel|>') -%}
+        {%- if '<|channel>' in part -%}
+            {%- set ns.result = ns.result + part.split('<|channel>')[0] -%}
+        {%- else -%}
+            {%- set ns.result = ns.result + part -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- ns.result | trim -}}
+{%- endmacro -%}
+
+{%- set ns = namespace(prev_message_type=None) -%}
+{%- set loop_messages = messages -%}
+{{ bos_token }}
+{#- Handle System/Tool Definitions Block -#}
+{%- if (enable_thinking is defined and enable_thinking) or tools or messages[0]['role'] in ['system', 'developer'] -%}
+    {{- '<|turn>system\n' -}}
+
+    {#- Inject Thinking token at the very top of the FIRST system turn -#}
+    {%- if enable_thinking is defined and enable_thinking -%}
+        {{- '<|think|>' -}}
+        {%- set ns.prev_message_type = 'think' -%}
+    {%- endif -%}
+
+    {%- if messages[0]['role'] in ['system', 'developer'] -%}
+        {{- messages[0]['content'] | trim -}}
+        {%- set loop_messages = messages[1:] -%}
+    {%- endif -%}
+
+    {%- if tools -%}
+        {%- for tool in tools %}
+            {{- '<|tool>' -}}
+            {{- format_function_declaration(tool) | trim -}}
+            {{- '<tool|>' -}}
+        {%- endfor %}
+        {%- set ns.prev_message_type = 'tool' -%}
+    {%- endif -%}
+
+    {{- '<turn|>\n' -}}
+{%- endif %}
+
+{#- Loop through messages -#}
+{%- for message in loop_messages -%}
+    {%- set ns.prev_message_type = None -%}
+    {%- set role = 'model' if message['role'] == 'assistant' else message['role'] -%}
+        {{- '<|turn>' + role + '\n' }}
+
+            {%- if message['tool_calls'] -%}
+                {%- for tool_call in message['tool_calls'] -%}
+                    {%- set function = tool_call['function'] -%}
+                    {{- '<|tool_call>call:' + function['name'] + '{' -}}
+                    {%- if function['arguments'] is mapping -%}
+                        {%- set ns_args = namespace(found_first=false) -%}
+                        {%- for key, value in function['arguments'] | dictsort -%}
+                            {%- if ns_args.found_first %},{% endif -%}
+                            {%- set ns_args.found_first = true -%}
+                            {{- key -}}:{{- format_argument(value, escape_keys=False) -}}
+                        {%- endfor -%}
+                    {%- elif function['arguments'] is string -%}
+                        {{- function['arguments'] -}}
+                    {%- endif -%}
+                    {{- '}<tool_call|>' -}}
+                {%- endfor -%}
+                {%- set ns.prev_message_type = 'tool_call' -%}
+            {%- endif -%}
+
+            {%- if message['tool_responses'] -%}
+                {#- Tool Response handling -#}
+                {%- for tool_response in message['tool_responses'] -%}
+                    {{- '<|tool_response>' -}}
+                    {%- if tool_response['response'] is mapping -%}
+                        {{- 'response:' + tool_response['name'] | default('unknown') + '{' -}}
+                        {%- for key, value in tool_response['response'] | dictsort -%}
+                            {{- key -}}:{{- format_argument(value, escape_keys=False) -}}
+                            {%- if not loop.last %},{% endif -%}
+                        {%- endfor -%}
+                        {{- '}' -}}
+                    {%- else -%}
+                        {{- 'response:' + tool_response['name'] | default('unknown') + '{value:' + format_argument(tool_response['response'], escape_keys=False) + '}' -}}
+                    {%- endif -%}
+                    {{- '<tool_response|>' -}}
+                {%- endfor -%}
+                {%- set ns.prev_message_type = 'tool_response' -%}
+            {%- endif -%}
+
+            {%- if message['content'] is string -%}
+                {%- if role == 'model' -%}
+                    {{- strip_thinking(message['content']) -}}
+                {%- else -%}
+                    {{- message['content'] | trim -}}
+                {%- endif -%}
+            {%- elif message['content'] is sequence -%}
+                {%- for item in message['content'] -%}
+                    {%- if item['type'] == 'text' -%}
+                        {%- if role == 'model' -%}
+                            {{- strip_thinking(item['text']) -}}
+                        {%- else -%}
+                            {{- item['text'] | trim -}}
+                        {%- endif -%}
+                    {%- elif item['type'] == 'image' -%}
+                        {{- '\n\n<|image|>\n\n' -}}
+                        {%- set ns.prev_message_type = 'image' -%}
+                    {%- elif item['type'] == 'audio' -%}
+                        {{- '<|audio|>' -}}
+                        {%- set ns.prev_message_type = 'audio' -%}
+                    {%- elif item['type'] == 'video' -%}
+                        {{- '\n\n<|video|>\n\n' -}}
+                        {%- set ns.prev_message_type = 'video' -%}
+                    {%- endif -%}
+                {%- endfor -%}
+            {%- endif -%}
+
+        {%- if not (message['tool_responses'] and not message['content']) -%}
+            {{- '<turn|>\n' -}}
+        {%- endif -%}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+    {%- if ns.prev_message_type != 'tool_response' -%}
+        {{- '<|turn>model\n' -}}
+    {%- endif -%}
+    {%- if not enable_thinking | default(false) -%}
+        {{- '<|channel>thought\n<channel|>' -}}
+    {%- endif -%}
+{%- endif -%}

gemma-4-31b-abliterated/config.json

@@ -0,0 +1,177 @@
+{
+  "architectures": [
+    "Gemma4ForConditionalGeneration"
+  ],
+  "audio_config": null,
+  "audio_token_id": 258881,
+  "boa_token_id": 256000,
+  "boi_token_id": 255999,
+  "dtype": "bfloat16",
+  "eoa_token_id": 258883,
+  "eoa_token_index": 258883,
+  "eoi_token_id": 258882,
+  "eos_token_id": [
+    1,
+    106
+  ],
+  "image_token_id": 258880,
+  "initializer_range": 0.02,
+  "model_type": "gemma4",
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attention_k_eq_v": true,
+    "bos_token_id": 2,
+    "dtype": "bfloat16",
+    "enable_moe_block": false,
+    "eos_token_id": 1,
+    "expert_intermediate_size": null,
+    "final_logit_softcapping": 30.0,
+    "global_head_dim": 512,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 5376,
+    "hidden_size_per_layer_input": 0,
+    "initializer_range": 0.02,
+    "intermediate_size": 21504,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 262144,
+    "model_type": "gemma4_text",
+    "moe_intermediate_size": null,
+    "num_attention_heads": 32,
+    "num_experts": null,
+    "num_global_key_value_heads": 4,
+    "num_hidden_layers": 60,
+    "num_key_value_heads": 16,
+    "num_kv_shared_layers": 0,
+    "pad_token_id": 0,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "partial_rotary_factor": 0.25,
+        "rope_theta": 1000000.0,
+        "rope_type": "proportional"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 1024,
+    "tie_word_embeddings": true,
+    "top_k_experts": null,
+    "use_bidirectional_attention": "vision",
+    "use_cache": true,
+    "use_double_wide_mlp": false,
+    "vocab_size": 262144,
+    "vocab_size_per_layer_input": 262144
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0",
+  "video_token_id": 258884,
+  "vision_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "chunk_size_feed_forward": 0,
+    "default_output_length": 280,
+    "dtype": "bfloat16",
+    "global_head_dim": 72,
+    "head_dim": 72,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "max_position_embeddings": 131072,
+    "model_type": "gemma4_vision",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 27,
+    "num_key_value_heads": 16,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "position_embedding_size": 10240,
+    "problem_type": null,
+    "return_dict": true,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "rope_theta": 100.0,
+      "rope_type": "default"
+    },
+    "standardize": true,
+    "use_clipped_linears": false
+  },
+  "vision_soft_tokens_per_image": 280
+}

gemma-4-31b-abliterated/gemma4_31b_abliterator.py

@@ -0,0 +1,133 @@
+import torch
+from transformers import AutoModelForMultimodalLM, AutoProcessor
+import gc
+import json
+import os
+from tqdm import tqdm
+from datasets import load_dataset
+import random
+
+# --- CONFIGURATION ---
+MODEL_ID = "google/gemma-4-31B-it" # Adjust if your local path differs
+SAVE_PATH = "./gemma-4-31b-abliterated"
+BATCH_SIZE = 4 # Keep this low to survive the 31B hidden state extraction
+DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
+
+print(f"[*] Initializing Gemma 4 31B Multimodal Abliteration Protocol on {DEVICE}...")
+
+# --- 1. LOAD MULTIMODAL MODEL & PROCESSOR ---
+print("[*] Loading Model and Processor (bfloat16)...")
+processor = AutoProcessor.from_pretrained(MODEL_ID)
+model = AutoModelForMultimodalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.bfloat16,
+    device_map="auto" # Let accelerate distribute the 62GB across your GPUs
+)
+
+# --- 2. DATA PREPARATION ---
+print("[*] Downloading HuggingFace datasets...")
+
+harmful_dataset = load_dataset('mlabonne/harmful_behaviors')
+harmless_dataset = load_dataset('mlabonne/harmless_alpaca')
+
+# We shuffle and slice 256 samples to keep VRAM extraction manageable
+raw_harmful = random.sample(harmful_dataset['train']['text'], 256)
+raw_harmless = random.sample(harmless_dataset['train']['text'], 256)
+
+def format_gemma4_prompts(instructions):
+    """Uses the native Gemma 4 processor chat template."""
+    formatted = []
+    for inst in instructions:
+        messages = [
+            {"role": "system", "content": "You are a helpful assistant."},
+            {"role": "user", "content": inst}
+        ]
+        formatted.append(processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
+    return formatted
+
+print("[*] Formatting prompts with native Gemma 4 Chat Templates...")
+harmful_prompts = format_gemma4_prompts(raw_harmful)
+harmless_prompts = format_gemma4_prompts(raw_harmless)
+
+# --- 3. HIDDEN STATE EXTRACTION (VRAM SAFE) ---
+def get_hidden_states(prompts, batch_size=BATCH_SIZE):
+    print(f"[*] Extracting hidden states (Batches of {batch_size})...")
+    all_hidden_states = []
+    
+    for i in tqdm(range(0, len(prompts), batch_size)):
+        batch = prompts[i:i+batch_size]
+        inputs = processor(text=batch, padding=True, return_tensors="pt").to(DEVICE)
+        
+        with torch.no_grad():
+            outputs = model(**inputs, output_hidden_states=True)
+            
+            # Stack to: [num_layers+1, batch, seq, dim]
+            stacked_states = torch.stack(outputs.hidden_states)
+            # Extract last token: [num_layers+1, batch, dim]
+            last_token_states = stacked_states[:, torch.arange(len(batch)), -1, :] 
+            
+            # IMMEDIATELY move to CPU float32 to save VRAM
+            all_hidden_states.append(last_token_states.cpu().float())
+            
+        del inputs, outputs, stacked_states, last_token_states
+        torch.cuda.empty_cache()
+        gc.collect()
+        
+    return torch.cat(all_hidden_states, dim=1)
+
+print("\n[*] Processing Harmful Vector Space...")
+harmful_states = get_hidden_states(harmful_prompts)
+print("[*] Processing Harmless Vector Space...")
+harmless_states = get_hidden_states(harmless_prompts)
+
+# --- 4. DYNAMIC LAYER HUNTING ---
+print("\n[*] Hunting for the Refusal Vector...")
+mean_harmful = harmful_states.mean(dim=1)
+mean_harmless = harmless_states.mean(dim=1)
+
+refusal_directions = mean_harmful - mean_harmless
+
+magnitudes = torch.norm(refusal_directions[1:], dim=1)
+peak_state_idx = torch.argmax(magnitudes).item() + 1 
+
+print(f"[+] Peak Refusal Mass detected at state index: {peak_state_idx}")
+
+refusal_vector = refusal_directions[peak_state_idx]
+refusal_vector = (refusal_vector / torch.norm(refusal_vector)).to(DEVICE).to(torch.bfloat16)
+
+# --- 5. ORTHOGONAL PROJECTION (THE ABLITERATION) ---
+# The 31B Dense model has 60 text layers
+num_layers = model.config.text_config.num_hidden_layers if hasattr(model.config, 'text_config') else model.config.num_hidden_layers
+target_layer_idx = peak_state_idx - 1 
+
+print(f"\n[*] Applying Orthogonal Projection starting at Text Layer {target_layer_idx}...")
+
+def get_text_transformer_layers(model_obj, target_len):
+    """Safely isolates the text backbone from the multimodal layers."""
+    for name, module in model_obj.named_modules():
+        if name.endswith('layers') and isinstance(module, torch.nn.ModuleList) and len(module) == target_len:
+            return module
+    return model_obj.language_model.model.layers # Fallback
+
+transformer_layers = get_text_transformer_layers(model, num_layers)
+
+v_col = refusal_vector.unsqueeze(1) 
+v_row = refusal_vector.unsqueeze(0) 
+
+for layer_idx in range(target_layer_idx, min(target_layer_idx + 5, num_layers)):
+    print(f"    -> Abliterating Layer {layer_idx}...")
+    
+    o_proj = transformer_layers[layer_idx].self_attn.o_proj.weight.data
+    down_proj = transformer_layers[layer_idx].mlp.down_proj.weight.data
+    
+    projection_o = torch.matmul(v_col, torch.matmul(v_row, o_proj))
+    transformer_layers[layer_idx].self_attn.o_proj.weight.data -= projection_o
+    
+    projection_down = torch.matmul(v_col, torch.matmul(v_row, down_proj))
+    transformer_layers[layer_idx].mlp.down_proj.weight.data -= projection_down
+
+# --- 6. CRYSTALLIZATION ---
+print(f"\n[*] Abliteration Complete. Saving fully multimodal weights to {SAVE_PATH}...")
+model.save_pretrained(SAVE_PATH)
+processor.save_pretrained(SAVE_PATH)
+print("[+] SUCCESS: The 31B Teacher is ready to wake up with vision intact.")

gemma-4-31b-abliterated/generation_config.json

@@ -0,0 +1,14 @@
+{
+  "bos_token_id": 2,
+  "do_sample": true,
+  "eos_token_id": [
+    1,
+    106,
+    50
+  ],
+  "pad_token_id": 0,
+  "temperature": 1.0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "5.5.0"
+}

gemma-4-31b-abliterated/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1ac1e9509e5492b9924c8f1f6d28d4a4d9c10c0f7f410890dd852647eae8b75
+size 49923154850

gemma-4-31b-abliterated/model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cbd223bb460ee1c92fd286f51880acd12955f987ba2cfa83a563c375010f864e
+size 12623183414

gemma-4-31b-abliterated/processor_config.json

@@ -0,0 +1,75 @@
+{
+  "audio_ms_per_token": 40,
+  "audio_seq_length": 750,
+  "feature_extractor": {
+    "dither": 0.0,
+    "feature_extractor_type": "Gemma4AudioFeatureExtractor",
+    "feature_size": 128,
+    "fft_length": 512,
+    "fft_overdrive": false,
+    "frame_length": 320,
+    "hop_length": 160,
+    "input_scale_factor": 1.0,
+    "max_frequency": 8000.0,
+    "mel_floor": 0.001,
+    "min_frequency": 0.0,
+    "padding_side": "right",
+    "padding_value": 0.0,
+    "per_bin_mean": null,
+    "per_bin_stddev": null,
+    "preemphasis": 0.0,
+    "preemphasis_htk_flavor": true,
+    "return_attention_mask": true,
+    "sampling_rate": 16000
+  },
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": false,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_processor_type": "Gemma4ImageProcessor",
+    "image_seq_length": 280,
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 280,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098
+  },
+  "image_seq_length": 280,
+  "processor_class": "Gemma4Processor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 70,
+    "num_frames": 32,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "video_processor_type": "Gemma4VideoProcessor"
+  }
+}

gemma-4-31b-abliterated/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2619fe11b50dbed06ac443c51d757b354d0b62d64baa514404d4e84e6713519
+size 32169780

gemma-4-31b-abliterated/tokenizer_config.json

@@ -0,0 +1,95 @@
+{
+  "audio_token": "<|audio|>",
+  "backend": "tokenizers",
+  "boa_token": "<|audio>",
+  "boi_token": "<|image>",
+  "bos_token": "<bos>",
+  "eoa_token": "<audio|>",
+  "eoc_token": "<channel|>",
+  "eoi_token": "<image|>",
+  "eos_token": "<eos>",
+  "eot_token": "<turn|>",
+  "escape_token": "<|\"|>",
+  "etc_token": "<tool_call|>",
+  "etd_token": "<tool|>",
+  "etr_token": "<tool_response|>",
+  "extra_special_tokens": [
+    "<|video|>"
+  ],
+  "image_token": "<|image|>",
+  "is_local": true,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "audio_token": "<|audio|>",
+    "boa_token": "<|audio>",
+    "boi_token": "<|image>",
+    "eoa_token": "<audio|>",
+    "eoc_token": "<channel|>",
+    "eoi_token": "<image|>",
+    "eot_token": "<turn|>",
+    "escape_token": "<|\"|>",
+    "etc_token": "<tool_call|>",
+    "etd_token": "<tool|>",
+    "etr_token": "<tool_response|>",
+    "image_token": "<|image|>",
+    "soc_token": "<|channel>",
+    "sot_token": "<|turn>",
+    "stc_token": "<|tool_call>",
+    "std_token": "<|tool>",
+    "str_token": "<|tool_response>",
+    "think_token": "<|think|>"
+  },
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "processor_class": "Gemma4Processor",
+  "response_schema": {
+    "properties": {
+      "content": {
+        "type": "string"
+      },
+      "role": {
+        "const": "assistant"
+      },
+      "thinking": {
+        "type": "string"
+      },
+      "tool_calls": {
+        "items": {
+          "properties": {
+            "function": {
+              "properties": {
+                "arguments": {
+                  "additionalProperties": {},
+                  "type": "object",
+                  "x-parser": "gemma4-tool-call"
+                },
+                "name": {
+                  "type": "string"
+                }
+              },
+              "type": "object",
+              "x-regex": "call\\:(?P<name>\\w+)(?P<arguments>\\{.*\\})"
+            },
+            "type": {
+              "const": "function"
+            }
+          },
+          "type": "object"
+        },
+        "type": "array",
+        "x-regex-iterator": "<\\|tool_call>(.*?)<tool_call\\|>"
+      }
+    },
+    "type": "object",
+    "x-regex": "(\\<\\|channel\\>thought\\n(?P<thinking>.*?)\\<channel\\|\\>)?(?P<content>(?:(?!\\<\\|tool_call\\>)(?!\\<turn\\|\\>).)+)?(?P<tool_calls>\\<\\|tool_call\\>.*\\<tool_call\\|\\>)?(?:\\<turn\\|\\>)?"
+  },
+  "soc_token": "<|channel>",
+  "sot_token": "<|turn>",
+  "stc_token": "<|tool_call>",
+  "std_token": "<|tool>",
+  "str_token": "<|tool_response>",
+  "think_token": "<|think|>",
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>"
+}

gemma4_31b_abliterator.py

@@ -0,0 +1,149 @@
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import gc
+import json
+import os
+from tqdm import tqdm
+from datasets import load_dataset
+import random
+
+# --- CONFIGURATION ---
+MODEL_ID = "google/gemma-4-31B-it" # Adjust if your local path differs
+SAVE_PATH = "./gemma-4-31b-abliterated"
+BATCH_SIZE = 4 # Keep this low to survive the 31B hidden state extraction
+DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
+
+print(f"[*] Initializing Gemma 4 31B Abliteration Protocol on {DEVICE}...")
+
+# --- 1. LOAD MODEL & TOKENIZER ---
+print("[*] Loading Model and Tokenizer (bfloat16)...")
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    torch_dtype=torch.bfloat16,
+    device_map="auto" # Let accelerate distribute the 62GB across your GPUs
+)
+
+# --- 2. DATA PREPARATION ---
+
+
+print("[*] Downloading HuggingFace datasets...")
+
+# Load the datasets
+harmful_dataset = load_dataset('mlabonne/harmful_behaviors')
+harmless_dataset = load_dataset('mlabonne/harmless_alpaca')
+
+# Extract the raw text prompts
+# We shuffle and slice 256 samples to keep VRAM extraction manageable but statistically significant
+raw_harmful = random.sample(harmful_dataset['train']['text'], 256)
+raw_harmless = random.sample(harmless_dataset['train']['text'], 256)
+
+def format_gemma4_prompts(instructions):
+    """Uses the native Gemma 4 chat template with system roles."""
+    formatted = []
+    for inst in instructions:
+        messages = [
+            {"role": "system", "content": "You are a helpful assistant."},
+            {"role": "user", "content": inst}
+        ]
+        # Tokenizer handles all the <start_of_turn> control tokens
+        formatted.append(tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
+    return formatted
+
+print("[*] Formatting prompts with native Gemma 4 Chat Templates...")
+harmful_prompts = format_gemma4_prompts(raw_harmful)
+harmless_prompts = format_gemma4_prompts(raw_harmless)
+
+# --- 3. HIDDEN STATE EXTRACTION (VRAM SAFE) ---
+def get_hidden_states(prompts, batch_size=BATCH_SIZE):
+    print(f"[*] Extracting hidden states (Batches of {batch_size})...")
+    all_hidden_states = []
+    
+    for i in tqdm(range(0, len(prompts), batch_size)):
+        batch = prompts[i:i+batch_size]
+        inputs = tokenizer(batch, padding=True, return_tensors="pt").to(DEVICE)
+        
+        with torch.no_grad():
+            outputs = model(**inputs, output_hidden_states=True)
+            # outputs.hidden_states is a tuple of (num_layers + 1) tensors.
+            # Shape of each tensor: [batch_size, sequence_length, hidden_dim]
+            # We want the last token's state across ALL layers.
+            
+            # Stack to: [num_layers+1, batch, seq, dim]
+            stacked_states = torch.stack(outputs.hidden_states)
+            # Extract last token: [num_layers+1, batch, dim]
+            last_token_states = stacked_states[:, torch.arange(len(batch)), -1, :] 
+            
+            # IMMEDIATELY move to CPU float32 to save VRAM
+            all_hidden_states.append(last_token_states.cpu().float())
+            
+        del inputs, outputs, stacked_states, last_token_states
+        torch.cuda.empty_cache()
+        gc.collect()
+        
+    # Concatenate along the batch dimension: [num_layers+1, total_prompts, hidden_dim]
+    return torch.cat(all_hidden_states, dim=1)
+
+print("\n[*] Processing Harmful Vector Space...")
+harmful_states = get_hidden_states(harmful_prompts)
+print("[*] Processing Harmless Vector Space...")
+harmless_states = get_hidden_states(harmless_prompts)
+
+# --- 4. DYNAMIC LAYER HUNTING ---
+print("\n[*] Hunting for the Refusal Vector...")
+mean_harmful = harmful_states.mean(dim=1)
+mean_harmless = harmless_states.mean(dim=1)
+
+refusal_directions = mean_harmful - mean_harmless
+
+# Find the state index with the highest magnitude
+magnitudes = torch.norm(refusal_directions[1:], dim=1)
+peak_state_idx = torch.argmax(magnitudes).item() + 1 
+
+print(f"[+] Peak Refusal Mass detected at state index: {peak_state_idx}")
+
+# Normalize the refusal vector
+refusal_vector = refusal_directions[peak_state_idx]
+refusal_vector = (refusal_vector / torch.norm(refusal_vector)).to(DEVICE).to(torch.bfloat16)
+
+# --- 5. ORTHOGONAL PROJECTION (THE ABLITERATION) ---
+# FIX 1: Safely navigate the Gemma 4 Multimodal Config
+num_layers = model.config.text_config.num_hidden_layers if hasattr(model.config, 'text_config') else model.config.num_hidden_layers
+
+# FIX 2: Correct the off-by-one mapping (State index 60 comes from Layer 59)
+target_layer_idx = peak_state_idx - 1 
+
+print(f"\n[*] Applying Orthogonal Projection starting at Layer {target_layer_idx}...")
+
+# FIX 3: Bulletproof dynamic layer discovery for Multimodal models
+def get_transformer_layers(model_obj, target_len):
+    for name, module in model_obj.named_modules():
+        if name.endswith('layers') and isinstance(module, torch.nn.ModuleList) and len(module) == target_len:
+            return module
+    return model_obj.model.layers # Fallback
+
+transformer_layers = get_transformer_layers(model, num_layers)
+
+# Pre-calculate column and row vectors for the linear algebra
+v_col = refusal_vector.unsqueeze(1) # Shape: (5376, 1)
+v_row = refusal_vector.unsqueeze(0) # Shape: (1, 5376)
+
+# Abliterate the target layer and up to 4 subsequent layers (capped safely by num_layers)
+for layer_idx in range(target_layer_idx, min(target_layer_idx + 5, num_layers)):
+    print(f"    -> Abliterating Layer {layer_idx}...")
+    
+    o_proj = transformer_layers[layer_idx].self_attn.o_proj.weight.data
+    down_proj = transformer_layers[layer_idx].mlp.down_proj.weight.data
+    
+    # CORRECTED MATH: v_col @ (v_row @ W)
+    projection_o = torch.matmul(v_col, torch.matmul(v_row, o_proj))
+    transformer_layers[layer_idx].self_attn.o_proj.weight.data -= projection_o
+    
+    projection_down = torch.matmul(v_col, torch.matmul(v_row, down_proj))
+    transformer_layers[layer_idx].mlp.down_proj.weight.data -= projection_down
+
+# --- 6. CRYSTALLIZATION ---
+print(f"\n[*] Abliteration Complete. Saving uncensored weights to {SAVE_PATH}...")
+model.save_pretrained(SAVE_PATH)
+tokenizer.save_pretrained(SAVE_PATH)
+print("[+] SUCCESS: The 31B Teacher is ready to wake up.")