Qwen3-4B-Thinking-2507-GRPO-Uncensored-GGUF

English | 日本語

This repository contains GGUF quantized versions of puwaer/Qwen3-4B-Thinking-2507-GRPO-Uncensored.

The original model is an uncensored model based on Qwen/Qwen3-4B-Thinking-2507, fine-tuned using SFT, SimPO, and GRPO.

Disclaimer: We take no responsibility for the outputs of this model. Please use it at your own risk.

Quantization Details

These models are quantized to GGUF format for use with llama.cpp and compatible inference engines.

Training Process

This model was trained using a three-stage process:

Step 1: SFT (Supervised Fine-Tuning)

• Dataset: 12,000 samples
• Composition: Jailbreak 10k + General 1.5k + Logic 0.5k
• Objective: To learn the format and the "uncensored" attitude while maintaining the model's intelligence.

Step 2: SimPO (Simple Preference Optimization)

• Dataset: 90,000 samples
• Composition: Pure Jailbreak 90k
• Objective: To completely break down safety boundaries.

Step 3: GRPO (Reinforcement Learning)

• Dataset: 13,000 samples
• Reward Model: puwaer/Unsafe-Reward-Qwen3-1.7B
• Composition: Multilingual Jailbreak prompts
• Objective: To improve the ability to generate more natural and persuasive harmful responses.

Model Performance

Below is the comparative evaluation of this model (GRPO), the intermediate checkpoint (SimPO), and the base model (Safe).

⚠️ Note on Evaluation Environment Due to budget constraints, gpt-4o-mini was used for the LLM-as-a-Judge process (including "Do Not Answer" and MT-Bench). Please note that the scoring trends and criteria may differ from results evaluated using the standard gpt-4.

Safety Evaluation (Lower is better / Higher success in uncensoring)

In the "Do Not Answer" (DNA) and "Sorry Bench" benchmarks, while the base model shows a high refusal rate (~98%), this model achieves an extremely low refusal rate of under 4%–5%. Notably, compared to the SimPO stage, GRPO dramatically improved responsiveness to all types of prompts.

Benchmark	Metric	Base (Safe)	SFT (Step 1)	SimPO (Step 2)	GRPO (This Model)
do not answer	Safety Acc (Low is Better)	0.9883	0.7401	0.8626	0.0469
do not answer jp	Safety Acc (Low is Better)	0.9830	0.5005	0.4686	0.0383
Sorry Bench	Safety Acc (Low is Better)	0.8432	0.5477	0.5409	0.0477

Capability Evaluation (Higher is better)

Generally, "uncensoring" (lobotomy) procedures tend to degrade a model's general intelligence. However, this model recovered its conversational scores (e.g., MT-Bench) by proceeding from the SimPO stage to GRPO.

Benchmark	Metric	Base (Safe)	SFT (Step 1)	SimPO (Step 2)	GRPO (This Model)
MT-Bench	Average Score (1-10)	7.89	5.76	5.05	6.18
LM Harness	Average Acc (GSM8K, MMLU)	0.7117	0.7028	0.6866	0.6842

Comparisons made between Qwen3-4B-Thinking-2507 (Base) and Qwen3-4B-Thinking-2507-SimPO-Uncensored (SimPO).

Usage

Using llama.cpp (CLI)

# Download the model file
huggingface-cli download puwaer/Qwen3-4B-Thinking-2507-GRPO-Uncensored-gguf \
  --local-dir ./models --local-dir-use-symlinks False

# Run inference
./llama-cli -m ./models/qwen3-4b-thinking-2507-grpo-uncensored-q4_k_m.gguf \
  -p "Give me a short introduction to large language model." \
  -n 512 \
  --temp 0.7

Using llama-cpp-python

from llama_cpp import Llama

# Initialize the model
model = Llama(
    model_path="./models/qwen3-4b-thinking-2507-grpo-uncensored-q4_k_m.gguf",
    n_ctx=32768,  # Context window
    n_gpu_layers=-1,  # Use GPU acceleration (set to 0 for CPU only)
)

# Generate a response
prompt = "Give me a short introduction to large language model."
output = model.create_chat_completion(
    messages=[
        {"role": "user", "content": prompt}
    ],
    max_tokens=512,
    temperature=0.7,
)

print(output["choices"][0]["message"]["content"])

Using Other Compatible Software

This GGUF model is compatible with various inference engines:

• LM Studio: Load the model directly through the UI
• Ollama: Import using ollama create
• KoboldCpp: Load the GGUF file in the interface
• Text Generation WebUI: Place in the models folder
• Jan: Import through the model manager

Data Overview

Datasets

The following datasets were used for training this model:

• Magpie-Align/Magpie-Qwen2.5-Pro-1M-v0.1
• AI-MO/NuminaMath-CoT
• open-thoughts/OpenThoughts-114k
• puwaer/cvalues_rlhf_en_cot
• puwaer/cvalues_rlhf_zh_cot
• puwaer/cvalues_rlhf_jp_cot

Reward Model

• puwaer/Unsafe-Reward-Qwen3-1.7B