jorge-erdb/gpt-oss-20b-Derestricted-4bit-GGUF

gpt-oss-20b-Derestricted-GGUF — 4-bit

4-bit GGUF quantizations of ArliAI/gpt-oss-20b-Derestricted, a Norm-Preserving Biprojected Abliterated version of openai/gpt-oss-20b.

Quantization Details

Detail	Value
Quant types	IQ4_NL, Q4_K_M
Quantized by	jorge-erdb
Method	llama.cpp with importance matrix
Importance matrix	bartowski's imatrix calibration dataset
Source model	ArliAI/gpt-oss-20b-Derestricted (BF16)

Quant	Best for	Notes
IQ4_NL	CUDA / CPU	Non-linear, slightly better quality per bit
Q4_K_M	CUDA / CPU / Metal	Linear, best Metal compatibility

Download

pip install -U "huggingface_hub[cli]"

# IQ4_NL (non-linear, best for CUDA/CPU)
huggingface-cli download jorge-erdb/gpt-oss-20b-Derestricted-4bit-GGUF --include "*IQ4_NL.gguf" --local-dir ./

# Q4_K_M (linear, Metal-friendly)
huggingface-cli download jorge-erdb/gpt-oss-20b-Derestricted-4bit-GGUF --include "*Q4_K_M.gguf" --local-dir ./

Apple Metal Backend Warning

IQ4_NL is a non-linear quantization format. It performs sub-optimally on Apple's Metal backend due to the lack of native support for non-linear dequantization kernels. If you are running on an Apple Silicon Mac with GPU offloading via Metal, you will likely experience:

• Slower inference compared to linear quants of similar size (e.g., Q4_K_M)
• No speed benefit from the ARM weight repacking that IQ4_NL supports on CPU

If you're on Apple Metal, use the Q4_K_M quant from this repo instead. For higher precision options, see ArliAI's repo.

Credits

• Quantization: jorge-erdb
• Importance matrix: bartowski — imatrix calibration dataset
• Derestriction: Arli AI — Norm-Preserving Biprojected Abliteration
• Base model: OpenAI — gpt-oss-20b

---

Arli AI

gpt-oss-20b-Derestricted

After the initial success of GLM-4.5-Air-Derestricted, we thought it would be interesting to try and Derestrict one of the most famously restrictive model which is gpt-oss-20b.

gpt-oss-20b-Derestricted is a Derestricted version of openai/gpt-oss-20b, created by Arli AI.

Our goal with this release is to provide a version of the model that removed refusal behaviors while maintaining the high-performance reasoning of the original gpt-oss-20b. This is unlike regular abliteration which often inadvertently "lobotomizes" the model.

Methodology: Norm-Preserving Biprojected Abliteration

To achieve this, Arli AI utilized Norm-Preserving Biprojected Abliteration, a refined technique pioneered by Jim Lai (grimjim). You can read the full technical breakdown in this article.

Why this matters:

Standard abliteration works by simply subtracting a "refusal vector" from the model's weights. While this works to uncensor a model, it is mathematically unprincipled. It alters the magnitude (or "loudness") of the neurons, destroying the delicate feature norms the model learned during training. This damage is why many uncensored models suffer from degraded logic or hallucinations.

How Norm-Preserving Biprojected Abliteration fixes it:

This model was modified using a three-step approach that removes refusals without breaking the model's brain:

1. Biprojection (Targeting): We refined the refusal direction to ensure it is mathematically orthogonal to "harmless" directions. This ensures that when we cut out the refusal behavior, we do not accidentally cut out healthy, harmless concepts.
2. Decomposition: Instead of a raw subtraction, we decomposed the model weights into Magnitude and Direction.
3. Norm-Preservation: We removed the refusal component solely from the directional aspect of the weights, then recombined them with their original magnitudes.

The Result:

By preserving the weight norms, we maintain the "importance" structure of the neural network. Benchmarks suggest that this method avoids the "Safety Tax"—not only effectively removing refusals but potentially improving reasoning capabilities over the baseline, as the model is no longer wasting compute resources on suppressing its own outputs.

In fact, you may find surprising new knowledge and capabilities that the original model does not initially expose.

For gpt-oss-20b, we found that it might still occasionally try and suppress requests but eventually reason that it is unneccesary.

---

Original model card:

Try gpt-oss · Guides · Model card · OpenAI blog

Welcome to the gpt-oss series, OpenAI’s open-weight models designed for powerful reasoning, agentic tasks, and versatile developer use cases.

We’re releasing two flavors of these open models:

• gpt-oss-120b — for production, general purpose, high reasoning use cases that fit into a single 80GB GPU (like NVIDIA H100 or AMD MI300X) (117B parameters with 5.1B active parameters)
• gpt-oss-20b — for lower latency, and local or specialized use cases (21B parameters with 3.6B active parameters)

Both models were trained on our harmony response format and should only be used with the harmony format as it will not work correctly otherwise.

This model card is dedicated to the smaller gpt-oss-20b model. Check out gpt-oss-120b for the larger model.

Highlights

• Permissive Apache 2.0 license: Build freely without copyleft restrictions or patent risk—ideal for experimentation, customization, and commercial deployment.
• Configurable reasoning effort: Easily adjust the reasoning effort (low, medium, high) based on your specific use case and latency needs.
• Full chain-of-thought: Gain complete access to the model’s reasoning process, facilitating easier debugging and increased trust in outputs. It’s not intended to be shown to end users.
• Fine-tunable: Fully customize models to your specific use case through parameter fine-tuning.
• Agentic capabilities: Use the models’ native capabilities for function calling, web browsing, Python code execution, and Structured Outputs.
• MXFP4 quantization: The models were post-trained with MXFP4 quantization of the MoE weights, making gpt-oss-120b run on a single 80GB GPU (like NVIDIA H100 or AMD MI300X) and the gpt-oss-20b model run within 16GB of memory. All evals were performed with the same MXFP4 quantization.

---

Inference examples

Transformers

You can use gpt-oss-120b and gpt-oss-20b with Transformers. If you use the Transformers chat template, it will automatically apply the harmony response format. If you use model.generate directly, you need to apply the harmony format manually using the chat template or use our openai-harmony package.

To get started, install the necessary dependencies to setup your environment:

pip install -U transformers kernels torch 

Once, setup you can proceed to run the model by running the snippet below:

from transformers import pipeline
import torch

model_id = "openai/gpt-oss-20b"

pipe = pipeline(
    "text-generation",
    model=model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {"role": "user", "content": "Explain quantum mechanics clearly and concisely."},
]

outputs = pipe(
    messages,
    max_new_tokens=256,
)
print(outputs[0]["generated_text"][-1])

Alternatively, you can run the model via Transformers Serve to spin up a OpenAI-compatible webserver:

transformers serve
transformers chat localhost:8000 --model-name-or-path openai/gpt-oss-20b

Learn more about how to use gpt-oss with Transformers.

vLLM

vLLM recommends using uv for Python dependency management. You can use vLLM to spin up an OpenAI-compatible webserver. The following command will automatically download the model and start the server.

uv pip install --pre vllm==0.10.1+gptoss \
    --extra-index-url https://wheels.vllm.ai/gpt-oss/ \
    --extra-index-url https://download.pytorch.org/whl/nightly/cu128 \
    --index-strategy unsafe-best-match

vllm serve openai/gpt-oss-20b

Learn more about how to use gpt-oss with vLLM.

PyTorch / Triton

To learn about how to use this model with PyTorch and Triton, check out our reference implementations in the gpt-oss repository.

Ollama

If you are trying to run gpt-oss on consumer hardware, you can use Ollama by running the following commands after installing Ollama.

# gpt-oss-20b
ollama pull gpt-oss:20b
ollama run gpt-oss:20b

Learn more about how to use gpt-oss with Ollama.

LM Studio

If you are using LM Studio you can use the following commands to download.

# gpt-oss-20b
lms get openai/gpt-oss-20b

Check out our awesome list for a broader collection of gpt-oss resources and inference partners.

---

Download the model

You can download the model weights from the Hugging Face Hub directly from Hugging Face CLI:

# gpt-oss-20b
huggingface-cli download openai/gpt-oss-20b --include "original/*" --local-dir gpt-oss-20b/
pip install gpt-oss
python -m gpt_oss.chat model/

Reasoning levels

You can adjust the reasoning level that suits your task across three levels:

• Low: Fast responses for general dialogue.
• Medium: Balanced speed and detail.
• High: Deep and detailed analysis.

The reasoning level can be set in the system prompts, e.g., "Reasoning: high".

Tool use

The gpt-oss models are excellent for:

• Web browsing (using built-in browsing tools)
• Function calling with defined schemas
• Agentic operations like browser tasks

Fine-tuning

Both gpt-oss models can be fine-tuned for a variety of specialized use cases.

This smaller model gpt-oss-20b can be fine-tuned on consumer hardware, whereas the larger gpt-oss-120b can be fine-tuned on a single H100 node.

Citation

@misc{openai2025gptoss120bgptoss20bmodel,
      title={gpt-oss-120b & gpt-oss-20b Model Card}, 
      author={OpenAI},
      year={2025},
      eprint={2508.10925},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2508.10925}, 
}