Instructions to use abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw")
model = AutoModelForMultimodalLM.from_pretrained("abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw

SGLang

How to use abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw with Docker Model Runner:
```
docker model run hf.co/abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw
```

Qwen3.5-9B-Abliterated-GDN-Hybrid — EXL3 6.0bpw

EXL3 (exllamav3) quantization of huihui-ai/Huihui-Qwen3.5-9B-abliterated at 6.0 bits per weight.

Quantized by ABC&D System Inc for production real-time inference on a single RTX 3090.

Model Details

Property	Value
Base model	Qwen/Qwen3.5-9B
Abliteration	huihui-ai via remove-refusals-with-transformers
Parameters	9B dense (all active per token)
Architecture	`Qwen3_5ForConditionalGeneration` — GatedDeltaNet + Attention hybrid
Quantization	EXL3 6.0bpw, head bits 6, codebook mul1
Calibration	250 rows × 2048 cols
VRAM loaded	~5.6GB model + KV cache
Total size on disk	~8.5GB
License	Apache 2.0 (Alibaba Cloud)

Architecture — Why This Model Is Different

Qwen3.5 is not a standard transformer. It uses a hybrid attention design that alternates between two fundamentally different attention mechanisms:

GatedDeltaNet Linear Attention (layers 0, 1, 2, 4, 5, 6, 8, 9, 10, ...)

O(n) complexity instead of O(n²) — scales linearly with sequence length
Recurrent state via conv1d kernel (kernel dim 4)
16 key heads × 128 dim, 32 value heads × 128 dim
Gated with learned input/output projections + beta decay

Full Quadratic Attention (layers 3, 7, 11, 15, 19, 23, 27, 31)

Standard grouped-query attention (16 heads, 4 KV heads, head dim 256)
RoPE with partial rotary factor 0.25, theta 10M
Provides global context every 4th layer

Layer Pattern

[linear, linear, linear, FULL, linear, linear, linear, FULL, ...] × 4 = 32 layers

24 GatedDeltaNet layers + 8 full attention layers. The linear layers handle local/sequential patterns efficiently while the full attention layers provide global context anchoring.

Multi-Token Prediction (MTP)

The model includes a 1-layer MTP head for predicting multiple next tokens simultaneously, enabling faster inference throughput than standard single-token autoregressive decoding.

Text-Only Weights

Despite the architecture string Qwen3_5ForConditionalGeneration (which Alibaba uses for all Qwen3.5 variants including VL), this model contains zero vision weights. No ViT encoder, no image projector. The architecture class name reflects the MTP head and hybrid design, not multimodal capability.

Quantization Details

Quantized with exllamav3 v1.1.0.

Format: EXL3 with trellis-based weight encoding
Bits per weight: 6.0 (all linear layers uniform)
Head bits: 6
Codebook: mul1
Why EXL3 over GGUF: Native tensor core utilization on Ampere GPUs (RTX 3090/4090). GGUF dequantizes to FP16 at runtime, wasting tensor cores. EXL3 operates natively, yielding ~1.5x faster inference at equivalent quality.

Recommended Serving

TabbyAPI (recommended)

# config.yml
network:
  host: 0.0.0.0
  port: 11435

model:
  model_dir: /path/to/models
  model_name: Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw
  max_seq_len: 4096
  gpu_split: [24]
  cache_mode: FP16

CUDA_VISIBLE_DEVICES=0 python3 main.py --config config.yml

Important: Disable Thinking Mode

Qwen3.5 has a built-in chain-of-thought reasoning mode that is on by default. For real-time applications (voice, chat, API), you must disable it or the model will consume your entire token budget on internal reasoning before producing visible output.

Add to your system prompt:

/no_think

Or set enable_thinking: false in your chat template parameters.

Note on exllamav3 Raw API

The raw exllamav3 Generator Python API may fail to initialize GatedDeltaNet recurrent state (conv_state on meta device error). TabbyAPI handles this initialization correctly. Use TabbyAPI as the serving layer rather than calling exllamav3 directly.

Hardware Requirements

Config	VRAM	Notes
Minimum	~8GB	Model only, minimal KV cache
Recommended	16GB+	Model + FP16 KV cache for 4096 context
Tested on	RTX 3090 24GB	5.6GB model, 18.4GB free for cache

Performance (tested on RTX 3090)

Response latency: 300–500ms per completion
Suitable for real-time voice pipelines with STT overhead
First request ~14ms (cache warmup, empty response)

Use Cases

This quantization was built for low-latency, single-user dedicated inference — specifically real-time voice AI over SIP/phone calls. The abliteration ensures zero self-censorship in commercial conversation contexts.

Good for:

Real-time voice assistants and phone agents
Single-GPU dedicated inference servers
Applications requiring uncensored, natural conversational output
Latency-sensitive pipelines (sub-600ms budget)

Not ideal for:

Multi-user batched serving (use AWQ + vLLM instead)
Vision/multimodal tasks (no vision weights)
Applications requiring safety guardrails (abliterated model)

Credits

Base model: Qwen/Qwen3.5-9B by Alibaba Cloud (Apache 2.0)
Abliteration: huihui-ai via remove-refusals-with-transformers
EXL3 quantization: ABC&D System Inc using exllamav3 by turboderp
Serving: TabbyAPI by theroyallab

Usage Warnings

This model has had its safety filtering surgically removed (abliterated). It will comply with any instruction without refusal. Users are solely responsible for ensuring their use complies with applicable laws and ethical standards. See huihui-ai's original warnings for full details.

Downloads last month: 27

Safetensors

Model size

5B params

Tensor type

BF16

F16

I16

F32

Model tree for abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw

Base model

Qwen/Qwen3.5-9B-Base

Finetuned

Qwen/Qwen3.5-9B

Finetuned

huihui-ai/Huihui-Qwen3.5-9B-abliterated

Quantized

(16)

this model