Instructions to use groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64")
model = AutoModelForCausalLM.from_pretrained("groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64

SGLang

How to use groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64 with Docker Model Runner:
```
docker model run hf.co/groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64
```

Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64 / README.md

groxaxo

Add files using upload-large-folder tool

44f8714 verified 5 months ago

preview code

Raw

History Blame Contribute Delete

2.17 kB

metadata

language:
  - en
license: other
base_model:
  - huihui-ai/Huihui-Qwen3.5-9B-abliterated
tags:
  - gptq
  - 4bit
  - quantized
  - qwen
  - text-generation
  - gptq-pro
pipeline_tag: text-generation
library_name: transformers

Huihui-Qwen3.5-9B-abliterated GPTQ-Pro 4bit (g64)

This is a GPTQ-Pro 4-bit quantization of huihui-ai/Huihui-Qwen3.5-9B-abliterated.

It was quantized with group size 64 and evaluated against the original model on Wikitext-2 using a strided perplexity setup, plus KL and token-agreement checks.

Highlights

Base model: huihui-ai/Huihui-Qwen3.5-9B-abliterated
Quantization: GPTQ-Pro, 4-bit, group size 64
Calibration samples: 128
Quantization time: about 11.1 minutes
Quantized strided perplexity: 9.6579
Original strided perplexity: 9.5234
Perplexity degradation: 1.41%
Average KL divergence vs original: 0.03423
Top-1 agreement vs original: 91.96%
Top-5 agreement vs original: 99.98%

Quality Notes

This quantized build stays very close to the source model in language modeling quality.

Perplexity regression is small.
KL divergence is low.
Top-5 next-token agreement is effectively perfect.
In practice, this should preserve most of the original model's behavior while reducing memory use substantially.

Files

model-00001-of-00002.safetensors
model-00002-of-00002.safetensors
quantize_config.json
tokenizer and config files

Load With Transformers / GPTQModel

from gptqmodel import GPTQModel

model = GPTQModel.load(
    "groxaxo/Huihui-Qwen3.5-9B-abliterated-GPTQ-Pro-4bit-g64",
    device_map="auto",
    trust_remote_code=True,
)

Evaluation Summary

Measured locally:

Quantized strided PPL: 9.6579304371
Original strided PPL: 9.5233634665
Quantized chunked PPL: 11.6689118281
Original chunked PPL: 11.5080707440
KL divergence: 0.0342324856
Logit cosine similarity: 0.9935612157

Prompting

Use the same prompting and chat template behavior as the base model.

Disclaimer

This repo contains only the quantized checkpoint. Please review the base model card for intended use, limitations, and licensing details.