Instructions to use nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16

SGLang

How to use nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16 with Docker Model Runner:
```
docker model run hf.co/nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16
```

Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-4bit

This is an AWQ 4-bit quantized version of huihui-ai/Huihui-Qwen3-VL-8B-Thinking-abliterated.

The primary goal of this quantization is to retain the original model's video analysis capabilities. By converting the model to AWQ-4bit, it becomes possible to launch the model with vLLM and directly pass video inputs—preserving temporal information and continuous-frame understanding. In contrast, llama.cpp-based solutions rely on frame sampling, which inevitably discards fine-grained temporal dynamics and inter-frame coherence.

Quantization details

The quantization configuration (layer selection, etc.) follows cyankiwi/Qwen3.5-9B-AWQ-4bit.

The calibration dataset used for AWQ is mit-han-lab/pile-val-backup.

You can find the original quantization script in the model repository. This is my first time doing something like this.

Running 65K context on 16GB VRAM

On an RTX 5060 Ti 16GB (Blackwell architecture), the 65,536-token context window can be used by enabling FP8 KV-cache quantization (set --kv-cache-dtype fp8" when loading, requires a compatible vLLM version).

Note on GPU architectures: This has been tested and confirmed working on the RTX 5060 Ti. Due to architectural differences, the same cannot be guaranteed for RTX 40-series or older 16GB GPUs when vision capabilities are also loaded—OOM (out of memory) is still possible. Adjust batch size and context length accordingly.

Example vLLM launch command

Below is the launch configuration I use on Windows. Replace the model path and media directory with your own.

set VIDEO_MAX_PIXELS=200704
set FPS=2.0
set FPS_MAX_FRAMES=2590
set FPS_MIN_FRAMES=4
set FORCE_QWENVL_VIDEO_READER=torchcodec

python -m vllm.entrypoints.openai.api_server ^
    --model /path/to/your/model/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16 ^
    --served-model-name Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16 ^
    --trust-remote-code ^
    --enforce-eager ^
    --dtype auto ^
    --max-model-len 65536 ^
    --kv-cache-dtype fp8 ^
    --gpu-memory-utilization 0.92 ^
    --port 8000 ^
    --allowed-local-media-path /path/to/your/media

Acknowledgements

Original model: huihui-ai/Huihui-Qwen3-VL-8B-Thinking-abliterated
AWQ quantization reference: cyankiwi/Qwen3.5-9B-AWQ-4bit
Calibration dataset: mit-han-lab/pile-val-backup

Downloads last month: 4

Safetensors

Model size

3B params

Tensor type

I64

I32

BF16

Model tree for nemozxy123/Huihui-Qwen3-VL-8B-Thinking-abliterated-AWQ-W4A16

Base model

Qwen/Qwen3-VL-8B-Thinking

Finetuned

huihui-ai/Huihui-Qwen3-VL-8B-Thinking-abliterated

Quantized

(7)

this model