Instructions to use Minachist/Qwen3.6-27B-INT8-Autoround-V2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Minachist/Qwen3.6-27B-INT8-Autoround-V2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Minachist/Qwen3.6-27B-INT8-Autoround-V2")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Minachist/Qwen3.6-27B-INT8-Autoround-V2")
model = AutoModelForMultimodalLM.from_pretrained("Minachist/Qwen3.6-27B-INT8-Autoround-V2", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Minachist/Qwen3.6-27B-INT8-Autoround-V2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Minachist/Qwen3.6-27B-INT8-Autoround-V2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Minachist/Qwen3.6-27B-INT8-Autoround-V2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Minachist/Qwen3.6-27B-INT8-Autoround-V2

SGLang

How to use Minachist/Qwen3.6-27B-INT8-Autoround-V2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Minachist/Qwen3.6-27B-INT8-Autoround-V2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Minachist/Qwen3.6-27B-INT8-Autoround-V2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Minachist/Qwen3.6-27B-INT8-Autoround-V2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Minachist/Qwen3.6-27B-INT8-Autoround-V2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Minachist/Qwen3.6-27B-INT8-Autoround-V2 with Docker Model Runner:
```
docker model run hf.co/Minachist/Qwen3.6-27B-INT8-Autoround-V2
```

Qwen3.6-27B-INT8-Autoround-V2

This is an unofficial mixed-precision INT8 quantized version of Qwen3.6-27B. It was created using AutoRound.

Quantization details

Some layers are not quantized as quantizing them causes a heavy downgrade, or AutoRound simply doesn't support quantizing them.

Precision	Layer Count	Target Layers
INT8 (group_size=-1, symmetric)	410	`gate/up/down_proj`, `q/k/out_proj`, `in_proj_qkv`, `in_proj_z`, `mtp`
FP16 (unquantized)	98	`in_proj_a` / `in_proj_b` of linear_attn, `embed_tokens`, `visual`

AutoRound doesn't support quantizing in_proj_a and in_proj_b.
Calibration configuration: iters=250, nsamples=1024, batch_size=4, with gradient_accumulate_steps=2.

You need to change a line in utils.py of the AutoRound library

from:

layer_config[match] = val

to: layer_config[match] = copy.deepcopy(val)

in order to quantize linear_attn.

To whoever wants to open a pull request upstream: if you believe this is actually a bug, I encourage you to do so.

Evaluation Results

1. KL Divergence (KLD)

The output distribution of the quantized model was compared directly to that of the base model across 122,640 tokens.

Metric @ctx 512	Value
Mean KLD	0.00182 ± 0.00014
Median KLD	0.000672
P90 / P95 / P99 KLD	0.00232 / 0.00349 / 0.00931
P99.9 / Max KLD	0.0668 / 9.156
Top-1 Agreement	98.32%
PPL (Base → Quantized)	7.7415 → 7.7442
ln(PPL Ratio)	0.000343

Evaluation set: WikiText-2 (test), 240 sequences × 512 tokens = 122,640 scored next-token positions. Values are in nats.

I heard quantizing linear_attn impacts the performance quite badly, but that's not the case here. Maybe not quantizing in_proj_a / in_proj_b is the reason.

2. Benchmarks

Measured via lm-evaluation-harness (0.4.12) using the vLLM OpenAI-compatible API.

Settings: 0-shot, temperature=0.6, top_p=0.95, max generation tokens 65,536.
Comparison baseline: Official metrics from the Qwen3.6-27B model card.

Benchmark	This Model	Base Model (Official)	Difference	n
MMLU-Redux	91.9% ± 1.6	93.5	-1.6	285
MMLU-Pro	83.8% ± 1.8	86.2	-2.4	420

Performance drops in knowledge-focused benchmarks (MMLU-Redux, MMLU-Pro) are minor and lie mostly within the standard error. I wanted to run more benchmarks, but it's quite hot here and AC doesn't seem to work well. You have to do it yourself.

How to Use

This model has been tested using the latest vllm-openai:nightly.

On a 2x RTX 3090 setup, you can get 204,800 context without kv-cache quantization.

You can append --kv-cache-dtype fp8 to optimize memory further.

Acknowledgements

Alibaba / Qwen team for the base Qwen3.6-27B model.
Intel AutoRound team for the quantization framework.
vLLM project for the inference engine and Qwen3.x MTP support.

Downloads last month: 41

Safetensors

Model size

9B params

Tensor type

I32

BF16

F16

Model tree for Minachist/Qwen3.6-27B-INT8-Autoround-V2

Base model

Qwen/Qwen3.6-27B

Quantized

(640)

this model