Instructions to use XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1")
model = AutoModelForMultimodalLM.from_pretrained("XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1

SGLang

How to use XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 with Docker Model Runner:
```
docker model run hf.co/XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1
```

Request for MTP version

by hondaru - opened Jun 16

Discussion

hondaru

Jun 16

Hello XReyRobert,

Could you please provide the model card for the model mentioned:
XReyRobert/Qwopus3.6-27B-v2-MTP-GPTQ-Pro-v1?

While 131K might impose some limitations, I believe it could still be useful for shorter context sizes. Thank you for your assistance.

XReyRobert

Owner Jun 17

•

edited Jun 17

Hi there,
Kind of depressing that I finally get some feedback after 5K+ downloads but only for a model I scraped because it wasn't delivering what I expected :)

Let me see if I can rebuild this for you, as those aren't quantized I can probably do some patching... stay tuned...

Just please confirm your need bellow so that I don't do it for nothing if you lost interest since yesterday !

hondaru

Jun 18

Dear XReyRobert,
Since there are relatively few contributors creating AWQ and/or GPTQ models, I believe your work is receiving significant attention!
Actually, with another Qwen3.6-27B model, I'm getting around 60 TPS on an RTX 3090 with MTP, which made me wonder if this model might perform similarly well.
Thank you for your hard work.

XReyRobert

Owner Jun 18

•

edited Jun 18

Published! (untested) Let me know how it goes. (this is a patch so MTP are BF16 but that shouldn't have much of an impact)
https://huggingface.co/XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-MTP-BF16

hondaru

Jun 22

Dear XReyRobert,
After comparing the two models, the NON-MTP model achieved an average throughput of 45 TPS, while the MTP(2) model reached approximately 55 TPS. While the non-MTP model maintained consistent performance around 45, the MTP model produced results between 50-60 TPS. Based on these results, it appears the MTP enhancement does indeed provide measurable benefits.
Also MTP(2) with turboquant dropped to 40TPS with eager mode but 131k cache.

Regarding accuracy, however, the MTP model produced more distorted text. Why might this be?
Could it be an effect of the GPTQ quantization process?

Regardless, I've gained invaluable experience through this experiment, and I'm truly grateful for the opportunity.

XReyRobert

Owner Jun 22

Sounds great. On my tests MTP didn't bring anything, please share your configuration details (vllm command line) .

As for your distorded text it might more be an effect of MTP than quantization as I patched this model with an unquantized MTP head...

hondaru

Jun 22

Here's how I configure the Docker command:

docker run -d --gpus all
--name vllm-server
--ipc=host
-v ~/.cache/huggingface:/root/.cache/huggingface
-e HF_TOKEN="********"
-e VLLM_ATTENTION_BACKEND=TRITON_ATTN
vllm/vllm-openai:latest
XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-MTP-BF16
--gpu-memory-utilization 0.97
--max-model-len 42280
--max-num-seqs 1
--reasoning-parser qwen3
--kv-cache-dtype fp8
--max-num-batched-tokens 2048
--enable-chunked-prefill
--enable-prefix-caching
--language-model-only
--no-enable-flashinfer-autotune
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

XReyRobert

Owner Jun 25

Thanks for sharing. 42280 of context is really low for agentic use, that why I came to the conclusion of not using it.
I think I managed to use higher context that 42K but tok/s were on pare with the non MTP anyway so it wasn't worth it.
Again that's on a single 3090 if you have 2 thats' a different story.

hondaru

Jun 26

Thanks for the message!
You're right - with Agentic use, context size becomes crucial.
However, having MTP enabled lets you easily increase context size in TurboQuant with minimal performance degradation, which is definitely beneficial. Even with higher prediction buffer sizes, throughput shouldn't decrease significantly.
Having a single RTX 3090 is incredibly versatile - I'm really glad I have it!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment