Instructions to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4")
model = AutoModelForMultimodalLM.from_pretrained("palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4

SGLang

How to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4
```

palmfuture commited on Apr 26

Commit

a413772

verified ·

1 Parent(s): f32841f

Update README.md

Browse files

Files changed (1) hide show

README.md +44 -1

README.md CHANGED Viewed

@@ -25,6 +25,23 @@ GPTQ Int4 quantization of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwe
 This v2 release ships **MTP (Multi-Token Prediction) speculative decoding weights** verified working on both vLLM 0.19.1 and SGLang 0.5.10.
 ## Quality
 | Metric | Value |
@@ -134,6 +151,29 @@ vllm serve palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 \
 Requires `trust_remote_code=True` for the Qwen3.5-MoE architecture.
 ## Speculative decoding (MTP)
 This release ships **MTP (Multi-Token Prediction) weights** in the per-expert split format expected by vLLM and SGLang loaders (785 MTP keys total, all BF16). Speculative decoding is verified working on both engines.
@@ -164,6 +204,8 @@ Verified on vLLM 0.19.1 + 4× RTX 3060 12GB:
 | Mean acceptance length | 2.4–2.8 / 2 draft tokens |
 | KV pool | 160,800 tokens (262K max-model-len, 2.34× concurrency) |
 ### SGLang with MTP (EAGLE)
 ```bash
@@ -204,6 +246,7 @@ The full per-module quantization log is published as [`quant_log.csv`](./quant_l
 - **Cold experts.** 157 of 256 expert IDs fell back to RTN in at least one layer; 99 always got GPTQ. Top cold experts: 235, 249, 234, 197, 237. These rarely route at inference.
 - **Ampere-only limitations.** On RTX 3060 (SM86), fp8 KV cache is storage-only (dequantized for attention). No FP8 compute path.
 - **vLLM `--quantization` flag.** Do not pass `--quantization moe_wna16` to vLLM — it triggers a `KeyError` in the MTP loader path. SGLang requires the flag; vLLM must auto-detect.
 ## Credits
@@ -214,4 +257,4 @@ The full per-module quantization log is published as [`quant_log.csv`](./quant_l
 ---
-Quantized by [@palmfuture](https://huggingface.co/palmfuture).

 This v2 release ships **MTP (Multi-Token Prediction) speculative decoding weights** verified working on both vLLM 0.19.1 and SGLang 0.5.10.
+## ⚠️ Important: Thinking-mode infinite loops
+Qwen3.6 has a documented [infinite-loop issue in thinking mode](https://github.com/QwenLM/Qwen3.6/issues/88) that affects all variants (BF16, FP8, AWQ, GPTQ) — this is a model-level issue, not a quantization defect. On hard problems, reasoning can run away and consume the entire `max_tokens` budget with repeated phrases like "Wait, but actually...".
+**Recommended mitigation:** set a server-side `thinking_token_budget` using [vllm-default-thinking-budget](https://github.com/palmfuture/vllm-default-thinking-budget):
+```bash
+git clone https://github.com/palmfuture/vllm-default-thinking-budget
+./vllm-default-thinking-budget/install.sh /path/to/your/vllm/venv
+export VLLM_DEFAULT_THINKING_BUDGET=8192      # cap reasoning tokens
+export VLLM_DEFAULT_PRESENCE_PENALTY=1.0      # repetition suppression
+vllm serve palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 ...
+```
+This works around vLLM [issue #28070](https://github.com/vllm-project/vllm/issues/28070) where `--override-generation-config` silently drops `thinking_token_budget` and `presence_penalty`. See [Recommended sampling config](#recommended-sampling-config) below for tuning by workload.
 ## Quality
 | Metric | Value |
 Requires `trust_remote_code=True` for the Qwen3.5-MoE architecture.
+## Recommended sampling config
+For thinking mode, follow [official Qwen3.6 guidance](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) plus a server-side `thinking_token_budget` to prevent runaway reasoning:
+```bash
+vllm serve palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 \
+  --override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0}' \
+  --reasoning-parser qwen3 \
+  --reasoning-config '{"reasoning_start_str": "<think>", "reasoning_end_str": " I need to give the final answer now.</think>"}' \
+  ...
+```
+Combine with [vllm-default-thinking-budget](https://github.com/palmfuture/vllm-default-thinking-budget) for the budget defaults:
+| Workload | `VLLM_DEFAULT_THINKING_BUDGET` | Notes |
+|----------|--------------------------------|-------|
+| Agent / tool calls | 2048–4096 | Low reasoning need, fast response |
+| General chat | 8192 | Balanced default |
+| Coding tasks | 8192–16384 | Includes refactor and debug |
+| Math / hard reasoning | 16384–32768 | Approaching official 80K guidance |
+The `reasoning_end_str` injection ensures the model produces a coherent final answer when the budget is exceeded, instead of cutting off mid-thought.
 ## Speculative decoding (MTP)
 This release ships **MTP (Multi-Token Prediction) weights** in the per-expert split format expected by vLLM and SGLang loaders (785 MTP keys total, all BF16). Speculative decoding is verified working on both engines.
 | Mean acceptance length | 2.4–2.8 / 2 draft tokens |
 | KV pool | 160,800 tokens (262K max-model-len, 2.34× concurrency) |
+> **Note:** MTP and `thinking_token_budget` cannot be used together on vLLM 0.19.x ([vLLM issue #39573](https://github.com/vllm-project/vllm/issues/39573)). Choose MTP for throughput on stable workloads, or drop MTP and use `thinking_token_budget` for loop protection on reasoning-heavy workloads.
 ### SGLang with MTP (EAGLE)
 ```bash
 - **Cold experts.** 157 of 256 expert IDs fell back to RTN in at least one layer; 99 always got GPTQ. Top cold experts: 235, 249, 234, 197, 237. These rarely route at inference.
 - **Ampere-only limitations.** On RTX 3060 (SM86), fp8 KV cache is storage-only (dequantized for attention). No FP8 compute path.
 - **vLLM `--quantization` flag.** Do not pass `--quantization moe_wna16` to vLLM — it triggers a `KeyError` in the MTP loader path. SGLang requires the flag; vLLM must auto-detect.
+- **Thinking-mode loops.** Affects all Qwen3.6-35B-A3B variants regardless of quantization. Use [vllm-default-thinking-budget](https://github.com/palmfuture/vllm-default-thinking-budget) for server-side mitigation.
 ## Credits
 ---
+Quantized by [@palmfuture](https://huggingface.co/palmfuture).