Instructions to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4")
model = AutoModelForMultimodalLM.from_pretrained("palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4

SGLang

How to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4
```

Model requantized with improved recipe (v2)

by palmfuture - opened Apr 20

Discussion

palmfuture

Owner Apr 20

Model requantized with improved recipe (v2)

Hi everyone — just pushed a fresh quantization to replace the previous weights. If you've been using this model, here's what changed and why you might want to re-download.

What changed

Quality

Metric	Previous	New	Change
GPTQ success rate	69.68%	97.42%	+27.7pp
RTN fallback rate	30.32%	2.58%	11.7× lower
Loss mean	3.56e-04	1.38e-04	2.58× lower
Loss median	2.32e-04	9.29e-05	2.50× lower
Loss max	1.00e-02	2.14e-03	4.67× lower

The old version fell back to RTN on ~30% of modules. The new version falls back on ~2.6%. The remaining RTN cases are cold experts that rarely route at inference anyway.

Calibration

Previous: single-source (C4 only).
New: domain-mixed — C4 + Tulu-3 SFT + CodeParrot + MATH-500, 256 samples total.

This is the main reason RTN dropped so dramatically. MoE routing sends different input types to different experts, so single-domain calibration leaves many experts under-activated. Mixed calibration covers more of the routing surface.

Algorithm

Upgraded to GPTQModel 6.0.3 with v2=True (GPTQ v2 algorithm) and act_group_aware=True (auto-enabled with desc_act=False). Both contribute measurable quality improvements over the previous version.

Size

21.2 GB (down from 22.8 GB). All MoE expert layers 0–39 now properly quantized. Previous version had a config bug that skipped layer 0's experts — this is fixed.

Breaking changes

None for usage — same URL, same tokenizer, same architecture, same API. SGLang and vLLM load it the same way.

If you had it cached: you'll need to clear HF cache to pick up the new weights:

rm -rf ~/.cache/huggingface/hub/models--palmfuture--Qwen3.6-35B-A3B-GPTQ-Int4/

Or use force_download=True / --revision main depending on your loader.

Hardware note

This version was quantized on 4× RTX 3060 12GB (consumer) instead of an A100, using Python 3.13t no-GIL for multi-GPU data-parallel quantization. Runtime ~4h 20m. Full reproducibility notes in the README.

Transparency

quant_log.csv is included in the repo — 30,720 rows, one per module, with GPTQ loss or RTN fallback marker, sample count, and wall-clock time. Anyone can audit the run.

Known limitations

Layers 32–39 show higher RTN density (~7–10% vs ~1–3% in early layers). Consistent with late-layer MoE routing concentration and some memory pressure late in the run. With more RAM + more calibration samples, this would drop further.
157 of 256 expert IDs had RTN fallback in at least one layer. 99 got GPTQ across every layer. Top cold experts: 235, 249, 234, 197, 237.

Feedback welcome

If you notice regressions vs the old version (unlikely given the quality metrics, but possible on specific workloads), please drop a comment with specifics and I'll investigate.

Thanks to everyone who's been using this model.

Reasonable

Apr 20

•

edited Apr 20

Does it seem right that model-00001-of-00006.safetensors would not have been updated as part of this? It's still showing the same hash as before.

Also, thank you for working on this model. I get a pretty steady 80+ Tokens/s generation on my aging hardware (4x RTX 2080 Ti 22GB) which feels a bit like flying. One things are all dialed in and three quarters of those tokens aren't logic loops and repetitive thought, I think it's going to be quite capable.

I'm using vllm, and I'm having better success with this new model while implementing this fix: https://github.com/vllm-project/vllm/issues/39056

I suspect we're going to continue to see improvements with vllm and this model as time goes by.

palmfuture

Owner Apr 21

Yes, that's expected. Shard 1 only contains layers that GPTQ doesn't quantize (visual tower, embeddings, attention, shared experts, layernorms) — they get copied from the base model as-is, so the bytes come out identical every run regardless of calibration. The actual quantized experts live in shards 2–6, which did change between v1 and v2.
And thank you for the kind words! 80+ tokens/s on 4× 2080 Ti 22GB is great to hear — Turing really holds up for MoE inference. Good find on issue #39056 too, that one tripped up a few people. Enjoy!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment