Instructions to use Qwen/Qwen3-235B-A22B-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Qwen/Qwen3-235B-A22B-GPTQ-Int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Qwen/Qwen3-235B-A22B-GPTQ-Int4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-235B-A22B-GPTQ-Int4")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-235B-A22B-GPTQ-Int4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Qwen/Qwen3-235B-A22B-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen3-235B-A22B-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-235B-A22B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen3-235B-A22B-GPTQ-Int4

SGLang

How to use Qwen/Qwen3-235B-A22B-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen3-235B-A22B-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-235B-A22B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen3-235B-A22B-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen3-235B-A22B-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen3-235B-A22B-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen3-235B-A22B-GPTQ-Int4
```

feihu.hf commited on May 19, 2025

Commit

7c0ea6e

2 Parent(s): 5c4d7dc fe68aef

Merge branch 'main' of hf.co:Qwen/Qwen3-235B-A22B-GPTQ-Int4

Browse files

Files changed (38) hide show

README.md +1 -1
config.json +6 -14
generation_config.json +5 -12
model-00001-of-00005.safetensors +0 -3
model-00005-of-00005.safetensors → model-00001-of-00032.safetensors +2 -2
model-00002-of-00005.safetensors → model-00002-of-00032.safetensors +2 -2
model-00003-of-00005.safetensors → model-00003-of-00032.safetensors +2 -2
model-00004-of-00005.safetensors → model-00004-of-00032.safetensors +2 -2
model-00005-of-00032.safetensors +3 -0
model-00006-of-00032.safetensors +3 -0
model-00007-of-00032.safetensors +3 -0
model-00008-of-00032.safetensors +3 -0
model-00009-of-00032.safetensors +3 -0
model-00010-of-00032.safetensors +3 -0
model-00011-of-00032.safetensors +3 -0
model-00012-of-00032.safetensors +3 -0
model-00013-of-00032.safetensors +3 -0
model-00014-of-00032.safetensors +3 -0
model-00015-of-00032.safetensors +3 -0
model-00016-of-00032.safetensors +3 -0
model-00017-of-00032.safetensors +3 -0
model-00018-of-00032.safetensors +3 -0
model-00019-of-00032.safetensors +3 -0
model-00020-of-00032.safetensors +3 -0
model-00021-of-00032.safetensors +3 -0
model-00022-of-00032.safetensors +3 -0
model-00023-of-00032.safetensors +3 -0
model-00024-of-00032.safetensors +3 -0
model-00025-of-00032.safetensors +3 -0
model-00026-of-00032.safetensors +3 -0
model-00027-of-00032.safetensors +3 -0
model-00028-of-00032.safetensors +3 -0
model-00029-of-00032.safetensors +3 -0
model-00030-of-00032.safetensors +3 -0
model-00031-of-00032.safetensors +3 -0
model-00032-of-00032.safetensors +3 -0
model.safetensors.index.json +1 -1
tokenizer_config.json +1 -0

README.md CHANGED Viewed

@@ -44,7 +44,7 @@ For more details, including benchmark evaluation, hardware requirements, and inf
 > Currently, ``transformers`` has issues with multi-GPU inference for GPTQ quantized models. We recommend using SGLang or vLLM for deployment.
-For deployment, you can use `sglang>=0.4.6.post1` or `vllm>=0.8.5` or to create an OpenAI-compatible API endpoint:
 - SGLang:
     ```shell
     python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-GPTQ-Int4 --reasoning-parser qwen3 --tp 4

 > Currently, ``transformers`` has issues with multi-GPU inference for GPTQ quantized models. We recommend using SGLang or vLLM for deployment.
+For deployment, you can use `sglang>=0.4.6.post1` or `vllm==0.8.4` or to create an OpenAI-compatible API endpoint:
 - SGLang:
     ```shell
     python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-GPTQ-Int4 --reasoning-parser qwen3 --tp 4

config.json CHANGED Viewed

@@ -27,23 +27,15 @@
   "quantization_config": {
     "bits": 4,
     "checkpoint_format": "gptq",
     "desc_act": false,
     "group_size": 128,
-    "lm_head": false,
-    "meta": {
-      "damp_auto_increment": 0.0025,
-      "damp_percent": 0.01,
-      "mse": 0.0,
-      "quantizer": [
-        "gptqmodel:2.2.0"
-      ],
-      "static_groups": false,
-      "true_sequential": true,
-      "uri": "https://github.com/modelcloud/gptqmodel"
-    },
-    "pack_dtype": "int32",
     "quant_method": "gptq",
-    "sym": true
   },
   "rms_norm_eps": 1e-06,
   "rope_scaling": null,

   "quantization_config": {
     "bits": 4,
     "checkpoint_format": "gptq",
+    "damp_percent": 0.01,
     "desc_act": false,
     "group_size": 128,
+    "model_file_base_name": null,
+    "model_name_or_path": null,
     "quant_method": "gptq",
+    "static_groups": false,
+    "sym": true,
+    "true_sequential": true
   },
   "rms_norm_eps": 1e-06,
   "rope_scaling": null,

generation_config.json CHANGED Viewed

@@ -1,13 +1,6 @@
 {
-    "bos_token_id": 151643,
-    "do_sample": true,
-    "eos_token_id": [
-        151645,
-        151643
-    ],
-    "pad_token_id": 151643,
-    "temperature": 0.6,
-    "top_k": 20,
-    "top_p": 0.95,
-    "transformers_version": "4.51.0"
-}

 {
+  "_from_model_config": true,
+  "eos_token_id": 151645,
+  "pad_token_id": 151643,
+  "transformers_version": "4.51.3"
+}

model-00001-of-00005.safetensors DELETED Viewed

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:1cef7c90a889a0ddc6092800984d784467673a417dea06a7e7b0901c18bbe10c
-size 30004176672

model-00005-of-00005.safetensors → model-00001-of-00032.safetensors RENAMED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b52fd70a71d949db173a4087580c0c39d5aff9dd94e0989ffbfc120ec581ef7d
-size 4534206216

 version https://git-lfs.github.com/spec/v1
+oid sha256:e31da9bc1725cb89fa1ceace52b06fcaf9ce4dd3a56e162ade4bf62dc6b3f1ed
+size 3998000432

model-00002-of-00005.safetensors → model-00002-of-00032.safetensors RENAMED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:81b9c7a7b70664a9f241955aa88bc857b957d9495bf4f7717d008e126f72e7ba
-size 30003678080

 version https://git-lfs.github.com/spec/v1
+oid sha256:3df75d9069fbba562413edb4d456e4c09e8c07b4ad0a36135cb58b0be5d721f4
+size 4000547568

model-00003-of-00005.safetensors → model-00003-of-00032.safetensors RENAMED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:847d53a1d667fa23f5bdf17bc05a49822e6ca1e9c727f390e53765acceded10f
-size 30003563152

 version https://git-lfs.github.com/spec/v1
+oid sha256:7d7cb04f447b1104fcf49eedaa0b656bb13b70ba780d3b4678f1a70177ae15f4
+size 4000537336

model-00004-of-00005.safetensors → model-00004-of-00032.safetensors RENAMED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9ee54370ecab9fef20539a72071c1b52105dc19f0379df965bddec5db9a897c5
-size 30003563152

 version https://git-lfs.github.com/spec/v1
+oid sha256:61ef7766943e43164c6a92bc79dea9e318ed00f9e4c2fe7137a3536f732b5bd4
+size 4000539432

model-00005-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3b25c2698a5c0ff5130b4e9572d7d1f6847d46b5612d1bc6d76732c903a313e1
+size 4000552368

model-00006-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d5da86a06feb3a1810e84da6481542f724c32e85ce9238e3304c671a3827c44a
+size 4000542136

model-00007-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2c0babeaa9800cb7d24003f27b4b9bd50782efb6e1e6cc19dd29c4dd3a3de469
+size 4000542128

model-00008-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c677020129e3bed540e06fdc71ff0056b5185efb6e0967d5c9f758d493f8df8e
+size 4000552368

model-00009-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:94dc2ce10e56d7c46a72bf9fc5be6bb76b7ad7efed7b4e5692d977d30a4f43f0
+size 4000542136

model-00010-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:699e58086bf7465b4c1b42c3c1a9e915c8c8d42a50ff942345009421586f619a
+size 4000542224

model-00011-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b9dc3fe098f86e932cbc1cf67364db5023cc87b1009d0d925e6fd8951af6099d
+size 4000552496

model-00012-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2635504eef849688b2d97a730957f859ed9a997fc735e5d92713ada3e1a7ad28
+size 3987427192

model-00013-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2a469964ee0143a9fd2817a9ea710bad9498273049306e212776dc20bc0d1f6a
+size 3999344896

model-00014-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7c16aba7820e51ca33c0569fb170488802f400b245ab02ca421566ee5155e46b
+size 4000542096

model-00015-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c46d69af8333dd0d4fba8214e5872a65b5d428bc645ab2026a44384d7fd290e8
+size 4000542128

model-00016-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:ab8f03db925b9af1563b7dcee7e8cc2cdd83adbd0b244d584b913ca49c43538b
+size 4000552368

model-00017-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:635631e51b554a2756590dbe158527a8ed995ab1edc0f203fd1c4d5cc5cf060a
+size 4000542136

model-00018-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1335e2fc6e0546d15d0859a1468a976cac475bd4f397388939ec96488f54da05
+size 4000542128

model-00019-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:46270693fd4ba48aa6f5a05746b70a746ead9e84a0ae13a0e763b4bbf420aab2
+size 4000552368

model-00020-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:02d7bf6837a35f631b976c191569997aa89d88e8bab18876101cf70b5b5ef2da
+size 4000542136

model-00021-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2bac214c217591ba97c33d5c37d0575feb26752094dcab88c3f1490cbfa7b90a
+size 4000542128

model-00022-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:46efe0f85a124d844cd5601e368c973c017bcb571db922aa3c20fadb75f39df7
+size 4000552400

model-00023-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:371ec11eec1b410c02a0c3a54383e88770713783db971069e848788b9cfaabec
+size 4000542264

model-00024-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a0a7bbe6b9637cdcb4c2f5e476ba8929f3cc5cd978ee380f2633314de7a2d58b
+size 4000542256

model-00025-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bb39cd2d5f249c7ad2c3f5b5e27ad8e9dd83b5e8fdbe8aefb9e0a1539d359fd9
+size 3999361608

model-00026-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:53bd114a4e9ffeb72330b1a44f9a96cbc181dc2cd880f23b72303b1949489f2a
+size 4000542048

model-00027-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d194433e3662f34a81102f0fa11a71862870e02414d3a20ff90d5add64f91c80
+size 4000542128

model-00028-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b257d596ab34e5cdd32bd259e55d2c7d53d7a8ce2bd5f436f40dcf82f26d6a1b
+size 4000552368

model-00029-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bb4ae0c6893b599e5b21a8d1db8f8a750603758a43adb8a2f9e0095d206502d6
+size 4000542136

model-00030-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e9c24c522a4a84eec494a5d46f152f9aa10232ad4f241b5c3b6486a17e04f243
+size 4000542128

model-00031-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3773b95a3594f2ede533d64564efe4908e2f874b6eda540a7c83265899b144ef
+size 3305963560

model-00032-of-00032.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:28e42c8e703cca812a08af985b44d3bd34091327e263d1a2e78a361f8a6643e1
+size 1244659840

model.safetensors.index.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:129eced7ff2b4e1d5d45f13ffcf9aed1671f4fc8cbd013fd55501d112802ac80
 size 13211435

 version https://git-lfs.github.com/spec/v1
+oid sha256:27fbdfeb4565eeb580455d7d3ab67261f9bfa9bd7a687511f64d409287b38aab
 size 13211435

tokenizer_config.json CHANGED Viewed

@@ -231,6 +231,7 @@
   "clean_up_tokenization_spaces": false,
   "eos_token": "<|im_end|>",
   "errors": "replace",
   "model_max_length": 131072,
   "pad_token": "<|endoftext|>",
   "split_special_tokens": false,

   "clean_up_tokenization_spaces": false,
   "eos_token": "<|im_end|>",
   "errors": "replace",
+  "extra_special_tokens": {},
   "model_max_length": 131072,
   "pad_token": "<|endoftext|>",
   "split_special_tokens": false,