Instructions to use Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Local Apps Settings

How to use Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4

SGLang

How to use Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4
```

可亲 commited on Sep 24, 2024

Commit

288c348

1 Parent(s): 669a942

fix(pad zero) pad intermediate_size to 29696 to make sure quantized model can use 8 tensor-parallel in vllm

Browse files

Files changed (12) hide show

config.json +10 -16
model-00002-of-00011.safetensors +2 -2
model-00003-of-00011.safetensors +2 -2
model-00004-of-00011.safetensors +2 -2
model-00005-of-00011.safetensors +2 -2
model-00006-of-00011.safetensors +2 -2
model-00007-of-00011.safetensors +2 -2
model-00008-of-00011.safetensors +2 -2
model-00009-of-00011.safetensors +2 -2
model-00010-of-00011.safetensors +2 -2
model-00011-of-00011.safetensors +2 -2
model.safetensors.index.json +1 -1

config.json CHANGED Viewed

@@ -1,20 +1,16 @@
 {
-  "_name_or_path": "Qwen/Qwen2-VL-72B-Instruct-GPTQ-Int4",
   "architectures": [
     "Qwen2VLForConditionalGeneration"
   ],
   "attention_dropout": 0.0,
   "bos_token_id": 151643,
   "eos_token_id": 151645,
-  "vision_start_token_id": 151652,
-  "vision_end_token_id": 151653,
-  "vision_token_id": 151654,
-  "image_token_id": 151655,
-  "video_token_id": 151656,
   "hidden_act": "silu",
   "hidden_size": 8192,
   "initializer_range": 0.02,
-  "intermediate_size": 29568,
   "max_position_embeddings": 32768,
   "max_window_layers": 80,
   "model_type": "qwen2_vl",
@@ -48,17 +44,15 @@
   "transformers_version": "4.45.0.dev0",
   "use_cache": true,
   "use_sliding_window": false,
   "vision_config": {
-    "depth": 32,
-    "embed_dim": 1280,
-    "mlp_ratio": 4,
-    "num_heads": 16,
-    "in_chans": 3,
     "hidden_size": 8192,
-    "patch_size": 14,
-    "spatial_merge_size": 2,
-    "spatial_patch_size": 14,
-    "temporal_patch_size": 2
   },
   "vocab_size": 152064
 }

 {
+  "_name_or_path": "Qwen/Qwen2-VL-72B-Instruct-315-fix-dim-pad",
   "architectures": [
     "Qwen2VLForConditionalGeneration"
   ],
   "attention_dropout": 0.0,
   "bos_token_id": 151643,
   "eos_token_id": 151645,
   "hidden_act": "silu",
   "hidden_size": 8192,
+  "image_token_id": 151655,
   "initializer_range": 0.02,
+  "intermediate_size": 29696,
   "max_position_embeddings": 32768,
   "max_window_layers": 80,
   "model_type": "qwen2_vl",
   "transformers_version": "4.45.0.dev0",
   "use_cache": true,
   "use_sliding_window": false,
+  "video_token_id": 151656,
   "vision_config": {
     "hidden_size": 8192,
+    "in_chans": 3,
+    "model_type": "qwen2_vl",
+    "spatial_patch_size": 14
   },
+  "vision_end_token_id": 151653,
+  "vision_start_token_id": 151652,
+  "vision_token_id": 151654,
   "vocab_size": 152064
 }

model-00002-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e70b86ad87b5c9acb4ed4c40a1d827aa34c64eb2d3c9049094cc6439d918fe84
-size 3902575872

 version https://git-lfs.github.com/spec/v1
+oid sha256:45d3b76058c8eff6d2ebf879f82ceff8fabefcb369bbc674b47ef9e51e634184
+size 3916744448

model-00003-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:02a7968a92a35f325117dca0be501de123230da3ea257a0f39296428df51be6a
-size 3981209392

 version https://git-lfs.github.com/spec/v1
+oid sha256:ccb329fd14b0ca16bc1b14a58e8a859fa3a6f2905a03388be9b92bc955346c2a
+size 3995377968

model-00004-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:00f64105f70d48805450f3e62d7184ddb285e5081f1e6480494f9635b9318041
-size 3981123936

 version https://git-lfs.github.com/spec/v1
+oid sha256:d8d6e3c26d69b39cbde61744bf0b60adf1fe07e751ded341924c6ce6f6ad9869
+size 3995292000

model-00005-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9e8a923b3ed7a42f07c4aa1a1386c7932e2fdf76dc65d638798d2150b69fba6c
-size 3902576144

 version https://git-lfs.github.com/spec/v1
+oid sha256:ae941c3aae7310f6447f00162fb2a5dc3eef83446e8f6d394ec99e7e53b91f08
+size 3916744720

model-00006-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:98f99c6b09d3e2d9a62334e7ed7ef22702b37f744d163c637d809c468bd64387
-size 3981209432

 version https://git-lfs.github.com/spec/v1
+oid sha256:a38f2e3f7dd0c6e4b02b25bd7353cd092fd099e4860419305d82cec3b8e3a744
+size 3995378008

model-00007-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:988e04dbd234bb629a30199ad18b62b311f2b39fde400ef93c3cc22968e2105b
-size 3981123936

 version https://git-lfs.github.com/spec/v1
+oid sha256:00704e00f9deef3aad2d0a5703b2ee94c3bc0456bc597c2b12c97450cbc21354
+size 3995292000

model-00008-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f9d7fddeb804d3cf3efb6935a529575694fa76970cdc8319457d7c238bf7bacb
-size 3902576144

 version https://git-lfs.github.com/spec/v1
+oid sha256:086663beec63d0cdd63cf43839e4e84ec7d3903bc39ca100590094eb3fff9596
+size 3916744720

model-00009-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ad670d4c0712886048b89d67c5a3e30ed7f74697cf18d68366596d2ca54c71fc
-size 3981209432

 version https://git-lfs.github.com/spec/v1
+oid sha256:c5a23a028d90c731b76d1710140512cef54ba2fc2a0559382ac59ec6af727750
+size 3995378008

model-00010-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0d82198941e145fc7367d9a3d322ca02a512bdd396df3d7e4c13331087930227
-size 3981123936

 version https://git-lfs.github.com/spec/v1
+oid sha256:9ce98548c5ad9c0d1bd960d601d2edbbb994610d1049622762f98bab672c95c2
+size 3995292000

model-00011-of-00011.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aaed771e993864e5c4739a9037185d9a3e9f9195e95678847f39b78b26e12b73
-size 3325518160

 version https://git-lfs.github.com/spec/v1
+oid sha256:c4b6514d5795ed72e8921df2af722ab7f06b9825a7f2e187e8cb4b44d4f6e35b
+size 3328787792

model.safetensors.index.json CHANGED Viewed

@@ -1,6 +1,6 @@
 {
   "metadata": {
-    "total_size": 42888678400
   },
   "weight_map": {
     "lm_head.weight": "model-00011-of-00011.safetensors",

 {
   "metadata": {
+    "total_size": 43019463680
   },
   "weight_map": {
     "lm_head.weight": "model-00011-of-00011.safetensors",