Instructions to use simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors")
model = AutoModelForMultimodalLM.from_pretrained("simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors

SGLang

How to use simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors with Docker Model Runner:
```
docker model run hf.co/simaai/Qwen2.5-VL-3B-Instruct-GPTQ-Safetensors
```

Qwen2.5-VL-3B-Instruct-GPTQ

This is a local post-training quantized checkpoint of Qwen/Qwen2.5-VL-3B-Instruct for later LLiMa compilation and deployment on Sima.ai hardware. It must not be uploaded as part of the current VLM batch.

Source revision: Not captured in the flattened local source directory. An immutable upstream revision must be recovered and recorded before any release or redistribution.

Source license metadata: qwen-research. This checkpoint remains subject to the upstream license, intended-use guidance, and limitations.

Quantization

Component	Method	Weight format	Effective targets
Decoder Linear layers and `lm_head`	GPTQ	symmetric INT4, group size 256, static act-order	253
Vision encoder Linear layers	GPTQ	symmetric INT8, per-channel, static act-order	162
Projector/architecture exceptions	none	source BF16	Explicitly recorded in `recipe.yaml`

The lm_head follows the default GPTQ INT4/G256 policy; no INT8 escalation was made. Calibration used lmms-lab/flickr30k test[:512], deterministic dataset order, 512 image-text samples, sequence length 2048, batch size 1, no concatenation, and no padding to maximum length. The exact target names and effective settings are saved in recipe.yaml.

Evaluation

Full MMStar used all 1,500 examples, VLMEvalKit commit 7055d3010c38ccb5dcae1bc9535ca19c7fe5d79f, deterministic generation, and exact_matching without an API judge.

Checkpoint	Overall accuracy
Source	54.1333%
Quantized	53.5333%
Absolute change	-0.6000 percentage points
Relative change	-1.1084%

Evaluation date: 2026-07-16. Slurm jobs: source 4641, quantized 4642. The aggregate evidence row is stored in vlmeval_results/mmstar_results.csv; raw predictions and status files remain under vlmeval_results/mmstar_full/.

Validation: MMStar inference passed (1,500/1,500).

Full matched WikiText-2 word perplexity (EleutherAI/wikitext_document_level, wikitext-2-raw-v1, no example limit; 2026-07-18): source 13.8655; this checkpoint 15.5220; absolute degradation +1.6565; relative degradation +11.95%. Raw JSON evidence: perplexity_results/vlm_full_wikitext/qwen25_vl_3b_{source,gptq_w4g256_visiongptq_w8}_v3.json.

Reproduction

This directory contains the exact quantize.py, recipe.yaml, and versions.txt used for the artifact:

python quantize.py \
  --model-path /path/to/source-model \
  --output-dir /path/to/new-quantized-model

Environment

Python: 3.13.2
torch: 2.11.0+cu128
CUDA: 12.8
transformers: 5.10.1
llm-compressor: 0.12.0
auto-round: 0.13.0
compressed-tensors: 0.17.1

Deployment

This is the pre-LLiMa quantized Hugging Face artifact. LLiMa compiler output must remain a separate deployment artifact unless its format and redistribution are approved. No Hugging Face upload is authorized for this batch.

Limitations

Quantization quality can vary by language, visual domain, prompt format, context length, and downstream runtime. The MMStar result does not replace application-specific validation. Audio components, when present, and explicitly excluded projectors remain in source precision as documented in recipe.yaml.