Instructions to use JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only")
model = AutoModelForMultimodalLM.from_pretrained("JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only

SGLang

How to use JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only with Docker Model Runner:
```
docker model run hf.co/JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only
```

Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only

This is an MXFP4 W4A4 quantized version of Qwen/Qwen2.5-VL-3B-Instruct, created using llm-compressor.

Only the LLM decoder is quantized. The Vision Transformer (ViT) encoder remains in BF16 precision.

Model Summary

Property	Value
Base Model	Qwen/Qwen2.5-VL-3B-Instruct
Quantization	MXFP4 W4A4 (4-bit float weights, 4-bit float activations)
Quantization Scope	LLM decoder only (ViT encoder in BF16)
Strategy	Per-tensor-group (group_size=32), symmetric (minmax observer)
Format	`compressed-tensors` (`mxfp4-pack-quantized`)
Model Size	~3.3 GB (1 shard)
Ignored Layers	`lm_head`, all `model.visual.*` layers
Tool	llm-compressor
Supported Runtime	vLLM (with `compressed-tensors`)

Quantization Details

Weights: FP4 E2M1 (4-bit float), per-tensor-group (group_size=32) with E8M0 (uint8 exponent) block scales, static quantization
Activations: FP4 E2M1 (4-bit float), per-tensor-group (group_size=32), dynamic quantization
Ignored: lm_head (kept in BF16) and all ViT encoder layers (model.visual.*)
Calibration: 512 samples from CNN/DailyMail, max_seq_length=2048
Standard: OCP MX Specification

Quantization Recipe

quant_stage:
  quant_modifiers:
    QuantizationModifier:
      ignore: ["lm_head", "re:model.visual.*"]
      scheme: "MXFP4"
      targets: ["Linear"]

Hardware Requirements

MXFP4 inference requires NVIDIA Blackwell (SM120+) GPUs with CUDA 12.8+ for native CUTLASS MXFP4 GEMM support.

Usage

With vLLM

export VLLM_ATTENTION_BACKEND=TORCH_SDPA

vllm serve JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only \
    --trust-remote-code \
    --max-model-len 4096 \
    --enforce-eager

With Transformers

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only",
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

messages = [{"role": "user", "content": [
    {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
    {"type": "text", "text": "Describe this image in detail."},
]}]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=256)
result = processor.batch_decode(output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(result[0])

Model Architecture

Architecture: Qwen2_5_VLForConditionalGeneration
LLM Hidden Size: 2048
LLM Intermediate Size: 11008
LLM Layers: 36
LLM Attention Heads: 16
LLM KV Heads: 2 (GQA)
Vision Encoder Depth: 32
Vision Hidden Size: 1280
Vocabulary Size: 151936

Differences from NVFP4

Feature	MXFP4	NVFP4
Scale Format	E8M0 (uint8 exponent)	E4M3 + FP32 global scale
Group Size	32	16
Standard	OCP MX Specification	NVIDIA proprietary
Hardware	SM120+ (Blackwell)	SM89+ (Ada/Hopper/Blackwell)

Acknowledgments

Base model by Qwen Team
Quantization powered by llm-compressor and compressed-tensors

Downloads last month: 5

Safetensors

Model size

2B params

Tensor type

BF16

Model tree for JongYeop/Qwen2.5-VL-3B-Instruct-MXFP4-W4A4-LM-Only

Base model

Qwen/Qwen2.5-VL-3B-Instruct

Quantized

(92)

this model