Qwen3-VL-4B-Instruct-GPTQ-W4A16

This repository provides a GPTQ post-training quantized version of Qwen3-VL-4B-Instruct for efficient multimodal inference and evaluation.

Overview

This model is a third-party compressed checkpoint built on top of Qwen3-VL-4B-Instruct, mainly for efficient deployment, benchmarking, and PTQ baseline construction.

The current release uses GPTQ W4A16 quantization in the llm-compressor workflow, with group-wise low-bit weight compression for the language-side transformer modules.

Compared with the original checkpoint layout, this release also reduces storage footprint in a practical way.

• Original size: 4,850,810 KB + 3,816,885 KB
• Quantized size: 3,400,962 KB
• Compression: -60.774%

Base Model

• Base model: Qwen/Qwen3-VL-4B-Instruct
• Model family: Qwen3-VL
• Quantization method: GPTQ
• Quantization format: W4A16
• Framework: llm-compressor

Quantization Setup

This release follows a GPTQ-based post-training quantization workflow in llm-compressor, where the compressed checkpoint is produced by reconstructing low-bit weights layer-by-layer using calibration statistics.

Quantization Recipe

recipe = GPTQModifier(
    ignore=[
        "re:.*lm_head", "re:.*visual.*"
    ],
    block_size=128,
    dampening_frac=0.01,
    actorder="static",
    offload_hessians=False,
    config_groups={
        "group_0": {
            "targets": ["Linear"],
            "weights": {
                "num_bits": 4,
                "type": "int",
                "symmetric": True,
                "group_size": 128,
                "strategy": "group",
                "dynamic": False,
                "actorder": None,
            },
        },
    },
)

Notes

• The checkpoint uses GPTQ W4A16 as a practical low-bit PTQ baseline.
• Quantization is applied to Linear layers with 4-bit symmetric integer weights and group-wise compression (group_size=128).
• block_size=128 controls the GPTQ reconstruction granularity during compression.
• dampening_frac=0.01 is used to stabilize Hessian-based quantization.
• actorder="static" is enabled for better accuracy recovery with no extra runtime cost.
• lm_head and visual modules are excluded from quantization in this release.

Calibration Setup

Calibration data was constructed from the Flickr30k image-caption dataset.

For GPTQ calibration, 128 samples were selected from local Flickr30k parquet files after dataset loading and random shuffling with a fixed seed (seed=42). Each sample was converted into a multimodal chat-style input containing one image and one paired caption, and then processed into model inputs such as input_ids, attention_mask, pixel_values, and image_grid_thw.

Calibration Details

• Dataset: Flickr30k
• Data format: local parquet files
• Number of calibration samples: 128
• Sampling strategy: shuffled subset with fixed random seed
• Max sequence length: 2048
• Purpose: multimodal activation/statistics collection for GPTQ PTQ

Evaluation Configuration

For evaluation in VLMEvalKit, the following model entry can be added to VLMEvalKit/vlmeval/config.py:

'Qwen3-VL-4B-Instruct-GPTQ-W4A16': partial(
    vlm.Qwen3VLChat,
    model_path='/home/lml/models/Qwen3-VL-4B-Instruct-GPTQ-W4A16-g128-llmcompressor',
    min_pixels=256 * 28 * 28,
    max_pixels=1280 * 28 * 28,
    use_custom_prompt=False,
    use_vllm=True,
    temperature=0.7,
    max_new_tokens=8192,
    repetition_penalty=1.0,
    presence_penalty=1.5,
    top_p=0.8,
    top_k=20,
    max_model_len=16384,
    gpu_utils=0.90,
    enable_thinking=False,
)

Intended Use

This release is intended for:

• Efficient multimodal inference
• PTQ baseline construction for Qwen3-VL
• Evaluation with VLMEvalKit
• Serving experiments with vLLM
• Research on VLM post-training quantization

Disclaimer

This is a third-party quantized checkpoint and is not an official release from the Qwen team.

Quantization may affect model quality on some multimodal tasks, especially fine-grained visual understanding and reasoning benchmarks.

Citation

If you use this model, please cite the original Qwen3-VL report, GPTQ, VLMEvalKit, and the calibration dataset when appropriate.

@article{bai2025qwen3vl,
  title={Qwen3-VL Technical Report},
  author={Bai, Shuai and Cai, Yuxuan and Zhu, Keming and others},
  journal={arXiv preprint arXiv:2511.21631},
  year={2025}
}

@inproceedings{frantar2023gptq,
  title={GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers},
  author={Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2023}
}

@misc{duan2024vlmevalkit,
  title={VLMEvalKit: An Open-Source Toolkit for Evaluating Large Vision-Language Models},
  author={OpenCompass Team},
  howpublished={\url{https://github.com/open-compass/VLMEvalKit}},
  year={2024}
}

@article{young2014image,
  title={From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions},
  author={Young, Peter and Lai, Alice and Hodosh, Micah and Hockenmaier, Julia},
  journal={Transactions of the Association for Computational Linguistics},
  volume={2},
  pages={67--78},
  year={2014},
  publisher={MIT Press}
}

Acknowledgement

This repository builds upon the following open-source projects:

• Qwen3-VL
• llm-compressor
• AWQ
• VLMEvalKit
• vLLM