---
license: apache-2.0
base_model:
- Qwen/Qwen2.5-3B-Instruct
tags:
- openvino
- int4
- nncf
- ovms
- text-generation
library_name: openvino
pipeline_tag: text-generation
---

# Qwen2.5-3B-Instruct-int4-ov

[Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) converted to **OpenVINO IR** with
**INT4** weight-only quantization (NNCF, group_size=128, asymmetric, ratio=1.0).
Intended for serving with **OpenVINO Model Server (OVMS)** on CPU / Intel GPU.

## Serve with OVMS

```bash
docker run -d --rm -p 8000:8000 -v $(pwd)/models:/models --device /dev/dri \
  --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) \
  openvino/model_server:2026.2.1-gpu \
  --rest_port 8000 --model_repository_path /models \
  --source_model Prog-up/Qwen2.5-3B-Instruct-int4-ov --target_device GPU --task text_generation
```

Then call the OpenAI-compatible endpoint at `POST /v3/chat/completions`.

## Benchmark

~26.1 tok/s decode throughput (INT4, Intel iGPU, OVMS 2026.2.1, 256-token greedy decode).