machine-made-Fibre/Qwen3-4B-Instruct-2507-heretic-OpenVINO-INT8-NPU

Qwen3-4B-Instruct-2507-heretic OpenVINO INT8 for Intel NPU

This repository contains a ready-to-run OpenVINO IR export of p-e-w/Qwen3-4B-Instruct-2507-heretic, prepared for local inference on Intel NPU through OpenVINO Model Server.

This is not a fine-tune. It is an OpenVINO INT8 runtime export of the source model above.

Source model

• Source model: p-e-w/Qwen3-4B-Instruct-2507-heretic
• Original base model: Qwen/Qwen3-4B-Instruct-2507
• Architecture: Qwen3ForCausalLM
• Task: text generation
• License: Apache-2.0, inherited from the source model metadata

OpenVINO export

Compression metadata from this export:

{
  "mode": "int8_sym",
  "nncf_mode": "INT8_SYM",
  "group_size": -1,
  "ratio": 1.0
}

Known local artifact size:

• openvino_model.bin: about 3.75 GiB

Tested Intel NPU runtime

Tested locally on Windows with OpenVINO Model Server / OpenVINO GenAI:

• Target device: NPU
• OVMS task: text_generation
• Runtime prompt limit: 16384
• Max concurrent sequences: 1
• Cache interval multiplier: 64

Example OVMS command:

ovms.exe `
  --model_path Q:/llm/models/OpenVINO/p-e-w--Qwen3-4B-Instruct-2507-heretic-text-fp16-int8-sym-cw-ov `
  --model_name p-e-w--Qwen3-4B-Instruct-2507-heretic-int8-npu `
  --rest_port 8000 `
  --rest_bind_address 0.0.0.0 `
  --task text_generation `
  --target_device NPU `
  --max_prompt_len 16384 `
  --max_num_seqs 1 `
  --cache_interval_multiplier 64 `
  --tool_parser hermes3

Local benchmark

Measured on the local Intel NPU setup above with the OVMS OpenAI-compatible chat completions endpoint.

Quantization	Load time	Avg output speed	Working set after runs	Private memory after runs
INT8_SYM	371.7 s	9.04 tok/s	9.29 GiB	2.56 GiB

The benchmark prompt was a short three-sentence OpenVINO explanation request with max_tokens=128.