---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: OpenGVLab/InternVL3_5-4B-HF
base_model_relation: quantized
language:
  - en
  - zh
  - ko
tags:
  - internvl
  - internvl3.5
  - vision-language
  - multimodal
  - vllm
  - compressed-tensors
  - awq
  - int4
  - w4a16
  - ampere
  - wsl2
---

# InternVL3.5-4B-HF AWQ W4A16 (group size 128)

A compressed-tensors **AWQ W4A16** quantization of
[OpenGVLab/InternVL3_5-4B-HF](https://huggingface.co/OpenGVLab/InternVL3_5-4B-HF),
built for serving a 4B vision-language model on an 8 GB consumer GPU with vLLM.

A companion FP8 build of the same base model is at
[hsmin92/internvl35-fp8](https://huggingface.co/hsmin92/internvl35-fp8).
See [Choosing between the FP8 and AWQ builds](#choosing-between-the-fp8-and-awq-builds).

## Quantization scope

Quantized to asymmetric INT4:

- **252** language decoder `Linear` modules
  (144 attention: `q_proj` / `k_proj` / `v_proj` / `o_proj`,
  108 MLP: `gate_proj` / `up_proj` / `down_proj`)
- Scheme: `W4A16_ASYM`, algorithm: AWQ (`duo_scaling`, `n_grid=20`)
- Weight format: `pack-quantized`, group size **128**
- Activations: **not** quantized (A16)

Kept in BF16:

- Vision tower (`vision_tower`)
- Multimodal projector (`multi_modal_projector`)
- Input embeddings
- `lm_head`
- Normalization layers and other protected parameters

Generated from the base-model revision:

```text
6bd4487402110ef9889ba50eb7aefeb302526fed
```

Checkpoint contents verified: 1,597 tensors, 252 packed INT4 modules with
matching scale / zero-point tensors, no packed weights under `vision_tower`,
`multi_modal_projector`, `lm_head`, or the embeddings, and no NaN/Inf in any
scale. See [`quantization/recipe.py`](./quantization/recipe.py).

## Calibration

AWQ fits per-channel scales against a calibration set, so the calibration
distribution matters. This build used:

| | |
|---|---|
| Dataset | `lmms-lab/flickr30k` |
| Samples | 128 (0 skipped) |
| Image size | 448×448, 1 patch per sample |
| Sequence length | 278–288 tokens |
| Prompts | generic English / Korean scene-description instructions |

**This is a general-purpose photo set.** If your target domain is far from
everyday photography — industrial inspection, thermal imagery, medical,
document OCR, CCTV at unusual angles — re-run AWQ with in-domain calibration
images rather than assuming this checkpoint transfers. The FP8 build needs no
calibration and does not carry this caveat.

## Verified environment

| Component | Version / value |
|---|---|
| GPU | NVIDIA GeForce RTX 3070 8 GB (Ampere, SM 8.6) |
| Host | Windows 11 + WSL2 (Ubuntu 24.04), Docker Desktop |
| NVIDIA driver | 591.86 |
| Serving image | `vllm/vllm-openai:v0.26.0` |
| vLLM | 0.26.0 (V1 engine) |
| PyTorch | 2.11.0+cu130 |
| Transformers | 5.14.1 |
| Quantization backend | compressed-tensors 0.17.0 |
| Attention backend | FlashAttention (auto-selected) |

Startup with the command below:

```text
Model loading took 3.84 GiB memory and 5.08 seconds
GPU KV cache size: 12,288 tokens
Maximum concurrency for 8,192 tokens per request: 1.50x
Graph capturing finished in 1 secs, took 0.09 GiB
```

Total process footprint is about **6.6 GB**, leaving roughly 0.9 GB free on an
8 GB card that is also driving a desktop. CUDA graphs capture successfully and
a full FP16 KV cache is used at an 8,192-token context — neither is possible
with the FP8 build on the same card.

## Serving with vLLM

`--dtype bfloat16` is required; see [Do not use `--dtype half`](#do-not-use---dtype-half).

```bash
vllm serve hsmin92/internvl35-4b-awq-w4a16-g128 \
  --served-model-name internvl35-awq \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 4096 \
  --limit-mm-per-prompt '{"image":2,"video":0}' \
  --gpu-memory-utilization 0.79 \
  --kv-cache-memory-bytes 1728M \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser hermes
```

Then use the OpenAI-compatible endpoint at `http://127.0.0.1:8000/v1`.

Each flag is explained below — several of them are not obvious, and two of
them fail in ways that a health check will not catch.

### Do not use `--dtype half`

On this checkpoint family `--dtype half` produces a server that starts
normally, returns HTTP 200 from `/health`, reports a healthy container
health check — and answers every request with `!!!!!!!!`. Dequantization
overflows the FP16 range, the logits become NaN, and argmax selects token 0.
It fails silently, so a liveness probe will not catch it.

The checkpoint declares `bfloat16` in `config.json` and Ampere supports BF16
natively at the same memory cost. Always send a real request after a
configuration change, not just a health check.

### Pin the KV cache instead of raising `--gpu-memory-utilization`

Without `--kv-cache-memory-bytes`, vLLM expands the KV cache to fill whatever
`--gpu-memory-utilization` allows. On this model that wastes VRAM for no gain
and makes the cache size drift between restarts with whatever the desktop
happens to be using:

| | utilization only (0.86) | pinned KV (0.79 + 1728M) |
|---|---|---|
| KV cache | 16,128 – 19,344 tokens (varies per restart) | **12,288 tokens (fixed)** |
| Process VRAM | 7.9 GB | **6.6 GB** |
| Throughput, 16 concurrent | 126.1 tok/s | 124.4 tok/s |

1.3 GB is recovered for a throughput difference inside measurement noise.
Size the cache from the workload instead: 12,288 tokens holds one full-length
8,192-token request with room to spare, or about 11 typical requests of a
960×544 frame plus a normalized crop (~1,084 tokens each).

**`--gpu-memory-utilization` is a fraction of the free memory the process can
see, which is not what host `nvidia-smi` reports.** Under WSL2 the container
saw 6.93 GiB while the Windows host reported 7.48 GiB free, so the effective
ceiling was 0.866, and 0.87 was rejected outright:

```text
Free memory on device cuda:0 (6.93/8.0 GiB) on startup is less than
desired GPU memory utilization (0.87, 6.96 GiB)
```

### `--max-num-seqs`: leaving it at the default costs throughput

The vLLM default is 128. When more concurrent requests arrive than the KV
cache can hold, vLLM preempts and recomputes, and throughput collapses below
what a smaller limit achieves. Measured at ~1,084 prompt tokens per request
with a 16,128-token cache:

| Concurrent | `--max-num-seqs 128` | `--max-num-seqs 8` |
|---|---|---|
| 1 | 0.84 s / 63.3 tok/s | 0.84 s / 63.4 tok/s |
| 2 | 1.19 s / 87.4 tok/s | 1.18 s / 88.2 tok/s |
| 4 | 1.87 s / 105.8 tok/s | 1.19 s / 167.2 tok/s |
| 8 | 3.26 s / 119.7 tok/s | 3.22 s / 120.5 tok/s |
| **16** | **11.82 s / 68.9 tok/s** | **6.54 s / 126.1 tok/s** |

Pick a limit the cache can actually sustain. Do **not** derive it from the
`Maximum concurrency for 8,192 tokens per request` line in the startup log —
that figure assumes every request fills the full context window, which real
requests rarely do.

### `--max-num-batched-tokens` also sizes the multimodal encoder cache

This is not a VRAM tuning knob. The vLLM default of 2048 rejects large images
outright, on any card:

```text
image item with 2816 embedding tokens, which exceeds the
pre-allocated encoder cache size 2048
```

InternVL dynamic tiling allows up to 12 patches plus a thumbnail, so a single
image can reach **3,329 embedding tokens**. Use at least 4096.

### Tool-call flags are needed even without tool calling

`--enable-auto-tool-choice` and `--tool-call-parser` are required for the
server to *accept* requests that carry a `tool_choice` field. Open WebUI sends
`tool_choice: "auto"` by default, so omitting the flags makes it fail with:

```text
"auto" tool choice requires --enable-auto-tool-choice and
--tool-call-parser to be set
```

Actual tool calling still does not work: the bundled `chat_template.jinja` has
no tool rendering, so `tools=[...]` is silently ignored.

### Image token cost is driven by aspect ratio, not size

Tile count is chosen from the aspect ratio, so token cost is not monotonic in
resolution. Measured on this model:

| Input size | Image tokens | Tiles |
|---|---|---|
| 448×448 | 257 | 1 |
| 896×448 | 769 | 3 |
| 960×544 | 769 | 3 |
| 800×450 | 769 | 3 |
| 2048×2048 | 2,561 | 10 |
| **200×150** | **3,329** | **13** |
| 4000×300 | 3,329 | 13 |

A 200×150 crop costs **4.3× more** than an 800×450 image, because 4:3 maps
exactly onto a 4×3 tile grid and the crop is upscaled to 1792×1344 for no
added information. Across randomly sized crops of a 960×544 frame, the total
prompt cost swung between 1,040 and 4,112 tokens.

When sending region crops, letterbox them into a fixed 448×448 canvas
(preserving aspect ratio, no upscaling) to make cost constant at ~1,065 tokens
per request. On a small-text reading task this was 4× faster than sending the
raw crop, with no loss of accuracy down to 14 px glyphs.

## Choosing between the FP8 and AWQ builds

Measured on the same RTX 3070, same vLLM version, each tuned to its own best
settings on this card:

| | FP8 Dynamic | **AWQ W4A16 G128** |
|---|---|---|
| Weights on GPU | 5.51 GiB | **3.84 GiB** |
| KV cache | 9,088 tokens (fp8 KV) | **12,288 tokens (fp16 KV)** |
| KV cache dtype needed for 8192 ctx | fp8 (compromise) | **fp16** |
| CUDA graphs on 8 GB | not possible | **yes** |
| `--gpu-memory-utilization` needed | 0.85 (near the 0.866 ceiling) | **0.79** |
| Process VRAM | 7.8 GB | **6.6 GB** |
| Text generation, single request | 12–18 tok/s | **~2–4× faster** (18–63 tok/s, varies with warm-up) |
| Calibration data required | **no** | yes (see above) |
| Weight precision | 8-bit | 4-bit |

AWQ is decisively better on memory and speed, and it is the only one of the
two that fits comfortably on an 8 GB card without compromises.

**Accuracy has not been compared between the two builds, or against the BF16
base model.** 4-bit weights and out-of-domain calibration are both reasons to
expect the AWQ build to degrade first on hard inputs. Benchmark both on your
own data before choosing.

## Limitations

- A quantized derivative, not an independently trained model.
- **No accuracy benchmark has been published for this checkpoint.** Functional
  validation only: text generation, a 7k-token context, and multi-image
  (full frame + crop) requests all produce coherent, correct answers on simple
  synthetic probes, both from a local directory and downloaded from this repo.
- Calibration is general-purpose photography; see [Calibration](#calibration).
- The vision tower and output head remain BF16 and are a meaningful share of
  the loaded weights, so the size reduction is smaller than 4/16 would suggest.
- The bundled `chat_template.jinja` has no tool-calling support.
- VRAM figures depend on driver, desktop applications, context length,
  multimodal limits, and vLLM version.

## Attribution and license

A quantized derivative of
[OpenGVLab/InternVL3_5-4B-HF](https://huggingface.co/OpenGVLab/InternVL3_5-4B-HF).
The original project and this derivative are distributed under the Apache-2.0
license. Review the upstream model card for original training details,
limitations, and citation information.

## Citation

```bibtex
@article{wang2025internvl3_5,
  title={InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency},
  author={Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others},
  journal={arXiv preprint arXiv:2508.18265},
  year={2025}
}
```