hsmin92/internvl35-4b-awq-w4a16-g128

InternVL3.5-4B-HF AWQ W4A16 (group size 128)

A compressed-tensors AWQ W4A16 quantization of OpenGVLab/InternVL3_5-4B-HF, built for serving a 4B vision-language model on an 8 GB consumer GPU with vLLM.

A companion FP8 build of the same base model is at hsmin92/internvl35-fp8. See Choosing between the FP8 and AWQ builds.

Quantization scope

Quantized to asymmetric INT4:

• 252 language decoder Linear modules (144 attention: q_proj / k_proj / v_proj / o_proj, 108 MLP: gate_proj / up_proj / down_proj)
• Scheme: W4A16_ASYM, algorithm: AWQ (duo_scaling, n_grid=20)
• Weight format: pack-quantized, group size 128
• Activations: not quantized (A16)

Kept in BF16:

• Vision tower (vision_tower)
• Multimodal projector (multi_modal_projector)
• Input embeddings
• lm_head
• Normalization layers and other protected parameters

Generated from the base-model revision:

6bd4487402110ef9889ba50eb7aefeb302526fed

Checkpoint contents verified: 1,597 tensors, 252 packed INT4 modules with matching scale / zero-point tensors, no packed weights under vision_tower, multi_modal_projector, lm_head, or the embeddings, and no NaN/Inf in any scale. See quantization/recipe.py.

Calibration

AWQ fits per-channel scales against a calibration set, so the calibration distribution matters. This build used:

Dataset	lmms-lab/flickr30k
Samples	128 (0 skipped)
Image size	448×448, 1 patch per sample
Sequence length	278–288 tokens
Prompts	generic English / Korean scene-description instructions

This is a general-purpose photo set. If your target domain is far from everyday photography — industrial inspection, thermal imagery, medical, document OCR, CCTV at unusual angles — re-run AWQ with in-domain calibration images rather than assuming this checkpoint transfers. The FP8 build needs no calibration and does not carry this caveat.

Verified environment

Component	Version / value
GPU	NVIDIA GeForce RTX 3070 8 GB (Ampere, SM 8.6)
Host	Windows 11 + WSL2 (Ubuntu 24.04), Docker Desktop
NVIDIA driver	591.86
Serving image	vllm/vllm-openai:v0.26.0
vLLM	0.26.0 (V1 engine)
PyTorch	2.11.0+cu130
Transformers	5.14.1
Quantization backend	compressed-tensors 0.17.0
Attention backend	FlashAttention (auto-selected)

Startup with the command below:

Model loading took 3.84 GiB memory and 5.08 seconds
GPU KV cache size: 12,288 tokens
Maximum concurrency for 8,192 tokens per request: 1.50x
Graph capturing finished in 1 secs, took 0.09 GiB

Total process footprint is about 6.6 GB, leaving roughly 0.9 GB free on an 8 GB card that is also driving a desktop. CUDA graphs capture successfully and a full FP16 KV cache is used at an 8,192-token context — neither is possible with the FP8 build on the same card.

Serving with vLLM

--dtype bfloat16 is required; see Do not use --dtype half.

vllm serve hsmin92/internvl35-4b-awq-w4a16-g128 \
  --served-model-name internvl35-awq \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 4096 \
  --limit-mm-per-prompt '{"image":2,"video":0}' \
  --gpu-memory-utilization 0.79 \
  --kv-cache-memory-bytes 1728M \
  --trust-remote-code \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

Then use the OpenAI-compatible endpoint at http://127.0.0.1:8000/v1.

Each flag is explained below — several of them are not obvious, and two of them fail in ways that a health check will not catch.

Do not use --dtype half

On this checkpoint family --dtype half produces a server that starts normally, returns HTTP 200 from /health, reports a healthy container health check — and answers every request with !!!!!!!!. Dequantization overflows the FP16 range, the logits become NaN, and argmax selects token 0. It fails silently, so a liveness probe will not catch it.

The checkpoint declares bfloat16 in config.json and Ampere supports BF16 natively at the same memory cost. Always send a real request after a configuration change, not just a health check.

Pin the KV cache instead of raising --gpu-memory-utilization

Without --kv-cache-memory-bytes, vLLM expands the KV cache to fill whatever --gpu-memory-utilization allows. On this model that wastes VRAM for no gain and makes the cache size drift between restarts with whatever the desktop happens to be using:

	utilization only (0.86)	pinned KV (0.79 + 1728M)
KV cache	16,128 – 19,344 tokens (varies per restart)	12,288 tokens (fixed)
Process VRAM	7.9 GB	6.6 GB
Throughput, 16 concurrent	126.1 tok/s	124.4 tok/s

1.3 GB is recovered for a throughput difference inside measurement noise. Size the cache from the workload instead: 12,288 tokens holds one full-length 8,192-token request with room to spare, or about 11 typical requests of a 960×544 frame plus a normalized crop (~1,084 tokens each).

--gpu-memory-utilization is a fraction of the free memory the process can see, which is not what host nvidia-smi reports. Under WSL2 the container saw 6.93 GiB while the Windows host reported 7.48 GiB free, so the effective ceiling was 0.866, and 0.87 was rejected outright:

Free memory on device cuda:0 (6.93/8.0 GiB) on startup is less than
desired GPU memory utilization (0.87, 6.96 GiB)

--max-num-seqs: leaving it at the default costs throughput

The vLLM default is 128. When more concurrent requests arrive than the KV cache can hold, vLLM preempts and recomputes, and throughput collapses below what a smaller limit achieves. Measured at ~1,084 prompt tokens per request with a 16,128-token cache:

Concurrent	--max-num-seqs 128	--max-num-seqs 8
1	0.84 s / 63.3 tok/s	0.84 s / 63.4 tok/s
2	1.19 s / 87.4 tok/s	1.18 s / 88.2 tok/s
4	1.87 s / 105.8 tok/s	1.19 s / 167.2 tok/s
8	3.26 s / 119.7 tok/s	3.22 s / 120.5 tok/s
16	11.82 s / 68.9 tok/s	6.54 s / 126.1 tok/s

Pick a limit the cache can actually sustain. Do not derive it from the Maximum concurrency for 8,192 tokens per request line in the startup log — that figure assumes every request fills the full context window, which real requests rarely do.

--max-num-batched-tokens also sizes the multimodal encoder cache

This is not a VRAM tuning knob. The vLLM default of 2048 rejects large images outright, on any card:

image item with 2816 embedding tokens, which exceeds the
pre-allocated encoder cache size 2048

InternVL dynamic tiling allows up to 12 patches plus a thumbnail, so a single image can reach 3,329 embedding tokens. Use at least 4096.

Tool-call flags are needed even without tool calling

--enable-auto-tool-choice and --tool-call-parser are required for the server to accept requests that carry a tool_choice field. Open WebUI sends tool_choice: "auto" by default, so omitting the flags makes it fail with:

"auto" tool choice requires --enable-auto-tool-choice and
--tool-call-parser to be set

Actual tool calling still does not work: the bundled chat_template.jinja has no tool rendering, so tools=[...] is silently ignored.

Image token cost is driven by aspect ratio, not size

Tile count is chosen from the aspect ratio, so token cost is not monotonic in resolution. Measured on this model:

Input size	Image tokens	Tiles
448×448	257	1
896×448	769	3
960×544	769	3
800×450	769	3
2048×2048	2,561	10
200×150	3,329	13
4000×300	3,329	13

A 200×150 crop costs 4.3× more than an 800×450 image, because 4:3 maps exactly onto a 4×3 tile grid and the crop is upscaled to 1792×1344 for no added information. Across randomly sized crops of a 960×544 frame, the total prompt cost swung between 1,040 and 4,112 tokens.

When sending region crops, letterbox them into a fixed 448×448 canvas (preserving aspect ratio, no upscaling) to make cost constant at ~1,065 tokens per request. On a small-text reading task this was 4× faster than sending the raw crop, with no loss of accuracy down to 14 px glyphs.

Choosing between the FP8 and AWQ builds

Measured on the same RTX 3070, same vLLM version, each tuned to its own best settings on this card:

	FP8 Dynamic	AWQ W4A16 G128
Weights on GPU	5.51 GiB	3.84 GiB
KV cache	9,088 tokens (fp8 KV)	12,288 tokens (fp16 KV)
KV cache dtype needed for 8192 ctx	fp8 (compromise)	fp16
CUDA graphs on 8 GB	not possible	yes
--gpu-memory-utilization needed	0.85 (near the 0.866 ceiling)	0.79
Process VRAM	7.8 GB	6.6 GB
Text generation, single request	12–18 tok/s	~2–4× faster (18–63 tok/s, varies with warm-up)
Calibration data required	no	yes (see above)
Weight precision	8-bit	4-bit

AWQ is decisively better on memory and speed, and it is the only one of the two that fits comfortably on an 8 GB card without compromises.

Accuracy has not been compared between the two builds, or against the BF16 base model. 4-bit weights and out-of-domain calibration are both reasons to expect the AWQ build to degrade first on hard inputs. Benchmark both on your own data before choosing.

Limitations

• A quantized derivative, not an independently trained model.
• No accuracy benchmark has been published for this checkpoint. Functional validation only: text generation, a 7k-token context, and multi-image (full frame + crop) requests all produce coherent, correct answers on simple synthetic probes, both from a local directory and downloaded from this repo.
• Calibration is general-purpose photography; see Calibration.
• The vision tower and output head remain BF16 and are a meaningful share of the loaded weights, so the size reduction is smaller than 4/16 would suggest.
• The bundled chat_template.jinja has no tool-calling support.
• VRAM figures depend on driver, desktop applications, context length, multimodal limits, and vLLM version.

Attribution and license

A quantized derivative of OpenGVLab/InternVL3_5-4B-HF. The original project and this derivative are distributed under the Apache-2.0 license. Review the upstream model card for original training details, limitations, and citation information.

Citation

@article{wang2025internvl3_5,
  title={InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency},
  author={Wang, Weiyun and Gao, Zhangwei and Gu, Lixin and Pu, Hengjun and Cui, Long and Wei, Xingguang and Liu, Zhaoyang and Jing, Linglin and Ye, Shenglong and Shao, Jie and others},
  journal={arXiv preprint arXiv:2508.18265},
  year={2025}
}