Update model card with right-sized 8 GB serving config and measured tuning data

README.md

@@ -95,19 +95,19 @@ calibration and does not carry this caveat.
 | Quantization backend | compressed-tensors 0.17.0 |
 | Attention backend | FlashAttention (auto-selected) |
 
-Measured at startup with `--gpu-memory-utilization 0.86 --max-model-len 8192
---max-num-seqs 1 --max-num-batched-tokens 4096 --dtype bfloat16`:
+Startup with the command below:
 
 ```text
-Model loading took 3.84 GiB memory
-Available KV cache memory: 2.66 GiB
-GPU KV cache size: 19,344 tokens
-Maximum concurrency for 8,192 tokens per request: 2.36x
-Graph capturing finished in 1 secs, took 0.00 GiB
+Model loading took 3.84 GiB memory and 5.08 seconds
+GPU KV cache size: 12,288 tokens
+Maximum concurrency for 8,192 tokens per request: 1.50x
+Graph capturing finished in 1 secs, took 0.09 GiB
 ```
 
-CUDA graphs capture successfully and full FP16 KV cache fits at an 8,192-token
-context — neither is possible with the FP8 build on the same 8 GB card.
+Total process footprint is about **6.6 GB**, leaving roughly 0.9 GB free on an
+8 GB card that is also driving a desktop. CUDA graphs capture successfully and
+a full FP16 KV cache is used at an 8,192-token context — neither is possible
+with the FP8 build on the same card.
 
 ## Serving with vLLM
 
@@ -118,20 +118,26 @@ vllm serve hsmin92/internvl35-4b-awq-w4a16-g128 \
   --served-model-name internvl35-awq \
   --dtype bfloat16 \
   --max-model-len 8192 \
-  --max-num-seqs 1 \
+  --max-num-seqs 8 \
   --max-num-batched-tokens 4096 \
   --limit-mm-per-prompt '{"image":2,"video":0}' \
-  --gpu-memory-utilization 0.86 \
-  --trust-remote-code
+  --gpu-memory-utilization 0.79 \
+  --kv-cache-memory-bytes 1728M \
+  --trust-remote-code \
+  --enable-auto-tool-choice \
+  --tool-call-parser hermes
 ```
 
 Then use the OpenAI-compatible endpoint at `http://127.0.0.1:8000/v1`.
 
+Each flag is explained below — several of them are not obvious, and two of
+them fail in ways that a health check will not catch.
+
 ### Do not use `--dtype half`
 
 On this checkpoint family `--dtype half` produces a server that starts
-normally, returns HTTP 200 from `/health`, reports a healthy Docker
-healthcheck — and answers every request with `!!!!!!!!`. Dequantization
+normally, returns HTTP 200 from `/health`, reports a healthy container
+health check — and answers every request with `!!!!!!!!`. Dequantization
 overflows the FP16 range, the logits become NaN, and argmax selects token 0.
 It fails silently, so a liveness probe will not catch it.
 
@@ -139,18 +145,80 @@ The checkpoint declares `bfloat16` in `config.json` and Ampere supports BF16
 natively at the same memory cost. Always send a real request after a
 configuration change, not just a health check.
 
-### `--max-num-batched-tokens` and multi-image requests
+### Pin the KV cache instead of raising `--gpu-memory-utilization`
+
+Without `--kv-cache-memory-bytes`, vLLM expands the KV cache to fill whatever
+`--gpu-memory-utilization` allows. On this model that wastes VRAM for no gain
+and makes the cache size drift between restarts with whatever the desktop
+happens to be using:
+
+| | utilization only (0.86) | pinned KV (0.79 + 1728M) |
+|---|---|---|
+| KV cache | 16,128 – 19,344 tokens (varies per restart) | **12,288 tokens (fixed)** |
+| Process VRAM | 7.9 GB | **6.6 GB** |
+| Throughput, 16 concurrent | 126.1 tok/s | 124.4 tok/s |
+
+1.3 GB is recovered for a throughput difference inside measurement noise.
+Size the cache from the workload instead: 12,288 tokens holds one full-length
+8,192-token request with room to spare, or about 11 typical requests of a
+960×544 frame plus a normalized crop (~1,084 tokens each).
 
-This value also sizes the multimodal encoder cache. InternVL dynamic tiling
-allows up to 12 patches plus a thumbnail, so a single image can reach
-**3,329 embedding tokens**. With a smaller value, large images are rejected:
+**`--gpu-memory-utilization` is a fraction of the free memory the process can
+see, which is not what host `nvidia-smi` reports.** Under WSL2 the container
+saw 6.93 GiB while the Windows host reported 7.48 GiB free, so the effective
+ceiling was 0.866, and 0.87 was rejected outright:
+
+```text
+Free memory on device cuda:0 (6.93/8.0 GiB) on startup is less than
+desired GPU memory utilization (0.87, 6.96 GiB)
+```
+
+### `--max-num-seqs`: leaving it at the default costs throughput
+
+The vLLM default is 128. When more concurrent requests arrive than the KV
+cache can hold, vLLM preempts and recomputes, and throughput collapses below
+what a smaller limit achieves. Measured at ~1,084 prompt tokens per request
+with a 16,128-token cache:
+
+| Concurrent | `--max-num-seqs 128` | `--max-num-seqs 8` |
+|---|---|---|
+| 1 | 0.84 s / 63.3 tok/s | 0.84 s / 63.4 tok/s |
+| 2 | 1.19 s / 87.4 tok/s | 1.18 s / 88.2 tok/s |
+| 4 | 1.87 s / 105.8 tok/s | 1.19 s / 167.2 tok/s |
+| 8 | 3.26 s / 119.7 tok/s | 3.22 s / 120.5 tok/s |
+| **16** | **11.82 s / 68.9 tok/s** | **6.54 s / 126.1 tok/s** |
+
+Pick a limit the cache can actually sustain. Do **not** derive it from the
+`Maximum concurrency for 8,192 tokens per request` line in the startup log —
+that figure assumes every request fills the full context window, which real
+requests rarely do.
+
+### `--max-num-batched-tokens` also sizes the multimodal encoder cache
+
+This is not a VRAM tuning knob. The vLLM default of 2048 rejects large images
+outright, on any card:
 
 ```text
 image item with 2816 embedding tokens, which exceeds the
 pre-allocated encoder cache size 2048
 ```
 
-Use at least 4096 if you send high-resolution or wide-aspect images.
+InternVL dynamic tiling allows up to 12 patches plus a thumbnail, so a single
+image can reach **3,329 embedding tokens**. Use at least 4096.
+
+### Tool-call flags are needed even without tool calling
+
+`--enable-auto-tool-choice` and `--tool-call-parser` are required for the
+server to *accept* requests that carry a `tool_choice` field. Open WebUI sends
+`tool_choice: "auto"` by default, so omitting the flags makes it fail with:
+
+```text
+"auto" tool choice requires --enable-auto-tool-choice and
+--tool-call-parser to be set
+```
+
+Actual tool calling still does not work: the bundled `chat_template.jinja` has
+no tool rendering, so `tools=[...]` is silently ignored.
 
 ### Image token cost is driven by aspect ratio, not size
 
@@ -162,35 +230,45 @@ resolution. Measured on this model:
 | 448×448 | 257 | 1 |
 | 896×448 | 769 | 3 |
 | 960×544 | 769 | 3 |
-| 200×150 | **3,329** | **13** |
 | 800×450 | 769 | 3 |
-| 4000×300 | **3,329** | **13** |
+| 2048×2048 | 2,561 | 10 |
+| **200×150** | **3,329** | **13** |
+| 4000×300 | 3,329 | 13 |
 
 A 200×150 crop costs **4.3× more** than an 800×450 image, because 4:3 maps
 exactly onto a 4×3 tile grid and the crop is upscaled to 1792×1344 for no
-added information. When sending region crops, letterbox them into a fixed
-448×448 canvas (preserving aspect ratio, no upscaling) to make cost constant.
+added information. Across randomly sized crops of a 960×544 frame, the total
+prompt cost swung between 1,040 and 4,112 tokens.
+
+When sending region crops, letterbox them into a fixed 448×448 canvas
+(preserving aspect ratio, no upscaling) to make cost constant at ~1,065 tokens
+per request. On a small-text reading task this was 4× faster than sending the
+raw crop, with no loss of accuracy down to 14 px glyphs.
 
 ## Choosing between the FP8 and AWQ builds
 
-Measured on the same RTX 3070, same vLLM version, same settings:
+Measured on the same RTX 3070, same vLLM version, each tuned to its own best
+settings on this card:
 
 | | FP8 Dynamic | **AWQ W4A16 G128** |
 |---|---|---|
 | Weights on GPU | 5.51 GiB | **3.84 GiB** |
-| KV cache available | 0.61 GiB | **2.66 GiB** |
-| KV cache capacity | 8,928 tokens | **19,344 tokens** |
-| Max concurrency @ 8192 ctx | 1.09× | **2.36×** |
+| KV cache | 9,088 tokens (fp8 KV) | **12,288 tokens (fp16 KV)** |
 | KV cache dtype needed for 8192 ctx | fp8 (compromise) | **fp16** |
 | CUDA graphs on 8 GB | not possible | **yes** |
-| Text generation | 13–17 tok/s | **~55 tok/s** |
+| `--gpu-memory-utilization` needed | 0.85 (near the 0.866 ceiling) | **0.79** |
+| Process VRAM | 7.8 GB | **6.6 GB** |
+| Text generation, single request | 12–18 tok/s | **~2–4× faster** (18–63 tok/s, varies with warm-up) |
 | Calibration data required | **no** | yes (see above) |
 | Weight precision | 8-bit | 4-bit |
 
-AWQ is decisively better on memory and speed. **Accuracy has not been compared
-between the two builds, or against the BF16 base model.** 4-bit weights and
-out-of-domain calibration are both reasons to expect the AWQ build to degrade
-first on hard inputs. Benchmark both on your own data before choosing.
+AWQ is decisively better on memory and speed, and it is the only one of the
+two that fits comfortably on an 8 GB card without compromises.
+
+**Accuracy has not been compared between the two builds, or against the BF16
+base model.** 4-bit weights and out-of-domain calibration are both reasons to
+expect the AWQ build to degrade first on hard inputs. Benchmark both on your
+own data before choosing.
 
 ## Limitations
 
@@ -198,12 +276,11 @@ first on hard inputs. Benchmark both on your own data before choosing.
 - **No accuracy benchmark has been published for this checkpoint.** Functional
   validation only: text generation, a 7k-token context, and multi-image
   (full frame + crop) requests all produce coherent, correct answers on simple
-  synthetic probes.
+  synthetic probes, both from a local directory and downloaded from this repo.
 - Calibration is general-purpose photography; see [Calibration](#calibration).
 - The vision tower and output head remain BF16 and are a meaningful share of
   the loaded weights, so the size reduction is smaller than 4/16 would suggest.
-- The bundled `chat_template.jinja` has no tool-calling support. Passing
-  `tools=[...]` is silently ignored regardless of vLLM's tool-parser flags.
+- The bundled `chat_template.jinja` has no tool-calling support.
 - VRAM figures depend on driver, desktop applications, context length,
   multimodal limits, and vLLM version.