KV-Cache Receipts — nine ≈128K users on one A100 (vLLM)

fraQtl 's Collections

updated Aug 25

Three-arm, needle-gated serving receipts (fp16 / fp8-KV / compressed) — reproducible for ≈$12 with one command.

fraQtl/qwen3-4b-instruct-2507-kv-sidecars

Updated Aug 25 • 14 • 1

Note Start here — the 9-user receipt: nine 128K users on one A100, 134.1 tok/s aggregate (fp16: 2 users, 66.6), 9/9 needles
fraQtl/mistral-7b-instruct-v0.3-kv-sidecars

Updated Aug 21 • 8

Note 2.4× the fp16 KV pool at ≈95-100% decode speed, 8K/32K, byte-exact independent re-run
fraQtl/fraqtl-sm80-runtime

Updated Aug 21 • 6

Note The prebuilt A100 wheel + one-command repro script — verify every number yourself, ≈$12
fraQtl/fraqtl-membrane-llamacpp-runtime

Updated Aug 25
fraQtl/mistral-nemo-instruct-2407-kv-sidecars

Updated Aug 25