anoane/DeepSeek-V4-Flash-vllm-moet-sm120-cache

DeepSeek-V4-Flash — vLLM-Moet 2-bit cache (SM120)

Prebuilt 2-bit routed-expert planes + FP4 delta/recovery pool for DeepSeek-V4-Flash (159B MoE), so you can serve the model on a single SM120 GPU — RTX 6000 Pro (96 GB) or RTX 5090 (32 GB) — with vLLM-Moet without re-quantizing the 149 GB fp8 checkpoint on every boot.

The native DeepSeek-V4-Flash checkpoint is fp8 (149 GB) — too big for 96 GB. vLLM-Moet serves it as 2-bit routed-expert planes on SM120 SASS + an optional FP4 delta pool. Building that cache from the fp8 checkpoint means a full re-quant + a ~51 GiB host-RAM staging transient on every start (which forces swap on a small single-GPU host). This repo ships the prebuilt cache so you skip that entirely: with the loader-skip in vLLM-Moet, boot loads these files directly.

Contents

path	size	what
planescache/tp1-rank0/*.bin	~202 GB	resident 2-bit base planes (per-layer). Both configs boot from this — VLLM_MOE_W2_PLANES_CACHE.
moet_store/base.rank0of1.pack	~74 GB	2-bit base experts in pack form (VLLM_MOE_W2_STORE_DIR).
moet_store/fp4.rank0of1.pack	~140 GB	full FP4 experts.
moet_store/delta.rank0of1.pack	~132 GB	FP4 delta pool — the recovery pool Config A loads.
moet_store/pool-heat.*.json, poolmon.json.*	tiny	pool-heat / warm-start metadata.

You still need the model config + tokenizer from the original deepseek-ai/DeepSeek-V4-Flash (mounted as the model directory). This repo provides only the quantized expert cache — everything the 149 GB fp8 → 2-bit re-quant would otherwise rebuild on each boot.

Download & place

hf download anoane/DeepSeek-V4-Flash-vllm-moet-sm120-cache --local-dir /data/ds4-cache
# → /data/ds4-cache/planescache/...   and   /data/ds4-cache/moet_store/...

Mount into the container and point the env vars at the mounts:

• planescache/ → -v /data/ds4-cache/planescache:/planescache → VLLM_MOE_W2_PLANES_CACHE=/planescache
• moet_store/ → -v /data/ds4-cache/moet_store:/store → VLLM_MOE_W2_STORE_DIR=/store

Serving on 1× RTX 6000 Pro (96 GB, SM120)

Uses the vLLM-Moet image (vllm-moet-sm120:v024) with the moe_w2 loader-skip that boots directly from this cache. ⚠️ The loader-skip currently lives on the branch anoane/vLLM-Moet @ feat/moe-w2-loader-skip — it is not on any main yet (a PR to the upstream repo is open), so build the image from that branch.

config	weights	MTP	sampling	decode (seqs 1)	aggregate	rec. ctx
A best quality	2-bit base + 6 GiB FP4 pool	off	temp 1 / top_p 1 / min_p 0.05	~75 tok/s	—	256K
B max throughput	2-bit base (no pool)	off	temp 1 / top_p 1	~89 tok/s	~2085 tok/s @ C=256	256K

Config A — best quality (fills the 96 GB card, ~75 tok/s @ 256K)

Resident 2-bit base + a 6 GiB VRAM FP4 recovery pool, MTP off, min_p=0.05.

docker run -d --name ds4-quality --gpus '"device=0"' --network host --ipc host --shm-size 64g \
  -v /path/to/DeepSeek-V4-Flash:/model:ro \
  -v /data/ds4-cache/planescache:/planescache \
  -v /data/ds4-cache/moet_store:/store \
  -e VLLM_MOE_W2=1 \
  -e VLLM_MOE_W2_PLANES_CACHE=/planescache \
  -e VLLM_MOE_W2_STORE_DIR=/store \
  -e VLLM_MOE_W2_DELTA_GB=6 \
  -e VLLM_MOE_W2_GATE=1 \
  -e VLLM_MOE_W2_GATE_MAX_PROMOTE=64 \
  -e VLLM_MOE_W2_FORCE_POOL=1 \
  -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \
  vllm-moet-sm120:v024 \
  --model /model --served-model-name deepseek-v4-flash --trust-remote-code \
  --kv-cache-dtype fp8 --block-size 256 --max-model-len 262144 \
  --gpu-memory-utilization 0.99 --max-num-batched-tokens 1024 --max-num-seqs 1 \
  --tokenizer-mode deepseek_v4 --no-scheduler-reserve-full-isl \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --port 8000
# MTP intentionally OFF. Per-request sampling: temperature=1.0, top_p=1.0, min_p=0.05.

Why each value (measured on this card):

• VLLM_MOE_W2=1 — 2-bit routed-expert path; native fp8 (149 GB) does not fit 96 GB.
• VLLM_MOE_W2_PLANES_CACHE=/planescache — boot the 2-bit base from this cache (loader-skip): no per-boot re-quant, no host staging, no swap.
• VLLM_MOE_W2_STORE_DIR=/store — serve the FP4 delta pool from the on-disk pack (pread, VRAM/RAM/SSD split), not pinned shmem.
• VLLM_MOE_W2_DELTA_GB=6 — VRAM FP4 recovery pool. 6 GiB is the ceiling on 96 GB — 72.7 GB 2-bit base + 6 GB pool + KV + activations fills the card at util 0.99 (DELTA=7 OOMs at model load). With MTP off, 6 GB = 512 slots ≥ the 438-slot fire floor → full-fidelity fires. A DELTA sweep (0/4/6) showed coding quality flat within seed noise, so 6 is chosen for its long-context-retrieval benefit at no cost.
• VLLM_MOE_W2_GATE=1 — uncertainty gate: on a low-confidence decode step, re-forward it with the routed experts upgraded to FP4 (recovers quality over pure 2-bit).
• VLLM_MOE_W2_GATE_MAX_PROMOTE=64 — cap on experts force-promoted per fired step (optional; unset = uncapped also works).
• VLLM_MOE_W2_FORCE_POOL=1 — downgrade the boot fire-floor check to a warning (we operate right at the floor by design).
• VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 — the graph-memory profiler over-reserves ~2 GiB while the real graph pool is ~0.06 GiB; disabling reclaims it for pool + KV so DELTA=6 fits at 256K.
• --gpu-memory-utilization 0.99 — DELTA=6 is tight; 0.99 vs 0.95 buys ~4 GiB more KV. Safe only because the graph profiler is disabled.
• --max-num-batched-tokens 1024 — prefill-chunk sweet spot for DELTA=6: 2048 OOMs, 256 is slower; 1024 cuts 256K prefill ~30% (65 s vs ~95 s).
• --max-num-seqs 1 — single stream by design: the fire floor scales with min(seqs, 4); seqs>1 pushes the 6 GB pool sub-floor and loses full fidelity.
• --kv-cache-dtype fp8 / --block-size 256 — DS4's fp8_ds_mla attention asserts fp8 KV (compact ~1.8 KB/token).
• MTP — off (--speculative-config omitted): MTP's (1+n_spec)=3 triples the fire floor → forces the pool sub-floor (+110% token inflation → slower, 58 vs 75 tok/s, and degraded), and blocks min_p.
• min_p=0.05 (sampling) — filters low-probability wrong tokens → ~+1 rung on the hardest coding rungs. Only possible with MTP off; ≥ 0.10 over-filters.

Measured: ~75 tok/s single-stream at 262144 ctx, VRAM ~97 GB (fills the card), full-fidelity FP4 fires (sub-floor cleared).

Config B — throughput / concurrency (no pool)

Pure 2-bit base, no FP4 pool — moet_store is not needed. VLLM_MOE_W2_DELTA_GB=0 frees the whole card for KV + activations; --max-num-batched-tokens 2048 + a small --max-model-len 32768 batch to saturate the GPU. Whether to enable MTP depends on how many parallel streams you serve (see below).

docker run -d --name ds4-throughput --gpus '"device=0"' --network host --ipc host --shm-size 64g \
  -v /path/to/DeepSeek-V4-Flash:/model:ro \
  -v /data/ds4-cache/planescache:/planescache \
  -e VLLM_MOE_W2=1 \
  -e VLLM_MOE_W2_PLANES_CACHE=/planescache \
  -e VLLM_MOE_W2_DELTA_GB=0 \
  vllm-moet-sm120:v024 \
  --model /model --served-model-name deepseek-v4-flash --trust-remote-code \
  --kv-cache-dtype fp8 --block-size 256 --max-model-len 32768 \
  --gpu-memory-utilization 0.95 --max-num-batched-tokens 2048 \
  --tokenizer-mode deepseek_v4 --no-scheduler-reserve-full-isl \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","custom_ops":["all"]}' \
  --port 8000 \
  --max-num-seqs 256                       # MTP OFF → max aggregate throughput (default here)

# --- Low-latency / few-stream variant: MTP k=2 (swap the last line for these two) ---
#   --max-num-seqs 192 \
#   --speculative-config '{"method":"deepseek_mtp","num_speculative_tokens":2}'

MTP is a concurrency knob. Aggregate decode (tok/s) vs concurrent streams C (DELTA=0, ctx 32K):

C (streams)	1	8	32	64	96	128	192	256
MTP k=2 (seqs 192)	115	234	442	1122	—	858	927	—
MTP off (seqs 256)	89	220	513	752	1097	1368	1748	2085

• Few streams / latency-sensitive — up to ~64 concurrent → MTP k=2. 30% faster single-stream (115 vs 89 tok/s), and aggregate peaks at **1122 tok/s @ C=64**.
• High concurrency — ~96 streams and up → MTP off. Once the batch saturates the GPU, MTP's draft+verify is wasted compute in a diverse batch (little acceptance), so it collapses past its C=64 peak (927 @ C=192 vs 1748 off); MTP-off scales monotonically to ~2085 @ C=256 — the only way to reach the ~2000 tok/s headline.

With no pool there is no gate firing to destabilise attention, so Config B is also the more reliable long-context retrieval path.

Context size & needle-in-a-haystack retrieval

A unique code is planted at early (depth 0.05), middle (0.50) and late (0.95) positions and queried; the recommended length is the largest where all three depths retrieve reliably (16 samples/cell, 2 reps × N=8), and the absolute-max is the largest --max-model-len that still boots (KV fits).

length	needle pass rate	note
192K	~92%	robust at all depths
256K	~90%	recommended — reliable across depths; majority-vote best-of-N recovers to ~100%
288K	~75%	marginal — the early-depth needle degrades and best-of-N no longer recovers it

• Recommended context: --max-model-len 262144 (256K) for both configs — the robust ceiling.
• Config A (FP4 pool) degrades long-context retrieval above ~256K. The pool's async gate/fire perturbs attention; in a controlled A/B at 300K where only the pool differs, the pool config misses the needle most of the time while the no-pool config retrieves it consistently. Keep Config A ≤ 256K for dependable recall; for retrieval-heavy long-context work use Config B (no pool), which stays reliable to ~300K.
• Absolute-max — boots for capacity, but retrieval is NOT reliable there. Config A --max-model-len 913920 (~914K), Config B --max-model-len 1048576 (1M). ⚠️ capacity ceilings only: above 256K you cannot trust retrieval near the start of the context even with best-of-N.
• ⚠️ Single-run needle numbers on this engine are noisy (the pass rate swings run-to-run), so use best-of-N / repeated sampling for anything beyond 256K.

Acknowledgements

Huge thanks to @kacper-daftcode, author of the original vLLM-Moet — the SM120 2-bit routed-expert MoE engine that makes single-GPU DeepSeek-V4-Flash (and GLM-5.2 / Kimi-K2.7) possible in the first place. All the hard work — the 2-bit SASS GEMMs, the FP4 delta pool, the tiered store — is theirs.

This repo just packages a prebuilt cache + the single-card serving configs; the moe_w2 loader-skip and these configs live in the fork branch anoane/vLLM-Moet @ feat/moe-w2-loader-skip.

Base model: deepseek-ai/DeepSeek-V4-Flash · Engine: kacper-daftcode/vLLM-Moet · SM120 (RTX 6000 Pro / RTX 5090).