DeepSeek-V4-Flash-Vision-Exp-NVFP4

This is a mixed-precision derivative of deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at revision 86f746b36186f0e567729a5c06a8c918caba82a9.

Only the regular routed MoE experts are represented as NVFP4 W4A4 with group size 16. Attention, the router gate, shared experts, embeddings, the language model head, MTP/DSpark blocks, and the vision tower retain their source checkpoint formats.

The source expert weights are MXFP4. They are converted with NVIDIA ModelOpt's DeepSeek V4 lossless MXFP4-to-NVFP4 cast; activation input scales are calibrated with 64 samples from each of cnn_dailymail and nvidia/Nemotron-Post-Training-Dataset-v2, sequence length 512, MP=2 on two NVIDIA DGX Spark systems. The calibration-only tokenizer copy uses left padding, as recommended by ModelOpt; the published source tokenizer files remain unchanged. Model loading uses a tensor-at-a-time wrapper around the official PTQ entrypoint to avoid materializing a second 86 GB checkpoint in DGX Spark unified memory. For calibration on GB10 (SM121), the wrapper uses an isolated copy of the source inference code with the sparse-attention tile block reduced from 64 to 32: the source kernel requests 104,448 bytes of dynamic shared memory while GB10 permits 101,376 bytes. This calibration-only compatibility change does not modify the source or published inference files. ModelOpt is pinned to 029c67f27e67088fb19ac0a9af241dc2bc740650.

The source reference runtime also declares fast_hadamard_transform without a version constraint. Because that extension is absent from the pinned ARM64 runtime, the same isolated calibration copy supplies a normalized seven-stage PyTorch BF16 butterfly for the model's 128-wide indexer rotations. Its output is checked against the dense Walsh-Hadamard definition before calibration.

Status

• Structural validation: passed for all 48 shards.
• Lossless routed-expert weight cast: passed for 33024 tensors and 8657043456 NVFP4 blocks.
• Runtime support: validated on two DGX Spark systems with the vLLM fork at 70b65679e6f10b81e42e030d767ae9c4e7acfb4d and tensor parallelism 2.
• Generation validation: passed for both text-only and image inputs through the OpenAI-compatible API. The original image smoke test correctly identified an ear of corn; the final long-context runtime also identified a robot icon. Both completed with finish_reason=stop.
• Quality and throughput A/B: passed under the conditions below. The selected runtime remains within the 3-point GSM8K gate and improves decode speed.
• Long-context qualification: passed 12/12 sequential gates from 32K through 250K in both growing-prefix and isolated modes. Every request returned its exact marker placed near the beginning, stopped normally, and passed an immediate short post-probe.

Do not infer benchmark equivalence from the lossless weight cast alone: activation quantization remains a calibrated change. The results below are bounded to the stated sample and serving configuration.

Measured results

Comparison	Quality baseline	Quality candidate	Delta	Throughput baseline	Throughput candidate	Delta
Upstream checkpoint -> NVFP4 S0, GSM8K	97.0%	97.0%	+0.0 pt	11.512 tok/s	12.236 tok/s	+0.725 tok/s
NVFP4 S0 -> selected T3, GSM8K	97.0%	97.0%	+0.0 pt	12.236 tok/s	18.655 tok/s	+6.419 tok/s
NVFP4 S0 -> selected T3, fixed serving workload	n/a	n/a	n/a	14.358 tok/s	23.734 tok/s	+9.376 tok/s
Selected T3 -> long-context T5, GSM8K	97.0%	96.0%	-1.0 pt	18.655 tok/s	18.251 tok/s	-0.405 tok/s
Selected T3 -> long-context T5, fixed serving workload	n/a	n/a	n/a	23.734 tok/s	23.692 tok/s	-0.042 tok/s
T5 -> DSpark T6 k=3, GSM8K	96.0%	94.0%	-2.0 pt	18.251 tok/s	14.995 tok/s	-3.256 tok/s
T5 -> DSpark T6 k=3, fixed serving workload	n/a	n/a	n/a	23.692 tok/s	16.343 tok/s	-7.349 tok/s
T5 -> DSpark T7 k=5, GSM8K	96.0%	96.0%	+0.0 pt	18.251 tok/s	14.769 tok/s	-3.482 tok/s
T5 -> DSpark T7 k=5, fixed serving workload	n/a	n/a	n/a	23.692 tok/s	14.653 tok/s	-9.039 tok/s

GSM8K uses the first 100 official vLLM test questions, 8-shot prompting, max_tokens=256, temperature 0, seed 42, and concurrency 1. The fixed serving workload requests input length 256 and output length 128 for five prompts after one warmup at concurrency 1; chat rendering produced 339 actual input tokens. The throughput value is the median of three runs. Results were measured on two DGX Spark systems with TP=2 and the pinned vLLM fork above.

DSpark k=3 and k=5 remained inside the 3-point quality gate, but their median draft-token acceptance rates were only 2.00% and 1.67%. Both candidates were rejected for performance, and the non-speculative T5 recipe remains selected. The DSpark-only measurements used fork revision 1e12c3e9db9c5f156ca8abe75e1340a5f439fd49; T5 remains pinned to the non-speculative revision stated above. Its winner recheck produced 23.590 tok/s on one fixed-workload run and passed an isolated 249,935-token prompt with exact marker retrieval and an immediate post-context probe.

The selected recipe is vllm-optimized-sm121-tp2.yaml: prefix caching and vLLM compile are disabled, while Breakable CUDA Graph is enabled for full decode only. It serves a 262,144-token context ceiling and explicitly reserves 12 GiB for KV cache. At startup vLLM normalized the requested fp8 cache to DeepSeek's fp8_ds_mla layout and profiled 2,298,856 KV-cache tokens, or 8.77 times one 262,144-token request. The active sparse-MLA backend does not support an NVFP4 KV-cache dtype, so no unsupported lower-precision cache was forced.

The long-context harness calibrated each request with the runtime tokenizer and reserved 64 tokens within each boundary. Growing-prefix passed 6/6 boundaries (32,768, 65,536, 100,000, 131,072, 200,000, and 250,000); isolated prompts passed the same 6/6 boundaries with unique markers. At the largest boundary the observed prompt lengths were 249,934 and 249,935 tokens respectively.

Published optimization artifacts

• Selected two-node vLLM recipe
• Spark Arena v2 benchmark summary
• Canonicalized Spark Arena CSV
• Canonicalized Spark Arena JSON

The benchmark files preserve the measured values from completed submission sub1788319379140. Only the model identity was canonicalized from the local runtime path to this Hugging Face repository ID; no timing or throughput field was rewritten.

License

The source repository declares the MIT License. The original license and ancillary files are retained in this derivative.