--- pipeline_tag: image-text-to-text base_model: - deepseek-ai/DeepSeek-V4-Flash-Vision-Exp license: mit library_name: Model Optimizer tags: - deepseek-v4 - vision - multimodal - modelopt - nvfp4 - quantized - dgx-spark --- # DeepSeek-V4-Flash-Vision-Exp-NVFP4 This is a mixed-precision derivative of `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` at revision `86f746b36186f0e567729a5c06a8c918caba82a9`. Only the regular routed MoE experts are represented as NVFP4 W4A4 with group size 16. Attention, the router gate, shared experts, embeddings, the language model head, MTP/DSpark blocks, and the vision tower retain their source checkpoint formats. The source expert weights are MXFP4. They are converted with NVIDIA ModelOpt's DeepSeek V4 lossless MXFP4-to-NVFP4 cast; activation input scales are calibrated with 64 samples from each of `cnn_dailymail` and `nvidia/Nemotron-Post-Training-Dataset-v2`, sequence length 512, MP=2 on two NVIDIA DGX Spark systems. The calibration-only tokenizer copy uses left padding, as recommended by ModelOpt; the published source tokenizer files remain unchanged. Model loading uses a tensor-at-a-time wrapper around the official PTQ entrypoint to avoid materializing a second 86 GB checkpoint in DGX Spark unified memory. For calibration on GB10 (SM121), the wrapper uses an isolated copy of the source inference code with the sparse-attention tile block reduced from 64 to 32: the source kernel requests 104,448 bytes of dynamic shared memory while GB10 permits 101,376 bytes. This calibration-only compatibility change does not modify the source or published inference files. ModelOpt is pinned to `029c67f27e67088fb19ac0a9af241dc2bc740650`. The source reference runtime also declares `fast_hadamard_transform` without a version constraint. Because that extension is absent from the pinned ARM64 runtime, the same isolated calibration copy supplies a normalized seven-stage PyTorch BF16 butterfly for the model's 128-wide indexer rotations. Its output is checked against the dense Walsh-Hadamard definition before calibration. ## Status - Structural validation: passed for all 48 shards. - Lossless routed-expert weight cast: passed for 33024 tensors and 8657043456 NVFP4 blocks. - Runtime support: validated on two DGX Spark systems with the vLLM fork at `70b65679e6f10b81e42e030d767ae9c4e7acfb4d` and tensor parallelism 2. - Generation validation: passed for both text-only and image inputs through the OpenAI-compatible API. The original image smoke test correctly identified an ear of corn; the final long-context runtime also identified a robot icon. Both completed with `finish_reason=stop`. - Quality and throughput A/B: passed under the conditions below. The selected runtime remains within the 3-point GSM8K gate and improves decode speed. - Long-context qualification: passed 12/12 sequential gates from 32K through 250K in both growing-prefix and isolated modes. Every request returned its exact marker placed near the beginning, stopped normally, and passed an immediate short post-probe. Do not infer benchmark equivalence from the lossless weight cast alone: activation quantization remains a calibrated change. The results below are bounded to the stated sample and serving configuration. ## Measured results | Comparison | Quality baseline | Quality candidate | Delta | Throughput baseline | Throughput candidate | Delta | |---|---:|---:|---:|---:|---:|---:| | Upstream checkpoint -> NVFP4 S0, GSM8K | 97.0% | 97.0% | +0.0 pt | 11.512 tok/s | 12.236 tok/s | +0.725 tok/s | | NVFP4 S0 -> selected T3, GSM8K | 97.0% | 97.0% | +0.0 pt | 12.236 tok/s | 18.655 tok/s | +6.419 tok/s | | NVFP4 S0 -> selected T3, fixed serving workload | n/a | n/a | n/a | 14.358 tok/s | 23.734 tok/s | +9.376 tok/s | | Selected T3 -> long-context T5, GSM8K | 97.0% | 96.0% | -1.0 pt | 18.655 tok/s | 18.251 tok/s | -0.405 tok/s | | Selected T3 -> long-context T5, fixed serving workload | n/a | n/a | n/a | 23.734 tok/s | 23.692 tok/s | -0.042 tok/s | | T5 -> DSpark T6 k=3, GSM8K | 96.0% | 94.0% | -2.0 pt | 18.251 tok/s | 14.995 tok/s | -3.256 tok/s | | T5 -> DSpark T6 k=3, fixed serving workload | n/a | n/a | n/a | 23.692 tok/s | 16.343 tok/s | -7.349 tok/s | | T5 -> DSpark T7 k=5, GSM8K | 96.0% | 96.0% | +0.0 pt | 18.251 tok/s | 14.769 tok/s | -3.482 tok/s | | T5 -> DSpark T7 k=5, fixed serving workload | n/a | n/a | n/a | 23.692 tok/s | 14.653 tok/s | -9.039 tok/s | GSM8K uses the first 100 official vLLM test questions, 8-shot prompting, `max_tokens=256`, temperature 0, seed 42, and concurrency 1. The fixed serving workload requests input length 256 and output length 128 for five prompts after one warmup at concurrency 1; chat rendering produced 339 actual input tokens. The throughput value is the median of three runs. Results were measured on two DGX Spark systems with TP=2 and the pinned vLLM fork above. DSpark k=3 and k=5 remained inside the 3-point quality gate, but their median draft-token acceptance rates were only 2.00% and 1.67%. Both candidates were rejected for performance, and the non-speculative T5 recipe remains selected. The DSpark-only measurements used fork revision `1e12c3e9db9c5f156ca8abe75e1340a5f439fd49`; T5 remains pinned to the non-speculative revision stated above. Its winner recheck produced 23.590 tok/s on one fixed-workload run and passed an isolated 249,935-token prompt with exact marker retrieval and an immediate post-context probe. The selected recipe is `vllm-optimized-sm121-tp2.yaml`: prefix caching and vLLM compile are disabled, while Breakable CUDA Graph is enabled for full decode only. It serves a 262,144-token context ceiling and explicitly reserves 12 GiB for KV cache. At startup vLLM normalized the requested `fp8` cache to DeepSeek's `fp8_ds_mla` layout and profiled 2,298,856 KV-cache tokens, or 8.77 times one 262,144-token request. The active sparse-MLA backend does not support an NVFP4 KV-cache dtype, so no unsupported lower-precision cache was forced. The long-context harness calibrated each request with the runtime tokenizer and reserved 64 tokens within each boundary. Growing-prefix passed 6/6 boundaries (32,768, 65,536, 100,000, 131,072, 200,000, and 250,000); isolated prompts passed the same 6/6 boundaries with unique markers. At the largest boundary the observed prompt lengths were 249,934 and 249,935 tokens respectively. ## Published optimization artifacts - [Selected two-node vLLM recipe](recipes/vllm-optimized-sm121-tp2.yaml) - [Spark Arena v2 benchmark summary](benchmarks/README.md) - [Canonicalized Spark Arena CSV](benchmarks/spark-arena-v2.csv) - [Canonicalized Spark Arena JSON](benchmarks/spark-arena-v2.json) The benchmark files preserve the measured values from completed submission `sub1788319379140`. Only the model identity was canonicalized from the local runtime path to this Hugging Face repository ID; no timing or throughput field was rewritten. ## License The source repository declares the MIT License. The original license and ancillary files are retained in this derivative.