--- license: mit library_name: transformers pipeline_tag: image-text-to-text base_model: deepseek-ai/DeepSeek-V4.1-Flash base_model_relation: quantized tags: - nvfp4 - fp4 - deepseek - moe - multimodal - libertai --- # DeepSeek-V4.1-Flash-NVFP4 > **🚧 Coming soon — this repository is a placeholder.** The weights are being built > right now and will land here shard by shard. The numbers below marked ⬜ are not > measured yet. Everything marked ✅ is already verified against the source checkpoint. > Watch this repo; the card is updated as each piece is uploaded. An FP4 re-pack of [`deepseek-ai/DeepSeek-V4.1-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) by [LibertAI](https://libertai.io). ## ⚠️ Read this first: V4.1-Flash already ships in 4-bit If you came here expecting the usual ~70% NVFP4 cut, **it does not exist for this model.** DeepSeek quantized V4.1-Flash themselves. From the upstream `config.json`: ```json "quantization_config": { "quant_method": "fp8", "weight_block_size": [32, 32], "scale_fmt": "ue8m0", "expert_dtype": "fp4" } ``` ✅ Measured across all 48 upstream shards / 96,085 tensors (475.2 GiB): | component | size | share | dtype as shipped | |---|---:|---:|---| | routed experts | 268.9 GiB | 56.6% | **E2M1 packed 2/byte + E8M0 scales, block 1×32** | | Engram tables | 189.1 GiB | 39.8% | E4M3 + E8M0 scales, block 1×32 | | MTP (3 layers) | 7.4 GiB | 1.6% | E2M1, as above | | attention / dense | 4.9 GiB | 1.0% | E4M3, block 32×32 | | embeddings, head, vision | 4.0 GiB | 0.8% | BF16 | This is the break from V4-Flash. `DeepSeek-V4-Flash-0731` shipped FP8 block-128×128 with no FP4 anywhere — which is why the NVFP4 conversions of *that* model were worth ~2×. Here the 57% of the checkpoint that is routed experts is already at 4 bits, so quantizing it again buys nothing. ## What this repo actually does Two things, and it is worth being precise about which is a format change and which is a quantization. ### 1. Experts: MXFP4 → NVFP4, losslessly ✅ The upstream experts are MX-style: 32-element blocks with a power-of-two E8M0 scale. NVFP4 is 16-element blocks with an E4M3 scale plus an FP32 per-tensor global scale. The transcode is **bit-exact**, and for a structural reason worth stating: NVFP4's 16-element blocks are a strict refinement of the source's 32-element blocks, so both NVFP4 half-blocks inherit one source scale `s`. Emit the nibbles **unchanged** and force `weight_scale × weight_scale_2 == s`, and every reconstructed value is identical to the source. E8M0 scales are powers of two, and E4M3 represents 2^k exactly for k ∈ [−9, 8], so the identity holds for every block whose scale lands in that window — the global scale is chosen per tensor to centre it. ⬜ Fraction of blocks outside the window: to be reported here per tensor (expected 0). ✅ Verified on shard 4: 1,152 expert weights transcoded, and an independent reconstruction from both representations found **0 mismatches over 2.36 billion elements, `max|delta| = 0` exactly**, with **0 of 849,346,560 block scales** falling outside E4M3's window. **This costs size.** NVFP4 is 4 + 8/16 = 4.5 bits/weight against MXFP4's 4 + 8/32 = 4.25, so the experts *grow*: shard 4 went from 7,389,759,032 to 7,814,559,376 bytes, **+5.75%**. Take this variant only if your engine's Blackwell MoE kernels want NVFP4 and have no MX path — it is a compatibility shim, not a compression win, and we would rather say so than let you find out from a disk-full error. To be clear about prior art: this losslessness is not something we invented, it is what any careful implementation of this transcode gets, for the structural reason above. We checked [`s-zaizen/DeepSeek-V4.1-Flash-NVFP4`](https://huggingface.co/s-zaizen/DeepSeek-V4.1-Flash-NVFP4), published the same day, by reconstructing `layers.1.ffn.experts.0.w1` from both their checkpoint and the source: **their nibbles are identical and all 11,796,480 elements match exactly too.** Their repo is **491.1 GiB — larger than the 475.2 GiB original** — because the expert transcode grows the file and the Engram tables are left at FP8. That is the gap this repo is trying to close, and it is the only thing that meaningfully separates the two. ### 2. Engram: FP8 → FP4, the one real size lever ⬜ The Engram conditional-memory tables are 189.1 GiB — 39.8% of the checkpoint — and they are the only large block DeepSeek left at 8 bits. Two tables of ~384M rows × 256, at E4M3 with block-32 E8M0 scales. We take them to E2M1 **keeping the source's own block-32 / E8M0 scale layout**, so a table row ends up stored exactly the way DeepSeek stores an expert weight and any kernel that can dequantize an expert can dequantize the table. Scales are recomputed as a power of two ≥ amax/6 rather than reused, so no block clips. **189.1 → ~97 GiB.** ⬜ Accuracy impact is a real question and is being measured — this is a lossy step on a memory table, unlike step 1. Mean cosine similarity per row will be reported here before this variant is recommended for anything. Engram is a hashed n-gram **gather**, not a GEMM — roughly 48 rows × 256 B ≈ 12 KB per token — so it dequantizes cheaply on lookup and is NVMe-friendly if you offload it. Upstream's own `convert.py` already shards `engram.embed` along dim 0 across ranks. ### Net effect ⬜ | | upstream | s-zaizen NVFP4 | this repo | |---|---:|---:|---:| | routed experts + MTP | 276.3 GiB | ~292 GiB | ~292 GiB | | Engram | 189.1 GiB | 189.1 GiB | ~97 GiB | | everything else | 8.9 GiB | 8.9 GiB | 8.9 GiB | | **total** | **475.2 GiB** | **491.1 GiB** ✅ | **~398 GiB** ⬜ | The Engram column is the whole point. If you need NVFP4 experts and do not care about Engram, s-zaizen's repo is a perfectly good checkpoint and was there first. ## Hardware notes ⬜ Even at ~398 GiB this is a large model, and the Engram offload does not change the resident-weight picture as much as you would hope: the backbone alone (experts + MTP + attention + vision + embeddings) is ~301 GiB / 323 GB resident with the entire Engram table pushed to NVMe. Two 120 GB unified-memory boxes cannot hold it. A 4×B200 or 8×96 GB node can. ⬜ A **REAP-pruned variant** targeting 2-node 120 GB-class hardware is in progress and will be published separately. ## About the base model DeepSeek-V4.1-Flash is a multimodal MoE with a **552B-parameter backbone** plus **196B of sparsely-accessed Engram memory**, activating **8B parameters per token during prefill and 16B during decode**, with a **1M-token** context. It uses a **Causal Encoder-Decoder** layout (40 layers = a 20-layer causal encoder feeding a 20-layer decoder, so the decoder's global KV is projected from final encoder states), **Compressed Sparse Attention 2** with per-layer Full/Reindex/Reuse modes and a hierarchical sparse indexer, **FP4 main KV caching** (E2M1, one E4M3 scale per 16 channels) for a global KV footprint of **890 bytes/token**, Single-Pass mHC, and **DSpark** speculative decoding. 384 routed experts + 1 shared, 6 routed per token. Reasoning effort is continuously controllable from 1 to 100. See the [technical report](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf) and the upstream card for evaluations. We have run none of our own yet. ## Engine support ⬜ As of 2026-09-10 no released engine serves `deepseek_v41`. vLLM has open PRs ([#56214](https://github.com/vllm-project/vllm/pull/56214), [#56228](https://github.com/vllm-project/vllm/pull/56228), [#56227](https://github.com/vllm-project/vllm/pull/56227), [#56208](https://github.com/vllm-project/vllm/pull/56208)) filed the morning the model dropped, and SGLang has merged a cookbook plus initial support. Serving instructions land here once we have actually served it. ## Provenance Built with [`dsv41_fp4_stream.py`](./dsv41_fp4_stream.py), a shard-streaming re-packer that never holds more than one shard in RAM. The E2M1 table and nibble order are taken verbatim from the checkpoint's own `inference/convert.py`, not reconstructed from the spec. License: MIT, inherited from the base model.