File size: 3,298 Bytes
c7e88d7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | # MMLU-Pro Evaluation: DeepSeek-V4-Flash-DSpark (top_k=4 vs top_k=6)
## Setup
All evaluations use the following configuration:
- **Inference mode**: `chat` (no `<think>` prefix)
- **`max_tokens`**: 20 (single-letter answer extraction)
- **vLLM version**: 0.23.0
- **GPU**: single NVIDIA B300 SXM6 (sm_103a, 275 GB VRAM)
- **DeepGEMM**: enabled (`VLLM_USE_DEEP_GEMM` unset, all 2562 kernels JIT-compiled at warmup)
- **KV cache dtype**: `fp8`
- **Dataset**: MMLU-Pro (12,032 questions, A–J categories)
- **Backend**: `flashinfer::trtllm_fp4_block_scale_moe` for MXFP4 MoE, `DeepGemmFp8BlockScaledMM` for FP8 linear
### Model Configuration Detail
Both runs use the **same checkpoint files**. The only difference is the `num_experts_per_tok` parameter and how the `tid2eid` (hash-based expert routing table) weight tensors are handled:
| Setting | `num_experts_per_tok` | `tid2eid` handling |
|---------|----------------------|--------------------|
| top_k=4 | 4 | Sliced from `[vocab_size, 6]` → `[vocab_size, 4]` (first 4 columns retained) |
| top_k=6 | 6 | Loaded as-is from checkpoint (`[vocab_size, 6]`) |
The checkpoint stores `tid2eid` weights with shape `[129280, 6]` for the first 3 hash-based MoE routing layers. When `num_experts_per_tok=4`, the model parameter shape is `[129280, 4]`; the weight is sliced to match.
## Results
### Overall Accuracy
| Configuration | Accuracy | Correct / Total | Generation Time | Engine Init Time |
|---------------|----------|-----------------|-----------------|------------------|
| **top_k=4** | **42.30%** | 5,090 / 12,032 | **73.0 s** | **167.9 s** |
| **top_k=6** | **39.27%** | 4,725 / 12,032 | 89.0 s | 186.9 s |
| *Δ (4−6)* | *+3.03%* | *+365* | *−16.0 s* | *−19.0 s* |
Using `num_experts_per_tok=4` (with tid2eid slicing) gives a **+3.03 percentage point** lift over `num_experts_per_tok=6`, with faster generation and engine initialization.
### Per-Category Breakdown (top_k=4)
Only the top_k=4 run recorded per-category metrics:
| Category | Accuracy |
|----------|----------|
| biology | 492/717 = 68.6% |
| business | 325/789 = 41.2% |
| chemistry | 214/1132 = 18.9% |
| computer science | 194/410 = 47.3% |
| economics | 557/844 = 66.0% |
| engineering | 262/969 = 27.0% |
| health | 483/818 = 59.0% |
| history | 191/381 = 50.1% |
| law | 369/1101 = 33.5% |
| math | 393/1351 = 29.1% |
| other | 507/924 = 54.9% |
| philosophy | 263/499 = 52.7% |
| physics | 291/1299 = 22.4% |
| psychology | 549/798 = 68.8% |
### Comparison: DSpark vs 4E (both top_k=4)
| Model | Accuracy | Generation Time |
|-------|----------|-----------------|
| **DeepSeek-V4-Flash-DSpark** | **42.30%** | 73.0 s |
| **DeepSeek-V4-Flash-4E** | **41.75%** | 122.0 s |
| *Δ (DSpark−4E)* | *+0.55%* | *−49.0 s* |
DSpark at top_k=4 slightly outperforms the 4E reference model (+0.55%) while generating answers **40% faster** (73 s vs 122 s).
## Key Takeaway
**`num_experts_per_tok=4` is the correct setting for DeepSeek-V4-Flash-DSpark on MMLU-Pro.** Despite the checkpoint storing 6-column `tid2eid` weights (originally for top_k=6 evaluation), forcing top_k=6 degrades accuracy by ~3 percentage points. The canonical top_k=4 with sliced weight loading achieves the best accuracy (42.30%) and ties or exceeds the comparable 4E model.
|