# HumanEval Evaluation: DeepSeek-V4-Flash-DSpark (top_k=4 vs top_k=6)

## Setup

All evaluations use the following configuration:

- **Inference mode**: `thinking` (with `<think>` reasoning tags)
- **`max_tokens`**: 4096
- **vLLM version**: 0.23.0
- **GPU**: single NVIDIA B300 SXM6 (sm_103a, 275 GB VRAM)
- **DeepGEMM**: enabled (all 2562 kernels JIT-compiled at warmup)
- **KV cache dtype**: `fp8`
- **Dataset**: OpenAI HumanEval (164 Python coding problems)
- **Sampling**: temperature=0.0, top_p=0.95
- **Evaluation**: pass@1 measured by executing generated code against hidden test cases
- **Backend**: `flashinfer::trtllm_fp4_block_scale_moe` for MXFP4 MoE, `DeepGemmFp8BlockScaledMM` for FP8 linear

### Model Configuration Detail

| Setting | `num_experts_per_tok` | `tid2eid` handling |
|---------|----------------------|--------------------|
| top_k=4 | 4 | Sliced from `[vocab_size, 6]` → `[vocab_size, 4]` (first 4 columns retained) |
| top_k=6 | 6 | Loaded as-is from checkpoint (`[vocab_size, 6]`) |

The checkpoint stores `tid2eid` weights with shape `[129280, 6]` for the first 3 hash-based MoE routing layers. When `num_experts_per_tok=4`, weights are sliced to match the parameter shape.

## Results

### Overall Pass@1

| Configuration | Pass@1 | Generation Time | Per-Sample Time |
|---------------|--------|-----------------|-----------------|
| **DSpark top_k=4** | 155/164 = **94.51%** | **55.78 s** | **0.34 s** |
| **DSpark top_k=6** | 157/164 = **95.73%** | 64.07 s | 0.39 s |
| *Δ (6−4)* | *+1.22%* | *+8.29 s (+14.9%)* | |

### Comparison with DeepSeek-V4-Flash-4E

| Model | Config | Pass@1 | Generation Time |
|-------|--------|--------|-----------------|
| **DSpark** | top_k=4 | **94.51%** | 55.78 s |
| **DSpark** | top_k=6 | **95.73%** | 64.07 s |
| **4E** | top_k=4 | **95.73%** | 56.83 s |
| **4E** | top_k=6 | **95.73%** | 64.06 s |

Key observations:
- DSpark top_k=6 ties 4E at **95.73%** (157/164) — identical accuracy.
- DSpark top_k=4 is slightly lower at **94.51%** (155/164), missing **2 extra problems** compared to top_k=6.
- Generation time scales similarly across both models (~14–15% slower at top_k=6).

### Failure Analysis

**DSpark top_k=4 failures (9 total):**

| Task ID | Entry Point | Error |
|---------|------------|-------|
| HumanEval/0 | `has_close_elements` | SyntaxError |
| HumanEval/38 | `decode_cyclic` | NameError |
| HumanEval/50 | `decode_shift` | NameError |
| HumanEval/94 | `skjkasdkd` | SyntaxError |
| HumanEval/116 | `sort_array` | SyntaxError |
| HumanEval/129 | `minPath` | SyntaxError |
| HumanEval/132 | `is_nested` | SyntaxError |
| HumanEval/145 | `order_by_points` | SyntaxError |
| HumanEval/160 | `do_algebra` | SyntaxError |

**DSpark top_k=6 failures (7 total):**

| Task ID | Entry Point | Error |
|---------|------------|-------|
| HumanEval/0 | `has_close_elements` | SyntaxError |
| HumanEval/38 | `decode_cyclic` | NameError |
| HumanEval/65 | `circular_shift` | AssertionError |
| HumanEval/129 | `minPath` | SyntaxError |
| HumanEval/132 | `is_nested` | SyntaxError |
| HumanEval/145 | `order_by_points` | SyntaxError |
| HumanEval/160 | `do_algebra` | SyntaxError |

**Shared failures (5 problems):**
`HumanEval/0`, `HumanEval/38`, `HumanEval/129`, `HumanEval/132`, `HumanEval/145`, `HumanEval/160` — these 6 problems fail in **both** configurations. `HumanEval/38` is a NameError (missing `decode_cyclic` in namespace); the remaining 5 are SyntaxError (the model generated syntactically invalid code).

**Unique failures:**

| Task | Fails in | Description |
|------|----------|-------------|
| HumanEval/50 (`decode_shift`) | top_k=4 only | NameError |
| HumanEval/94 (`skjkasdkd`) | top_k=4 only | SyntaxError |
| HumanEval/116 (`sort_array`) | top_k=4 only | SyntaxError |
| HumanEval/65 (`circular_shift`) | top_k=6 only | AssertionError (logic error, but close) |

### Speed Analysis

| Phase | top_k=4 | top_k=6 | Difference |
|-------|---------|---------|------------|
| Model loading | 21.92 s | 22.12 s | +0.20 s |
| Engine initialization | 179.73 s | 180.46 s | +0.73 s |
| Generation | **55.78 s** | **64.07 s** | **+8.29 s (+14.9%)** |

top_k=6 activates 50% more experts per token (6 vs 4), increasing compute by ~50% for the MoE feed-forward layers. The observed 14.9% generation time increase is lower than the naive 50% estimate, likely because the MoE computation is partially bandwidth-bound on the B300 GPU.

## Conclusion

1. **DSpark top_k=6 matches 4E at 95.73% Pass@1** on HumanEval — identical accuracy to the 4E reference.
2. **DSpark top_k=4 achieves 94.51%**, a minor 1.22% drop vs top_k=6, losing 2 extra problems (both due to syntax/code quality issues rather than logic errors).
3. **Generation is ~13–15% faster at top_k=4** (55.78 s vs 64.07 s), consistent with fewer activated experts.
4. For code generation tasks where maximum accuracy is critical, **top_k=6 is recommended** for DSpark. For throughput-sensitive workloads, top_k=4 offers a reasonable speed-accuracy trade-off.

## Raw Data

- `humaneval_results_dspark_topk4.jsonl`: Full results for top_k=4 (155/164 passed)
- `humaneval_results_dspark_topk6.jsonl`: Full results for top_k=6 (157/164 passed)
- `evaluate_humaneval.py`: Evaluation script

## HumanEval MMLU-Pro Cross-Comparison

| Model | Config | HumanEval | MMLU-Pro |
|-------|--------|-----------|----------|
| DSpark | top_k=4 | 94.51% | 42.30% |
| DSpark | top_k=6 | 95.73% | 39.27% |
| 4E | top_k=4 | 95.73% | 41.75% |

Unlike MMLU-Pro (where top_k=4 significantly outperformed top_k=6 by +3.03%), on HumanEval, **top_k=6 slightly outperforms top_k=4** (+1.22%). This suggests that code generation benefits from the additional expert diversity, while multiple-choice knowledge retrieval does not.