# Serving DeepSeek-V4-Flash + EAGLE-3.1 (vLLM)

This draft head **requires** a small vLLM overlay because upstream `deepseek_v4` does
not implement the EAGLE-3 aux-capture interface. Without the overlay, serve fails with
`Model does not support EAGLE3 interface`.

## Requirements

- NVIDIA **Hopper or Blackwell** GPU (DeepSeek V4 is not supported on A100)
- [vLLM](https://github.com/vllm-project/vllm) **≥ 0.21.0** (validated on **0.22.0**)
- Target: [`deepseek-ai/DeepSeek-V4-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)
- Draft: this repository

## Install overlay (required)

Copy the overlay from the Maniac training repo and install it before importing vLLM:

```bash
git clone https://github.com/ManiacIncorporated/maniac-desktop.git
OVERLAY=maniac-desktop/training/eagle3-v4flash/patches/vllm_overlay
sudo mkdir -p /opt/eagle3-vllm-overlay
sudo cp "$OVERLAY/deepseek_v4_eagle3_aux.py" "$OVERLAY/eagle3_draft_kv_cache.py" /opt/eagle3-vllm-overlay/
SITE=$(python3 -c "import site; print(site.getsitepackages()[0])")
echo 'import sys; sys.path.insert(0, "/opt/eagle3-vllm-overlay"); import deepseek_v4_eagle3_aux as _o; _o.install()' \
  | sudo tee "$SITE/zz_eagle3_v4_vllm_overlay.pth"
```

`eagle3_draft_kv_cache.py` is imported automatically by `deepseek_v4_eagle3_aux.install()`.

Environment:

```bash
export EAGLE3_DRAFT_KV_CACHE_DTYPE=auto
export VLLM_ALLOW_INSECURE_SERIALIZATION=1
```

## Python API (minimal)

```python
from vllm import LLM, SamplingParams

llm = LLM(
    model="deepseek-ai/DeepSeek-V4-Flash",
    trust_remote_code=True,
    tensor_parallel_size=4,
    enable_expert_parallel=True,
    enforce_eager=True,
    kv_cache_dtype="fp8",
    gpu_memory_utilization=0.6,
    max_model_len=8192,
    speculative_config={
        "method": "eagle3",
        "model": "ManiacLabs/DeepSeek-V4-Flash-EAGLE3.1",
        "num_speculative_tokens": 3,
    },
)

prompts = ["Explain how a hash map works in two sentences."]
outs = llm.generate(prompts, SamplingParams(temperature=0.0, max_tokens=128))
print(outs[0].outputs[0].text)
```

## Notes

- **`enforce_eager=True`** is required when using the runtime overlay (forward hooks are
  not `torch.compile`-safe on DeepSeek V4).
- **`kv_cache_dtype="fp8"`** is required for the target; the overlay decouples draft KV
  to `auto` via `EAGLE3_DRAFT_KV_CACHE_DTYPE`.
- Use the **DeepSeek chat template** for production prompts; raw-string benchmarks
  understate acceptance vs held-out eval.
- Greedy EAGLE output matches target greedy **within the same run**; cross-run FP8 +
  expert-parallel noise can make text diffs look worse than token-level parity.

## Reproduce wall-clock benchmark

From the Maniac repo (Modal GPU):

```bash
EAGLE3_VLLM_IMAGE=vllm/vllm-openai:v0.22.0 EAGLE3_EVAL_GPU=B200:4 \
  python3 -m modal run training/eagle3-v4flash/vllm_eagle_speedup_eval.py::speedup_run \
  --draft-path /ckpt/v4-flash-eagle3.1-genv3-blend/draft_model_hf
```

Or point `--draft-path` at a local checkout of this HF repo after download.

## Upstream status

If vLLM wires `deepseek_v4` for EAGLE-3 natively, the overlay can be removed. Track:
`training/eagle3-v4flash/patches/vllm_overlay/README.md` in the Maniac repo.