AGENTS.md

# Agent runbook — set up `LordNeel/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8` end-to-end

**For**: an autonomous code agent (Claude Code, Codex, Cursor, Aider, etc.) running on a fresh-ish Linux box with the right hardware. Stop and ask the human only when explicitly told.

**Goal**: a vLLM OpenAI-compatible server on port 8000 serving the MTP-enabled model at the validated 524k profile, with smoke tests passing.

---

## Phase 0 — verify hardware (HARD STOP if not satisfied)

Run `nvidia-smi --query-gpu=name,memory.total --format=csv,noheader` and `nvidia-smi topo -m`.

Validated SKUs (any of these works without code changes):
- 2× **RTX PRO 6000 Blackwell Max-Q Workstation Edition** (96 GB ea, sm_120, no NVLink) ← reference platform
- 2× RTX PRO 6000 Blackwell Server (96 GB ea, sm_120, with NVLink) ← also expected to work
- 2× DGX Spark / GB10 (sm_121)
- 8× H200 SXM (sm_90)

**Stop and ask the human if any of the following is true**:
- Fewer than 2 GPUs with ≥ 96 GB each (TP=2 with 524k KV cache won't fit)
- More than 2 visible GPUs and you'd need to pick which 2 (set `CUDA_VISIBLE_DEVICES` deliberately, don't guess)
- GPU is on a topology where `nvidia-smi topo -m` reports `SYS` (cross-NUMA) — this can cause more comm overhead; not a blocker but flag it

Detect Max-Q-specific quirks: if the GPU name contains "Max-Q" you MUST pass `--disable-custom-all-reduce` later. The serve script already does this.

---

## Phase 1 — system prereqs (no sudo)

Working dir: pick `$HOME/dsv4-local` (the reference layout). All paths in this doc assume that.

```bash
mkdir -p $HOME/dsv4-local
cd $HOME/dsv4-local
```

Required:
- Python 3.12 (system or via miniforge — agent's choice; reference uses miniforge3)
- Driver: NVIDIA 580.x or newer (check `nvidia-smi`)
- gcc/g++ 13.x (system) — used as host compiler for nvcc
- Git, curl, jq, ca-certificates (apt should already have these)

Forbidden without explicit human approval:
- Modifying `/etc`, kernel modules, drivers, or system CUDA
- Installing system packages with sudo (use conda for CUDA toolkit instead)
- Touching `/proc/sys/*` or `nvidia-smi -pl`/`-lgc` (those need sudo)

---

## Phase 2 — local CUDA toolkit via conda (no sudo)

```bash
# 2a. Miniforge3 (only if conda not already available)
test -x $HOME/dsv4-local/miniforge3/bin/conda || (
  curl -fsSLO https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
  bash Miniforge3-Linux-x86_64.sh -b -p $HOME/dsv4-local/miniforge3
)
$HOME/dsv4-local/miniforge3/bin/conda init bash >/dev/null

# 2b. CUDA toolkit 12.9 in a self-contained env
$HOME/dsv4-local/miniforge3/bin/conda create -y -p $HOME/dsv4-local/conda-cuda \
  -c nvidia -c conda-forge cuda-toolkit=12.9 ccache

# 2c. Verify
$HOME/dsv4-local/conda-cuda/bin/nvcc --version | tail -3
```

Common gotcha: conda's default put CUDA headers under `targets/x86_64-linux/include/` instead of `include/`. The vLLM build will fail with "Could NOT find CUDA". Fix by symlinking once:

```bash
CUDA_HOME=$HOME/dsv4-local/conda-cuda
for f in $(ls $CUDA_HOME/targets/x86_64-linux/include 2>/dev/null); do
  test -e $CUDA_HOME/include/$f || ln -s ../targets/x86_64-linux/include/$f $CUDA_HOME/include/$f
done
```

---

## Phase 3 — Python venv + base deps

```bash
mkdir -p $HOME/dsv4-local/venvs
python3.12 -m venv $HOME/dsv4-local/venvs/vllm-dsv4
source $HOME/dsv4-local/venvs/vllm-dsv4/bin/activate
python -m pip install --upgrade pip wheel setuptools
pip install \
  "torch==2.11.0+cu128" --index-url https://download.pytorch.org/whl/cu128
pip install \
  "transformers==5.8.0" \
  "compressed-tensors==0.15.0.1" \
  "safetensors>=0.7" \
  "huggingface_hub>=1.14" \
  hf_transfer \
  "openai" \
  "rich" \
  "datasets"
```

(If conda gcc-14 is on PATH, force system gcc/g++ for the upcoming vLLM build:
`export CC=/usr/bin/gcc CXX=/usr/bin/g++ CUDAHOSTCXX=/usr/bin/g++ NVCC_PREPEND_FLAGS="-ccbin /usr/bin/g++"`.)

---

## Phase 4 — patched vLLM fork

```bash
mkdir -p $HOME/dsv4-local/src
cd $HOME/dsv4-local/src
git clone https://github.com/jasl/vllm.git vllm
cd vllm
# Validated pin. This is itself a cherry-pick of neuralmagic/vllm@f910a73 ("support
# ct quantization" — Kyle Sayers); pinning here means you do NOT need the extra
# cherry-pick step that older versions of this runbook documented.
git checkout b158e5001f097b193b4fa8c4fc7bb32bb4e32e9e

# pasta-paul's packed_modules_mapping patch (adds the class-level dict on
# DeepseekV4ForCausalLM that the cherry-pick references but doesn't define).
curl -fsSL https://raw.githubusercontent.com/pasta-paul/dsv4-flash-w4a16-fp8/main/scripts/patch_v4_packed_mapping.py \
  -o /tmp/patch_packed.py
python /tmp/patch_packed.py vllm/model_executor/models/deepseek_v4.py
```

### Acti's MTP-loader patches (~30 lines, three edits in `vllm/model_executor/models/deepseek_v4_mtp.py`)

Apply each of these or you'll hit the six load errors documented in the model card.

#### Patch 4a — pass `prefix=` to e_proj / h_proj

In `class DeepSeekV4MultiTokenPredictorLayer.__init__`, change both ReplicatedLinear constructions:

```python
self.e_proj = ReplicatedLinear(
    config.hidden_size, config.hidden_size,
    bias=False, return_bias=False,
    quant_config=quant_config,
    prefix=f"{prefix}.e_proj",          # <-- ADD THIS
)
self.h_proj = ReplicatedLinear(
    config.hidden_size, config.hidden_size,
    bias=False, return_bias=False,
    quant_config=quant_config,
    prefix=f"{prefix}.h_proj",          # <-- ADD THIS
)
```

#### Patch 4b — `packed_modules_mapping` on `DeepSeekV4MTP`

Add a class attribute right above `__init__`:

```python
class DeepSeekV4MTP(nn.Module):
    packed_modules_mapping = {                          # <-- ADD THIS BLOCK
        "fused_wqa_wkv": ["wq_a", "wkv"],
        "fused_wkv_wgate": ["wkv", "wgate"],
        "gate_up_proj": ["w1", "w3"],
    }

    def __init__(self, *, vllm_config, prefix=""):
        ...
```

#### Patch 4c — `.weight_scale` (no `_inv`) in MTP loader

In `DeepSeekV4MTP.load_weights` find the suffix pick logic and change:

```python
suffix = (
    expert_scale_suffix
    if _EXPERT_SCALE_RE.search(name)
    else ".weight_scale"                # <-- was ".weight_scale_inv"
)
```

If anything goes sideways, sanity-check against the local diff after applying — `git diff vllm/model_executor/models/deepseek_v4_mtp.py` should show three hunks: (a) `prefix=f"{prefix}.e_proj"` and `prefix=f"{prefix}.h_proj"` added in the `DeepSeekV4MultiTokenPredictorLayer.__init__` constructor calls, (b) the `packed_modules_mapping` dict added as a class attribute on `DeepSeekV4MTP` immediately above its `__init__`, and (c) the `else ".weight_scale_inv"` line in `DeepSeekV4MTP.load_weights` changed to `else ".weight_scale"`.

### Build vLLM

```bash
cd $HOME/dsv4-local/src/vllm
export CUDA_HOME=$HOME/dsv4-local/conda-cuda
export PATH=/usr/bin:$CUDA_HOME/bin:$PATH
export LD_LIBRARY_PATH=$CUDA_HOME/lib:$LD_LIBRARY_PATH
export CPATH=$CUDA_HOME/include:$CPATH
export TORCH_CUDA_ARCH_LIST=12.0a
export CUDA_ARCH_LIST=120a
export CC=/usr/bin/gcc CXX=/usr/bin/g++ CUDAHOSTCXX=/usr/bin/g++
export NVCC_PREPEND_FLAGS="-ccbin /usr/bin/g++"
pip install -e .            # ~20-30 min on a fast box
python -c "import vllm; print(vllm.__version__)"
python -c "from vllm.model_executor.models.deepseek_v4 import DeepseekV4ForCausalLM; print('OK')"
```

If the build fails: the most common causes are (a) host gcc too new for nvcc 12.9, fixed by exporting CC/CXX above; (b) Triton can't find `pyconfig.h`, fix by `export CPATH=$CUDA_HOME/include:/usr/include/x86_64-linux-gnu/python3.12:$CPATH`. Stop and ask the human if those don't fix it — don't pivot to vanilla vLLM.

---

## Phase 5 — model download (143 GB)

```bash
mkdir -p $HOME/dsv4-local/models
HF_HUB_ENABLE_HF_TRANSFER=1 hf download \
  LordNeel/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8 \
  --local-dir $HOME/dsv4-local/models/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8
```

This is the v2 GPTQ release. Expect ~30-60 min on a 100 MB/s connection. The 4 base shards (~50 GB each) will hash-dedupe against pasta-paul's repo if you've already downloaded that.

Verification:

```bash
ls $HOME/dsv4-local/models/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8/*.safetensors | wc -l    # → 5
du -sh $HOME/dsv4-local/models/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8                       # → ~146G
python -c "
from safetensors import safe_open
import json
idx = json.load(open('$HOME/dsv4-local/models/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8/model.safetensors.index.json'))
keys = list(idx['weight_map'])
print('total tensors:', len(keys))
print('mtp.0.* tensors:', sum(1 for k in keys if k.startswith('mtp.')))
print('mtp.0.ffn.experts.0.w1.weight_packed present:',
      'mtp.0.ffn.experts.0.w1.weight_packed' in keys)
"
```

Expected: ~102k tensors, 2338 mtp.* tensors, w1.weight_packed True.

---

## Phase 6 — write serve script + env

```bash
mkdir -p $HOME/dsv4-local/scripts
cat > $HOME/dsv4-local/scripts/serve_524k.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
ROOT=$HOME/dsv4-local
source "$ROOT/venvs/vllm-dsv4/bin/activate"

export CUDA_VISIBLE_DEVICES=0,1
export CUDA_HOME=$ROOT/conda-cuda
export PATH=/usr/bin:$CUDA_HOME/bin:$PATH
export TRITON_PTXAS_PATH=$CUDA_HOME/bin/ptxas
export LD_LIBRARY_PATH=$CUDA_HOME/lib:${LD_LIBRARY_PATH:-}
export CPATH=$CUDA_HOME/include:${CPATH:-}
export TORCH_CUDA_ARCH_LIST=12.0a
export CUDA_ARCH_LIST=120a

export VLLM_USE_FLASHINFER_SAMPLER=0
export VLLM_ENABLE_DEEPSEEK_V4_SPARSE_MLA_WARMUP=0
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export PYTHONUNBUFFERED=1

# NCCL: validated for Max-Q PCIe topology
export NCCL_DEBUG=WARN
export NCCL_P2P_DISABLE=1
export NCCL_SHM_DISABLE=0
export NCCL_IB_DISABLE=1
# small-msg latency tuning (drops TTFT 154ms -> 91ms on Max-Q at zero decode cost)
export NCCL_PROTO=LL
export NCCL_ALGO=Ring
export NCCL_MIN_NCHANNELS=8
export NCCL_NTHREADS=512

MODEL=$ROOT/models/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8

exec vllm serve "$MODEL" \
  --served-model-name deepseek-v4-flash deepseek-v4-flash-mtp DSV4-W4A16-FP8 deepseek-ai/DeepSeek-V4-Flash \
  --tensor-parallel-size 2 \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --max-model-len 524288 \
  --max-num-seqs 2 --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.93 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --trust-remote-code \
  --disable-custom-all-reduce \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --host 0.0.0.0 --port 8000
EOF
chmod +x $HOME/dsv4-local/scripts/serve_524k.sh
```

**Hardware-specific edits to make**:
- If running on RTX PRO 6000 **Server** (with NVLink): you can drop `--disable-custom-all-reduce` AND you can drop the `NCCL_P2P_DISABLE`/`NCCL_PROTO=LL` block. CustomAllreduce works on NVLink and is a bit faster.
- If running on H200: same as Server (NVLink available); also you'll likely run TP=4 or higher — adjust `--tensor-parallel-size` and `CUDA_VISIBLE_DEVICES` accordingly. **Do NOT use TP≥4 with W4A16 quants** — there's an upstream MoE scale-sharding bug ([vllm-project/vllm#41511](https://github.com/vllm-project/vllm/issues/41511)).
- If running on DGX Spark / GB10: same as Max-Q (no NVLink). Keep all flags.

---

## Phase 7 — start the server, wait for ready

```bash
setsid bash $HOME/dsv4-local/scripts/serve_524k.sh > $HOME/dsv4-local/serve.log 2>&1 &
echo "started pid=$!"

# Wait for readiness (≤5 min cold)
until grep -q "Application startup complete" $HOME/dsv4-local/serve.log 2>/dev/null; do
  if grep -qE "(Traceback|RuntimeError|Engine core init.*failed|Worker proc.*died)" $HOME/dsv4-local/serve.log; then
    echo "FAILED — see $HOME/dsv4-local/serve.log"
    tail -50 $HOME/dsv4-local/serve.log
    exit 1
  fi
  sleep 2
done
curl -fsS http://127.0.0.1:8000/health -o /dev/null && echo "health OK"
```

Expected timing on a 2× Max-Q: weight load ~30 s, KV cache + warmup ~30 s, graph capture ~5 s, total ~75 s wall.

### Failure modes to recognize and remediate

| Symptom in log | Cause | Fix |
|---|---|---|
| `NotImplementedError: DeepSeekV4 requires FP8 attention quantization` | Loaded a non-FP8-attn variant (e.g. Intel/AutoRound) | Use the LordNeel repo, NOT a different DSV4 quant |
| `Unable to find matching target for ''` | Patch 4a (e_proj/h_proj prefix) missing | Apply Patch 4a |
| `KeyError: attn.fused_wqa_wkv.weight_scale_inv` (mtp_block) | Patch 4b (`packed_modules_mapping` on `DeepSeekV4MTP`) missing | Apply Patch 4b |
| `ValueError: Unable to find matching target for model.layers.43.e_proj` | Out-of-date config.json (ignore list missing layer 43 prefixes) | Re-pull the model — that bug is fixed in v2 |
| Hangs at `gpu_worker.py:619 CUDA graph pool memory` then loops `shm_broadcast.py:681` | CustomAllreduce deadlock on Max-Q | Add `--disable-custom-all-reduce` |
| `KeyError: experts.w13_weight` | Loaded the v1 RTN ckpt by mistake | Pull v2 (current main branch on Hub) |

---

## Phase 8 — smoke tests

```bash
# Non-streaming chat
curl -sS -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer EMPTY" \
  --data '{
    "model":"deepseek-v4-flash","stream":false,"max_tokens":64,"temperature":0.0,
    "messages":[{"role":"user","content":"Say OK."}]
  }' | python -c "import json,sys; d=json.load(sys.stdin); print(d['choices'][0]['message'].get('content') or '(empty content — check reasoning_content)')"

# Streaming
curl -sS -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" -H "Authorization: Bearer EMPTY" \
  --data '{
    "model":"deepseek-v4-flash","stream":true,"max_tokens":32,"temperature":0.0,
    "messages":[{"role":"user","content":"hi"}]
  }' | head -20
```

If `content` is empty even on simple prompts, that's the deepseek_v4 reasoning-parser putting output into `reasoning_content`. Read both fields:

```python
msg = response["choices"][0]["message"]
text = msg.get("content") or msg.get("reasoning_content") or ""
```

---

## Phase 9 — hand-off

Once Phase 8 passes, the agent should:

1. Print the listen address: `http://<host>:8000/v1`
2. Print the served model names: `deepseek-v4-flash`, `deepseek-v4-flash-mtp`, `DSV4-W4A16-FP8`, `deepseek-ai/DeepSeek-V4-Flash`
3. Note that the API key is the literal string `EMPTY` (vLLM's default for unauthenticated)
4. Print expected single-stream decode TPS for the configured profile (524k → ~85; 128k → ~111)
5. Tell the human about reasoning_parser and `--reasoning-parser deepseek_v4` (their client may need to read both `content` and `reasoning_content`)

Do NOT proceed to fine-tuning, hot-swapping, or modifying the running server without explicit human approval.

---

## Phase 10 — optional: GPU clock unlock (sudo, +0-1% TPS)

If on Max-Q and the human wants the last 1%:

```bash
# REQUIRES SUDO. Stop and ask the human first.
sudo nvidia-smi -pm 1
sudo nvidia-smi -pl 325 -i 0; sudo nvidia-smi -pl 325 -i 1
sudo nvidia-smi -lgc 2700 -i 0; sudo nvidia-smi -lgc 2700 -i 1
# revert: sudo nvidia-smi -rgc -i 0,1; sudo nvidia-smi -pl 300 -i 0,1
```

Honest expectation: Max-Q firmware caps the achievable boost at ~2325 MHz under sustained 99% SM load regardless of the lock value. The +1% gain is real but small. Skip this unless the human asks for it.

---

## Quick reference: where things live in the reference layout

```
$HOME/dsv4-local/
├── conda-cuda/                      # CUDA 12.9 toolkit (no system change)
├── miniforge3/                      # conda installer
├── venvs/vllm-dsv4/                 # the only Python venv you need
├── src/vllm/                        # patched jasl/vllm fork
├── models/DeepSeek-V4-Flash-Acti-MTP-W4A16-FP8/
└── scripts/serve_524k.sh
```

## Questions an agent should ask the human (only when triggered)

| Trigger | Ask |
|---|---|
| < 2× 96 GB GPUs found | "Hardware doesn't fit a 524k TP=2 profile. Want to try smaller profiles or stop?" |
| Build fails with cuda/nvcc errors after the documented fixes | "Build broken. Want me to pivot to vanilla vLLM (won't load this model) or stop?" |
| Phase 7 hangs > 5 min with no log progress | "Server isn't responding; want me to dump py-spy stacks (needs sudo) or kill and retry with `--enforce-eager`?" |
| Phase 10 (sudo clock unlock) | always ask before running |
| User asks about TP > 2 | "Upstream W4A16 MoE scale-sharding bug — issue #41511. TP=4 will OOM/error. Want to stop or proceed anyway?" |