anoane/Laguna-S-2.1-NVFP4-RTX6000PRO-vLLM

Laguna-S-2.1-NVFP4 on RTX PRO 6000 (Blackwell / sm120) — vLLM serving stack

Everything needed to rebuild the poolside/Laguna-S-2.1-NVFP4 production serving stack on a clean Ubuntu 24.04 LTS host with a single RTX PRO 6000 Blackwell (sm_120, 96 GB) GPU: vLLM built from latest main, the flashinfer sm120 fix, the poolside_v1 reasoning-parser patch, DFlash n=7 speculative decoding, native 256K context, thinking-on, and a transparent proxy that keeps thinking-on safe at any context depth. Full step-by-step guide + rationale + benchmarks: LAGUNA_S_2.1_VLLM.md (§23 is the reproduction guide).

file	purpose
build_vllm.sh	build vLLM from source inside the CUDA-13.3/sm120 base image (§3)
poolside_v1_reasoning_parser.py	patched reasoning parser — drop into vllm/reasoning/ (§14.4)
laguna_poolside_reasoning_parser.patch	the same change as a git diff
laguna-serve.service	systemd unit: vLLM server, native 256K, DFlash n=7, thinking-on, no output cap (§17)
laguna_budget_proxy.py	transparent proxy that injects an adaptive thinking_token_budget so thinking-on is safe near the context ceiling — works for any OpenAI-compatible client (§20)
laguna-budget-proxy.service	systemd unit for the proxy
caddy-laguna.example	domain-agnostic TLS-edge example (bring your own cert + host)

TL;DR

1. Build engine image: build_vllm.sh → flashinfer 0.6.15 jit-cache → drop in poolside_v1_reasoning_parser.py → docker commit vllm-laguna-sm120:latest.
2. hf download poolside/Laguna-S-2.1-NVFP4 + …-DFlash-NVFP4 into /root/workspace.
3. laguna-serve.service (vLLM :8000).
4. venv (fastapi uvicorn[standard] httpx) + laguna_budget_proxy.py + laguna-budget-proxy.service (:8001).
5. Front :8001 with your own TLS reverse proxy → clients hit https://<host>/v1, model laguna-s-2.1.

Key gotchas: DFlash needs --gpu-memory-utilization 0.93 + PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and --max-num-seqs 32; never send min_p/logit_bias under DFlash; clients read chain-of-thought from the reasoning field (not reasoning_content).

Testing notes (how I got here)

I tested the first NVFP4 revision first — a lot of the early chapters (§1–§13 and the §10x benches) are that rev1 work and some of it is now outdated, so skip ahead to §14 for the current state. Then poolside re-published a second revision (spinquantless-norot, §14.1), which is what I run in production. It took two fixes to make it solid.

1. THINK FIX — </think> with no opening tag (§14.4)

On rev2 the chat template primes <think> in the prompt and the model only ever emits the closing </think> — so vLLM's stock reasoning parser never enters the reasoning channel and the whole chain-of-thought gets misclassified as content (or dropped). That breaks most harnesses. Fix:

• poolside_v1_reasoning_parser.py → drop-in replacement for vllm/reasoning/poolside_v1_reasoning_parser.py (also shipped as a git diff, laguna_poolside_reasoning_parser.patch). Full diagnosis: §14.4.

2. LOOP FIX — thinking_token_budget via a proxy (§20)

Even with the think fix I still got reasoning loops near the context ceiling (see the §15 tests). I solved it with a nice workaround: on the same box running vLLM I run a small proxy that injects a per-request thinking_token_budget (§20). This drives vLLM's stock vllm/v1/sample/thinking_budget_state.py — no changes to that file, it's just for reference (the feature already ships in vLLM main; the proxy just sets the field, since no off-the-shelf harness sends it). With the proxy injecting the budget I get zero loops — §22 has the before/after comparison (the exact rows that looped / came back empty in §15 all complete once the budget is injected).

Needle retrieval / YaRN (stable past 256K)

Needle-retrieval is quite stable with YaRN on and off, even at contexts bigger than 256K. I tested these 4 serve configs:

f80-off = serve laguna-nvfp4-560k     --max-model-len 573440 --moe-backend flashinfer_cutlass --gpu-memory-utilization 0.95
          [factor-80 YaRN] · spec: none
f48-on  = serve laguna-nvfp4-360k     --max-model-len 360448 --gpu-memory-utilization 0.93 -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
          [factor-48 YaRN] · spec: DFlash-NVFP4 num_spec=7 method=dflash
nat-off = serve laguna-nvfp4-official --max-model-len 262144 --gpu-memory-utilization 0.95
          [native factor-32] · spec: none
nat-on  = serve laguna-nvfp4-official --max-model-len 262144 --gpu-memory-utilization 0.93
          [native factor-32] · spec: DFlash-NVFP4 num_spec=7 method=dflash

I use nat-on as my daily config, but f48-on (350K via factor-48 YaRN) is also quite stable with the patches + workarounds above. Needle ladders + RULER/NoLiMa curves: §10b / §10d.

Production config + speed

My production config is §17: native 256K context + DFlash on + thinking/reasoning enabled. Single-session I get a stable ~140–190 tok/s (up to ~210 measured single-stream in §24).

Concurrency / aggregate speed (§24)

vLLM batches concurrent requests dynamically (PagedAttention — nothing is pre-reserved per slot, unlike llama.cpp). Measured through the proxy on the production config:

concurrent	aggregate tok/s	per-stream tok/s	TTFT	DFlash accept
1	209	210	0.06 s	0.46
16	990	95	0.16 s	0.46
32	1,494	81	0.20 s	0.45
40	1,762	82	7.8 s (8 queued)	0.47

~8.4× aggregate scaling out to 40 concurrent, DFlash accept steady at ~0.45 the whole way, and sub-0.2 s first-token up to 32 (= --max-num-seqs; beyond that requests queue). Full tables (latency p50/p95, power, energy, scaling efficiency): §24.

Agent test

Testing on the pi.dev agent right now — no loops so far, and I like it. So far it's a good contender against Qwen3.6-27B and a (highly quantized) DeepSeek-V4-Flash (my RTX 6000 PRO recipe for that one: https://huggingface.co/anoane/DeepSeek-V4-Flash-vllm-moet-sm120-cache).

DGX (sm121)

For a DGX I'd just search-and-replace sm120 → sm121 and sm_120 → sm_121 — I expect it to work out of the box. You'll have room for way more concurrent contexts (proportionally more RAM). The §17 production config stays totally unchanged; the only tweak worth trying is bumping --gpu-memory-utilization 0.93 → 0.94 — the DGX has proportionally more RAM, so the DFlash workspace should still fit at 0.94 if you want to squeeze out the last bit of KV.