poolside

/

Laguna-S-2.1-NVFP4

like 88

Follow

Poolside 815

I'm testing since few minutes only for now but it seems pretty good :)
I'm surprised it's super fast between each turn, the ttft is weirdly too fast, it practically replies before I even send the message :D

The only downside for now is the decoding of the filler data, which is very poor; perhaps there is a way to optimize the dflash?

Here is my tests on a single Dgx Spark:

  🔍 Engine: vLLM 0.23.1rc1.dev1353+g81f51a780.d20260721

╭──────────────────────────────────────────────────────────────────────────────────────────────────────────── 🔮 Speculative Decoding Benchmark ─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ poolside/Laguna-S-2.1-NVFP4                                                                                                                                                                                                                                │
│ tg=128  depth=[0, 16384, 32768]  prompts=['filler', 'code', 'structured']  method=auto                                                                                                                                                                     │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Prometheus /metrics acceptance-rate counters are server-wide aggregates. If other models are serving concurrent traffic on this endpoint, per-request acceptance rate measurements will be inaccurate. For clean measurements: use a single-model server with no concurrent load.

  ✓     filler @ d0  19.2 eff t/s  19.9 stream t/s  α=7.7%  waste=92%  τ=1.2  win=15
  ✓       code @ d0  48.3 eff t/s  53.6 stream t/s  α=36.3%  waste=64%  τ=5.5  win=15
  ✓ structured @ d0  85.6 eff t/s  104.4 stream t/s  α=82.0%  waste=18%  τ=12.3  win=15
  ✓     filler @ d16384  19.2 eff t/s  19.9 stream t/s  α=7.9%  waste=92%  τ=1.2  win=15
  ✓       code @ d16384  45.2 eff t/s  49.6 stream t/s  α=33.7%  waste=66%  τ=5.0  win=15
  ✓ structured @ d16384  85.3 eff t/s  103.7 stream t/s  α=82.0%  waste=18%  τ=12.3  win=15
  ✓     filler @ d32768  21.3 eff t/s  22.3 stream t/s  α=10.3%  waste=90%  τ=1.5  win=15
  ✓       code @ d32768  42.7 eff t/s  46.8 stream t/s  α=32.2%  waste=68%  τ=4.8  win=15
  ✓ structured @ d32768  85.3 eff t/s  103.6 stream t/s  α=82.0%  waste=18%  τ=12.3  win=15

  vllm serve poolside/Laguna-S-2.1-NVFP4 \
    --host 0.0.0.0 \
    --moe-backend marlin \
    --port 8000 \
    --max-model-len 180000 \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 32 \
    --gpu-memory-utilization 0.85\
    --dtype auto \
    --attention-backend flashinfer \
    --tokenizer-mode auto \
    --load-format instanttensor \
    --async-scheduling \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --trust-remote-code \
    --reasoning-parser poolside_v1 \
    --enable-auto-tool-choice \
    --tool-call-parser poolside_v1 \
    --kv-cache-dtype fp8_e4m3 \
    --speculative-config '{"method":"dflash","num_speculative_tokens":15,"model":"poolside/Laguna-S-2.1-DFlash-NVFP4"}' \
    --override-generation-config '{"temperature":0.7,"top_p":0.95}'

EDIT
After few hours I got an issue, there is a looping, he thinks, he writes, he uses tool, rethink (the same reasoning), writes (the same text), he uses tool (the same) infinitly...