Relativ3pa1n/Laguna-S-2.1-W8A16

Laguna S 2.1 — W8A16 (group_size=32) — tested on 8× RTX 3090

A W8A16 quantization of poolside/Laguna-S-2.1 (118B total / 8B active, 256-expert MoE, OpenMDW-1.1). The high-fidelity companion to Laguna-S-2.1-W4A16 — same model, 8-bit weights for higher quality at the cost of size/speed.

~38 tok/s decode · 120 GB · 256K context (3.18× concurrent) · clean output with thinking on · tested on 8× RTX 3090 (Ampere, 192 GB VRAM)

TL;DR

Size	120 GB (from 219 GB BF16)
Format	compressed-tensors W8A16, group_size=32, symmetric, Marlin kernels
Serves on	8× RTX 3090 (192 GB VRAM), TP=8
Throughput	~38 tok/s single-stream (CUDA graphs on)
Context / concurrency	256K, 3.18× concurrent
Output	Clean — including technical/markdown content, thinking on
Terminal-Bench (core 0.1.1, 18-task subset, terminus, pass@1)	5/18 = 27.8% (vs W4A16's 4/18 = 22.2%)

Why this quant: the high-fidelity variant

The W4A16 is the small/fast option (69 GB, ~92 tok/s). This W8A16 is the higher-fidelity option (8-bit weights preserve more capability — it scored +1 task on Terminal-Bench). Both serve clean on consumer Ampere (3090).

The non-obvious part: group_size=32 is the Laguna-MoE sweet spot for both 4-bit AND 8-bit. The default g128 produces token-artifacts (byfusion, import torchdec) even at 8-bit; g32 clears them — same fix as the 4-bit. And asymmetric 8-bit is kernel-blocked in vLLM (the Marlin WNA16 MoE backend rejects it — is_supported_config → _supports_quant_scheme returns False for asym-8-bit; llm-compressor #2628), so symmetric g32 is the path. The full quant story (OOM wall → per-layer loop → stale-base → group_size → cudagraph) is in the W4A16 blog.

Serve (vLLM)

vLLM 0.25.1+, 8× 24 GB Ampere:

python -m vllm.entrypoints.openai.api_server \
  --model . --served-model-name laguna \
  --tensor-parallel-size 8 --device-ids 0,1,2,3,4,5,6,7 \
  --quantization compressed-tensors \
  --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --max-model-len 262144 --gpu-memory-utilization 0.90 \
  --max-num-seqs 16 --trust-remote-code \
  --host 0.0.0.0 --port 8085

Key: --quantization compressed-tensors (required), no --enforce-eager (cudagraphs), --gpu-memory-utilization 0.90, poolside_v1 parsers, enable_thinking.

Terminal-Bench — W8A16 vs W4A16 (same 18 tasks)

	W4A16 (4-bit)	W8A16 (8-bit)
Score	4/18 = 22.2%	5/18 = 27.8%
Solved	crack-7z-hash.easy, sqlite-with-gcov, prove-plus-comm, sanitize-git-repo	+ path-tracing (W4 failed 4/4; W8 solved all 4 sub-tests)
Near-solve	swe-bench-astropy-2 (8/9)	swe-bench-astropy-2 (10/11) — even closer

W8A16 per-task breakdown:

Task	Result	Reason
crack-7z-hash.easy	✅ PASS	solved (2 sub-tests)
sqlite-with-gcov	✅ PASS	solved (3)
prove-plus-comm	✅ PASS	solved (4)
sanitize-git-repo	✅ PASS	solved (3)
path-tracing	✅ PASS	solved (4) — W4 failed this
swe-bench-astropy-2	❌ FAIL	10/11 sub-tests (near-solve)
blind-maze-explorer-5x5	❌ FAIL	1/2 sub-tests
security-vulhub-minio	❌ FAIL	1/1 sub-tests
sqlite-db-truncate	❌ FAIL	1/1 sub-tests
extract-moves-from-video	❌ FAIL	2/2 sub-tests
password-recovery	❌ FAIL	2/2 sub-tests
qemu-alpine-ssh	❌ FAIL	1/1 (hard infra)
build-linux-kernel-qemu	❌ FAIL	agent 900s timeout (hard infra)
download-youtube	❌ FAIL	agent 900s timeout
chess-best-move	❌ FAIL	agent 900s timeout
polyglot-c-py	❌ FAIL	agent 900s timeout
pytorch-model-cli.hard	❌ FAIL	test-harness 120s timeout
cron-broken-network	❌ FAIL	test output unparseable

License & attribution

• Model weights: OpenMDW-1.1 (poolside). Derivative quant — same license.
• Built on an 8×3090 rig (192 GB VRAM) with llmcompressor 0.12 / transformers 5.10.1 / vLLM 0.25.1.