rdtand/Hy3-295B-A21B-PrismaQuant-5.3bit-2xSpark-vllm

Hy3 295B-A21B — PrismaQuant 5.3-bit mixed NVFP4/FP8/BF16 for 2× DGX Spark

Tencent Hy3 (July 2026 production release, Apache 2.0) — 295B total / 21B active MoE, 80 layers, 192 routed experts top-8, including the MTP draft layer for speculative decoding — quantized by PrismaQuant from the BF16 source to a ~190 GB compressed-tensors checkpoint (5.28 bits/param over quantizable weights) sized to serve on two NVIDIA DGX Sparks under vLLM pipeline parallelism with no custom kernels and no forked runtime.

Sibling artifact: the single-Spark 2.8-bit GGUF is at rdtand/Hy3-295B-A21B-PrismaQuant-2.8bit-gguf-vllm.

What's inside

• Measured mixed-precision allocation (per-tensor formats from a knapsack solve over measured per-(tensor, format) error weighted by an empirical Fisher probe of the BF16 model): NVFP4 on the routed-expert bulk, FP8 (E4M3, dynamic activations) on most attention/shared tensors, BF16 on the small sensitive remainder. Full map in allocation/layer_config.json.
• Byte-budget selection: sized so weights fit 2×128 GB with KV headroom (~99 GB/node under pp2), not tuned to a curve heuristic.
• Deliberate rendering throughout: GPTQ + static activation ordering + joint scale optimization on dense Linears; batched GPTQ on every one of the 15,360 routed experts (rendered inline during the streaming export — the model never fits in memory at any point in the pipeline).
• MTP included: model.layers.80 ships verbatim BF16 and loads into vLLM's HYV3 MTP path (--speculative-config '{"method":"mtp",...}').
• Embeddings/head BF16; norms/router FP32.

Validation — read this

No quality claims are made for this artifact, and the author has one Spark: the full 2-node topology has not been run. What was validated:

• A 6-layer truncated twin of this exact artifact (same quantization metadata, real tensors, MTP remapped) passes vLLM load + generate and MTP speculative-decode smokes on a single Spark. pp2 loads each rank through the same unsharded per-layer path the twin exercises.
• A static tp2 shard audit (shipped: serving/tp2_shard_audit.py) verifies every quantized tensor's packed dims, NVFP4 group-16 boundaries, and scale shapes divide cleanly at world size 2.
• Per-tensor packing is bit-exact against the pipeline's own emulation.

Community measurements (2×Spark throughput, KL/perplexity vs BF16, task suites) are welcome and will be linked here.

Serving

See serving/README.md for the two-node pp2 launch (recommended) and the tp2 alternative.

Provenance

Quantized with PrismaQuant. Source: tencent/Hy3 (BF16). Allocation and Pareto data in allocation/. Contact: robert.tand@icloud.com