rdtand/Hy3-295B-A21B-PrismaQuant-2.8bit-gguf-vllm

Hy3 295B-A21B — PrismaQuant 2.8-bit GGUF (vLLM-servable)

Tencent Hy3 (the July 2026 production release, Apache 2.0 — not the preview) — 295B total / 21B active MoE, 80 layers, 192 routed experts top-8 — quantized by PrismaQuant from the BF16 source to a 103.7 GB GGUF (2.80 bits/param over quantizable weights) that serves on a single NVIDIA DGX Spark (128 GB unified memory) via vLLM + the GGUF plugin.

This supersedes the earlier Hy3-preview-based 2.9-bit artifact: production base model, Apache 2.0 license, and a quantization menu extended with the GGUF IQ formats (IQ2_XXS…IQ4_XS, E8-lattice codebooks) alongside the k-quants — the IQ rungs measured strictly better per byte than the k-rungs they displace at these bit-rates.

How it was made

• Measured allocation: per-tensor formats chosen by a knapsack solve over measured per-(tensor, format) quantization error weighted by an empirical Fisher probe of the BF16 model (32×1024 calibration, streaming — the model never fits in memory), with per-format calibration factors derived from end-to-end KL measurements on a smaller proxy model.
• Byte-budget selection: the bit-rate targets a 103.5 GB file (single-Spark serving with KV-cache headroom) rather than a curve heuristic. The measured rate-distortion knee sits at ~2.3 bpp — this artifact rides well above it.
• Format mix: routed-expert bulk on IQ2_XS/IQ3_XXS/IQ4_XS; small attention/shared tensors on Q5_K/Q6_K/Q8_0; embeddings Q4_K; output head Q6_K; norms/router F32. Full map in allocation/layer_config.json.
• imatrix-weighted quantization (per-column activation second moments from the same calibration corpus), exhaustive GPU codeword search for IQ formats.
• The MTP layer (model.layers.80) is not included (GGUF serving does not use speculative decoding here).

Validation — read this

No quality claims are made for this artifact. Models of this scale cannot be KL-validated against their BF16 teacher on the target hardware. Validation performed: vLLM loads the artifact, greedy and chat generation are coherent across smoke prompts (math, prose, technical explanation), and the pipeline's per-tensor packing is bit-exact against its own emulation and gguf-py decoding. Community measurements welcome.

Serving (vLLM + GGUF plugin, single DGX Spark)

See serving/: serve.sh launches a vLLM container with the vllm-gguf-plugin and two small patches (container_patches.py fixes an upstream embed_tokens quant-config gap; hy_v3_gguf_adapter.py maps this artifact's HF-checkpoint-verbatim tensor names onto vLLM's Hy3 loader, including the stacked 192-expert MoE tensors). Note: all 2-D weight tensors in this artifact are quantized (no BF16 Linears) — required by the current plugin load path. llama.cpp does not implement this architecture at time of writing; serving is vLLM-only.

serving/serve.sh hy3-prismaquant-2.8bpp-00001-of-00003.gguf ./config

Provenance

Quantized with PrismaQuant (streaming direct-from-safetensors GGUF exporter). Source checkpoint: tencent/Hy3 (BF16). Allocation, Pareto sweep, and per-format calibration factors are in allocation/. Contact: robert.tand@icloud.com