---
license: apache-2.0
base_model: tencent/Hy3
tags:
- gguf
- quantized
- mixed-precision
- prismaquant
- vllm
---

# Hy3 295B-A21B — PrismaQuant 2.8-bit GGUF (vLLM-servable)

Tencent **Hy3** (the July 2026 production release, Apache 2.0 — not the
preview) — 295B total / 21B active MoE, 80 layers, 192 routed experts top-8 —
quantized by [PrismaQuant](https://github.com/RobTand/prismaquant) from the
**BF16 source** to a **103.7 GB** GGUF (2.80 bits/param over quantizable
weights) that serves on a **single NVIDIA DGX Spark** (128 GB unified memory)
via vLLM + the GGUF plugin.

This supersedes the earlier `Hy3-preview`-based 2.9-bit artifact: production
base model, Apache 2.0 license, and a quantization menu extended with the
GGUF **IQ formats** (IQ2_XXS…IQ4_XS, E8-lattice codebooks) alongside the
k-quants — the IQ rungs measured strictly better per byte than the k-rungs
they displace at these bit-rates.

## How it was made

- **Measured allocation:** per-tensor formats chosen by a knapsack solve over
  measured per-(tensor, format) quantization error weighted by an empirical
  Fisher probe of the BF16 model (32×1024 calibration, streaming — the model
  never fits in memory), with per-format calibration factors derived from
  end-to-end KL measurements on a smaller proxy model.
- **Byte-budget selection:** the bit-rate targets a 103.5 GB file (single-Spark
  serving with KV-cache headroom) rather than a curve heuristic. The measured
  rate-distortion knee sits at ~2.3 bpp — this artifact rides well above it.
- **Format mix:** routed-expert bulk on IQ2_XS/IQ3_XXS/IQ4_XS; small
  attention/shared tensors on Q5_K/Q6_K/Q8_0; embeddings Q4_K; output head
  Q6_K; norms/router F32. Full map in `allocation/layer_config.json`.
- **imatrix-weighted quantization** (per-column activation second moments from
  the same calibration corpus), exhaustive GPU codeword search for IQ formats.
- The MTP layer (`model.layers.80`) is not included (GGUF serving does not use
  speculative decoding here).

## Validation — read this

**No quality claims are made for this artifact.** Models of this scale cannot
be KL-validated against their BF16 teacher on the target hardware. Validation
performed: vLLM loads the artifact, greedy and chat generation are coherent
across smoke prompts (math, prose, technical explanation), and the pipeline's
per-tensor packing is bit-exact against its own emulation and gguf-py
decoding. Community measurements welcome.

## Serving (vLLM + GGUF plugin, single DGX Spark)

See `serving/`: `serve.sh` launches a vLLM container with the
[vllm-gguf-plugin](https://github.com/vllm-project/vllm-gguf-plugin) and two
small patches (`container_patches.py` fixes an upstream `embed_tokens`
quant-config gap; `hy_v3_gguf_adapter.py` maps this artifact's
HF-checkpoint-verbatim tensor names onto vLLM's Hy3 loader, including the
stacked 192-expert MoE tensors). Note: all 2-D weight tensors in this
artifact are quantized (no BF16 Linears) — required by the current plugin
load path. llama.cpp does not implement this architecture at time of writing;
serving is vLLM-only.

```bash
serving/serve.sh hy3-prismaquant-2.8bpp-00001-of-00003.gguf ./config
```

## Provenance

Quantized with PrismaQuant (streaming direct-from-safetensors GGUF exporter).
Source checkpoint: `tencent/Hy3` (BF16). Allocation, Pareto sweep, and
per-format calibration factors are in `allocation/`. Contact:
robert.tand@icloud.com