ox-ox/Hy3-295B-Instruct-w2q3exp-AProjQ8-SExpQ8-OutQ8-MTP-mlx

Hy3-295B-Instruct — MLX (w2q3 experts / Q8 rest / MTP)

An MLX port of tencent/Hy3 (Tencent Hunyuan 3, hy_v3) — a 295B-parameter Mixture-of-Experts model (21B active, 192 experts top-8, GQA + QK-norm, 256K context) with a Multi-Token-Prediction (MTP) layer for self-speculative decoding.

Quantized in the asymmetric style of antirez/deepseek-v4-gguf: crush the routed experts (they are most of the weights), keep the decision-making parts high precision. The filename is the spec. Runs on a single 128 GB Apple Silicon machine.

⚠️ Needs a patched mlx-lm. Upstream mlx-lm doesn't ship hy_v3 yet, and even PR #1211 (which adds the base model) strips the MTP layer. The branch below = PR #1211 (base model + tool parsers + tokenizer fix, by @kernelpool) plus the MTP self-speculative decoding added here.

Installation

One command. The branch = PR #1211 (hy_v3 base + tool parsers + tokenizer fix, by @kernelpool) plus the MTP self-speculative decoding added here. The transformers pin avoids a known crash on 5.13+ (AutoTokenizer.register).

pip install "transformers>=5.7,<5.13" \
  "mlx-lm @ git+https://github.com/eauchs/mlx-lm.git@hy_v3-mtp"

Read this before running (memory). The weights must stay fully resident in RAM. This model is ~110 GB on a 128 GB Mac — tight. For usable speed:

1. Keep the model on the internal SSD (external USB works but pages slowly).
2. Close other big RAM users (Ollama, LM Studio, browsers).

Fully resident it does ~5–6 tok/s; paging from an external USB SSD it can drop to <1 tok/s. This is a hardware/memory limit, not the quant — see Performance and Troubleshooting.

Usage

mlx_lm.generate --model ox-ox/Hy3-295B-Instruct-w2q3exp-AProjQ8-SExpQ8-OutQ8-MTP-mlx \
  --prompt "Explain mixture-of-experts routing in one paragraph." \
  --max-tokens 300 --temp 0.0

mlx_lm.chat --model ox-ox/Hy3-295B-Instruct-w2q3exp-AProjQ8-SExpQ8-OutQ8-MTP-mlx

from mlx_lm import load, generate

model, tokenizer = load("ox-ox/Hy3-295B-Instruct-w2q3exp-AProjQ8-SExpQ8-OutQ8-MTP-mlx")
messages = [{"role": "user", "content": "Hello! Introduce yourself briefly."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=300))

Reasoning modes

Hy3 supports no_think (default) and high reasoning effort via the chat template. The #1211 tokenizer fix + hy_v3_opensource parser handle the release checkpoint's :opensource special tokens (<think:opensource>, <tool_calls:opensource>, ...):

prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
    chat_template_kwargs={"reasoning_effort": "high"},
)

Files

Repo	Size	Routed experts	Everything else
this repo	~110 GB	2-bit (gate, up) + 3-bit (down)	Q8 attn proj / shared expert / output head / router / dense-layer-0; bf16 embeddings; bf16/f32 norms & expert_bias; MTP layer included

Quantization recipe

The filename is the spec (group_size = 64, MLX affine, verified against the tensors):

Tensor class	Quant	Notes
experts gate_proj, up_proj	2-bit	routed-expert up/gate
experts down_proj	3-bit	routed-expert down (extra bit for quality)
shared_mlp.*	Q8 (8-bit)	shared expert
self_attn.{q,k,v,o}_proj	Q8	all attention projections (GQA)
lm_head	Q8	output head
router.gate	Q8	learned router (antirez keeps this F16)
dense MLP (layer 0)	Q8	first dense layer
embed_tokens	bf16	input embedding
all *_norm, expert_bias	bf16 / f32
MTP layer	same scheme	self-speculative decoding

Motivation for the asymmetry (per antirez): the routed experts are the majority of the parameter count, but each expert handles only a fraction of tokens, so aggressive quantization on them costs less average quality than the same treatment of the router, projections, or shared expert. Keeping the decision-makers at Q8 preserves behaviour; crushing the experts buys the size.

Differences from antirez's DS4 recipe

• Router: Q8 here vs F16 for antirez.
• Method: plain round-to-nearest (RTN), no calibration/imatrix. Expect a visible quality drop at 2-bit on the expert gate/up projections vs the source model. A calibrated pass would need the source (FP8/BF16) weights.

Multi-Token Prediction (self-speculative decoding)

Because num_nextn_predict_layers > 0, the patched mlx_lm uses the built-in MTP head for self-speculative decoding — no external draft model. Each step drafts one token with the MTP head and verifies it with a single target pass, up to two tokens per pass.

Verification is distribution-preserving and honors your sampler (temperature, top-p, top-k) — it is no longer greedy-only, so temp = 0.9 (Hy3's recommended sampling) works correctly. Two rules, picked automatically:

• Residual (Leviathan) — at temp > 0 with a standard sampler: accept the draft x ~ q with probability min(1, p(x)/q(x)) and resample from the normalized relu(p − q) on rejection. Accepts strictly more often (Σ min(p,q) ≥ Σ p·q) than plain matching, for the exact same output distribution.
• Match — the sampler-agnostic fallback (custom samplers / XTC / active logits processors), and what temp = 0 reduces to (greedy draft-equals-target).

Acceptance is checkpoint-bound: the MTP drafter here is quantized about as hard as the target, so it reflects quant precision more than the method's ceiling. ≈ 35–40% at greedy on this quant; at temp > 0 the residual rule accepts more often than matching on the same draws (higher-precision drafters reach ~64–66% at greedy).

Performance

Mode	Speed	Peak RAM
Standard decoding	~4.8 tok/s	~107 GB
MTP self-speculative	~6.3 tok/s	~114 GB

These assume the model is fully resident. At ~114/128 GB it's borderline: any other large RAM user, or serving the weights from a slow external disk, makes the mmap'd pages page in from storage and throughput collapses (seen as low as ~0.7 tok/s). Prefer the internal SSD and close other apps. (Note: forcing mx.set_wired_limit to pin ~110 GB from an external USB SSD trips the Metal GPU watchdog, so wiring is left to the OS.)

Troubleshooting

• AttributeError: 'str' object has no attribute '__module__' on import — a transformers 5.13+ incompatibility in mlx-lm. Use transformers<5.13 (already pinned in the install command above).
• GPU Timeout Error (kIOGPUCommandBufferCallbackErrorTimeout) at load / first token — mlx-lm pins (wires) the whole model in RAM, and pinning ~110 GB from a slow external USB SSD trips Metal's watchdog. Fix: put the model on the internal SSD (fast enough to wire), or free enough RAM that it stays resident. This is a hardware/storage limit, not the quantization.

Limitations

• Requires ~128 GB unified memory; not usable on smaller Macs.
• RTN quantization (no calibration/imatrix) — reduced quality vs. the source model.
• MTP acceptance is capped by the drafter's precision (2–3-bit experts here), not the method.
• Depends on the hy_v3-mtp mlx-lm branch until hy_v3 (+ MTP) lands upstream.

Credits & license

• Base model: tencent/Hy3 — Apache 2.0.
• hy_v3 mlx-lm support + tool parsers + tokenizer fix: @kernelpool, mlx-lm PR #1211.
• Asymmetric low-bit MoE recipe inspired by antirez/deepseek-v4-gguf.
• Quantization + MTP self-speculative decoding (distribution-preserving verification: residual/Leviathan + match): ox-ox (mlx-lm PR #1485).
• Released under Apache 2.0, following the base model's terms.