bleysg/Qwen3.5-122B-A10B-int4-fp8-hybrid

Qwen3.5-122B-A10B — hybrid INT4 + FP8 (DGX Spark)

A hybrid-quantized checkpoint of Qwen3.5-122B-A10B for bandwidth-optimal decode on the NVIDIA DGX Spark (GB10 / SM121, 128 GB unified) under vLLM. Routed experts and attention stay INT4; the BF16 shared experts (dense — read on every token) are replaced with calibrated FP8 from the official FP8 release.

Composition

• Routed experts + attention: INT4 (GPTQ / AutoRound) — from Intel/Qwen3.5-122B-A10B-int4-AutoRound.
• Shared experts: FP8 E4M3 block-128 — from Qwen/Qwen3.5-122B-A10B-FP8.
• Embeddings / norms / head: unchanged from the INT4 base.

144 shared-expert layers convert BF16→FP8 — an always-on bandwidth lever worth +28 % at base / no-spec decode on GB10 (28.2 → 36.0 tok/s), diminishing under high-acceptance speculative decode (see the project's amortization law).

Usage

Built for and served by qwen3.5-122B-A10B-on-spark (--profile dense). It requires that project's vLLM 0.23 hybrid-FP8 dispatch patch — an INCConfig.maybe_update_config override that dispatches Fp8LinearMethod for the FP8 shared-expert layers. Stock vLLM will not dispatch the mixed INT4/FP8 scheme.

Provenance & credit

Recipe (hybrid INT4+FP8, INT8 lm-head, MTP) and build script: albond/DGX_Spark_Qwen3.5-122B-A10B-AR-INT4. Built with build-hybrid-checkpoint.py, which merges FP8 non-expert tensors into the INT4 checkpoint.

License

Derived work — inherits the upstream Qwen3.5-122B-A10B and Intel AutoRound terms. See the base-model repositories linked above.