Brooooooklyn/Qwen3.6-35B-A3B-UD-NVFP4_K_XL-mlx

Qwen3.6-35B-A3B — UD-NVFP4_K_XL (mlx-node)

NVFP4 (NVIDIA Blackwell FP4) quantization of Qwen/Qwen3.6-35B-A3B for Apple Silicon, using the Unsloth Dynamic quantization strategy via mlx-node.

	Original (BF16)	UD-Q4_K_XL (affine)	This Model
Size	~66 GB	22 GB	22 GB
Format	SafeTensors	SafeTensors	SafeTensors
Precision	BF16 uniform	4-bit affine + BF16	NVFP4 (E4M3 scales) + mixed affine + BF16
FFN group size	—	64	16 (nvfp4) / 64 (affine)
Biases	—	yes	no (FFN nvfp4); yes (affine layers)

What is NVFP4?

NVFP4 is NVIDIA's Blackwell FP4 micro-scaling format. Each group of 16 elements shares a single 8-bit E4M3 scale (a full FP8 number with mantissa, not just an exponent), and elements themselves are stored as E2M1 FP4 values. Compared to MXFP4 (OCP):

• Higher fidelity scale: E4M3 scale has 3 mantissa bits, MXFP4's E8M0 only encodes a power-of-two
• Smaller group: 16 vs. 32 — fewer outliers per scale, better dynamic range tracking
• Higher scale density: 2× the scales per weight (16 elements vs 32 per scale), so per-byte overhead is roughly the same as MXFP4 despite the richer scale type
• No biases: zero-point implicit (FP4 covers ±range)

For dense LLM weights, NVFP4 typically beats MXFP4 on perplexity at the same nominal bit budget, with the trade-off of slightly more metadata per weight.

Note on router gates: MoE router gates (mlp.gate, mlp.shared_expert_gate) stay 8-bit affine under both --q-mxfp and --q-mode nvfp4. FP4 quantization noise on a 256-expert router flips top-K expert selection and destroys generation quality. mlx-lm hardcodes router gates to affine for the same reason.

All Variants

Repo	GGUF Equivalent	Size	Decode (tok/s)
Brooooooklyn/Qwen3.6-35B-A3B-UD-Q2_K_XL-mlx	UD-Q2_K_XL	14 GB	63.7
Brooooooklyn/Qwen3.6-35B-A3B-UD-Q3_K_XL-mlx	UD-Q3_K_XL	18 GB	59.2
Brooooooklyn/Qwen3.6-35B-A3B-UD-MXFP4_K_XL-mlx	—	21 GB	54.4
Brooooooklyn/Qwen3.6-35B-A3B-UD-NVFP4_K_XL-mlx (this model)	—	22 GB	59.1
Brooooooklyn/Qwen3.6-35B-A3B-UD-Q4_K_XL-mlx	UD-Q4_K_XL	22 GB	55.1
Brooooooklyn/Qwen3.6-35B-A3B-UD-Q5_K_XL-mlx	UD-Q5_K_XL	26 GB	51.8
Brooooooklyn/Qwen3.6-35B-A3B-UD-Q6_K_XL-mlx	UD-Q6_K_XL	31 GB	54.4
Brooooooklyn/Qwen3.6-35B-A3B-UD-MXFP8_K_XL-mlx	—	35 GB	47.6
Brooooooklyn/Qwen3.6-35B-A3B-UD-Q8_K_XL-mlx	UD-Q8_K_XL	36 GB	45.9

Benchmarked on Apple M3 Max 128GB via examples/lm.ts (best decode tok/s across turns 2–4, steady-state).

Performance

Steady-state decode: 59.1 tok/s on Apple M3 Max 128GB (median best-of-T2–T4 across 3 runs, examples/lm.ts capitals chat with reasoningEffort: 'low').

Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only 8 of 256 experts per token (~3B active out of 35.9B total), and the compiled C++ forward graph fuses the per-layer dispatch.

Note: this build is ~6% slower than the prior uniform-NVFP4 release (63.0 tok/s) because the Unsloth recipe now adds 5/6/8-bit affine dequant per layer for sensitive tensors that the previous build was missing. The quality–perf trade-off is small but consistent.

Per-Tensor Bit Assignments (N=4)

Weight	Mode	Bits	Group	Rationale
embed_tokens	6-bit affine	6	64	Loader is affine-only; nvfp upgrade skipped (unsloth base+2)
lm_head	8-bit affine	8	64	Loader is affine-only; nvfp upgrade skipped (unsloth base+3 snapped 7→8)
self_attn.q/k/v_proj	6-bit affine + AWQ	6	64	AWQ via input_layernorm; preserved at higher bits (unsloth base+2)
linear_attn.in_proj_qkv/z	6-bit affine + AWQ	6	64	AWQ via input_layernorm; preserved at higher bits (unsloth base+2)
self_attn.o_proj	bf16	—	—	NOT AWQ-correctable
linear_attn.out_proj	bf16	—	—	KLD ~6.0 — worst tensor, kept full-precision
mlp.switch_mlp.down_proj	5-bit affine	5	64	"Slightly more sensitive" (unsloth base+1) — applies per-expert across all 256 experts
mlp.shared_expert.down_proj	5-bit affine	5	64	Same as switch_mlp.down_proj
mlp.switch_mlp.gate_proj, up_proj	nvfp4	4	16	Unsloth UD-Q4 base — promoted to NVFP4
mlp.shared_expert.gate_proj, up_proj	nvfp4	4	16	Same as switch_mlp
Router gates (mlp.gate, shared_expert_gate)	8-bit affine	8	64	MoE routing accuracy — FP4 noise breaks top-K
GDN params (A_log, etc)	bf16	—	—	State-space dynamics

252 per-layer overrides total (vs 80 in the previous uniform-NVFP4 build — the recipe now protects every sensitive tensor class).

Quantization Strategy

Built on Unsloth Dynamic 2.0 per-tensor KLD analysis. At --q-bits 4 the unsloth recipe's per-layer bit offsets become 4-bit FFN gate/up (promoted to NVFP4), 5-bit down_proj (per-expert across all 256 + shared expert), 6-bit attn/SSM projections with AWQ pre-scaling, 6-bit embed_tokens, and 8-bit lm_head + router gates. Then --q-mode nvfp4 orthogonally promotes only the 4-bit affine decisions to NVFP4 (mode="nvfp4", bits=4, group_size=16) — non-4-bit decisions stay affine at their original bit width, AWQ-uncorrectable projections (o_proj, out_proj) stay bf16, and router gates stay 8-bit affine to preserve top-K accuracy.

imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead). AWQ-correctable projections (q/k/v, in_proj_qkv/z) get the AWQ pass; non-AWQ-correctable projections (o_proj, out_proj) stay bf16 — their inputs come from attention/GDN computation, not from a norm layer.

Architecture

Parameter	Value
Total parameters	35.9B (3B active per token)
Hidden size	2,048
Layers	40 (30 linear + 10 full attention)
Attention heads	16 (2 KV heads, GQA 8:1)
Head dimension	256
Experts	256 per MoE layer, top-8 routing
Vocab size	248,320
Max context	262,144 tokens

Usage

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Qwen3.6-35B-A3B-UD-NVFP4_K_XL-mlx');

for await (const event of session.sendStream('Explain NVFP4 vs MXFP4 quantization.', {
  config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
  if (!event.done) process.stdout.write(event.text);
}

How It Was Made

mlx convert \
  -i Qwen3.6-35B-A3B \
  -o Qwen3.6-35B-A3B-UD-NVFP4_K_XL-mlx \
  -q --q-mode nvfp4 --q-recipe unsloth \
  --imatrix-path imatrix_unsloth.gguf

--q-mode nvfp4 selects NVIDIA's NVFP4 micro-scaling format as the global dequantizer (bits=4, group_size=16). Combined with --q-recipe unsloth, the recipe emits per-tensor affine decisions for sensitive layers (q/k/v + AWQ, down_proj, lm_head, embed_tokens, router gates at higher bits; o_proj/out_proj/GDN as bf16), and the converter promotes the remaining 4-bit affine decisions (gate_proj/up_proj on switch_mlp and shared_expert) to NVFP4. --q-bits 4 is implicit when --q-mode nvfp4 is set.

Acknowledgments

• Unsloth — Quantization strategy based on their per-layer KLD benchmarks and Dynamic 2.0 methodology
• NVIDIA NVFP4 — For the NVFP4 specification
• Qwen Team — For the Qwen3.6 model family
• Apple MLX — For the Metal-accelerated ML framework

License

Apache 2.0 (inherited from base model).