README.md

metadata

language:
  - en
  - ko
  - zh
license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
base_model_relation: quantized
library_name: mlx
pipeline_tag: image-text-to-text
tags:
  - mlx
  - qwen3_5_moe
  - quantized
  - mixed-precision
  - mixture-of-experts
  - apple-silicon
  - dwq
  - hybrid-attention
  - conversational

Qwen3.6-35B-A3B-Alis-MLX-Dynamic

Apple Silicon (MLX) mixed-precision quantizations of Qwen/Qwen3.6-35B-A3B — a 35B MoE with 3B active parameters, 3:1 Gated DeltaNet : full attention hybrid (qwen3_5_moe). One repo, one card — each quantization lives on its own branch.

branch	eff. bpw	size	target Mac	decode*	KL vs bf16 (top-1 flip)	status
main	4.000	17.33 GB	24 GB+	102.1 tok/s	0.292 (19.7%)	✅ golden — GPQA-D 76.3%
16gb	2.700	11.70 GB	16 GB (sysctl)	110.8 tok/s	1.111 (32.9%)	⚠️ under review — see note
vision	4.087	17.93 GB	24 GB+	~100 tok/s	(same text weights)	✅ multimodal — vision tower restored at 8-bit
mtp-bf16	bf16	1.69 GB	—	—	—	MTP head sidecar for self-speculative decoding

* single-stream decode on an M3 Ultra, short prompt. A3B routing makes this the fastest model class per GB in the lineup — ~3× the decode rate of the dense 27B sibling.

Everything below is measured, not projected — KL / top-1 flip on a fixed 4096-token EN/code/ZH/KO slice (teacher-forced vs bf16), strided perplexity (ctx 2048 / stride 1024), and single-stream decode on an M3 Ultra.

Which build?

• 24 GB+ Mac → main. 16 GiB weights, 17.5 GB measured peak at short context, 102 tok/s.

• 16 GB Mac → 16gb. 11 GiB weights, 11.8 GB measured peak — exceeds the default ~10.7 GiB wired limit; raise it first:

  sudo sysctl iogpu.wired_limit_mb=13312
  

  With int8 KV that leaves ~1.5 GiB of cache ≈ 150 K tokens of context (KV here is tiny — see below). 111 tok/s decode makes this the best speed-per-GB option for 16 GB machines; prefer it over the dense 27B 16 GB build when throughput matters more than per-token quality.

Multimodal (default)

main carries the full multimodal stack — 4.0bpw text weights + the vision tower at 8-bit (verified with EN/KO OCR tests). Plain mlx-lm loads main fine for text-only use; the text branch saves 0.60 GB if you never need images.

pip install mlx-vlm
python -m mlx_vlm generate --model avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic \
  --image photo.jpg --prompt "이미지의 텍스트를 읽어줘." --max-tokens 300

Usage

from mlx_lm import load, generate

model, tokenizer = load("avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic")                  # main
model, tokenizer = load("avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic", revision="16gb")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "MoE 라우팅을 두 문장으로 설명해줘."}],
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

CLI / server:

mlx_lm.generate --model avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic --prompt "..." -m 512
hf download avlp12/Qwen3.6-35B-A3B-Alis-MLX-Dynamic --revision 16gb --local-dir qwen36-35b-16gb
mlx_lm.server --model ./qwen36-35b-16gb --port 8080

Long context: --kv-bits 8 --kv-group-size 64 in the generate path (measured quality-free, below).

⚠️ 16gb under review. The dense-27B sibling's same-calibration low-bit tier was withdrawn after free-running generation showed early stops / repetition loops on long-CoT problems (teacher-forced KL/PPL did not catch it). This branch shares that calibration recipe and is being re-verified generatively; treat it as experimental until this notice is removed.

Recipes (verified from each build's config.json)

92% of the parameters live in 256 fused routed experts per MoE layer (stored as 3D SwitchLinear tensors — the sensitivity scan samples 8 experts/layer/matrix). Per-tensor bits assigned by marginal utility from a 2–6-bit + nvfp4 scan of all 472 text tensors:

tier	bulk	rest	pins
main (4.0bpw)	50.3% affine 3-bit g64	49.5% nvfp4 g16	routers/shared-expert gates 6-bit; embed & lm_head ≥4-bit
16gb (2.7bpw)	83.6% affine 2-bit g64	12.9% 3-bit + 3.4% nvfp4	same

nvfp4 vs affine — measured both ways. On the dense 27B sibling, affine-only beat nvfp4 decisively (its card has the numbers). Here the reverse holds: an affine-only 4.0bpw rebuild (50.3% a3 + 49.4% a4) scored KL 0.309 vs main's nvfp4-mix 0.292 — nvfp4 g16 genuinely suits the small fused expert matrices (2048×512). The format choice is architecture-dependent; main stands as-is.

DWQ. Layerwise DWQ (KL distillation vs bf16, 260-sample EN/KO/ZH/code chat mix, seq 512, validation-gated — alis-dwq): at 4.0bpw every round was rejected (already teacher-close), so main carries conversion-time weights; at 2.7bpw DWQ improved valid KL 0.305 → 0.268 (−12%).

Quality

KL(bf16‖quant) / top-1 flip on the fixed 4096-token slice, per domain:

build	overall	EN	code	ZH	KO
main 4.0bpw	0.292 / 19.7%	0.100 / 11.2%	0.622 / 19.7%	0.266 / 27.9%	0.181 / 19.9%
16gb 2.7bpw	1.111 / 32.9%	0.365 / 21.8%	2.454 / 27.3%	1.004 / 47.7%	0.622 / 34.8%

Strided perplexity (ctx 2048, stride 1024; wikitext / code / Korean):

build	wikitext	code	KO
main	6.16 [6.07, 6.25]	2.12 [2.10, 2.14]	7.09 [7.02, 7.17]
16gb	7.65 [7.54, 7.76]	2.57 [2.54, 2.60]	9.04 [8.94, 9.14]

Generative benchmark (GPQA-Diamond, our harness, n=198, temp 1.0, 32K cap)

main scores 76.3% (avg 14,742 thinking tokens, 7 no-answer). For scale: a dense 27B at 4.6bpw scores 72.7 on the same harness, and quantized short-thinking models reach 83.3 — see the cross-family curve on the ThinkingCap repo. The 16gb 2.7bpw tier fails hard reasoning generatively (MMLU-300 75.3% but GPQA non-convergent) — hence the warning above; treat it as a light/general-use build only.

KV cache — measured recommendations

Only 10 of 40 layers carry KV (the rest are Gated DeltaNet recurrent state), so KV is tiny to begin with; quantizing it is still measured quality-free:

KV config	KL vs bf16-KV	KV bytes/token	~ctx in 1.5 GiB
bf16	0	20 KB	75 K
int8 g64 (recommended)	0.011	10 KB	150 K
K4/V8 (long-context)	0.016	7.5 KB	200 K

Notes & provenance

• Text-only conversion: the base checkpoint's vision tower and MTP head are not included (mlx-lm text path). An mtp-bf16 sidecar branch may land later.
• Calibration/eval data: tulu-3 SFT chat (EN), Korean chat, code/ZH mix — chat-template rendered.
• Built on an M3 Ultra 512 GB · mlx-lm 0.31.3 · mlx 0.31.2 · 2026-07.
• Sibling repo: Qwen3.6-27B-Alis-MLX-Dynamic (dense, higher per-token quality; its 16gb-thinkingcap branch fits 16 GB Macs without sysctl).

Credits

• Qwen team — the Qwen3.6-35B-A3B base model and its hybrid DeltaNet MoE architecture.
• Apple MLX team — mlx / mlx-lm, including the DWQ trainer this pipeline builds on.
• Unsloth — their NVFP4 releases motivated the diverse-chat calibration mix and served as quality reference points.
• AllenAI — tulu-3 SFT mixture used in the calibration set.
• Quantization recipe, DWQ retune and evals: Alis (avlp12) with the alis-dwq pipeline; built with assistance from Claude (Anthropic).