--- license: apache-2.0 language: - en - zh base_model: Qwen/Qwen3.6-27B tags: - mlx - mlx-node - quantized - awq - mtp - 4-bit - qwen3.6 - hybrid-attention - gated-delta-net - apple-silicon - unsloth-dynamic library_name: mlx-node quantized_by: igorvibes pipeline_tag: text-generation model_type: qwen3_5 --- # Qwen3.6-27B — UD-Q4_K_XL + AWQ + MTP (MLX) Mixed-precision 4-bit quantization of [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) for Apple Silicon, built with [mlx-node](https://github.com/mlx-node/mlx-node) using the **Unsloth Dynamic** class map plus **AWQ pre-scaling** from an activation imatrix. **The multi-token-prediction (MTP) head is preserved.** All 15 `mtp.*` tensors are kept inline in the main shards, unquantized in BF16, so a runtime that supports Qwen3.5/3.6 speculative decoding can use them. This is the one deliberate difference from the [Brooooooklyn UD-Q*_K_XL](https://huggingface.co/Brooooooklyn/Qwen3.6-27B-UD-Q4_K_XL-mlx) builds this recipe otherwise follows — I checked their Q4 checkpoint and it carries zero `mtp.*` tensors. Other MLX conversions of this model do ship MTP; this is not a claim to be the only one. ## Contents | Component | Tensors | On disk | Precision | |---|---:|---:|---| | Language model | 1,847 | 17.81 GiB | mixed 4–8 bit | | Vision tower | 333 | 0.86 GiB | BF16 (unquantized) | | MTP head | 15 | 0.79 GiB | BF16 (unquantized) | | **Total** | **2,195** | **19.45 GiB** | | affine quantization, group size 64, 370 per-tensor overrides. 27,781,427,952 total parameters. ## Effective bits per weight **6.18 BPW** — 20,889,186,434 bytes on disk over 27,020,391,152 logical parameters. This is well above the nominal 4, and that is expected rather than a defect. `--q-bits 4` sets the *base* class in the Unsloth Dynamic map, not the average. Only `gate_proj` and `up_proj` actually sit at 4-bit; attention projections, the GatedDeltaNet input projections, `down_proj`, the embeddings and `lm_head` are all promoted. The BF16 vision tower and BF16 MTP head add further weight that no quantizer touches. Any "Q4" MLX quant of this family that reports a similar size is doing the same thing. ## Precision map | Module | Width | Scope | |---|---|---| | `down_proj` | 5-bit | 64 layers (0–63) | | `embed_tokens` | 6-bit | single tensor | | `in_proj_a` | 8-bit | 48 layers (0–62) | | `in_proj_b` | 8-bit | 48 layers (0–62) | | `in_proj_qkv` | 6-bit | 48 layers (0–62) | | `in_proj_z` | 6-bit | 48 layers (0–62) | | `k_proj` | 6-bit | 16 layers (3–63) | | `lm_head` | 8-bit | single tensor | | `o_proj` | 8-bit | 16 layers (3–63) | | `out_proj` | 8-bit | 48 layers (0–62) | | `q_proj` | 6-bit | 16 layers (3–63) | | `v_proj` | 6-bit | 16 layers (3–63) | | `gate_proj`, `up_proj` | **4-bit** (base) | all 64 layers | Left in BF16 throughout: all RMSNorms, `q_norm`/`k_norm`, the GatedDeltaNet `A_log` / `conv1d` / `dt_bias` state parameters, the entire vision tower, and the entire MTP head. ## AWQ calibration AWQ pre-scaling amplifies activation-important weight columns and folds the inverse into the preceding norm — an output-preserving reparametrization that moves quantization error onto channels that matter less. Applied here across four dependency groups: norm→gate/up, up-rows→down-cols, input_layernorm→q/k/v, and input_layernorm→GatedDeltaNet `in_proj_*`. The importance matrix is Unsloth's own calibration for **this exact model** — `imatrix_unsloth.gguf_file` from [unsloth/Qwen3.6-27B-GGUF](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF), internally tagged `unsloth_calibration_Qwen3.6-27B.txt`, 992 tensors over 76 chunks. An imatrix from a different checkpoint would apply without error and silently degrade the result, so this provenance matters. ## Reproducing ```bash mlx convert \ --input \ --output Qwen3.6-27B-UD-Q4_K_XL-AWQ-MTP-mlx \ --model-type qwen3_5 \ --quantize \ --q-bits 4 \ --q-group-size 64 \ --q-recipe unsloth \ --imatrix-path imatrix_unsloth.gguf_file \ --q-mtp off ``` `--q-mtp off` means "do not split the MTP head into a separate drafter directory" — the tensors stay inline and unquantized. Use `--q-mtp split` instead if you want a standalone drafter. Conversion cost on a 36 GB M-series Mac: **99 s**, peak 15.85 GiB process RSS, 22.7 GiB MLX allocator peak. Swap: 7.89 GiB paged out / 5.76 GiB paged in (`vm_stat`, sampled across the run). This needs a build of mlx-node containing the bounded-memory conversion fix ([PR #118](https://github.com/mlx-node/mlx-node/pull/118)); before it, the AWQ path materialized the whole BF16 checkpoint at once and would not complete in 36 GB. ## Variants | Repo | Base width | Size | BPW | |---|---|---:|---:| | [igorvibes/Qwen3.6-27B-UD-Q5_K_XL-AWQ-MTP-mlx](https://huggingface.co/igorvibes/Qwen3.6-27B-UD-Q5_K_XL-AWQ-MTP-mlx) | 5-bit | 22.95 GiB | 7.38 | | **This model** | **4-bit** | **19.45 GiB** | **6.18** | ## Sources - Base weights — [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) - Importance matrix — [unsloth/Qwen3.6-27B-GGUF](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF) - Quantization strategy — [Unsloth Dynamic](https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks) - Conversion tool — [mlx-node](https://github.com/mlx-node/mlx-node) - Recipe reference — [Brooooooklyn/Qwen3.6-27B-UD-Q4_K_XL-mlx](https://huggingface.co/Brooooooklyn/Qwen3.6-27B-UD-Q4_K_XL-mlx), whose flag set this follows apart from MTP retention ## Not tested Stated plainly so you can weigh it: - **No benchmarks were run on this build.** No perplexity, no task evals, no throughput numbers. Nothing here claims a quality or speed result. - **MTP speculative decoding is preserved, not verified.** The tensors are present and correctly shaped; whether your runtime engages them is on your runtime. - **The vision tower is carried through unquantized but untested.** No image or video input was exercised. - License is stated as Apache-2.0 following the upstream Qwen3 convention; verify against the base repo if it matters to you.