File size: 3,502 Bytes
f3c7034 419d634 f3c7034 d8c333b f3c7034 419d634 c8bbdee 419d634 f3c7034 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
license: apache-2.0
base_model: EigenLabs/Qwen3.8-27B-MTP-bf16
tags:
- mlx
- qwen3.8
- multi-token-prediction
- speculative-decoding
- qwen3_5
- apple-silicon
- 4-bit
library_name: mlx
pipeline_tag: text-generation
---
# Qwen3.8-27B MTP head — mixed precision (bf16 `fc`, 4-bit/g64 elsewhere)
A drafting head for Qwen 3.8 27B native-MTP speculative decode on MLX /
Apple Silicon. This is an **optional experiment, not a speed product**. Acceptance on hard, varied prose is **unmeasured**. Do not quote tok/s for this file. Derived from the organizer-pinned
`EigenLabs/Qwen3.8-27B-MTP-bf16` @ `26a328e070875b0314d652a039b6b59902690f03`.
## Pairing
Hub collection: [https://huggingface.co/collections/junafinity/qwen-38-27b-uncensored-apple-silicon-6a896c726b52be3a0b63400e](https://huggingface.co/collections/junafinity/qwen-38-27b-uncensored-apple-silicon-6a896c726b52be3a0b63400e)
This is a **drafting head only**, not a standalone model. Pair it with a Qwen3.8-27B MLX backbone (for example [Qwen-3.8-27B-Uncensored-8-Bit-MLX](https://huggingface.co/junafinity/Qwen-3.8-27B-Uncensored-8-Bit-MLX)) in a loader that accepts a scales-keyed mixed-precision head.
```text
backbone: junafinity/Qwen-3.8-27B-Uncensored-8-Bit-MLX
draft head: junafinity/qwen38-mtp-head-fc-bf16-4bit
```
Exact loader flags: **[PLACEHOLDER]** (depends on your mlx / speculative-decode entrypoint). Acceptance-rate benefit on hard, varied prose is **unmeasured**; see the caveat below.
## What is different
Existing 4-bit heads for this model quantize all eight head linears. This one
keeps **`fc` in bf16** and quantizes only the other seven (q/k/v/o_proj,
gate/up/down_proj) to MLX affine 4-bit, group size 64 — the geometry that
matches the 4-bit backbone, so a stock scales-keyed loader converts exactly
the modules that carry a `.scales` sibling and leaves `fc` a dense `Linear`
with no code changes.
## Why `fc`
Qwen's own official FP8 release, `Qwen/Qwen3.8-27B-FP8`, lists `mtp.fc` in
`modules_to_not_convert` — it declines to quantize that matrix while
quantizing the head's other large linears. `fc` is the projection that fuses
the embedding stream with the backbone hidden state
(`concat[embed, hidden]` at 10240 → 5120), so representation error there
propagates into every downstream head computation at every draft position.
This artifact tests whether respecting that exclusion improves draft
acceptance enough to pay for the extra bandwidth.
## Cost/benefit
`fc` in bf16 is 104.9 MB versus 29.5 MB at 4-bit — about **+75 MB of weight
traffic per draft step** (total head read ≈ 314 MB vs ≈ 239 MB). Break-even
needs only a small acceptance gain at mid/deep draft positions, since an
accepted draft token amortizes a full ~15.6 GB target verify pass on this
dense 27B model.
## Contents
29 tensors: `fc.weight` (bf16, `[5120, 10240]`), seven bf16 norm vectors, and
weight/scales/biases triples for the seven quantized linears. Single
`model.safetensors` (314,300,360 bytes) plus `config.json` and an index,
carried unchanged from the pinned head so the head loader accepts the tree.
## Caveat on evaluation
Short local decode windows over a model's own greedy self-continuation
saturate draft acceptance (measured: effective draft length 5.4, accept rate
1.000, identical for this head and a uniformly 4-bit one). Such a setup can
measure this artifact's cost but not its benefit; evaluation needs harder,
more varied prose where acceptance is genuinely below 1.
|