Hugging Face's logo

junafinity
/
qwen38-mtp-head-fc-bf16-4bit

Text Generation
MLX
Safetensors
qwen3_5_mtp
qwen3.8
multi-token-prediction
speculative-decoding
qwen3_5
apple-silicon
4-bit precision
Model card Files
xet
 Community
qwen38-mtp-head-fc-bf16-4bit / README.md
junafinity's picture
junafinity
docs: honesty block — not unique vision-bf16, MTP drop, no invented tok/s
d8c333b verified
preview code
|
Raw
History Blame Contribute Delete
3.5 kB
metadata 
license: apache-2.0
base_model: EigenLabs/Qwen3.8-27B-MTP-bf16
tags:
  - mlx
  - qwen3.8
  - multi-token-prediction
  - speculative-decoding
  - qwen3_5
  - apple-silicon
  - 4-bit
library_name: mlx
pipeline_tag: text-generation

Qwen3.8-27B MTP head — mixed precision (bf16 fc, 4-bit/g64 elsewhere)

A drafting head for Qwen 3.8 27B native-MTP speculative decode on MLX / Apple Silicon. This is an optional experiment, not a speed product. Acceptance on hard, varied prose is unmeasured. Do not quote tok/s for this file. Derived from the organizer-pinned EigenLabs/Qwen3.8-27B-MTP-bf16 @ 26a328e070875b0314d652a039b6b59902690f03.

Pairing

Hub collection: https://huggingface.co/collections/junafinity/qwen-38-27b-uncensored-apple-silicon-6a896c726b52be3a0b63400e

This is a drafting head only, not a standalone model. Pair it with a Qwen3.8-27B MLX backbone (for example Qwen-3.8-27B-Uncensored-8-Bit-MLX) in a loader that accepts a scales-keyed mixed-precision head.

backbone: junafinity/Qwen-3.8-27B-Uncensored-8-Bit-MLX
draft head: junafinity/qwen38-mtp-head-fc-bf16-4bit

Exact loader flags: [PLACEHOLDER] (depends on your mlx / speculative-decode entrypoint). Acceptance-rate benefit on hard, varied prose is unmeasured; see the caveat below.

What is different

Existing 4-bit heads for this model quantize all eight head linears. This one keeps fc in bf16 and quantizes only the other seven (q/k/v/o_proj, gate/up/down_proj) to MLX affine 4-bit, group size 64 — the geometry that matches the 4-bit backbone, so a stock scales-keyed loader converts exactly the modules that carry a .scales sibling and leaves fc a dense Linear with no code changes.

Why fc

Qwen's own official FP8 release, Qwen/Qwen3.8-27B-FP8, lists mtp.fc in modules_to_not_convert — it declines to quantize that matrix while quantizing the head's other large linears. fc is the projection that fuses the embedding stream with the backbone hidden state (concat[embed, hidden] at 10240 → 5120), so representation error there propagates into every downstream head computation at every draft position. This artifact tests whether respecting that exclusion improves draft acceptance enough to pay for the extra bandwidth.

Cost/benefit

fc in bf16 is 104.9 MB versus 29.5 MB at 4-bit — about +75 MB of weight traffic per draft step (total head read ≈ 314 MB vs ≈ 239 MB). Break-even needs only a small acceptance gain at mid/deep draft positions, since an accepted draft token amortizes a full ~15.6 GB target verify pass on this dense 27B model.

Contents

29 tensors: fc.weight (bf16, [5120, 10240]), seven bf16 norm vectors, and weight/scales/biases triples for the seven quantized linears. Single model.safetensors (314,300,360 bytes) plus config.json and an index, carried unchanged from the pinned head so the head loader accepts the tree.

Caveat on evaluation

Short local decode windows over a model's own greedy self-continuation saturate draft acceptance (measured: effective draft length 5.4, accept rate 1.000, identical for this head and a uniformly 4-bit one). Such a setup can measure this artifact's cost but not its benefit; evaluation needs harder, more varied prose where acceptance is genuinely below 1.