Hugging Face's logo

junafinity
/
qwen38-mtp-head-fc-bf16-4bit

Text Generation
MLX
Safetensors
qwen3_5_mtp
qwen3.8
multi-token-prediction
speculative-decoding
qwen3_5
apple-silicon
4-bit precision
Model card Files
xet
 Community
junafinity commited on
Commit
f3c7034
·
verified ·
1 Parent(s): e73c746

Mixed-precision MTP head: bf16 fc, 4-bit/g64 elsewhere

Browse files
Files changed (4) hide show
  1. README.md +58 -0
  2. config.json +116 -0
  3. model.safetensors +3 -0
  4. model.safetensors.index.json +36 -0
README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: EigenLabs/Qwen3.8-27B-MTP-bf16
4
+ tags:
5
+ - mlx
6
+ - qwen3.8
7
+ - multi-token-prediction
8
+ - speculative-decoding
9
+ ---
10
+
11
+ # Qwen3.8-27B MTP head — mixed precision (bf16 `fc`, 4-bit/g64 elsewhere)
12
+
13
+ A drafting head for Qwen 3.8 27B native-MTP speculative decode on MLX /
14
+ Apple Silicon. Derived from the organizer-pinned
15
+ `EigenLabs/Qwen3.8-27B-MTP-bf16` @ `26a328e070875b0314d652a039b6b59902690f03`.
16
+
17
+ ## What is different
18
+
19
+ Existing 4-bit heads for this model quantize all eight head linears. This one
20
+ keeps **`fc` in bf16** and quantizes only the other seven (q/k/v/o_proj,
21
+ gate/up/down_proj) to MLX affine 4-bit, group size 64 — the geometry that
22
+ matches the 4-bit backbone, so a stock scales-keyed loader converts exactly
23
+ the modules that carry a `.scales` sibling and leaves `fc` a dense `Linear`
24
+ with no code changes.
25
+
26
+ ## Why `fc`
27
+
28
+ Qwen's own official FP8 release, `Qwen/Qwen3.8-27B-FP8`, lists `mtp.fc` in
29
+ `modules_to_not_convert` — it declines to quantize that matrix while
30
+ quantizing the head's other large linears. `fc` is the projection that fuses
31
+ the embedding stream with the backbone hidden state
32
+ (`concat[embed, hidden]` at 10240 → 5120), so representation error there
33
+ propagates into every downstream head computation at every draft position.
34
+ This artifact tests whether respecting that exclusion improves draft
35
+ acceptance enough to pay for the extra bandwidth.
36
+
37
+ ## Cost/benefit
38
+
39
+ `fc` in bf16 is 104.9 MB versus 29.5 MB at 4-bit — about **+75 MB of weight
40
+ traffic per draft step** (total head read ≈ 314 MB vs ≈ 239 MB). Break-even
41
+ needs only a small acceptance gain at mid/deep draft positions, since an
42
+ accepted draft token amortizes a full ~15.6 GB target verify pass on this
43
+ dense 27B model.
44
+
45
+ ## Contents
46
+
47
+ 29 tensors: `fc.weight` (bf16, `[5120, 10240]`), seven bf16 norm vectors, and
48
+ weight/scales/biases triples for the seven quantized linears. Single
49
+ `model.safetensors` (314,300,360 bytes) plus `config.json` and an index,
50
+ carried unchanged from the pinned head so the head loader accepts the tree.
51
+
52
+ ## Caveat on evaluation
53
+
54
+ Short local decode windows over a model's own greedy self-continuation
55
+ saturate draft acceptance (measured: effective draft length 5.4, accept rate
56
+ 1.000, identical for this head and a uniformly 4-bit one). Such a setup can
57
+ measure this artifact's cost but not its benefit; evaluation needs harder,
58
+ more varied prose where acceptance is genuinely below 1.
config.json ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "block_size": 3,
3
+ "model_type": "qwen3_5_mtp",
4
+ "text_config": {
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "attn_output_gate": true,
8
+ "bos_token_id": 248044,
9
+ "dtype": "bfloat16",
10
+ "eos_token_id": 248044,
11
+ "full_attention_interval": 4,
12
+ "head_dim": 256,
13
+ "hidden_act": "silu",
14
+ "hidden_size": 5120,
15
+ "initializer_range": 0.02,
16
+ "intermediate_size": 17408,
17
+ "layer_types": [
18
+ "linear_attention",
19
+ "linear_attention",
20
+ "linear_attention",
21
+ "full_attention",
22
+ "linear_attention",
23
+ "linear_attention",
24
+ "linear_attention",
25
+ "full_attention",
26
+ "linear_attention",
27
+ "linear_attention",
28
+ "linear_attention",
29
+ "full_attention",
30
+ "linear_attention",
31
+ "linear_attention",
32
+ "linear_attention",
33
+ "full_attention",
34
+ "linear_attention",
35
+ "linear_attention",
36
+ "linear_attention",
37
+ "full_attention",
38
+ "linear_attention",
39
+ "linear_attention",
40
+ "linear_attention",
41
+ "full_attention",
42
+ "linear_attention",
43
+ "linear_attention",
44
+ "linear_attention",
45
+ "full_attention",
46
+ "linear_attention",
47
+ "linear_attention",
48
+ "linear_attention",
49
+ "full_attention",
50
+ "linear_attention",
51
+ "linear_attention",
52
+ "linear_attention",
53
+ "full_attention",
54
+ "linear_attention",
55
+ "linear_attention",
56
+ "linear_attention",
57
+ "full_attention",
58
+ "linear_attention",
59
+ "linear_attention",
60
+ "linear_attention",
61
+ "full_attention",
62
+ "linear_attention",
63
+ "linear_attention",
64
+ "linear_attention",
65
+ "full_attention",
66
+ "linear_attention",
67
+ "linear_attention",
68
+ "linear_attention",
69
+ "full_attention",
70
+ "linear_attention",
71
+ "linear_attention",
72
+ "linear_attention",
73
+ "full_attention",
74
+ "linear_attention",
75
+ "linear_attention",
76
+ "linear_attention",
77
+ "full_attention",
78
+ "linear_attention",
79
+ "linear_attention",
80
+ "linear_attention",
81
+ "full_attention"
82
+ ],
83
+ "linear_conv_kernel_dim": 4,
84
+ "linear_key_head_dim": 128,
85
+ "linear_num_key_heads": 16,
86
+ "linear_num_value_heads": 48,
87
+ "linear_value_head_dim": 128,
88
+ "mamba_ssm_dtype": "float32",
89
+ "max_position_embeddings": 262144,
90
+ "model_type": "qwen3_5_text",
91
+ "mtp_num_hidden_layers": 1,
92
+ "mtp_use_dedicated_embeddings": false,
93
+ "num_attention_heads": 24,
94
+ "num_hidden_layers": 64,
95
+ "num_key_value_heads": 4,
96
+ "output_gate_type": "swish",
97
+ "pad_token_id": null,
98
+ "partial_rotary_factor": 0.25,
99
+ "rms_norm_eps": 1e-06,
100
+ "rope_parameters": {
101
+ "mrope_interleaved": true,
102
+ "mrope_section": [
103
+ 11,
104
+ 11,
105
+ 10
106
+ ],
107
+ "partial_rotary_factor": 0.25,
108
+ "rope_theta": 10000000,
109
+ "type": "default"
110
+ },
111
+ "tie_word_embeddings": false,
112
+ "use_cache": true,
113
+ "vocab_size": 248320
114
+ },
115
+ "tie_word_embeddings": false
116
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:341e6bf529455401595ac10c7166c492abbeb75cd5fbe95d1007446f1e13d5d1
3
+ size 314300360
model.safetensors.index.json ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 314297344
4
+ },
5
+ "weight_map": {
6
+ "fc.weight": "model.safetensors",
7
+ "layers.0.input_layernorm.weight": "model.safetensors",
8
+ "layers.0.mlp.down_proj.biases": "model.safetensors",
9
+ "layers.0.mlp.down_proj.scales": "model.safetensors",
10
+ "layers.0.mlp.down_proj.weight": "model.safetensors",
11
+ "layers.0.mlp.gate_proj.biases": "model.safetensors",
12
+ "layers.0.mlp.gate_proj.scales": "model.safetensors",
13
+ "layers.0.mlp.gate_proj.weight": "model.safetensors",
14
+ "layers.0.mlp.up_proj.biases": "model.safetensors",
15
+ "layers.0.mlp.up_proj.scales": "model.safetensors",
16
+ "layers.0.mlp.up_proj.weight": "model.safetensors",
17
+ "layers.0.post_attention_layernorm.weight": "model.safetensors",
18
+ "layers.0.self_attn.k_norm.weight": "model.safetensors",
19
+ "layers.0.self_attn.k_proj.biases": "model.safetensors",
20
+ "layers.0.self_attn.k_proj.scales": "model.safetensors",
21
+ "layers.0.self_attn.k_proj.weight": "model.safetensors",
22
+ "layers.0.self_attn.o_proj.biases": "model.safetensors",
23
+ "layers.0.self_attn.o_proj.scales": "model.safetensors",
24
+ "layers.0.self_attn.o_proj.weight": "model.safetensors",
25
+ "layers.0.self_attn.q_norm.weight": "model.safetensors",
26
+ "layers.0.self_attn.q_proj.biases": "model.safetensors",
27
+ "layers.0.self_attn.q_proj.scales": "model.safetensors",
28
+ "layers.0.self_attn.q_proj.weight": "model.safetensors",
29
+ "layers.0.self_attn.v_proj.biases": "model.safetensors",
30
+ "layers.0.self_attn.v_proj.scales": "model.safetensors",
31
+ "layers.0.self_attn.v_proj.weight": "model.safetensors",
32
+ "norm.weight": "model.safetensors",
33
+ "pre_fc_norm_embedding.weight": "model.safetensors",
34
+ "pre_fc_norm_hidden.weight": "model.safetensors"
35
+ }
36
+ }