Hugging Face's logo

divydeep
/
granite-speech-4.1-2b-mlx-5bit

Automatic Speech Recognition
MLX
Safetensors
granite_speech
mlx-audio
speech-to-text
speech-translation
granite
granite-speech
autoregressive
quantized
5-bit
Model card Files
xet
 Community
divydeep commited on
Commit
da53ca8
·
verified ·
1 Parent(s): f6194f1

Improve mixed-precision quantization and add model card

Browse files
Files changed (4) hide show
  1. README.md +159 -0
  2. config.json +810 -2
  3. model.safetensors +2 -2
  4. model.safetensors.index.json +1 -5
README.md ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: mlx
4
+ pipeline_tag: automatic-speech-recognition
5
+ base_model: ibm-granite/granite-speech-4.1-2b
6
+ base_model_relation: quantized
7
+ language:
8
+ - multilingual
9
+ - en
10
+ - fr
11
+ - de
12
+ - es
13
+ - pt
14
+ - ja
15
+ tags:
16
+ - mlx
17
+ - mlx-audio
18
+ - automatic-speech-recognition
19
+ - speech-to-text
20
+ - speech-translation
21
+ - granite
22
+ - granite-speech
23
+ - autoregressive
24
+ - quantized
25
+ - 5-bit
26
+ ---
27
+
28
+ # Granite Speech 4.1 2B MLX 5-bit
29
+
30
+ Quality-oriented mixed-precision MLX conversion of
31
+ [`ibm-granite/granite-speech-4.1-2b`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b)
32
+ for Apple Silicon. This is the **autoregressive Granite Speech model**, not the
33
+ separate NAR variant.
34
+
35
+ The model runs with
36
+ [`mlx-audio`](https://github.com/Blaizzy/mlx-audio) and supports multilingual
37
+ automatic speech recognition (ASR), speech translation (AST), punctuation and
38
+ capitalization, and keyword-biased transcription.
39
+
40
+ ## Quantization
41
+
42
+ This conversion uses post-training weight quantization only. No training,
43
+ fine-tuning, calibration dataset, or importance matrix was used.
44
+
45
+ | Component | Precision |
46
+ |---|---|
47
+ | 16-layer Conformer speech encoder | BF16 |
48
+ | 2-layer Q-Former speech projector | BF16 |
49
+ | Granite language model, ordinary eligible internal linear layers | MLX affine 5-bit, group size 64 |
50
+ | All `v_proj` and `down_proj` layers | MLX affine 6-bit, group size 64 |
51
+ | Token embedding | BF16 |
52
+ | Language-model output head | BF16 |
53
+ | Norms, biases, and unsupported tensors | BF16 |
54
+
55
+ Value and down projections receive extra precision because mixed-tensor
56
+ quantizers commonly protect these sensitive paths. The acoustic stack, token
57
+ embedding, and output head remain BF16. This is conceptually similar to medium
58
+ mixed-tensor quantizations such as GGUF Q5_K_M, but the file format and
59
+ numerical representation are MLX affine quantization rather than GGUF K-quants.
60
+
61
+ - Source revision: `de575db64086f84fdc79da4932d1076e965bc546`
62
+ - Effective average reported by MLX: 9.691 bits per weight
63
+ - `model.safetensors`: approximately 2.5 GB
64
+
65
+ ## Requirements
66
+
67
+ - Apple Silicon Mac
68
+ - macOS 14 or later
69
+ - `mlx-audio >= 0.4.5`
70
+
71
+ ```bash
72
+ pip install -U mlx-audio
73
+ ```
74
+
75
+ ## Usage
76
+
77
+ ```bash
78
+ python -m mlx_audio.stt.generate \
79
+ --model divydeep/granite-speech-4.1-2b-mlx-5bit \
80
+ --audio audio.wav \
81
+ --output-path transcript \
82
+ --format txt \
83
+ --prompt "transcribe the speech with proper punctuation and capitalization."
84
+ ```
85
+
86
+ Python:
87
+
88
+ ```python
89
+ from mlx_audio.stt.utils import load_model
90
+
91
+ model = load_model("divydeep/granite-speech-4.1-2b-mlx-5bit")
92
+ result = model.generate(
93
+ "audio.wav",
94
+ prompt="transcribe the speech with proper punctuation and capitalization.",
95
+ temperature=0.0,
96
+ )
97
+ print(result.text)
98
+ ```
99
+
100
+ Greedy decoding with `temperature=0.0` is recommended for transcription.
101
+
102
+ ## Prompts
103
+
104
+ | Task | Prompt |
105
+ |---|---|
106
+ | Raw ASR | `can you transcribe the speech into a written format?` |
107
+ | Punctuated ASR | `transcribe the speech with proper punctuation and capitalization.` |
108
+ | Keyword-biased ASR | `transcribe the speech to text. Keywords: <kw1>, <kw2>, ...` |
109
+ | Speech translation | `translate the speech to <language>.` |
110
+ | Punctuated translation | `translate the speech to <language> with proper punctuation and capitalization.` |
111
+
112
+ For non-English punctuation, translation, and keyword biasing, use the English
113
+ prompt forms recommended by IBM.
114
+
115
+ ## Validation
116
+
117
+ The conversion was validated by:
118
+
119
+ - Strictly loading all weights with `mlx-audio`.
120
+ - Confirming the architecture is `granite_speech`, not `granite_speech_nar`.
121
+ - Running greedy transcription on IBM's bundled `multilingual_sample.wav`.
122
+ - Checking preservation of English and French text, punctuation, accents, and
123
+ capitalization in the generated transcript.
124
+
125
+ This is a conversion smoke test, not a full WER benchmark. The upstream IBM
126
+ evaluation results describe the original model and should not be interpreted as
127
+ measured results for this quantized conversion.
128
+
129
+ ## Limitations
130
+
131
+ - Quantization may affect rare words, names, punctuation, casing, keyword
132
+ biasing, translation, and difficult or noisy audio.
133
+ - This model has not been independently evaluated on the complete IBM benchmark
134
+ suite.
135
+ - The model is autoregressive and does not provide the throughput behavior of
136
+ the separate NAR architecture.
137
+ - Refer to the upstream model card for intended use, language coverage, safety
138
+ considerations, training data, and architectural details.
139
+
140
+ ## References
141
+
142
+ - [Original model and authoritative model card](https://huggingface.co/ibm-granite/granite-speech-4.1-2b)
143
+ - [Granite Speech paper](https://arxiv.org/abs/2505.08699)
144
+ - [MLX Audio](https://github.com/Blaizzy/mlx-audio)
145
+
146
+ ## License
147
+
148
+ Apache-2.0, matching the original model.
149
+
150
+ ## Citation
151
+
152
+ ```bibtex
153
+ @misc{granite-speech-4.1-2b,
154
+ title={Granite 4.1 Speech},
155
+ author={IBM Granite Speech Team},
156
+ year={2026},
157
+ url={https://huggingface.co/ibm-granite/granite-speech-4.1-2b}
158
+ }
159
+ ```
config.json CHANGED
@@ -3,6 +3,14 @@
3
  "GraniteSpeechForConditionalGeneration"
4
  ],
5
  "audio_token_index": 100352,
 
 
 
 
 
 
 
 
6
  "downsample_rate": 5,
7
  "dtype": "bfloat16",
8
  "encoder_config": {
@@ -47,12 +55,812 @@
47
  "quantization": {
48
  "group_size": 64,
49
  "bits": 5,
50
- "mode": "affine"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
51
  },
52
  "quantization_config": {
53
  "group_size": 64,
54
  "bits": 5,
55
- "mode": "affine"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  },
57
  "text_config": {
58
  "_name_or_path": "ibm-granite/granite-4.0-1b-base",
 
3
  "GraniteSpeechForConditionalGeneration"
4
  ],
5
  "audio_token_index": 100352,
6
+ "conversion": {
7
+ "base_model": "ibm-granite/granite-speech-4.1-2b",
8
+ "base_revision": "de575db64086f84fdc79da4932d1076e965bc546",
9
+ "method": "post-training weight quantization",
10
+ "calibration_dataset": null,
11
+ "training": false,
12
+ "variant": "5bit"
13
+ },
14
  "downsample_rate": 5,
15
  "dtype": "bfloat16",
16
  "encoder_config": {
 
55
  "quantization": {
56
  "group_size": 64,
57
  "bits": 5,
58
+ "mode": "affine",
59
+ "language_model.model.layers.0.self_attn.v_proj": {
60
+ "group_size": 64,
61
+ "bits": 6,
62
+ "mode": "affine"
63
+ },
64
+ "language_model.model.layers.0.mlp.down_proj": {
65
+ "group_size": 64,
66
+ "bits": 6,
67
+ "mode": "affine"
68
+ },
69
+ "language_model.model.layers.1.self_attn.v_proj": {
70
+ "group_size": 64,
71
+ "bits": 6,
72
+ "mode": "affine"
73
+ },
74
+ "language_model.model.layers.1.mlp.down_proj": {
75
+ "group_size": 64,
76
+ "bits": 6,
77
+ "mode": "affine"
78
+ },
79
+ "language_model.model.layers.2.self_attn.v_proj": {
80
+ "group_size": 64,
81
+ "bits": 6,
82
+ "mode": "affine"
83
+ },
84
+ "language_model.model.layers.2.mlp.down_proj": {
85
+ "group_size": 64,
86
+ "bits": 6,
87
+ "mode": "affine"
88
+ },
89
+ "language_model.model.layers.3.self_attn.v_proj": {
90
+ "group_size": 64,
91
+ "bits": 6,
92
+ "mode": "affine"
93
+ },
94
+ "language_model.model.layers.3.mlp.down_proj": {
95
+ "group_size": 64,
96
+ "bits": 6,
97
+ "mode": "affine"
98
+ },
99
+ "language_model.model.layers.4.self_attn.v_proj": {
100
+ "group_size": 64,
101
+ "bits": 6,
102
+ "mode": "affine"
103
+ },
104
+ "language_model.model.layers.4.mlp.down_proj": {
105
+ "group_size": 64,
106
+ "bits": 6,
107
+ "mode": "affine"
108
+ },
109
+ "language_model.model.layers.5.self_attn.v_proj": {
110
+ "group_size": 64,
111
+ "bits": 6,
112
+ "mode": "affine"
113
+ },
114
+ "language_model.model.layers.5.mlp.down_proj": {
115
+ "group_size": 64,
116
+ "bits": 6,
117
+ "mode": "affine"
118
+ },
119
+ "language_model.model.layers.6.self_attn.v_proj": {
120
+ "group_size": 64,
121
+ "bits": 6,
122
+ "mode": "affine"
123
+ },
124
+ "language_model.model.layers.6.mlp.down_proj": {
125
+ "group_size": 64,
126
+ "bits": 6,
127
+ "mode": "affine"
128
+ },
129
+ "language_model.model.layers.7.self_attn.v_proj": {
130
+ "group_size": 64,
131
+ "bits": 6,
132
+ "mode": "affine"
133
+ },
134
+ "language_model.model.layers.7.mlp.down_proj": {
135
+ "group_size": 64,
136
+ "bits": 6,
137
+ "mode": "affine"
138
+ },
139
+ "language_model.model.layers.8.self_attn.v_proj": {
140
+ "group_size": 64,
141
+ "bits": 6,
142
+ "mode": "affine"
143
+ },
144
+ "language_model.model.layers.8.mlp.down_proj": {
145
+ "group_size": 64,
146
+ "bits": 6,
147
+ "mode": "affine"
148
+ },
149
+ "language_model.model.layers.9.self_attn.v_proj": {
150
+ "group_size": 64,
151
+ "bits": 6,
152
+ "mode": "affine"
153
+ },
154
+ "language_model.model.layers.9.mlp.down_proj": {
155
+ "group_size": 64,
156
+ "bits": 6,
157
+ "mode": "affine"
158
+ },
159
+ "language_model.model.layers.10.self_attn.v_proj": {
160
+ "group_size": 64,
161
+ "bits": 6,
162
+ "mode": "affine"
163
+ },
164
+ "language_model.model.layers.10.mlp.down_proj": {
165
+ "group_size": 64,
166
+ "bits": 6,
167
+ "mode": "affine"
168
+ },
169
+ "language_model.model.layers.11.self_attn.v_proj": {
170
+ "group_size": 64,
171
+ "bits": 6,
172
+ "mode": "affine"
173
+ },
174
+ "language_model.model.layers.11.mlp.down_proj": {
175
+ "group_size": 64,
176
+ "bits": 6,
177
+ "mode": "affine"
178
+ },
179
+ "language_model.model.layers.12.self_attn.v_proj": {
180
+ "group_size": 64,
181
+ "bits": 6,
182
+ "mode": "affine"
183
+ },
184
+ "language_model.model.layers.12.mlp.down_proj": {
185
+ "group_size": 64,
186
+ "bits": 6,
187
+ "mode": "affine"
188
+ },
189
+ "language_model.model.layers.13.self_attn.v_proj": {
190
+ "group_size": 64,
191
+ "bits": 6,
192
+ "mode": "affine"
193
+ },
194
+ "language_model.model.layers.13.mlp.down_proj": {
195
+ "group_size": 64,
196
+ "bits": 6,
197
+ "mode": "affine"
198
+ },
199
+ "language_model.model.layers.14.self_attn.v_proj": {
200
+ "group_size": 64,
201
+ "bits": 6,
202
+ "mode": "affine"
203
+ },
204
+ "language_model.model.layers.14.mlp.down_proj": {
205
+ "group_size": 64,
206
+ "bits": 6,
207
+ "mode": "affine"
208
+ },
209
+ "language_model.model.layers.15.self_attn.v_proj": {
210
+ "group_size": 64,
211
+ "bits": 6,
212
+ "mode": "affine"
213
+ },
214
+ "language_model.model.layers.15.mlp.down_proj": {
215
+ "group_size": 64,
216
+ "bits": 6,
217
+ "mode": "affine"
218
+ },
219
+ "language_model.model.layers.16.self_attn.v_proj": {
220
+ "group_size": 64,
221
+ "bits": 6,
222
+ "mode": "affine"
223
+ },
224
+ "language_model.model.layers.16.mlp.down_proj": {
225
+ "group_size": 64,
226
+ "bits": 6,
227
+ "mode": "affine"
228
+ },
229
+ "language_model.model.layers.17.self_attn.v_proj": {
230
+ "group_size": 64,
231
+ "bits": 6,
232
+ "mode": "affine"
233
+ },
234
+ "language_model.model.layers.17.mlp.down_proj": {
235
+ "group_size": 64,
236
+ "bits": 6,
237
+ "mode": "affine"
238
+ },
239
+ "language_model.model.layers.18.self_attn.v_proj": {
240
+ "group_size": 64,
241
+ "bits": 6,
242
+ "mode": "affine"
243
+ },
244
+ "language_model.model.layers.18.mlp.down_proj": {
245
+ "group_size": 64,
246
+ "bits": 6,
247
+ "mode": "affine"
248
+ },
249
+ "language_model.model.layers.19.self_attn.v_proj": {
250
+ "group_size": 64,
251
+ "bits": 6,
252
+ "mode": "affine"
253
+ },
254
+ "language_model.model.layers.19.mlp.down_proj": {
255
+ "group_size": 64,
256
+ "bits": 6,
257
+ "mode": "affine"
258
+ },
259
+ "language_model.model.layers.20.self_attn.v_proj": {
260
+ "group_size": 64,
261
+ "bits": 6,
262
+ "mode": "affine"
263
+ },
264
+ "language_model.model.layers.20.mlp.down_proj": {
265
+ "group_size": 64,
266
+ "bits": 6,
267
+ "mode": "affine"
268
+ },
269
+ "language_model.model.layers.21.self_attn.v_proj": {
270
+ "group_size": 64,
271
+ "bits": 6,
272
+ "mode": "affine"
273
+ },
274
+ "language_model.model.layers.21.mlp.down_proj": {
275
+ "group_size": 64,
276
+ "bits": 6,
277
+ "mode": "affine"
278
+ },
279
+ "language_model.model.layers.22.self_attn.v_proj": {
280
+ "group_size": 64,
281
+ "bits": 6,
282
+ "mode": "affine"
283
+ },
284
+ "language_model.model.layers.22.mlp.down_proj": {
285
+ "group_size": 64,
286
+ "bits": 6,
287
+ "mode": "affine"
288
+ },
289
+ "language_model.model.layers.23.self_attn.v_proj": {
290
+ "group_size": 64,
291
+ "bits": 6,
292
+ "mode": "affine"
293
+ },
294
+ "language_model.model.layers.23.mlp.down_proj": {
295
+ "group_size": 64,
296
+ "bits": 6,
297
+ "mode": "affine"
298
+ },
299
+ "language_model.model.layers.24.self_attn.v_proj": {
300
+ "group_size": 64,
301
+ "bits": 6,
302
+ "mode": "affine"
303
+ },
304
+ "language_model.model.layers.24.mlp.down_proj": {
305
+ "group_size": 64,
306
+ "bits": 6,
307
+ "mode": "affine"
308
+ },
309
+ "language_model.model.layers.25.self_attn.v_proj": {
310
+ "group_size": 64,
311
+ "bits": 6,
312
+ "mode": "affine"
313
+ },
314
+ "language_model.model.layers.25.mlp.down_proj": {
315
+ "group_size": 64,
316
+ "bits": 6,
317
+ "mode": "affine"
318
+ },
319
+ "language_model.model.layers.26.self_attn.v_proj": {
320
+ "group_size": 64,
321
+ "bits": 6,
322
+ "mode": "affine"
323
+ },
324
+ "language_model.model.layers.26.mlp.down_proj": {
325
+ "group_size": 64,
326
+ "bits": 6,
327
+ "mode": "affine"
328
+ },
329
+ "language_model.model.layers.27.self_attn.v_proj": {
330
+ "group_size": 64,
331
+ "bits": 6,
332
+ "mode": "affine"
333
+ },
334
+ "language_model.model.layers.27.mlp.down_proj": {
335
+ "group_size": 64,
336
+ "bits": 6,
337
+ "mode": "affine"
338
+ },
339
+ "language_model.model.layers.28.self_attn.v_proj": {
340
+ "group_size": 64,
341
+ "bits": 6,
342
+ "mode": "affine"
343
+ },
344
+ "language_model.model.layers.28.mlp.down_proj": {
345
+ "group_size": 64,
346
+ "bits": 6,
347
+ "mode": "affine"
348
+ },
349
+ "language_model.model.layers.29.self_attn.v_proj": {
350
+ "group_size": 64,
351
+ "bits": 6,
352
+ "mode": "affine"
353
+ },
354
+ "language_model.model.layers.29.mlp.down_proj": {
355
+ "group_size": 64,
356
+ "bits": 6,
357
+ "mode": "affine"
358
+ },
359
+ "language_model.model.layers.30.self_attn.v_proj": {
360
+ "group_size": 64,
361
+ "bits": 6,
362
+ "mode": "affine"
363
+ },
364
+ "language_model.model.layers.30.mlp.down_proj": {
365
+ "group_size": 64,
366
+ "bits": 6,
367
+ "mode": "affine"
368
+ },
369
+ "language_model.model.layers.31.self_attn.v_proj": {
370
+ "group_size": 64,
371
+ "bits": 6,
372
+ "mode": "affine"
373
+ },
374
+ "language_model.model.layers.31.mlp.down_proj": {
375
+ "group_size": 64,
376
+ "bits": 6,
377
+ "mode": "affine"
378
+ },
379
+ "language_model.model.layers.32.self_attn.v_proj": {
380
+ "group_size": 64,
381
+ "bits": 6,
382
+ "mode": "affine"
383
+ },
384
+ "language_model.model.layers.32.mlp.down_proj": {
385
+ "group_size": 64,
386
+ "bits": 6,
387
+ "mode": "affine"
388
+ },
389
+ "language_model.model.layers.33.self_attn.v_proj": {
390
+ "group_size": 64,
391
+ "bits": 6,
392
+ "mode": "affine"
393
+ },
394
+ "language_model.model.layers.33.mlp.down_proj": {
395
+ "group_size": 64,
396
+ "bits": 6,
397
+ "mode": "affine"
398
+ },
399
+ "language_model.model.layers.34.self_attn.v_proj": {
400
+ "group_size": 64,
401
+ "bits": 6,
402
+ "mode": "affine"
403
+ },
404
+ "language_model.model.layers.34.mlp.down_proj": {
405
+ "group_size": 64,
406
+ "bits": 6,
407
+ "mode": "affine"
408
+ },
409
+ "language_model.model.layers.35.self_attn.v_proj": {
410
+ "group_size": 64,
411
+ "bits": 6,
412
+ "mode": "affine"
413
+ },
414
+ "language_model.model.layers.35.mlp.down_proj": {
415
+ "group_size": 64,
416
+ "bits": 6,
417
+ "mode": "affine"
418
+ },
419
+ "language_model.model.layers.36.self_attn.v_proj": {
420
+ "group_size": 64,
421
+ "bits": 6,
422
+ "mode": "affine"
423
+ },
424
+ "language_model.model.layers.36.mlp.down_proj": {
425
+ "group_size": 64,
426
+ "bits": 6,
427
+ "mode": "affine"
428
+ },
429
+ "language_model.model.layers.37.self_attn.v_proj": {
430
+ "group_size": 64,
431
+ "bits": 6,
432
+ "mode": "affine"
433
+ },
434
+ "language_model.model.layers.37.mlp.down_proj": {
435
+ "group_size": 64,
436
+ "bits": 6,
437
+ "mode": "affine"
438
+ },
439
+ "language_model.model.layers.38.self_attn.v_proj": {
440
+ "group_size": 64,
441
+ "bits": 6,
442
+ "mode": "affine"
443
+ },
444
+ "language_model.model.layers.38.mlp.down_proj": {
445
+ "group_size": 64,
446
+ "bits": 6,
447
+ "mode": "affine"
448
+ },
449
+ "language_model.model.layers.39.self_attn.v_proj": {
450
+ "group_size": 64,
451
+ "bits": 6,
452
+ "mode": "affine"
453
+ },
454
+ "language_model.model.layers.39.mlp.down_proj": {
455
+ "group_size": 64,
456
+ "bits": 6,
457
+ "mode": "affine"
458
+ }
459
  },
460
  "quantization_config": {
461
  "group_size": 64,
462
  "bits": 5,
463
+ "mode": "affine",
464
+ "language_model.model.layers.0.self_attn.v_proj": {
465
+ "group_size": 64,
466
+ "bits": 6,
467
+ "mode": "affine"
468
+ },
469
+ "language_model.model.layers.0.mlp.down_proj": {
470
+ "group_size": 64,
471
+ "bits": 6,
472
+ "mode": "affine"
473
+ },
474
+ "language_model.model.layers.1.self_attn.v_proj": {
475
+ "group_size": 64,
476
+ "bits": 6,
477
+ "mode": "affine"
478
+ },
479
+ "language_model.model.layers.1.mlp.down_proj": {
480
+ "group_size": 64,
481
+ "bits": 6,
482
+ "mode": "affine"
483
+ },
484
+ "language_model.model.layers.2.self_attn.v_proj": {
485
+ "group_size": 64,
486
+ "bits": 6,
487
+ "mode": "affine"
488
+ },
489
+ "language_model.model.layers.2.mlp.down_proj": {
490
+ "group_size": 64,
491
+ "bits": 6,
492
+ "mode": "affine"
493
+ },
494
+ "language_model.model.layers.3.self_attn.v_proj": {
495
+ "group_size": 64,
496
+ "bits": 6,
497
+ "mode": "affine"
498
+ },
499
+ "language_model.model.layers.3.mlp.down_proj": {
500
+ "group_size": 64,
501
+ "bits": 6,
502
+ "mode": "affine"
503
+ },
504
+ "language_model.model.layers.4.self_attn.v_proj": {
505
+ "group_size": 64,
506
+ "bits": 6,
507
+ "mode": "affine"
508
+ },
509
+ "language_model.model.layers.4.mlp.down_proj": {
510
+ "group_size": 64,
511
+ "bits": 6,
512
+ "mode": "affine"
513
+ },
514
+ "language_model.model.layers.5.self_attn.v_proj": {
515
+ "group_size": 64,
516
+ "bits": 6,
517
+ "mode": "affine"
518
+ },
519
+ "language_model.model.layers.5.mlp.down_proj": {
520
+ "group_size": 64,
521
+ "bits": 6,
522
+ "mode": "affine"
523
+ },
524
+ "language_model.model.layers.6.self_attn.v_proj": {
525
+ "group_size": 64,
526
+ "bits": 6,
527
+ "mode": "affine"
528
+ },
529
+ "language_model.model.layers.6.mlp.down_proj": {
530
+ "group_size": 64,
531
+ "bits": 6,
532
+ "mode": "affine"
533
+ },
534
+ "language_model.model.layers.7.self_attn.v_proj": {
535
+ "group_size": 64,
536
+ "bits": 6,
537
+ "mode": "affine"
538
+ },
539
+ "language_model.model.layers.7.mlp.down_proj": {
540
+ "group_size": 64,
541
+ "bits": 6,
542
+ "mode": "affine"
543
+ },
544
+ "language_model.model.layers.8.self_attn.v_proj": {
545
+ "group_size": 64,
546
+ "bits": 6,
547
+ "mode": "affine"
548
+ },
549
+ "language_model.model.layers.8.mlp.down_proj": {
550
+ "group_size": 64,
551
+ "bits": 6,
552
+ "mode": "affine"
553
+ },
554
+ "language_model.model.layers.9.self_attn.v_proj": {
555
+ "group_size": 64,
556
+ "bits": 6,
557
+ "mode": "affine"
558
+ },
559
+ "language_model.model.layers.9.mlp.down_proj": {
560
+ "group_size": 64,
561
+ "bits": 6,
562
+ "mode": "affine"
563
+ },
564
+ "language_model.model.layers.10.self_attn.v_proj": {
565
+ "group_size": 64,
566
+ "bits": 6,
567
+ "mode": "affine"
568
+ },
569
+ "language_model.model.layers.10.mlp.down_proj": {
570
+ "group_size": 64,
571
+ "bits": 6,
572
+ "mode": "affine"
573
+ },
574
+ "language_model.model.layers.11.self_attn.v_proj": {
575
+ "group_size": 64,
576
+ "bits": 6,
577
+ "mode": "affine"
578
+ },
579
+ "language_model.model.layers.11.mlp.down_proj": {
580
+ "group_size": 64,
581
+ "bits": 6,
582
+ "mode": "affine"
583
+ },
584
+ "language_model.model.layers.12.self_attn.v_proj": {
585
+ "group_size": 64,
586
+ "bits": 6,
587
+ "mode": "affine"
588
+ },
589
+ "language_model.model.layers.12.mlp.down_proj": {
590
+ "group_size": 64,
591
+ "bits": 6,
592
+ "mode": "affine"
593
+ },
594
+ "language_model.model.layers.13.self_attn.v_proj": {
595
+ "group_size": 64,
596
+ "bits": 6,
597
+ "mode": "affine"
598
+ },
599
+ "language_model.model.layers.13.mlp.down_proj": {
600
+ "group_size": 64,
601
+ "bits": 6,
602
+ "mode": "affine"
603
+ },
604
+ "language_model.model.layers.14.self_attn.v_proj": {
605
+ "group_size": 64,
606
+ "bits": 6,
607
+ "mode": "affine"
608
+ },
609
+ "language_model.model.layers.14.mlp.down_proj": {
610
+ "group_size": 64,
611
+ "bits": 6,
612
+ "mode": "affine"
613
+ },
614
+ "language_model.model.layers.15.self_attn.v_proj": {
615
+ "group_size": 64,
616
+ "bits": 6,
617
+ "mode": "affine"
618
+ },
619
+ "language_model.model.layers.15.mlp.down_proj": {
620
+ "group_size": 64,
621
+ "bits": 6,
622
+ "mode": "affine"
623
+ },
624
+ "language_model.model.layers.16.self_attn.v_proj": {
625
+ "group_size": 64,
626
+ "bits": 6,
627
+ "mode": "affine"
628
+ },
629
+ "language_model.model.layers.16.mlp.down_proj": {
630
+ "group_size": 64,
631
+ "bits": 6,
632
+ "mode": "affine"
633
+ },
634
+ "language_model.model.layers.17.self_attn.v_proj": {
635
+ "group_size": 64,
636
+ "bits": 6,
637
+ "mode": "affine"
638
+ },
639
+ "language_model.model.layers.17.mlp.down_proj": {
640
+ "group_size": 64,
641
+ "bits": 6,
642
+ "mode": "affine"
643
+ },
644
+ "language_model.model.layers.18.self_attn.v_proj": {
645
+ "group_size": 64,
646
+ "bits": 6,
647
+ "mode": "affine"
648
+ },
649
+ "language_model.model.layers.18.mlp.down_proj": {
650
+ "group_size": 64,
651
+ "bits": 6,
652
+ "mode": "affine"
653
+ },
654
+ "language_model.model.layers.19.self_attn.v_proj": {
655
+ "group_size": 64,
656
+ "bits": 6,
657
+ "mode": "affine"
658
+ },
659
+ "language_model.model.layers.19.mlp.down_proj": {
660
+ "group_size": 64,
661
+ "bits": 6,
662
+ "mode": "affine"
663
+ },
664
+ "language_model.model.layers.20.self_attn.v_proj": {
665
+ "group_size": 64,
666
+ "bits": 6,
667
+ "mode": "affine"
668
+ },
669
+ "language_model.model.layers.20.mlp.down_proj": {
670
+ "group_size": 64,
671
+ "bits": 6,
672
+ "mode": "affine"
673
+ },
674
+ "language_model.model.layers.21.self_attn.v_proj": {
675
+ "group_size": 64,
676
+ "bits": 6,
677
+ "mode": "affine"
678
+ },
679
+ "language_model.model.layers.21.mlp.down_proj": {
680
+ "group_size": 64,
681
+ "bits": 6,
682
+ "mode": "affine"
683
+ },
684
+ "language_model.model.layers.22.self_attn.v_proj": {
685
+ "group_size": 64,
686
+ "bits": 6,
687
+ "mode": "affine"
688
+ },
689
+ "language_model.model.layers.22.mlp.down_proj": {
690
+ "group_size": 64,
691
+ "bits": 6,
692
+ "mode": "affine"
693
+ },
694
+ "language_model.model.layers.23.self_attn.v_proj": {
695
+ "group_size": 64,
696
+ "bits": 6,
697
+ "mode": "affine"
698
+ },
699
+ "language_model.model.layers.23.mlp.down_proj": {
700
+ "group_size": 64,
701
+ "bits": 6,
702
+ "mode": "affine"
703
+ },
704
+ "language_model.model.layers.24.self_attn.v_proj": {
705
+ "group_size": 64,
706
+ "bits": 6,
707
+ "mode": "affine"
708
+ },
709
+ "language_model.model.layers.24.mlp.down_proj": {
710
+ "group_size": 64,
711
+ "bits": 6,
712
+ "mode": "affine"
713
+ },
714
+ "language_model.model.layers.25.self_attn.v_proj": {
715
+ "group_size": 64,
716
+ "bits": 6,
717
+ "mode": "affine"
718
+ },
719
+ "language_model.model.layers.25.mlp.down_proj": {
720
+ "group_size": 64,
721
+ "bits": 6,
722
+ "mode": "affine"
723
+ },
724
+ "language_model.model.layers.26.self_attn.v_proj": {
725
+ "group_size": 64,
726
+ "bits": 6,
727
+ "mode": "affine"
728
+ },
729
+ "language_model.model.layers.26.mlp.down_proj": {
730
+ "group_size": 64,
731
+ "bits": 6,
732
+ "mode": "affine"
733
+ },
734
+ "language_model.model.layers.27.self_attn.v_proj": {
735
+ "group_size": 64,
736
+ "bits": 6,
737
+ "mode": "affine"
738
+ },
739
+ "language_model.model.layers.27.mlp.down_proj": {
740
+ "group_size": 64,
741
+ "bits": 6,
742
+ "mode": "affine"
743
+ },
744
+ "language_model.model.layers.28.self_attn.v_proj": {
745
+ "group_size": 64,
746
+ "bits": 6,
747
+ "mode": "affine"
748
+ },
749
+ "language_model.model.layers.28.mlp.down_proj": {
750
+ "group_size": 64,
751
+ "bits": 6,
752
+ "mode": "affine"
753
+ },
754
+ "language_model.model.layers.29.self_attn.v_proj": {
755
+ "group_size": 64,
756
+ "bits": 6,
757
+ "mode": "affine"
758
+ },
759
+ "language_model.model.layers.29.mlp.down_proj": {
760
+ "group_size": 64,
761
+ "bits": 6,
762
+ "mode": "affine"
763
+ },
764
+ "language_model.model.layers.30.self_attn.v_proj": {
765
+ "group_size": 64,
766
+ "bits": 6,
767
+ "mode": "affine"
768
+ },
769
+ "language_model.model.layers.30.mlp.down_proj": {
770
+ "group_size": 64,
771
+ "bits": 6,
772
+ "mode": "affine"
773
+ },
774
+ "language_model.model.layers.31.self_attn.v_proj": {
775
+ "group_size": 64,
776
+ "bits": 6,
777
+ "mode": "affine"
778
+ },
779
+ "language_model.model.layers.31.mlp.down_proj": {
780
+ "group_size": 64,
781
+ "bits": 6,
782
+ "mode": "affine"
783
+ },
784
+ "language_model.model.layers.32.self_attn.v_proj": {
785
+ "group_size": 64,
786
+ "bits": 6,
787
+ "mode": "affine"
788
+ },
789
+ "language_model.model.layers.32.mlp.down_proj": {
790
+ "group_size": 64,
791
+ "bits": 6,
792
+ "mode": "affine"
793
+ },
794
+ "language_model.model.layers.33.self_attn.v_proj": {
795
+ "group_size": 64,
796
+ "bits": 6,
797
+ "mode": "affine"
798
+ },
799
+ "language_model.model.layers.33.mlp.down_proj": {
800
+ "group_size": 64,
801
+ "bits": 6,
802
+ "mode": "affine"
803
+ },
804
+ "language_model.model.layers.34.self_attn.v_proj": {
805
+ "group_size": 64,
806
+ "bits": 6,
807
+ "mode": "affine"
808
+ },
809
+ "language_model.model.layers.34.mlp.down_proj": {
810
+ "group_size": 64,
811
+ "bits": 6,
812
+ "mode": "affine"
813
+ },
814
+ "language_model.model.layers.35.self_attn.v_proj": {
815
+ "group_size": 64,
816
+ "bits": 6,
817
+ "mode": "affine"
818
+ },
819
+ "language_model.model.layers.35.mlp.down_proj": {
820
+ "group_size": 64,
821
+ "bits": 6,
822
+ "mode": "affine"
823
+ },
824
+ "language_model.model.layers.36.self_attn.v_proj": {
825
+ "group_size": 64,
826
+ "bits": 6,
827
+ "mode": "affine"
828
+ },
829
+ "language_model.model.layers.36.mlp.down_proj": {
830
+ "group_size": 64,
831
+ "bits": 6,
832
+ "mode": "affine"
833
+ },
834
+ "language_model.model.layers.37.self_attn.v_proj": {
835
+ "group_size": 64,
836
+ "bits": 6,
837
+ "mode": "affine"
838
+ },
839
+ "language_model.model.layers.37.mlp.down_proj": {
840
+ "group_size": 64,
841
+ "bits": 6,
842
+ "mode": "affine"
843
+ },
844
+ "language_model.model.layers.38.self_attn.v_proj": {
845
+ "group_size": 64,
846
+ "bits": 6,
847
+ "mode": "affine"
848
+ },
849
+ "language_model.model.layers.38.mlp.down_proj": {
850
+ "group_size": 64,
851
+ "bits": 6,
852
+ "mode": "affine"
853
+ },
854
+ "language_model.model.layers.39.self_attn.v_proj": {
855
+ "group_size": 64,
856
+ "bits": 6,
857
+ "mode": "affine"
858
+ },
859
+ "language_model.model.layers.39.mlp.down_proj": {
860
+ "group_size": 64,
861
+ "bits": 6,
862
+ "mode": "affine"
863
+ }
864
  },
865
  "text_config": {
866
  "_name_or_path": "ibm-granite/granite-4.0-1b-base",
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:55ef6da65f5f97aafd708b17f3d4bfcd3a7d2f2b4c87c7d158d32ff567b20b37
3
- size 2215392490
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f9800db1b68ebf112d29b337a9d923892ff0e7a36b8b56f3718be057d04008d4
3
+ size 2802075924
model.safetensors.index.json CHANGED
@@ -1,6 +1,6 @@
1
  {
2
  "metadata": {
3
- "total_size": 2215208376,
4
  "total_parameters": 2313204060
5
  },
6
  "weight_map": {
@@ -522,11 +522,7 @@
522
  "encoder.out.weight": "model.safetensors",
523
  "encoder.out_mid.bias": "model.safetensors",
524
  "encoder.out_mid.weight": "model.safetensors",
525
- "language_model.lm_head.biases": "model.safetensors",
526
- "language_model.lm_head.scales": "model.safetensors",
527
  "language_model.lm_head.weight": "model.safetensors",
528
- "language_model.model.embed_tokens.biases": "model.safetensors",
529
- "language_model.model.embed_tokens.scales": "model.safetensors",
530
  "language_model.model.embed_tokens.weight": "model.safetensors",
531
  "language_model.model.layers.0.input_layernorm.weight": "model.safetensors",
532
  "language_model.model.layers.0.mlp.down_proj.biases": "model.safetensors",
 
1
  {
2
  "metadata": {
3
+ "total_size": 2801892024,
4
  "total_parameters": 2313204060
5
  },
6
  "weight_map": {
 
522
  "encoder.out.weight": "model.safetensors",
523
  "encoder.out_mid.bias": "model.safetensors",
524
  "encoder.out_mid.weight": "model.safetensors",
 
 
525
  "language_model.lm_head.weight": "model.safetensors",
 
 
526
  "language_model.model.embed_tokens.weight": "model.safetensors",
527
  "language_model.model.layers.0.input_layernorm.weight": "model.safetensors",
528
  "language_model.model.layers.0.mlp.down_proj.biases": "model.safetensors",