Granite Speech 4.1 2B MLX 5-bit

Quality-oriented mixed-precision MLX conversion of ibm-granite/granite-speech-4.1-2b for Apple Silicon. This is the autoregressive Granite Speech model, not the separate NAR variant.

The model runs with mlx-audio and supports multilingual automatic speech recognition (ASR), speech translation (AST), punctuation and capitalization, and keyword-biased transcription.

Quantization

This conversion uses post-training weight quantization only. No training, fine-tuning, calibration dataset, or importance matrix was used.

Component	Precision
16-layer Conformer speech encoder	BF16
2-layer Q-Former speech projector	BF16
Granite language model, ordinary eligible internal linear layers	MLX affine 5-bit, group size 64
All `v_proj` and `down_proj` layers	MLX affine 6-bit, group size 64
Token embedding	BF16
Language-model output head	BF16
Norms, biases, and unsupported tensors	BF16

Value and down projections receive extra precision because mixed-tensor quantizers commonly protect these sensitive paths. The acoustic stack, token embedding, and output head remain BF16. This is conceptually similar to medium mixed-tensor quantizations such as GGUF Q5_K_M, but the file format and numerical representation are MLX affine quantization rather than GGUF K-quants.

Source revision: de575db64086f84fdc79da4932d1076e965bc546
Effective average reported by MLX: 9.691 bits per weight
model.safetensors: approximately 2.5 GB

Requirements

Apple Silicon Mac
macOS 14 or later
mlx-audio >= 0.4.5

pip install -U mlx-audio

Usage

python -m mlx_audio.stt.generate \
  --model divydeep/granite-speech-4.1-2b-mlx-5bit \
  --audio audio.wav \
  --output-path transcript \
  --format txt \
  --prompt "transcribe the speech with proper punctuation and capitalization."

Python:

from mlx_audio.stt.utils import load_model

model = load_model("divydeep/granite-speech-4.1-2b-mlx-5bit")
result = model.generate(
    "audio.wav",
    prompt="transcribe the speech with proper punctuation and capitalization.",
    temperature=0.0,
)
print(result.text)

Greedy decoding with temperature=0.0 is recommended for transcription.

Prompts

Task	Prompt
Raw ASR	`can you transcribe the speech into a written format?`
Punctuated ASR	`transcribe the speech with proper punctuation and capitalization.`
Keyword-biased ASR	`transcribe the speech to text. Keywords: <kw1>, <kw2>, ...`
Speech translation	`translate the speech to <language>.`
Punctuated translation	`translate the speech to <language> with proper punctuation and capitalization.`

For non-English punctuation, translation, and keyword biasing, use the English prompt forms recommended by IBM.

Validation

The conversion was validated by:

Strictly loading all weights with mlx-audio.
Confirming the architecture is granite_speech, not granite_speech_nar.
Running greedy transcription on IBM's bundled multilingual_sample.wav.
Checking preservation of English and French text, punctuation, accents, and capitalization in the generated transcript.

This is a conversion smoke test, not a full WER benchmark. The upstream IBM evaluation results describe the original model and should not be interpreted as measured results for this quantized conversion.

Limitations

Quantization may affect rare words, names, punctuation, casing, keyword biasing, translation, and difficult or noisy audio.
This model has not been independently evaluated on the complete IBM benchmark suite.
The model is autoregressive and does not provide the throughput behavior of the separate NAR architecture.
Refer to the upstream model card for intended use, language coverage, safety considerations, training data, and architectural details.

References

License

Apache-2.0, matching the original model.

Citation

@misc{granite-speech-4.1-2b,
  title={Granite 4.1 Speech},
  author={IBM Granite Speech Team},
  year={2026},
  url={https://huggingface.co/ibm-granite/granite-speech-4.1-2b}
}