divydeep/granite-speech-4.1-2b-mlx-4bit

Granite Speech 4.1 2B MLX 4-bit

Memory-oriented MLX conversion of ibm-granite/granite-speech-4.1-2b for Apple Silicon. This is the autoregressive Granite Speech model, not the NAR variant.

Quantization

This conversion uses post-training weight quantization without training, calibration data, or an importance matrix.

Component	Precision
16-layer Conformer speech encoder	BF16
2-layer Q-Former speech projector	BF16
Eligible internal language-model linear layers	MLX affine 4-bit, group size 64
Token embedding and language-model output head	BF16
Norms, biases, and unsupported tensors	BF16

• Source revision: de575db64086f84fdc79da4932d1076e965bc546
• Effective average reported by MLX: 8.911 bits per weight
• model.safetensors: approximately 2.46 GB

Usage

pip install -U mlx-audio

python -m mlx_audio.stt.generate \
  --model /path/to/granite-speech-4.1-2b-mlx-4bit \
  --audio audio.wav \
  --output-path transcript \
  --format txt \
  --prompt "transcribe the speech with proper punctuation and capitalization."

Validation

The checkpoint was strictly loaded by mlx-audio and run with greedy decoding on IBM's bundled multilingual_sample.wav. It preserved every reference word and accent, but omitted three French punctuation or hyphen marks compared with the BF16 reference. This is a smoke test, not a complete WER benchmark.

Quantization can affect names, rare words, punctuation, casing, translation, keyword biasing, and difficult or noisy audio. Use the 8-bit variant when closer agreement with the original model is more important than memory usage.

License

Apache-2.0, matching the original model.