Qwen3.5-2B-oQ8-mtp

This repository contains an MLX/oMLX-compatible oQ8 quantization of Qwen/Qwen3.5-2B. It is a derivative quantized release; all credit for the original model architecture, training, and base release belongs to Qwen.

Summary

• Base model: Qwen/Qwen3.5-2B
• Quantization: oMLX/oQ level 8, affine 8-bit, group size 64
• Runtime target: oMLX / MLX
• Format: MLX safetensors
• Dtype for preserved/unquantized tensors: bfloat16
• Multimodal: yes; vision tensors preserved
• Native MTP: yes; MTP tensors preserved
• License: Apache-2.0, inherited from the base model

Files

README.md
LICENSE
chat_template.jinja
config.json
merges.txt
model.safetensors
preprocessor_config.json
tokenizer.json
tokenizer_config.json
vocab.json

Verification

Verified before public release:

• model.safetensors opens successfully
• safetensors metadata: {"format": "mlx"}
• model.safetensors size: 2,735,417,896 bytes
• tensor count: 1020
• MTP tensor count: 29
• vision tensor count: 297
• clean private Hugging Face download: pass
• clean oMLX download/load from Hugging Face: pass
• text smoke test: prompt Reply with exactly OK. returned OK
• image smoke test: red square returned red

See PUBLISH_CHECK.md for release verification details.

Use with oMLX/OpenAI-compatible API

Example request once the model is downloaded into an oMLX server:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3.5-2B-oQ8-mtp",
    "messages": [{"role": "user", "content": "Reply with exactly OK."}],
    "max_tokens": 16,
    "temperature": 0
  }'

For vision use, send OpenAI-style multimodal chat content with an image_url item.

Quantization details

{
  "model_path": "/Users/dylanrapanan/.omlx/models/Qwen/Qwen3.5-2B",
  "oq_level": 8,
  "group_size": 64,
  "sensitivity_model_path": "",
  "text_only": false,
  "dtype": "bfloat16",
  "preserve_mtp": true,
  "auto_proxy_sensitivity": true
}

Notes

• This is not the original base model; it is a quantized derivative for local MLX/oMLX use.
• Native MTP is enabled by preserving the model's MTP heads. In oMLX, keep mtp_enabled on; do not confuse this with external VLM drafter settings.
• The model was published private-first, then clean-downloaded, tensor-verified, clean-loaded in oMLX, smoke-tested, and only then made public.

Attribution

All credit for the original model goes to Qwen/Alibaba Cloud. This repository only provides a quantized MLX/oMLX artifact for convenience.