randmaru/Mellum2.1-12B-A2.5B-Thinking-mlx-mxfp4
This is an MXFP4 MLX quantization of JetBrains/Mellum2.1-12B-A2.5B-Thinking for Apple Silicon inference.
The base model is a 12B mixture-of-experts decoder (2.5B active parameters, 64 experts / top-8, 28 layers, 128K context). Weights are quantized to 4-bit MXFP4 with group size 32; the router gates are kept at 8-bit (group size 64). The result is a single model.safetensors of 6.46 GB, averaging ≈4.25 bits per weight.
MXFP4 vs 4Bit Quantization Comparison
| Parameter | MXFP4 |
4Bit (GGUF Q4_K_M) |
|---|---|---|
| Reference | this repo (MLX) | JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF |
| Quantization format | 4‑bit floating point with microscaling, group 32, shared exponent E8M0 | 4‑bit k‑quant (mixed 4/6‑bit blocks) |
| Tensor types | U32 (packed weights), U8 (scales), BF16 | GGUF k‑quant blocks with FP scales |
| Weights file size | ≈6.46 GB (6,457,111,871 bytes) | ≈8.07 GB (8,071,295,264 bytes) |
| Total download size | ≈6.46 GB (6,464,259,055 bytes) | ≈8.07 GB (8,071,295,264 bytes) |
| Effective bits per weight | 4.25 | ≈5.3 |
| Router gates | kept at 8‑bit (group 64) | folded into the k‑quant |
| Runtime | mlx-lm, etc. |
llama.cpp, etc. |
| Hardware support | Apple Silicon GPU via MLX (Metal) | Metal, CUDA, ROCm, CPU |
| Quality | good for its bit‑rate (4.25 bpw) | higher bit‑rate (5.3 bpw) tends to preserve accuracy better on outliers |
Key takeaways:
- Parameter size: The MXFP4
safetensorsis ≈1.6 GB smaller than the 4‑bit GGUFQ4_K_M(≈6.46 GB vs ≈8.07 GB). Most of the difference is bit‑rate, not packing magic: MXFP4 averages 4.25 bits/weight whileQ4_K_Mkeeps several tensors at higher precision (≈5.3 bits/weight overall). Treat it as a size/quality trade‑off. - Total download: MXFP4 ships as a few small files (config, tokenizer, one
model.safetensors) totalling ≈6.46 GB, versus the single ≈8.07 GB GGUF. - Apple Silicon: MXFP4 runs natively on the Apple GPU through MLX (Metal); the GGUF build runs through llama.cpp's Metal backend.
- Quality: the higher bit‑rate of
Q4_K_Mgenerally retains a little more accuracy; MXFP4 trades some precision for a smaller footprint. Evaluate on your own task. - GGUF alternative: the same base model is also available as
MXFP4_MOE(7.03 GB, ≈4.6 bits/weight) in the GGUF repo.
Actual speed depends on the backend, GPU, batch size, and quantization implementation.
- Downloads last month
- 36
Model size
12B params
Tensor type
U32
·
BF16 ·
Hardware compatibility
Log In to add your hardware
4-bit
Model tree for randmaru/Mellum2.1-12B-A2.5B-Thinking-mlx-mxfp4
Base model
JetBrains/Mellum2-12B-A2.5B-Base Finetuned
JetBrains/Mellum2.1-12B-A2.5B-Thinking
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("randmaru/Mellum2.1-12B-A2.5B-Thinking-mlx-mxfp4") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True)