konic-labs/LFM2.5-VL-3B-ffn8192-int4-MLX-optimized

LFM2.5-VL-3B-ffn8192-int4 — MLX Optimized

5.1–5.6× faster decode on Apple Silicon (M3, 16 GB) with quality preserved.

An inference-optimized MLX port of konic-labs/LFM2.5-VL-3B-ffn8192-int4: the compressed-tensors INT4/INT8 weights are mapped losslessly into MLX QuantizedLinear layers (no dequantization, no re-quantization), fused at load time, and driven by a lean custom decode loop.

⚠️ Fused checkpoint layout. The weights in this repo use fused names (e.g. w1+w3, q+k+v) and require the loader in the companion code repo — not the stock mlx_vlm.utils.load.

Companion code

github.com/konic-labs/lfm25-mlx-optimized — packer/loader (quant_map.py), lean decode loop (fastgen.py), REPL (chat_lfm2.py), OpenAI-compatible server (serve_lfm2.py), benchmark and quality gates.

Benchmarks

On Apple M3 (16 GB unified memory), greedy decode, 128 tokens, prompt "Write a detailed paragraph about the history of the Pyrenees mountains." (machine-load dependent; full run log in the code repo's .auto/log.jsonl).

Metric	Source checkpoint (this repo's base)	This port	Δ
Decode throughput	12.16 tok/s	62–68 tok/s	+410–460%
Text TTFT	167 ms	62 ms	−63%
Image TTFT	~1.25 s	~0.88 s	−30%
Multi-turn TTFT (KV reuse)	~188 ms	~64 ms	−66%
Prefill	~200 tok/s	~600 tok/s	+200%
Checkpoint size	2.79 GB	1.63 GB	−42%
Peak RSS	~4.8 GB	~1.9 GB	−60%

Optimizations

Component	Bits	Notes
FFN w1/w3/w2	INT4	checkpoint-native; 4-bit is the quality floor — 3-bit deterministically breaks multi-turn tool calling
Conv in/out_proj	4-bit	−185 MB decode bandwidth
Tied embeddings / lm_head	4-bit	the fp16 128k×2048 matmul was the dominant decode cost
LM attention (q/k/v/o)	4-bit	−33 MB decode bandwidth
Vision fc2	INT8 padded	4304→4352 zero-extended packed words, exact checkpoint values
Vision projector	8-bit
patch_embedding	fp16 (kept)	8-bit there destroys vision
Scales / biases	fp16	checkpoint scales are natively fp16

Plus load-time linear fusion (w1+w3, q+k+v; bit-identical math, −24% RSS), fp16 quantized_matmul outputs, and a lean decode loop: no greedy logsumexp, argpartition+categorical sampling, bool-sync EOS checks, last-position-only lm_head in prefill, and prefix-verified prompt-cache KV reuse.

Usage

Requires Python ≥ 3.12, mlx, mlx-vlm, and the loader from the companion code repo.

git clone git@github.com:konic-labs/lfm25-mlx-optimized.git
cd lfm25-mlx-optimized
.venv/bin/python chat_lfm2.py --model konic-labs/LFM2.5-VL-3B-ffn8192-int4-MLX-optimized

Or serve it as an OpenAI-compatible endpoint:

.venv/bin/python serve_lfm2.py --port 8080 --model konic-labs/LFM2.5-VL-3B-ffn8192-int4-MLX-optimized

Files

LFM2.5-VL-3B-ffn8192-int4-MLX-optimized/
  config.json
  model.safetensors
  chat_template.jinja
  tokenizer.json
  tokenizer_config.json
  generation_config.json
  processor_config.json
  README.md

Quality gates

The port passes automated quality gates: greedy text (Paris), greedy single-turn tool calling, image description (red square on white), and a 16-trial multi-turn tool-calling gate (discriminates the 3-bit cliff: ~40% call rate vs ~0% at 3-bit).

Research

The full experiment trail (15 kept runs, 12.16 → ~62–68 tok/s) is in the code repo under .auto/ (playbook, run log, ideas, dead ends).

License

Apache-2.0 (model weights as per the base model).