mlboydaisuke/MiniCPM-V-4.6-CoreAI

MiniCPM-V-4.6 — Core AI

On-device vision-language model for iPhone / Apple Silicon. A Core AI port of openbmb/MiniCPM-V-4.6 — the strongest sub-2B open VLM — running fully local on the GPU via the Core AI pipelined engine: pick a photo, ask about it, stream the answer.

Verified on iPhone 17 Pro: image → grounded answer at ~51.5 tok/s decode, all local.

Fridge photo → recipe ideas, fully on-device on an iPhone 17 Pro (CoreAIChat).

Architecture

MiniCPM-V-4.6 (1.3B) = a SigLIP So400m vision tower (980px / patch 14 / 27 layers, with a window-attention insert-merger @ layer 6 + a downsample-MLP merger → ÷16 = 64 visual tokens per 448px slice) + a Qwen3.5-hybrid text backbone (qwen3_5_text: 0.8B, 24 layers, GatedDeltaNet linear attention ×3 : full attention ×1, head_dim 256, vocab 248094, tied head). Connector = 2×2 spatial merges + MLP, spliced into the text embeddings at <image> positions (masked_scatter).

Bundles

path	what	dtype	size
gpu-pipelined/minicpmv46_vlm_decode_int8lin/	VLM text decoder (input_ids → logits + a static image_embeds[64,1024] buffer; in-graph gather ids ≥ V ? image_embeds[ids-V] : embed[ids])	int8 (per-block-32 linear)	~1.0 GB
gpu-pipelined/minicpmv46_vision/	fixed-grid SigLIP vision encoder (pixel_values[1,3,448,448] → image_features[64,1024])	fp16	~1.0 GB

The decoder is a complete qwen3.5-hybrid text LLM when image_embeds is zero — same bundle, no image needed.

How a VLM rides the text-only engine

The pipelined engine knows nothing about images. The whole multimodal state rides the static-input hook (image_embeds buffer) + an id-space trick — the graph stays ids + positions → logits:

• The host runs the vision encoder once per image (resize 448, normalize x/127.5−1) and writes image_embeds [64,1024] into one owned MTLBuffer the engine binds on every step.
• The prompt's <|image_pad|> ids are rewritten to extension ids V + slot (slot 0..63). In-graph: embed = ids < V ? table[ids] : image_embeds[ids − V].
• Positions are plain 1D (no M-RoPE / no rope-shift), the qwen3.5-hybrid KV + conv + recurrent states are the engine's; nothing else changes.

Simpler than the Qwen3-VL port (no deepstack, no M-RoPE).

Measured (iPhone 17 Pro, iOS 27 beta, release)

• iPhone 17 Pro decode ~51.5 tok/s (text core 53.4) · M4 Max ~224 tok/s (text core, llm-benchmark), engine cold-spec ~3–5 s, ~1.5 GB resident (jetsam-safe).
• Numerics: fp32-torch parity bit-exact (vision cos 1.000000, full overlay logits cos 1.00004); Core AI engine ≡ python ≡ HF (text 24/24; image path reproduces the HF description modulo one int8 near-tie token, then reconverges).
• Real-photo example (kakigōri): "a bowl of shaved ice ... chunks of mango ... a dark blue saucer ... a menu or a book, hinting at a café ... a wooden table" — accurate, fully on-device.

Use it

apps/CoreAIChat and the standalone MiniCPMVLM app have a MiniCPM-V 4.6 mode with a photo picker: pick an image, ask, stream. The vision tower runs once per image (~hundreds of ms); each turn re-prefills (S=1).

Conversion + gates: see coreai-model-zoo / minicpm-v-4.6.

License: Apache-2.0 (inherited from openbmb/MiniCPM-V-4.6).