---
license: apache-2.0
base_model: openbmb/MiniCPM-V-4.6
tags:
- core-ai
- coreai
- vision-language
- vlm
- on-device
- iphone
- apple
pipeline_tag: image-text-to-text
language:
- en
- zh
---

# MiniCPM-V-4.6 — Core AI

**On-device vision-language model for iPhone / Apple Silicon.** A Core AI port of
[`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) — the strongest
sub-2B open VLM — running fully local on the GPU via the Core AI **pipelined engine**:
pick a photo, ask about it, stream the answer.

Verified on **iPhone 17 Pro**: image → grounded answer at **~51.5 tok/s** decode, all local.

<p align="center">
  <img src="https://github.com/user-attachments/assets/c4baa524-5217-4bb3-a23f-b0acd6249bd4" width="300" alt="MiniCPM-V 4.6 on iPhone — a fridge photo becomes recipe ideas, fully on-device in CoreAIChat">
</p>
<p align="center"><em>Fridge photo → recipe ideas, fully on-device on an iPhone 17 Pro (CoreAIChat).</em></p>

## Architecture

MiniCPM-V-4.6 (1.3B) = a **SigLIP So400m vision tower** (980px / patch 14 / 27 layers, with a
window-attention insert-merger @ layer 6 + a downsample-MLP merger → ÷16 = 64 visual tokens per
448px slice) + a **Qwen3.5-hybrid text backbone** (`qwen3_5_text`: 0.8B, 24 layers, GatedDeltaNet
linear attention ×3 : full attention ×1, head_dim 256, vocab 248094, tied head). Connector =
2×2 spatial merges + MLP, spliced into the text embeddings at `<image>` positions (`masked_scatter`).

## Bundles

| path | what | dtype | size |
|---|---|---|---|
| `gpu-pipelined/minicpmv46_vlm_decode_int8lin/` | VLM text decoder (`input_ids → logits` + a static `image_embeds[64,1024]` buffer; in-graph gather `ids ≥ V ? image_embeds[ids-V] : embed[ids]`) | int8 (per-block-32 linear) | ~1.0 GB |
| `gpu-pipelined/minicpmv46_vision/` | fixed-grid SigLIP vision encoder (`pixel_values[1,3,448,448] → image_features[64,1024]`) | fp16 | ~1.0 GB |

The decoder is a complete qwen3.5-hybrid text LLM when `image_embeds` is zero — same bundle, no image needed.

## How a VLM rides the text-only engine

The pipelined engine knows nothing about images. The whole multimodal state rides the
**static-input hook** (`image_embeds` buffer) + an id-space trick — the graph stays `ids + positions → logits`:

- The host runs the vision encoder **once per image** (resize 448, normalize `x/127.5−1`) and writes
  `image_embeds [64,1024]` into one owned MTLBuffer the engine binds on every step.
- The prompt's `<|image_pad|>` ids are rewritten to **extension ids** `V + slot` (slot 0..63).
  In-graph: `embed = ids < V ? table[ids] : image_embeds[ids − V]`.
- Positions are **plain 1D** (no M-RoPE / no rope-shift), the qwen3.5-hybrid KV + conv + recurrent
  states are the engine's; nothing else changes.

Simpler than the [Qwen3-VL port](https://huggingface.co/mlboydaisuke/Qwen3-VL-2B-CoreAI) (no deepstack, no M-RoPE).

## Measured (iPhone 17 Pro, iOS 27 beta, release)

- **iPhone 17 Pro decode ~51.5 tok/s** (text core 53.4) · **M4 Max ~224 tok/s** (text core, `llm-benchmark`),
  engine cold-spec ~3–5 s, ~1.5 GB resident (jetsam-safe).
- **Numerics**: fp32-torch parity bit-exact (vision cos 1.000000, full overlay logits cos 1.00004);
  Core AI engine ≡ python ≡ HF (text 24/24; image path reproduces the HF description modulo one int8
  near-tie token, then reconverges).
- Real-photo example (kakigōri): *"a bowl of shaved ice ... chunks of mango ... a dark blue saucer ...
  a menu or a book, hinting at a café ... a wooden table"* — accurate, fully on-device.

## Use it

`apps/CoreAIChat` and the standalone `MiniCPMVLM` app have a **MiniCPM-V 4.6 mode with a photo picker**:
pick an image, ask, stream. The vision tower runs once per image (~hundreds of ms); each turn re-prefills (S=1).

Conversion + gates: see [coreai-model-zoo / minicpm-v-4.6](https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/minicpm-v-4.6.md).

License: Apache-2.0 (inherited from `openbmb/MiniCPM-V-4.6`).