--- license: apache-2.0 tags: - executorch - xnnpack - pte - on-device - image-text-to-text base_model: - HuggingFaceTB/SmolVLM2-500M-Video-Instruct base_model_relation: quantized --- # SmolVLM2-500M — ExecuTorch `smolvlm2_500m_video_instruct_xnnpack_8da8w.pte` (518 MB) - **Source**: HuggingFaceTB/SmolVLM2-500M-Video-Instruct — a SigLIP vision tower (768 wide, 12 layers) and a Llama decoder (960 wide, 32 layers) - **License**: Apache-2.0 - **Input**: a 512×512 picture as `[1, 3, 512, 512]`, and token ids for the words around it - **Output**: logits over the 49,280-token vocabulary One file, three entry points — the shape ExecuTorch's multimodal runner asks for: | method | in | out | |---|---|---| | `vision_encoder` | `[1, 3, 512, 512]` | `[1, 64, 960]` rows in the decoder's embedding space | | `token_embeddings` | token ids | embeddings | | `text_model` | embeddings, positions | logits, and its own cache | A decoder that only takes token ids cannot be told about a picture. Splitting a vision-language model into a vision `.pte` and a text `.pte` runs aground there; `MultimodalPrefiller::load` asks one module for `token_embeddings` and `text_model` and uses `vision_encoder` if it finds it. `vision_encoder` here takes a picture rather than a patch sequence, which is what that runner hands it — so this file can be driven by the runner as well as method by method. ## Verification (Mac arm64, 2026-08-21) | check | result | |---|---| | vision half vs the untouched tower, 24 photographs | worst **corr 0.99794** | | first-step logits vs eager | **corr 0.99536**, same top-1 | | greedy tokens vs eager, 12 steps | **12/12** | The wrappers themselves are exact: run in eager without quantization they agree with the model at corr 1.000000, so what these numbers measure is the weights, not the wiring. ## Why the decoder is int8 and not int4 The vision tower is int8 per output channel and the decoder is int8 over groups of 32. int4 on a decoder this size does not hold: | | int4 decoder | int8 decoder | |---|---|---| | size | 497 MB | 518 MB | | first-step logits vs eager | corr 0.89878 | corr 0.99406 | | greedy tokens vs eager | 6/12 | 12/12 | int4 keeps the first word and then drifts. Only the int8 build is here. Those two columns were measured before the embedding table was quantised, when this file was 657 MB. ## Ask for one tile The processor cuts a picture into tiles by aspect ratio and by size, and a 512×512 photograph becomes **seventeen** of them — 1088 image tokens, against the 64 this graph produces and the 512 of context it was built with. `do_image_splitting = False` on the processor (or its image processor) gives the one tile that matches. ```bash VL_CKPT=HuggingFaceTB/SmolVLM2-500M-Video-Instruct \ VL_PTE=smolvlm2_500m_video_instruct_xnnpack_8da8w.pte \ python convert/run_vl.py "What is in this picture?" ``` The driver builds the prompt with the model's own processor and writes the picture's rows wherever the processor put an image token, which is what the model does internally. ## It invents text on signs Asked about a London street, this model answers with a shop name that is not there. That is the model and not the conversion: eager, unquantized, invents a different one from the same photograph. If reading signs matters, LFM2.5-VL reads them ([450M](https://huggingface.co/mlboydaisuke/LFM2.5-VL-450M-ExecuTorch)). ## Conversion `convert/export_vl_bundle.py`. Two things needed re-authoring: - **The position embeddings are chosen with `torch.bucketize`**, so that a picture filling part of the grid still lands on the right ones. There is no `bucketize` kernel in the runtime, and a full square grid has one answer anyway: patch *i* takes position *i*. The export checks that against the model's own code rather than assuming it. - **The cache has to live inside `text_model`.** ExecuTorch copies a mutable buffer into each method that names it (`Program::load_mutable_subsegment_into` writes into the method's own memory), so a prefill method and a decode method would each get their own and neither would see the other's writes. One graph serves both. `StaticCache` does not survive lowering, and neither does a cache held by any object outside the module tree — its tensors get lifted a second time as constants, and `run_decompositions` then returns a function where a GraphModule was expected. ## The number that decides whether it runs on a phone `CONTEXT`, the upper bound on the dynamic sequence dimension. The memory planner sizes its arena for the bound, not for what a picture costs: at 4096 that arena runs to gigabytes and iOS kills the process with signal 9 before the first method has finished loading. One picture is 64 rows here, so 512 leaves room for a long question and a long answer. (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) · iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples)) ## The token embedding table The table is an `nn.Embedding`, and the quantisation filter in `convert/export_vl_bundle.py` was written as `isinstance(module, nn.Linear)`, so it went out in fp32 — 189 MB of a 657 MB file. `tie_word_embeddings` does not save it either: `quantize_` swaps `lm_head.weight` for a new tensor and the embedding keeps pointing at the old fp32 one. (LFM2.5-VL ties and SmolVLM2 does not; both leaked.) It is now int8 over groups of 64 along each row, written out by hand because torchao's `IntxWeightOnlyConfig` reaches an `nn.Embedding` but will not lower (`Missing out variants: {'torchao::dequantize_affine'}`). On this model the first-step logits improve slightly — corr 0.99536 against 0.99406 for the fp32 table — while the twelfth greedy token differs where the fp32-table build matched all twelve. The file drops from 657 MB to 518 MB. Both numbers are here rather than the flattering one. One scale per row is not enough for a narrow table: on SmolVLM2-256M, whose rows are 576 long, per-row scales moved the first-step logits from corr 0.99197 to 0.97811 and flipped the top-1 token, so that model keeps its fp32 table. Groups of 64 are what make this one free.