LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Phi-4-mini-reasoning — LiteRT-LM (blockwise int4)

microsoft/Phi-4-mini-reasoning converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

Phi-4-mini-reasoning is a dense 3.8B math/reasoning model from Microsoft (implemented as Phi3ForCausalLM, 32 layers) — it solves problems with a <think>…</think> chain-of-thought, then the answer.

File	model.litertlm — int4 block 32 (~2.6 GB)
Quantization	int4 weights (symmetric) + OCTAV optimal-clipping; embeddings INT8 (externalized section)
Compute	integer
Context (KV cache)	4096
Base model	microsoft/Phi-4-mini-reasoning

⚠️ It's a reasoning model — give it room to think

This model emits a <think>…</think> chain-of-thought, then a \boxed{} answer. Run it with max_tokens ≥ 2048 — at a short limit it gets cut off before the answer. (All quality numbers below were measured at 2048.)

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Device	Backend	Prefill (256)	Decode	TTFT	Load	Peak footprint
Apple M4 Max (macOS)	CPU	113 tok/s	19.8 tok/s	2.62 s	—	—
Apple M4 Max (macOS)	GPU (Metal)	1168 tok/s	82.8 tok/s	0.24 s	—	—

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The model loads and generates correct answers on an iPhone 17 Pro, but no on-device timing was taken, so none is quoted. Android phone rows (Galaxy S26, Pixel 8a) are below.

Accuracy note

Measured on GSM8K (n=100, greedy, 0-shot chain-of-thought, max_tokens 2048, identical prompt and answer-extraction for every row).

Configuration	GSM8K
bf16 (reference)	89.0%
LiteRT int4 — block 32	81.0% (−8 pt)

int4 (block 32) is at parity (−8 pt). Why block 32 (not block 128)? This is a precision-sensitive math model: the coarser block-128 int4 dropped to 74% (−15 pt) and degenerated on some prompts, while block 32 holds at 81%. So only the block-32 build is published.

Galaxy S26 — GPU and CPU

The published bundle runs on the Android GPU backend and generates.

file	GPU backend	delegation	peak
model.litertlm	runs	3227 / 3227 ops across 2 subgraphs on LiteRT GPU	1334 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

The op counts above are the LiteRT GPU partitions. XNNPACK additionally takes 1 of the 4 nodes in decode_embedder and 1 of the 4 nodes in prefill_embedder_128; the runtime accepts that split.

Speed (measured 2026-09-05)

Protocol: litert_lm_advanced_main built from the litert-lm v0.16.0 tree (2026-08-17 build) with --benchmark --sampler_backend=cpu, the published model.litertlm, a real 499-token prompt read from a file (--input_prompt_file; 511 tokens after the chat template), --max_output_tokens=256, one run at a time with 45 s idle between runs, every listed run shown and the median reported. Before every run the phone sat idle until dumpsys thermalservice reported thermal status 1 or lower, and the status was logged again after each run (every run ended at status 2). The model hit the 256-token cap on every run, so decode is measured over 256 tokens. Init time is excluded; the first GPU run per phone also compiled the GPU program cache.

backend	prefill tok/s (511 tokens)	decode tok/s (256 tokens)	time to first token	runs (prefill ; decode)
CPU (XNNPACK)	71.3	8.40	7.28 s	71.35 / 68.62 / 74.13 ; 8.40 / 6.66 / 8.48
GPU (LiteRT GPU / OpenCL)	277.6	13.10	1.92 s	274.17 / 277.56 / 280.19 ; 10.88 / 13.39 / 13.10

On this handset the GPU backend is faster at prefill and faster at decode than the CPU backend for this bundle (13.1 vs 8.4 tok/s decode).

GPU wiring, including the Gallery import toggle: GPU guide.

Pixel 8a — CPU and GPU

Same bundle, same binary and protocol as the Galaxy S26 rows above, on a Google Pixel 8a (Tensor G3, 8 GB, Android 16). Measured 2026-09-05.

backend	prefill tok/s (511 tokens)	decode tok/s (256 tokens)	time to first token	runs (prefill ; decode)
CPU (XNNPACK)	19.4	3.77	29.70 s	13.12 / 25.66 ; 3.51 / 4.03
GPU (LiteRT GPU / OpenCL)	76.6	6.83	6.81 s	75.89 / 77.33 ; 6.75 / 6.91

On this handset the GPU backend is faster at prefill and faster at decode than the CPU backend for this bundle (6.8 vs 3.8 tok/s decode). One further Pixel 8a CPU run ended without a benchmark report (the adb session dropped mid-run) and is not listed.

Usage

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "A bat and a ball cost \$1.10. The bat costs \$1.00 more than the ball. How much is the ball?"

The .litertlm bundle carries the tokenizer and prompt template (Phi format — <|user|>…<|end|><|assistant|>), so no separate tokenizer files are needed.

Run on Android

Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) — no computer or adb needed. The manual steps below are only required on older builds or for sideloading a local file.

The official Google AI Edge Gallery app runs .litertlm models on-device:

1. Install a recent Gallery (package com.google.ai.edge.gallery, 1.0.15+ supports .litertlm).
2. Download model.litertlm and push it: adb push model.litertlm /sdcard/Download/
3. In the app tap +, pick the file, choose the GPU backend, and raise the max-tokens setting (≥2048).

Run on desktop (LiteRT-LM CLI)

The same .litertlm bundle runs on macOS / Linux / Windows with the official LiteRT-LM CLI — including as a local OpenAI-compatible API server:

pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/Phi-4-mini-reasoning model.litertlm phi-4-mini-reasoning
litert-lm run phi-4-mini-reasoning     # interactive chat in the terminal
litert-lm serve           # local OpenAI-compatible API server

Run on iPhone

Verified on iPhone 17 Pro (LiteRT-LM Swift runtime): loads and generates correct answers. This is a ~2.6 GB bundle (Phi's 200K-token vocab makes a large externalized embedder), so it sits near the iOS memory ceiling — if you hit "embedding lookup model is not initialized" (a low-memory symptom), reboot the phone to free RAM and reload.

Conversion

Converted with the official litert-torch converter. Phi-4-mini uses the Phi3ForCausalLM arch with LongRoPE + a (nominal) sliding window; two export-time adjustments are needed for current litert-torch:

1. LongRoPE: replace Phi3RotaryEmbedding.forward with a static version (the @dynamic_rope_update seq-len branch is data-dependent under torch.export; for cache ≤ original_max=4096 the short factor is always correct).
2. Sliding window: set config.sliding_window=None (it is 262144 ≫ context, i.e. full-causal) so the standard causal mask path is used.

Recipe: blockwise-32 int4 + OCTAV, embeddings INT8, KV cache 4096, externalize_embedder=True.

2026-08-28 — start_token fix (weights unchanged)

The LiteRT-LM engine prepends the metadata start_token to every prompt, but this model's reference prompt has no leading BOS at all — the bundle was feeding an extra <|endoftext|> the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

2026-08-29 — default system prompt restored (weights unchanged)

The upstream chat template emits a default system turn whenever the caller sends no system message — for this model: Your name is Phi, an AI math expert developed by Microsoft.. The converter's template probe renders the template with a system message already present, so that block was never seen and never reached the bundle: with no system message the model was running without the default system turn it was tuned with. The chat template in model.litertlm now emits the block exactly once when no system message is given. In model.litertlm, the block is not emitted when you pass a system message; the upstream template also wraps a caller's system message in its own preamble, and this file passes it through unchanged, exactly as it did before. The restored block adds 15 prefill tokens to a conversation that sends no system message, so time-to-first-token grows by that much; per-token speed is unchanged.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file per token — only the file's own sha256 differs. What changed is the input: with no system message, the prompt now renders byte-identical to the upstream chat template's output, verified on the LiteRT-LM runtime. A system message you pass yourself renders as before. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-29, re-download.

2026-08-31 — thought channel declared (metadata only, weights unchanged)

model.litertlm now declares the reasoning channel in its metadata (LlmMetadata.channels: channel name thought, markers <think>…</think> exactly as this model emits them). Without the declaration the runtime has no way to tell the reasoning apart from the answer: the raw thinking streamed inline into the visible text, and a thinking_token_budget was silently ignored (the API returns OK and only logs a warning). With the channel declared, LiteRT-LM returns the reasoning separated in channels["thought"] and the thinking budget takes effect.

Metadata-only change: every section of the bundle except LlmMetadata is byte-identical to the previous file (verified per section, tokenizer included), so the weights, the graph, the tokenizer and the chat template are unchanged and the speed and accuracy numbers on this card still describe this file — only the file's own sha256 differs. Verified on the LiteRT-LM runtime (litert-lm-api 0.16.1): the visible answer stays clean, the reasoning lands in channels["thought"], and on one bundle of this batch thinking_token_budget=16 was confirmed to truncate the reasoning at exactly 16 tokens where it was a no-op before. If you downloaded before 2026-08-31, re-download to get the channel-aware file.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
model.litertlm	14.1 (13.9–14.2)	1.6 (1.6–1.6)	21.1 s	4.2 GB

License

MIT, inherited from the base model microsoft/Phi-4-mini-reasoning.