RECIPE.md

Conversion recipe

This document is the reproducibility record for the published asset. The conversion used Apple's public coreai-models project, the public community-maintained john-rocky/coreai-model-zoo, and the 2.6B-specific wrapper and gates shipped here as lfm2.5-2.6b-coreai-conversion-e0e375b.tar.gz.

The toolkit contains only the wrapper, targeted configuration fixes, gates, and locked Python environment from this conversion workspace. It does not vendor either upstream repository.

Prerequisites

Component	Pin	Public?
LiquidAI/LFM2.5-2.6B	dca1825886789bd40b94368f53b1d9ada4c94598	yes
apple/coreai-models (converter base)	b1cb71b8522d99408059fa0b98b8742171bcb0b8	yes, official Apple
apple/coreai-models (runtime base)	5ed9981303b38d5a44aa6b45509bc4f6945029f5	yes, official Apple
john-rocky/coreai-model-zoo — LFM2 exporter/overlay and runtime patches	95a29d41affed3bdf3ea5992ece094a908e21a04	yes, community-maintained
2.6B conversion toolkit in this repository	e0e375b; SHA-256 665285044be7c0171e1b737167b9892f2799bc2e01e3c5441212887903456f2d	yes
coreai-core / coreai-torch / coreai-opt	1.0.0b2 / 0.4.1 / 0.2.1	yes

Toolchain used: macOS 27.0 (build 26A5388g), Xcode 27.0 (27A5228h), Python 3.11.15, torch 2.9.0, coremltools 9.0.

The public community zoo is required in both directions:

• Conversion needs the zoo's overlay, because that overlay is what carries the LFM2 authoring module (models/macos/lfm2.py)—Apple's converter base alone does not know this architecture. The shipped wrapper additionally fixes the nested LFM2.5 RoPE configuration, local-checkpoint loading, tokenizer staging, and the measured attention-precision choice.
• Inference needs exactly two zoo patches: apps/coreai-shared-product.patch followed by apps/coreai-pipelined-extra-states.patch. The extra-state patch carries LFM's fixed-shape convolution state beside the growing KV pair. This model does not require the zoo's per-token-input, static-input, or prefix-cache patches.

Recreate the converter workspace

tar -xzf lfm2.5-2.6b-coreai-conversion-e0e375b.tar.gz
cd lfm2.5-2.6b-coreai-conversion-e0e375b

git clone https://github.com/apple/coreai-models.git /path/to/coreai-models
git -C /path/to/coreai-models checkout b1cb71b8522d99408059fa0b98b8742171bcb0b8

git clone https://github.com/john-rocky/coreai-model-zoo.git /path/to/coreai-model-zoo
git -C /path/to/coreai-model-zoo checkout 95a29d41affed3bdf3ea5992ece094a908e21a04

COREAI_MODELS_REPO=/path/to/coreai-models \
COREAI_ZOO_REPO=/path/to/coreai-model-zoo \
  bash scripts/setup-vendor.sh

uv run python scripts/export_bundle.py --mode int8hu -- --head-sym --tag _attnfp16
uv run python scripts/authored_parity.py
uv run python scripts/quant_reference.py --mode int8hu -- --head-sym --tag _attnfp16
uv run python scripts/gate_bundle.py --bundle <exported-bundle>

The source checkpoint itself is not included in the toolkit; obtain LiquidAI/LFM2.5-2.6B at the pinned revision under its upstream license.

Quantization

Applied to the authored module before export, then exported to the Core AI dialect.

Tensor group	Precision	Detail
Linear / MLP weights	int8	blockwise, block size 32, per-block scales
lm_head	int8	blockwise 32, symmetric; the head is untied and is ~0.5 GB
Attention q,k,v,out projections	fp16	overlay default is fp32; overridden
Token embedding	fp16	left unquantized
Norms, RoPE tables, indices	fp16 / int32	untouched

The resulting compiled storage budget, which is the check that a rebuild matched:

Int8     2,621,243,392
Float16    428,342,276
Float32             34
Int32              312
UInt32              71
UInt64               1

Graph shape: input_ids [1,1] static, position_ids dynamic, KV cache dynamic on the sequence axis, max_context_length = 4096. Decode-only; no chunked prefill entrypoint.

Two deviations from the zoo recipe's defaults were measured rather than inherited:

1. Attention projections fp16 instead of fp32. The overlay promotes these to fp32 for GPU-delegate exactness. On this model that precision is not needed, and fp32 costs ~168 MB of reads on every decode step.
2. Attention projections were not taken to int8. That is a further ~2.5 % throughput for one lost position in 125; the higher-fidelity option was shipped instead.

One correctness fix was required on the overlay, and it matters more than either:

The checkpoint carries no top-level rope_theta. It ships rope_parameters.rope_theta = 1e7 (the transformers ≥ 5 layout). Code that reads only the legacy key silently falls back to 1e6 — a 10× wrong RoPE that still produces fluent short completions and only clearly breaks at long context. Both the overlay and transformers 4.x hit this. Any reproduction must read the nested key.

A second, latent one: the checkpoint spells tying tie_word_embeddings, not tie_embedding. The default is correct here, so nothing breaks on this model, but it would flip silently on an untied checkpoint.

Gates

Four separate questions, deliberately not collapsed into one number.

1. Authoring fidelity — the re-authored module vs Hugging Face Lfm2ForCausalLM, both fp32, teacher-forced. Result 21/21 top-1, cosine 1.000000. This is the gate that caught the RoPE bug.
2. Quantization damage — the quantized module vs an independent fp32 Hugging Face reference (transformers ≥ 5.2), teacher-forced over 5 sequences / 125 positions. Result 122/125 top-1, minimum per-position cosine 0.997050.
3. Conversion fidelity — the exported bundle vs its own quantized weights run eagerly, greedy, 5 prompts. Result 5/5 exact. Comparing the bundle to fp32 here would conflate quantization damage with conversion bugs, so it is compared to the thing it is supposed to equal.
4. Throughput — measured before any gate loads the model, because loading first cost ~10 % on an identical bundle.

Quality is teacher-forced throughout. Free-running text is not usable as a gate on this model: every probe prompt contains at least one step with a sub-0.05 top-2 margin, so transcripts diverge on near-ties without indicating damage.

Measurement protocol

Comparisons below ~5 % are meaningless without this. Early runs showed ~3 % spread on a byte-identical bundle.

• Clear the Core AI specialization cache entry for this asset only, for the producing binary. The asset's own main.hash is the content key.
• One throwaway load + short generation to absorb cold specialization.
• 60 s settle so the SoC sheds export and compile heat.
• 5 trials, prompt 64 tokens, generate 128, fixed seed.
• Report between-run spread across independent runs. Within-run standard deviation of adjacent trials is repeatability, not a population statistic, and quoting it as though it bounded the mean overstates confidence badly.

Two environment notes that changed results materially:

• COREAI_CHUNK_THRESHOLD=1.
• Ahead-of-time compilation must name one architecture. Compiling without that builds all 20 (~8 GB each). --expect-frequent-reshapes measured 84 tok/s against 160 and 8.3 GB against 3.3 GB, so it is off.

Rejected

Attempt	Outcome
int4 blockwise 32	minimum cosine 0.51–0.66 — a different model
int4 blockwise 32, conv projections rescued to int8	cosine 0.662, 16/21 top-1; rescuing conv does not protect the MLP bulk, which is where both the bytes and the damage are
int4 blockwise 16	quality recovers, 42 tok/s — ~3× slower than int8, dequantization dominates
int8 token embedding	throughput-neutral, −214 MB; not shipped because it is not a win
--preferred-compute neural-engine	no-op; the compiled asset holds an MPSGraph delegate either way. A dynamic KV dimension is not an ANE-shaped graph
Speculative decoding, static-S verify graph	exports and gates its contract, but per-position logits do not match stepped decode; not published

Reproduction is verified by the gates and the storage budget above, not by hashing. The exporter names each externalized call site with a generated UUID — 391 such names in this graph — so two exports of identical weights differ in a few bytes and therefore in SHA-256.