RECIPE.md

# Conversion recipe

This document is the reproducibility record for the published asset. The conversion used
Apple's public `coreai-models` project, the public community-maintained
`john-rocky/coreai-model-zoo`, and the 2.6B-specific wrapper and gates shipped here as
`lfm2.5-2.6b-coreai-conversion-e0e375b.tar.gz`.

The toolkit contains only the wrapper, targeted configuration fixes, gates, and locked Python
environment from this conversion workspace. It does not vendor either upstream repository.

## Prerequisites

| Component | Pin | Public? |
| --- | --- | --- |
| `LiquidAI/LFM2.5-2.6B` | `dca1825886789bd40b94368f53b1d9ada4c94598` | yes |
| [`apple/coreai-models`](https://github.com/apple/coreai-models) (converter base) | `b1cb71b8522d99408059fa0b98b8742171bcb0b8` | yes, official Apple |
| [`apple/coreai-models`](https://github.com/apple/coreai-models) (runtime base) | `5ed9981303b38d5a44aa6b45509bc4f6945029f5` | yes, official Apple |
| [`john-rocky/coreai-model-zoo`](https://github.com/john-rocky/coreai-model-zoo) — LFM2 exporter/overlay and runtime patches | `95a29d41affed3bdf3ea5992ece094a908e21a04` | yes, community-maintained |
| 2.6B conversion toolkit in this repository | `e0e375b`; SHA-256 `665285044be7c0171e1b737167b9892f2799bc2e01e3c5441212887903456f2d` | yes |
| `coreai-core` / `coreai-torch` / `coreai-opt` | `1.0.0b2` / `0.4.1` / `0.2.1` | yes |

Toolchain used: macOS 27.0 (build `26A5388g`), Xcode 27.0 (`27A5228h`), Python 3.11.15,
torch 2.9.0, coremltools 9.0.

The public community zoo is required in both directions:

- **Conversion** needs the zoo's overlay, because that overlay is what carries the LFM2
  authoring module (`models/macos/lfm2.py`)—Apple's converter base alone does not know this
  architecture. The shipped wrapper additionally fixes the nested LFM2.5 RoPE configuration,
  local-checkpoint loading, tokenizer staging, and the measured attention-precision choice.
- **Inference** needs exactly two zoo patches:
  `apps/coreai-shared-product.patch` followed by
  `apps/coreai-pipelined-extra-states.patch`. The extra-state patch carries LFM's fixed-shape
  convolution state beside the growing KV pair. This model does not require the zoo's
  per-token-input, static-input, or prefix-cache patches.

### Recreate the converter workspace

```bash
tar -xzf lfm2.5-2.6b-coreai-conversion-e0e375b.tar.gz
cd lfm2.5-2.6b-coreai-conversion-e0e375b

git clone https://github.com/apple/coreai-models.git /path/to/coreai-models
git -C /path/to/coreai-models checkout b1cb71b8522d99408059fa0b98b8742171bcb0b8

git clone https://github.com/john-rocky/coreai-model-zoo.git /path/to/coreai-model-zoo
git -C /path/to/coreai-model-zoo checkout 95a29d41affed3bdf3ea5992ece094a908e21a04

COREAI_MODELS_REPO=/path/to/coreai-models \
COREAI_ZOO_REPO=/path/to/coreai-model-zoo \
  bash scripts/setup-vendor.sh

uv run python scripts/export_bundle.py --mode int8hu -- --head-sym --tag _attnfp16
uv run python scripts/authored_parity.py
uv run python scripts/quant_reference.py --mode int8hu -- --head-sym --tag _attnfp16
uv run python scripts/gate_bundle.py --bundle <exported-bundle>
```

The source checkpoint itself is not included in the toolkit; obtain
`LiquidAI/LFM2.5-2.6B` at the pinned revision under its upstream license.

## Quantization

Applied to the authored module before export, then exported to the Core AI dialect.

| Tensor group | Precision | Detail |
| --- | --- | --- |
| Linear / MLP weights | **int8** | blockwise, block size 32, per-block scales |
| `lm_head` | **int8** | blockwise 32, **symmetric**; the head is untied and is ~0.5 GB |
| Attention `q,k,v,out` projections | **fp16** | overlay default is fp32; overridden |
| Token embedding | **fp16** | left unquantized |
| Norms, RoPE tables, indices | fp16 / int32 | untouched |

The resulting compiled storage budget, which is the check that a rebuild matched:

```
Int8     2,621,243,392
Float16    428,342,276
Float32             34
Int32              312
UInt32              71
UInt64               1
```

Graph shape: `input_ids [1,1]` static, `position_ids` dynamic, KV cache dynamic on the
sequence axis, `max_context_length = 4096`. Decode-only; no chunked prefill entrypoint.

Two deviations from the zoo recipe's defaults were **measured** rather than inherited:

1. **Attention projections fp16 instead of fp32.** The overlay promotes these to fp32 for
   GPU-delegate exactness. On this model that precision is not needed, and fp32 costs
   ~168 MB of reads on every decode step.
2. **Attention projections were *not* taken to int8.** That is a further ~2.5 % throughput
   for one lost position in 125; the higher-fidelity option was shipped instead.

One correctness fix was required on the overlay, and it matters more than either:

> The checkpoint carries no top-level `rope_theta`. It ships
> `rope_parameters.rope_theta = 1e7` (the transformers ≥ 5 layout). Code that reads only the
> legacy key silently falls back to `1e6` — a 10× wrong RoPE that still produces fluent short
> completions and only clearly breaks at long context. Both the overlay and transformers 4.x
> hit this. Any reproduction must read the nested key.

A second, latent one: the checkpoint spells tying `tie_word_embeddings`, not `tie_embedding`.
The default is correct here, so nothing breaks on this model, but it would flip silently on an
untied checkpoint.

## Gates

Four separate questions, deliberately not collapsed into one number.

1. **Authoring fidelity** — the re-authored module vs Hugging Face `Lfm2ForCausalLM`, both
   fp32, teacher-forced. Result 21/21 top-1, cosine 1.000000. This is the gate that caught
   the RoPE bug.
2. **Quantization damage** — the quantized module vs an *independent* fp32 Hugging Face
   reference (transformers ≥ 5.2), teacher-forced over 5 sequences / 125 positions. Result
   122/125 top-1, minimum per-position cosine 0.997050.
3. **Conversion fidelity** — the exported bundle vs **its own quantized weights run eagerly**,
   greedy, 5 prompts. Result 5/5 exact. Comparing the bundle to fp32 here would conflate
   quantization damage with conversion bugs, so it is compared to the thing it is supposed to
   equal.
4. **Throughput** — measured *before* any gate loads the model, because loading first cost
   ~10 % on an identical bundle.

Quality is teacher-forced throughout. Free-running text is not usable as a gate on this
model: every probe prompt contains at least one step with a sub-0.05 top-2 margin, so
transcripts diverge on near-ties without indicating damage.

## Measurement protocol

Comparisons below ~5 % are meaningless without this. Early runs showed ~3 % spread on a
*byte-identical* bundle.

- Clear the Core AI specialization cache entry **for this asset only**, for the producing
  binary. The asset's own `main.hash` is the content key.
- One throwaway load + short generation to absorb cold specialization.
- 60 s settle so the SoC sheds export and compile heat.
- 5 trials, prompt 64 tokens, generate 128, fixed seed.
- Report **between-run** spread across independent runs. Within-run standard deviation of
  adjacent trials is repeatability, not a population statistic, and quoting it as though it
  bounded the mean overstates confidence badly.

Two environment notes that changed results materially:

- `COREAI_CHUNK_THRESHOLD=1`.
- Ahead-of-time compilation must name one architecture. Compiling without that builds all 20
  (~8 GB each). `--expect-frequent-reshapes` measured 84 tok/s against 160 and 8.3 GB against
  3.3 GB, so it is off.

## Rejected

| Attempt | Outcome |
| --- | --- |
| int4 blockwise 32 | minimum cosine 0.51–0.66 — a different model |
| int4 blockwise 32, conv projections rescued to int8 | cosine 0.662, 16/21 top-1; rescuing conv does not protect the MLP bulk, which is where both the bytes and the damage are |
| int4 blockwise 16 | quality recovers, 42 tok/s — ~3× *slower* than int8, dequantization dominates |
| int8 token embedding | throughput-neutral, −214 MB; not shipped because it is not a win |
| `--preferred-compute neural-engine` | no-op; the compiled asset holds an `MPSGraph` delegate either way. A dynamic KV dimension is not an ANE-shaped graph |
| Speculative decoding, static-S verify graph | exports and gates its contract, but per-position logits do not match stepped decode; not published |

Reproduction is verified by the **gates and the storage budget above, not by hashing**. The
exporter names each externalized call site with a generated UUID — 391 such names in this
graph — so two exports of identical weights differ in a few bytes and therefore in SHA-256.