README.md

---
license: other
license_name: lfm-open-license-v1.0
license_link: https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/dca1825886789bd40b94368f53b1d9ada4c94598/LICENSE
base_model: LiquidAI/LFM2.5-2.6B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: coreai
language:
- ar
- zh
- en
- fr
- de
- hi
- id
- it
- ja
- ko
- pl
- pt
- ru
- es
- th
- vi
tags:
- coreai
- aimodel
- aimodelc
- apple-silicon
- lfm2
- lfm2.5
- liquid
- edge
- int8
---

# LFM2.5-2.6B — Core AI (int8)

An int8 Core AI conversion of **[LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B)**
for Apple silicon. This repository contains no trained weights of its own: it is a quantized
format conversion of Liquid AI's model, and all model credit belongs to **Liquid AI**.

Converted from source revision `dca1825886789bd40b94368f53b1d9ada4c94598`. Both upstream
safetensors shards were SHA-256 verified against that revision before conversion.

## Read this first: running it needs a runtime you may not be able to get

**This asset will not run on a stock Core AI runtime.** It requires a patch stack that lives in
Apple's `coreai-model-zoo`, which is **not a public repository**. Specifically:

- The graph uses a **per-token input contract** (`input_ids [1,1]` plus explicit KV state) that
  an unpatched runtime at the pinned commit rejects. The patches involved are named
  `coreai-pipelined-per-token-inputs`, `-static-inputs`, `-extra-states`,
  `coreai-prefix-cache` and `coreai-shared-product`.
- Re-exporting additionally needs that repository's LFM2 **overlay**, which is what teaches the
  converter this architecture at all.

Those patches are Apple's, not mine, so they are **not redistributed here** and this repository
cannot make them available to you. Practically:

- **With access to `coreai-model-zoo`:** everything needed to build the runtime and reproduce
  the conversion is pinned in [`RECIPE.md`](RECIPE.md).
- **Without it:** you can download and inspect these weights, read every measurement, and reuse
  the recipe — but you will not be able to execute the asset today. That is a real limitation,
  stated here rather than buried.

For the same reason **no conversion toolkit is shipped**: those scripts import modules from that
non-public repo, so publishing them would either redistribute code that is not mine or hand you
something that cannot run. [`RECIPE.md`](RECIPE.md) describes the recipe precisely instead.

## Which file do I want?

Take the portable `lfm2_5_2_6b_decode_int8hu_attnfp16_block32_sym/` directory. It runs on any
Apple silicon Mac, given the runtime above. `aimodelc-h16c/` is the same model pre-compiled for
one GPU architecture — identical output and speed, about half the cold-load time, and the
runtime rejects it on a different architecture.

## What this is and is not

- It **is** a decode-optimized single-token-step graph, the shape a chat/completion loop uses.
- It **is not** a chunked-prefill or batch-serving asset.
- Quality here means **teacher-forced top-1 agreement and cosine similarity against an
  independent fp32 Hugging Face reference**, over 5 sequences / 125 positions. That is a
  regression probe, **not** a benchmark suite. No MMLU/GSM8K-style numbers are claimed.

## Artifacts

Full SHA-256 of each file as published.

| File | Bytes | SHA-256 | Recipe | Hardware scope | Use |
| --- | ---: | --- | --- | --- | --- |
| `lfm2_5_2_6b_decode_int8hu_attnfp16_block32_sym.aimodel/main.mlirb` | 3469367807 | `80540b2ee9183b756adb1ce51f334a94f13c093a554a2538aec66f9d9f3a07b3` | int8 blockwise-32, fp16 attention + embedding | any Apple silicon | **recommended** |
| `aimodelc-h16c/…h16c.aimodelc/…/resources.bin` | 3468949676 | `793907c101a01331ffa72aff7c9db5049c89a02ea03acee3174513a93c394e22` | same weights, AOT compiled | **`h16c` only** | faster cold load |
| `aimodelc-h16c/…h16c.aimodelc/…/original_model_0.mpsgraph` | 363905 | `74a24870a8853bf797a6f12f40e7fcb988de014d98166da772b24cab643d5d4f` | compiled graph | `h16c` only | part of the above |
| `tokenizer/tokenizer.json` | 17905598 | `695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41` | upstream, unmodified | — | required |

- **The two `tokenizer/tokenizer.json` copies are byte-identical** (same hash above). The one
  under `aimodelc-h16c/tokenizer/` is a **convenience copy**; you do not need both.
- Portable and compiled are **the same model**, published together because the compiled one
  halves cold load but only runs on one architecture, so neither dominates.
- Both were produced from the same pinned commit and have identical compiled storage budgets.

## Recipe, and what was rejected

Oracle for every quality number: **an independent fp32 reference, Hugging Face's own
`Lfm2ForCausalLM` at transformers ≥ 5.2**, teacher-forced over 5 sequences / 125 positions.
Cosine is the minimum per-position cosine.

| Variant | Bundle | Top-1 vs fp32 oracle | Min cosine | Conversion | Shipped |
| --- | ---: | ---: | ---: | ---: | --- |
| **int8 blockwise-32, fp16 attention + embedding** | 3.25 GB | **122/125** | **0.997050** | 5/5 | **yes** |
| + attention q/k/v/out to int8 | 3.19 GB | 121/125 | 0.996949 | 5/5 | no |
| + embedding to int8 | 3.03 GB | 123/125 | 0.996848 | 5/5 | no |
| attention at fp32 (converter default) | 3.42 GB | — | 0.997210 | 5/5 | no |
| int4 blockwise-32 family | 2.07–2.34 GB | — | 0.51–0.80 | 4–5/5 | no |

The shipped arm has the highest minimum cosine of the int8 arms. The ±1 position differences
between the three int8 rows are near-tie argmax flips at 125 positions, not a systematic
ordering — do not read the embedding row's 123 as "better". int4 is excluded on **quality**, not
size: a minimum cosine of 0.51–0.80 is a different model. **Rejected variants are documented and
deliberately not uploaded.** Full detail in [`RECIPE.md`](RECIPE.md).

## Performance, and an unresolved caveat

Decode here is **memory-bandwidth bound** — throughput tracks bytes read per token.

| Measurement | Protocol | Result |
| --- | --- | --- |
| Development runs of this recipe | cold cache + 60 s settle, 5 trials × 3 independent runs, quiet machine | **138.15 tok/s**, between-run spread 0.31 % |
| Re-measurement of *this published artifact* | same protocol, 5 trials | **107.98 tok/s**, sd 0.500 — machine demonstrably busy: load average 9–17, background disk 50–1400 MB/s |

Both are real; neither is cherry-picked; **the gap is not explained**. Established: this
artifact's storage budget is byte-identical to the one measured at 138, and its quality and
conversion gate reproduce exactly, so these are measurements of the same recipe rather than of
two different models. Not established: the cause. A bandwidth-bound workload losing throughput
to competing memory traffic is the obvious candidate, but the machine never went quiet again
during the session, so it was never isolated and another regression cannot be ruled out.

Treat 138 as what this recipe has done on an idle M4 Max and 108 as what it did under the stated
load, and **measure on your own hardware rather than trusting either.**

For reference, on the same Mac, MLX reported ~100 tok/s at 8-bit and ~60 tok/s at BF16. Those
were throughput-only observations; MLX quality was not measured, so no quality comparison against
MLX is claimed.

AOT compilation is throughput-neutral within noise and halves cold load, 9.8 s → 5.1 s.

## Reproduce

Every pin, the full quantization spec, the four gates, the measurement protocol and the rejected
variants are in **[`RECIPE.md`](RECIPE.md)**. Summary:

```
Source model : LiquidAI/LFM2.5-2.6B @ dca1825886789bd40b94368f53b1d9ada4c94598          (public)
Converter    : github.com/apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8  (public)
Runtime      : github.com/apple/coreai-models @ 5ed9981303b38d5a44aa6b45509bc4f6945029f5  (public)
coreai-torch : c89f6a44713249a12a84beec9f3e0cf2206ecc38                                   (public)
LFM2 overlay + runtime patch stack :
               apple/coreai-model-zoo @ ebef921a1f358af66c9ff67e8c6e7d4e24efad0d      (NOT public)
Toolchain    : macOS 27.0 (26A5388g), Xcode 27.0 (27A5228h), Python 3.11.15,
               torch 2.9.0, coremltools 9.0
```

**Verify a rebuild by the gates and the storage budget, not by hashing.** The exporter names each
externalized call site with a generated UUID (391 in this graph), so two exports of identical
weights differ in a few bytes and therefore in SHA-256. The budget to match is
`Int8 2,621,243,392` + `Float16 428,342,276` + `Float32 34` + small index types.

## Download

```bash
# Recommended artifact only (~3.5 GB), not the whole repository.
hf download harshav/LFM2.5-2.6B-CoreAI \
  --include 'lfm2_5_2_6b_decode_int8hu_attnfp16_block32_sym/*' 'tokenizer/*' \
  --local-dir ./LFM2.5-2.6B-CoreAI
```

With a runtime built as described above, the asset is driven as a pipelined Core AI language
model with `COREAI_CHUNK_THRESHOLD=1`, greedy decoding, using the model's own chat template at
`tokenizer/chat_template.jinja`. Operational notes:

- **macOS 27.0+** on **Apple silicon**. Validated on M4 Max.
- Runs on the **GPU** via an `MPSGraph` delegate. This is **not** an ANE asset: the KV dimension
  is dynamic, and `--preferred-compute` does not change the emitted delegate.
- Budget roughly bundle size plus KV cache, about **5 GB** at 4096 context.
- `aimodelc-h16c/` is **architecture-locked to `h16c`**; the runtime names the architecture it
  wanted when it refuses.

## Validation evidence

Machine-readable under `evidence/`: `authored_parity_vs_huggingface.json`, `fp32_reference.json`,
`recipe_quality.json`, `conversion_gate.json`, `benchmark.json`, `compiled_storage_stats.json`.

Four **separate** questions, not interchangeable:

- **Authoring fidelity** — re-authored module vs Hugging Face, fp32: 21/21 top-1, cosine
  1.000000. This caught a real bug: the checkpoint sets `rope_parameters.rope_theta = 1e7`, and
  code reading only the legacy top-level key silently defaults to `1e6` — a 10× wrong RoPE that
  still produces fluent short text. Both the converter overlay and transformers 4.x hit it.
- **Quantization damage** — vs the fp32 oracle: 122/125, min cosine 0.997050.
- **Conversion fidelity** — bundle vs **its own** quantized weights run eagerly: 5/5 exact.
  Comparing to fp32 here would conflate quantization damage with conversion bugs.
- **Throughput** — see the caveat above.

## Limitations and negative results

- **Not runnable without the non-public patch stack.** See the top of this card.
- No task-benchmark evaluation; quality is a 125-position regression probe.
- Multilingual support is inherited from upstream and was **not** re-verified per language; the
  probe is English.
- int4 rejected at block 32 (cosine 0.51–0.66) and at block 16 (quality recovers, 42 tok/s,
  ~3× slower than int8).
- `--expect-frequent-reshapes` measured 84 tok/s and 8.3 GB; not used.
- Speculative decoding not shipped: a static-S verify graph exports and its contract gates, but
  per-position logits do not match stepped decode.
- 300 tok/s was a target and was not reached by any tested configuration.

## License and attribution

The model is **Liquid AI's**. This repository redistributes a converted, quantized copy under the
upstream license, and claims **no authorship of the model**.

- Upstream: [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B) by **Liquid AI**.
- Upstream license: **LFM Open License v1.0** — pinned copy
  [here](https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/dca1825886789bd40b94368f53b1d9ada4c94598/LICENSE),
  included verbatim as [`LICENSE.upstream`](LICENSE.upstream). Your use of these weights is
  governed by it.
- Conversion tooling: Apple's `coreai-models`, `coreai-torch` and `coreai-model-zoo`, at the
  commits pinned above. **No Apple source is redistributed here.**
- This repository contributes the conversion recipe, the gates, and the measurements.