--- license: other license_name: lfm-open-license-v1.0 license_link: https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/dca1825886789bd40b94368f53b1d9ada4c94598/LICENSE base_model: LiquidAI/LFM2.5-2.6B base_model_relation: quantized pipeline_tag: text-generation library_name: coreai language: - ar - zh - en - fr - de - hi - id - it - ja - ko - pl - pt - ru - es - th - vi tags: - coreai - aimodel - aimodelc - apple-silicon - lfm2 - lfm2.5 - liquid - edge - int8 --- # LFM2.5-2.6B — Core AI (int8) An int8 Core AI conversion of **[LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B)** for Apple silicon. This repository contains no trained weights of its own: it is a quantized format conversion of Liquid AI's model, and all model credit belongs to **Liquid AI**. Converted from source revision `dca1825886789bd40b94368f53b1d9ada4c94598`. Both upstream safetensors shards were SHA-256 verified against that revision before conversion. ## Read this first: running it needs a runtime you may not be able to get **This asset will not run on a stock Core AI runtime.** It requires a patch stack that lives in Apple's `coreai-model-zoo`, which is **not a public repository**. Specifically: - The graph uses a **per-token input contract** (`input_ids [1,1]` plus explicit KV state) that an unpatched runtime at the pinned commit rejects. The patches involved are named `coreai-pipelined-per-token-inputs`, `-static-inputs`, `-extra-states`, `coreai-prefix-cache` and `coreai-shared-product`. - Re-exporting additionally needs that repository's LFM2 **overlay**, which is what teaches the converter this architecture at all. Those patches are Apple's, not mine, so they are **not redistributed here** and this repository cannot make them available to you. Practically: - **With access to `coreai-model-zoo`:** everything needed to build the runtime and reproduce the conversion is pinned in [`RECIPE.md`](RECIPE.md). - **Without it:** you can download and inspect these weights, read every measurement, and reuse the recipe — but you will not be able to execute the asset today. That is a real limitation, stated here rather than buried. For the same reason **no conversion toolkit is shipped**: those scripts import modules from that non-public repo, so publishing them would either redistribute code that is not mine or hand you something that cannot run. [`RECIPE.md`](RECIPE.md) describes the recipe precisely instead. ## Which file do I want? Take the portable `lfm2_5_2_6b_decode_int8hu_attnfp16_block32_sym/` directory. It runs on any Apple silicon Mac, given the runtime above. `aimodelc-h16c/` is the same model pre-compiled for one GPU architecture — identical output and speed, about half the cold-load time, and the runtime rejects it on a different architecture. ## What this is and is not - It **is** a decode-optimized single-token-step graph, the shape a chat/completion loop uses. - It **is not** a chunked-prefill or batch-serving asset. - Quality here means **teacher-forced top-1 agreement and cosine similarity against an independent fp32 Hugging Face reference**, over 5 sequences / 125 positions. That is a regression probe, **not** a benchmark suite. No MMLU/GSM8K-style numbers are claimed. ## Artifacts Full SHA-256 of each file as published. | File | Bytes | SHA-256 | Recipe | Hardware scope | Use | | --- | ---: | --- | --- | --- | --- | | `lfm2_5_2_6b_decode_int8hu_attnfp16_block32_sym.aimodel/main.mlirb` | 3469367807 | `80540b2ee9183b756adb1ce51f334a94f13c093a554a2538aec66f9d9f3a07b3` | int8 blockwise-32, fp16 attention + embedding | any Apple silicon | **recommended** | | `aimodelc-h16c/…h16c.aimodelc/…/resources.bin` | 3468949676 | `793907c101a01331ffa72aff7c9db5049c89a02ea03acee3174513a93c394e22` | same weights, AOT compiled | **`h16c` only** | faster cold load | | `aimodelc-h16c/…h16c.aimodelc/…/original_model_0.mpsgraph` | 363905 | `74a24870a8853bf797a6f12f40e7fcb988de014d98166da772b24cab643d5d4f` | compiled graph | `h16c` only | part of the above | | `tokenizer/tokenizer.json` | 17905598 | `695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41` | upstream, unmodified | — | required | - **The two `tokenizer/tokenizer.json` copies are byte-identical** (same hash above). The one under `aimodelc-h16c/tokenizer/` is a **convenience copy**; you do not need both. - Portable and compiled are **the same model**, published together because the compiled one halves cold load but only runs on one architecture, so neither dominates. - Both were produced from the same pinned commit and have identical compiled storage budgets. ## Recipe, and what was rejected Oracle for every quality number: **an independent fp32 reference, Hugging Face's own `Lfm2ForCausalLM` at transformers ≥ 5.2**, teacher-forced over 5 sequences / 125 positions. Cosine is the minimum per-position cosine. | Variant | Bundle | Top-1 vs fp32 oracle | Min cosine | Conversion | Shipped | | --- | ---: | ---: | ---: | ---: | --- | | **int8 blockwise-32, fp16 attention + embedding** | 3.25 GB | **122/125** | **0.997050** | 5/5 | **yes** | | + attention q/k/v/out to int8 | 3.19 GB | 121/125 | 0.996949 | 5/5 | no | | + embedding to int8 | 3.03 GB | 123/125 | 0.996848 | 5/5 | no | | attention at fp32 (converter default) | 3.42 GB | — | 0.997210 | 5/5 | no | | int4 blockwise-32 family | 2.07–2.34 GB | — | 0.51–0.80 | 4–5/5 | no | The shipped arm has the highest minimum cosine of the int8 arms. The ±1 position differences between the three int8 rows are near-tie argmax flips at 125 positions, not a systematic ordering — do not read the embedding row's 123 as "better". int4 is excluded on **quality**, not size: a minimum cosine of 0.51–0.80 is a different model. **Rejected variants are documented and deliberately not uploaded.** Full detail in [`RECIPE.md`](RECIPE.md). ## Performance, and an unresolved caveat Decode here is **memory-bandwidth bound** — throughput tracks bytes read per token. | Measurement | Protocol | Result | | --- | --- | --- | | Development runs of this recipe | cold cache + 60 s settle, 5 trials × 3 independent runs, quiet machine | **138.15 tok/s**, between-run spread 0.31 % | | Re-measurement of *this published artifact* | same protocol, 5 trials | **107.98 tok/s**, sd 0.500 — machine demonstrably busy: load average 9–17, background disk 50–1400 MB/s | Both are real; neither is cherry-picked; **the gap is not explained**. Established: this artifact's storage budget is byte-identical to the one measured at 138, and its quality and conversion gate reproduce exactly, so these are measurements of the same recipe rather than of two different models. Not established: the cause. A bandwidth-bound workload losing throughput to competing memory traffic is the obvious candidate, but the machine never went quiet again during the session, so it was never isolated and another regression cannot be ruled out. Treat 138 as what this recipe has done on an idle M4 Max and 108 as what it did under the stated load, and **measure on your own hardware rather than trusting either.** For reference, on the same Mac, MLX reported ~100 tok/s at 8-bit and ~60 tok/s at BF16. Those were throughput-only observations; MLX quality was not measured, so no quality comparison against MLX is claimed. AOT compilation is throughput-neutral within noise and halves cold load, 9.8 s → 5.1 s. ## Reproduce Every pin, the full quantization spec, the four gates, the measurement protocol and the rejected variants are in **[`RECIPE.md`](RECIPE.md)**. Summary: ``` Source model : LiquidAI/LFM2.5-2.6B @ dca1825886789bd40b94368f53b1d9ada4c94598 (public) Converter : github.com/apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8 (public) Runtime : github.com/apple/coreai-models @ 5ed9981303b38d5a44aa6b45509bc4f6945029f5 (public) coreai-torch : c89f6a44713249a12a84beec9f3e0cf2206ecc38 (public) LFM2 overlay + runtime patch stack : apple/coreai-model-zoo @ ebef921a1f358af66c9ff67e8c6e7d4e24efad0d (NOT public) Toolchain : macOS 27.0 (26A5388g), Xcode 27.0 (27A5228h), Python 3.11.15, torch 2.9.0, coremltools 9.0 ``` **Verify a rebuild by the gates and the storage budget, not by hashing.** The exporter names each externalized call site with a generated UUID (391 in this graph), so two exports of identical weights differ in a few bytes and therefore in SHA-256. The budget to match is `Int8 2,621,243,392` + `Float16 428,342,276` + `Float32 34` + small index types. ## Download ```bash # Recommended artifact only (~3.5 GB), not the whole repository. hf download harshav/LFM2.5-2.6B-CoreAI \ --include 'lfm2_5_2_6b_decode_int8hu_attnfp16_block32_sym/*' 'tokenizer/*' \ --local-dir ./LFM2.5-2.6B-CoreAI ``` With a runtime built as described above, the asset is driven as a pipelined Core AI language model with `COREAI_CHUNK_THRESHOLD=1`, greedy decoding, using the model's own chat template at `tokenizer/chat_template.jinja`. Operational notes: - **macOS 27.0+** on **Apple silicon**. Validated on M4 Max. - Runs on the **GPU** via an `MPSGraph` delegate. This is **not** an ANE asset: the KV dimension is dynamic, and `--preferred-compute` does not change the emitted delegate. - Budget roughly bundle size plus KV cache, about **5 GB** at 4096 context. - `aimodelc-h16c/` is **architecture-locked to `h16c`**; the runtime names the architecture it wanted when it refuses. ## Validation evidence Machine-readable under `evidence/`: `authored_parity_vs_huggingface.json`, `fp32_reference.json`, `recipe_quality.json`, `conversion_gate.json`, `benchmark.json`, `compiled_storage_stats.json`. Four **separate** questions, not interchangeable: - **Authoring fidelity** — re-authored module vs Hugging Face, fp32: 21/21 top-1, cosine 1.000000. This caught a real bug: the checkpoint sets `rope_parameters.rope_theta = 1e7`, and code reading only the legacy top-level key silently defaults to `1e6` — a 10× wrong RoPE that still produces fluent short text. Both the converter overlay and transformers 4.x hit it. - **Quantization damage** — vs the fp32 oracle: 122/125, min cosine 0.997050. - **Conversion fidelity** — bundle vs **its own** quantized weights run eagerly: 5/5 exact. Comparing to fp32 here would conflate quantization damage with conversion bugs. - **Throughput** — see the caveat above. ## Limitations and negative results - **Not runnable without the non-public patch stack.** See the top of this card. - No task-benchmark evaluation; quality is a 125-position regression probe. - Multilingual support is inherited from upstream and was **not** re-verified per language; the probe is English. - int4 rejected at block 32 (cosine 0.51–0.66) and at block 16 (quality recovers, 42 tok/s, ~3× slower than int8). - `--expect-frequent-reshapes` measured 84 tok/s and 8.3 GB; not used. - Speculative decoding not shipped: a static-S verify graph exports and its contract gates, but per-position logits do not match stepped decode. - 300 tok/s was a target and was not reached by any tested configuration. ## License and attribution The model is **Liquid AI's**. This repository redistributes a converted, quantized copy under the upstream license, and claims **no authorship of the model**. - Upstream: [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B) by **Liquid AI**. - Upstream license: **LFM Open License v1.0** — pinned copy [here](https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/dca1825886789bd40b94368f53b1d9ada4c94598/LICENSE), included verbatim as [`LICENSE.upstream`](LICENSE.upstream). Your use of these weights is governed by it. - Conversion tooling: Apple's `coreai-models`, `coreai-torch` and `coreai-model-zoo`, at the commits pinned above. **No Apple source is redistributed here.** - This repository contributes the conversion recipe, the gates, and the measurements.