--- license: other license_name: lfm-open-license-v1.0 license_link: https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/dca1825886789bd40b94368f53b1d9ada4c94598/LICENSE base_model: LiquidAI/LFM2.5-2.6B base_model_relation: quantized pipeline_tag: text-generation library_name: coreai language: - ar - zh - en - fr - de - hi - id - it - ja - ko - pl - pt - ru - es - th - vi tags: - coreai - aimodel - aimodelc - apple-silicon - lfm2 - lfm2.5 - liquid - edge - int8 - community-model-zoo --- # LFM2.5-2.6B — Core AI (int8) An int8 Core AI conversion of **[LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B)** for Apple silicon. This repository contains no trained weights of its own: it is a quantized format conversion of Liquid AI's model, and all model credit belongs to **Liquid AI**. > **Runtime status:** runnable with public tooling, but not with an unmodified > `apple/coreai-models` checkout. LFM's fixed-shape convolution state requires two patches from > the public, community-maintained > [`john-rocky/coreai-model-zoo`](https://github.com/john-rocky/coreai-model-zoo). Exact public > commits and commands are below. Converted from source revision `dca1825886789bd40b94368f53b1d9ada4c94598`. Both upstream safetensors shards were SHA-256 verified against that revision before conversion. ## Runtime requirements This graph has the normal growing key/value cache plus one fixed-shape `conv_state`. The stock pipelined engine handles only the KV pair. Apply these **two** public community patches—no per-token-input, static-input, or prefix-cache patch is required for this model: 1. `apps/coreai-shared-product.patch` 2. `apps/coreai-pipelined-extra-states.patch` Both come from [`john-rocky/coreai-model-zoo@95a29d41`](https://github.com/john-rocky/coreai-model-zoo/tree/95a29d41affed3bdf3ea5992ece094a908e21a04), a public community repository maintained by Daisuke Majima (`john-rocky`). It is not an Apple repository. The patches target Apple's public [`coreai-models`](https://github.com/apple/coreai-models) runtime at `5ed9981303b38d5a44aa6b45509bc4f6945029f5`. ```bash git clone https://github.com/apple/coreai-models.git git -C coreai-models checkout 5ed9981303b38d5a44aa6b45509bc4f6945029f5 git clone https://github.com/john-rocky/coreai-model-zoo.git git -C coreai-model-zoo checkout 95a29d41affed3bdf3ea5992ece094a908e21a04 git -C coreai-models apply ../coreai-model-zoo/apps/coreai-shared-product.patch git -C coreai-models apply ../coreai-model-zoo/apps/coreai-pipelined-extra-states.patch swift build --package-path coreai-models -c release --product llm-runner ``` The public community zoo also supplies the LFM2 exporter and its `coreai-models` overlay. The small toolkit published here contains the 2.6B-specific wrapper, tokenizer/config corrections, quality gates, and locked Python environment used for this artifact. Full reproduction details are in [`RECIPE.md`](RECIPE.md). ## Which file do I want? **The repository root *is* the portable bundle.** The runtime wants a directory containing `metadata.json`, the `*.aimodel/` graph and `tokenizer/`, and those sit at the top level here. So the recommended download is the root minus the compiled variant — the exact command is under [Download](#download). `aimodelc-h16c/` is that same model pre-compiled for one GPU architecture: identical output and speed, about half the cold-load time, and the runtime rejects it on a different architecture. Take it only if you are on `h16c` and care about load time. ## What this is and is not - It **is** a decode-optimized single-token-step graph, the shape a chat/completion loop uses. - It **is not** a chunked-prefill or batch-serving asset. - Quality here means **teacher-forced top-1 agreement and cosine similarity against an independent fp32 Hugging Face reference**, over 5 sequences / 125 positions. That is a regression probe, **not** a benchmark suite. No MMLU/GSM8K-style numbers are claimed. ## Artifacts Full SHA-256 of each file as published. | File | Bytes | SHA-256 | Recipe | Hardware scope | Use | | --- | ---: | --- | --- | --- | --- | | `lfm2_5_2_6b_decode_int8hu_attnfp16_block32_sym.aimodel/main.mlirb` | 3469367807 | `80540b2ee9183b756adb1ce51f334a94f13c093a554a2538aec66f9d9f3a07b3` | int8 blockwise-32, fp16 attention + embedding | any Apple silicon | **recommended** | | `aimodelc-h16c/…h16c.aimodelc/…/resources.bin` | 3468949676 | `793907c101a01331ffa72aff7c9db5049c89a02ea03acee3174513a93c394e22` | same weights, AOT compiled | **`h16c` only** | faster cold load | | `aimodelc-h16c/…h16c.aimodelc/…/original_model_0.mpsgraph` | 363905 | `74a24870a8853bf797a6f12f40e7fcb988de014d98166da772b24cab643d5d4f` | compiled graph | `h16c` only | part of the above | | `tokenizer/tokenizer.json` | 17905598 | `695be7802a0e4b8a81048f0ff5ebb7fc811a0ba5a6be63dbb24deb5a81096f41` | upstream, unmodified | — | required | | `lfm2.5-2.6b-coreai-conversion-e0e375b.tar.gz` | 57118 | `665285044be7c0171e1b737167b9892f2799bc2e01e3c5441212887903456f2d` | exact wrapper + gates at local conversion commit `e0e375b` | — | reproduction | - **The two `tokenizer/tokenizer.json` copies are byte-identical** (same hash above). The one under `aimodelc-h16c/tokenizer/` is a **convenience copy**; you do not need both. - Portable and compiled are **the same model**, published together because the compiled one halves cold load but only runs on one architecture, so neither dominates. - Both were produced from the same pinned commit and have identical compiled storage budgets. ## Recipe, and what was rejected Oracle for every quality number: **an independent fp32 reference, Hugging Face's own `Lfm2ForCausalLM` at transformers ≥ 5.2**, teacher-forced over 5 sequences / 125 positions. Cosine is the minimum per-position cosine. | Variant | Bundle | Top-1 vs fp32 oracle | Min cosine | Conversion | Shipped | | --- | ---: | ---: | ---: | ---: | --- | | **int8 blockwise-32, fp16 attention + embedding** | 3.25 GB | **122/125** | **0.997050** | 5/5 | **yes** | | + attention q/k/v/out to int8 | 3.19 GB | 121/125 | 0.996949 | 5/5 | no | | + embedding to int8 | 3.03 GB | 123/125 | 0.996848 | 5/5 | no | | attention at fp32 (converter default) | 3.42 GB | — | 0.997210 | 5/5 | no | | int4 blockwise-32 family | 2.07–2.34 GB | — | 0.51–0.80 | 4–5/5 | no | The shipped arm has the highest minimum cosine of the int8 arms. The ±1 position differences between the three int8 rows are near-tie argmax flips at 125 positions, not a systematic ordering — do not read the embedding row's 123 as "better". int4 is excluded on **quality**, not size: a minimum cosine of 0.51–0.80 is a different model. **Rejected variants are documented and deliberately not uploaded.** Full detail in [`RECIPE.md`](RECIPE.md). ## Performance Decode here is **memory-bandwidth bound** — throughput tracks bytes read per token. | Measurement | Protocol | Result | | --- | --- | --- | | This published artifact | 128 prompt tokens, 256 generated, greedy, 5 trials, M4 Max | **107.98 decode tok/s**, sd 0.500; **111.86 prompt tok/s** | | Earlier development build of the same recipe | separate runs on the same M4 Max | 138.15 decode tok/s | Only **107.98 tok/s** is attached to the published artifact in `evidence/benchmark.json`. The earlier 138.15 result was not reproduced after publication, so it is retained as historical context rather than advertised as this download's performance. The workload is memory-bandwidth-bound and was measured while the machine was busy; measure on your own hardware. For reference, on the same Mac, MLX reported ~100 tok/s at 8-bit and ~60 tok/s at BF16. Those were throughput-only observations; MLX quality was not measured, so no quality comparison against MLX is claimed. AOT compilation is throughput-neutral within noise and halves cold load, 9.8 s → 5.1 s. ## Reproduce Every pin, the full quantization spec, the four gates, the measurement protocol and the rejected variants are in **[`RECIPE.md`](RECIPE.md)**. Summary: ``` Source model : LiquidAI/LFM2.5-2.6B @ dca1825886789bd40b94368f53b1d9ada4c94598 (public) Converter : github.com/apple/coreai-models @ b1cb71b8522d99408059fa0b98b8742171bcb0b8 (public) Runtime : github.com/apple/coreai-models @ 5ed9981303b38d5a44aa6b45509bc4f6945029f5 (public) Community LFM2 exporter, overlay and two runtime patches: github.com/john-rocky/coreai-model-zoo @ 95a29d41affed3bdf3ea5992ece094a908e21a04 (public) 2.6B wrapper : lfm2.5-2.6b-coreai-conversion-e0e375b.tar.gz SHA-256 665285044be7c0171e1b737167b9892f2799bc2e01e3c5441212887903456f2d Toolchain : macOS 27.0 (26A5388g), Xcode 27.0 (27A5228h), Python 3.11.15, torch 2.9.0, coreai-core 1.0.0b2, coreai-torch 0.4.1, coreai-opt 0.2.1, coremltools 9.0 ``` **Verify a rebuild by the gates and the storage budget, not by hashing.** The exporter names each externalized call site with a generated UUID (391 in this graph), so two exports of identical weights differ in a few bytes and therefore in SHA-256. The budget to match is `Int8 2,621,243,392` + `Float16 428,342,276` + `Float32 34` + small index types. ## Download ```bash # The portable bundle: the .aimodel graph, the tokenizer and the bundle metadata. # This is the whole repository except the compiled h16c variant (~3.5 GB rather than ~7 GB). hf download harshav/LFM2.5-2.6B-CoreAI \ --exclude 'aimodelc-h16c/*' \ --local-dir ./LFM2.5-2.6B-CoreAI # The directory you then point the runtime at is ./LFM2.5-2.6B-CoreAI itself. ``` With a runtime built as described above, the asset is driven as a pipelined Core AI language model with `COREAI_CHUNK_THRESHOLD=1`, greedy decoding, using the model's own chat template at `tokenizer/chat_template.jinja`. Operational notes: ```bash COREAI_CHUNK_THRESHOLD=1 coreai-models/.build/release/llm-runner \ --model ./LFM2.5-2.6B-CoreAI \ --prompt "The capital of France is" \ --max-tokens 64 --temperature 0 \ --inference-engine-variant coreai-pipelined \ --warmup exact --warmup-length 1 ``` - **macOS 27.0+** on **Apple silicon**. Validated on M4 Max. - Runs on the **GPU** via an `MPSGraph` delegate. This is **not** an ANE asset: the KV dimension is dynamic, and `--preferred-compute` does not change the emitted delegate. - Budget roughly bundle size plus KV cache, about **5 GB** at 4096 context. - `aimodelc-h16c/` is **architecture-locked to `h16c`**; the runtime names the architecture it wanted when it refuses. ## Validation evidence Machine-readable under `evidence/`: `authored_parity_vs_huggingface.json`, `fp32_reference.json`, `recipe_quality.json`, `conversion_gate.json`, `benchmark.json`, `compiled_storage_stats.json`. Four **separate** questions, not interchangeable: - **Authoring fidelity** — re-authored module vs Hugging Face, fp32: 21/21 top-1, cosine 1.000000. This caught a real bug: the checkpoint sets `rope_parameters.rope_theta = 1e7`, and code reading only the legacy top-level key silently defaults to `1e6` — a 10× wrong RoPE that still produces fluent short text. Both the converter overlay and transformers 4.x hit it. - **Quantization damage** — vs the fp32 oracle: 122/125, min cosine 0.997050. - **Conversion fidelity** — bundle vs **its own** quantized weights run eagerly: 5/5 exact. Comparing to fp32 here would conflate quantization damage with conversion bugs. - **Throughput** — see the caveat above. ## Limitations and negative results - Requires the two public community runtime patches listed above; it does not run on an unmodified `apple/coreai-models` runtime. - No task-benchmark evaluation; quality is a 125-position regression probe. - Multilingual support is inherited from upstream and was **not** re-verified per language; the probe is English. - int4 rejected at block 32 (cosine 0.51–0.66) and at block 16 (quality recovers, 42 tok/s, ~3× slower than int8). - `--expect-frequent-reshapes` measured 84 tok/s and 8.3 GB; not used. - Speculative decoding not shipped: a static-S verify graph exports and its contract gates, but per-position logits do not match stepped decode. - 300 tok/s was a target and was not reached by any tested configuration. ## License and attribution The model is **Liquid AI's**. This repository redistributes a converted, quantized copy under the upstream license, and claims **no authorship of the model**. - Upstream: [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B) by **Liquid AI**. - Upstream license: **LFM Open License v1.0** — pinned copy [here](https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/dca1825886789bd40b94368f53b1d9ada4c94598/LICENSE), included verbatim as [`LICENSE.upstream`](LICENSE.upstream). Your use of these weights is governed by it. - Official Apple tooling: [`apple/coreai-models`](https://github.com/apple/coreai-models), `coreai-core`, `coreai-torch`, and `coreai-opt`. - Community conversion/runtime work: public [`john-rocky/coreai-model-zoo`](https://github.com/john-rocky/coreai-model-zoo), maintained by **Daisuke Majima**, under its BSD-3-Clause license. Its repository includes third-party notices for portions derived from Apple's BSD-licensed `coreai-models` project. - This repository contributes the conversion recipe, the gates, and the measurements.