mosesman/LFM2-2.6B-openvino-int4-npu

LFM2-2.6B — OpenVINO INT4 (Intel AI Boost / NPU)

OpenVINO IR export of LiquidAI/LFM2-2.6B quantized for Intel AI Boost NPU.

Default INT4 exports of LFM2 often produce garbled tokens on NPU. This build uses Intel’s documented NPU-friendly recipe:

--weight-format int4 --sym --group-size 128 --backup-precision int8_sym

Validated on:

• CPU: Intel Core Ultra 7 256V (Lunar Lake)
• NPU: Intel AI Boost (NPU4000)
• iGPU: Intel Arc 140V
• Runtime: OpenVINO / GenAI 2026.3.0, NPU driver 32.0.100.4841

Why this exists

Export	Backup layers	NPU
Typical INT4 (int8_asym backup)	asymmetric / zero-points	❌ garbled
This repo (int8_sym backup)	symmetric INT8	✅ coherent

~95% of weights remain INT4; only layers that cannot stay INT4 use INT8-symmetric backup.

Quick start (NPU) — OpenVINO GenAI

pip install openvino openvino-genai transformers

import openvino_genai as ov_genai
from transformers import AutoTokenizer

model_id = "mosesman/LFM2-2.6B-openvino-int4-npu"
device = "NPU"  # or "GPU" / "CPU"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "What is 17*19? Reply with only the number."}],
    tokenize=False,
    add_generation_prompt=True,
)

pipe = ov_genai.LLMPipeline(model_id, device)
config = ov_genai.GenerationConfig()
config.max_new_tokens = 128
config.do_sample = False

print(pipe.generate(prompt, config))

NPU-only tip: pass "NPU" (not AUTO / HETERO). Tokenizer/detokenizer glue may still use CPU; model compute stays on NPU.

First-time NPU compile vs later starts

The first NPU load can take ~1–2 minutes while OpenVINO / the NPU driver compiles the graph. OpenVINO caches the compiled blob (e.g. .blob / driver caches such as .cl_cache under local temp / cache dirs), so subsequent cold starts usually launch in seconds instead of recompiling from scratch. Later generates in the same process are much faster either way.

Alternative: Optimum-Intel (OVModelForCausalLM)

pip install "optimum[openvino]" transformers

from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoTokenizer, pipeline

model_id = "mosesman/LFM2-2.6B-openvino-int4-npu"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = OVModelForCausalLM.from_pretrained(model_id, device="NPU")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
messages = [{"role": "user", "content": "What is 17*19? Reply with only the number."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(pipe(prompt, max_new_tokens=32, do_sample=False)[0]["generated_text"])

Local helper script

This folder includes run_lfm_device.py:

python run_lfm_device.py --device NPU --greedy

Reproduce the export

optimum-cli export openvino \
  --model LiquidAI/LFM2-2.6B \
  --trust-remote-code \
  --weight-format int4 \
  --sym \
  --group-size 128 \
  --backup-precision int8_sym \
  ./LFM2-2.6B-openvino-int4-npu

Tooling used for this build:

• optimum-intel 2.1.0
• nncf 3.3.0
• openvino / openvino-genai 2026.3.0

Benchmark notes (single machine)

Greedy decode, ~120 new tokens, corrected INT4 IR:

Device	Approx. tok/s	Notes
Arc 140V (GPU)	~59	fastest
CPU	~35	quick load
AI Boost (NPU)	~20	lowest power; slow first compile

Exact numbers vary by driver, prompt length, and power mode.

Requirements

• Intel Core Ultra with Intel AI Boost (for NPU)
• Recent Intel NPU driver (this build tested on 32.0.100.4841; OpenVINO notes recommend ≥ 32.0.100.4621 on Windows)
• OpenVINO 2026.3+ recommended (LFM2 NPU / NPUW support)

Also runs on Intel GPU / CPU via the same IR if needed.

Files

OpenVINO GenAI layout:

• openvino_model.xml / .bin
• openvino_tokenizer.xml / .bin
• openvino_detokenizer.xml / .bin
• tokenizer + chat template configs

License / attribution

Derivative of LiquidAI/LFM2-2.6B under the LFM Open License v1.0 (lfm1.0).

• License text: see LICENSE (copied from the base model)
• Upstream: https://huggingface.co/LiquidAI/LFM2-2.6B
• License FAQ: https://www.liquid.ai/lfm-license

This repository redistributes a quantized derivative. Downstream commercial use remains subject to Liquid AI’s license terms (including the commercial revenue threshold).

Disclaimer

Provided as-is for interoperability with Intel NPU / OpenVINO. Not affiliated with Liquid AI or Intel. Validate quality for your own workload before production use.

Citation

If you use LFM2, please cite Liquid AI’s model card / paper for LiquidAI/LFM2-2.6B.