--- license: other license_name: lfm1.0 license_link: LICENSE language: - en pipeline_tag: text-generation tags: - liquid - edge - lfm2 - lfm2.5 - moe - mixture-of-experts - onnx - onnxruntime base_model: - LiquidAI/LFM2.5-8B-A1B ---

Try LFM • Docs • LEAP • Discord

# LFM2.5-8B-A1B-ONNX ONNX export of [LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B) for cross-platform inference. LFM2.5-8B-A1B is a Mixture of Experts model with 8B total parameters and about 1B active parameters per token. It uses 32 experts with 4 experts activated per token, combining the efficiency of sparse models with the quality of larger dense models. ## Recommended Variants | Precision | Size | Use Case | |-----------|------|----------| | Q4F16 | ~4.7GB | Recommended (Q4 MoE + FP16 dense) | | FP16 | ~15.8GB | Higher quality | | Q4 | ~5.2GB | Smallest size | | Q8 | ~30.4GB | Highest-fidelity quantized variant | Note: This model is too large for WebGPU browser inference. ## Validation This export was validated against the local PyTorch reference for `LiquidAI/LFM2.5-8B-A1B`. - FP32 padded-batch parity passed for both left and right padding, with cosine similarity `1.0000` and top-5 overlap `5/5` at the last valid token for each row. - Q4 decoder and coherence checks passed the repository thresholds. Average coherence similarity: `0.7144`. - Q4F16 was runtime-validated on `CPUExecutionProvider` and matched the same decoder/coherence thresholds as Q4. Average coherence similarity: `0.7145`. - Q8 decoder and coherence checks passed, and stayed very close to the PyTorch reference. Average coherence similarity: `0.9975`. ## Model Files ``` onnx/ ├── model.onnx # FP32 model graph ├── model.onnx_data* # FP32 weights ├── model_fp16.onnx # FP16 model graph ├── model_fp16.onnx_data* # FP16 weights ├── model_q4.onnx # Q4 model graph ├── model_q4.onnx_data* # Q4 weights ├── model_q4f16.onnx # Q4 MoE experts + FP16 dense (recommended) ├── model_q4f16.onnx_data* # Q4F16 weights ├── model_q8.onnx # Q8 model graph └── model_q8.onnx_data* # Q8 weights * Large models split weights across multiple files: model.onnx_data, model.onnx_data_1, model.onnx_data_2, etc. All data files must be in the same directory as the .onnx file. ``` ## Python ### Installation ```bash pip install onnxruntime transformers numpy huggingface_hub # or with GPU support: pip install onnxruntime-gpu transformers numpy huggingface_hub ``` ### Inference ```python from huggingface_hub import snapshot_download from transformers import AutoConfig, AutoTokenizer import numpy as np import onnxruntime # 1. Load config, tokenizer, and model model_id = "LiquidAI/LFM2.5-8B-A1B-ONNX" config = AutoConfig.from_pretrained(model_id) tokenizer = AutoTokenizer.from_pretrained(model_id) eos_token_id = config.eos_token_id filename = "model_q4f16.onnx" # Options: model.onnx, model_fp16.onnx, model_q4.onnx, model_q4f16.onnx, model_q8.onnx model_path = snapshot_download(repo_id=model_id, allow_patterns=f"onnx/{filename}*") session = onnxruntime.InferenceSession(f"{model_path}/onnx/{filename}") input_names = {inp.name for inp in session.get_inputs()} # 2. Prepare inputs prompt = "What is C. elegans?" messages = [{"role": "user", "content": prompt}] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="np", ) input_ids = inputs["input_ids"] attention_mask = inputs["attention_mask"] batch_size = input_ids.shape[0] past_cache_values = {} for inp in session.get_inputs(): name = inp.name shape = inp.shape dtype = np.float32 if inp.type == "tensor(float)" else np.float16 if name.startswith("past_key_values"): past_cache_values[name] = np.zeros([batch_size, shape[1], 0, shape[3]], dtype=dtype) elif name.startswith("past_conv"): past_cache_values[name] = np.zeros([batch_size, shape[1], shape[2]], dtype=dtype) position_ids = np.arange(input_ids.shape[1], dtype=np.int64).reshape(1, -1) # 3. Generation loop max_new_tokens = 256 generated_tokens = np.array([[]], dtype=np.int64) cur_len = input_ids.shape[1] for i in range(max_new_tokens): if i == 0: ids = input_ids pos = position_ids else: ids = generated_tokens[:, -1:] pos = np.array([[cur_len - 1]], dtype=np.int64) feed = { "input_ids": ids, "attention_mask": attention_mask, **past_cache_values, } if "position_ids" in input_names: feed["position_ids"] = pos outputs = session.run(None, feed) logits = outputs[0] next_token = logits[:, -1].argmax(-1, keepdims=True) generated_tokens = ( next_token if generated_tokens.shape[1] == 0 else np.concatenate([generated_tokens, next_token], axis=-1) ) attention_mask = np.concatenate( [attention_mask, np.ones_like(next_token, dtype=np.int64)], axis=-1, ) output_names = [out.name for out in session.get_outputs()] cache_outputs = { name: value for name, value in zip(output_names[1:], outputs[1:]) } for key in past_cache_values: present_key = key.replace("past_key_values", "present").replace("past_conv", "present_conv") past_cache_values[key] = cache_outputs[present_key] cur_len += 1 if np.isin(next_token, eos_token_id).any(): break print(tokenizer.decode(next_token[0]), end="", flush=True) print() # 4. Output result print(tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]) ``` ## License This model is released under the [LFM 1.0 License](LICENSE).