Instructions to use FastFlowLM/LFM2.5-1.2B-Thinking-NPU2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FastFlowLM/LFM2.5-1.2B-Thinking-NPU2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="FastFlowLM/LFM2.5-1.2B-Thinking-NPU2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("FastFlowLM/LFM2.5-1.2B-Thinking-NPU2")
model = AutoModelForCausalLM.from_pretrained("FastFlowLM/LFM2.5-1.2B-Thinking-NPU2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FastFlowLM/LFM2.5-1.2B-Thinking-NPU2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FastFlowLM/LFM2.5-1.2B-Thinking-NPU2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FastFlowLM/LFM2.5-1.2B-Thinking-NPU2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FastFlowLM/LFM2.5-1.2B-Thinking-NPU2

SGLang

How to use FastFlowLM/LFM2.5-1.2B-Thinking-NPU2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FastFlowLM/LFM2.5-1.2B-Thinking-NPU2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FastFlowLM/LFM2.5-1.2B-Thinking-NPU2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FastFlowLM/LFM2.5-1.2B-Thinking-NPU2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FastFlowLM/LFM2.5-1.2B-Thinking-NPU2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use FastFlowLM/LFM2.5-1.2B-Thinking-NPU2 with Docker Model Runner:
```
docker model run hf.co/FastFlowLM/LFM2.5-1.2B-Thinking-NPU2
```

FastFlowLM commited on Jan 19

Commit

7d0657b

verified ·

1 Parent(s): 67f9fad

Update README.md

Browse files

Files changed (1) hide show

README.md +35 -15

README.md CHANGED Viewed

@@ -201,21 +201,41 @@ LFM2.5-1.2B-Thinking offers extremely fast inference speed on CPUs with a low me
 In addition, we are partnering with AMD, Qualcomm, Nexa AI, and FastFlowLM to bring the LFM2.5 family to NPUs. These optimized models are available through our partners, enabling highly efficient on-device inference.
-We report the following numbers with 1K prefill and 100 decode tokens:
-| Device                                               | Inference | Framework        | Model                | Prefill (tok/s) | Decode (tok/s) | Memory |
-| ---------------------------------------------------- | --------- | ---------------- | -------------------- | --------------- | -------------- | ------ |
-| AMD Ryzen AI 395+                                    | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking | 1487            | 60             | 1700MB |
-| AMD Ryzen AI 9 HX 370                                | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking | 1487            | 57             | 1700MB |
-| AMD Ryzen AI 7 HX 350                                | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking | 1431            | 63             | 1700MB |
-| AMD Ryzen AI 5 HX 340                                | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking | 1431            | 63             | 1700MB |
-| AMD Ryzen AI 9 HX 370                                | GPU       | llama.cpp (Q4_0) | LFM2.5-1.2B-Thinking | 2975            | 116            | 856MB  |
-| Qualcomm Snapdragon® X Elite                         | NPU       | NexaML           | LFM2.5-1.2B-Thinking | 2591            | 63             | 0.9GB  |
-| Qualcomm Snapdragon® Gen4 (ROG Phone9 Pro)           | NPU       | NexaML           | LFM2.5-1.2B-Thinking | 4391            | 82             | 0.9GB  |
-| Qualcomm Dragonwing IQ9 (IQ-9075) (IoT)              | NPU       | NexaML           | LFM2.5-1.2B-Thinking | 2143            | 53             | 0.9 GB |
-| Qualcomm Snapdragon® Gen4 (Samsung Galaxy S25 Ultra) | CPU       | llama.cpp (Q4_0) | LFM2.5-1.2B-Thinking | 335             | 70             | 719MB  |
-LFM2.5-1.2B-Thinking works well with long contexts. On AMD NPU, it delivers **~59 tok/s at 4K** context and **~52 tok/s at 16K** during decoding, and reaches a **prefill speed of ~2,226 tok/s with a 4K-token prompt**. Check the detailed benchmark results [here](https://fastflowlm.com/docs/benchmarks/lfm2_results/).
 These capabilities unlock new deployment scenarios across various devices, including vehicles, mobile devices, laptops, IoT devices, and embedded systems.

 In addition, we are partnering with AMD, Qualcomm, Nexa AI, and FastFlowLM to bring the LFM2.5 family to NPUs. These optimized models are available through our partners, enabling highly efficient on-device inference.
+#### **Prefill Performance**
+We report prefill throughput evaluated over a range of prompt lengths.
+| Platform / Device                                  | Inference | Framework        | Model              | 1K Prefill (tok/s) | 4K Prefill (tok/s) | 16K Prefill (tok/s) |  Memory |
+|----------------------------------------------------|-----------|------------------|-------------------|-------------------:|-------------------:|--------------------:| -------:|
+| AMD Ryzen™ AI 395+                                 | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking| 1,487              | 2,226          | 1,670           | 1.7 GB |
+| AMD Ryzen™ AI 9 HX 370                              | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking| 1,487              | 2,226          |1,670          | 1.7 GB |
+| AMD Ryzen™ AI 7 HX 350                              | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking| 1,431              | 2,032          |1,519          | 1.7 GB |
+| AMD Ryzen™ AI 5 HX 340                              | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking| 1,431              | 2,032          |1,519          | 1.7 GB |
+| AMD Ryzen™ AI 9 HX 370                              | CPU       | llama.cpp (Q4_0) | LFM2.5-1.2B-Thinking| 2,975          | N/A                | N/A                | 856 MB |
+| Qualcomm Snapdragon® X Elite                       | NPU       | NexaML           | LFM2.5-1.2B-Thinking| 2,591              | N/A                | N/A                 | 0.9 GB |
+| Qualcomm Snapdragon® Gen4 (ROG Phone 9 Pro)        | NPU       | NexaML           | LFM2.5-1.2B-Thinking| 4,391          | N/A                | N/A                 | 0.9 GB |
+| Qualcomm Dragonwing IQ9 (IQ-9075, IoT)              | NPU       | NexaML           | LFM2.5-1.2B-Thinking| 2,143              | N/A                | N/A                | 0.9 GB |
+| Qualcomm Snapdragon® Gen4 (Galaxy S25 Ultra)       | CPU       | llama.cpp (Q4_0) | LFM2.5-1.2B-Thinking| 335                | N/A                | N/A                 | 719 MB |
+#### **Decode Performance**
+The reported results correspond to decoding 100 tokens at different context lengths.
+| Platform / Device                                  | Inference | Framework        | Model              | Decode @1K (tok/s) | Decode @4K (tok/s) | Decode @16K (tok/s) |  Memory |
+|----------------------------------------------------|-----------|------------------|-------------------|-------------------:|-------------------:|--------------------:| -------:|
+| AMD Ryzen™ AI 395+                                 | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking| 60              | 54          | 49           | 1.7 GB |
+| AMD Ryzen™ AI 9 HX 370                              | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking| 57              | 54          |49          | 1.7 GB |
+| AMD Ryzen™ AI 7 HX 350                              | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking| 63              | 59          |52          | 1.7 GB |
+| AMD Ryzen™ AI 5 HX 340                              | NPU       | FastFlowLM       | LFM2.5-1.2B-Thinking| 63              | 59          |52          | 1.7 GB |
+| AMD Ryzen™ AI 9 HX 370                              | CPU       | llama.cpp (Q4_0) | LFM2.5-1.2B-Thinking| 116          | N/A                | N/A                | 856 MB |
+| Qualcomm Snapdragon® X Elite                       | NPU       | NexaML           | LFM2.5-1.2B-Thinking| 63              | N/A                | N/A                 | 0.9 GB |
+| Qualcomm Snapdragon® Gen4 (ROG Phone 9 Pro)        | NPU       | NexaML           | LFM2.5-1.2B-Thinking| 82          | N/A                | N/A                 | 0.9 GB |
+| Qualcomm Dragonwing IQ9 (IQ-9075, IoT)              | NPU       | NexaML          | LFM2.5-1.2B-Thinking | 53              | N/A                | N/A                | 0.9 GB |
+| Qualcomm Snapdragon® Gen4 (Galaxy S25 Ultra)       | CPU       | llama.cpp (Q4_0) | LFM2.5-1.2B-Thinking| 70                | N/A                | N/A                 | 719 MB |
+**LFM2.5-1.2B-Thinking excels at long-context inference.**
+On AMD NPUs with FastFlowLM, decoding throughput sustains ~46 tok/s even at the full 32K context, indicating robust long-context scalability.
+See detailed benchmark results (up to full context length) [here](https://fastflowlm.com/docs/benchmarks/lfm2_results/).
 These capabilities unlock new deployment scenarios across various devices, including vehicles, mobile devices, laptops, IoT devices, and embedded systems.