Instructions to use meituan-longcat/LongCat-Flash-Lite with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meituan-longcat/LongCat-Flash-Lite with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="meituan-longcat/LongCat-Flash-Lite", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("meituan-longcat/LongCat-Flash-Lite", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use meituan-longcat/LongCat-Flash-Lite with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meituan-longcat/LongCat-Flash-Lite"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meituan-longcat/LongCat-Flash-Lite",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/meituan-longcat/LongCat-Flash-Lite

SGLang

How to use meituan-longcat/LongCat-Flash-Lite with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meituan-longcat/LongCat-Flash-Lite" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meituan-longcat/LongCat-Flash-Lite",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meituan-longcat/LongCat-Flash-Lite" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meituan-longcat/LongCat-Flash-Lite",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use meituan-longcat/LongCat-Flash-Lite with Docker Model Runner:
```
docker model run hf.co/meituan-longcat/LongCat-Flash-Lite
```

GGUF quants and llama.cpp inference?

by ljupco - opened Feb 27

Discussion

ljupco

Feb 27

Does anyone know if there is any hope of bringing LongCat-Flash-Lite into the llama.cpp codebase? So inference is working under llama.cpp, and we get GGUF quants? For me on an old MacBook Pro M2 (but with 96gb ram), the mlx-lm inference is x5-x10 times slower than equivalent GGUF models quants running llama.cpp llama-server. IDK but I can only presume that it's flash attention, 8-bit KV caches, prompts caching etc that make the huge difference.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment