Instructions to use Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip")
model = AutoModelForCausalLM.from_pretrained("Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip

SGLang

How to use Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip with Docker Model Runner:
```
docker model run hf.co/Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip
```

Qwen2.5-3B-Instruct-Add-Speech-Token-4096-Nostrip

Introduction

This repo contains the Qwen2.5-3B-Instruct-Add-Speech-Token-4096-Nostrip model utilized to train the EMOVA series of models. Based on the original Qwen2.5-3B-Instruct checkpoint, we further insert speech tokens into its vocabulary for end-to-end omni-modal alignment as follows. The total number of speech tokens in EMOVA speech tokenizer is 4096. Therefore, it should be utilized as initialization in the Stage 2: Omni-modal text-centric alignment of EMOVA training.

# Source code can be found https://github.com/emova-ollm/EMOVA#insert-speech-tokens-into-llm-vocabulary
python scripts/insert_speech_token.py \
  --origin_model_path Qwen/Qwen2.5-3B-Instruct \
  --saved_model_path ./Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip \
  --num_speech_tokens 4096

Usage

To train EMOVA with Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip, we need to create a new model config, and set the language_model parameters as follows. An example is provided here. Check more details on training EMOVA in our github repo.

language_model=dict(
  type='EmovaQwen2ForCausalLM',                                              -- Wrapper class type for EMOVA
  pretrained_model_name_or_path='Emova-ollm/Qwen2.5-3B-Instruct_add_speech_token_4096_nostrip',  -- HuggingFace repo of pre-trained LLM
  attn_implementation="flash_attention_2",                                   -- Attention type
  from_pretrained=True,                                                      -- Load pre-trained weights
),

Citation

@article{chen2024emova,
  title={Emova: Empowering language models to see, hear and speak with vivid emotions},
  author={Chen, Kai and Gou, Yunhao and Huang, Runhui and Liu, Zhili and Tan, Daxin and Xu, Jing and Wang, Chunwei and Zhu, Yi and Zeng, Yihan and Yang, Kuo and others},
  journal={arXiv preprint arXiv:2409.18042},
  year={2024}
}

@article{qwen2.5,
    title   = {Qwen2.5 Technical Report}, 
    author  = {An Yang and Baosong Yang and Beichen Zhang and Binyuan Hui and Bo Zheng and Bowen Yu and Chengyuan Li and Dayiheng Liu and Fei Huang and Haoran Wei and Huan Lin and Jian Yang and Jianhong Tu and Jianwei Zhang and Jianxin Yang and Jiaxi Yang and Jingren Zhou and Junyang Lin and Kai Dang and Keming Lu and Keqin Bao and Kexin Yang and Le Yu and Mei Li and Mingfeng Xue and Pei Zhang and Qin Zhu and Rui Men and Runji Lin and Tianhao Li and Tingyu Xia and Xingzhang Ren and Xuancheng Ren and Yang Fan and Yang Su and Yichang Zhang and Yu Wan and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zihan Qiu},
    journal = {arXiv preprint arXiv:2412.15115},
    year    = {2024}
}