Instructions to use NousResearch/Hermes-2-Pro-Mistral-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NousResearch/Hermes-2-Pro-Mistral-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NousResearch/Hermes-2-Pro-Mistral-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("NousResearch/Hermes-2-Pro-Mistral-7B")
model = AutoModelForCausalLM.from_pretrained("NousResearch/Hermes-2-Pro-Mistral-7B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use NousResearch/Hermes-2-Pro-Mistral-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NousResearch/Hermes-2-Pro-Mistral-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Hermes-2-Pro-Mistral-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/NousResearch/Hermes-2-Pro-Mistral-7B

SGLang

How to use NousResearch/Hermes-2-Pro-Mistral-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NousResearch/Hermes-2-Pro-Mistral-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Hermes-2-Pro-Mistral-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NousResearch/Hermes-2-Pro-Mistral-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Hermes-2-Pro-Mistral-7B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use NousResearch/Hermes-2-Pro-Mistral-7B with Docker Model Runner:
```
docker model run hf.co/NousResearch/Hermes-2-Pro-Mistral-7B
```

Custom Quantization Types

by christopherthompson81 - opened Mar 14, 2024

Discussion

christopherthompson81

Mar 14, 2024

I would like to make my own quants but the vocab is shown as incomplete when converting with llama.cpp. This likely means that the tokenizer implementation is unsupported in llama.cpp:main. Is there a llama.cpp PR or a fork I could look at that does support it? Like the nous-llama.cpp repo?

Suparious

Mar 14, 2024

What quant format? I was able to make AWQ without any issue.

Joseph717171

Mar 14, 2024

•

edited Mar 14, 2024

Use the --pad-vocab option when converting to gguf (this will resolve the issue):

python convert.py $model --pad-vocab --outtype f16

christopherthompson81

Mar 14, 2024

Padding the vocab ends up with dummy tokens being generated pretty frequently and is therefore not useful.

qnguyen3

Mar 14, 2024

you can use convert-hf-to-gguf as well

teknium

NousResearch org Mar 14, 2024

Is this all good now? We resized vocab to a multiple of 32 even though we only added 2 tokens because it causes less issues with tensor parallelism and should make the model inference faster

Joseph717171

Mar 14, 2024

•

edited Mar 15, 2024

It was fine to begin with. All anyone had to do to successfully convert the model to GGUF was use —pad-vocab. Doing so would resolve the issue of unequal sized vocabs and allows the conversion process to successfully complete. 😁

deleted

Mar 14, 2024

This comment has been hidden

teknium changed discussion status to closed Mar 14, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment