Instructions to use OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17")
model = AutoModelForCausalLM.from_pretrained("OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17

SGLang

How to use OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17 with Docker Model Runner:
```
docker model run hf.co/OpenLLM-Ro/RoMistral-7b-Instruct-2024-05-17
```

O idee generală cu privire la conținutul folosit pentru antrenare?

by scaraliu - opened May 17, 2024

Discussion

scaraliu

May 17, 2024

Salutare și felicitări!
Aș fi fost interesat la modul general cu privire la conținutul folosit pentru antrenare.
Mulțumesc.

trebedea

OpenLLM-Ro org May 17, 2024

Salutare,

Sigur, toate detaiile sunt in model card si in lucrare: https://arxiv.org/abs/2405.07703
Vom publica si seturile de date in romana in urmtoarele saptamani.

Traian

scaraliu

May 17, 2024

Salutare,
Am citit rapid paper-ul, trebuie să recunosc că m-am grăbit...
Practic am plasat întrebarea pe nerăsuflate.
Mulțumesc și felicitări echipei!
PS - mai sunt locuri?

mihaimasala

OpenLLM-Ro org Aug 14, 2024

Salut! Am publicat si datele folosite pentru finetune.

mihaimasala changed discussion status to closed Aug 14, 2024

scaraliu

Aug 14, 2024

cool!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment