Instructions to use knifeayumu/Cydonia-v1.3-Magnum-v4-22B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use knifeayumu/Cydonia-v1.3-Magnum-v4-22B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="knifeayumu/Cydonia-v1.3-Magnum-v4-22B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("knifeayumu/Cydonia-v1.3-Magnum-v4-22B")
model = AutoModelForCausalLM.from_pretrained("knifeayumu/Cydonia-v1.3-Magnum-v4-22B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use knifeayumu/Cydonia-v1.3-Magnum-v4-22B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "knifeayumu/Cydonia-v1.3-Magnum-v4-22B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "knifeayumu/Cydonia-v1.3-Magnum-v4-22B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/knifeayumu/Cydonia-v1.3-Magnum-v4-22B

SGLang

How to use knifeayumu/Cydonia-v1.3-Magnum-v4-22B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "knifeayumu/Cydonia-v1.3-Magnum-v4-22B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "knifeayumu/Cydonia-v1.3-Magnum-v4-22B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "knifeayumu/Cydonia-v1.3-Magnum-v4-22B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "knifeayumu/Cydonia-v1.3-Magnum-v4-22B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use knifeayumu/Cydonia-v1.3-Magnum-v4-22B with Docker Model Runner:
```
docker model run hf.co/knifeayumu/Cydonia-v1.3-Magnum-v4-22B
```

Feedback

by bigfish3 - opened Dec 13, 2024

Discussion

bigfish3

Dec 13, 2024

•

edited Dec 13, 2024

Good stuff. Any hope for a Cydonia-v1.25-Magnum-v4-22B? (A merge between the two merges.) Adventure is great, but somehow Cydonia-v1.2-Magnum-v4-22B elaborated more it also felt less rushed. For example a character could be doing a mildly time consuming task (be it supposed filing documents, supposed cooking, playing with their imaginary cat) in v1.2 and it'd eventually finish after a couple replies. v1.3 on the other hand they may finish after half as many replies and quickly imply moving onto something else (not that, that isn't technically realistic) partially changing the current situation at a quicker pace. Which isn't exactly bad, but would be nice if there were a tad less adventurous version a v1.2 and a half lol. I did like the adventurous aspects in this model being mixed with magnum though.