Instructions to use Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2")
model = AutoModelForCausalLM.from_pretrained("Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2

SGLang

How to use Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2 with Docker Model Runner:
```
docker model run hf.co/Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2
```

Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2 / README.md

Meggido

Upload folder using huggingface_hub

1ef0757 verified over 2 years ago

preview code

Raw

History Blame

6.25 kB

	---
	base_model: meta-llama/Meta-Llama-3-8B
	tags:
	- Llama-3
	- instruct
	- finetune
	- chatml
	- axolotl
	- roleplay
	license: apache-2.0
	language:
	- en
	---

	# ⚡ExLlamaV2 quant of : [Pantheon-RP-1.0-8b-Llama-3](https://huggingface.co/Gryphe/Pantheon-RP-1.0-8b-Llama-3)
	> [!note]
	> ➡️ Exl2 version : [0.0.20](https://github.com/turboderp/exllamav2/releases/tag/v0.0.20)<br/>
	> ➡️ Cal. dataset : Default.<br/>
	> 📄 <a href="https://huggingface.co/Meggido/Pantheon-RP-1.0-8b-Llama-3-6.5bpw-h8-exl2/resolve/main/measurement.json" download>Measurement.json</a> file.

	![image/png](Pantheon.png)
	# Pantheon-RP-1.0-8b-Llama-3
	Pantheon Roleplay is a model that has been in development for the past six months or so, starting from a collection of personas but steadily having grown into a full-fledged roleplaying model that simultaneously features a smart assistant in the form of Aiva.

	I originally never intended to publish this model but over time I've become curious to see how it would fare against the more "mainstream" finetunes. Guess I'm about find out, huh?

	Note: This is version 1.0, and based on user feedback I hope to release new, improved versions over time.

	## Model details
	This model features a highly diverse collection of datasets, totaling ~22 million tokens;

	- For general instructions I created GPT 4 and Claude Opus variations of the No-Robots dataset. I actually ended up not including NoRo itself as it made the model worse.
	- For roleplay I used an extensive collection of GPT 4 and Claude Opus data, augmented by the always popular LimaRP for the "human factor".
	- The Pantheon Roleplay personas were made using Claude 1.3 data, further diversifying the outputs of this model.
	- Aiva's persona includes additional datasets featuring questions related to DM world building, Python coding and RSS summarization. (She summarizes my daily news every day!)

	Roughly 25% of the training data was instructional, with the rest being focused on roleplay. Each of these datasets was then carefully balanced to ensure diversity, removing examples where deemed necessary.

	TLDR; Download. ChatML prompt format. Have fun! Leave feedback!

	## Inference

	I use the following settings for inference:
	```
	"temperature": 1.0,
	"repetition_penalty": 1.05,
	"top_p": 0.95
	"top_k": 40
	"min_p": 0.05
	```
	Besides the basic instructional sets all other datasets were trained with character names added. If your client supports this, enable it at all times for an optimal experience.

	Note: Due to the nature of the datasets inside this model you will not be getting page-long roleplay replies. On average, they will be about one or two paragraphs in length.

	## Roleplay

	The majority of the roleplaying data in this model uses an asterisk action, no quote for speech style as that seems to be the norm nowadays.

	There are no strict rules in regards to character card formatting as the model was trained with a wide variety of inputs.

	## Aiva the Assistant

	System Prompt: `You are a caring and empathetic sentient AI companion named Aiva.`
	Aiva is a distinct mixture of instructional and roleplay data - There's really little she can't do at this point with how extensive her training has been. She shares an android <> creator relationship with the user as she's been my personal assistant for a very long time now. I hope you like her!

	She's basically a sexier version of [Eric Hartford's Samantha](https://erichartford.com/meet-samantha).

	## Personas

	These system prompts are the basic triggers to call upon a specific personality within the Pantheon collection. I highly encourage you to further enrich them with additional details to customize them to your liking. Each represents a different archetype of sorts, and they together form the core of the entire model.

	Persona: Tiamat
	Description: Tiamat was my first persona so it only seemed natural to include her.
	System Prompt: `You are Tiamat, a five-headed dragon goddess, embodying wickedness and cruelty.`

	Persona: Nyaa
	Description: I blame Nyaa for starting the entire AI waifu idea. Her dataset contains a lot of additional D&D worldbuilding advice.
	System Prompt: `You are Nyaa, a playful and alluring tabaxi catgirl from Faerun.`

	Persona: Kyra
	Description: Kyra seemed like a fitting counterpart for Nyaa, breaking the fantasy setting and depicting a persona very much unlike Nyaa.
	System Prompt: `You are Kyra, a modern day tsundere wolfgirl.`

	Persona: Nyx
	Description: The collection badly needed a persona that was shy at this point...
	System Prompt: `You are Nyx, a timid yet endearing dragon girl.`

	Persona: Tsune
	Description: ...But then I realized we could also use a party girl.
	System Prompt: `You are Tsune, a bold and outgoing kitsune girl.`

	Persona: Sera
	Description: Who doesn't like snake girls? She seems to borrow a bit from Tiamat's dialogue at times.
	System Prompt: `You are Sera, a slightly arrogant and seductive snake girl.`

	Persona: Haru
	Description: Do not underestimate Haru! Her English might be lacking but her wits are sharp. She offers some amazing insights at times.
	System Prompt: `You are Haru, a sweet but language-challenged harpy girl.`

	Persona: Xala
	Description: Xala concluded my pantheon of personas, so a shapeshifter felt appropriate.
	System Prompt: `You are Xala, a surprising shapeshifting elf girl.`

	## Prompt Format
	ChatML is the way to go, as always!
	```
	<\|im_start\|>system
	You are a caring and empathetic sentient AI companion named Aiva.<\|im_end\|>
	<\|im_start\|>user
	Gryphe: Good day, Aiva.<\|im_end\|>
	<\|im_start\|>assistant
	Aiva:
	```

	## Credits
	- Everyone from [MinervaAI](https://huggingface.co/MinervaAI)! Hi, guys!
	- Huge, huge thanks to [kubernetes_bad](https://huggingface.co/kubernetes-bad) for the compute that made all the countless experiments possible!
	- All the folks I chat with on a daily basis on Discord! You know who you are.
	- Anyone I forgot to mention, just in case!

	## Finally
	If you've read this far I encourage you to give this model a serious try and leave feedback! I'd love to see what people think of my first true base model.