Instructions to use art-from-the-machine/Mantella-Skyrim-Llama-3-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use art-from-the-machine/Mantella-Skyrim-Llama-3-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="art-from-the-machine/Mantella-Skyrim-Llama-3-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("art-from-the-machine/Mantella-Skyrim-Llama-3-8B")
model = AutoModelForCausalLM.from_pretrained("art-from-the-machine/Mantella-Skyrim-Llama-3-8B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use art-from-the-machine/Mantella-Skyrim-Llama-3-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "art-from-the-machine/Mantella-Skyrim-Llama-3-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "art-from-the-machine/Mantella-Skyrim-Llama-3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/art-from-the-machine/Mantella-Skyrim-Llama-3-8B

SGLang

How to use art-from-the-machine/Mantella-Skyrim-Llama-3-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "art-from-the-machine/Mantella-Skyrim-Llama-3-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "art-from-the-machine/Mantella-Skyrim-Llama-3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "art-from-the-machine/Mantella-Skyrim-Llama-3-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "art-from-the-machine/Mantella-Skyrim-Llama-3-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Studio

How to use art-from-the-machine/Mantella-Skyrim-Llama-3-8B with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for art-from-the-machine/Mantella-Skyrim-Llama-3-8B to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for art-from-the-machine/Mantella-Skyrim-Llama-3-8B to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for art-from-the-machine/Mantella-Skyrim-Llama-3-8B to start chatting

Load model with FastModel

pip install unsloth
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained(
    model_name="art-from-the-machine/Mantella-Skyrim-Llama-3-8B",
    max_seq_length=2048,
)

Docker Model Runner
How to use art-from-the-machine/Mantella-Skyrim-Llama-3-8B with Docker Model Runner:
```
docker model run hf.co/art-from-the-machine/Mantella-Skyrim-Llama-3-8B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Skyrim Fine-Tune for Meta Llama 3 8B Instruct

GGUF models: https://huggingface.co/art-from-the-machine/Mantella-Skyrim-Llama-3-8B-GGUF

Introduction

This fine-tune of Meta's Llama 3 8B Instruct model is intended to be used as a custom-tailored model for the Mantella Skyrim mod.

Dataset

An Alpaca-style dataset of 8,800+ player <-> NPC interactions has been used in this fine-tune. Details on this dataset can be found in the Mantella LLM Fine-Tuning repo.

Example:

{
    "instruction": "Generate dialogue in the style of Skyrim.",
    "input": "Where can I find fire salts?",
    "output": "A flame atronach's body might provide fire salt. They're dangerous creatures that can be summoned by wizards. Of course, it would be much easier to check with an alchemist. They occasionally have them for sale."
}

Specifications

Developed by: art-from-the-machine
License: apache-2.0
Fine-tuned from model: unsloth/llama-3-8b-Instruct-bnb-4bit
Context length: 8,192

Example Output

The fine-tuned Mantella-Skyrim-Llama-3-8B-Q4_K_M.gguf model has been tested in Mantella v0.11.2's debug mode, with the model running locally in koboldcpp. The character tested is Hulda. A list of questions are tested against both the fine-tuned model as well as an equivalent Q4_K_M Llama 3 Instruct base model for comparison. For both models, Mantella's max_response_sentences setting has been set to 4.

Mantella-Skyrim-Llama-3-8B-Q4_K_M.gguf

[
    {
        "role": "assistant",
        "content": "Ah, welcome to the Bannered Mare! Take a seat by the fire and I'll have one of the girls bring you a drink. What can I get for you?"
    },
    {
        "role": "user",
        "content": "What's life like in Whiterun?"
    },
    {
        "role": "assistant",
        "content": "It's a good life. The people are friendly, the food is good, and the Jarl is fair. Of course, there are some who grumble about the Empire, but I think they're just jealous of the Jarl's fine wine cellar. What about you? What brings you to Whiterun?"
    },
    {
        "role": "user",
        "content": "What do you think about the civil war in Skyrim?"
    },
    {
        "role": "assistant",
        "content": "I think it's a shame. I've lived in Whiterun all my life, and I've seen the Empire bring peace and prosperity to our land. The Stormcloaks just want to tear everything down and start over. I don't think that's the way to build a better future. But what do I know? I'm just a simple innkeeper."
    },
    {
        "role": "user",
        "content": "I'm looking for work. Got any leads?"
    },
    {
        "role": "assistant",
        "content": "Well, the Companions are always looking for new recruits. They're a bit rough, but they're good people. And of course, there's the Jarl's court. If you're looking for work, you might want to try talking to his steward. He's always looking for someone to do odd jobs for him."
    },
    {
        "role": "user",
        "content": "I'm looking for an Elder Scroll. Do you know anything about them?"
    },
    {
        "role": "assistant",
        "content": "Elder Scrolls? Ha! You must be joking. Those are ancient relics, hidden away in the ruins of old. You won't find one in Whiterun, that's for sure. If you're looking for something like that, you should try talking to the Greybeards. They're the ones who live in the mountains and meditate all day."
    }
]

Meta-Llama-3-8B-Instruct-Q4_K_M.gguf

[
    {
        "role": "assistant",
        "content": "Ah, welcome to The Bannered Mare! Have a seat by the fire, and I'll have one of the barmaids bring you a pint of our finest ale. What brings you to Whiterun today?"
    },
    {
        "role": "user",
        "content": "What's life like in Whiterun?"
    },
    {
        "role": "assistant",
        "content": "Life in Whiterun is good, thank you for asking. The Jarl's been doing a fine job of keeping the city prosperous. Of course, there's always something going on, whether it's the Companions looking for new members or the Thieves Guild trying to... well, steal things. But overall, it's a good place to live."
    },
    {
        "role": "user",
        "content": "What do you think about the civil war in Skyrim?"
    },
    {
        "role": "assistant",
        "content": "The civil war? Well, I think it's a shame, to be honest. I've got friends on both sides of the conflict, and it's hard to see them fighting each other. But I suppose it's not my place to say who's right or wrong. I just hope it ends soon and we can all get back to living our lives in peace."
    },
    {
        "role": "user",
        "content": "I'm looking for work. Got any leads?"
    },
    {
        "role": "assistant",
        "content": "Work, you say? Well, the city's always looking for people to help with the harvest, and the Jarl's guards could use some extra hands. If you're looking for something more... adventurous, you might want to talk to the Companions. They're always looking for new members to join their ranks. Just be warned, though - it's not for the faint of heart."
    },
    {
        "role": "user",
        "content": "I'm looking for an Elder Scroll. Do you know anything about them?"
    },
    {
        "role": "assistant",
        "content": "Elder Scrolls? Ah, yes... I've heard rumors about those ancient artifacts. Some say they hold the secrets of the world, while others claim they're cursed. I don't know much about them, but I've heard that the Greybeards in the mountains might know more. They're a reclusive bunch, but if you're willing to brave the wilderness, you might be able to learn something from them."
    }
]