Instructions to use ockerman0/MN-12B-Starcannon-v5-unofficial with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ockerman0/MN-12B-Starcannon-v5-unofficial with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ockerman0/MN-12B-Starcannon-v5-unofficial")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("ockerman0/MN-12B-Starcannon-v5-unofficial")
model = AutoModelForMultimodalLM.from_pretrained("ockerman0/MN-12B-Starcannon-v5-unofficial")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ockerman0/MN-12B-Starcannon-v5-unofficial with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ockerman0/MN-12B-Starcannon-v5-unofficial"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ockerman0/MN-12B-Starcannon-v5-unofficial",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ockerman0/MN-12B-Starcannon-v5-unofficial

SGLang

How to use ockerman0/MN-12B-Starcannon-v5-unofficial with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ockerman0/MN-12B-Starcannon-v5-unofficial" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ockerman0/MN-12B-Starcannon-v5-unofficial",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ockerman0/MN-12B-Starcannon-v5-unofficial" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ockerman0/MN-12B-Starcannon-v5-unofficial",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ockerman0/MN-12B-Starcannon-v5-unofficial with Docker Model Runner:
```
docker model run hf.co/ockerman0/MN-12B-Starcannon-v5-unofficial
```

Feedback/request

by IZA09 - opened Aug 28, 2024

Discussion

IZA09

Aug 28, 2024

•

edited Aug 28, 2024

Sillytavern (kobold ccp) Q6K

I don't know if i just haven't found the correct settings, but i can only seem to get either beautifully creative prose and emotionless lacking dialogue, or brief, bland prose and beautifully creative dialogue. i cant seem to find an acceptable middle ground. regardless i genuinely love this model if not for the vivid descriptions it can create when the perimeters are tuned (specifically in NSFW). bit in general, there seems to be this "lifelessness" or a sort of disconnect i cant really explain. im coming from the original V3 starcannon where everything and everyone just had this fun liveliness to them like they ARE char and their portrayed personality is near perfect especially with accents, but your V5 feels like the AI knows its not char and is just reciting their lines like an unenthusiastic theater actor, do you know what i mean?

it honestly feels exactly like Nous-Capybara from back in the day. it was surprisingly creative, but it just lacked color.

I love the creativity of this model's prose, mentioning subtle outside aspects into mundane scenes of just watching TV like the sound of rain or the lighting of the room, or even just the mention of dust motes that just really bring color to the scene, and i love the lifelike dialogue of V3... could i request you take V3, and mix it with like 25% of what you used to make this one so vivid in prose? i have no idea if thats how it works lol but V3 would be near perfect if it had just a dash of V5's creativity/intelligence.

ockerman0

Owner Aug 28, 2024

I think I understand what you mean with the unenthusiastic theatre actor part. After using this model for a while the responses can sometimes feel bland and uninspiring. I'll have a mess around and see if I can make anything better. Thanks for the feedback!

ockerman0

Owner Aug 30, 2024

I made a new merge roughly based on what you were thinking, try this one and see if it's any better.

https://huggingface.co/ockerman0/MN-12B-Starcannon-v5.5-unofficial

IZA09

Aug 30, 2024

I made a new merge roughly based on what you were thinking, try this one and see if it's any better.

https://huggingface.co/ockerman0/MN-12B-Starcannon-v5.5-unofficial

you are a saint!. ill mess around with it and provide my thoughts over on that one soon

IZA09

Aug 31, 2024

I made a new merge roughly based on what you were thinking, try this one and see if it's any better.

https://huggingface.co/ockerman0/MN-12B-Starcannon-v5.5-unofficial

you are a saint!. ill mess around with it and provide my thoughts over on that one soon

Whoopsie daisy just realized ill need it quantized lol.

ockerman0

Owner Aug 31, 2024

•

edited Aug 31, 2024

Whoopsie daisy just realized ill need it quantized lol.

My bad, I'll submit a request to Mradermacher

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment