Instructions to use LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2")
model = AutoModelForCausalLM.from_pretrained("LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2

SGLang

How to use LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2 with Docker Model Runner:
```
docker model run hf.co/LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2
```

Norobara-ZLoss-8x7B

This is an experimental instruct-tuned mistralai/Mixtral-8x7B-v0.1-based model trained using Charles Goddard's ZLoss and Megablocks-based fork of transformers.

It primarily uses the Capybara and No Robots datasets (thus the name). The goal was to create an uncensored general instruction following model, as well as test various loss implementations while we figure out how the heck to train Mixtral properly.

Exl2 Quants

Usage:

The intended prompt format is a modified multi-turn Alpaca instruction format:

### Instruction:
{system prompt}

### Input:
{user message}

### Response:
{model response}

### Input:
{user message}

### Response:
{model response}

(etc.)

Bias, Risks, and Limitations

The model will show biases present in the base model. No ethical alignment was applied to prevent the generation of toxic or harmful outputs (in fact the opposite, with examples from toxic-DPO included), so generate at your own risk.

Training Details

This model was trained as a QLora adapter for 3 epochs using a single H100 GPU for around 13 hours.

Downloads last month: 13

LoneStriker
/

Norobara-ZLoss-8x7B-3.5bpw-h6-exl2

Norobara-ZLoss-8x7B

Usage:

Bias, Risks, and Limitations

Training Details

Datasets used to train LoneStriker/Norobara-ZLoss-8x7B-3.5bpw-h6-exl2