Instructions to use EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3")
model = AutoModelForCausalLM.from_pretrained("EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3

SGLang

How to use EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3 with Docker Model Runner:
```
docker model run hf.co/EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

(!) early testing shows refusals still, needs more examples

wassname (updated baukit) implementation of the paper: https://www.alignmentforum.org/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction applied to llama3 8b instruct

The Model is meant purely for alignment research and exploration of alignmentforum theory
The Model is provided ""AS IS"" and ""AS AVAILABLE"" without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, title, or non-infringement.
The Provider disclaims all liability for any damages or losses resulting from the use or misuse of the Model, including but not limited to any damages or losses arising from the use of the Model for purposes other than those intended by the Provider.
The Provider does not endorse or condone the use of the Model for any purpose that violates applicable laws, regulations, or ethical standards.
The Provider does not warrant that the Model will meet your specific requirements or that it will be error-free or that it will function without interruption.
You assume all risks associated with the use of the Model, including but not limited to any loss of data, loss of business, or damage to your reputation.

Downloads last month: 8

Safetensors

Model size

8B params

Tensor type

F16

Model tree for EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3

Quantizations

5 models

EdgerunnersArchive
/

Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3

Model tree for EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3

Spaces using EdgerunnersArchive/Llama-3-8B-Instruct-ortho-baukit-toxic-n128-v3 9