Instructions to use TheBloke/Llama2-70B-OASST-SFT-v10-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheBloke/Llama2-70B-OASST-SFT-v10-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="TheBloke/Llama2-70B-OASST-SFT-v10-GGUF")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("TheBloke/Llama2-70B-OASST-SFT-v10-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use TheBloke/Llama2-70B-OASST-SFT-v10-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use TheBloke/Llama2-70B-OASST-SFT-v10-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheBloke/Llama2-70B-OASST-SFT-v10-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Llama2-70B-OASST-SFT-v10-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M

SGLang

How to use TheBloke/Llama2-70B-OASST-SFT-v10-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "TheBloke/Llama2-70B-OASST-SFT-v10-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Llama2-70B-OASST-SFT-v10-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "TheBloke/Llama2-70B-OASST-SFT-v10-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheBloke/Llama2-70B-OASST-SFT-v10-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use TheBloke/Llama2-70B-OASST-SFT-v10-GGUF with Ollama:
```
ollama run hf.co/TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use TheBloke/Llama2-70B-OASST-SFT-v10-GGUF with Docker Model Runner:
```
docker model run hf.co/TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M
```

Lemonade

How to use TheBloke/Llama2-70B-OASST-SFT-v10-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheBloke/Llama2-70B-OASST-SFT-v10-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Llama2-70B-OASST-SFT-v10-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

tensor 'token_embd.weight' has wrong shape

by rambocoder - opened Sep 21, 2023

Discussion

rambocoder

Sep 21, 2023

When running ./main -ngl 32 -gqa 8 -m /tmp/llama2-70b-oasst-sft-v10.Q8_0.gguf --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant"

error occurs:
error loading model: create_tensor: tensor 'token_embd.weight' has wrong shape; expected 8192, 32007, got 8192, 32128, 1, 1
llama_load_model_from_file: failed to load model
llama_init_from_gpt_params: error: failed to load model '/tmp/llama2-70b-oasst-sft-v10.Q8_0.gguf'
main: error: unable to load model

Any ideas how to get around this error?

rambocoder

Sep 21, 2023

The GGML version has these attributes:
llama_model_load_internal: n_vocab = 32007
llama_model_load_internal: n_embd = 8192

TheBloke

Owner Sep 21, 2023

Ah I know what this is. Andreas did something to the model to pad it to 128 tokens, but the result is the model is actually broken. I had the same issue when I made the AWQ the other day - I had to go back to an earlier commit on the source model, before he did that padding. See here: https://huggingface.co/OpenAssistant/llama2-70b-oasst-sft-v10/discussions/5

I'll have to re-make the GGUF as well I guess

rambocoder

Sep 21, 2023

This comment has been hidden

anjakuzev

Sep 22, 2023

•

edited Sep 22, 2023

How can i convert a finetuned model with LoRA to GGUF?

when i try with the convert.py i get this error:
Loading model file lora/adapter_model.bin
Traceback (most recent call last):
File "convert.py", line 1208, in
main()
File "convert.py", line 1157, in main
params = Params.load(model_plus)
File "convert.py", line 292, in load
params = Params.guessed(model_plus.model)
File "convert.py", line 166, in guessed
n_vocab, n_embd = model["model.embed_tokens.weight"].shape if "model.embed_tokens.weight" in model else model["tok_embeddings.weight"].shape
KeyError: 'tok_embeddings.weight

vmirea

Oct 25, 2023

•

edited Oct 25, 2023

How can i convert a finetuned model with LoRA to GGUF?

when i try with the convert.py i get this error:
Loading model file lora/adapter_model.bin
Traceback (most recent call last):
File "convert.py", line 1208, in
main()
File "convert.py", line 1157, in main
params = Params.load(model_plus)
File "convert.py", line 292, in load
params = Params.guessed(model_plus.model)
File "convert.py", line 166, in guessed
n_vocab, n_embd = model["model.embed_tokens.weight"].shape if "model.embed_tokens.weight" in model else model["tok_embeddings.weight"].shape
KeyError: 'tok_embeddings.weight

Have you succeeded in fixing this? I get the same error on convert.py

newsletter

Nov 13, 2023

It is still not usable, apparently: Using llama2-70b-oasst-sft-v10.Q5_K_M.gguf in oobabooga/text-generation-webui fails with the following error:
error loading model: create_tensor: tensor 'token_embd.weight' has wrong shape; expected 8192, 32007, got 8192, 32128, 1, 1

Is there any way to fix this?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment