Instructions to use astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit")
model = AutoModelForCausalLM.from_pretrained("astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit

SGLang

How to use astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit with Docker Model Runner:
```
docker model run hf.co/astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit
```

This model is generously created and made open source by Astronomer.

Astronomer is the de facto company for Apache Airflow, the most trusted open-source framework for data orchestration and MLOps.

Llama-3-8B-Instruct-GPTQ-8-Bit

Original Model creator: Meta Llama from Meta
Original model: meta-llama/Meta-Llama-3-8B-Instruct
Built with Meta Llama 3
Quantized by Astronomer

Important Note About Serving with vLLM & oobabooga/text-generation-webui

For loading this model onto vLLM, make sure all requests have "stop_token_ids":[128001, 128009] to temporarily address the non-stop generation issue.
- vLLM does not yet respect generation_config.json.
- vLLM team is working on a a fix for this https://github.com/vllm-project/vllm/issues/4180
For oobabooga/text-generation-webui
- Load the model via AutoGPTQ, with no_inject_fused_attention enabled. This is a bug with AutoGPTQ library.
- Under Parameters -> Generation -> Skip special tokens: turn this off (deselect)
- Under Parameters -> Generation -> Custom stopping strings: add "<|end_of_text|>","<|eot_id|>" to the field

Description

This repo contains 8 Bit quantized GPTQ model files for meta-llama/Meta-Llama-3-8B-Instruct.

This model can be loaded with just over 10GB of VRAM (compared to the original 16.07GB model) and can be served lightning fast with the cheapest Nvidia GPUs possible (Nvidia T4, Nvidia K80, RTX 4070, etc).

The 8 bit GPTQ quant has minimum quality degradation from the original bfloat16 model due to its higher bitrate.

GPTQ Quantization Method

This model is quantized by utilizing the AutoGPTQ library, following best practices noted by GPTQ paper
Quantization is calibrated and aligned with random samples from the specified dataset (wikitext for now) for minimum accuracy loss.

Branch	Bits	Group Size	Act Order	Damp %	GPTQ Dataset	Sequence Length	VRAM Size	ExLlama	Description
main	8	32	Yes	0.1	wikitext	8192	9.09 GB	No	8-bit, with Act Order and group size 32g. Minimum accuracy loss with decent VRAM usage reduction.
More variants to come	TBD	TBD	TBD	TBD	TBD	TBD	TBD	TBD	May upload additional variants of GPTQ 8 bit models in the future using different parameters such as 128g group size and etc.

Serving this GPTQ model using vLLM

Tested serving this model via vLLM using an Nvidia T4 (16GB VRAM).

Tested with the below command

python -m vllm.entrypoints.openai.api_server --model astronomer-io/Llama-3-8B-Instruct-GPTQ-8-Bit --max-model-len 8192 --dtype float16

For the non-stop token generation bug, make sure to send requests with stop_token_ids":[128001, 128009] to vLLM endpoint Example:

{
    "model": "astronomer-io/Llama-3-8B-Instruct-GPTQ-8-Bit",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who created Llama 3?"}
        ],
    "max_tokens": 2000,
    "stop_token_ids":[128001,128009]
}

Prompt Template

<|begin_of_text|><|start_header_id|>user<|end_header_id|>
{{prompt}}<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

Contributors

Quantized by David Xue, Machine Learning Engineer from Astronomer

Downloads last month: 21

Safetensors

Model size

8B params

Tensor type

I32

BF16

Model tree for astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit

Base model

meta-llama/Meta-Llama-3-8B-Instruct

Quantized

(272)

this model

Adapters

1 model

Quantizations

1 model

Dataset used to train astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit

Collection including astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit

Llama-3 GPTQ Quantized Models

Collection

A collection of Llama 3 models quantized by Astronomer • 4 items • Updated Apr 21, 2024 • 2

Paper for astronomer/Llama-3-8B-Instruct-GPTQ-8-Bit

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Paper • 2210.17323 • Published Oct 31, 2022 • 10