# Orpheus-3b-0.1 AWQ 4-bit Quantized

This repository contains a 4-bit AWQ quantized version of [canopylabs/orpheus-3b-0.1-pretrained](https://huggingface.co/canopylabs/orpheus-3b-0.1-pretrained).

## Model Details

- **Original Model**: [canopylabs/orpheus-3b-0.1-pretrained](https://huggingface.co/canopylabs/orpheus-3b-0.1-pretrained)
- **Quantization Method**: AWQ (Activation-aware Weight Quantization)
- **Bit Precision**: 4-bit
- **Group Size**: 128
- **Zero Point**: Enabled
- **Version**: GEMM

## Usage

You can load this model using the AutoAWQ library:

```python
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "chatboo/orpheus-3b-0.1-awq-4bit"
model = AutoAWQForCausalLM.from_quantized(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

# Example usage
inputs = tokenizer("Hello, how are you?", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Quantization Configuration

The model was quantized with the following configuration:

```python
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM"
}
```

## Benefits of Quantization

- **Reduced Size**: The model size is significantly reduced compared to the original.
- **Memory Efficiency**: Lower memory requirements for inference.
- **Faster Inference**: Potentially faster inference especially on hardware with int4 acceleration.
- **Similar Performance**: Maintains most of the quality of the original model.