---
base_model:
- meta-llama/Llama-2-70b-hf
license: llama2
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
---

# GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance

This repository contains a quantized version of `meta-llama/Llama-2-70b-hf` using the **GuidedQuant** method. GuidedQuant is a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. It consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, it introduces a novel non-uniform scalar quantization algorithm, **LNQ**, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category.

- **Paper**: [GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance](https://arxiv.org/abs/2505.07004)
- **Project Page**: [https://jusjinuk.me/blog/guidedquant/](https://jusjinuk.me/blog/guidedquant/)
- **Code**: [https://github.com/snu-mllab/GuidedQuant](https://github.com/snu-mllab/GuidedQuant)

### Model Details

- **Base model**: `meta-llama/Llama-2-70b-hf`
- **Quantization method**: SqueezeLLM (with GuidedQuant enhancement)
- **Target bit-width**: 4
- **Backend kernel**: Any-Precision-LLM kernel (`ap-gemv`)
- **Calibration data**: RedPajama (1024 sentences / 4096 tokens)
- **Calibration objective**: Next-token prediction

## How to Use

You can easily load and test this quantized model using the `AnyPrecisionForCausalLM` class, as shown in the following example (runs on one RTX 3090).

```python
from any_precision.modules.AnyPrecisionForCausalLM import AnyPrecisionForCausalLM
from transformers import AutoTokenizer, TextStreamer
import torch

# This specific model within the repository (adjust if the model ID on the Hub is different):
quantized_model_name = "jusjinuk/Llama-2-70b-hf-4bit-guidedquant-lnq"
# Use float16 for Llama models, and bfloat16 for Qwen / Gemma models
dtype = torch.float16 if "llama" in quantized_model_name.lower() else torch.bfloat16

model = AnyPrecisionForCausalLM.from_quantized(quantized_model_name, torch_dtype=dtype)
tokenizer = AutoTokenizer.from_pretrained(quantized_model_name)
streamer = TextStreamer(tokenizer)

prompt = "Write me a short and concise story about Harry, Ron, and Hermione.
"
chat = [
    {"role": "system", "content": "You are a helpful assistant.
"},
    {"role": "user", "content": prompt},
]

inputs = tokenizer.apply_chat_template(
    chat, tokenize=True, return_tensors="pt", add_generation_prompt=True
).to(model.device)

model.generate(inputs,
    max_new_tokens=200, do_sample=False, temperature=1.0, streamer=streamer, pad_token_id=tokenizer.eos_token_id
)
```

For detailed installation instructions and advanced usage (e.g., inference speed-up, different quantization types, evaluation), please refer to the [official GitHub repository](https://github.com/snu-mllab/GuidedQuant).

## Citation

Please cite our paper if you find our work useful:

```bibtex
@inproceedings{kim2025guidedquant,
      title={GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance},
      author={Jinuk Kim and Marwa El Halabi and Wonpyo Park and Clemens JS Schaefer and Deokjae Lee and Yeonhong Park and Jae W. Lee and Hyun Oh Song},
      booktitle = {International Conference on Machine Learning (ICML)},
      year={2025},
}
```