---
base_model:
- meta-llama/Llama-2-70b-hf
license: mit
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
---

# GuidedQuant: Llama-2-70B

This model is a 3-bit quantized version of `meta-llama/Llama-2-70b-hf` using **GuidedQuant**, a novel post-training quantization approach. GuidedQuant integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies. This method consistently boosts the performance of state-of-the-art quantization techniques across various settings.

The model was presented in the paper [**GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance**](https://huggingface.co/papers/2505.07004).

-   **Project Page**: [https://jusjinuk.me/blog/guidedquant/](https://jusjinuk.me/blog/guidedquant/)
-   **Code**: [https://github.com/snu-mllab/GuidedQuant](https://github.com/snu-mllab/GuidedQuant)

## Model Details
-   Base model: `meta-llama/Llama-2-70b-hf`
-   Quantization method: BlockLDLQ with GuidedQuant Hessian
-   Target bit-width: 3
-   Backend kernel: QTIP kernel (HYB variant)
-   Calibration data: RedPajama (1024 sentences / 4096 tokens)
-   Calibration objective: Next-token prediction
-   num_groups (for GuidedQuant Hessian): 2

## Usage

You can easily load and test this quantized model using the `AnyPrecisionForCausalLM` class, which integrates seamlessly with the Hugging Face `transformers` library.

```python
from any_precision.modules.AnyPrecisionForCausalLM import AnyPrecisionForCausalLM
from transformers import AutoTokenizer, TextStreamer
import torch

quantized_model_name = "jusjinuk/Llama-3.3-70B-Instruct-2bit-GuidedQuant-LNQ" # Example model, replace with current model name if different
# Use float16 for Llama models, and bfloat16 for Qwen / Gemma models
dtype = torch.float16 if "llama" in quantized_model_name.lower() else torch.bfloat16

model = AnyPrecisionForCausalLM.from_quantized(quantized_model_name, torch_dtype=dtype)
tokenizer = AutoTokenizer.from_pretrained(quantized_model_name)
streamer = TextStreamer(tokenizer)

prompt = "Write me a short and concise story about Harry, Ron, and Hermione.
"
chat = [
    {"role": "system", "content": "You are a helpful assistant.
"},
    {"role": "user", "content": prompt},
]

inputs = tokenizer.apply_chat_template(
    chat, tokenize=True, return_tensors="pt", add_generation_prompt=True
).to(model.device)

model.generate(inputs, 
    max_new_tokens=200, do_sample=False, temperature=1.0, streamer=streamer, pad_token_id=tokenizer.eos_token_id
)
```

For more comprehensive instructions on installation, advanced usage, and reproduction of results, please refer to the [GuidedQuant GitHub repository](https://github.com/snu-mllab/GuidedQuant) and the [QTIP kernel repository](https://github.com/Cornell-RelaxML/qtip).

## Citation

Please cite our paper if you find our work useful:

```
@inproceedings{kim2025guidedquant,
      title={GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance}, 
      author={Jinuk Kim and Marwa El Halabi and Wonpyo Park and Clemens JS Schaefer and Deokjae Lee and Yeonhong Park and Jae W. Lee and Hyun Oh Song},
      booktitle = {International Conference on Machine Learning (ICML)},
      year={2025},
}
```