Improve model card: Add pipeline tag, library, license, paper, project page, and usage
Browse filesThis PR improves the model card for the `Llama-2-70b-hf-3bit-GuidedQuant-QTIP` model by adding key metadata and enhancing its content:
- Adds `pipeline_tag: text-generation` for better discoverability and to enable the text generation inference widget.
- Adds `library_name: transformers` to correctly identify the library used for loading and interacting with the model, ensuring better integration with the Hugging Face ecosystem.
- Corrects the license from `llama2` to `mit`, aligning with the explicit license stated in the original GuidedQuant GitHub repository for the project's artifacts.
- Updates the paper link to the official Hugging Face Papers page: [GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance](https://huggingface.co/papers/2505.07004).
- Adds a direct link to the project page: [https://jusjinuk.me/blog/guidedquant/](https://jusjinuk.me/blog/guidedquant/).
- Includes a clear Python usage example for quick inference, derived from the project's quick start guide.
|
@@ -1,21 +1,75 @@
|
|
| 1 |
---
|
| 2 |
base_model:
|
| 3 |
- meta-llama/Llama-2-70b-hf
|
|
|
|
| 4 |
base_model_relation: quantized
|
| 5 |
-
|
|
|
|
| 6 |
---
|
| 7 |
-
# Model Card
|
| 8 |
|
| 9 |
-
|
| 10 |
-
- Quantization method: BlockLDLQ with GuidedQuant Hessian
|
| 11 |
-
- Target bit-width: 3
|
| 12 |
-
- Backend kernel: QTIP kernel (HYB variant)
|
| 13 |
-
- Calibration data: RedPajama (1024 sentences / 4096 tokens)
|
| 14 |
-
- Calibration objective: Next-token prediction
|
| 15 |
-
- num_groups (for GuidedQuant Hessian): 2
|
| 16 |
|
| 17 |
-
|
| 18 |
-
- Follow the instruction in https://github.com/snu-mllab/GuidedQuant and https://github.com/Cornell-RelaxML/qtip
|
| 19 |
|
| 20 |
-
|
| 21 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
base_model:
|
| 3 |
- meta-llama/Llama-2-70b-hf
|
| 4 |
+
license: mit
|
| 5 |
base_model_relation: quantized
|
| 6 |
+
pipeline_tag: text-generation
|
| 7 |
+
library_name: transformers
|
| 8 |
---
|
|
|
|
| 9 |
|
| 10 |
+
# GuidedQuant: Llama-2-70B
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
|
| 12 |
+
This model is a 3-bit quantized version of `meta-llama/Llama-2-70b-hf` using **GuidedQuant**, a novel post-training quantization approach. GuidedQuant integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies. This method consistently boosts the performance of state-of-the-art quantization techniques across various settings.
|
|
|
|
| 13 |
|
| 14 |
+
The model was presented in the paper [**GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance**](https://huggingface.co/papers/2505.07004).
|
| 15 |
+
|
| 16 |
+
- **Project Page**: [https://jusjinuk.me/blog/guidedquant/](https://jusjinuk.me/blog/guidedquant/)
|
| 17 |
+
- **Code**: [https://github.com/snu-mllab/GuidedQuant](https://github.com/snu-mllab/GuidedQuant)
|
| 18 |
+
|
| 19 |
+
## Model Details
|
| 20 |
+
- Base model: `meta-llama/Llama-2-70b-hf`
|
| 21 |
+
- Quantization method: BlockLDLQ with GuidedQuant Hessian
|
| 22 |
+
- Target bit-width: 3
|
| 23 |
+
- Backend kernel: QTIP kernel (HYB variant)
|
| 24 |
+
- Calibration data: RedPajama (1024 sentences / 4096 tokens)
|
| 25 |
+
- Calibration objective: Next-token prediction
|
| 26 |
+
- num_groups (for GuidedQuant Hessian): 2
|
| 27 |
+
|
| 28 |
+
## Usage
|
| 29 |
+
|
| 30 |
+
You can easily load and test this quantized model using the `AnyPrecisionForCausalLM` class, which integrates seamlessly with the Hugging Face `transformers` library.
|
| 31 |
+
|
| 32 |
+
```python
|
| 33 |
+
from any_precision.modules.AnyPrecisionForCausalLM import AnyPrecisionForCausalLM
|
| 34 |
+
from transformers import AutoTokenizer, TextStreamer
|
| 35 |
+
import torch
|
| 36 |
+
|
| 37 |
+
quantized_model_name = "jusjinuk/Llama-3.3-70B-Instruct-2bit-GuidedQuant-LNQ" # Example model, replace with current model name if different
|
| 38 |
+
# Use float16 for Llama models, and bfloat16 for Qwen / Gemma models
|
| 39 |
+
dtype = torch.float16 if "llama" in quantized_model_name.lower() else torch.bfloat16
|
| 40 |
+
|
| 41 |
+
model = AnyPrecisionForCausalLM.from_quantized(quantized_model_name, torch_dtype=dtype)
|
| 42 |
+
tokenizer = AutoTokenizer.from_pretrained(quantized_model_name)
|
| 43 |
+
streamer = TextStreamer(tokenizer)
|
| 44 |
+
|
| 45 |
+
prompt = "Write me a short and concise story about Harry, Ron, and Hermione.
|
| 46 |
+
"
|
| 47 |
+
chat = [
|
| 48 |
+
{"role": "system", "content": "You are a helpful assistant.
|
| 49 |
+
"},
|
| 50 |
+
{"role": "user", "content": prompt},
|
| 51 |
+
]
|
| 52 |
+
|
| 53 |
+
inputs = tokenizer.apply_chat_template(
|
| 54 |
+
chat, tokenize=True, return_tensors="pt", add_generation_prompt=True
|
| 55 |
+
).to(model.device)
|
| 56 |
+
|
| 57 |
+
model.generate(inputs,
|
| 58 |
+
max_new_tokens=200, do_sample=False, temperature=1.0, streamer=streamer, pad_token_id=tokenizer.eos_token_id
|
| 59 |
+
)
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
For more comprehensive instructions on installation, advanced usage, and reproduction of results, please refer to the [GuidedQuant GitHub repository](https://github.com/snu-mllab/GuidedQuant) and the [QTIP kernel repository](https://github.com/Cornell-RelaxML/qtip).
|
| 63 |
+
|
| 64 |
+
## Citation
|
| 65 |
+
|
| 66 |
+
Please cite our paper if you find our work useful:
|
| 67 |
+
|
| 68 |
+
```
|
| 69 |
+
@inproceedings{kim2025guidedquant,
|
| 70 |
+
title={GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance},
|
| 71 |
+
author={Jinuk Kim and Marwa El Halabi and Wonpyo Park and Clemens JS Schaefer and Deokjae Lee and Yeonhong Park and Jae W. Lee and Hyun Oh Song},
|
| 72 |
+
booktitle = {International Conference on Machine Learning (ICML)},
|
| 73 |
+
year={2025},
|
| 74 |
+
}
|
| 75 |
+
```
|