nielsr HF Staff commited on
Commit
be3a9da
·
verified ·
1 Parent(s): 8ebc3e1

Improve model card: Add pipeline tag, library, license, paper, project page, and usage

Browse files

This PR improves the model card for the `Llama-2-70b-hf-3bit-GuidedQuant-QTIP` model by adding key metadata and enhancing its content:

- Adds `pipeline_tag: text-generation` for better discoverability and to enable the text generation inference widget.
- Adds `library_name: transformers` to correctly identify the library used for loading and interacting with the model, ensuring better integration with the Hugging Face ecosystem.
- Corrects the license from `llama2` to `mit`, aligning with the explicit license stated in the original GuidedQuant GitHub repository for the project's artifacts.
- Updates the paper link to the official Hugging Face Papers page: [GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance](https://huggingface.co/papers/2505.07004).
- Adds a direct link to the project page: [https://jusjinuk.me/blog/guidedquant/](https://jusjinuk.me/blog/guidedquant/).
- Includes a clear Python usage example for quick inference, derived from the project's quick start guide.

Files changed (1) hide show
  1. README.md +67 -13
README.md CHANGED
@@ -1,21 +1,75 @@
1
  ---
2
  base_model:
3
  - meta-llama/Llama-2-70b-hf
 
4
  base_model_relation: quantized
5
- license: llama2
 
6
  ---
7
- # Model Card
8
 
9
- - Base model: `meta-llama/Llama-2-70b-hf`
10
- - Quantization method: BlockLDLQ with GuidedQuant Hessian
11
- - Target bit-width: 3
12
- - Backend kernel: QTIP kernel (HYB variant)
13
- - Calibration data: RedPajama (1024 sentences / 4096 tokens)
14
- - Calibration objective: Next-token prediction
15
- - num_groups (for GuidedQuant Hessian): 2
16
 
17
- # How to run
18
- - Follow the instruction in https://github.com/snu-mllab/GuidedQuant and https://github.com/Cornell-RelaxML/qtip
19
 
20
- # References
21
- - [Model Paper](https://arxiv.org/abs/2505.07004)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  base_model:
3
  - meta-llama/Llama-2-70b-hf
4
+ license: mit
5
  base_model_relation: quantized
6
+ pipeline_tag: text-generation
7
+ library_name: transformers
8
  ---
 
9
 
10
+ # GuidedQuant: Llama-2-70B
 
 
 
 
 
 
11
 
12
+ This model is a 3-bit quantized version of `meta-llama/Llama-2-70b-hf` using **GuidedQuant**, a novel post-training quantization approach. GuidedQuant integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies. This method consistently boosts the performance of state-of-the-art quantization techniques across various settings.
 
13
 
14
+ The model was presented in the paper [**GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance**](https://huggingface.co/papers/2505.07004).
15
+
16
+ - **Project Page**: [https://jusjinuk.me/blog/guidedquant/](https://jusjinuk.me/blog/guidedquant/)
17
+ - **Code**: [https://github.com/snu-mllab/GuidedQuant](https://github.com/snu-mllab/GuidedQuant)
18
+
19
+ ## Model Details
20
+ - Base model: `meta-llama/Llama-2-70b-hf`
21
+ - Quantization method: BlockLDLQ with GuidedQuant Hessian
22
+ - Target bit-width: 3
23
+ - Backend kernel: QTIP kernel (HYB variant)
24
+ - Calibration data: RedPajama (1024 sentences / 4096 tokens)
25
+ - Calibration objective: Next-token prediction
26
+ - num_groups (for GuidedQuant Hessian): 2
27
+
28
+ ## Usage
29
+
30
+ You can easily load and test this quantized model using the `AnyPrecisionForCausalLM` class, which integrates seamlessly with the Hugging Face `transformers` library.
31
+
32
+ ```python
33
+ from any_precision.modules.AnyPrecisionForCausalLM import AnyPrecisionForCausalLM
34
+ from transformers import AutoTokenizer, TextStreamer
35
+ import torch
36
+
37
+ quantized_model_name = "jusjinuk/Llama-3.3-70B-Instruct-2bit-GuidedQuant-LNQ" # Example model, replace with current model name if different
38
+ # Use float16 for Llama models, and bfloat16 for Qwen / Gemma models
39
+ dtype = torch.float16 if "llama" in quantized_model_name.lower() else torch.bfloat16
40
+
41
+ model = AnyPrecisionForCausalLM.from_quantized(quantized_model_name, torch_dtype=dtype)
42
+ tokenizer = AutoTokenizer.from_pretrained(quantized_model_name)
43
+ streamer = TextStreamer(tokenizer)
44
+
45
+ prompt = "Write me a short and concise story about Harry, Ron, and Hermione.
46
+ "
47
+ chat = [
48
+ {"role": "system", "content": "You are a helpful assistant.
49
+ "},
50
+ {"role": "user", "content": prompt},
51
+ ]
52
+
53
+ inputs = tokenizer.apply_chat_template(
54
+ chat, tokenize=True, return_tensors="pt", add_generation_prompt=True
55
+ ).to(model.device)
56
+
57
+ model.generate(inputs,
58
+ max_new_tokens=200, do_sample=False, temperature=1.0, streamer=streamer, pad_token_id=tokenizer.eos_token_id
59
+ )
60
+ ```
61
+
62
+ For more comprehensive instructions on installation, advanced usage, and reproduction of results, please refer to the [GuidedQuant GitHub repository](https://github.com/snu-mllab/GuidedQuant) and the [QTIP kernel repository](https://github.com/Cornell-RelaxML/qtip).
63
+
64
+ ## Citation
65
+
66
+ Please cite our paper if you find our work useful:
67
+
68
+ ```
69
+ @inproceedings{kim2025guidedquant,
70
+ title={GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance},
71
+ author={Jinuk Kim and Marwa El Halabi and Wonpyo Park and Clemens JS Schaefer and Deokjae Lee and Yeonhong Park and Jae W. Lee and Hyun Oh Song},
72
+ booktitle = {International Conference on Machine Learning (ICML)},
73
+ year={2025},
74
+ }
75
+ ```