README.md

metadata

base_model:
  - meta-llama/Llama-2-70b-hf
base_model_relation: quantized
license: llama2

Model Card

• Base model: meta-llama/Llama-2-70b-hf
• Quantization method: BlockLDLQ with GuidedQuant Hessian
• Target bit-width: 3
• Backend kernel: QTIP kernel (HYB variant)
• Calibration data: RedPajama (1024 sentences / 4096 tokens)
• Calibration objective: Next-token prediction
• num_groups (for GuidedQuant Hessian): 2

How to run

• Follow the instruction in https://github.com/snu-mllab/GuidedQuant and https://github.com/Cornell-RelaxML/qtip

References

• Model Paper