KronQ: LLM Quantization via Kronecker-Factored Hessian
Paper • 2607.07964 • Published • 33
Llama-2-13b — KronQ W4A16 g128 (packed int4)
Paper: arXiv:2607.07964 · Code: GitHub
Llama-2-13b quantized to 4-bit weights / 16-bit activations, group size 128, with KronQ (Kronecker-factored Hessian quantization). Packed int4 (~7.0 GB); a fused dequant + bidirectional-incoherence (BiIP) CUDA kernel with per-group scales unpacks on the fly.
Results (WikiText-2, seqlen 2048)
Perplexity: 4.935
Zero-shot accuracy:
| PIQA | ARC-E | ARC-C | HellaSwag | WinoGrande | BoolQ | OBQA | Average |
|---|---|---|---|---|---|---|---|
| 80.25 | 75.72 | 48.63 | 78.96 | 72.06 | 80.76 | 45.00 | 68.77 |
(lm-evaluation-harness 0-shot; acc_norm for PIQA/HellaSwag/ARC, acc for WinoGrande/BoolQ.)
Usage
python eval_pretrained.py meta-llama/Llama-2-13b-hf donghyunli/Llama-2-13b-KronQ-W4A16-g128 --ppl --zs
Recipe
Group-128 asymmetric W4, weight-only (a_bits=16), --alpha 0.25, --act_order, BiIP, raw H_G. 128 WikiText-2 calibration sequences.
License
Derivative of Llama-2-13b — llama2 license.
- Downloads last month
- 136
Model tree for donghyunli/Llama-2-13b-KronQ-W4A16-g128
Base model
meta-llama/Llama-2-13b-hf