donghyunli/Llama-2-13b-KronQ-W2A16-g128

Llama-2-13b — KronQ W2A16 g128 (packed int2)

Paper: arXiv:2607.07964 · Code: GitHub

Llama-2-13b quantized to 2-bit weights / 16-bit activations, group size 128, with KronQ (Kronecker-factored Hessian quantization). Packed int2 (~4.0 GB); a fused dequant + bidirectional-incoherence (BiIP) CUDA kernel with per-group scales unpacks on the fly.

Results (WikiText-2, seqlen 2048)

Perplexity: 6.508

Zero-shot accuracy:

PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	OBQA	Average
72.58	68.86	38.14	63.88	65.98	66.02	36.80	58.89

(lm-evaluation-harness 0-shot; acc_norm for PIQA/HellaSwag/ARC, acc for WinoGrande/BoolQ.)

Usage

python eval_pretrained.py meta-llama/Llama-2-13b-hf donghyunli/Llama-2-13b-KronQ-W2A16-g128 --ppl --zs

Recipe

Group-128 asymmetric W2, weight-only (a_bits=16), --alpha 0.25, --act_order, BiIP, raw H_G. 128 WikiText-2 calibration sequences.

License

Derivative of Llama-2-13b — llama2 license.