mirai-community/granite-3.3-2b-instruct-int4-gram

Granite-3.3-2b-instruct, int4 (Hessian-aware rounding, one-sided Gram)

int4 weight-quantized ibm-granite/granite-3.3-2b-instruct in lalamo format, runs in lalamo and on-device in uzu.

Every linear is rounded against its own input activation Gram (X^T X), i.e. GPTQ / LDLQ-grade curvature, through lalamo's yaqa_round_blockwise.

This is not true YAQA. The preconditioner is an activation Gram, not a gradient Fisher, and only the input block is used (the output block is identity). The name matters because the recipe is the point: everything below is reproducible from the base model.

Results

MMLU-Pro, canonical TIGER-Lab 5-shot CoT, greedy, paired, full set n=12032, both rows generated through the same engine (lalamo reply_many):

checkpoint	MMLU-Pro	Δ vs bf16
bf16 (reference)	33.55%	—
int4, one-sided Gram (this checkpoint)	31.24%	−2.31 pt
int4, quant-aware distillation (for contrast)	29.38%	−4.17 pt

wikitext held-out PPL (1024 tokens): bf16 5.162 · this 5.457 · int4 min/max PTQ 5.396 · int4 QAD 4.860.

Note the PPL/MMLU inversion: this checkpoint's LM perplexity is slightly worse than naive min/max int4, yet its downstream accuracy is much better (on a paired n=1000 subset: this 33.3 vs min/max PTQ 30.0, bf16 34.0). Perplexity is a poor selector between int4 rounding methods here; the reasoning benchmark separates them.

Recipe

field	value
method	int4 weight-only, Hessian-aware rounding (LDL, GPTQ-grade)
bits / group size	4 / 64, asymmetric, layout OUTPUT_INPUT
preconditioner	per-linear input activation Gram X^T X, one-sided (output block = identity). Not a Fisher.
damping	0.01 * mean(diag(G)) * I
quantized layers	all 160 decoder linears (qkv, out, gate/up, down); embeddings and lm_head left unquantized (F32)
calibration	wikitext-2-raw-v1 (train), 12288 tokens, accumulated over 24 chunks × 512 tokens
calibration rank	full-rank by construction: 12288 calib tokens > 8192, the largest in-dim (down_projection)
RHT / incoherence processing	none
base model	ibm-granite/granite-3.3-2b-instruct @ 707f574c62054322f6b5b04b6d075f0a8f05e0f0
lalamo	124849b (pkg 0.13.1)

Reproduce

python make_int_checkpoint.py \
  models/granite-3.3-2b-instruct models/granite-3.3-2b-instruct-int4-gram \
  --yaqa --calib-file calib_wikitext.txt --calib-tokens 12288 --chunk-len 512 \
  --bits 4 --group-size 64

make_int_checkpoint.py is a thin driver over lalamo.compressed: it accumulates the per-linear input Gram over the calibration chunks and passes it as the preconditioner to IntSpec.compress. down/out inputs are recovered by re-running each layer's own mlp/mixer with the trailing projection swapped for an identity, so no core lalamo changes are needed.

Caveats

• Calibration rank is load-bearing. Below full rank (calib tokens < in-dim) the Gram is rank-deficient, rounding in its null space is unconstrained, and the checkpoint collapses (we measured PPL 57487 at 242 calib tokens). Keep calibration tokens above 8192 for this model.
• Evaluated with the public TIGER-Lab MMLU-Pro protocol, not Mirai's internal reasoning-enabled harness, so the delta vs bf16 is the transferable number, not the absolute score.