# ADR-0002: Use Affine INT8 for the LM Head

Status: Accepted  
Date: 2026-07-20  
Amends: ADR-0001 for `language_model.lm_head` only

Evidence status: this records a historical candidate-construction decision,
not current release approval. Its calibration and evaluation reused the same
12 samples. The schema-3 gate now requires disjoint calibration and held-out
evaluation data before this decision can approve publication.

## Context

The group sensitivity run showed that the vision encoder must remain BF16, while
the projector and token embedding could be quantized without degrading CER,
digit CER, or table score. Joint validation then exposed a trade-off isolated
group sensitivity could not predict:

| LM-head mode | Mean CER | Digit CER | Decode tok/s | Release result |
|---|---:|---:|---:|---|
| BF16 | 0.231904 | 0.082224 | 271.1 | throughput failed |
| MXFP8 | 0.260170 | 0.076961 | 307.4 | CER failed |
| affine INT8 | 0.237384 | 0.085733 | 301.2 | development calibration passed |

The common BF16 baseline CER was 0.239188 and digit CER was 0.087340. The
Sahil-reference throughput was 312.7 tok/s. The selection procedure used the
pre-existing quality and 90% throughput limits; thresholds were not changed
after observing the results.

## Decision

Use MXFP8, group size 32, for 119 eligible modules. Use 8-bit affine
quantization, group size 32, for `language_model.lm_head`. Keep the vision
encoder and non-quantizable normalization/routing state in BF16.

`quantization/calibrate_precision.py` selects the fastest candidate that passes
every existing release constraint and records the decision in
`calibration_results.json`. The generated precision map consumes that record,
so the affine exception is reproducible and is not a manual release edit.

## Alternatives considered

### Keep the LM head in BF16

Rejected for this release because measured throughput was 86.7% of the Sahil
reference, below the approved 90% floor.

### Quantize the LM head to MXFP8

Rejected because mean CER degraded by 0.02098 absolute versus BF16, above the
0.01 ceiling, even though throughput passed.

### Lower the throughput or CER threshold

Rejected. Changing a gate after seeing a failing candidate would invalidate the
fail-closed release design.

## Consequences

- The checkpoint remains predominantly MXFP8 but is accurately described as a
  mixed MXFP8/affine8 model.
- The historical same-dataset rerun reached 297.2 tok/s, 95.0% of the Sahil
  reference, while mean CER and digit CER were slightly better than BF16. This
  does not replace a disjoint held-out release evaluation.
- Consumers must use a recent MLX/`mlx-vlm` version that supports per-module
  affine and MXFP8 quantization metadata.