# Agents-A1 imatrix quality evaluation

This is a local sanity benchmark, not a substitute for MMLU, HellaSwag, or human review.
No files were uploaded.

## Setup

- Accuracy benchmark: `eval/quality_multiple_choice.bin`
- KL holdout text: `eval/kl_holdout.txt`
- KL context/chunks: 128 ctx, 2 chunks
- Backend: native llama.cpp, CPU-only (`-dev none -ngl 0`)

## Accuracy Retention

F16 baseline accuracy: 89.5833%

| Variant | Accuracy | Retention vs F16 | Random chance | Status |
|---|---:|---:|---:|---|
| A1-Q2_K-imatrix | 87.5000% | 97.6745% | 25.0000% | passed |
| A1-IQ3_M-imatrix | 89.5833% | 100.0000% | 25.0000% | passed |
| A1-Q3_K_M-imatrix | 89.5833% | 100.0000% | 25.0000% | passed |
| A1-Q4_K_M-imatrix | 87.5000% | 97.6745% | 25.0000% | passed |
| A1-IQ4_XS-imatrix | 87.5000% | 97.6745% | 25.0000% | passed |

## KL Divergence

Baseline logits file: `/Users/oz/Documents/AgentsA1 Quantization/a1-local-quant-pipeline/benchmark_results/quality/baseline_ctx128_chunks2.kld`
Baseline PPL on KL holdout: 13.0194

| Variant | Mean KLD | PPL ratio | PPL delta | Same top p | RMS delta p | Status |
|---|---:|---:|---:|---:|---:|---|
| A1-Q2_K-imatrix | 0.128242 | 1.158739 | 2.0667 | 81.7460% | 8.8070% | passed |
| A1-IQ3_M-imatrix | 0.045312 | 1.001578 | 0.0205 | 92.0630% | 5.0240% | passed |
| A1-Q3_K_M-imatrix | 0.046980 | 0.984125 | -0.2067 | 88.0950% | 5.2290% | passed |
| A1-Q4_K_M-imatrix | 0.015182 | 0.986453 | -0.1764 | 93.6510% | 2.7470% | passed |
| A1-IQ4_XS-imatrix | 0.020185 | 0.998199 | -0.0234 | 92.0630% | 3.1110% | passed |

## Notes

- Accuracy is a local multiple-choice benchmark generated in `eval/quality_multiple_choice.json`.
- Retention is quant accuracy divided by F16 baseline accuracy on the same local benchmark.
- KL divergence compares quant logit distributions to the F16 no-MTP GGUF baseline on `eval/kl_holdout.txt`.
- Smaller KLD is better; higher same-top-p is better.