---
---
license: mit
language:
- en
base_model:
- Minthy/ToriiGate-0.5
base_model_relation: quantized
quantization: exl3
library_name: exllamav3
tags:
- multimodal
- vision
- image-text-to-text
---
Quantization was performed using [exllamav3 v0.0.28](https://github.com/turboderp-org/exllamav3) (commit `ea87af6`).

| Quant | Size (GB) | Actual bpw | PPL | KL-div (q→o) | KL-div (o→q) | Top-1 | Top-2 | Top-3 | Top-4 | Top-5 |
|---|---|---|---|---|---|---|---|---|---|---|
| [4.0bpw](https://huggingface.co/NeuroSenko/ToriiGate-0.5-exl3/tree/4.0bpw) | 3.94 | 4.00 | 29.100 | 0.0150 | 0.0150 | 93.1% | 80.3% | 64.8% | 49.4% | 35.8% |
| [5.0bpw](https://huggingface.co/NeuroSenko/ToriiGate-0.5-exl3/tree/5.0bpw) | 4.51 | 5.00 | 28.854 | 0.0042 | 0.0042 | 96.2% | 88.6% | 78.6% | 67.2% | 55.8% |
| [6.0bpw](https://huggingface.co/NeuroSenko/ToriiGate-0.5-exl3/tree/6.0bpw) | 4.92 | 6.00 | 28.666 | 0.0013 | 0.0013 | 97.9% | 93.7% | 87.6% | 80.1% | 71.8% |
| [7.0bpw](https://huggingface.co/NeuroSenko/ToriiGate-0.5-exl3/tree/7.0bpw) | 5.34 | 7.00 | 28.610 | 0.0004 | 0.0004 | 98.7% | 96.0% | 92.2% | 87.2% | 81.4% |
| [8.0bpw](https://huggingface.co/NeuroSenko/ToriiGate-0.5-exl3/tree/8.0bpw) | 5.75 | 8.00 | 28.621 | 0.0002 | 0.0002 | 99.1% | 97.2% | 94.4% | 90.8% | 86.4% |
| original | 9.66 | 16.00 | 28.596 | — | — | — | — | — | — | — |

### Metrics
- **PPL** (Perplexity) — how well the model predicts the next token. Lower is better. The original model's PPL is the baseline.
- **KL-div** (Kullback-Leibler divergence) — measures how the quant's probability distribution differs from the original. Lower is better. Shown in both directions (quant→orig, orig→quant); asymmetry indicates where the quant over/under-estimates probabilities.
- **Top-K agreement** — probability that the quant's top-K predicted tokens match the original's top-K. Higher is better. Top-1 is the most important (does the quant pick the same best token?), higher K values show agreement across less likely candidates.

### Example

4.0bpw performance example using [gradio script](https://huggingface.co/Minthy/ToriiGate-0.5/blob/main/scripts/gradio_interface.py) from origin's repo.

![image](https://cdn-uploads.huggingface.co/production/uploads/63cbee15f488db9bb3c1cd4b/wku3bXPMIauzjIhdHODkB.png)