Note: DEPRECATED. PLEASE CHECK THE YAML FILE [models/model_targets.yaml](models/model_targets.yaml)

# Model performance and accuracy

Performance and token accuracy using teacher forcing is collected from [demo/simple_text_demo.py](demo/simple_text_demo.py) with the `ci-token-matching` test case. You can generate this table by running these tests with the `lt` tool (tell it to run `table` or `pareto`) and pressing `m` whilst in the results section to export to markdown.

Note that token accuracy parses the below to determine expected values +- 0.5. In May 2025 we switched the default to measuring the accuracy by prefilling 512 tokens and generating another 511, rather than generating 128 tokens in earlier versions. This caused overall accuracy values to drop slightly.

Also note that all the performance metrics below were taken for a maximum generation of 200 tokens, i.e., 200 decode iterations.

## Performance

This configuration uses bfp4 MLP and bfp8 attention weights for all models except:
* Qwen-2.5-7B, which uses bfp8 MLP and bfp16 attention weights in all decoder layers
* Llama-3.1-8B which uses bfp8 MLP in only the 32nd decoder layer and bfp4 MLP elsewhere

| Model             | Device      | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|-------------------|-------------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B      | N150        | 79        | 97        | 87.8          | 26        |
| Llama-3.2-1B      | N300        | 79        | 97        | 105.9         | 22        |
| Llama-3.2-1B      | T3K         | 80        | 97        | 119.8         | 32        |
| Llama-3.2-1B      | TG          | 77        | 96        | 51.0          |           |
| Llama-3.2-3B      | N150        | 89        | 98        | 54.0          | 55        |
| Llama-3.2-3B      | N300        | 89        | 98        | 68.0          | 39        |
| Llama-3.2-3B      | T3K         | 91        | 99        | 68.5          | 52        |
| Llama-3.2-3B      | TG          | 87        | 97        | 33.5          |           |
| Llama-3.1-8B      | N150        | 90        | 97        | 28.3          | 104       |
| Llama-3.1-8B      | N300        | 90        | 97        | 44.2          | 67        |
| Llama-3.1-8B      | P100        | 90        | 98        | 29.5          | 84        |
| Llama-3.1-8B      | P150        | 90        | 98        | 33.6          | 76        |
| Llama-3.1-8B      | T3K         | 90        | 98        | 64.3          | 53        |
| Llama-3.1-8B      | T3K  (DP=4) |           |           | 39.6          | 58        |
| Llama-3.1-8B      | T3K  (DP=8) |           |           | 24.9          | 86        |
| Llama-3.1-8B      | TG          | 88        | 97        | 29.5          |           |
| Llama-3.2-11B     | N150        | 90        | 98        | 55.5          | 58        |
| Llama-3.2-11B     | N300        | 90        | 98        | 44.1          | 67        |
| Llama-3.2-11B     | T3K         | 90        | 98        | 62.7          | 47        |
| Llama-3.2-11B     | TG          | 87        | 97        | 29.5          |           |
| Llama-3.1-70B     | T3K         | 96        | 100       | 16.6          | 164       |
| Llama-3.3-70B     | T3K         | 96        | 100       | 16.6          | 164       |
| Llama-3.1-70B     | TG          | 95        | 100       | 12.7          |           |
| Llama-3.3-70B     | TG          | 95        | 100       | 12.7          |           |
| Llama-3.1-70B     | TG   (DP=4) |           |           | 14.8          | 189       |
| Llama-3.2-90B     | T3K         | 96        | 100       | 6             | 5535      |
| Qwen2.5-7B        | N300        | 84        | 96        | 24.6          | 92        |
| Qwen2.5-72B       | T3K         | 99        | 100       | 15.2          | 225       |
| Qwen2.5-32B       | T3K         | 98        | 99        | 22.4          | 190       |
| Qwen2.5-72B       | P150x8      | 99        | 100       |               |           |
| Qwen2.5-32B       | P150x8      | 97        | 99        |               |           |
| Qwen2.5-Coder-32B | T3K         | 96        | 99        | 22.4          | 190       |
| Qwen3-32B         | T3K         | 89        | 97        | 22.9          | 123       |
| QwQ-32B           | T3K         | 96        | 100       | 20.7          | 105       |
| Phi3.5-mini       | N150        |           |           | 43.2          | 98        |
| Phi3.5-mini       | N300        |           |           | 57.8          | 62        |
| Phi3.5-mini       | T3K         |           |           | 48.8          | 51        |
| Mistral-7B        | N150        | 95        | 99        | 29.75         | 100.24    |
| Mistral-7B        | N300        | 95        | 100       | 47.01         | 65.95     |
| Mistral-7B        | T3K         | 95        | 100       | 67.82         | 53.93     |
| Mistral-Small-3.1-24B | T3K      | 95        | 99        |               |           |
| Phi-3-mini-128k-instruct | N150        | 89        | 99        | 45.0          | 73.32     |
| Phi-3-mini-128k-instruct | N300        | 89        | 99        | 60.87         | 114.94    |
| Phi-4 | N300 | 97 | 100 | 37.34 | 123.33 |
| Mixtral-8x7B-v0.1 | T3K         | 98        | 100       | 67.82         | 53.93     |
| Ministral-8B      | N300        | 93        | 98        | 22.15         | 79.3      |

## Accuracy

This configuration uses bfp8 MLP and BF16 attention weights (70B+ models use bfp8 attention and bfp4 MLP).
Llama 3 models test as insensitive to attention precision and so we use bfp8 attention and kv-cache for them even in accuracy mode.

| Model             | Device      | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|-------------------|-------------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B      | N150        | 87        | 99        | 84.7          | 29        |
| Llama-3.2-1B      | N300        | 87        | 98        | 102.8         | 21        |
| Llama-3.2-1B      | T3K         | 88        | 99        | 120.5         | 28        |
| Llama-3.2-1B      | TG          | 85        | 98        | 48.4          |           |
| Llama-3.2-3B      | N150        | 96        | 100       | 47.6          | 63        |
| Llama-3.2-3B      | N300        | 96        | 100       | 63.5          | 41        |
| Llama-3.2-3B      | T3K         | 96        | 100       | 67.9          | 69        |
| Llama-3.2-3B      | TG          | 92        | 99        | 33.6          |           |
| Llama-3.1-8B      | N150        | 96        | 100       | 25.2          | 138       |
| Llama-3.1-8B      | N300        | 96        | 100       | 38.8          | 79        |
| Llama-3.1-8B      | T3K         | 97        | 100       | 60.8          | 81        |
| Llama-3.1-8B      | TG          | 95        | 100       | 29.5          |           |
| Llama-3.2-11B     | N150        | 95        | 100       | 56.7          | 62        |
| Llama-3.2-11B     | N300        | 95        | 100       | 38.3          | 78        |
| Llama-3.2-11B     | T3K         | 96        | 100       | 61.4          | 53        |
| Llama-3.2-11B     | TG          | 94        | 100       | 29.5          |           |
| Llama-3.1-70B     | T3K         | 96        | 100       | 16.5          | 168       |
| Llama-3.1-70B     | TG          | 95        | 100       | 12.7          |           |
| Llama-3.2-90B     | T3K         | 96        | 100       | 6             | 5600      |
| Qwen2.5-7B        | N300        | 84        | 96        | 24.6          | 92        |
| Qwen2.5-72B       | T3K         | 99        | 100       | 15.1          | 216       |
| Qwen2.5-32B       | T3K         | 95        | 99        | 19.7          | 183       |
| Qwen2.5-72B       | P150x8      | 99        | 100       |               |           |
| Qwen2.5-32B       | P150x8      | 97        | 99        |               |           |
| Qwen2.5-Coder-32B | T3K         | 95        | 99        | 19.7          | 183       |
| Qwen3-32B         | T3K         | 95        | 100       | 19.6          | 119       |
| QwQ-32B           | T3K         | 99        | 100       | 18.3          | 120       |
| Phi3.5-mini       | N150        |           |           | 38.8          | 92        |
| Phi3.5-mini       | N300        |           |           | 53.9          | 63        |
| Phi3.5-mini       | T3K         |           |           | 48.6          | 53        |
| Mistral-7B        | N150        | 96        | 100       | 29.75         | 100.24    |
| Mistral-7B        | N300        | 97        | 100       | 47.01         | 65.95     |
| Mistral-7B        | T3K         | 98        | 100       | 67.82         | 53.93     |
| Mistral-Small-3.1-24B | T3K      | 95        | 99        |               |           |
| Phi-3-mini-128k-instruct | N150        | 94        | 99        | 40.41         | 82.58     |
| Phi-3-mini-128k-instruct | N300        | 94        | 99        | 57.0          | 115.36    |
| Phi-4 | N300 | 99 | 100 | 20.48 | 146.32 |
| Mixtral-8x7B-v0.1 | T3K         | 95        | 100       | 67.82         | 53.93     |
| Ministral-8B      | N300        | 97        | 100       | 19.95         | 93.22     |

##  Long-context (64K Tokens)

This configuration uses bfp4 MLP FF1+FF3 for all models. **Batch_size=1 and prefill_length is 64k tokens.**

| Model          | Device | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|---------------|-----------|
| Llama-3.2-1B   | N150   | 53.0          | 20066     |
| Llama-3.2-1B   | N300   | 65.2          | 10949     |
| Llama-3.2-1B   | T3K    | 73.7          | 5271      |
| Llama-3.2-1B   | TG     |               |           |
| Llama-3.2-3B   | N150   | 25.3          | 46743     |
| Llama-3.2-3B   | N300   | 34.8          | 22921     |
| Llama-3.2-3B   | T3K    | 41.0          | 10677     |
| Llama-3.2-3B   | TG     |               |           |
| Llama-3.1-8B   | N150   | 16.9          | 64385     |
| Llama-3.1-8B   | N300   | 26.1          | 36229     |
| Llama-3.1-8B   | T3K    | 38.1          | 16165     |
| Llama-3.1-8B   | TG     |               |           |
| Llama-3.2-11B  | N300   | 26.1          | 36247     |
| Llama-3.2-11B  | T3K    | 38.4          | 16167     |
| Llama-3.2-11B  | TG     |               |           |
| Llama-3.1-70B  | T3K    | 11.9          | 74363     |
| Llama-3.1-70B  | TG     |               |           |
| Qwen2.5-7B     | N300   |               |           |
| Qwen2.5-72B    | T3K    |               |           |

##  Long-context (32K Tokens)

This configuration uses bfp4 MLP FF1+FF3 for all models. **Batch_size=1 and prefill_length is 32k tokens.**

| Model                     | Device | Speed (t/s/u) | TTFT (ms) |
|---------------------------|--------|---------------|-----------|
| Phi-3-mini-128k-instruct  | N300   | 26.1          | 10072     |

## Short-Context, Batch-32

This configuration uses bfp4 MLP FF1+FF3 for all models. **Batch_size=32 and prefill_length is 128 tokens.**

| Model          | Device | Speed (t/s/u) | avg TTFT (ms) |
|----------------|--------|---------------|---------------|
| Llama-3.2-1B   | N150   | 54.7          | 38            |
| Llama-3.2-1B   | N300   | 64.2          | 34            |
| Llama-3.2-1B   | T3K    | 69.9          | 42            |
| Llama-3.2-1B   | TG     |               |               |
| Llama-3.2-3B   | N150   | 36.5          | 69            |
| Llama-3.2-3B   | N300   | 45.8          | 51            |
| Llama-3.2-3B   | T3K    | 47.8          | 63            |
| Llama-3.2-3B   | TG     |               |               |
| Llama-3.1-8B   | N150   | 22.3          | 119           |
| Llama-3.1-8B   | N300   | 33.5          | 80            |
| Llama-3.1-8B   | T3K    | 45.6          | 64            |
| Llama-3.1-8B   | TG     |               |               |
| Llama-3.2-11B  | N300   | 33.4          | 79            |
| Llama-3.2-11B  | T3K    | 45.1          | 64            |
| Llama-3.2-11B  | TG     |               |               |
| Llama-3.1-70B  | T3K    | 14.8          | 192           |
| Llama-3.1-70B  | TG     |               |               |
| Qwen2.5-7B     | N300   |               |               |
| Qwen2.5-72B    | T3K    |               |               |
| Phi-3-mini-128k-instruct  | 150    | 25.66         | 68.58         |
| Phi-3-mini-128k-instruct  | N300   | 39.4          | 85.99         |

# Llama 3 model precision and math fidelity

## precision_cfg = {ff1_3: bfp4, ff2: bfp4, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: lofi, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 85        | 98        | 100.3         | 69        |

## precision_cfg = {ff1_3: bfp4, ff2: bfp8, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: lofi, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 88        | 98        | 100.3         | 55        |

## precision_cfg = {ff1_3: bfp4, ff2: bf16, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: lofi, li_ff2: hifi4, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 87        | 98        | 96.8          | 51        |

## precision_cfg = {ff1_3: bfp8, ff2: bfp4, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 87        | 98        | 98.5          | 50        |

## precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 91        | 98        | 99.0          | 60        |

## precision_cfg = {ff1_3: bfp8, ff2: bf16, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi4, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 89        | 99        | 95.2          | 49        |

## precision_cfg = {ff1_3: bf16, ff2: bfp4, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi4, li_ff2: lofi, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 89        | 98        | 95.2          | 53        |

## precision_cfg = {ff1_3: bf16, ff2: bfp8, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi4, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 91        | 98        | 94.4          | 57        |

## precision_cfg = {ff1_3: bf16, ff2: bf16, wqkv: bfp8, wo: bfp8, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi4, li_ff2: hifi4, li_qkv_decode: hifi2, li_o_decode: hifi2, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: hifi2fp16, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 90        | 98        | 91.2          | 60        |

## precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp4, kv_cache: bfp8, activation: bf16}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: lofi, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: lofi, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 88        | 98        | 98.2          | 45        |

## precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp4, kv_cache: bfp8, activation: bfp8}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: lofi, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: lofi, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 90        | 98        | 101.0         | 57        |

## precision_cfg = {ff1_3: bfp8, ff2: bfp8, wqkv: bfp8, wo: bfp4, kv_cache: bfp8, activation: mixed}, fidelity_cfg = {li_ff1_3: hifi2, li_ff2: hifi2, li_qkv_decode: hifi2, li_o_decode: lofi, sdpa_decode: hifi2na, li_qkv_prefill: hifi2, li_o_prefill: lofi, sdpa_prefill: hifi4}

| Model          | Device | Top-1 (%) | Top-5 (%) | Speed (t/s/u) | TTFT (ms) |
|----------------|--------|-----------|-----------|---------------|-----------|
| Llama-3.2-1B   | N300   | 90        | 98        | 99.4          | 73        |