Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -194,11 +194,11 @@ This is only a quick local parity check, not a full benchmark suite. It is inten
|
|
| 194 |
|
| 195 |
The table below compares three locally validated TensorRT-LLM variants built for the same GPU family and the same local engine limits (`max_batch_size=1`, `max_seq_len=1024`, `max_num_tokens=256`).
|
| 196 |
|
| 197 |
-
| Variant | Checkpoint | Engine | `short_chat_42_64` | `balanced_128_128` | `long_generation_42_192` | Quick-check overall |
|
| 198 |
-
|---|---:|---:|---:|---:|---:|---:|---
|
| 199 |
-
| `FP16` | `3.4 GB` | `3.4 GB` | `105.48 tok/s` | `105.49 tok/s` | `105.70 tok/s` | `0.75` | `
|
| 200 |
-
| `FP8` | `2.1 GB` | `2.2 GB` | `166.72 tok/s` | `144.37 tok/s` | `151.36 tok/s` | `0.75` | `
|
| 201 |
-
| `NVFP4` | `1.6 GB` | `1.2 GB` | `199.58 tok/s` | `200.09 tok/s` | `200.28 tok/s` | `0.60` | `
|
| 202 |
|
| 203 |
This comparison is intentionally local and narrow. It should not be treated as a universal benchmark across all prompts, datasets, GPUs, or TensorRT-LLM versions.
|
| 204 |
|
|
|
|
| 194 |
|
| 195 |
The table below compares three locally validated TensorRT-LLM variants built for the same GPU family and the same local engine limits (`max_batch_size=1`, `max_seq_len=1024`, `max_num_tokens=256`).
|
| 196 |
|
| 197 |
+
| Variant | Checkpoint | Engine | `short_chat_42_64` | `balanced_128_128` | `long_generation_42_192` | Quick-check overall | Quick-check change vs FP16 | Practical reading |
|
| 198 |
+
|---|---:|---:|---:|---:|---:|---:|---|---|
|
| 199 |
+
| `FP16` | `3.4 GB` | `3.4 GB` | `105.48 tok/s` | `105.49 tok/s` | `105.70 tok/s` | `0.75` | `baseline` | Most conservative variant |
|
| 200 |
+
| `FP8` | `2.1 GB` | `2.2 GB` | `166.72 tok/s` | `144.37 tok/s` | `151.36 tok/s` | `0.75` | `no drop on this quick-check` | Best balance in these local tests |
|
| 201 |
+
| `NVFP4` | `1.6 GB` | `1.2 GB` | `199.58 tok/s` | `200.09 tok/s` | `200.28 tok/s` | `0.60` | `-15 pts on this quick-check` | Fastest and smallest, but with visible quality drop |
|
| 202 |
|
| 203 |
This comparison is intentionally local and narrow. It should not be treated as a universal benchmark across all prompts, datasets, GPUs, or TensorRT-LLM versions.
|
| 204 |
|