Add TRT-LLM W8A8_SQ + INT8_KV checkpoint and detailed model card

README.md

@@ -0,0 +1,92 @@
+---
+license: llama3.1
+base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
+library_name: tensorrt_llm
+tags:
+  - text-generation
+  - tensorrt-llm
+  - llama
+  - smoothquant
+  - int8
+  - kv-cache-quantization
+---
+
+# Meta-Llama-3.1-8B-Instruct TensorRT-LLM checkpoint (W8A8 SmoothQuant + INT8 KV)
+
+TensorRT-LLM **checkpoint** for **Meta-Llama-3.1-8B-Instruct**, with **W8A8 SmoothQuant** quantization for model compute and **INT8** KV cache. Use with `trtllm-build` to produce an engine for inference.
+
+## Model details
+
+| Item | Value |
+|---|---|
+| **Base model** | Meta-Llama-3.1-8B-Instruct |
+| **Framework** | TensorRT-LLM (checkpoint format) |
+| **Weight/activation quantization** | W8A8 SmoothQuant (`W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN`) |
+| **KV cache** | INT8 |
+| **Producer** | TensorRT-LLM `convert_checkpoint.py` |
+| **Key conversion flags** | `--smoothquant 0.5 --per_token --per_channel --int8_kv_cache` |
+| **Calibration size** | 512 samples (`--calib_size 512`) |
+| **Architecture** | LlamaForCausalLM (decoder-only) |
+
+## Build (how to produce this checkpoint)
+
+This checkpoint is produced using the TensorRT-LLM Llama converter with SmoothQuant and INT8 KV cache enabled:
+
+```bash
+python TensorRT-LLM/examples/models/core/llama/convert_checkpoint.py \
+  --model_dir /path/to/Meta-Llama-3.1-8B-Instruct \
+  --output_dir ./llama-3.1-8b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8 \
+  --dtype float16 \
+  --tp_size 1 \
+  --smoothquant 0.5 \
+  --per_token \
+  --per_channel \
+  --int8_kv_cache \
+  --calib_size 512
+```
+
+### Environment note
+
+In this environment, loading the slow tokenizer path (`use_fast=False`) returns an invalid object for this model. During generation, tokenizer loading is forced to `use_fast=True` at runtime. This only affects tokenizer loading compatibility in the conversion process and does not change the target quantization configuration.
+
+## Output
+
+After conversion, `--output_dir` contains:
+
+- `config.json` - TensorRT-LLM checkpoint config
+- `rank0.safetensors` - rank-0 checkpoint weights (single-GPU)
+
+## Upload (how to upload to Hugging Face)
+
+```bash
+cd ./llama-3.1-8b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8
+
+huggingface-cli repo create rungalileo/llama-3.1-8b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8 --repo-type model
+huggingface-cli upload rungalileo/llama-3.1-8b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8 . --repo-type model
+```
+
+## How to use
+
+### 1. Build engine
+
+```bash
+git clone https://huggingface.co/rungalileo/llama-3.1-8b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8
+cd llama-3.1-8b-instruct-trtllm-ckpt-wq_w8a8sq-kv_int8
+
+trtllm-build --checkpoint_dir . --output_dir ./engine \
+  --max_batch_size 1 --max_input_len 512 --max_seq_len 1024
+```
+
+### 2. Run inference
+
+Use a tokenizer from the base model:
+
+```bash
+trtllm-serve ./engine --tokenizer meta-llama/Meta-Llama-3.1-8B-Instruct --port 8000
+# OpenAI-compatible API: http://localhost:8000/v1/completions
+```
+
+## References
+
+- TensorRT-LLM
+- Meta-Llama-3.1-8B-Instruct

config.json

@@ -0,0 +1,85 @@
+{
+    "mlp_bias": false,
+    "attn_bias": false,
+    "rotary_base": 500000.0,
+    "rotary_scaling": {
+        "factor": 8.0,
+        "high_freq_factor": 4.0,
+        "low_freq_factor": 1.0,
+        "original_max_position_embeddings": 8192,
+        "rope_type": "llama3"
+    },
+    "residual_mlp": false,
+    "disable_weight_only_quant_plugin": false,
+    "moe": {
+        "num_experts": 0,
+        "shared_expert_intermediate_size": 0,
+        "top_k": 0,
+        "normalization_mode": null,
+        "sparse_mixer_epsilon": 0.01,
+        "tp_mode": 0,
+        "device_limited_n_group": 0,
+        "device_limited_topk_group": 0,
+        "device_limited_routed_scaling_factor": 1.0
+    },
+    "remove_duplicated_kv_heads": false,
+    "fc_after_embed": false,
+    "use_input_layernorm_in_first_layer": true,
+    "use_last_layernorm": true,
+    "layer_idx_offset": 0,
+    "embedding_multiplier": 1.0,
+    "attention_multiplier": 1.0,
+    "residual_multiplier": 1.0,
+    "output_multiplier_scale": 1.0,
+    "has_partial_lora_mask": false,
+    "architecture": "LlamaForCausalLM",
+    "dtype": "float16",
+    "vocab_size": 128256,
+    "hidden_size": 4096,
+    "num_hidden_layers": 32,
+    "num_attention_heads": 32,
+    "hidden_act": "silu",
+    "logits_dtype": "float32",
+    "norm_epsilon": 1e-05,
+    "runtime_defaults": null,
+    "position_embedding_type": "rope_gpt_neox",
+    "num_key_value_heads": 8,
+    "intermediate_size": 14336,
+    "max_position_embeddings": 131072,
+    "mapping": {
+        "world_size": 1,
+        "gpus_per_node": 8,
+        "cp_size": 1,
+        "tp_size": 1,
+        "pp_size": 1,
+        "moe_tp_size": 1,
+        "moe_cluster_size": 1,
+        "moe_ep_size": 1,
+        "attn_tp_size": 1,
+        "attn_cp_size": 1,
+        "cp_config": {},
+        "auto_parallel": false,
+        "enable_attention_dp": false,
+        "enable_lm_head_tp_in_adp": false
+    },
+    "quantization": {
+        "quant_algo": "W8A8_SQ_PER_CHANNEL_PER_TOKEN_PLUGIN",
+        "kv_cache_quant_algo": "INT8",
+        "group_size": 128,
+        "smoothquant_val": 0.5,
+        "clamp_val": null,
+        "use_meta_recipe": false,
+        "has_zero_point": false,
+        "pre_quant_scale": false,
+        "exclude_modules": null,
+        "mamba_ssm_cache_dtype": null
+    },
+    "use_parallel_embedding": false,
+    "embedding_sharding_dim": 0,
+    "head_size": 128,
+    "qk_layernorm": false,
+    "rotary_embedding_dim": 128,
+    "tie_word_embeddings": false,
+    "quant_ckpt_path": null,
+    "load_model_on_cpu": false
+}

rank0.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6c29d8d1ec8ada985f054c60e20e30a83892c46637fc0c92cd4259a4722a064d
+size 9089128848