| --- |
| license: apache-2.0 |
| tags: |
| - neuron |
| - trainium |
| - aws |
| - compiled |
| - laguna |
| base_model: poolside/Laguna-XS.2 |
| --- |
| |
| # Laguna-XS.2 — Pre-compiled for AWS Neuron (trn2.3xlarge) |
|
|
| Pre-compiled and pre-sharded model artifacts for serving [poolside/Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2) on AWS Trainium2 using NxD Inference. |
|
|
| ## Configuration |
|
|
| - **Instance**: trn2.3xlarge (LNC=2, 4 logical cores) |
| - **TP degree**: 4 |
| - **Batch size**: 4 (TKG), 1 (CTE) |
| - **Max sequence length**: 4096 |
| - **Precision**: BF16 |
| - **SDK**: Neuron SDK 2.29 (neuronx-cc 2.24, NxDI 0.9.17334) |
|
|
| ## Files |
|
|
| | File | Size | Description | |
| |------|------|-------------| |
| | | 4.3 GB | Compiled NEFFs (6 CTE + 6 TKG buckets) | |
| | | 12 KB | NxDI inference configuration | |
| | | 16 GB | Sharded weights for TP rank 0 | |
| | | 16 GB | Sharded weights for TP rank 1 | |
| | | 16 GB | Sharded weights for TP rank 2 | |
| | | 16 GB | Sharded weights for TP rank 3 | |
|
|
| ## Usage with vLLM |
|
|
|
|
|
|
| ## Performance |
|
|
| | Metric | Value | |
| |--------|-------| |
| | Throughput (BS=1) | ~50 tok/s (via vLLM) | |
| | Throughput (BS=4, raw) | 223 tok/s | |
| | Throughput (BS=8, raw) | 310 tok/s | |
| | TPOT (BS=1) | 11 ms | |
|
|
| ## Requirements |
|
|
| - AWS trn2.3xlarge instance |
| - Neuron SDK 2.29 (DLAMI 20260410) |
| - [NxDI fork with Laguna contrib](https://github.com/aws-neuron/neuronx-distributed-inference/pull/158) |
|
|
