--- library_name: transformers license: mit language: - en - vi - id - th - fil - ta - ms - my tags: - aisingapore - nvidia - pytorch - FP8 arxiv: "2609.18310" pipeline_tag: text-generation track_downloads: true base_model: - aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B --- ![Banner!](Nemotron-SEA-LION-v4.8-30B-A3B-FP8.png "Nemotron-SEA-LION-v4.8-30B-A3B-FP8")

Technical Report 👁️

# Nemotron-SEA-LION-v4.8-30B-A3B-FP8 *Last updated: 2026-09-18* This repository contains the W8A8 FP8 (E4M3) quantized weights for **[aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B](https://huggingface.co/aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B)**. The model is optimized for Southeast Asian (SEA) languages, cultures, and contexts. This repository provides quantized models that have significantly compressed memory footprints while preserving model accuracy, enabling low-latency MoE inference across high-throughput production environments. ## Model Details ### Model Description SEA-LION stands for Southeast Asian Languages In One Network. The model compression process includes quantizing all backbone `nn.Linear` layers, spanning the Mamba, attention, and MoE expert projections (2,944 routed-expert matrices). The weights use a single symmetric FP8 scale per tensor, and input activations are quantized to FP8 with static per-tensor scales frozen from calibration. The calibration was performed on 512 instructions from [garage-bAInd/Open-Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus). This model was quantized using [vllm llm-compressor](https://github.com/vllm-project/llm-compressor.git) library. For tokenization, the model employs the default tokenizer used in [nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16). - **Developed by:** AI Products Pillar, AI Singapore - **Funded by:** National Research Foundation Singapore - **Shared by:** AI Products Pillar, AI Singapore - **Model type:** Instruction-tuned language model - **Architecture:** Mamba2-Transformer Hybrid MoE - **Context length:** 262,144 tokens - **Language(s):** Burmese, English, Indonesian, Filipino, Malay, Tamil, Thai, and Vietnamese - **License:** [MIT](https://mit-license.org/) - **Parent model:** [aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B](https://huggingface.co/aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B) ### Model Sources - **Collection:** [SEA-LION v4.8 - an aisingapore Collection](https://huggingface.co/collections/aisingapore/sea-lion-v48) ## Available Quantized Versions We provide multiple quantization formats to optimize deployment trade-offs between memory footprint and output quality. - [Nemotron-SEA-LION-v4.8-30B-A3B-NVFP4](https://huggingface.co/aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B-NVFP4) - [Nemotron-SEA-LION-v4.8-30B-A3B-FP8](https://huggingface.co/aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B-FP8) - [Nemotron-SEA-LION-v4.8-30B-A3B-GGUF](https://huggingface.co/aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B-GGUF) - [Nemotron-SEA-LION-v4.8-120B-A12B-NVFP4](https://huggingface.co/aisingapore/Nemotron-SEA-LION-v4.8-120B-A12B-NVFP4) - [Nemotron-SEA-LION-v4.8-120B-A12B-FP8](https://huggingface.co/aisingapore/Nemotron-SEA-LION-v4.8-120B-A12B-FP8) - [Nemotron-SEA-LION-v4.8-120B-A12B-GGUF](https://huggingface.co/aisingapore/Nemotron-SEA-LION-v4.8-120B-A12B-GGUF) ## Usage ### vLLM You can serve the model using [vLLM](https://docs.vllm.ai/en/stable/features/quantization): ```python from vllm import LLM, SamplingParams llm = LLM( model="aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B-FP8", trust_remote_code=True, ) messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Where is Lau Pa Sat? Tell me in a single sentence."}, ] params = SamplingParams(temperature=0.7, top_p=0.9 ) outputs = llm.chat(messages, params) print(outputs[0].outputs[0].text) ``` ### SGLang You can serve the model using [SGLang](https://docs.sglang.io/docs/advanced_features/quantized_kv_cache): ```python from sglang import Engine, SamplingParams # SGLang natively detects the pre-quantized FP8 checkpoint llm = Engine( model_path="aisingapore/Nemotron-SEA-LION-v4.8-30B-A3B-FP8", trust_remote_code=True, ) messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Where is Lau Pa Sat? Tell me in a single sentence."}, ] # Configure generation constraints params = SamplingParams(temperature=0.7, top_p=0.9) # Execute inference using the OpenAI-compatible chat message format outputs = llm.chat(messages, params) print(outputs["text"]) ``` ## Evaluation ### Throughput Test Each checkpoint was served with vLLM on NVIDIA H200 GPUs and benchmarked over the OpenAI-compatible chat endpoint with GuideLLM, an open-source throughput benchmark from the [vLLM](https://github.com/vllm-project/guidellm) project. The prompts are our own Southeast Asian instructions from the SEA-instruct dataset, streamed with a fixed output length per request so that every checkpoint generates the same number of tokens. Time to first token is the time taken from sending the request until the first content token arrives, so it covers prompt processing (prefill) and network overhead on localhost. Tokens per second is a metric that measures the number of output tokens divided by that request's total time from send to final token. Both are reported as mean ± standard deviation over all requests completed in a 120-second run. The VRAM Usage is taken from vLLM upon loading. | Model Variant | Number of GPUs | Actual VRAM Usage (GB) | Time to First Token (s) | Tokens per Second | | --- | --- | --- | --- | --- | | 30B-A3B(BF16) | 1x H200 GPU(s) | 63.3 GB | 0.0456 | 311.10 ± 25.12 | | 30B-A3B-FP8 | 1x H200 GPU(s) | 33.4 GB | 0.0350 | 365.18 ± 31.50 | | 30B-A3B-NVFP4 | 1x H200 GPU(s) | 20.5 GB | 0.0339 | 340.29 ± 8.68 | | 120B-A12B(BF16) | 2x H200 GPU(s) | 242.5 GB | 0.0936 | 150.31 ± 12.80 | | 120B-A12B-FP8 | 2x H200 GPU(s) | 128.6 GB | 0.0449 | 161.36 ± 3.05 | | 120B-A12B-NVFP4 | 2x H200 GPU(s) | 80.4 GB | 0.0446 | 156.36 ± 2.64 | *Note: Benchmarks were captured on Hopper architecture (NVIDIA H200) GPUs; NVFP4 precision formats yield further hardware-level acceleration when executed natively on NVIDIA Blackwell infrastructure.* All models were served at their full context length of 262,144 tokens with default vLLM settings. The tokens per second are reported as the aggregate across the GPUs used. FP8 checkpoints are a good default for reduced-precision deployment. NVFP4 checkpoints offer a 4-bit alternative designed for NVIDIA's Blackwell architecture. The figures reflect Hopper execution, and the checkpoints have not yet been benchmarked on Blackwell. We recommend measuring on your own workload before choosing a format. For details on Nemotron-SEA-LION-v4.8 performance, please refer to the [SEA-HELM Leaderboard](https://leaderboard.sea-lion.ai/). ## Technical Specifications ### Technical Report For training details, see the [SEA-LION-v4.8 Technical Report](https://arxiv.org/abs/2609.18310). ### Model Architecture The architecture is based on the highly efficient Nemotron-3-Nano foundation. The detailed architecture can be found at [nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 documentation](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Nano-Technical-Report.pdf). ## Uses ### Out-of-Scope Use The model has not been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claims, damages, or other liabilities arising from the use of the released weights and codes. ### Bias, Risks, and Limitations The model was not tested for robustness against adversarial prompting. It is important for users to be aware that our model exhibits certain limitations that warrant consideration. Like many LLMs, the model can hallucinate and occasionally generates irrelevant content, introducing fictional elements that are not grounded in the provided context. Users should also exercise caution in interpreting and validating the model's responses due to the potential inconsistencies. ## Citation **BibTeX:** ```bibtex @misc{aisingapore2026sealionv48technicalreport, title={SEA-LION-v4.8: A Technical Report}, author={Adila Aulia and Ahmed Dabeer and Ahn Jeongmi and Antonyrex Sajeban and Chan Hok Teng Adwin and Cheng Zi Yi Nicholas and Choa Hsueh Mei Esther and Heng Jonathan and Jann Railey Estrada Montalan and Lee Chwan Ren and Leong Wai Yi and Leong Wei Qi and Liew Rachel and Limkonchotiwat Peerat and Muhammad Ridzuan Bin Mokhtar and Nagarajan Karthik and Ng Boon Cheong Raymond and Ngee Chia Tai and Ngui Jian Gang and Nguyen Thanh Ngan and Ong Tat-Wee David and Pereira Mark and Phang Shi Wei Benjamin and Poon Joseph and Rengarajan Hamsawardhini and Susanto Yosephine and Sutaveephamochanon Anocha and Tan Choon Meng and Tan Chor Phin Evelyn and Tan Le Min Sheryl and Tan Siao Wei Jessica and Tan Yixian and Tasawong Panuthep and Tee Jun Yun and Teng Kok Wai Walter and Teo Eng Sipp Leslie and Tjhi William and Tuchinda Pume and Wu Donghang and Yong Xianbin and Zhang Zhou}, year={2026}, eprint={2609.18310}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2609.18310}, } ``` ## Team AI Products Pillar, AI Singapore ## Acknowledgement This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA), Singapore under its National Large Language Model Funding Initiative. ## Contact [**sealion@aisingapore.org**](mailto:sealion@aisingapore.org)