---
base_model:
- sarvamai/sarvam-30b
---
## Quantization Details

This model was quantized to 8-bit floating-point (FP8) using the [llmcompressor](https://github.com/vllm-project/llm-compressor) library. 

To prioritize fast processing and maintain high accuracy without requiring a calibration dataset, we utilized a zero-shot Round-to-Nearest (RTN) approach with block-wise weight quantization.

**Quantization Configuration:**
* **Algorithm:** Round-to-Nearest (RTN)
* **Weight Scheme:** FP8 with a Block Size of 128 (`FP8_BLOCK`)
* **Activation Scheme:** Dynamic FP8 (quantized dynamically during inference)
* **Calibration Data:** None (Zero-shot)
* **Preserved Layers:** The `lm_head` and `mlp.gate` layers were explicitly ignored and kept in original precision to protect the structural integrity and routing accuracy of the model.

**Usage:**
This FP8-quantized model is natively supported by `vLLM` and `transformers`. It is highly recommended to run this model in environments optimized for FP8 computation (e.g., NVIDIA Hopper or Ada Lovelace architectures) to achieve the best performance memory-bandwidth reductions.