SaltyYean/sarvam-30b-FP8-BLOCK

Quantization Details

This model was quantized to 8-bit floating-point (FP8) using the llmcompressor library.

To prioritize fast processing and maintain high accuracy without requiring a calibration dataset, we utilized a zero-shot Round-to-Nearest (RTN) approach with block-wise weight quantization.

Quantization Configuration:

• Algorithm: Round-to-Nearest (RTN)
• Weight Scheme: FP8 with a Block Size of 128 (FP8_BLOCK)
• Activation Scheme: Dynamic FP8 (quantized dynamically during inference)
• Calibration Data: None (Zero-shot)
• Preserved Layers: The lm_head and mlp.gate layers were explicitly ignored and kept in original precision to protect the structural integrity and routing accuracy of the model.

Usage: This FP8-quantized model is natively supported by vLLM and transformers. It is highly recommended to run this model in environments optimized for FP8 computation (e.g., NVIDIA Hopper or Ada Lovelace architectures) to achieve the best performance memory-bandwidth reductions.