File size: 2,186 Bytes
68c5c3a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
license: other
license_name: deepseek-license
license_link: LICENSE
base_model: deepseek-ai/DeepSeek-V2-Lite
tags:
- deepseek
- mla
- moe
- fp8
- group-quantization
- compressed-tensors
library_name: transformers
---
# DeepSeek-V2-Lite-FP8-Group
Per-group FP8 quantized version of [deepseek-ai/DeepSeek-V2-Lite](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite), created with [llm-compressor](https://github.com/vllm-project/llm-compressor).
## Quantization Details
| Property | Value |
|----------|-------|
| Base model | deepseek-ai/DeepSeek-V2-Lite |
| Parameters | 15.7B total (2.4B active) |
| Architecture | DeepSeek-V2 (MLA + MoE, 64 experts, top-6) |
| Quantization | Per-group FP8 (E4M3), dynamic activations |
| Weight strategy | Group, group_size=64 |
| Activation strategy | Per-token, dynamic |
| Format | compressed-tensors (float-quantized) |
| Ignored layers | lm_head |
| Model size | ~16 GB |
| Tool | llm-compressor 0.10.0 |
This model uses the same per-group FP8 quantization scheme as DeepSeek-V3 (`weight_block_size: [1, 64]`), making it useful for testing and validating group FP8 inference paths (e.g., MLA attention + group FP8 fusion in vLLM) without needing a 671B model.
## Evaluation
GSM8K accuracy (100 samples, via lm_eval harness):
| Model | exact_match |
|-------|-------------|
| Baseline (BF16) | 0.300 |
| FP8-Group (this model) | 0.330 |
No precision degradation observed from group FP8 quantization.
## Usage
### With vLLM
```bash
vllm serve carlyou/DeepSeek-V2-Lite-FP8-Group --trust-remote-code
```
### With Transformers
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"carlyou/DeepSeek-V2-Lite-FP8-Group",
torch_dtype="auto",
trust_remote_code=True,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
"carlyou/DeepSeek-V2-Lite-FP8-Group",
trust_remote_code=True,
)
```
## Reproduction
```bash
pip install llmcompressor transformers
python quantize.py --model deepseek-ai/DeepSeek-V2-Lite --scheme fp8-group
```
See [carlyou/llm-quant](https://github.com/carlyou/llm-quant) for the quantization script.
|