DeepSeek-V2-Lite-FP8-Group

Per-group FP8 quantized version of deepseek-ai/DeepSeek-V2-Lite, created with llm-compressor.

Quantization Details

Property	Value
Base model	deepseek-ai/DeepSeek-V2-Lite
Parameters	15.7B total (2.4B active)
Architecture	DeepSeek-V2 (MLA + MoE, 64 experts, top-6)
Quantization	Per-group FP8 (E4M3), dynamic activations
Weight strategy	Group, group_size=64
Activation strategy	Per-token, dynamic
Format	compressed-tensors (float-quantized)
Ignored layers	lm_head
Model size	~16 GB
Tool	llm-compressor 0.10.0

This model uses the same per-group FP8 quantization scheme as DeepSeek-V3 (weight_block_size: [1, 64]), making it useful for testing and validating group FP8 inference paths (e.g., MLA attention + group FP8 fusion in vLLM) without needing a 671B model.

Evaluation

GSM8K accuracy (100 samples, via lm_eval harness):

Model	exact_match
Baseline (BF16)	0.300
FP8-Group (this model)	0.330

No precision degradation observed from group FP8 quantization.

Usage

With vLLM

vllm serve carlyou/DeepSeek-V2-Lite-FP8-Group --trust-remote-code

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "carlyou/DeepSeek-V2-Lite-FP8-Group",
    torch_dtype="auto",
    trust_remote_code=True,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "carlyou/DeepSeek-V2-Lite-FP8-Group",
    trust_remote_code=True,
)

Reproduction

pip install llmcompressor transformers
python quantize.py --model deepseek-ai/DeepSeek-V2-Lite --scheme fp8-group

See carlyou/llm-quant for the quantization script.