---
license: apache-2.0
base_model: Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24
datasets:
- Vikhrmodels/GrandMaster-PRO-MAX
language:
- ru
- en
tags:
- awq
- compressed-tensors
- vllm
- quantized
- rag
pipeline_tag: text-generation
---

# Vikhr-Nemo-12B-Instruct-R — AWQ (W4A16)

4-bit AWQ quantization of [Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24](https://huggingface.co/Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24), a Russian-language grounded-RAG model. Produced with [llm-compressor](https://github.com/vllm-project/llm-compressor) in the compressed-tensors format for vLLM.

## Quantization details

- Method: AWQ, scheme W4A16, `lm_head` kept unquantized
- Tool: llm-compressor (`AWQModifier`, one-shot)
- Calibration: 256 samples x 2048 tokens from [GrandMaster-PRO-MAX](https://huggingface.co/datasets/Vikhrmodels/GrandMaster-PRO-MAX), rendered through the model chat template

## Usage

### Ampere and newer (sm80+): dtype is picked from config automatically
```bash
vllm serve ConstDemi/vikhr-nemo-12b-instruct-r-awq
```

### Turing (sm75, e.g. Quadro RTX 6000): no native bfloat16
```bash
vllm serve ConstDemi/vikhr-nemo-12b-instruct-r-awq \
    --dtype float16
```

Verified on vLLM 0.25.1, sm75 GPU, `--dtype float16`. The model follows the two-step grounded generation protocol of the original Vikhr-R (document selection as JSON, then the answer); see the original model card for the prompt format.

## Limitations

- No perplexity or benchmark-based quality evaluation was performed; the checkpoint was validated on an internal Russian grounded-RAG task (answer quality, source citation, and the JSON document-selection step).
- Calibrated on general instruction data, not domain-specific text.
- bf16 scale tensors: use `--dtype float16` on GPUs without native bf16 support.