Update README.md
Browse files
README.md
CHANGED
|
@@ -19,8 +19,6 @@ pipeline_tag: text-generation
|
|
| 19 |
|
| 20 |
4-bit AWQ quantization of [Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24](https://huggingface.co/Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24), a Russian-language grounded-RAG model. Produced with [llm-compressor](https://github.com/vllm-project/llm-compressor) in the compressed-tensors format for vLLM.
|
| 21 |
|
| 22 |
-
Main motivation: on pre-Ampere GPUs (sm75, e.g. Quadro RTX 6000 / RTX 20xx) GPTQ checkpoints fall back to slow non-Marlin kernels in vLLM, while this checkpoint runs via `MarlinLinearKernel for CompressedTensorsWNA16`, giving a large speedup in prefill and batched decode.
|
| 23 |
-
|
| 24 |
## Quantization details
|
| 25 |
|
| 26 |
- Method: AWQ, scheme W4A16, `lm_head` kept unquantized
|
|
|
|
| 19 |
|
| 20 |
4-bit AWQ quantization of [Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24](https://huggingface.co/Vikhrmodels/Vikhr-Nemo-12B-Instruct-R-21-09-24), a Russian-language grounded-RAG model. Produced with [llm-compressor](https://github.com/vllm-project/llm-compressor) in the compressed-tensors format for vLLM.
|
| 21 |
|
|
|
|
|
|
|
| 22 |
## Quantization details
|
| 23 |
|
| 24 |
- Method: AWQ, scheme W4A16, `lm_head` kept unquantized
|