Update README.md

README.md

@@ -14,6 +14,29 @@ tags:
 
 This is a variant of `moonshotai/Kimi-K3` quantized with FP8_BLOCK quantization applied to MLP and experts layers. Attention layers were not quantized due to shape mismatches, namely with `self_attn.b_proj.weight`
 
+## Usage ##
+```bash
+docker run --gpus all \
+  --privileged --ipc=host -p 8000:8000 \
+  -v ~/.cache/huggingface:/root/.cache/huggingface \
+  -e GLOO_SOCKET_IFNAME=$IFACE_NAME \
+  -e NCCL_SOCKET_IFNAME=$IFACE_NAME \
+  -e VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 \
+  vllm/vllm-openai:kimi-k3 RedHatAI/Kimi-K3-FP8-BLOCK \
+  --trust-remote-code \
+  --load-format fastsafetensors \
+  --gpu-memory-utilization 0.95 \
+  --tensor-parallel-size 16 \
+  --nnodes 2 \
+  --node-rank 0 \
+  --master-addr $HEAD_IP \
+  --no-enable-flashinfer-autotune \
+  --disable-custom-all-reduce \
+  --enable-auto-tool-choice \
+  --tool-call-parser kimi_k3 \
+  --reasoning-parser kimi_k3
+```
+
 ## Creation Process
 
 Quantized using [llm-compressor](https://github.com/vllm-project/llm-compressor):