| --- |
| license: other |
| license_name: kimi-k3 |
| license_link: https://huggingface.co/moonshotai/Kimi-K3/blob/9f62e4e9fffbd0a83ddd60e1c209d828994b3569/LICENSE |
| base_model: moonshotai/Kimi-K3 |
| base_model_relation: quantized |
| tags: |
| - compressed-tensors |
| - nvfp4 |
| - nvfp4a16 |
| - kimi-k3 |
| - moe |
| --- |
| |
| # Kimi-K3 NVFP4A16 — Standard Requantization |
|
|
| Weight-only NVFP4A16 derivative of source revision |
| [`moonshotai/Kimi-K3@9f62e4e9fffbd0a83ddd60e1c209d828994b3569`](https://huggingface.co/moonshotai/Kimi-K3/tree/9f62e4e9fffbd0a83ddd60e1c209d828994b3569). |
|
|
| ## Method |
|
|
| MXFP4 values are dequantized per tensor and requantized with the compressed-tensors memoryless_minmax NVFP4 algorithm. Gate/up experts share one global scale as required by fused MoE loaders; w2 uses its own tensor global scale. |
| |
| Pinned shard 49, expert 0 real-weight sample versus MXFP4: w1 relative RMSE 0.064053 / cosine 0.998657480; w2 0.063678 / 0.998681610; w3 0.063978 / 0.998660965. All pass relative RMSE < 0.10 and cosine similarity > 0.99. |
| |
| - Format: `compressed-tensors`, `nvfp4-pack-quantized` |
| - Weights: FP4 E2M1, tensor-group strategy, group size 16 |
| - Local scales: FP8 E4M3FN |
| - Activations: unquantized (`input_activations: null`) |
| - Quantized modules: routed MoE experts only; all ignored K3 modules remain bit-identical |
| - Source revision: `9f62e4e9fffbd0a83ddd60e1c209d828994b3569` |
| - Source provenance: `pinned-staging-config-and-commit-window` |
| - Lossless over-span policy: `fail; never silently requantize` |
| - Tensor data size: 1.497 TiB |
| - Size change versus source tensor data: +5.45% |
| - Companion artifact: [`GrEarl/Kimi-K3-NVFP4A16-Transcoded`](https://huggingface.co/GrEarl/Kimi-K3-NVFP4A16-Transcoded) |
|
|
| This is **NVFP4A16**, not calibrated W4A4 NVFP4. Kimi K3 was trained |
| with MXFP8 activations; FP4 activation quantization would require a separate |
| calibration and quality-validation run. |
|
|
| ## Runtime validation (2026-07-27 UTC) |
|
|
| A **complete Stable LatentMoE block passed on vLLM for this published |
| artifact** using the official `KimiMoE` class. The test first loaded and ran the |
| block through a purpose-built layout path, then rebuilt it and loaded the same |
| 8,072 source tensors through the real K3 model-class chain: |
| `KimiK3ForConditionalGeneration.load_weights` -> `AutoWeightsLoader` -> |
| `KimiLinearForCausalLM.load_weights` -> `KimiLinearModel.load_weights` -> the |
| generic `process_weights_after_loading` traversal. |
|
|
| - Evidence run: `20260727T234215Z` |
| - Earlier routed-expert-core run: `20260727T220141Z` |
| - Tested artifact revision: `6d82c0bfeb91dec56fbd1b0726ccb45d7692caf4` |
| - Checkpoint shard: `model-00049-of-000096.safetensors` (17,916,197,160 bytes) |
| - Runtime image: `vllm/vllm-openai:kimi-k3` |
| - Pinned linux/amd64 image: `vllm/vllm-openai@sha256:fb16b180bd9727600067e16fcd6a6de43fb4db1baf4298ef20b4dbdf6bfa5a0e` |
| - vLLM: `0.1.dev19262+gb6bbf29dd.d20260727` |
| - Hardware: NVIDIA B200, SM100; torch `2.13.0+cu130`, CUDA 13.0 |
| - Complete path: FP32 router + correction bias, `7168 -> 3584` latent down, |
| 896 routed experts with real top-k 16 and `3584 -> 3072 -> 3584`, routed |
| RMSNorm + `3584 -> 7168` up, and 2 shared experts (`7168 -> 6144 -> 7168`) |
| - Loaded source tensors: 8,072 = 8 nonexpert block tensors + 8,064 routed-expert |
| quant tensors; all 896 experts |
| - SiTU: beta `4.0`, linear beta `25.0` |
| - Backend: `MARLIN` / `MarlinExperts` |
| - Observed operators: `vllm::moe_forward_shared`, |
| `_moe_C::moe_wna16_marlin_gemm`, `_moe_C::grouped_topk`, |
| `_C::situ_and_mul`, `vllm_ir::rms_norm`, routing alignment, and MoE sum |
| - Router logits: `[2, 896]`; selected IDs `[2, 16]`; 16 unique in-range |
| experts per token; finite normalized weights |
| - Both direct and real model-class-loader outputs: `[2, 7168]`, finite and |
| nonzero; sampled parameter fingerprints matched exactly and outputs were |
| bit-exact (`max_abs_difference=0`) |
| - Complete-block validation run cost: `$0.341178`; all attempts including the |
| externally canceled run: `$3.892933 < $4.5` |
|
|
| This verifies the real K3 outer/inner `load_weights` mapping and generic |
| post-load traversal on a reduced one-block wrapper. `DefaultModelLoader` |
| index/file traversal was not exercised, and the full model was not constructed. |
|
|
| **SGLang runtime was not executed.** A zero-cost gate against the official |
| `lmsysorg/sglang:kimi-k3` image (linux/amd64 digest |
| `sha256:2e8ef3746b2591287db0f37b4470910ab08974bdaf96fc5820bac35f6d3962bc`) |
| found that its compressed-tensors selector supports FLOAT tensor-group NVFP4 |
| only as W4A4 with input activation scales; it has no matching NVFP4A16 |
| FLOAT tensor-group MoE path for `input_activations: null`. The gate therefore |
| returned `BLOCKED / DO_NOT_RUN_GPU` (`20260727T221943Z`) before weight download |
| or GPU allocation. |
|
|
| Not verified: the full 93-layer / 2.8T model, all 96 shards resident together, |
| `DefaultModelLoader` index/file traversal, KDA/MLA/vision integration, TP16, |
| end-to-end token generation, serving quality, or SGLang load/forward/kernel |
| dispatch. |
|
|
| ## License |
|
|
| Inherits the [Kimi K3 License](https://huggingface.co/moonshotai/Kimi-K3/blob/9f62e4e9fffbd0a83ddd60e1c209d828994b3569/LICENSE). |
|
|
