README.md

---
license: other
license_name: kimi-k3
license_link: https://huggingface.co/moonshotai/Kimi-K3/blob/9f62e4e9fffbd0a83ddd60e1c209d828994b3569/LICENSE
base_model: moonshotai/Kimi-K3
base_model_relation: quantized
tags:
- compressed-tensors
- nvfp4
- nvfp4a16
- kimi-k3
- moe
---

# Kimi-K3 NVFP4A16 — Lossless Transcode

Weight-only NVFP4A16 derivative of source revision
[`moonshotai/Kimi-K3@9f62e4e9fffbd0a83ddd60e1c209d828994b3569`](https://huggingface.co/moonshotai/Kimi-K3/tree/9f62e4e9fffbd0a83ddd60e1c209d828994b3569).

## Method

FP4 packed bytes are preserved exactly. Each active (nonzero) MXFP4 E8M0 group-32 scale is duplicated for two NVFP4 group-16 blocks and represented exactly with an E4M3 scale plus a power-of-two global scale. An out-of-window scale attached only to an all-zero group may be canonicalized because its dequantized values remain zero.

Verified on shard 49, expert 0, real K3 w1/w2/w3 weights: all packed bytes are identical and every reconstructed value has zero mismatch and zero maximum absolute error.

- Format: `compressed-tensors`, `nvfp4-pack-quantized`
- Weights: FP4 E2M1, tensor-group strategy, group size 16
- Local scales: FP8 E4M3FN
- Activations: unquantized (`input_activations: null`)
- Quantized modules: routed MoE experts only; all ignored K3 modules remain bit-identical
- Source revision: `9f62e4e9fffbd0a83ddd60e1c209d828994b3569`
- Source provenance: `retroactive-hf-commit-window`
- Lossless over-span policy: `fail; never silently requantize`
- Tensor data size: 1.497 TiB
- Size change versus source tensor data: +5.45%
- Companion artifact: [`GrEarl/Kimi-K3-NVFP4A16-Requantized`](https://huggingface.co/GrEarl/Kimi-K3-NVFP4A16-Requantized)

This is **NVFP4A16**, not calibrated W4A4 NVFP4. Kimi K3 was trained
with MXFP8 activations; FP4 activation quantization would require a separate
calibration and quality-validation run.

## Runtime validation (2026-07-27 UTC)

A **complete Stable LatentMoE block passed on vLLM for this published
artifact** using the official `KimiMoE` class. The test first loaded and ran the
block through a purpose-built layout path, then rebuilt it and loaded the same
8,072 source tensors through the real K3 model-class chain:
`KimiK3ForConditionalGeneration.load_weights` -> `AutoWeightsLoader` ->
`KimiLinearForCausalLM.load_weights` -> `KimiLinearModel.load_weights` -> the
generic `process_weights_after_loading` traversal.

- Evidence run: `20260727T234215Z`
- Earlier routed-expert-core run: `20260727T220141Z`
- Tested artifact revision: `8c22f32993e94e063cb634a03b7fc2f9ff539621`
- Checkpoint shard: `model-00049-of-000096.safetensors` (17,916,197,160 bytes)
- Runtime image: `vllm/vllm-openai:kimi-k3`
- Pinned linux/amd64 image: `vllm/vllm-openai@sha256:fb16b180bd9727600067e16fcd6a6de43fb4db1baf4298ef20b4dbdf6bfa5a0e`
- vLLM: `0.1.dev19262+gb6bbf29dd.d20260727`
- Hardware: NVIDIA B200, SM100; torch `2.13.0+cu130`, CUDA 13.0
- Complete path: FP32 router + correction bias, `7168 -> 3584` latent down,
  896 routed experts with real top-k 16 and `3584 -> 3072 -> 3584`, routed
  RMSNorm + `3584 -> 7168` up, and 2 shared experts (`7168 -> 6144 -> 7168`)
- Loaded source tensors: 8,072 = 8 nonexpert block tensors + 8,064 routed-expert
  quant tensors; all 896 experts
- SiTU: beta `4.0`, linear beta `25.0`
- Backend: `MARLIN` / `MarlinExperts`
- Observed operators: `vllm::moe_forward_shared`,
  `_moe_C::moe_wna16_marlin_gemm`, `_moe_C::grouped_topk`,
  `_C::situ_and_mul`, `vllm_ir::rms_norm`, routing alignment, and MoE sum
- Router logits: `[2, 896]`; selected IDs `[2, 16]`; 16 unique in-range
  experts per token; finite normalized weights
- Both direct and real model-class-loader outputs: `[2, 7168]`, finite and
  nonzero; sampled parameter fingerprints matched exactly and outputs were
  bit-exact (`max_abs_difference=0`)
- Complete-block validation run cost: `$0.341178`; all attempts including the
  externally canceled run: `$3.892933 < $4.5`

This verifies the real K3 outer/inner `load_weights` mapping and generic
post-load traversal on a reduced one-block wrapper. `DefaultModelLoader`
index/file traversal was not exercised, and the full model was not constructed.

**SGLang runtime was not executed.** A zero-cost gate against the official
`lmsysorg/sglang:kimi-k3` image (linux/amd64 digest
`sha256:2e8ef3746b2591287db0f37b4470910ab08974bdaf96fc5820bac35f6d3962bc`)
found that its compressed-tensors selector supports FLOAT tensor-group NVFP4
only as W4A4 with input activation scales; it has no matching NVFP4A16
FLOAT tensor-group MoE path for `input_activations: null`. The gate therefore
returned `BLOCKED / DO_NOT_RUN_GPU` (`20260727T221943Z`) before weight download
or GPU allocation.

Not verified: the full 93-layer / 2.8T model, all 96 shards resident together,
`DefaultModelLoader` index/file traversal, KDA/MLA/vision integration, TP16,
end-to-end token generation, serving quality, or SGLang load/forward/kernel
dispatch.

## License

Inherits the [Kimi K3 License](https://huggingface.co/moonshotai/Kimi-K3/blob/9f62e4e9fffbd0a83ddd60e1c209d828994b3569/LICENSE).