Kimi K3 Model Overview: 2.8T Parameters, MXFP4 Quantization, and What the Open Weights Mean for the Community
A structured overview for researchers and MLOps practitioners
Moonshot AI publicly released Kimi K3 on July 16, 2026, with full open-source weights promised by July 27. At 2.8 trillion parameters, it is the first open-source model to reach the 3-trillion-parameter class. This article provides a structured model overview for the research and MLOps community.
Model Card Summary
| Field | Value |
|---|---|
| Developer | Moonshot AI |
| Model type | Autoregressive Mixture-of-Experts (MoE) transformer with native vision |
| Total parameters | 2.8 trillion |
| Active parameters | ~50B equivalent (16/896 experts per token) |
| Context length | 1,000,000 tokens |
| Training precision | Mixed (MXFP4 weights, MXFP8 activations) |
| Modalities | Text + Vision (native, not adapter-based) |
| Reasoning | Always-on thinking mode |
| License | Open weights (license TBD with weight release) |
| Release date | July 16, 2026 (API); July 27, 2026 (weights) |
Architecture
Kimi K3 introduces several architectural innovations that collectively yield approximately 2.5x scaling efficiency improvement over its predecessor K2.
Kimi Delta Attention (KDA)
A hybrid linear attention mechanism that replaces standard quadratic attention in a subset of layers. KDA maintains full expressiveness for critical layers while reducing the computational cost of attention across the model's 1M-token context window. For a detailed treatment of KDA's design and its relationship to prior linear attention work, see the architecture deep dive on Hashnode.
Attention Residuals (AttnRes)
A drop-in replacement for standard residual connections. Rather than uniformly accumulating layer outputs, AttnRes allows each layer to selectively retrieve representations from arbitrary earlier layers. This is particularly impactful in MoE architectures where different experts activate at different depths.
Stable LatentMoE
The MoE framework that manages 896 experts with 16 active per token. Key components include latent-space routing, Quantile Balancing for load management, and soft dropping for overflow tokens. For the full technical analysis, see the MoE deep dive on Zenn.dev.
Additional Components
- Per-Head Muon optimizer: Learning rate scheduling at the individual attention head level
- Sigmoid Tanh Unit (SiTU): Custom activation function replacing standard GeLU/SwiGLU
- Gated MLA: Gated Multi-head Latent Attention for memory-efficient KV-cache management
Quantization: MXFP4 Weights and MXFP8 Activations
K3 employs quantization-aware training (QAT) starting from the supervised fine-tuning stage, not post-training quantization. This is a critical distinction — the model learns to compensate for quantization error during training, resulting in significantly less quality degradation.
The format choices are deliberate:
- MXFP4 weights (Microscaling FP4): Each weight is stored in 4-bit floating point with per-block scaling factors. MXFP4 is supported natively by NVIDIA Blackwell GPUs and AMD MI400 accelerators, making deployment hardware-friendly.
- MXFP8 activations: Intermediate activations use 8-bit floating point, providing higher precision where it matters most for gradient flow during training and numerical stability during inference.
Deployment Implications
The MXFP4 weight format means the full 2.8T model requires approximately 1.4 TB of weight storage — substantially less than the ~5.6 TB that FP16 weights would demand. This brings self-hosting within reach of organizations with multi-node GPU clusters (e.g., 8-16 nodes of 8x H100/B200).
Moonshot's own serving uses the Mooncake disaggregated inference infrastructure, which separates prefill and decode across different node pools and achieves a reported 90% cache hit rate on coding workloads. This architecture enables the aggressive cached input pricing of $0.30/MTok.
Benchmark Results
General Intelligence
| Benchmark | K3 | Fable 5 Max | GPT-5.6 Sol Max | Opus 4.8 |
|---|---|---|---|---|
| GDPval-AA v2 | 1,687 | 1,815 | 1,747.8 | 1,600 |
| GPQA-Diamond | 93.5 | — | — | — |
| MMMU-Pro | 81.6 | — | — | — |
| MathVision | 94.3 | — | — | — |
Coding
| Benchmark | K3 | Notable |
|---|---|---|
| DeepSWE | 67.5 | |
| FrontierSWE | 81.2 | |
| Terminal-Bench 2.1 | 88.3 | 0.5 pts behind GPT-5.6 Sol |
| Program Bench | 77.8 | Best overall |
| SWE Marathon | 42.0 | Best overall |
Agentic
| Benchmark | K3 |
|---|---|
| BrowseComp | 91.2 |
| DeepSearchQA (F1) | 95.0 |
| Automation Bench | 30.8 |
K3 leads all models on SWE Marathon and Program Bench, suggesting particular strength in sustained coding sessions — consistent with the 1M-token context window enabling full-repository understanding.
What Open Weights Mean for the Community
Research Opportunities
Expert specialization analysis: With 896 experts, researchers can probe whether experts specialize by language, domain, task type, or some other dimension. This data could inform future MoE designs.
Pruning and distillation: Can the 896-expert pool be reduced to 256 or 128 experts for deployment on smaller clusters? What is the quality-compute tradeoff curve?
Attention Residuals ablation: AttnRes is described as a "drop-in replacement" for residual connections. Testing this in smaller models could validate whether the technique generalizes beyond K3's scale.
Fine-tuning dynamics: How does LoRA or QLoRA interact with an already-quantized MoE model? The MXFP4 base presents novel challenges for adapter-based fine-tuning.
MLOps Considerations
- Hardware requirements: Minimum 1.4 TB aggregate GPU memory for weight loading. Practical minimum: 8-node cluster with 8x 80GB GPUs each (5.12 TB total, leaving headroom for KV-cache and activations).
- Inference framework: Likely requires vLLM, TensorRT-LLM, or SGLang with MoE-aware scheduling. Standard frameworks may need patches for the 896-expert routing.
- Serving economics: At self-hosted scale, the MXFP4 quantization reduces memory bandwidth requirements by ~4x vs. FP16, making the cost structure viable even for medium-sized organizations.
Known Limitations
Moonshot acknowledges three areas where K3 falls short:
- Thinking history sensitivity: Agent harnesses that truncate or modify the model's chain-of-thought can cause significant quality degradation.
- Excessive proactiveness: In ambiguous scenarios, K3 tends to act rather than ask for clarification — a common MoE trait where specialized experts "overfit" to action-oriented training data.
- UX gap: Despite benchmark parity, the subjective user experience still trails Claude Fable 5 and GPT-5.6 Sol in conversational polish.
Conclusion
Kimi K3 represents a significant milestone: frontier-level performance in an open-source package. The combination of Stable LatentMoE, KDA, and quantization-aware training makes the 2.8T parameter count economically viable for both API serving and self-hosting.
The weight release on July 27 will be the real test — whether the community can reproduce Moonshot's benchmark numbers, adapt the model for domain-specific use cases, and advance the underlying architectural innovations.
For a broader analysis of what Kimi K3 means for the open-source AI ecosystem, see this comprehensive overview on wan27.org.
