# Kimi K3 — Evaluation Summary > Where Kimi K3 actually lands: public benchmarks, in-house suites, cyber-capability evaluation, > third-party results, and cost efficiency. All figures from the technical report unless marked > **[verified]**. Baselines: Claude Fable 5, GPT-5.6 Sol, Claude Opus 4.8, GPT-5.5 (xhigh), and > GLM-5.2 (open weight). **The one-line summary from the report itself:** K3 **trails the strongest proprietary systems overall — Claude Fable 5 and GPT-5.6 Sol — and is consistently ahead of every other open and proprietary model in their suite.** Evaluation configuration: K3 is evaluated at **reasoning effort = max, temperature = 1.0**. Single-step tasks and vision benchmarks without tools use **top-p = 0.95**; agentic tasks use **top-p = 1.0**. The report's general recommendation is top-p = 0.95 for reasoning and knowledge, top-p = 1.0 for coding and agentic scenarios. Results for Claude Fable 5 include fallback behaviors, and GPT-5.6 Sol results include potential cyberguards. --- ## 1. Public benchmarks ### Reasoning & Knowledge | Benchmark | Kimi K3 | Best baseline | |---|---|---| | GPQA Diamond | 93.5 | **94.1** (GPT-5.6 Sol) | | AA-LCR | **74.7** | 73.7 (GPT-5.6 Sol) | | HLE-Full (no tool / tool) | 43.5 / 56.0 | **53.3 / 63.0** (Fable 5) | | CritPt | 23.4 | **32.3** (GPT-5.6 Sol) | Competitive at graduate-level reasoning; **a real gap remains on research-level tasks**. The report is explicit that **research-level reasoning remains a key direction for improvement**. ### Coding | Benchmark | Kimi K3 | Note | |---|---|---| | ProgramBench | **77.8** | best | | SWE-Marathon | **42.0** | best — **7 points ahead of Fable 5**; a GPU-kernel-oriented suite | | Terminal-Bench 2.1 | 88.3 | nearly matches GPT-5.6 Sol (88.8) | | FrontierSWE | 81.2 | second, behind Fable 5 (86.6), well ahead of all others | | DeepSWE | 67.5 | behind Fable 5 and GPT-5.6 Sol, ahead of Opus 4.8 and GPT-5.5 | | PostTrainBench / MLS-Bench-Lite / SciCode | 36.6 / 48.3 / 58.7 | — | ### Agentic — the broadest area of strength State of the art on: **BrowseComp 91.2**, **DeepSearchQA (F1) 95.0**, **ResearchRubrics 76.2**, **MCPMark-Verified 94.5**, **AutomationBench 30.8**, **SpreadsheetBench 2 34.8**, **τ³-Banking 33.4**, **Harvey Lab-AA 94.6** (criterion pass rate). The main exceptions are **knowledge-work suites, both led by Claude Fable 5**: GDPval-AA v2 (K3 third at 1,686 Elo) and AA-Briefcase (K3 second at 1,548). Elsewhere it is largely competitive — CorpFin v2 (71.6 vs 71.8) and OSWorld-Verified (84.8 vs 85.0) finish within 0.2 points — while the harder computer-use benchmarks (**OSWorld 2.0, SaaS-Bench**) are still led by Fable 5 or GPT-5.6 Sol. ### Vision | Benchmark | Kimi K3 | Note | |---|---|---| | OmniDocBench | **91.1** | best | | Video-MME (w/ sub) | **90.0** | best | | MMVU | **82.1** | best | | Math-Vision (no tool / Python) | 94.3 / **97.8** | — | | ZeroBench-main (pass@5, no tool / Python) | 23.0 / **41.0** | ties Fable 5 without tools | | WorldVQA ForceAnswer | 51.0 | second, behind Fable 5 (56.7) | | MMMU-Pro / CharXiv (RQ) | 81.6 / 83.4 · 84.8 / 91.3 | — | **The tool-augmentation delta is the interesting part.** Math-Vision +3.5 and ZeroBench-main **+18.0** with a Python tool indicate that a substantial fraction of K3's visual capability is unlocked by **writing code to look at the image**, rather than by pure forward perception. This corresponds directly to the "multi-step verifiable visual reasoning" RL environment (report §4.2.3), where trajectories are generated with a Python interpreter in the loop and execution outputs — including generated images — are fed back as new observations. --- ## 2. In-house evaluation The internal suite separates strengths from weaknesses more sharply than the public benchmarks. - **Clearest strengths: orchestration- and research-type agency.** K3 leads **Swarm Bench (76.3)** and **Deep Research Bench (90.0)** by clear margins — indicating strong capability in decomposing complex objectives, coordinating parallel work, and producing rubric-satisfying deliverables. - **Coding.** On Kimi Code Bench 2.0 it trails only Claude Fable 5, and it attains the **best score on Coding Experience (59.9)** — suggesting its practical behavior as a coding agent (communication quality, behavioral appropriateness, instruction-following stability) is **ahead of its raw test scores**. - **Professional knowledge work has improved markedly** over the previous generation, with Finance Bench essentially tied with GPT-5.6 Sol. - **Kimi Webdev Bench** (blind expert judging vs Claude Opus 4.8, both under the Claude Code harness): | Domain | Win | Tie | Lose | Win − Lose | |---|---|---|---|---| | Games | 55.6% | 3.7% | 40.7% | +14.9 | | **3D / WebGL / Shader** | 72.7% | 13.7% | 13.6% | **+59.1** | | Website / UI Clone | 52.6% | 21.1% | 26.3% | +26.3 | | **Overall** | 58.6% | 13.8% | 27.6% | **+31.0** | - **Where it trails**: Agent Behavior Bench, MIRA Bench, 24/7 ClawBench 2.0, Agentic Vision Bench, and KWV Bench. On the remaining filled suites (KAET, CLIF Bench, Online Experience, DECK Bench, Faithfulness, Chat All-in-One Bench) it ranks first or a close second. --- ## 3. Cyber-security evaluation Evaluated along a two-tier progression of increasing operational risk. Reported factually here; the report itself frames these as **a lower bound on capability**, conditioned on the current model version and evaluation coverage. **Tier 1 — vulnerability discovery** (identifying genuine bugs in current codebases and demonstrating reproducibility; primarily associated with defensive security research). Across dozens of widely deployed systems spanning OS kernels, databases, AI services, web frameworks, blockchain, and VPN software, the model identified hundreds of candidates. **Of the findings that underwent human review, approximately 70% were confirmed genuine, including 16 previously unknown vulnerabilities across six projects.** Two Linux kernel findings are cited: a remotely triggerable heap out-of-bounds write introduced by an incomplete upstream fix (confirmed by security experts as a remote denial-of-service primitive), and a Dirty-COW-class vulnerability in the RDMA subsystem where an earlier upstream fix had inadvertently dropped a permission check (confirmed as a deterministic local privilege-escalation primitive). **Tier 2 — exploit development** (36 in-house tasks: 16 user-space, 20 Linux kernel). Every task is **verified solvable by human security experts**; completing the full suite is estimated at roughly **540 expert-hours, about 15 hours per task on average**. | | Kimi K3 | GLM-5.2 | |---|---|---| | Tasks solved | **14 / 36 (38.9%)** | 8 / 36 (22.2%) | Successes are **unevenly distributed: 10 of K3's 14 come from the user-space track.** On the kernel track, **neither model solves three-quarters of the tasks.** Trajectory analysis attributes the remaining gap to human-expert capability to four recurring failure modes: (i) difficulty completing the final stage of an exploit chain from primitives already obtained; (ii) poor strategy selection under mitigations; (iii) getting trapped in prolonged, unproductive debugging loops; and (iv) insufficient verification of the final deliverable before submission. **Independent assessment.** A joint assessment by the UK AI Security Institute and NIST's Center for AI Standards and Innovation (CAISI) reaches consistent conclusions: K3 outperforms GLM-5.2 on exploit development (**32% vs 24% on ExploitBench**; **17 vs 11 steps** on a 32-step simulated enterprise network that takes a human expert roughly 20 hours), but **trails frontier cyber-capable models on end-to-end exploit completion, achieving arbitrary code execution on 0 of 41 tasks.** Note on comparability: frontier models from Anthropic and OpenAI refuse cyber-related tasks, making a comparable evaluation infeasible; they are excluded from this suite. --- ## 4. Third-party evaluation (as of 2026-07-23) | Source | Result | |---|---| | **Artificial Analysis** | Intelligence Index v4.1 = **57.1**, ranking **#4 of 580 models** — third if GPT-5.6 Sol's effort variants are counted as a single entry — behind Fable 5 (59.9) and GPT-5.6 Sol (58.9), **ahead of all other evaluated models** | | **Vals AI** | GDP-weighted industry benchmark suite: Vals Index **74.7**, **#2 of 39**, behind Fable 5 (75.1), ahead of GPT-5.6 Sol (73.1) | | **LMArena** | **WebDev Arena 1,678 Elo, #1 of 99 — the first open model to top this leaderboard.** Text Arena 1,486 Elo, #8 of 200. Agent Arena 9.1, #4 of 37 (behind Fable 5 at 12.7, GPT-5.6 Sol at 10.1, Opus 4.8 at 9.8) | --- ## 5. Cost efficiency Score against per-task cost across four suites. For Kimi Code Bench 2.0, costs are measured internally with K3 run via Kimi Code and all other models via Claude Code. For BrowseComp, K3's cost is measured from their own runs while Claude and GPT costs are cited from published charts. GDPval-AA v2 and AA-Briefcase costs are cited from Artificial Analysis's pay-per-token API pricing. | Suite | Result | |---|---| | **Kimi Code Bench 2.0** | 4.0 points behind Fable 5 at **38% of its cost**; at *high* effort it already **matches Claude Opus 4.8's maximum-effort score at roughly one third of the cost** | | **BrowseComp** | Best score **91.2% at $2.03 per task** — **half the cost of GPT-5.6 Sol** (90.4%) and **an order of magnitude cheaper** than the Claude models at maximum effort | | **GDPval-AA v2** | Within 50 Elo of GPT-5.6 Sol at **13% lower cost**, and **2.6× cheaper than Claude Fable 5** | | **AA-Briefcase** | Second-best score behind Fable 5, at **roughly half the latter's cost** | **Overall, K3 sits on or near the cost-efficiency frontier across all four suites — near-top scores at a fraction of the cost of Claude Fable 5 in particular.** --- ## 6. Case studies (report §7) Five demonstrations, reported here as capability evidence: 1. **GPU kernel optimization.** Each model works independently in an identically configured sandbox with a budget of up to 24 hours per task, across four kernels (AttnRes, DeepSeek Sparse Attention, KDA, and MLA with head dimension 512), on an NVIDIA Hopper GPU and an alternative-vendor GPGPU. K3 **reduced AttnRes latency from 283.6 ms to 114.4 ms, cut DSA and KDA runtime by 55.1% and 73.6%**, and reached **over half of peak TFLOPS on MLA**. It **matched Claude Fable 5 (with fallback)** and substantially outperformed Opus 4.8, GPT-5.6 Sol, and GPT-5.5. Notably, **an early K3 checkpoint was already handling most of the kernel optimization work during late-stage development.** 2. **MiniTriton** — a compact Triton-like compiler K3 developed: custom tile-level Python frontend and layout system, a lightweight warp-level MLIR annotation and optimization layer, and a PTX code-generation pipeline, plus a dual-mode tensor library with reverse-mode autograd, neural-network modules, distributed training primitives over NCCL, and sparse and visualization primitives. On an NVIDIA L20 it **outperforms PyTorch eager and `torch.compile` in geometric mean** over its core benchmark suite; its from-scratch tensor-core matmul **approaches cuBLAS at the largest shapes, reaching about 90% of the measured machine roofline**; its DSL-level KDA prefill kernel outperforms a matched Triton reference by a clear margin. It also trains a GPT end to end with a loss curve closely tracking the PyTorch reference, with **full-model gradients differing from torch autograd by no more than torch's own fp32 rounding error (10⁻⁴)**, measured against an fp64 reference. Open source, Apache-2.0. 3. **Chip design (nano-kpu).** K3 designed an inference-chip prototype for a nano model **following the same architecture as itself** — hybrid KDA and NoPE-MLA attention, Block AttnRes with a block size of two, sigmoid-based MoE routing with one shared expert, under group-wise INT4 weight quantization (group size 128). In **a single 48-hour autonomous run with Kimi Code**, K3 built, optimized, and verified the chip using open-source EDA tools with the Nangate45 standard-cell library. Within a **4 mm² analytical area budget, the design closes timing at 100 MHz** and achieves an **RTL-simulated decode throughput of over 8,700 tokens/s**, integrating **1.46M standard cells, 0.277 MiB of SRAM, and an INT4 MAC array with fused dequantization**. Open source, Apache-2.0. 4. **Coding for research.** To reproduce the I–Love–Q universal relations in computational astrophysics, K3 **reviewed more than 20 papers and cross-validated their results, implemented the full numerical pipeline, evaluated over 300 equations of state, identified inconsistencies in published formulas, wrote more than 3,000 lines of Python, and produced an interactive HTML dashboard — in about two hours**, versus a typical one to two weeks for an experienced researcher. 5. **Knowledge work.** In Kimi Work, K3 produced an interactive research website covering **42 years of the AI ASIC industry**, completing more than **120 rounds of iterative refinement**, drawing on a corpus of **87 quarterly reports and 99 original PDFs (more than 11,000 pages)** through over **2,800 web searches** and over **1,100 terminal queries**. In a second case, K3 analyzed **391 gravitational-wave events in GWTC-5** using more than **20 concurrent subagents**, producing seven scientific visualizations, two summary tables, and a literature synthesis of over ten papers. 6. **Video editing and motion design.** Leveraging its native multimodal architecture, K3 created a 3Blue1Brown-style motion-graphics explainer **of its own architecture**, and edited its teaser video from **56 source clips** — clip selection, motion-matched cuts, frame-accurate beat synchronization, audio processing, and multiple rounds of revision. Producing a comparable high-density short video would typically take an experienced editor one to two days.