GrEarl/Kimi-K3-GGUF

Kimi-K3 GGUF — Q2_K experts / Q4_K dense (2.673 bpw)

864.81 GiB in 94 parts. Text model only. Regenerated 2026-07-28 to the llama.cpp PR #26185 (pwilkin/kimi-k3-text) tensor contract.

Source: moonshotai/Kimi-K3 at revision 9f62e4e9fffbd0a83ddd60e1c209d828994b3569 (pinned for every download).

Parts	94 (Kimi-K3-Q2_K-000NN-of-00094.gguf)
Size	864.81 GiB (source 1,453.8 GiB → 0.595x)
Tensors	2,573 (matches the contract exactly)
Effective rate	2.673 bpw
Architecture	kimi-k3
general.file_type	10 (MOSTLY_Q2_K)
Chat template	embedded, validated byte-exact against K3's own builder
imatrix	no (round-to-nearest only)

Quantization layout

ggml type	tensors	size	bpw	what
Q2_K	276	832.04 GiB	2.6250	routed expert stacks (ffn_{gate,up,down}_exps)
Q4_K	1,067	28.58 GiB	4.5000	dense 2-D weights, token_embd
F32	1,112	2.25 GiB	32	norms, router, ssm_a, ssm_conv1d, fused AttnRes scores
Q8_0	1	1.16 GiB	8.5	output.weight
F16	117	0.76 GiB	16	attn_k_b / attn_v_b (row length not a multiple of 256)

The experts hold 2.72 T of the 2.78 T parameters, so they set the overall rate. Dense weights are kept at Q4_K rather than Q2_K: they are 3.3% of the bytes, and spending 4.5 bpw there is cheap insurance for the layers that every token passes through. output.weight is promoted further, to Q8_0, because its error lands directly on the logits with nothing downstream to average it out; it costs 0.54 GiB, or 0.06% of the artifact, and upstream llama.cpp promotes the output head for low-bit file types as well.

Chat template

K3's tokenizer_config.json ships no chat template — conversation assembly lives in encoding_k3.py as code, so nothing could simply be copied across. The template embedded here is Xenova's Jinja port, validated before embedding by rendering it and diffing against K3's own build_chat_segments(): 20 of 20 cases byte-exact, across 5 message shapes (user only, system+user, multi-turn with assistant, CJK plus combining marks, attribute escaping) × 4 thinking settings (off, and on with thinking_effort None / max / low).

One divergence was found and resolved during that check: the Jinja port defaults thinking_effort to 'max' when the caller does not set it, while build_chat_segments() defaults to None and then emits no thinking-effort system message. With the argument passed explicitly on both sides the outputs are identical. The embedded template keeps the Jinja default of 'max'.

Rendering has never been exercised by llama.cpp itself, and the five tested message shapes do not cover tool calls, tool results, or images.

Measured quantization error

MXFP4 source dequantized to F32, requantized to Q2_K, then dequantized again and compared against the source. 92 expert tensors sampled, one per routed layer:

metric	value
relative RMSE	0.3309
cosine similarity	0.951450

For reference, the same measurement on the IQ1_S sibling is 0.5375 / 0.848230. This repo is 1.64x larger and noticeably closer to the source.

The Q2_K encoder is a numpy implementation of block_q2_K (84 B / 256 elements). Its output bytes are fed to gguf-py's own dequantizer and compared against our dequantizer: max absolute difference 0.0, 0 mismatching elements, on both synthetic data and a real K3 expert tensor. Sub-block scale search uses direct min/max rather than ggml's 15-step iterative make_qkx2_quants, so the byte format is identical but quality is marginally below what llama-quantize would produce.

Unrealised headroom in these files

A better sub-block scale search was implemented and measured but is not in this artifact. On a real expert tensor, at identical file size:

sub-block scale search	rel_rmse	cos
min/max (shipped here)	0.3313	0.951268
ggml make_qkx2_quants (16 candidates, MAD, weights=|x|)	0.3057	0.955525
same candidates, squared error, weights=|x|	0.2891	0.959019
same candidates, squared error, unweighted	0.2701	0.962870

Dropping ggml's weights=|x| helps because quantisation error propagates as Δy = ΔW·x, so E‖Δy‖² = Σ ΔW²·E[x²]; the correct weighting is the input second moment — an importance matrix — and the weight's own magnitude has no standing in that expansion. Absent an imatrix, an isotropic-input assumption makes plain ‖ΔW‖_F the consistent objective. Note that this last row optimises exactly the metric reported, so read it as a bound on L2 reconstruction, not as evidence of downstream quality.

Applying it costs 11x more compute per part, measured on production hardware, and was not spent. Sampling caveat: the numbers above come from a single expert tensor of one layer.

Status: not loadable yet

No released llama.cpp can load this. kimi-k3 support is not merged upstream; it exists only in PR #26185. This repo is built to that PR's contract so that it becomes loadable when K3 support lands.

What has been verified, and what has not:

level	status
GGUF structure, split metadata, tensor names/types, hparams, vocab	verified on the published files (headers read over HTTP Range)
Numeric agreement of our quant bytes with gguf-py's dequantizer	verified, exact
Actual llama.cpp model load	not verified — no implementation exists to test against
Token generation / perplexity	not measured

Do not read the structural checks as a runtime guarantee.

What was fixed relative to the previous 96-part upload

The earlier Q2_K release in this repo (96 parts, 938.59 GiB) could not have been loaded even with K3 support present. All four defects required regenerating every part, so this upload replaces it entirely.

1. Tensor names were raw Hugging Face names. 668 names across 92 of the 96 parts exceeded GGML_MAX_NAME (64). Now every name follows the PR contract; longest is 29 characters.
2. Part 1 had no model metadata. No hparams, no vocabulary. Now part 1 carries 66 KV entries: 25 required hparams plus the full vocabulary (163,840 tokens / 163,328 merges, gpt2 / pre kimi-k2, chkhsh verified).
3. split.tensors.count was UINT16 and per-part. llama-model-loader.cpp reads it as a required key and compares it against weights_map.size(), throwing corrupted model on mismatch. Now INT32 / 2573 in all 94 parts, equal to the actual sum of per-part tensor counts.
4. Vision tensors were mixed into the text model. HF shards 95 and 96 hold 168 vision_tower / mm_projector tensors. done_getting_tensors() is called with partial=false, so a single extra tensor triggers wrong number of tensors. Those two shards are excluded, which is why there are 94 parts and not 96.

Conversion details

Recorded in the file itself under kimi-k3.conversion.*:

• a_log_transform = -exp(A_log[:n_head]) — K3 stores A_log as 128 elements but only the first num_attention_heads = 96 are used
• kv_b_split = k_b(transposed)+v_b — kv_b_proj split into attn_k_b / attn_v_b
• attn_res_fused = res_norm*res_proj[0] in float32 — the AttnRes norm/proj pair is fused into one score vector, matching the reference _apply_attn_res()
• expert_stack_dim = 0 — 896 experts stacked on dim 0
• source_quant = compressed-tensors/mxfp4-pack-quantized
• defaults_used = rope_theta — the only value not present in config.json, taken from the configuration_kimi_k3.py class default (10000.0)

Usage (once upstream support exists)

Pass the first part; llama.cpp follows the split metadata to the rest.

llama-cli -m Kimi-K3-Q2_K-00001-of-00094.gguf -p "..." 

All 94 parts must be present in the same directory.

Cost

Regeneration: $4.242 of Modal compute (94 CPU containers, 8 cores / 16 GiB each, no GPU), 15.8 minutes wall clock, 0 failures. Per-part cost came from the returned per-container measurements, summed over the 94 result records.