README.md

---
license: other
license_name: kimi-k3
base_model: moonshotai/Kimi-K3
base_model_relation: quantized
library_name: mlx
pipeline_tag: text-generation
tags:
- mlx
- moe
- reap
- pruned
- kimi
- apple-silicon
---

# Kimi-K3-REAP73-zh-code-MLX-mxfp4-q8

**Kimi-K3, REAP expert-pruned (73%) and converted to MLX** — 451 GB,
sized to actually load on a 512 GB Apple Silicon machine.

Base: [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) — 2.78T total / 104B active,
native-multimodal MoE with Kimi Delta Attention, Attention Residuals and a 1M
context.

## Why this exists

Unpruned Kimi-K3 does not fit on any Mac. The full model is 1.56 TB; even a
2-bit quant is ~870 GB against a 512 GB ceiling. Fitting 4-bit into 512 GB would
need ≤1.38 bits/weight.

REAP ([Cerebras](https://github.com/CerebrasResearch/reap)) scores each expert by
`gate x ||expert_output||` over a calibration set and keeps the most salient.
This build keeps **242 of 896 experts per layer** across 92 MoE
layers → **793B params, 451 GB**.

## Precision: mxfp4, and it is lossless

K3's routed experts ship from Moonshot as **MXFP4** (`weight_packed` + e8m0
`weight_scale`, group 32). MLX's native `mxfp4` mode uses the identical encoding,
so the surviving experts here are a **bit-exact byte copy of the source** — the
only information lost in this repo is the pruning itself, not the quantization.

Requantizing those same weights to affine 4-bit would cost ~9.8% mean relative
error *and* be larger (4.5 vs 4.25 bits/weight), so no affine 4-bit tier is
published.

Non-expert tensors (mxfp4 global config; attention, shared experts,
latent projections, embeddings) are carried at higher precision.

## Measured behaviour

Loaded on a 512 GiB M3 Ultra (62 s, 451 GB peak). Verbatim, greedy, unedited:

```
prompt: 'def merge_intervals(intervals):\n    """Merge overlapping intervals."""\n'
--> if not intervals:
        return []
    intervals.sort(key=lambda x: x[0])
    merged = [list

prompt: 'The capital of France is'
--> Paris",
+    "The capital of Germany is Berlin",
    "The capital of Italy is Rome",
    "The

prompt: '机器学习的基本原理是'
--> ：通过训练数据，学习算法，然后对未知数据进行预测。机器学习的过程是：输入数据→学习算法→
```

Samples above are verbatim, greedy-decoded, and unedited — including the failure
modes. Degradation from the pruning shows up as drift into list-like or
source-file-like continuations rather than answering directly, and at heavier
prune ratios as outright repetition loops.

Code completions stay structurally correct across every ratio tested,
which matches the calibration data: code experts form a dense, self-similar
cluster (57% self-overlap in a top-242 set, versus a 27% chance baseline) and so
survive pruning better than more diffuse language capability.

## Speed — read this before downloading

**~0.14 tok/s** on a 512 GiB M3 Ultra. Not a typo, and not interactive.

Each decoded token reads roughly 87 GB of weights: 25.8 GB of routed
experts plus 60.8 GB of non-expert weights, which every token touches. This is a
bandwidth wall, not a headroom problem -- a 350 GB build with 160 GiB of spare
memory measured 0.20 tok/s versus 0.16 for a 451 GB build, i.e. gains track size
almost exactly. No prune ratio makes K3 interactive on this hardware.

Note the non-experts dominate per-token traffic despite being ~2% of parameters:
all of them are read every token, while only 16 of 242 experts are.

## Quality expectations — read this

This is an **aggressive** prune. Top-16 routing over 242 experts is
6.6% density, comparable to a REAP-50 of a 256-expert model. Expect
noticeable degradation versus full K3. It is the "fits on one machine" build, not
a quality build.

Two things work in its favour that a plain expert cull would not have: K3 keeps
**2 shared experts** that fire on every token regardless of pruning, and its
LatentMoE applies RMSNorm to the *combined* expert output, which partially
self-corrects the magnitude lost when experts are removed.

## Calibration — targeted at Chinese + code

This build is calibrated on **Chinese and code only**, not the full mixed
corpus. Kimi-K3's experts cluster by domain — measured over a top-242 set
against a 27% chance baseline, code-python↔code-multi overlap is 57.2% while
chinese↔code-python is 17.8%, i.e. *below* chance — so dropping the languages
you do not need frees expert slots for the ones you do. Saliency retained rises
from 59.1% (mixed) to 69.3% here at identical size.

The tradeoff is real: Japanese, Korean, Russian, Arabic, German, French and
Spanish are materially degraded relative to the mixed build. Use
`Kimi-K3-REAP73-MLX-mxfp4-q8` if you need those.

Measured against the mixed build on the same prompts: Chinese improves (the
mixed build drifts into restating the prompt by ~18 tokens; this one does not),
and code is unchanged.

Saliency was measured on a deliberately mixed 12.6 MB corpus — 40% code
(multi-language + real Python), 30% English web, 15% Chinese, 15% across
ja/ru/ko/de/fr/es/ar. The mix matters: whatever a calibration corpus
under-represents gets pruned away silently. An earlier attempt using C4's pooled
`multilingual` config left CJK at 0.03% of the corpus, which would have quietly
removed the experts handling Chinese.

## Usage

Requires **mlx-lm** plus the bundled `kimi_k3.py` loader (the architecture is not
upstream yet). On a 512 GB machine you must raise the GPU wired limit first —
the default is ~75% of RAM, below this model's footprint:

```bash
sudo sysctl iogpu.wired_limit_mb=480000
pip install mlx-lm

python - <<'PY'
import os, shutil, mlx_lm
from huggingface_hub import hf_hub_download
for f in ("kimi_k3.py",):
    dst = os.path.join(os.path.dirname(mlx_lm.__file__), "models", f)
    shutil.copy(hf_hub_download("pipenetwork/Kimi-K3-REAP73-zh-code-MLX-mxfp4-q8", f), dst)
PY

mlx_lm.generate --model pipenetwork/Kimi-K3-REAP73-zh-code-MLX-mxfp4-q8 --max-tokens 256 \
    --prompt "Write a Python function that merges overlapping intervals."
```

`kimi_k3_vision.py` and `kimi_k3_vl/` ship alongside for the vision tower; the
image path needs mlx-vlm and is not exercised by `mlx_lm.generate`.

## Provenance

Converted with [PipeNetwork/kimi-k3-mlx](https://github.com/PipeNetwork/kimi-k3-mlx):
a streaming converter (the model never fits in memory at any stage) and a
streaming REAP calibration harness. Weights remain under the Kimi K3 License.