Mati-3.7-173B-4.6bit-MLX

A REAP-pruned, mixed-precision-quantized MLX build of stepfun-ai/Step-3.7-Flash, built to run comfortably on a 128GB Apple Silicon Mac. Released under the base model's Apache 2.0 license.

~173B logical parameters (201B base, 245/288 experts kept per layer via 15% REAP). 4-bit routed experts; 8-bit shared expert / dense layers / router gates / output head (lm_head) + token embeddings; BF16 vision tower. ~93.5GB on disk, ~4.68 bits/weight effective (not uniform 4-bit -- see Quantization below). A slightly smaller 4-bit-head build (4.658 bpw) is on the shared8-4bit-head branch -- see Variants.

Every claim below is a measured result, not an assumption — see Recipe & measurements for the evidence trail.

Variants (two downloadable revisions)

revision	head precision	bpw	download
main (default)	8-bit lm_head/embed_tokens	~4.68	default — LM Studio and mlx-vlm get this
shared8-4bit-head (branch)	4-bit	4.658	--revision shared8-4bit-head

main is the recommended build: it bumps the output head + token embeddings to 8-bit for a measured text gain (−0.0086 NLL overall, mostly on tool_use/coding) over the 4-bit-head build, at +0.53 GB. The shared8-4bit-head branch is that slightly smaller 4-bit-head build (4.658 bpw); it is the one additionally validated on the 250-image multimodal set (main's head change is text-PPL validated, and quantizing an already-near-lossless head to 8-bit is not expected to affect vision — but it was not separately re-measured there). To fetch the smaller build:

huggingface-cli download True2456/Mati-3.7-173B-4.6bit-MLX --revision shared8-4bit-head
# or in code:  snapshot_download(repo_id=..., revision="shared8-4bit-head")

Most runtimes (including LM Studio) download main unless a revision is specified.

What this is

• Base: Step-3.7-Flash — StepFun VLM, ~201B logical parameters measured directly from the released checkpoint's tensor shapes (commonly quoted as "198B"), 45 decoder layers (dense 0–2, MoE 3–44), 288 routed experts/layer, top-k=8, 1 shared expert, sigmoid router, SwiGLU, RMSNorm.
• Pruned size: 190B of the base's ~201B params live in routed experts. Removing 43/288 per layer (15%) brings the total to **173B logical parameters** -- independent of quantization, which only changes storage size, not parameter count.
• Pruning: REAP at 15% (245/288 experts kept per layer), using a vision-blended saliency — text and image calibration saliency combined per-layer (0.7·text + 0.3·vision, mass-normalized) rather than text-only, to avoid pruning experts that specialize on image tokens. Measured to cut vision-saliency mass lost to pruning by ~31% (9.42% → 6.51%) versus a text-only REAP plan, at no measurable text-quality cost.
• Quantization: affine 4-bit (group size 64) on routed MoE experts; affine 8-bit on the shared expert, the three dense (non-MoE) decoder layers, all router gates, and (on main) the output head + token embeddings — these always-on, non-sparse weight classes turned out to carry almost all of the quantization damage. Vision tower left BF16, unquantized.

Calibration methodology

The REAP saliency underlying this checkpoint's expert-keep decisions was collected in two passes, both on rows drawn from the front of the calibration set (rows 5000+ were held out for evaluation and never touched during calibration -- see below):

• Sample-size validation (5k prompts): an initial saliency collection over 5,000 prompts, compared against a 1,000-prompt subset. Discarded router-saliency mass agreed to within 0.05% between the two -- confirming saliency stabilizes by roughly 1k prompts, well under 5k.
• Final saliency run (2500 prompts): the actual saliency this checkpoint's REAP plan is built from -- 2,500 prompts, max_tokens=1024, headtail truncation (keeps both task setup and answer for long agentic/tool-use prompts, see docs/FINDINGS.md section 3a for why head truncation alone was measured insufficient), 42 MoE layers, ~1.4h runtime. Separately, 300 images (200 ChartQA + 100 VQAv2) were run through the real vision path for the vision-blend saliency (see Pruning below).

2,500 text prompts is roughly 2.5x past the point sample-size validation showed saliency had already stabilized -- not a number picked without checking, and not the same run as the 5k validation pass. Raw saliency scores for every kept/pruned expert are in artifacts/*.json (see Full evidence trail).

Recipe & measurements

All measured on 500 held-out text prompts (rows 5000+ of the calibration set, never used for saliency collection or calibration) and 250 held-out images (ChartQA test split + VQAv2 validation, both disjoint from every calibration/saliency pass -- calibration used ChartQA train and the first 300 VQAv2 rows). head truncation, raw text (no chat-template double-wrapping) -- see docs/FINDINGS.md section 10b for two methodology bugs that were caught and fixed before any of these numbers were trusted.

Text PPL by category

category	BF16	REAP p15, uniform 4-bit	+ vision-blend saliency	+ 8-bit shared/dense/router (this model)
agentic	6.796	7.369	7.078	6.921
coding	8.016	7.540	7.429	6.488
general_instruction	6.184	6.002	6.248	5.115
reasoning_math	2.472	2.513	2.520	2.485
tool_use	43.855	36.662	36.621	27.177
OVERALL	6.653	6.607	6.554	5.930

Multimodal answer-token NLL by category (250 held-out images)

category	REAP p15, text-only saliency	+ vision-blend saliency	+ 8-bit shared/dense/router (this model)	answer tokens
chartqa	9.0017	8.9659	8.9489	438
vqa_natural	6.8574	6.7274	6.6529	141
OVERALL	8.4795	8.4208	8.3898	579

Where the 8-bit precision policy came from

Tested 7 variants against the vision-blend baseline: 6 windows of routed experts at 6-bit (one 7-layer band at a time, spanning the full depth), and one variant bumping the shared expert + 3 dense layers + all router gates to 8-bit. ΔNLL vs. the vision-blend baseline, per category:

variant	target	agentic	coding	gen_instr	reas_math	tool_use	OVERALL
w0	layers 3-9 (experts)	+0.006	+0.126	+0.084	+0.004	+0.243	+0.066
w1	layers 10-16 (experts)	+0.045	+0.052	+0.116	-0.000	+0.055	+0.048
w2	layers 17-23 (experts)	-0.012	-0.001	+0.002	-0.001	-0.006	-0.004
w3	layers 24-30 (experts)	-0.006	+0.005	+0.011	-0.000	+0.020	+0.003
w4	layers 31-37 (experts)	-0.009	+0.001	-0.006	+0.001	+0.009	-0.002
w5	layers 38-44 (experts)	+0.002	-0.002	+0.005	-0.002	+0.006	+0.001
shared8	shared+dense+router	-0.022	-0.135	-0.200	-0.014	-0.298	-0.099

Every depth-localized expert bump landed inside measurement noise, no consistent sign. Only the always-on, non-sparse weight class (shared expert / dense layers / router gates -- weights with no top-k sparsity to smooth over quantization noise the way routed experts get) showed a real, consistent, cross-category improvement -- about 10x any depth-window effect, for roughly +0.03 bpw / +1GB.

Capability validation (exact-answer accuracy, not perplexity)

Perplexity is a proxy: it rewards a confident distribution, which is not the same as a correct one. To check that the numbers above reflect real capability, this model was also scored right/wrong on exact-answer tasks where a good proxy cannot substitute for a correct answer — greedy decoding, all sampler fields pinned:

test	items	score
multi-step arithmetic + factual recall	24	24/24
tool selection + argument extraction (from a provided catalog)	15	14/15

These are spot checks, not a benchmark suite, but they confirm the model reasons and calls tools correctly, not merely that it has low perplexity. Item sets and per-item results: artifacts/acc_*.json, artifacts/tc_*.json.

Also tested and rejected: expert merging (REAM)

Merging the 43 lowest-saliency experts per layer into their nearest kept neighbour (instead of pruning them) lowered perplexity substantially — overall text NLL −0.194, with tool_use perplexity dropping 41% (26.5 → 15.6). On the exact-answer tests above, however, the merged model scored identically (24/24 → 23/24, tool-call 14/15 → 14/15): the perplexity gain was distribution smoothing with no capability behind it. A clean demonstration that perplexity can move without capability following — and the reason pruning, not merging, was shipped. Full write-up in docs/REAM-RESULT.md.

Also tested and rejected: per-expert mixed-precision tiering

Splitting each layer's 245 kept experts into 61 hot (6-bit) / 123 base (4-bit) / 61 cold (3-bit) banks by saliency rank made quality worse (overall text PPL 6.55 -> 7.35, +0.114 NLL) and the checkpoint came out larger (4.868 bpw vs. 4.632) -- the 3-bit cold bank's damage exceeded the 6-bit hot bank's gain. This architecture's expert saliency is too flat (no dead experts, no expendable low-saliency tail) for per-expert precision tiering to pay for itself. Full breakdown in docs/TIERED-EXPERTS-RESULT.md.

Full evidence trail

Every number above, plus the complete methodology, is included in this repo:

• docs/FINDINGS.md -- master findings doc: REAP ratio selection, the vision-blindness discovery, two calibration bugs found and fixed, DWQ investigation (and why it was shelved), quantization format decision.
• docs/PPL-DECOMPOSITION.md -- isolates reap-pruning damage from quantization damage (three-way BF16 / full-4bit / p15-4bit comparison).
• docs/VISION-BLEND-RESULT.md -- the vision-blended saliency result.
• docs/TOMOGRAPHY-RESULT.md -- the full 7-variant precision sweep.
• docs/SHARED8-RESULT.md -- this model's final validation.
• docs/TIERED-EXPERTS-RESULT.md -- the rejected per-expert tiering experiment.
• docs/REAM-RESULT.md -- the rejected expert-merging experiment and the perplexity-vs-capability demonstration behind it.
• docs/TOKENIZER-INVESTIGATION.md -- why the numeric behaviour is intrinsic tokenization and not a checkpoint defect (and why the base tokenizer must not be copied in).
• artifacts/acc_*.json, artifacts/tc_*.json -- per-item results for the exact-answer accuracy and tool-call spot checks, with artifacts/accuracy_eval.py and artifacts/toolcall_eval.py.
• artifacts/*.json -- raw per-category NLL/PPL from every eval run referenced above, and plan_p15_blend03.json, the exact REAP plan (per-layer kept/pruned expert indices and blended scores) used to build this checkpoint.
• artifacts/saliency-text-2500.json, artifacts/saliency-vision-300.json -- the raw per-expert, per-layer REAP saliency from the two source runs (2500 text prompts, 300 images) that plan_p15_blend03.json's blended scores were computed from.

Usage

MLX format — works with LM Studio (search this repo name) or directly via mlx-vlm:

pip install mlx-vlm
python -m mlx_vlm.generate --model <this-repo> --prompt "Describe this image" --image path/to/image.png

Sampling defaults

This repo ships a generation_config.json. The upstream checkpoint has none, which means every runtime falls back to its own defaults — and the generic ones are a poor fit for this tokenizer:

{"temperature": 0.5, "top_p": 0.9, "top_k": 100, "min_p": 0.05, "repetition_penalty": 1.0}

repetition_penalty is pinned to 1.0. It is the only common sampler setting that reorders candidates rather than merely filtering them, so it is the riskiest knob to raise — but note it was measured not to be the cause of the numeric behaviour described under Numeric handling below (a direct A/B at 1.02 vs 1.0 was identical). Keep it at 1.0 as a safe default; use min_p for loop suppression, raising it to 0.1 before reaching for a penalty.

Temperature is deliberately below StepFun's suggested 0.7–1.0. That guidance is tuned for prose diversity; this tokenizer spells numbers one digit at a time (see Numeric handling), which makes numeric output more temperature-sensitive than prose. Raise it for open-ended writing; keep it low for tool calls, structured output, and anything numeric.

Numeric handling (a base-model property, not a defect)

Step-3.7's tokenizer emits one token per digit (ids 19–28 for '0'–'9') plus a standalone space token (223). Numbers therefore carry no token-level redundancy: a single mis-sampled or mis-copied token corrupts an entire number, while prose has enough redundancy to absorb it. In long, number-dense contexts (IP/MAC tables, float-heavy source) this shows up as digits dropped or split by a delimiter — 2456 → 2 4 5 6, 12.0 → 1.2.0, sed -n '447,495p' → sed -n '4.4.7,4.9.5p' — with non-numeric output unaffected.

This is StepFun's tokenization design, not something introduced by the pruning or quantization here. It was investigated at length and each plausible in-checkpoint cause was ruled out with measurements:

• Not the quantized output head. Digit rows carry 1.03x the all-row reconstruction error; the quantization perturbation is 12% of the tightest inter-digit argmax margin.
• Not degraded digit copying. Teacher-forced probes (no sampler in the path) rank the correct next digit first at p = 0.91–0.9996.
• Not the sampler. A direct repetition_penalty A/B (1.02 vs 1.0) was identical.
• Not a reading failure. In the same context the model reports 18452 has 5 digits and echoes it back verbatim.

Do not "fix" this by replacing tokenizer.json with the base checkpoint's. The upstream stepfun-ai/Step-3.7-Flash tokenizer.json is not usable standalone — read directly by the tokenizers library (which is what MLX runtimes do) it yields zero tokens for Chinese (你好世界 → []) and mangles English word boundaries; it only works via transformers' load-time reconstruction. This checkpoint ships the correct, standalone-usable tokenizer.

Practical guidance. Keep temperature low for numeric/tool work; for agentic use, verify numeric arguments programmatically rather than relying on the model to re-check its own output. A helper (scripts/numeric_guard.py) and a Pi editor extension for exactly this are included in the source repo.

Reasoning

Step-3.7 is a <think>-style reasoning model, with one wrinkle that trips up most reasoning parsers.

The chat template opens the think block for you. With add_generation_prompt=True, the prompt ends with:

<|im_start|>assistant
<think>

So the model never emits an opening <think> tag — it generates reasoning immediately and emits only the closing </think>. A parser configured to look for a balanced <think>…</think> pair will not detect any reasoning and will render the raw chain-of-thought as visible output. Configure your reasoning parser to treat the response as starting inside the think block, or to split on </think> alone. Both tags are single tokens (<think> = 128798, </think> = 128799), as are <|im_start|> (128006), <|im_end|> (128007), and <tool_call> (128008).

Reasoning effort is a system-prompt line. If the template is rendered with a reasoning_effort variable, it is interpolated verbatim into a Reasoning: <value> line at the top of the system message. It is not a special token and the checkpoint does not enumerate the accepted values — those come from StepFun's base-model documentation.

Reasoning is dropped from history at the next user turn. The template re-renders the <think> block only for assistant messages after the last real user query; earlier assistant turns are rendered with their reasoning stripped. Note the "last user query" scan skips <tool_response> messages, so reasoning is carried across a multi-step tool-calling loop and discarded only when a genuinely new user message arrives. If you manage history yourself, an assistant message may carry a reasoning_content field; the template falls back to splitting content on </think> when it is absent.

Tool calls are XML, not JSON. The template instructs the model to emit <tool_call><function=name><parameter=key>value</parameter></function></tool_call>. This is a separate path from JSON-schema structured output — see Known limitations.

Known limitations

• Multimodal improvement (vision-blend + shared8 vs. text-only baseline) is measured on 579–600 answer tokens across 250 images — real and consistent in direction across categories, but treat the exact magnitude as indicative rather than precise.
• No BF16 multimodal reference was run, so absolute (not just relative) vision-task damage from quantization remains unmeasured.
• This checkpoint has not been evaluated on standard downstream benchmarks (MMLU, GSM8K, SWE-bench, etc.). The category tables are perplexity/NLL proxies; the Capability validation section adds small exact-answer accuracy spot checks (39 items), but these are not a substitute for a full benchmark suite.
• JSON-schema structured output can crash llguidance-based runtimes. This is a base-model tokenizer interaction, not a defect introduced by the pruning or quantization here — it reproduces with zero model weights loaded, on the stock stepfun-ai/Step-3.7-Flash tokenizer. A merged token decoding to ": collides with grammars that use a ": " key separator (as LM Studio's auto chat-title schema does), and the bitmask generator and token consumer contradict each other. Filed upstream as guidance-ai/llguidance#366. Native XML tool calling (see Reasoning) is unaffected.
• The original checkpoint's MTP (multi-token-prediction) weights (layers 45–47) are not included — this build only implements the main 0–44 decoder stack, matching current mlx-vlm support for this architecture.

Attribution

Base model: stepfun-ai/Step-3.7-Flash, Apache 2.0. This repository is a derivative work under the same license; see LICENSE for the full text. No affiliation with StepFun.