Step-3.7-Flash-oQ3e

Data-driven mixed-precision quantization (oMLX oQ3e, imatrix-calibrated, group size 128) of stepfun-ai/Step-3.7-Flash, in MLX format for Apple Silicon. Runs in mlx-vlm and oMLX.

• ~79 GB on disk (~3.3 bpw text backbone, down from 375 GB BF16)
• Full VLM — step3p7 text backbone (45 layers, MoE with 288 experts / top-8) in oQ3e gs128; vision encoder in 8-bit gs64 (~2.2 GB)
• Peak memory in my tests: ~85 GB text-only, ~87.5 GB with vision — 96GB will do, but you'll want a 128GB Mac for bigger context sizes
• Converted and tested on a MacBook Pro M5 Max 128GB 40 GPU

Quantization

oQ3e allocates bits per layer from an importance-matrix sensitivity pass over calibration data, instead of quantizing everything uniformly — the bulk of the model sits at 3 bits with group size 128, and sensitive layers are kept at 8 bits (group size 64). The vision encoder is quantized separately at 8 bits (group size 64), which is effectively lossless for vision towers. Output is standard MLX affine quantization — no custom kernels or runtime required.

Benchmarks

Benchmark	Score	Samples	Conditions
mmlu_pro	0.84	50	thinking on, greedy
mmlu_pro	0.64	50	thinking off (enable_thinking=false), greedy

At n=50 the standard error is ≈ 0.07 — treat these as indicative, not rankings. I haven't benchmarked vision quality yet; the encoder passed my qualitative smoke tests.

Variants

Variant	Size	bpw	mmlu_pro (n=50, thinking)	Observations
Step-3.7-Flash-oQ3e (this repo)	79 GB	~3.3	0.84	looks like this quant has impressive quality
Step-3.7-Flash-oQ2e	59 GB	~2.4	0.66	smaller and way cheaper on RAM; within noise of the API
Step-3.7-Flash (API)	—	—	0.60	via OpenRouter; provider quant not controlled, but it was probably BF16. I mean, I guess.

I'll run more tests later to make each quant's quality relative to the API clearer.

Thinking control

Step-3.7-Flash is a smart boi. It really likes to think. As a matter of fact, the upstream always enables thinking and doesn't let you turn it off — the chat template unconditionally opens a <think> block, and reasoning_effort barely moves the needle (in my probes, low/medium/high modulate thinking length by ~2×, and the default behaves like max).

I fixed this by implementing the off switch at the template level: when you pass enable_thinking: false (or reasoning_effort: "none"), the template prefills an empty <think>\n</think>\n\n block, so the model skips straight to the answer — no runtime patches, no custom parser, works in any stack that forwards chat_template_kwargs.

You pass	Behavior
(nothing)	Full thinking (default, longest)
reasoning_effort: "low"	Shorter thinking (weak effect, not a hard cap)
enable_thinking: false	No thinking — answers directly
reasoning_effort: "none"	Same as enable_thinking: false

Heads up: skipping thinking costs accuracy on hard tasks (0.84 → 0.64 on mmlu_pro in my runs) — but it also turns a 29-minute eval into a 39-second one. Pick your poison.

Changes from upstream

1. tokenizer_class fixed — upstream declares LlamaTokenizerFast for what is a ByteLevel BPE tokenizer, which breaks decode in transformers (literal Ġ/Ċ in output — it silently corrupted my eval scores until I caught it). This repo sets PreTrainedTokenizerFast, restoring correct encode and decode.
2. Chat template extended — enable_thinking=false / reasoning_effort="none" emit an empty-think prefill (see Thinking control). Everything else is unchanged.

Usage

# text and vision (mlx-vlm — plain mlx-lm doesn't support the step3p7 architecture)
mlx_vlm.generate --model mlx-community/Step-3.7-Flash-oQ3e --prompt "..."
mlx_vlm.generate --model mlx-community/Step-3.7-Flash-oQ3e \
  --image photo.jpg --prompt "Describe this image."

# server — thinking off per request:
# POST /v1/chat/completions
#   {"chat_template_kwargs": {"enable_thinking": false}, ...}

# oMLX — discovers the model from the HF cache
omlx serve