Whittle-Next-27B-A3B (research preview, shipped with caveats)

What this is, and is not. This is not a finished, general-purpose model. It is a working starting point for anyone who wants to build on the Qwen4-Next (qwen4_exp) architecture — a body that already loads and runs on stock llama.cpp, with hyper-connections, a hashed n-gram memory and per-layer-embedding tensors wired in, plus every piece needed to keep training it (frozen body, trainable checkpoint, table, hash contract, trainer, exporter). Treat the numbers as a baseline to improve on, not a product.

The successor to Whittle-Next-26B-A3B: the same Qwen3.8-Flash-Next-format (qwen4_exp) body built from Qwen3.6-35B-A3B, a 2B-row hashed n-gram memory trained in the memorisation regime it can actually serve, hyper-connection streams that are genuinely different from each other, a LoRA on the experts and attention of layers 2–12 held near the original by a self-anchor, and an on-policy reasoning distill from Qwen3.8-27B on complete thinking traces. Three training phases (v1, v2, v3 below) got it here; this release is the third. Runs on stock llama.cpp, no patches.

Read the caveats section before using it as anything but a research artefact.

	base 25B	Next-26B	Next-27B (this release)
total parameters	25.1B	26.1B	27.1B (25.1B + 2.0B memory)
active per token	~3B	~3B	~3B
GSM8K 200, 512-token cap, thinking off	86.5%	87.0% (4 truncated)	86.5% (173/200, 1 truncated)
GSM8K 200, 1024-token cap, thinking off	–	86.5% (0 truncated)	86.0% (172/200, 0 truncated)
GSM8K 200, 2048-token cap, thinking on	–	–	86.5% (173/200, 3 truncated)
stop probe (12 open replies, 400 cap)	–	10/12	12/12, max 4-gram repetition 0.055
code probes (fib / bash / LRU)	–	–	all three correct; 2 of 3 fenced
held-out CE, chat / corpus	–	1.182 / 2.052	1.219 / 2.084 (teacher on the same corpus rows: 1.826)

All numbers are from the shipped Q8_0 GGUF on stock llama.cpp with the serving sampler below; every GSM8K reply and probe log is in eval/v3/.

How it was built

The three phases are kept here because each one measured something that changed the plan. Every checkpoint, table copy and log of every phase, including the ones that were not shipped, is in train/ on this repo and on the 26B repo.

v1 — the 27B body: table doubled, hyper-connections unlocked (3 Sep)

1. Table 1B → 2B by row replication (bit-identical at the swap: the 26B's weights plus the doubled table is exactly the 26B).
2. Hyper-connection symmetry broken. The identity initialisation makes the four residual streams identical, and every HC weight then receives identical gradients per stream, so gradient descent can never differentiate them. Every earlier HC run was a fair test of nothing. v1 added independent per-stream noise to the mixing and injection weights (cost +0.047 held-out CE at step 0, two thirds recovered by the end) and trained HC at 2× the base learning rate.
3. Distilled 1314 steps (110 min) with the 26B's recipe: corpus forward KL + CE on 2048-token windows, chat CE, on-policy reverse KL on the student's own 64-token replies graded by Qwen3.8-27B, and a code stream (replies up to 384 tokens to prompts with language-tagged fenced references). Expert body frozen.

v1 measured 84.5% / 86.0% on GSM8K at 512 / 1024 (2 and 1 truncated), 9/12 on the stop probe, 1 of 3 code probes fully right. Tried and not shipped (logs under train/27b* on the 26B repo): a six-hour table-only night (introduced maths loops, 15/200 runaways), a code-only phase, and two LoRA-on-everything attempts at 1e-4 to 2e-4 that drifted on both held-out sets within 500 steps. The v1 GGUF is kept on this repo for comparison.

v2 — the memory pulls weight (3 Sep)

v1's 2B-row table had never lowered a held-out number. v2 made it measurably load-bearing and, for the first time, let something downstream of the table learn to read it.

Why the table was inert (measured, not guessed). llama.cpp's PLE path normalises value × gate after gating, so the learned gate is a no-op at inference: every written row injects at unit scale and only an exactly-zero row injects nothing. There is no confidence channel, so a table trained to write everywhere writes noise everywhere. A pruning sweep on the v1 table confirmed the corollary: all of its small benefit lived in a few thousand hot rows.

Sparse commit. v2 trains the table in the memorisation regime it can serve: a fixed 1M-token corpus slice, rows committed only after being visited 6 times (shadow table + visit counts), decay on committed rows, no pruning of the shipped v1 rows.

Adapting the readers. LoRA r8 on the expert and attention linears of layers 2–12 (the layers right after the injection point), lr 3e-5, with a self-anchor term (SelfAug, arXiv 2509.03934): KL(original student ‖ adapted student) at weight 1.0, so the adapted layers may learn to use the memory but not drift from the model they were. Hyper-connections trained throughout.

PLE gain (CE with memory off − CE with memory on)	v1	v2
training slice (text the table has seen)	+0.209	+0.315
held-out chat	+0.021	+0.008
held-out unseen corpus	−0.012	−0.020

Held-out CE moved less than 0.01 (the anchor held). The table became a real memory of what it was shown, at a cost of ~0.02 nats on text it has never seen. v2 shipped as full weights only (bf16-v2/, train/v2/), no GGUF and no behavioural eval.

A metric we got wrong, so you don't. Three earlier LoRA runs were stopped because held-out chat CE rose. The teacher's own CE on those same chat rows is 4.26 against the student's 1.19: under knowledge distillation from Qwen3.8-27B a rising chat CE is the student converging on the teacher's distribution, not damage. Behavioural gates (GSM8K with truncation count, the stop probe, code probes) decide what ships here; SFT CE does not.

v3 — reasoning distill to completion (4 Sep, this release)

v3 is the v2 state plus a short on-policy reasoning distill in which the student writes complete thinking traces and the teacher marks every token of them.

What changed. For every second training step, eight GSM8K or coding prompts are rendered with thinking on; the student samples each reply to its own end (EOS) or a 4096-token cap, in stages of 512/2048/4096 so long replies do not stall the batch; every reply passes format checks (closed think block, one EOS at the end, no stray tokens after it) or is excluded; the 27B teacher, also in thinking mode, then supplies its full-vocabulary next-token distribution at every reply position and the student is pulled toward it (reverse KL, chunked over 512-position blocks). Corpus forward-KL and the memory table continue as in v2, with the layer-2–12 LoRA held by the self-anchor. 101 steps, 141 minutes, 398 complete reasoning replies (231,609 reply tokens) graded, 14 cap-hits, 2 replies excluded by the format checks.

Why completion length. Every earlier on-policy phase capped replies at 64 to 384 tokens and taught the student the shape of truncated answers (the "repetition is failure to stop" finding). Grading whole traces means the decision to stop is graded too, and a runaway is training signal rather than waste.

What moved. Held-out CE barely changed (chat 1.192 → 1.219, corpus 2.082 → 2.084; PLE gains +0.314 / +0.004 / −0.017 on slice / chat / unseen). The behaviour did: stop probe 9/12 → 12/12, GSM8K @1024 truncations 1 → 0, GSM8K @512 84.5% → 86.5%, all three code probes correct, and with thinking enabled GSM8K reaches 86.5% at a 2048 cap. Talk, facts and two-turn name recall are clean. Reverse-KL per token stayed in the 0.13 to 0.38 band across the run with 8 of 8 replies graded on nearly every step.

Trainer changes shipped with v3 (logic65/mini-next-a100-kit/colab/): distill_common.py (chunked full-vocab reverse KL, staged run-to-completion sampling, format assertions, selftest), batched left-padded decoding through the cached hyper-connection forward (asserted equal to the single-row path), reasoning-prompt rendering with the thinking flag verified on both sides, and a checkpoint at the time-budget stop. Known limit: grading an 8k-token reply needs a segmented forward; this run capped at 4096.

Caveats, measured

• Maths is at the 26B's level, not above it. 86.5% vs 87.0% at a 512 cap, 86.0% vs 86.5% at 1024, with 1 and 0 truncated replies in 200. Thinking mode adds no accuracy on GSM8K (86.5%) and costs tokens; 3 of 200 traces hit the 2048 cap.
• The memory is a memory, not general knowledge yet. It lowers CE by 0.31 nats on the 1M-token slice it was trained on, is neutral on held-out chat (+0.004) and costs 0.017 nats on unseen corpus text. Growing the slice is the lever; that is the funded run below.
• The distillation is short. 141 minutes and 398 graded reasoning replies on top of ~4 hours of earlier phases. Published distills run one to two orders of magnitude longer. Expect the teacher's reasoning to show on GSM8K-shaped problems more than elsewhere.
• Teacher voice. Long graded replies pull the student toward the 27B's planning register on some prompts.
• Fencing is inconsistent: 2 of 3 code probes fenced their block.
• Sampling matters. Greedy decoding loops on this family; use the sampler below.

Run it

llama-server -m Whittle-Next-27B-A3B-v3-Q8_0.gguf -ngl 99 -c 8192 --jinja -fa on

Request body: temperature 0.7, top_p 0.8, top_k 20, repeat_penalty 1.05; chat_template_kwargs: {"enable_thinking": false} for direct answers, true for reasoning traces (v3 was distilled with thinking on). Sample, don't decode greedily.

Files

• Whittle-Next-27B-A3B-v3-Q8_0.gguf — the model, stock llama.cpp.
• bf16-v3/ — the same weights in full (qwen4_exp, LoRA 2–12 merged, v3 table). The base for further training.
• train/v3/ — checkpoints every 16 steps with their tables, the final trainable checkpoint (LoRA un-merged), hash contract, logs.
• eval/v3/ — every GSM8K run (512 and 1024 thinking off, 2048 thinking on), probe and server logs.
• History: Whittle-Next-27B-A3B-Q8_0.gguf + bf16/ + train/ + eval/ (v1); bf16-v2/ + train/v2/ (v2).

Train it further

logic65/Whittle-Next-26B-A3B/base-sigmoid/ is the frozen body; train/v3/ here holds everything trained on top, bf16-v3/ has it merged. Trainer, exporter and table tools: logic65/mini-next-a100-kit/colab/ (train_next36.py, distill_common.py, export_next36.py, grow_table.py). The trainer's knobs cover the LoRA (LORA_R, LORA_LAYERS, ANCHOR_W), hyper-connections (HC_PERTURB, HC_LR_MULT), the sparse-commit table (TBL_COMMIT_K, TBL_DECAY, CORPUS_LIMIT_TOKENS) and the on-policy stream (THINK, REASON_PROMPTS, ONPOLICY_MAXNEW, ONPOLICY_STAGES, ONPOLICY_BATCH, KL_CHUNK).

Next step: a full distillation — needs funding

Every run behind this model was a phase test of one to three hours. A full online distillation from Qwen3.8-27B at the budgets the literature uses (roughly 40–50M teacher-graded corpus tokens plus ~10k completion-length reasoning replies, about 40–45 hours on one RTX PRO 6000 Blackwell, ~400 Colab units) is the next step, and it is the step we cannot currently pay for. The pipeline is ready for it: resumable segments with checkpoints and the memory table uploaded as they are written, behavioural gates at every segment boundary, and a teacher-cache job that removes the teacher from every later pass.

Support this work

Whittle runs on one hobbyist's grocery budget and rented GPU hours. If this research is useful to you, or you want the full distillation to happen: ko-fi.com/davida81328 ☕

Authors

David Aylward (logic65) & Claude (Anthropic) — designed, debugged and verified together, one rented RTX PRO 6000 Blackwell, two days and a night.

Provenance

Body: Qwen/Qwen3.6-35B-A3B. Teacher: Qwen/Qwen3.8-27B. Format: Qwen3.8-Flash-Next (qwen4_exp) as implemented in llama.cpp. All Apache-2.0. Part of the Whittle project by logic65.