philipjohnbasile/wisp-coder-110m

Wisp Coder 110M

The Fieldbook / No. 001

CODE · CONTEXT · CURIOSITY

Mind
the gap.

A small code model.
A little room for possibility.

Enter the playground ↗

An open research experiment
by Philip John Basile

Small model. Open questions.

100.7M
TRUNK PARAMETERS

108.2M
INCLUDING MTP

2,048
TOKEN CONTEXT

01   Start here 02   Architecture 03   Usage 04   Evidence 05   Limits

❋

01 / START HERE

At the cursor.

THE CODE BEFORE. THE POSSIBILITY WITHIN. THE CODE AFTER.

YOU PROVIDE Prefix⌄ Code before the gap

The source context before the missing span.

<|fim_prefix|>

WISP SUGGESTS Middle⌄ The missing span

The model predicts this span using both sides of the gap. Review its suggestion before use.

<|fim_middle|>

YOU PROVIDE Suffix⌄ Code after the gap

The source context after the missing span. In PSM prompt order, this comes before the generated middle.

<|fim_suffix|>

Provide the code on both sides of a gap. Wisp predicts the middle. Explore it in the playground ↗

A FEW THINGS TO KNOW ALONG THE WAY

⌄ What is it for?

Research on code completion, fill-in-the-middle (FIM), and native multi-token prediction (MTP).

⌄ What did I build?

Philip John Basile's model trained from step zero: 100.7M trunk parameters, 108.2M including the MTP module.

⌄ What runs it?

The trunk uses standard Transformers or MLX-LM. MTP requires the explicit packaged MLX reference runtime; generic loaders do not activate the sidecar.

⌄ What was checked recently?

CPU float32 completions with PyTorch 2.13.0 and Transformers 5.14.1 on 2026-09-10. These are functional smokes, not a coding-quality or latency benchmark.

⌄ How much memory?

Trunk weight file: about 201 MB in BF16, or about 403 MB as FP32 tensors. Runtime overhead and the KV cache require additional RAM; these are weight sizes, not total-memory requirements.

⌄ What are the limits?

2,048-token context; limited completion quality; no chat tuning. The registered short-hole comparison established no superiority, and no production MTP speedup is claimed.

Quick start — ordinary completion on CPU

Run an ordinary completion on CPU

pip install "torch==2.13.0" "transformers==5.14.1"

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "philipjohnbasile/wisp-coder-110m"
revision = "186479fd8c1eb767e0d886014fb78b3426dc509a"  # pinned weights/tokenizer
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)
model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, dtype=torch.float32, trust_remote_code=False,
).eval()
inputs = tokenizer("def is_even(n):\n    return ", return_tensors="pt", add_special_tokens=False)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False,
                            pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Review generated code before using it. The demo also provides FIM prompts and displays raw suggestions without executing them. The full training, data-provenance, and evaluation disclosures follow.

The idea

A small code model built for one job: completing code at the cursor. It targets editor inline suggestions, but no production-runtime latency claim is made.

Two things make it unusual, and they are the reason it exists rather than a feature list.

It does fill in the middle natively. Each tokenized source document was split into chunks of at most 1024 tokens, and 70 percent of those chunks were independently transformed so the model predicts a middle span given both the prefix and the suffix. This is not a per-document or per-window rate. Cursor completion is the primary format, not an inference-time prompt trick added to an ordinary left-to-right checkpoint.

It has multi-token-prediction heads trained in from step zero. A shared MTP module, applied recursively in the Qwen3-Next style, predicts two or more tokens ahead from the trunk's own hidden state. The LM head is shared between trunk and MTP module, which ties the two output distributions to a common projection. That is a useful architectural choice, not a guarantee that draft and target distributions agree; how close they actually are is what acceptance measures.

What is and is not claimed here. Small models with MTP drafters exist: Google publishes gemma-4-E2B-it-assistant and related MTP assistant checkpoints. The artifact contribution is the combination at this scale, a small code model trained from step zero with both fill-in-the-middle on most tokenized chunks and a recursively shared MTP module, plus a paired measurement of how suffix information changes draft acceptance. A targeted search found no exact prior measurement, but that is not proof of absence. No exclusivity or first-of-its-kind claim is made.

❋

02 / WHAT IT IS NOT

Keep the expectations honest.

It is not a chat model, an instruction-following model, or an agent. It completes text. Asking it to refactor a module produces nonsense.

It is also not competitive with Qwen2.5-Coder on raw completion quality, and the arithmetic says it cannot be: Qwen2.5-Coder-0.5B saw roughly 5.5 trillion tokens and Wisp saw 5 billion, three orders of magnitude fewer. If you want the best completions available at small scale, use Qwen. Use Wisp to study FIM plus native speculative drafting on Apple Silicon, or how those two interact.

❋

03 / ARCHITECTURE

Under the petals.

Twelve layers. One shared module. A little further ahead.

01

02

03

04

05

06

07

08

09

10

11

12

12 TRUNK LAYERS

768 hidden dimensions · 12 attention heads
4 KV heads · SwiGLU 2,048

1 Shared.
Reused.

1 MTP TRANSFORMER BLOCK

Trained to depth 2.
Applied recursively at inference.

The trunk and MTP module share the LM head. The sidecar requires the explicit MTP runtime below.

ROUTE A / THE TRUNKAn ordinary Llama decoder.100.7M parameters. Standard Transformers and MLX-LM loading.Inspect the configuration ↗ ROUTE B / NATIVE MTPA shared drafting module.108.2M parameters with the sidecar. Use the explicit MLX correctness runtime.Read the reference runtime ↗

Export manifest ↗   ·   MTP configuration ↗

Parameters	108.2M total, 100.7M trunk
Layers	12, d_model 768, 12 heads / 4 KV heads, SwiGLU 2048
Context	2048, RoPE theta 100k
Vocab	32,768 byte-level BPE, digits split, FIM sentinels
MTP	1 shared module (1 transformer block), trained to depth 2, recursive at inference
Precision	bfloat16 weights, trained with float32 master weights

The trunk is an ordinary Llama decoder. It loads in transformers and mlx_lm with no custom modelling code and no trust_remote_code. The MTP module ships alongside as mtp.safetensors and is ignorable by runtimes that cannot use it. Generic Llama runtimes do not consume the sidecar automatically. The package therefore also ships wisp_mtp_model.py and wisp_mtp_reference.py, a manifest-bound MLX correctness decoder that requires the exact sidecar schema, verifies every declared package hash, exposes the resolved runtime contract and executed route, and fails unless its MTP path reproduces target greedy tokens. It recomputes full prefixes and is not a production speed path.

The export was verified against transformers numerically, not assumed: relative logit delta 2.16e-3 and argmax agreement 1.0000 against the MLX original on the same tokens, which pins the RoPE convention, the RMSNorm epsilon, and the grouped query head order.

❋

04 / TRAINING DATA

Where it grew.

The configured target mixture was 92 percent code from bigcode/starcoderdata and 8 percent HuggingFaceFW/fineweb-edu. This is a source percentage, not a permissive-license percentage. StarCoderData declares license: other, and its original repository terms and relevant attribution clauses remain applicable. FineWeb-Edu is ODC-By 1.0 and remains subject to Common Crawl terms. Run 1 retained no row-level source manifest, so no per-file licensing or attribution guarantee is made.

Training-data provenance notes

Every run 1 document received structural size, line-length, and character distribution filters. Because the loader expected path instead of StarCoderData's max_stars_repo_path, Python AST and JSON extension filtering did not activate for run 1. The schema-aware repair applies only to future source streaming, not to these weights. Corpus content was never executed. Read the published training-data disclosure and machine-checked training-data receipt.

The 32K tokenizer was trained on 400,000 documents sampled round-robin across the eleven source entries, not according to the later token-budget weights. Current shard bytes are fully attested after the build, but the receipt is not a raw-row manifest and cannot prove exact original example boundaries.

The Apache 2.0 metadata describes the released Wisp artifact. It does not override training-source terms or licenses applicable to generated code.

❋

05 / USAGE

Give it a few lines.

Plain completion:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("philipjohnbasile/wisp-coder-110m")
tok = AutoTokenizer.from_pretrained("philipjohnbasile/wisp-coder-110m")
print(tok.decode(model.generate(**tok("def quicksort(arr):", return_tensors="pt"),
                                max_new_tokens=64)[0]))

Fill in the middle. The model was trained with both orderings, evenly split:

# PSM: prefix, suffix, then generate the middle
prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"

Exercise the native sidecar explicitly. This is a correctness check, not a latency benchmark:

import os
import sys
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

package = snapshot_download(
    "philipjohnbasile/wisp-coder-110m",
    local_dir=os.path.abspath("wisp-coder-110m-package"),
)
sys.path.insert(0, package)
from wisp_mtp_reference import WispMTPReferenceRuntime

tok = Tokenizer.from_file(os.path.join(package, "tokenizer.json"))
ids = tok.encode(prompt, add_special_tokens=False).ids
runtime = WispMTPReferenceRuntime.load(package)
result = runtime.verify_greedy_parity(ids, max_new_tokens=16, depth=2)
print(result["mtp_route"])

For evaluation, the trunk can be configured as Continue.dev's autocomplete model while leaving a larger model as the chat and agent backend. Target-runtime batch-1 latency and editor user experience have not yet been measured, so this is not a sub-100 ms claim.

❋

06 / EVALUATION

Follow the evidence.

FUNCTIONAL CHECKS

CPU completions.

CPU completion smokes are documented above. They do not measure coding quality or latency.

SHORT-HOLE COMPARISON

No superiority.

No superiority was established. Read the comparison ↗

MTP RUNTIME

Speed is open.

No production MTP speedup is claimed. The packaged MLX runtime checks correctness.

Registered final results

Complete registered results

Measurement	Result
Final validation main NLL	1.3873 [1.3428, 1.4335], perplexity 4.00
Validation MTP depth 1 NLL	1.5727 [1.5236, 1.6234]
Validation MTP depth 2 NLL	1.6513 [1.6013, 1.7028]
FIM / shuffled-suffix acceptance, depth 2	1.0268 [1.0209, 1.0331], POSITIVE, 200 documents
Trained minus initialized acceptance-ratio lift	+0.0268 [+0.0210, +0.0331], CLEARS_CONTROL
Acceptance interpretation	POSITIVE_AND_CLEARS_CONTROL
FIM-training effect on shuffled-FIM minus L2R acceptance	+0.0814 [+0.0747, +0.0883], POSITIVE, 200 documents
FIM-training effect on true-suffix minus shuffled-suffix acceptance	+0.0049 [-0.0026, +0.0122], NULL: the interval includes zero
Adaptive minus fixed accepted drafts per verification	+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents
Adaptive minus fixed output tokens per target forward	+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents
Adaptive minus fixed drafts issued per output token	+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents (issuance proxy)
Adaptive minus fixed draft recursions per output token	+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents
Rollout policies selected on calibration	adaptive_h0.7 versus fixed_d4
Branch-local greedy replay	Exact argmax for all 38400 emitted tokens across 600 scored policy-document rollouts
Cross-policy output identity	Identical realized output branches for all 100 calibration/test documents across compared policies

Validation intervals measure Monte Carlo uncertainty from the frozen random-window sampler. Acceptance and rollout intervals resample paired target documents. They do not measure training-run or model uncertainty. Null and negative outcomes are retained rather than filtered from the release.

Read the sampler, replay and provenance notes

Rollout endpoint scope: The primary endpoint measures accepted drafts per verification. It does not establish verification-width cost or deployment latency. Draft recursions per output token is the registered drafter-work companion; issued drafts per output token is retained only as an issuance proxy. Target forwards exclude the added post-hoc branch-replay forward and independent verification pass.

Independent replay provenance scope: Unsigned local attestation bound to a pushed pre-execution receipt commit and the registered source, inputs, checkpoint, and argv; it is not a signed external or trusted-execution witness.

Format-ablation provenance limitation: This is repository-revision evidence, not a per-document raw-corpus manifest for run 1. Source drift between the run 1 build and the captured cache refs cannot be ruled out.

Format-ablation runtime limitation: Run 1 did not record source-file hashes or sampler state in its pre-fix checkpoints. A post-build receipt now hashes all 52 current shard files and binds deterministic visible-grammar normalization, but it does not prove that the bytes were unchanged since training began or recover exact original units. The step-300 process recovery reset the legacy sampler, causing 78,643,200 scheduled token positions, 1.57 percent of the training budget, to replay earlier random windows. sampler_reset_steps [300] matches that reset timing in both arms. Later checkpoints preserve exact sampler RNG state. Exact sampled-example equivalence still cannot be proven.

One control worth repeating here, because acceptance rate is easy to report dishonestly: an initialized model scored higher acceptance than a trained one in the E0 instrument pilot (0.482 against 0.331 for a model deliberately overfit on 100k tokens), because two near-uniform distributions agree trivially. Those are pilot diagnostics, not run 1 results. Any final acceptance number without its exact initialized floor beside it is uninterpretable.

❋

07 / LIMITATIONS

Room to grow.

• 5B training tokens. Small, and it shows on unfamiliar APIs, which it will hallucinate confidently.
• 2048 token context.
• Ten languages, weighted toward Python, JavaScript, and TypeScript. Weaker everywhere else, and untested outside that set.
• No safety tuning of any kind. It is a base completion model trained on public GitHub code, and it will reproduce patterns from that data, including insecure ones. Review what it writes.
• No row-level training-source manifest, license mapping, or attribution index. Exact aggregate train-token totals survive in the recovered final build log, but accepted rows, rejection counts, and per-source validation overshoot cannot be reconstructed from the shards.
• The tokenizer document sampler was round-robin by source entry rather than weighted like the training-token mixture.
• Training windows are sampled with replacement. The registered schedule's uniform-interval approximation covers about 62.57 percent of corpus positions at least once, rather than exposing every position once.
• The source-stratified validation shard is not repository- or document-disjoint. A chunked source document can cross the validation-to-training boundary. The separate publication holdout is repository-disjoint.
• The MTP sidecar is not consumed automatically by generic Llama runtimes, and no production-runtime MTP latency or speedup is claimed.
• Trained and evaluated on one machine, one seed, one run.

❋

08 / CITATION

Leave a trail.

@software{wisp_coder,
  title  = {Wisp: fill-in-the-middle and native multi-token prediction in a small code model},
  year   = {2026},
  url    = {https://huggingface.co/philipjohnbasile/wisp-coder-110m}
}

Explore the model guide · All public work

Try the interactive CPU demo · Short-hole evaluation · Selected work

SMALL MODEL. OPEN QUESTIONS.

Stay curious.

Return to the garden ↗   ·   More from Philip

_{The Wisp Fieldbook. Original AI-generated floral artwork shared with the Wisp demo. Artwork notes.}