Quazim0t0/Byrne-86M-JL

Byrne-86M

🔭 Jet-Long context extension (native 4K → 10K)

This is the Jet-Long edition of Byrne-86M. It extends the usable context from the native 4,096-token training window to 10,240 tokens with no fine-tuning and no change to short-context behaviour, by adding dynamic bifocal RoPE from Jet-Long (arXiv:2607.07740, NVIDIA).

What was applied

Jet-Long pairs a local window (w0 = 2048, classic RoPE) with a remote window whose position map aliases far-apart tokens back onto the pretrained rotation grid:

f(x) = floor(x / G),   G = max(1, ceil(L / 4096))

G adapts to the current sequence length L, so:

• L ≤ 4096 → G = 1 → f is the identity → the model is bit-for-bit the base model. (Verified: max |Δlogit| between Jet-Long on/off within the window is 0.000e+00.)
• L > 4096 → the remote window keeps every rotation in-distribution, so the model extrapolates instead of collapsing.

Implementation notes specific to this SpikeWhaleLM build:

• Only the decoupled RoPE partition (16 of 64 head dims) is aliased; the NoPE partition is untouched. Softmax attention (use_derf=False) — the standard Jet-Long merge applies.
• The remote view is realized by an on-the-fly correction rotation on the already-RoPE'd KV cache (RoPE composes additively), so the cache is never rewritten and decode is cheap.
• Enabled via config: use_jetlong=true, jetlong_w0=2048, jetlong_w_pretrained=4096, max_position_embeddings=10240. Set use_jetlong=false to recover the exact base model.
• The inclusion–exclusion / CuTe throughput kernel from the paper is not included (it targets 100K+ contexts on H100); at 86M params the bifocal attention is computed directly.

Measured (PG-19-style perplexity on held-out text, lower is better)

Context length	Base model	This Jet-Long model
≤ 4,096 (in-window)	(identical — Jet-Long is a no-op)	(identical)
10,240	48.00	11.36

Beyond the training window the base model's perplexity blows up while Jet-Long stays flat — and long-context generation stays grammatical where the base model degrades into word-salad.

Usage

Jet-Long is on by default in this repo. Pass explicit position_ids so RoPE gets true absolute positions during cached decode:

import torch
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M-JL", trust_remote_code=True)
ids = ...              # up to ~10,240 tokens
pos = torch.arange(ids.shape[1]).unsqueeze(0)
out = m(input_ids=ids, position_ids=pos, use_cache=True)   # prefill, then decode step-by-step

Method: Tang, Wang, Gu, Han, Cai — “Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE”, arXiv:2607.07740. Applied here zero-shot to SpikeWhaleLM; no weights were retrained.

The main Byrne-86M chat model (OPD v2) — the recommended model for general use. A ~86M-parameter, from-scratch SpikeWhaleLM decoder (Multi-head Latent Attention, n-gram engram memory, hash-lookup layers, hyper-connections, HRM refinement, MTP) with a custom ChatML-aware tokenizer. Trained with Modal credits during the Small Models, Big Adventures Hackathon.

Related: base model → Byrne-86M-Base

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("Quazim0t0/Byrne-86M", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("Quazim0t0/Byrne-86M", trust_remote_code=True)

Architecture

These models are built on SpikeWhaleLM, a custom ~86M-parameter decoder-only transformer (16 layers, hidden size 640, 4096-token context, 16,512 vocab, tied input/output embeddings). It combines several non-standard components:

• Multi-head Latent Attention (MLA + XSA) — queries and the output projection are LoRA-compressed (rank 128); each head splits into a decoupled RoPE part (dim 16) and a position-agnostic NoPE part (dim 48); 10 query heads share a single KV head (multi-query attention), with QK-norm for stable logits.
• Engram n-gram memory — a gated associative memory that hashes local n-grams (up to trigrams) into a learned 4,096-entry table and mixes the result back into the residual stream.
• Hash-lookup layers (×2) — multi-head content-addressable features alongside the token embeddings.
• Hyper-Connections — learned, width-expanded residual connections mixed via Sinkhorn-normalized routing, in place of the plain residual add.
• HRM refinement — a Hierarchical Reasoning Model block that performs an extra latent "think a bit more" refinement pass over the hidden states before the output head.
• Multi-Token Prediction (MTP) — a DeepSeek-V3-style auxiliary training head predicting more than one next token (no inference cost).
• Feed-forward is dense (the block is MoE-capable, but MoE is disabled in this release).

JEPA vs HRM. The Byrne models are Non-JEPA: they are trained with HRM refinement only (use_hrm_refine=True, use_jepa=False). The sibling Escarda models add a JEPA (Joint-Embedding Predictive) auxiliary objective on top of HRM refinement.

Tokenizer

These models use SpikeTokenizer, a custom byte-level "length-max" (greedy longest-match) tokenizer with a 16,512-token vocabulary — not a standard BPE/HF tokenizer. Text is UTF-8 encoded, each byte mapped to a latin-1 character, then greedily matched against the vocab using the longest key that fits at each position. It is ChatML-aware, with atomic special tokens for framing and reasoning/tool markers (<|im_start|>, <|im_end|>, <think>/</think>, <begin_solution>/<end_solution>, tool-call markers) plus <bos>/<eos>/<pad>/<unk>. It ships as a PreTrainedTokenizer subclass (spike_tokenizer.py) and loads via AutoTokenizer.from_pretrained(..., trust_remote_code=True).

Evaluation

log-likelihood, acc_norm = byte-length-normalized).

Task	acc	acc_norm
arc_easy	0.3670	0.3468
arc_challenge	0.1894	0.2355
hellaswag	0.2815	0.2858
winogrande	0.5201	—
piqa	0.5756	0.5593
openbookqa	0.1460	0.2440
boolq	0.3865	—

ArithMark-2.0 (AxiomicLabs) — official metric is raw acc: 0.3096.

Language modeling: WikiText-2 byte_ppl (↓) 2.6839 · BLiMP (↑) 0.7033.

Citation

If you use this model, please cite:

@misc{byrne86m,
  title        = {Byrne-86M: A ~86M-parameter SpikeWhaleLM},
  author       = {Dean Byrne (Quazim0t0)},
  year         = {2026},
  howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-86M}},
  note         = {Quazim0t0/Byrne-86M}
}

Update: format-blended SFT on the engram-repaired base

This revision applies the (behavior-preserving) engram repair, then a short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train (with '#### N' reasoning), and MMLU-style ('Answer: ') examples -- so chat fluency improves while the benchmark output-formats are preserved rather than overwritten. Held-out (test-split) before->after:

MMLU acc 0.260->0.254, format 0.918->0.951; GSM8K '####' 0.420->0.815

Note: these are fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.