bong-water-water-bong/halo-sparse-bitnet-3-4-500m

halo-sparse-bitnet-3:4-500m — research preview

Research preview. 3:4 Sparse-BitNet pretrained from Qwen2.5-0.5B skeleton. Not production-ready; shipped as a validation checkpoint for the 3:4 vs 2:4 N:M sparsity question. The canonical production model on the 1bit-systems stack remains halo-1bit-2b (2 B params, PPL 9.16).

Key findings

At 500 M params and 10 B training tokens (≈20 % of the paper gate of 50 B), on wikitext-103 test (145 chunks × 2048 tokens = 296 960 tokens scored):

Model	PPL	NLL/tok	Notes
Qwen2.5-0.5B fp16 dense (teacher baseline)	13.02	2.57	reference upper bound, not trained by us
3:4 Sparse-BitNet 500 M (this ckpt, step 9 600)	95.13	4.55	7.3× PPL gap vs teacher — undertrained at 10 B tokens
3:4 Sparse-BitNet 500 M (step 9 100 interim)	94.99	4.55	near-plateau, 500 steps earlier
2:4 Sparse-BitNet 500 M (paired run, step 9 600)	150.47	5.01	58 % worse than 3:4 at matched step

Findings:

1. 3:4 wins over 2:4 at 500 M / 10 B tokens by ~37 %. Paired training runs, same architecture + recipe + tokenizer, different N:M sparsity pattern.
2. 10 B tokens is undertrained. Sparse-BitNet paper endpoint is 10 B → PPL 26.31 at 6:8 (25 % dense) on wikitext-103 with the paper's larger model + full 50 B-token schedule. Our PPL 95.13 reflects scale + budget, not method.
3. Cosine plateau confirmed. Steps 9 100 → 9 600 moved PPL from 94.99 → 95.13 (within noise). The final 500 steps didn't help.

Takeaway: 3:4 is the pattern to scale at 2 B + full-budget. Published here so the ablation number is on record.

Recipe

• Base: Qwen2.5-0.5B tokenizer + skeleton (d_model=896, d_ffn=4864, 14 heads, 2 KV-heads, 24 layers, vocab 151 936, seq 2 048).
• Quantization: BitNet-1.58 ternary weights ({-1, 0, +1}) via per-row absmean scale.
• Sparsity: 3:4 N:M structured (AAzdi/Sparse-BitNet, arXiv 2603.05168). Every 4 contiguous weights keep exactly 3; the remaining 1 is masked zero.
• Training data: FineWeb-Edu streaming.
• Batch / seq: 16 × 2048 × grad-accum 32 = 1.05 M tokens / step.
• Target: 9 600 opt-steps ≈ 10.06 B tokens.
• Optimizer: AdamW, peak-lr 3e-4, min-lr 1e-5, warmup 200, cosine decay, β=(0.9, 0.95), eps 1e-5, weight-decay 0.1, grad-clip 1.0.
• Bail-outs active: rising-loss window, tok/s floor, mask-verify every 500 steps.

Loading + eval

This is a raw PyTorch state_dict, not a packaged .h1b. Serve path on the 1bit-systems stack requires a converter we haven't written yet — tools/sparse-bitnet-to-h1b/ is queued for a future release. Today, you load it in Python.

import torch
from llm_vendor.arch.model import Model, ModelArgs   # see 1bit-systems repo

args = ModelArgs(
    d_model=896, d_ffn=4864, head=14, kv_head=2,
    n_layers=24, vocab_size=151936,
    max_seq_len=2048, rope_theta=1000000.0, norm_eps=1e-6,
    weight_tying=True, bitlinear=True,
    use_weight_semi_sparse=True,
    sparse_n=3, sparse_m=4,
)
model = Model(args).to("cuda").to(torch.bfloat16)
model.load_state_dict(torch.load("model.pt", weights_only=False), strict=False)
model.eval()

ppl_eval.py script pattern in the 1bit-systems repo under benchmarks/.

What ships with this repo

model.pt                                  # 943 MB, bf16 raw state_dict (training artifact)
halo-sparse-bitnet-3-4-500m.safetensors   # 346 MB, 3:4 ternary 2-bpw packed + per-row bf16 scales (halo-ready)
pack_sparse_bitnet.py                     # the packer script — reproduces .safetensors from .pt
pack-summary.json                         # compression stats
meta.json                                 # training step pointer
training-loss.csv                         # per-50-step loss trace
ppl-qwen25-fp16-baseline.json             # teacher upper bound
ppl-step9600-final.json                   # our final PPL
ppl-step9100-interim.json                 # near-plateau interim
ppl-run5-baseline-for-comparison.json     # 2:4 twin for reference
SHA256SUMS                                # sha256 of every artifact

Packed vs raw:

• model.pt (943 MB bf16) is the exact training-loop output; use this if you want to fine-tune or resume.
• halo-sparse-bitnet-3-4-500m.safetensors (346 MB) is the halo-native layout: every linear is NAME.packed (uint8 2-bpw ternary, 4 codes per byte: 00=0, 01=+1, 10=-1) + NAME.row_scale (bfloat16 per-row scale). Norms, embeddings, and the tied LM head stay bf16.
• Packed form is 3.48× smaller and what the future halo serve path will consume.

Optimizer state (optim.pt, ~1.9 GB) not uploaded — if you want to resume training, open an issue. We can move it.

What does NOT ship

• .h1b packaged weights (conversion tool pending).
• Inference on 1bit-server. Serve path needs Qwen2.5 arch port
  • sparse-aware kernel — queued as v0.1.3 work in the 1bit-systems repo.
• NPU XDNA2 serve path. Queued as v0.1.2.

Why release this as a preview

1. Research validation: proves 3:4 > 2:4 at the small-scale ablation we could afford ($45 pod time). Cites into the AAzdi/Sparse-BitNet lineage with a reproducible numeric result.
2. Transparency: shows our Sparse-BitNet pipeline runs end-to-end on commodity RunPod H200 infrastructure.
3. Kickstarter / future funding signal: artifact is on record before the funding ask, not after.
4. Doesn't regress existing users: the production halo-1bit-2b model stays the install default. This ckpt is explicitly a research preview; the PPL gap between 95.13 and the 2 B production model's 9.16 is scale, not method.

Upstream

• AAzdi/Sparse-BitNet — paper repo, March 2026.
• microsoft/BitNet — the underlying 1.58-bit arch.
• Qwen/Qwen2.5-0.5B — tokenizer + arch skeleton.
• 1bit-systems — our stack; the full training script + recipe lives under scripts/pretrain_sparse_bitnet_qwen_0p5b.py.

License

MIT, matching the 1bit-systems stack. Derived weights + fine-tunes welcome.