kevinindustries/kevin-k2

KEVIN

A decoder-only transformer built entirely from scratch — every forward and backward pass derived and coded by hand, with no autograd, no PyTorch, no HuggingFace transformers. Implemented twice: once in NumPy, once in CUDA/C++ (this checkpoint is from the CUDA trainer).

Architecture

A LLaMA-style pre-norm decoder stack:

• 82.9M parameters
• d_model = 512
• 12 layers, 8 attention heads (d_head = 64)
• SwiGLU feed-forward (d_ff = 2026), with a learned Swish-gate β
• RMSNorm instead of LayerNorm
• Learned positional embeddings (no RoPE)
• Context window: 256 tokens, no KV cache — every generated token is a full forward pass over the whole context
• Untied input/output embeddings
• Vocab size: 32,005 (32,000-token byte-level BPE + 5 special tokens: <|endoftext|>, <|im_start|>, <|im_end|>, <tool_call>, </tool_call>)

Training

• Custom byte-level BPE tokenizer, also trained from scratch in C++
• Corpus: 4.01B tokens (10GB text)
• Optimizer: AdamW with linear warmup + cosine LR decay, global gradient-norm clipping, label smoothing 0.05, dropout 0.1
• Random-window sampling: each step draws a uniformly random contiguous 256-token window from anywhere in the corpus (not sequential epochs)
• This checkpoint: step 698,000 (training run complete), loss ~2.7, perplexity ~15

This is a base language model — pretrained on raw text only, not instruction-tuned or RLHF'd. It completes text fluently but has no grounded factual knowledge or chat behavior; treat its output as free-associative continuation, not assistant responses.

Checkpoint format

latest.ckpt is the CUDA trainer's native binary format (TFCKPT1 magic header), not a PyTorch state_dict. It carries its own architecture header (step, vocab size, d_model, heads, layers, d_ff, max_len) so it's self-describing. See the project repo for the loader (utils/ckpt_convert.py, cuda/include/checkpoint.cuh) and inference code (generate.py, serve.py).

Repo contents

• latest.ckpt — the checkpoint itself.
• tokenizer/tokenizer.bbpe (+ merges.txt, vocab.json) — the from-scratch byte-level BPE tokenizer this checkpoint was trained with. Must match exactly; a different tokenizer/vocab size will silently produce garbage.

The training corpus itself isn't included here (too large / not redistributable); bring your own plain-text corpus to resume training.

Download with:

hf download kevinindustries/kevin --local-dir kevin-transformer

Then resume training against your own corpus (see the project repo's docs/continue-training-from-usb.md for a full walkthrough of resuming from a checkpoint):

./build/train_transformer_cuda \
  --resume kevin-transformer/latest.ckpt \
  --corpus <your-corpus.txt> \
  --tokenizer bbpe --tokenizer-path kevin-transformer/tokenizer/tokenizer.bbpe \
  --batch-size 16 --steps 300000 --lr 1e-4 --min-lr 1e-5 --warmup-steps 500 \
  --grad-clip 1.0 --label-smoothing 0.05 --dropout 0.1 \
  --log-every 50 --checkpoint-every 2000 \
  --checkpoint-dir checkpoints --metrics-path checkpoints/metrics.csv

A chat/instruction-tuned variant, SFT'd from this checkpoint, is available at kevinindustries/kevin-chat.

Project repo: https://github.com/pstaykov/transformer