# safetensors

Three stages as safetensors, for harnesses that do not want a `.pt`.

| directory | stage | context | size |
|---|---|---|---|
| `base_62k/` | pretrained, step 62k | 1,024 | 457 MB |
| `sft_7100/` | SFT, step 7.1k | 4,096 | 460 MB |
| `dpo_3200/` | DPO, step 3.2k | 1,024 | 457 MB |

Each dir: `config.json`, `model.safetensors`, `tokenizer.json`,
`PROVENANCE.md`. Float32. Same bits as `checkpoints/*.pt` (502 tensors,
0 mismatches).

## Load

```bash
python safetensors/load.py sft_7100
```

```python
import json, sys
from safetensors.torch import load_file
sys.path.insert(0, ".")
from config import SpikeWhaleConfig
from model_v2 import SpikeWhaleLM

d = "safetensors/sft_7100"
cfg = json.load(open(f"{d}/config.json"))
cfg.pop("architectures", None)
model = SpikeWhaleLM(SpikeWhaleConfig(**cfg))
model.load_state_dict(load_file(f"{d}/model.safetensors"), strict=False)
model.tie_weights()
model.eval()
```

```python
from spike_tokenizer import SpikeTokenizer
tok = SpikeTokenizer("tokenizer.json")   # Length-MAX (Dong & Su, arXiv:2511.20849), 16512
ids = [tok.bos_token_id] + tok.encode("some text").tolist()
```

Tied head is not in the file. Safetensors will not store shared storage,
so export drops `lm_head.weight`. `tie_weights()` puts it back. Skip
that and logits are garbage from a random head. No error.

`AutoModelForCausalLM` will not load this. Architecture is `spike_whale`
(loop, MLA, fractal RoPE, depth attention, Memory Cache). Use the code
in the repo root.

Context is whatever that directory's `config.json` says: 1024 for base
and DPO, 4096 for SFT. Going past it raises. It does not wrap.

Training always started with `<bos>`. Leave it off and every token sits
one position left. No error, just worse numbers.

## Scoring vs generating

PPL / MC / likelihood: `use_cache=False`. Done.

Generation: `engine_chat.py` / `spike_infer`. A normal KV cache is wrong.
Memory Cache looks at the whole `x`; one-token windows silently compute
a different function (max |Δlogit| 7.24e-01 vs full recompute). Engine
path matches a full recompute. `python verify_cache.py`.

SFT/DPO want ChatML:
`<|im_start|>{role}\n{content}<|im_end|>\n`. Base continues text; score
it as an LM.

Generate at `temp 0.7, top_k 40, rep_pen 1.3`. `DECODING-DEFAULTS.md`.
Greedy is for tests.