File size: 2,767 Bytes
f9fbedd ebaa640 0dc82e0 ebaa640 f9fbedd 6c3b6a4 0dc82e0 98cd2a6 0dc82e0 ebaa640 98cd2a6 ebaa640 0dc82e0 ebaa640 0dc82e0 98cd2a6 ebaa640 98cd2a6 6c3b6a4 f9fbedd ebaa640 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | ---
license: cc-by-nc-4.0
library_name: pytorch
tags:
- checkpoint
- tr-hash
- pretraining
- work-in-progress
---
# TR-HASH MoE 200M — 70B Unique / 130B Replay Checkpoints
**Status: training complete.** This is the raw checkpoint backup repository,
not the finished model release. It exists as a safety net and a record of the
training trajectory — token-pack folders include model weights and resumable
optimizer/scheduler state.
- Checkpoints are uploaded automatically at token-pack boundaries and on
clean/interrupted shutdown by
[`scripts/sync_checkpoints_to_hf.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/sync_checkpoints_to_hf.py).
- Folder names follow `{tag}_{step}` (`token_pack_NNN_STEP`, `final_STEP`,
`interrupted_STEP`).
- The PIQA sweep below is exploratory, not a full validated evaluation suite.
Do not treat an individual checkpoint as a finished model release.
- The architecture config is tracked at
[`model_config.yaml`](model_config.yaml) in this repo (not embedded in
`checkpoint.pt`).
## Exploratory zero-shot checks (not a full evaluation)
Informal checks across the final token-pack checkpoints, via
[`scripts/convert_to_mlx.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/convert_to_mlx.py) +
[`scripts/eval_mlx_zero_shot.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/eval_mlx_zero_shot.py),
zero-shot causal-log-likelihood scoring, no chat template. Every row uses the
same tokenizer, MLX FP16 inference path, PIQA validation split, and 1,838
examples:
| Checkpoint | Tokens trained | Learning rate | PIQA acc | PIQA acc_norm |
|---|---:|---:|---:|---:|
| `token_pack_032_132239` | 104.00B | 3.00e-4 | 0.6736 | 0.6763 |
| `token_pack_033_136371` | 107.25B | 2.91e-4 | 0.6774 | 0.6746 |
| `token_pack_034_140504` | 110.50B | 2.62e-4 | 0.6768 | 0.6703 |
| **`token_pack_035_144636`** | **113.75B** | **2.19e-4** | **0.6741** | **0.6801** |
| `token_pack_036_148769` | 117.00B | 1.67e-4 | 0.6768 | 0.6774 |
| `token_pack_037_152901` | 120.25B | 1.14e-4 | 0.6676 | 0.6697 |
| `token_pack_038_157034` | 123.50B | 7.01e-5 | 0.6578 | 0.6638 |
| `token_pack_039_161166` | 126.75B | 4.04e-5 | 0.6567 | 0.6529 |
| `token_pack_040_165298` / final | 130.00B | 3.00e-5 | 0.6545 | 0.6561 |
`token_pack_035_144636` has the highest `acc_norm` in this sweep. The trajectory
is not monotonic, and PIQA alone must not be used as a complete model-quality
or checkpoint-selection criterion.
The finished base model is published separately as
[AETHORIA-AI/TR-HASH-MoE-200M-130B](https://huggingface.co/AETHORIA-AI/TR-HASH-MoE-200M-130B).
See [Complexity Framework](https://github.com/Complexity-ML/complexity-framework)
for the training code.
|