| --- |
| license: cc-by-nc-4.0 |
| library_name: pytorch |
| tags: |
| - checkpoint |
| - tr-hash |
| - pretraining |
| - work-in-progress |
| --- |
| |
| # TR-HASH MoE 200M — 70B Unique / 130B Replay Checkpoints |
|
|
| **Status: training complete.** This is the raw checkpoint backup repository, |
| not the finished model release. It exists as a safety net and a record of the |
| training trajectory — token-pack folders include model weights and resumable |
| optimizer/scheduler state. |
|
|
| - Checkpoints are uploaded automatically at token-pack boundaries and on |
| clean/interrupted shutdown by |
| [`scripts/sync_checkpoints_to_hf.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/sync_checkpoints_to_hf.py). |
| - Folder names follow `{tag}_{step}` (`token_pack_NNN_STEP`, `final_STEP`, |
| `interrupted_STEP`). |
| - The PIQA sweep below is exploratory, not a full validated evaluation suite. |
| Do not treat an individual checkpoint as a finished model release. |
| - The architecture config is tracked at |
| [`model_config.yaml`](model_config.yaml) in this repo (not embedded in |
| `checkpoint.pt`). |
|
|
| ## Exploratory zero-shot checks (not a full evaluation) |
|
|
| Informal checks across the final token-pack checkpoints, via |
| [`scripts/convert_to_mlx.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/convert_to_mlx.py) + |
| [`scripts/eval_mlx_zero_shot.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/eval_mlx_zero_shot.py), |
| zero-shot causal-log-likelihood scoring, no chat template. Every row uses the |
| same tokenizer, MLX FP16 inference path, PIQA validation split, and 1,838 |
| examples: |
|
|
| | Checkpoint | Tokens trained | Learning rate | PIQA acc | PIQA acc_norm | |
| |---|---:|---:|---:|---:| |
| | `token_pack_032_132239` | 104.00B | 3.00e-4 | 0.6736 | 0.6763 | |
| | `token_pack_033_136371` | 107.25B | 2.91e-4 | 0.6774 | 0.6746 | |
| | `token_pack_034_140504` | 110.50B | 2.62e-4 | 0.6768 | 0.6703 | |
| | **`token_pack_035_144636`** | **113.75B** | **2.19e-4** | **0.6741** | **0.6801** | |
| | `token_pack_036_148769` | 117.00B | 1.67e-4 | 0.6768 | 0.6774 | |
| | `token_pack_037_152901` | 120.25B | 1.14e-4 | 0.6676 | 0.6697 | |
| | `token_pack_038_157034` | 123.50B | 7.01e-5 | 0.6578 | 0.6638 | |
| | `token_pack_039_161166` | 126.75B | 4.04e-5 | 0.6567 | 0.6529 | |
| | `token_pack_040_165298` / final | 130.00B | 3.00e-5 | 0.6545 | 0.6561 | |
| |
| `token_pack_035_144636` has the highest `acc_norm` in this sweep. The trajectory |
| is not monotonic, and PIQA alone must not be used as a complete model-quality |
| or checkpoint-selection criterion. |
| |
| The finished base model is published separately as |
| [AETHORIA-AI/TR-HASH-MoE-200M-130B](https://huggingface.co/AETHORIA-AI/TR-HASH-MoE-200M-130B). |
| |
| See [Complexity Framework](https://github.com/Complexity-ML/complexity-framework) |
| for the training code. |
| |
