---
license: cc-by-nc-4.0
library_name: pytorch
tags:
- checkpoint
- tr-hash
- pretraining
- work-in-progress
---

# TR-HASH MoE 200M — 70B Unique / 130B Replay Checkpoints

**Status: training complete.** This is the raw checkpoint backup repository,
not the finished model release. It exists as a safety net and a record of the
training trajectory — token-pack folders include model weights and resumable
optimizer/scheduler state.

- Checkpoints are uploaded automatically at token-pack boundaries and on
  clean/interrupted shutdown by
  [`scripts/sync_checkpoints_to_hf.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/sync_checkpoints_to_hf.py).
- Folder names follow `{tag}_{step}` (`token_pack_NNN_STEP`, `final_STEP`,
  `interrupted_STEP`).
- The PIQA sweep below is exploratory, not a full validated evaluation suite.
  Do not treat an individual checkpoint as a finished model release.
- The architecture config is tracked at
  [`model_config.yaml`](model_config.yaml) in this repo (not embedded in
  `checkpoint.pt`).

## Exploratory zero-shot checks (not a full evaluation)

Informal checks across the final token-pack checkpoints, via
[`scripts/convert_to_mlx.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/convert_to_mlx.py) +
[`scripts/eval_mlx_zero_shot.py`](https://github.com/Complexity-ML/complexity-framework/blob/main/scripts/eval_mlx_zero_shot.py),
zero-shot causal-log-likelihood scoring, no chat template. Every row uses the
same tokenizer, MLX FP16 inference path, PIQA validation split, and 1,838
examples:

| Checkpoint | Tokens trained | Learning rate | PIQA acc | PIQA acc_norm |
|---|---:|---:|---:|---:|
| `token_pack_032_132239` | 104.00B | 3.00e-4 | 0.6736 | 0.6763 |
| `token_pack_033_136371` | 107.25B | 2.91e-4 | 0.6774 | 0.6746 |
| `token_pack_034_140504` | 110.50B | 2.62e-4 | 0.6768 | 0.6703 |
| **`token_pack_035_144636`** | **113.75B** | **2.19e-4** | **0.6741** | **0.6801** |
| `token_pack_036_148769` | 117.00B | 1.67e-4 | 0.6768 | 0.6774 |
| `token_pack_037_152901` | 120.25B | 1.14e-4 | 0.6676 | 0.6697 |
| `token_pack_038_157034` | 123.50B | 7.01e-5 | 0.6578 | 0.6638 |
| `token_pack_039_161166` | 126.75B | 4.04e-5 | 0.6567 | 0.6529 |
| `token_pack_040_165298` / final | 130.00B | 3.00e-5 | 0.6545 | 0.6561 |

`token_pack_035_144636` has the highest `acc_norm` in this sweep. The trajectory
is not monotonic, and PIQA alone must not be used as a complete model-quality
or checkpoint-selection criterion.

The finished base model is published separately as
[AETHORIA-AI/TR-HASH-MoE-200M-130B](https://huggingface.co/AETHORIA-AI/TR-HASH-MoE-200M-130B).

See [Complexity Framework](https://github.com/Complexity-ML/complexity-framework)
for the training code.