---
license: cc-by-nc-4.0
library_name: pytorch
language:
- en
pipeline_tag: text-generation
tags:
- continued-pretraining
- refinement
- tr-hash
- mixture-of-experts
- gqa
- custom-code
---

# TR-HASH MoE 200M — Interrupted 160B-Token Refinement Checkpoint

> **Phase 2 stopped at step 8,156 / 17,802.** The root
> `model.safetensors` is this intermediate checkpoint (45.8%, approximately
> 32.07B refinement tokens), **not a completed 70B-token refinement**.
> Together with the 130B-token base run, it represents roughly 162B trained
> token exposures and is reported as the 160B-token source checkpoint.

## Plan

Phase 2 takes the finished, fully-decayed TR-HASH 200M base checkpoint and
runs a full-parameter refinement pass over the **same 70B unique tokens**
used for base pretraining -- no replay, no augmentation, a fresh optimizer
and learning-rate schedule, initialized from the base checkpoint's weights
only.

| Setting | Value |
|---|---:|
| Base checkpoint | TR-HASH-200M-130B (post-decay) |
| Training tokens | 69,997,690,880 (70B unique, exact match required) |
| Training progress represented by current weights | Step 8,156 / 17,802 (45.8%, ~32.07B tokens) |
| Replay / augmentation | None |
| Optimizer | Fresh at phase-2 start (not resumed from base pretraining state) |
| Parameters refined | Full model (not LoRA) |

## Interim PIQA result

The step-8,156 checkpoint already exceeds the released base checkpoint on
PIQA. This is an **interim measurement**, not the final phase-2 result.

| Checkpoint | PIQA acc | PIQA acc_norm | Correct (acc) | Examples |
|---|---:|---:|---:|---:|
| TR-HASH-200M-130B base final | 65.45% | 65.61% | — | 1,838 |
| Phase 2, step 8,156 | **68.66%** | **68.39%** | 1,262 / 1,838 | 1,838 |

Evaluation protocol: full PIQA validation split, zero-shot causal
continuation log-likelihood, no chat template, maximum sequence length
2,048. The checkpoint was evaluated in FP16 through MLX using the same model
weights. `acc_norm` selects the continuation by length-normalized
log-likelihood. Final results will be measured again after step 17,802.

## Model summary

Same architecture as the base checkpoint: 201.2M parameters, 16 transformer
layers, GQA (14 query heads / 2 KV heads), deterministic token-ID routed
TR-Hash MoE (4 stored experts, top-2 active), tied embeddings. See
[TR-HASH-MoE-200M-130B](https://huggingface.co/AETHORIA-AI/TR-HASH-MoE-200M-130B) for
the full architecture and base pretraining details.

## Status

- [x] Base pretraining run complete
- [x] Phase-2 refinement launched
- [x] Intermediate step-8,156 weights, optimizer state, curves, and PIQA evaluation uploaded
- [x] Phase-2 run stopped at step 8,156
- [ ] Full 70B-token refinement completed (not reached)

No active process is expected to finish this repository. The root weights
remain the evaluated step-8,156 intermediate checkpoint.

## Instruction post-training

The earlier experimental LoRA repository has been retired. The next planned
stage is a full-parameter instruction SFT, documented in
[`AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT`](https://huggingface.co/AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT).

## Project links

- [AETHORIA-AI on Hugging Face](https://huggingface.co/AETHORIA-AI)
- [TR-HASH-200M-130B (base checkpoint)](https://huggingface.co/AETHORIA-AI/TR-HASH-MoE-200M-130B)
- [Complexity Framework](https://github.com/Complexity-ML/complexity-framework)

## License

The model release is CC BY-NC 4.0. Source datasets and individual documents
retain their own licenses and terms.