# temporal-moe-extras Supporting artifacts for the Temporal-MoE experiments, 248 files, 39.78 GiB. These are things that are expensive to regenerate but are not themselves training checkpoints. ## Layout ``` olmoe_adapt/ ckpt_bake_{CE,G,H,Er8,Er64,D}.pt full optimizer and training state behind the router bakes finetune_ids.pt tokenized finetune corpus token ids bpb_slice_ids.pt bits-per-byte evaluation slice cal2_gprime.pt calibration tensors oseries/, oseries_ce/ paired evaluation slices, val.f16, idx.u8, paired masks logs/ bake, sweep, calibration and evaluation logs merged_ce_model/ OLMoE backbone merged with the bake-CE router, loadable with transformers, includes config and tokenizer run_captures// router_log.pt per-run router traces, back the mechanistic interpretability and residency result tables delex_capture.pt delexicalization captures ablations/ 101 result tables as CSV, plus FINDINGS.md and README.md figures/ 43 figures tokenizer_tok16k/ the 16k tokenizer, also mirrored in the corpus repository ``` ## Notes `merged_ce_model/` is a derived artifact. It is included rather than regenerated because the merge is not guaranteed to be bit-reproducible. `ablations/` holds the result tables the paper draws on. `FINDINGS.md` and `README.md` in that directory are the written interpretation of those tables. The base OLMoE model is not mirrored here. It is public at [`allenai/OLMoE-1B-7B-0924`](https://huggingface.co/allenai/OLMoE-1B-7B-0924). ## Training configuration Every run directory contains `run.meta` and `train.log`. `run.meta` records the full knob set for that run in two lines, for example: ``` [run] temporal_fine_g3_1e19 N=185766400 iters=17112 warmup=856 min_lr=0.0003 eval@1711 [run] shape=s19opt H=800 L=14 ffn=4272 moe_ffn=184 grain=3 num_experts=192 shared_int=1100 dense=0 temporal=1 shared_mult=2 topk=18 heads=50 gb=256 mb=8 lr=3e-3 flops=1e19 ``` `train.log` is the full Megatron stdout, including the complete argument namespace dumped at startup. Read these before drawing any conclusion from a checkpoint. ## MANIFEST.csv and the `cited` column A manifest covering every file in all four repositories lives in the code repository. It has seven columns: `local_path`, `hf_repo`, `hf_path`, `bytes`, `sha256`, `run_name`, `cited`. The `cited` column marks whether a run is referenced by a results table in `results/ablations/*.csv` or by the paper: - `cited`, the run backs a published number. There are 58 of these. - `uncited`, the run is infrastructure validation, a smoke test, a throughput probe, or an aborted run. It is kept for completeness and reproducibility, not because a table depends on it. There are 13 of these. - empty, the file is not scoped to a single run, for example a batch log or an evaluation output. Every `sha256` in the manifest was computed on this disk before upload and each file was verified to exist remotely with a matching byte size. ## Links - Code: - Paper: Temporal-MoE (short paper), see the `paper/` directory in the code repository - Upstream platform this work forks: [FLAME-MoE](https://github.com/cmu-flame/FLAME-MoE), [arXiv:2505.20225](https://arxiv.org/abs/2505.20225) ## Companion repositories - [`ncylich/temporal-moe-ckpts`](https://huggingface.co/ncylich/temporal-moe-ckpts), Megatron training checkpoints - [`ncylich/temporal-moe-router-adapt`](https://huggingface.co/ncylich/temporal-moe-router-adapt), router adaptation safetensors - [`ncylich/temporal-moe-extras`](https://huggingface.co/ncylich/temporal-moe-extras), captures, merged model, result tables, figures - [`ncylich/temporal-moe-corpus`](https://huggingface.co/datasets/ncylich/temporal-moe-corpus), tokenized training corpus ## Provenance Trained with a personal fork of FLAME-MoE. Not affiliated with or endorsed by the FLAME-MoE authors.