Beyond Transcript Replay
Collection
Transcript-free persistent memory in event-stream models: the PNS-World benchmark and its causal evaluation. • 2 items • Updated
Persistent Event-State models
Assets for Learning Persistent Referential Identity for Two-Hop Retrieval in Event-Stream Models.
| Experiment | Assets |
|---|---|
| E1 / E3 | src/, eval/, configs/, checkpoints/, results/, reproduce/ |
| E8-FULL | e8_full/source/, nine final e8_full/checkpoints/, e8_full/results/, e8_full/reference/ |
| Reviewer-requested tests | rr2/campaign/, 15 model-only rr2/checkpoints/, rr2/reference/, rr2/frozen_evaluations/ |
Reproduce reviewer-requested tests
Python 3.13 and a CUDA GPU for evaluation/training.
pip install huggingface_hub
hf download nur-dev/pns-bind-25m --revision paper-v2.1 --local-dir pns-model
hf download nur-dev/pns-world --repo-type dataset --revision paper-v2.1 --local-dir pns-data
cd pns-model
pip install -r e8_full/requirements.txt
python rr2/reproduce.py verify --data ../pns-data/rr2
CUDA_VISIBLE_DEVICES=0 python rr2/reproduce.py evaluate --data ../pns-data/rr2 --suite compose --arm future --seed 821 --split confirmation --output rr2_eval_821
Repeat with seeds 822, 823; use --arm ordinary for the ordinary-only control.
Use --suite original --arm ordinary --split known_development for the original-name
control, or --suite gru --update 12000 for the semantic-only recurrent baseline.
Raw predictions are compared with published arrays. Checkpoints contain identical
model tensors with optimizer state omitted; hashes are in rr2/INFERENCE_EXPORT.json.
python rr2/reproduce.py prepare-training --data ../pns-data/rr2 --output rr2_train --workers 24
CUDA_VISIBLE_DEVICES=0 python rr2/reproduce.py train --data ../pns-data/rr2 --output rr2_train --suite compose --arm future --seed 821
Training corpora are regenerated deterministically and checked against frozen hashes.
rr2/GPU_RESULTS.json contains all primary and secondary criterion outcomes.
Reproduce E8-FULL
python e8_full/reproduce.py verify --data ../pns-data/e8_full
CUDA_VISIBLE_DEVICES=0 python e8_full/reproduce.py evaluate --data ../pns-data/e8_full --arm grounded_future --seed 821 --output e8_eval_821
Repeat for arms numeric_future, grounded_cut and seeds 822, 823.
Reproduce E1 / E3
pip install -r requirements.txt
PNS_DATA=../pns-data bash reproduce/reproduce_headline.sh
bash reproduce/reproduce_tables.sh
paper-v1.0 and paper-v2.0 retain the original releases.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support