Loading weights:   0%|          | 0/1951 [00:00<?, ?it/s]
Loading weights:   0%|          | 3/1951 [00:00<02:18, 14.10it/s]
Loading weights:   4%|4         | 82/1951 [00:00<00:05, 312.96it/s]
Loading weights:   9%|9         | 178/1951 [00:00<00:03, 538.91it/s]
Loading weights:  13%|#3        | 256/1951 [00:00<00:02, 604.69it/s]
Loading weights:  17%|#7        | 339/1951 [00:00<00:02, 674.33it/s]
Loading weights:  22%|##1       | 429/1951 [00:00<00:02, 742.71it/s]
Loading weights:  26%|##6       | 508/1951 [00:00<00:01, 754.85it/s]
Loading weights:  30%|###       | 587/1951 [00:00<00:01, 757.11it/s]
Loading weights:  35%|###4      | 677/1951 [00:01<00:01, 792.67it/s]
Loading weights:  39%|###8      | 758/1951 [00:08<00:36, 32.71it/s] 
Loading weights:  42%|####1     | 815/1951 [00:09<00:29, 38.70it/s]
Loading weights:  44%|####3     | 857/1951 [00:09<00:24, 44.47it/s]
Loading weights:  46%|####5     | 890/1951 [00:10<00:21, 49.64it/s]
Loading weights:  47%|####6     | 916/1951 [00:10<00:19, 53.82it/s]
Loading weights:  48%|####8     | 937/1951 [00:10<00:17, 59.01it/s]
Loading weights:  49%|####9     | 956/1951 [00:10<00:15, 64.09it/s]
Loading weights:  50%|####9     | 972/1951 [00:11<00:14, 69.12it/s]
Loading weights:  51%|#####     | 987/1951 [00:11<00:13, 73.17it/s]
Loading weights:  51%|#####1    | 1001/1951 [00:11<00:12, 76.20it/s]
Loading weights:  52%|#####1    | 1013/1951 [00:11<00:11, 78.68it/s]
Loading weights:  53%|#####2    | 1025/1951 [00:11<00:11, 77.24it/s]
Loading weights:  53%|#####3    | 1035/1951 [00:11<00:13, 68.59it/s]
Loading weights:  54%|#####3    | 1044/1951 [00:12<00:15, 58.71it/s]
Loading weights:  54%|#####4    | 1055/1951 [00:12<00:13, 66.71it/s]
Loading weights:  55%|#####4    | 1064/1951 [00:12<00:13, 63.84it/s]
Loading weights:  55%|#####4    | 1072/1951 [00:12<00:16, 54.36it/s]
Loading weights:  56%|#####5    | 1083/1951 [00:12<00:13, 62.18it/s]
Loading weights:  56%|#####5    | 1091/1951 [00:12<00:14, 60.54it/s]
Loading weights:  56%|#####6    | 1098/1951 [00:12<00:15, 55.16it/s]
Loading weights:  57%|#####6    | 1110/1951 [00:13<00:12, 68.72it/s]
Loading weights:  57%|#####7    | 1118/1951 [00:13<00:14, 57.69it/s]
Loading weights:  58%|#####7    | 1126/1951 [00:13<00:15, 54.89it/s]
Loading weights:  58%|#####8    | 1136/1951 [00:13<00:12, 63.84it/s]
Loading weights:  59%|#####8    | 1144/1951 [00:13<00:14, 53.99it/s]
Loading weights:  59%|#####9    | 1153/1951 [00:13<00:14, 54.44it/s]
Loading weights:  59%|#####9    | 1159/1951 [00:14<00:16, 47.96it/s]
Loading weights:  60%|#####9    | 1167/1951 [00:14<00:14, 53.51it/s]
Loading weights:  60%|######    | 1173/1951 [00:14<00:15, 49.39it/s]
Loading weights:  61%|######    | 1181/1951 [00:14<00:13, 55.32it/s]
Loading weights:  61%|######    | 1187/1951 [00:14<00:14, 51.05it/s]
Loading weights:  61%|######1   | 1195/1951 [00:14<00:13, 57.45it/s]
Loading weights:  62%|######1   | 1202/1951 [00:14<00:13, 53.80it/s]
Loading weights:  62%|######1   | 1209/1951 [00:14<00:12, 57.48it/s]
Loading weights:  62%|######2   | 1216/1951 [00:15<00:13, 54.24it/s]
Loading weights:  63%|######2   | 1223/1951 [00:15<00:13, 55.01it/s]
Loading weights:  63%|######2   | 1229/1951 [00:15<00:14, 49.98it/s]
Loading weights:  63%|######3   | 1237/1951 [00:15<00:12, 56.55it/s]
Loading weights:  64%|######3   | 1243/1951 [00:15<00:13, 50.95it/s]
Loading weights:  64%|######4   | 1251/1951 [00:15<00:12, 57.40it/s]
Loading weights:  64%|######4   | 1258/1951 [00:15<00:12, 53.90it/s]
Loading weights:  65%|######4   | 1265/1951 [00:16<00:11, 57.50it/s]
Loading weights:  65%|######5   | 1271/1951 [00:16<00:13, 51.43it/s]
Loading weights:  66%|######5   | 1279/1951 [00:16<00:11, 57.37it/s]
Loading weights:  66%|######5   | 1286/1951 [00:16<00:12, 53.46it/s]
Loading weights:  66%|######6   | 1292/1951 [00:16<00:12, 54.18it/s]
Loading weights:  70%|#######   | 1368/1951 [00:16<00:02, 234.32it/s]
Loading weights:  74%|#######3  | 1439/1951 [00:16<00:01, 360.77it/s]
Loading weights:  77%|#######7  | 1512/1951 [00:16<00:00, 457.85it/s]
Loading weights:  82%|########1 | 1594/1951 [00:16<00:00, 547.73it/s]
Loading weights:  86%|########5 | 1676/1951 [00:17<00:00, 614.66it/s]
Loading weights:  90%|########9 | 1754/1951 [00:17<00:00, 661.39it/s]
Loading weights:  94%|#########3| 1830/1951 [00:17<00:00, 684.86it/s]
Loading weights:  98%|#########8| 1912/1951 [00:17<00:00, 703.93it/s]
Loading weights: 100%|##########| 1951/1951 [00:17<00:00, 111.77it/s]
[pd] RESUMED from experiments/v8_nla_local/checkpoints/av_L23_clean_cont/step_000100 (extending grokking horizon)
[pd] data=experiments/v8_nla_local/data/stage3_balanced/av_sft_balanced.parquet rows=1356 uniform=True gamma=1.0 corpus_mean_weight=1.000
[pd] DOMAIN-AWARE negs: avg same-dom=16.0 diff-dom=16.0 per anchor (15 domains)
[pd] contrastive ON: beta=1.0 K=2 temp=1.0 domain_aware=True token_dropout=0.0 (InfoNCE over hard-neg activations)
[pd] INJECT at residual layer 23 (native scale); embed pass-through
[pd] inject_layer=23 inject_mode=raw
[pd] optimizer=adamw8bit lr=5e-05 wd=0.01 steps=500 ga=8 trainable_params=12.08M
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
C:\Users\caleb\deception-nanochat-sae-research\.venv-gemma4\Lib\site-packages\torch\_dynamo\eval_frame.py:1181: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. Starting in PyTorch 2.9, calling checkpoint without use_reentrant will raise an exception. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants.
  return fn(*args, **kwargs)
C:\Users\caleb\deception-nanochat-sae-research\experiments\v8_nla_local\train_prior_deviation_av.py:325: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
Consider using tensor.detach() first. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\pytorch\torch\csrc\autograd\generated\python_variable_methods.cpp:837.)
  (loss / ga).backward(); tot += float(loss) / ga
[pd] step 1/500 wloss=3.122 skipped=0