2026-09-23 21:35:22,772 INFO [train.py:1260] (0/2) Training started 2026-09-23 21:35:22,773 INFO [train.py:1270] (0/2) Device: cuda:0 2026-09-23 21:35:22,774 INFO [lexicon.py:171] (0/2) Converting L.pt to Linv.pt 2026-09-23 21:35:22,777 INFO [train.py:1298] (0/2) Using dtype=torch.float16 2026-09-23 21:35:22,777 INFO [train.py:1299] (0/2) Use AMP=True 2026-09-23 21:35:22,780 INFO [train.py:1303] (0/2) About to create model 2026-09-23 21:35:22,934 INFO [train.py:1307] (0/2) Number of model parameters: 22780647 2026-09-23 21:35:23,113 INFO [train.py:1329] (0/2) Using DDP 2026-09-23 21:35:23,549 INFO [multidataset.py:43] (0/2) About to get train cuts from all datasets 2026-09-23 21:35:23,549 INFO [multidataset.py:46] (0/2) Loading Spanish Common Voice in lazy mode 2026-09-23 21:35:23,550 INFO [multidataset.py:52] (0/2) Loading SLR72 dataset in lazy mode 2026-09-23 21:35:23,550 INFO [multidataset.py:57] (0/2) Loading TinyVox Spanish train split in lazy mode 2026-09-23 21:35:23,550 INFO [multidataset.py:62] (0/2) Combining all training datasets 2026-09-23 21:35:23,550 INFO [asr_datamodule.py:220] (0/2) Enable MUSAN 2026-09-23 21:35:23,550 INFO [asr_datamodule.py:221] (0/2) About to get Musan cuts 2026-09-23 21:35:24,128 INFO [asr_datamodule.py:223] (0/2) About to get Hallway noise cuts 2026-09-23 21:35:24,133 INFO [asr_datamodule.py:253] (0/2) Enable SpecAugment 2026-09-23 21:35:24,133 INFO [asr_datamodule.py:254] (0/2) Time warp factor: 80 2026-09-23 21:35:24,133 INFO [asr_datamodule.py:264] (0/2) Num frame mask: 10 2026-09-23 21:35:24,133 INFO [asr_datamodule.py:277] (0/2) About to create train dataset 2026-09-23 21:35:24,133 INFO [asr_datamodule.py:307] (0/2) Using DynamicBucketingSampler. 2026-09-23 21:35:24,414 INFO [asr_datamodule.py:322] (0/2) About to create train dataloader 2026-09-23 21:35:24,415 INFO [multidataset.py:67] (0/2) About to get validation cuts from all datasets 2026-09-23 21:35:24,415 INFO [multidataset.py:70] (0/2) Loading Spanish Common Voice test set as validation 2026-09-23 21:35:24,415 INFO [multidataset.py:75] (0/2) Loading SLR72 dataset test set as validation 2026-09-23 21:35:24,415 INFO [multidataset.py:80] (0/2) Loading TinyVox Spanish validation split 2026-09-23 21:35:24,415 INFO [multidataset.py:85] (0/2) Combining all validation datasets 2026-09-23 21:35:24,415 INFO [asr_datamodule.py:353] (0/2) About to create dev dataset 2026-09-23 21:35:24,667 INFO [asr_datamodule.py:370] (0/2) About to create dev dataloader 2026-09-23 21:35:24,667 INFO [train.py:1520] (0/2) Sanity check -- see if any of the batches in epoch 1 would cause OOM. 2026-09-23 21:35:51,046 INFO [train.py:1551] (0/2) Maximum memory allocated so far is 10941MB 2026-09-23 21:35:51,477 INFO [scaling.py:1024] (0/2) Whitening: name=None, num_groups=1, num_channels=192, metric=42.16 vs. limit=7.5 2026-09-23 21:35:51,603 INFO [train.py:1551] (0/2) Maximum memory allocated so far is 10941MB 2026-09-23 21:35:52,292 INFO [train.py:1551] (0/2) Maximum memory allocated so far is 10941MB 2026-09-23 21:35:53,050 INFO [train.py:1551] (0/2) Maximum memory allocated so far is 10941MB 2026-09-23 21:35:53,720 INFO [scaling.py:1024] (0/2) Whitening: name=None, num_groups=4, num_channels=128, metric=9.72 vs. limit=3.0 2026-09-23 21:35:53,796 INFO [train.py:1551] (0/2) Maximum memory allocated so far is 10941MB 2026-09-23 21:35:54,540 INFO [train.py:1551] (0/2) Maximum memory allocated so far is 11840MB 2026-09-23 21:35:59,519 INFO [train.py:1192] (0/2) Epoch 1, batch 0, loss[loss=3.842, simple_loss=3.468, pruned_loss=3.728, over 24578.00 frames. ], tot_loss[loss=3.842, simple_loss=3.468, pruned_loss=3.728, over 24578.00 frames. ], batch size: 137, lr: 2.00e-02, grad_scale: 1.0 2026-09-23 21:35:59,519 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 21:36:11,133 INFO [train.py:1224] (0/2) Epoch 1, validation: loss=3.681, simple_loss=3.318, pruned_loss=3.621, over 2564189.00 frames. 2026-09-23 21:36:11,133 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 11840MB 2026-09-23 21:36:13,713 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=0.0, ans=0.2 2026-09-23 21:36:17,338 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=33.333333333333336, ans=0.4984375 2026-09-23 21:36:18,019 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=8.46 vs. limit=7.525 2026-09-23 21:36:18,338 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 6.086e+02 7.883e+02 1.927e+03 5.224e+03 6.687e+03, threshold=7.709e+03, percent-clipped=0.0 2026-09-23 21:36:22,360 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=3.89 vs. limit=4.026666666666666 2026-09-23 21:36:22,703 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=66.66666666666667, ans=0.24625 2026-09-23 21:36:23,697 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 4.520e+02 6.170e+02 8.872e+02 3.026e+03 7.859e+03, threshold=3.549e+03, percent-clipped=0.0 2026-09-23 21:36:24,366 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=30.11 vs. limit=7.525 2026-09-23 21:36:25,919 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=66.66666666666667, ans=0.1975 2026-09-23 21:36:35,025 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 3.927e+02 5.130e+02 6.086e+02 8.872e+02 7.859e+03, threshold=2.434e+03, percent-clipped=0.0 2026-09-23 21:36:38,938 INFO [train.py:1192] (0/2) Epoch 1, batch 50, loss[loss=1.017, simple_loss=0.897, pruned_loss=1.067, over 24302.00 frames. ], tot_loss[loss=1.883, simple_loss=1.711, pruned_loss=1.657, over 1076808.10 frames. ], batch size: 125, lr: 2.20e-02, grad_scale: 0.25 2026-09-23 21:36:42,416 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=166.66666666666666, ans=0.4921875 2026-09-23 21:36:42,538 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=64.89 vs. limit=7.5625 2026-09-23 21:36:42,550 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=21.60 vs. limit=7.625 2026-09-23 21:36:45,343 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=33.49 vs. limit=7.65 2026-09-23 21:36:46,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=200.0, ans=0.298 2026-09-23 21:36:49,548 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=27.83 vs. limit=5.05 2026-09-23 21:36:51,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=233.33333333333334, ans=0.4890625 2026-09-23 21:36:52,675 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=233.33333333333334, ans=0.09475 2026-09-23 21:36:55,962 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=266.6666666666667, ans=0.19 2026-09-23 21:36:56,845 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=57.68 vs. limit=7.7 2026-09-23 21:36:58,143 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=266.6666666666667, ans=0.8906666666666667 2026-09-23 21:37:01,574 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=16.01 vs. limit=4.12 2026-09-23 21:37:02,257 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=12.36 vs. limit=4.12 2026-09-23 21:37:02,671 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=23.25 vs. limit=7.725 2026-09-23 21:37:03,077 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=300.0, ans=0.18875 2026-09-23 21:37:04,545 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=14.35 vs. limit=5.075 2026-09-23 21:37:07,588 INFO [train.py:1192] (0/2) Epoch 1, batch 100, loss[loss=1.005, simple_loss=0.875, pruned_loss=1.046, over 24611.00 frames. ], tot_loss[loss=1.446, simple_loss=1.295, pruned_loss=1.362, over 1905492.15 frames. ], batch size: 154, lr: 2.40e-02, grad_scale: 0.5 2026-09-23 21:37:09,322 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 9.555e+01 1.958e+02 3.763e+02 5.587e+02 7.859e+03, threshold=7.526e+02, percent-clipped=0.0 2026-09-23 21:37:10,207 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=13.43 vs. limit=4.133333333333334 2026-09-23 21:37:11,812 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=11.04 vs. limit=5.083333333333333 2026-09-23 21:37:11,949 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=80.32 vs. limit=7.625 2026-09-23 21:37:15,939 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=26.40 vs. limit=7.6375 2026-09-23 21:37:24,528 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=45.35 vs. limit=7.6625 2026-09-23 21:37:27,839 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.90 vs. limit=7.6625 2026-09-23 21:37:31,007 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=74.20 vs. limit=7.675 2026-09-23 21:37:32,742 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=466.6666666666667, ans=0.478125 2026-09-23 21:37:35,971 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=466.6666666666667, ans=0.1825 2026-09-23 21:37:38,204 INFO [train.py:1192] (0/2) Epoch 1, batch 150, loss[loss=0.9074, simple_loss=0.7784, pruned_loss=0.9434, over 24245.00 frames. ], tot_loss[loss=1.266, simple_loss=1.122, pruned_loss=1.229, over 2557662.00 frames. ], batch size: 125, lr: 2.60e-02, grad_scale: 0.5 2026-09-23 21:37:49,051 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=17.11 vs. limit=7.7 2026-09-23 21:37:56,523 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=25.70 vs. limit=7.7125 2026-09-23 21:37:58,964 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.36 vs. limit=3.09 2026-09-23 21:38:00,109 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=600.0, ans=0.17750000000000002 2026-09-23 21:38:00,970 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=9.25 vs. limit=5.15 2026-09-23 21:38:03,109 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.26 vs. limit=7.725 2026-09-23 21:38:04,890 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=118.22 vs. limit=7.7375 2026-09-23 21:38:08,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=633.3333333333334, ans=0.17625000000000002 2026-09-23 21:38:10,651 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=7.38 vs. limit=4.253333333333333 2026-09-23 21:38:11,808 INFO [train.py:1192] (0/2) Epoch 1, batch 200, loss[loss=1.035, simple_loss=0.8899, pruned_loss=0.9945, over 21070.00 frames. ], tot_loss[loss=1.177, simple_loss=1.032, pruned_loss=1.157, over 3054449.39 frames. ], batch size: 333, lr: 2.80e-02, grad_scale: 1.0 2026-09-23 21:38:12,073 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=8.0 2026-09-23 21:38:13,634 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 9.752e+01 1.933e+02 2.630e+02 3.377e+02 7.630e+02, threshold=5.261e+02, percent-clipped=1.0 2026-09-23 21:38:20,219 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=49.77 vs. limit=7.7625 2026-09-23 21:38:20,935 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=147.77 vs. limit=7.7625 2026-09-23 21:38:29,028 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=733.3333333333334, ans=0.465625 2026-09-23 21:38:32,220 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=17.44 vs. limit=7.7875 2026-09-23 21:38:35,182 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.59 vs. limit=8.075 2026-09-23 21:38:36,006 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.73 vs. limit=7.7875 2026-09-23 21:38:43,373 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=800.0, ans=0.0475 2026-09-23 21:38:45,288 INFO [train.py:1192] (0/2) Epoch 1, batch 250, loss[loss=1.104, simple_loss=0.931, pruned_loss=1.088, over 24306.00 frames. ], tot_loss[loss=1.126, simple_loss=0.9774, pruned_loss=1.113, over 3443054.52 frames. ], batch size: 234, lr: 3.00e-02, grad_scale: 1.0 2026-09-23 21:38:45,510 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=8.125 2026-09-23 21:38:48,894 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=833.3333333333334, ans=0.29166666666666663 2026-09-23 21:38:50,523 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=833.3333333333334, ans=5.416666666666667 2026-09-23 21:38:51,547 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=866.6666666666666, ans=0.459375 2026-09-23 21:38:55,798 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=8.93 vs. limit=8.15 2026-09-23 21:39:02,831 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=900.0, ans=0.291 2026-09-23 21:39:06,003 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=933.3333333333334, ans=0.29066666666666663 2026-09-23 21:39:09,527 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.00 vs. limit=8.2 2026-09-23 21:39:10,047 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=933.3333333333334, ans=0.45625 2026-09-23 21:39:12,881 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.15 vs. limit=7.8625 2026-09-23 21:39:12,941 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.08 vs. limit=7.8625 2026-09-23 21:39:16,347 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.14 vs. limit=8.25 2026-09-23 21:39:16,749 INFO [train.py:1192] (0/2) Epoch 1, batch 300, loss[loss=1.105, simple_loss=0.9199, pruned_loss=1.08, over 24554.00 frames. ], tot_loss[loss=1.093, simple_loss=0.9394, pruned_loss=1.081, over 3748865.65 frames. ], batch size: 204, lr: 3.20e-02, grad_scale: 2.0 2026-09-23 21:39:18,395 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=1000.0, ans=5.625 2026-09-23 21:39:18,852 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 1.125e+02 2.100e+02 2.680e+02 3.774e+02 1.103e+03, threshold=5.360e+02, percent-clipped=10.0 2026-09-23 21:39:33,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1066.6666666666667, ans=0.3666666666666667 2026-09-23 21:39:41,781 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=1133.3333333333333, ans=0.07450000000000001 2026-09-23 21:39:44,422 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=1133.3333333333333, ans=0.446875 2026-09-23 21:39:46,070 INFO [train.py:1192] (0/2) Epoch 1, batch 350, loss[loss=0.933, simple_loss=0.7692, pruned_loss=0.8967, over 24580.00 frames. ], tot_loss[loss=1.076, simple_loss=0.9163, pruned_loss=1.059, over 3992136.89 frames. ], batch size: 137, lr: 3.40e-02, grad_scale: 2.0 2026-09-23 21:39:47,588 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.26 vs. limit=8.375 2026-09-23 21:39:51,554 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.39 vs. limit=7.95 2026-09-23 21:39:53,861 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=1200.0, ans=0.44375 2026-09-23 21:40:02,785 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=10.28 vs. limit=7.9625 2026-09-23 21:40:08,511 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=1266.6666666666667, ans=0.440625 2026-09-23 21:40:09,527 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=1300.0, ans=0.287 2026-09-23 21:40:12,662 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.min_positive, batch_count=1300.0, ans=0.237 2026-09-23 21:40:12,858 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=20.72 vs. limit=7.9875 2026-09-23 21:40:13,242 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=1300.0, ans=0.287 2026-09-23 21:40:15,600 INFO [train.py:1192] (0/2) Epoch 1, batch 400, loss[loss=0.9936, simple_loss=0.8307, pruned_loss=0.8776, over 24561.00 frames. ], tot_loss[loss=1.06, simple_loss=0.8961, pruned_loss=1.027, over 4179317.93 frames. ], batch size: 170, lr: 3.60e-02, grad_scale: 4.0 2026-09-23 21:40:17,315 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 1.156e+02 2.069e+02 2.740e+02 4.120e+02 1.261e+03, threshold=5.479e+02, percent-clipped=17.0 2026-09-23 21:40:22,693 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.39 vs. limit=8.0125 2026-09-23 21:40:26,243 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.63 vs. limit=8.525 2026-09-23 21:40:29,496 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=18.00 vs. limit=8.025 2026-09-23 21:40:32,638 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=1400.0, ans=0.434375 2026-09-23 21:40:32,774 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.68 vs. limit=3.21 2026-09-23 21:40:36,057 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.75 vs. limit=5.358333333333333 2026-09-23 21:40:36,083 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=24.84 vs. limit=8.0375 2026-09-23 21:40:39,208 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=20.11 vs. limit=8.05 2026-09-23 21:40:41,922 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.86 vs. limit=5.733333333333333 2026-09-23 21:40:42,544 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=16.02 vs. limit=8.05 2026-09-23 21:40:43,691 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=14.37 vs. limit=8.6 2026-09-23 21:40:45,041 INFO [train.py:1192] (0/2) Epoch 1, batch 450, loss[loss=0.9903, simple_loss=0.8389, pruned_loss=0.8118, over 24628.00 frames. ], tot_loss[loss=1.042, simple_loss=0.8803, pruned_loss=0.9823, over 4311578.12 frames. ], batch size: 175, lr: 3.80e-02, grad_scale: 4.0 2026-09-23 21:40:46,941 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=1500.0, ans=0.4296875 2026-09-23 21:40:54,441 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=17.76 vs. limit=8.075 2026-09-23 21:40:56,048 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=7.05 vs. limit=5.383333333333333 2026-09-23 21:40:57,079 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=1566.6666666666667, ans=0.4265625 2026-09-23 21:41:04,907 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.54 vs. limit=8.1 2026-09-23 21:41:10,629 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=5.05 vs. limit=5.0 2026-09-23 21:41:14,882 INFO [train.py:1192] (0/2) Epoch 1, batch 500, loss[loss=1.008, simple_loss=0.859, pruned_loss=0.7894, over 24513.00 frames. ], tot_loss[loss=1.018, simple_loss=0.8609, pruned_loss=0.9266, over 4429096.59 frames. ], batch size: 218, lr: 4.00e-02, grad_scale: 8.0 2026-09-23 21:41:16,640 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 1.720e+02 2.882e+02 3.963e+02 5.717e+02 1.217e+03, threshold=7.926e+02, percent-clipped=27.0 2026-09-23 21:41:17,378 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=1666.6666666666667, ans=0.1375 2026-09-23 21:41:17,576 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.62 vs. limit=8.75 2026-09-23 21:41:18,052 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.25 vs. limit=8.75 2026-09-23 21:41:23,901 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=15.11 vs. limit=8.775 2026-09-23 21:41:29,138 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.62 vs. limit=5.866666666666666 2026-09-23 21:41:34,274 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.02 vs. limit=8.825 2026-09-23 21:41:36,419 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.53 vs. limit=5.441666666666666 2026-09-23 21:41:44,035 INFO [train.py:1192] (0/2) Epoch 1, batch 550, loss[loss=0.9717, simple_loss=0.8376, pruned_loss=0.7172, over 24259.00 frames. ], tot_loss[loss=0.9932, simple_loss=0.843, pruned_loss=0.87, over 4518474.41 frames. ], batch size: 257, lr: 3.99e-02, grad_scale: 8.0 2026-09-23 21:41:45,336 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=1833.3333333333333, ans=0.8358333333333333 2026-09-23 21:41:46,260 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.29 vs. limit=3.275 2026-09-23 21:41:46,824 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.56 vs. limit=8.1875 2026-09-23 21:41:47,946 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=5.42 vs. limit=5.458333333333333 2026-09-23 21:41:52,855 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=1866.6666666666667, ans=0.4125 2026-09-23 21:41:54,935 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=1900.0, ans=0.2625 2026-09-23 21:41:55,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=1900.0, ans=0.8335 2026-09-23 21:41:57,196 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=1900.0, ans=0.4109375 2026-09-23 21:41:59,934 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.91 vs. limit=8.2125 2026-09-23 21:42:02,773 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=16.93 vs. limit=8.225 2026-09-23 21:42:07,559 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=1966.6666666666667, ans=0.4078125 2026-09-23 21:42:11,819 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.13 vs. limit=8.975 2026-09-23 21:42:13,798 INFO [train.py:1192] (0/2) Epoch 1, batch 600, loss[loss=0.9539, simple_loss=0.8215, pruned_loss=0.6904, over 24340.00 frames. ], tot_loss[loss=0.9677, simple_loss=0.825, pruned_loss=0.8151, over 4584135.09 frames. ], batch size: 234, lr: 3.99e-02, grad_scale: 8.0 2026-09-23 21:42:15,430 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 1.756e+02 2.946e+02 3.974e+02 5.357e+02 1.166e+03, threshold=7.947e+02, percent-clipped=7.0 2026-09-23 21:42:19,150 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=5.84 vs. limit=6.016666666666667 2026-09-23 21:42:19,527 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.79 vs. limit=9.025 2026-09-23 21:42:23,573 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=2033.3333333333333, ans=0.2796666666666667 2026-09-23 21:42:24,709 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=2066.6666666666665, ans=6.291666666666666 2026-09-23 21:42:25,394 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.35 vs. limit=8.275 2026-09-23 21:42:27,121 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=8.50 vs. limit=8.275 2026-09-23 21:42:28,269 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.58 vs. limit=9.05 2026-09-23 21:42:31,398 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.81 vs. limit=9.075 2026-09-23 21:42:34,829 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.31 vs. limit=9.075 2026-09-23 21:42:42,063 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=2166.6666666666665, ans=0.11875000000000001 2026-09-23 21:42:42,551 INFO [train.py:1192] (0/2) Epoch 1, batch 650, loss[loss=0.8193, simple_loss=0.7256, pruned_loss=0.539, over 24571.00 frames. ], tot_loss[loss=0.9363, simple_loss=0.803, pruned_loss=0.7576, over 4649927.75 frames. ], batch size: 162, lr: 3.99e-02, grad_scale: 8.0 2026-09-23 21:42:53,584 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2233.3333333333335, ans=0.2776666666666667 2026-09-23 21:42:59,010 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.33 vs. limit=3.335 2026-09-23 21:43:02,282 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.37 vs. limit=9.2 2026-09-23 21:43:07,520 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=2300.0, ans=0.3921875 2026-09-23 21:43:07,659 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.25 vs. limit=5.575 2026-09-23 21:43:11,723 INFO [train.py:1192] (0/2) Epoch 1, batch 700, loss[loss=0.7976, simple_loss=0.7036, pruned_loss=0.5231, over 24553.00 frames. ], tot_loss[loss=0.9126, simple_loss=0.7867, pruned_loss=0.7113, over 4681821.00 frames. ], batch size: 154, lr: 3.99e-02, grad_scale: 8.0 2026-09-23 21:43:12,370 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=2333.3333333333335, ans=0.390625 2026-09-23 21:43:13,409 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.090e+02 3.037e+02 4.027e+02 5.498e+02 1.629e+03, threshold=8.054e+02, percent-clipped=8.0 2026-09-23 21:43:15,315 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=2333.3333333333335, ans=0.8183333333333334 2026-09-23 21:43:18,051 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2366.6666666666665, ans=0.2763333333333333 2026-09-23 21:43:23,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=2400.0, ans=0.3875 2026-09-23 21:43:29,158 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.40 vs. limit=8.4125 2026-09-23 21:43:35,290 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2466.6666666666665, ans=0.384375 2026-09-23 21:43:38,263 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=2466.6666666666665, ans=0.8136666666666666 2026-09-23 21:43:40,328 INFO [train.py:1192] (0/2) Epoch 1, batch 750, loss[loss=0.7855, simple_loss=0.7038, pruned_loss=0.4882, over 24555.00 frames. ], tot_loss[loss=0.8852, simple_loss=0.7672, pruned_loss=0.6661, over 4709826.92 frames. ], batch size: 170, lr: 3.99e-02, grad_scale: 8.0 2026-09-23 21:43:40,422 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2500.0, ans=0.27499999999999997 2026-09-23 21:43:42,299 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.14 vs. limit=5.625 2026-09-23 21:43:44,934 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=2500.0, ans=0.10625 2026-09-23 21:43:46,055 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=2533.3333333333335, ans=0.38125 2026-09-23 21:43:48,180 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=2533.3333333333335, ans=0.38125 2026-09-23 21:43:50,142 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=2533.3333333333335, ans=0.1833333333333333 2026-09-23 21:43:52,689 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2566.6666666666665, ans=0.3796875 2026-09-23 21:43:53,827 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=2566.6666666666665, ans=0.10375 2026-09-23 21:43:57,942 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 21:43:59,111 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=2600.0, ans=0.5 2026-09-23 21:44:03,748 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=2633.3333333333335, ans=0.3765625 2026-09-23 21:44:03,926 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.69 vs. limit=6.316666666666666 2026-09-23 21:44:07,744 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.24 vs. limit=9.475 2026-09-23 21:44:09,338 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=2666.6666666666665, ans=0.08333333333333334 2026-09-23 21:44:09,720 INFO [train.py:1192] (0/2) Epoch 1, batch 800, loss[loss=0.6793, simple_loss=0.6143, pruned_loss=0.4083, over 24519.00 frames. ], tot_loss[loss=0.8614, simple_loss=0.7506, pruned_loss=0.6266, over 4734285.36 frames. ], batch size: 137, lr: 3.99e-02, grad_scale: 16.0 2026-09-23 21:44:09,827 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2666.6666666666665, ans=0.2733333333333333 2026-09-23 21:44:11,237 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.531e+02 3.517e+02 4.281e+02 5.987e+02 1.290e+03, threshold=8.562e+02, percent-clipped=5.0 2026-09-23 21:44:16,527 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=2700.0, ans=0.3734375 2026-09-23 21:44:19,006 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=2700.0, ans=0.03925 2026-09-23 21:44:22,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=2733.3333333333335, ans=0.371875 2026-09-23 21:44:25,903 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.18 vs. limit=8.5375 2026-09-23 21:44:38,064 INFO [train.py:1192] (0/2) Epoch 1, batch 850, loss[loss=0.7787, simple_loss=0.7012, pruned_loss=0.4694, over 24609.00 frames. ], tot_loss[loss=0.8368, simple_loss=0.7336, pruned_loss=0.5887, over 4757669.48 frames. ], batch size: 198, lr: 3.99e-02, grad_scale: 16.0 2026-09-23 21:44:38,785 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.88 vs. limit=5.708333333333333 2026-09-23 21:45:01,426 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=2966.6666666666665, ans=0.2703333333333333 2026-09-23 21:45:02,033 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=2966.6666666666665, ans=0.3609375 2026-09-23 21:45:06,825 INFO [train.py:1192] (0/2) Epoch 1, batch 900, loss[loss=0.639, simple_loss=0.5939, pruned_loss=0.3539, over 24585.00 frames. ], tot_loss[loss=0.8167, simple_loss=0.7199, pruned_loss=0.5574, over 4772753.51 frames. ], batch size: 137, lr: 3.99e-02, grad_scale: 16.0 2026-09-23 21:45:08,876 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.471e+02 3.813e+02 4.524e+02 5.453e+02 1.532e+03, threshold=9.047e+02, percent-clipped=3.0 2026-09-23 21:45:15,101 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=3033.3333333333335, ans=0.17194166666666666 2026-09-23 21:45:15,157 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=3033.3333333333335, ans=0.21966666666666668 2026-09-23 21:45:15,320 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.22 vs. limit=5.758333333333334 2026-09-23 21:45:20,243 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=20.84 vs. limit=8.65 2026-09-23 21:45:23,081 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=3066.6666666666665, ans=0.35625 2026-09-23 21:45:31,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=3133.3333333333335, ans=0.08249999999999998 2026-09-23 21:45:34,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=3166.6666666666665, ans=0.08125 2026-09-23 21:45:34,896 INFO [train.py:1192] (0/2) Epoch 1, batch 950, loss[loss=0.8018, simple_loss=0.6754, pruned_loss=0.5436, over 11508.00 frames. ], tot_loss[loss=0.7951, simple_loss=0.7038, pruned_loss=0.5295, over 4712899.79 frames. ], batch size: 333, lr: 3.98e-02, grad_scale: 16.0 2026-09-23 21:45:38,257 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=3166.6666666666665, ans=0.08020833333333334 2026-09-23 21:45:39,543 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-1.pt 2026-09-23 21:45:44,542 INFO [train.py:1192] (0/2) Epoch 2, batch 0, loss[loss=0.7124, simple_loss=0.6442, pruned_loss=0.4189, over 24584.00 frames. ], tot_loss[loss=0.7124, simple_loss=0.6442, pruned_loss=0.4189, over 24584.00 frames. ], batch size: 137, lr: 3.91e-02, grad_scale: 32.0 2026-09-23 21:45:44,542 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 21:45:53,680 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([1.0274, 1.1104, 1.0488, 1.0449, 0.9819, 0.8827, 1.1300, 1.0332], device='cuda:0') 2026-09-23 21:45:55,881 INFO [train.py:1224] (0/2) Epoch 2, validation: loss=0.5047, simple_loss=0.4979, pruned_loss=0.2373, over 2564189.00 frames. 2026-09-23 21:45:55,882 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 21:45:56,772 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.53 vs. limit=9.895 2026-09-23 21:46:05,618 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=3226.6666666666665, ans=0.34875 2026-09-23 21:46:08,355 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.85 vs. limit=8.7225 2026-09-23 21:46:13,813 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=3293.3333333333335, ans=0.34562499999999996 2026-09-23 21:46:15,321 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=3293.3333333333335, ans=0.34562499999999996 2026-09-23 21:46:15,335 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=3293.3333333333335, ans=0.7847333333333334 2026-09-23 21:46:21,864 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.265e+02 3.080e+02 3.822e+02 5.393e+02 1.061e+03, threshold=7.645e+02, percent-clipped=3.0 2026-09-23 21:46:24,425 INFO [train.py:1192] (0/2) Epoch 2, batch 50, loss[loss=0.5749, simple_loss=0.5399, pruned_loss=0.309, over 24238.00 frames. ], tot_loss[loss=0.7373, simple_loss=0.6647, pruned_loss=0.4352, over 1076668.25 frames. ], batch size: 125, lr: 3.90e-02, grad_scale: 16.0 2026-09-23 21:46:33,455 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3393.3333333333335, ans=0.3409375 2026-09-23 21:46:37,387 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=3426.6666666666665, ans=0.339375 2026-09-23 21:46:38,540 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.42 vs. limit=5.370666666666667 2026-09-23 21:46:40,878 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=3460.0, ans=0.33781249999999996 2026-09-23 21:46:50,478 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.51 vs. limit=3.524 2026-09-23 21:46:52,345 INFO [train.py:1192] (0/2) Epoch 2, batch 100, loss[loss=0.6668, simple_loss=0.6151, pruned_loss=0.372, over 24596.00 frames. ], tot_loss[loss=0.7287, simple_loss=0.6614, pruned_loss=0.4227, over 1904726.18 frames. ], batch size: 154, lr: 3.90e-02, grad_scale: 16.0 2026-09-23 21:46:55,566 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=3526.6666666666665, ans=0.3346875 2026-09-23 21:46:59,575 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.25 vs. limit=8.835 2026-09-23 21:47:02,229 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=3560.0, ans=0.0199 2026-09-23 21:47:10,866 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=3626.6666666666665, ans=0.2637333333333333 2026-09-23 21:47:12,534 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=3626.6666666666665, ans=0.2637333333333333 2026-09-23 21:47:15,994 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=3660.0, ans=0.3284375 2026-09-23 21:47:17,959 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.567e+02 3.428e+02 4.005e+02 4.892e+02 1.099e+03, threshold=8.011e+02, percent-clipped=5.0 2026-09-23 21:47:20,060 INFO [train.py:1192] (0/2) Epoch 2, batch 150, loss[loss=0.557, simple_loss=0.5253, pruned_loss=0.2961, over 24292.00 frames. ], tot_loss[loss=0.7074, simple_loss=0.6465, pruned_loss=0.4036, over 2558558.16 frames. ], batch size: 125, lr: 3.90e-02, grad_scale: 16.0 2026-09-23 21:47:27,300 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.27 vs. limit=5.931666666666667 2026-09-23 21:47:31,556 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=3760.0, ans=0.32375 2026-09-23 21:47:33,265 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=3760.0, ans=0.32375 2026-09-23 21:47:36,511 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.67 vs. limit=3.564 2026-09-23 21:47:41,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=3793.3333333333335, ans=0.014649999999999996 2026-09-23 21:47:48,079 INFO [train.py:1192] (0/2) Epoch 2, batch 200, loss[loss=0.7664, simple_loss=0.6824, pruned_loss=0.4533, over 21099.00 frames. ], tot_loss[loss=0.6962, simple_loss=0.6387, pruned_loss=0.3931, over 3055681.69 frames. ], batch size: 333, lr: 3.90e-02, grad_scale: 16.0 2026-09-23 21:48:00,269 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=3926.6666666666665, ans=0.7892666666666667 2026-09-23 21:48:00,451 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.20 vs. limit=5.9816666666666665 2026-09-23 21:48:02,460 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=3926.6666666666665, ans=0.05274999999999999 2026-09-23 21:48:06,336 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=3960.0, ans=0.010899999999999993 2026-09-23 21:48:08,335 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.94 vs. limit=10.47 2026-09-23 21:48:09,243 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=3960.0, ans=0.010899999999999993 2026-09-23 21:48:14,869 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.848e+02 4.117e+02 5.175e+02 6.960e+02 1.473e+03, threshold=1.035e+03, percent-clipped=14.0 2026-09-23 21:48:16,980 INFO [train.py:1192] (0/2) Epoch 2, batch 250, loss[loss=0.7524, simple_loss=0.6896, pruned_loss=0.4211, over 24321.00 frames. ], tot_loss[loss=0.6874, simple_loss=0.633, pruned_loss=0.3845, over 3445247.46 frames. ], batch size: 234, lr: 3.90e-02, grad_scale: 16.0 2026-09-23 21:48:33,237 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=4126.666666666667, ans=0.7555666666666667 2026-09-23 21:48:37,749 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=4126.666666666667, ans=10.595 2026-09-23 21:48:39,339 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=4160.0, ans=0.07400000000000001 2026-09-23 21:48:44,528 INFO [train.py:1192] (0/2) Epoch 2, batch 300, loss[loss=0.683, simple_loss=0.6429, pruned_loss=0.3638, over 24534.00 frames. ], tot_loss[loss=0.6761, simple_loss=0.6253, pruned_loss=0.3745, over 3749651.96 frames. ], batch size: 204, lr: 3.90e-02, grad_scale: 16.0 2026-09-23 21:48:44,762 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.34 vs. limit=5.677333333333333 2026-09-23 21:48:50,529 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=4226.666666666667, ans=0.301875 2026-09-23 21:48:54,715 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=4226.666666666667, ans=0.025 2026-09-23 21:48:59,095 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=4260.0, ans=0.3003125 2026-09-23 21:49:10,536 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.695e+02 3.837e+02 4.969e+02 6.338e+02 1.168e+03, threshold=9.937e+02, percent-clipped=1.0 2026-09-23 21:49:11,643 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=4326.666666666667, ans=0.009928985507246376 2026-09-23 21:49:12,571 INFO [train.py:1192] (0/2) Epoch 2, batch 350, loss[loss=0.5659, simple_loss=0.5371, pruned_loss=0.2969, over 24570.00 frames. ], tot_loss[loss=0.6679, simple_loss=0.6207, pruned_loss=0.3661, over 3993450.14 frames. ], batch size: 137, lr: 3.89e-02, grad_scale: 16.0 2026-09-23 21:49:17,585 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=4393.333333333333, ans=0.7462333333333333 2026-09-23 21:49:18,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=4393.333333333333, ans=0.2659 2026-09-23 21:49:25,566 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=4426.666666666667, ans=0.07233333333333333 2026-09-23 21:49:39,580 INFO [train.py:1192] (0/2) Epoch 2, batch 400, loss[loss=0.6399, simple_loss=0.6041, pruned_loss=0.3387, over 24571.00 frames. ], tot_loss[loss=0.6564, simple_loss=0.6133, pruned_loss=0.3559, over 4181170.20 frames. ], batch size: 170, lr: 3.89e-02, grad_scale: 32.0 2026-09-23 21:49:41,399 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.86 vs. limit=10.895 2026-09-23 21:49:47,699 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.25 vs. limit=10.92 2026-09-23 21:49:47,825 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.03 vs. limit=6.14 2026-09-23 21:49:53,334 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=4593.333333333333, ans=0.04752777777777778 2026-09-23 21:49:59,072 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=4626.666666666667, ans=0.7380666666666666 2026-09-23 21:50:04,201 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=4660.0, ans=0.2815625 2026-09-23 21:50:04,635 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.866e+02 3.994e+02 4.572e+02 6.150e+02 1.146e+03, threshold=9.145e+02, percent-clipped=1.0 2026-09-23 21:50:05,462 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.79 vs. limit=7.33 2026-09-23 21:50:05,729 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=4660.0, ans=0.2815625 2026-09-23 21:50:06,682 INFO [train.py:1192] (0/2) Epoch 2, batch 450, loss[loss=0.625, simple_loss=0.6053, pruned_loss=0.3173, over 24620.00 frames. ], tot_loss[loss=0.649, simple_loss=0.609, pruned_loss=0.3489, over 4316032.68 frames. ], batch size: 175, lr: 3.89e-02, grad_scale: 32.0 2026-09-23 21:50:08,380 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=4693.333333333333, ans=0.28 2026-09-23 21:50:09,033 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.30 vs. limit=9.26 2026-09-23 21:50:20,587 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=4760.0, ans=0.276875 2026-09-23 21:50:34,667 INFO [train.py:1192] (0/2) Epoch 2, batch 500, loss[loss=0.6854, simple_loss=0.6519, pruned_loss=0.3586, over 24493.00 frames. ], tot_loss[loss=0.6383, simple_loss=0.6019, pruned_loss=0.34, over 4432491.37 frames. ], batch size: 218, lr: 3.89e-02, grad_scale: 32.0 2026-09-23 21:50:35,389 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=4860.0, ans=0.2514 2026-09-23 21:50:41,352 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=4893.333333333333, ans=0.7287333333333333 2026-09-23 21:50:46,736 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=4926.666666666667, ans=0.7275666666666667 2026-09-23 21:50:58,655 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=4993.333333333333, ans=0.25006666666666666 2026-09-23 21:50:59,670 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=4993.333333333333, ans=0.27490000000000003 2026-09-23 21:51:00,005 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 3.024e+02 3.952e+02 4.539e+02 5.350e+02 1.193e+03, threshold=9.079e+02, percent-clipped=2.0 2026-09-23 21:51:02,060 INFO [train.py:1192] (0/2) Epoch 2, batch 550, loss[loss=0.7025, simple_loss=0.6579, pruned_loss=0.3758, over 24238.00 frames. ], tot_loss[loss=0.633, simple_loss=0.5991, pruned_loss=0.335, over 4521719.59 frames. ], batch size: 257, lr: 3.88e-02, grad_scale: 32.0 2026-09-23 21:51:03,754 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=5026.666666666667, ans=0.264375 2026-09-23 21:51:04,030 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.16 vs. limit=9.385 2026-09-23 21:51:11,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=5060.0, ans=0.27590000000000003 2026-09-23 21:51:16,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=5093.333333333333, ans=0.26125 2026-09-23 21:51:21,469 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=5126.666666666667, ans=0.25968749999999996 2026-09-23 21:51:21,666 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.38 vs. limit=11.345 2026-09-23 21:51:27,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=5160.0, ans=0.258125 2026-09-23 21:51:27,505 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=5160.0, ans=0.258125 2026-09-23 21:51:29,341 INFO [train.py:1192] (0/2) Epoch 2, batch 600, loss[loss=0.6781, simple_loss=0.642, pruned_loss=0.3571, over 24417.00 frames. ], tot_loss[loss=0.6279, simple_loss=0.5966, pruned_loss=0.3302, over 4587894.30 frames. ], batch size: 235, lr: 3.88e-02, grad_scale: 32.0 2026-09-23 21:51:30,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=5193.333333333333, ans=0.24806666666666666 2026-09-23 21:51:51,092 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.29 vs. limit=7.663333333333334 2026-09-23 21:51:53,922 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.776e+02 3.843e+02 4.344e+02 5.247e+02 1.354e+03, threshold=8.688e+02, percent-clipped=5.0 2026-09-23 21:51:56,397 INFO [train.py:1192] (0/2) Epoch 2, batch 650, loss[loss=0.5531, simple_loss=0.5512, pruned_loss=0.2714, over 24555.00 frames. ], tot_loss[loss=0.6159, simple_loss=0.5894, pruned_loss=0.3205, over 4652644.39 frames. ], batch size: 162, lr: 3.88e-02, grad_scale: 32.0 2026-09-23 21:52:06,556 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=5426.666666666667, ans=0.7100666666666667 2026-09-23 21:52:07,025 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=5426.666666666667, ans=0.24562499999999998 2026-09-23 21:52:17,079 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=5460.0, ans=0.24406250000000002 2026-09-23 21:52:23,716 INFO [train.py:1192] (0/2) Epoch 2, batch 700, loss[loss=0.5403, simple_loss=0.535, pruned_loss=0.2683, over 24578.00 frames. ], tot_loss[loss=0.6102, simple_loss=0.5867, pruned_loss=0.3154, over 4684008.51 frames. ], batch size: 154, lr: 3.88e-02, grad_scale: 32.0 2026-09-23 21:52:27,005 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=5526.666666666667, ans=0.04363888888888889 2026-09-23 21:52:34,576 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=5593.333333333333, ans=0.06504166666666666 2026-09-23 21:52:43,029 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=5626.666666666667, ans=0.23625000000000002 2026-09-23 21:52:48,172 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 3.265e+02 4.700e+02 5.470e+02 6.470e+02 9.806e+02, threshold=1.094e+03, percent-clipped=5.0 2026-09-23 21:52:49,843 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=5693.333333333333, ans=0.009631884057971015 2026-09-23 21:52:50,326 INFO [train.py:1192] (0/2) Epoch 2, batch 750, loss[loss=0.5986, simple_loss=0.5857, pruned_loss=0.3027, over 24557.00 frames. ], tot_loss[loss=0.604, simple_loss=0.5827, pruned_loss=0.311, over 4711704.90 frames. ], batch size: 170, lr: 3.87e-02, grad_scale: 32.0 2026-09-23 21:52:50,526 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=5693.333333333333, ans=11.77 2026-09-23 21:52:53,665 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.94 vs. limit=7.846666666666666 2026-09-23 21:52:57,687 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=5726.666666666667, ans=0.2315625 2026-09-23 21:52:59,945 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=5726.666666666667, ans=0.24273333333333333 2026-09-23 21:53:04,380 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=5760.0, ans=0.22999999999999998 2026-09-23 21:53:07,740 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=5793.333333333333, ans=0.6972333333333334 2026-09-23 21:53:12,041 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=5826.666666666667, ans=0.6960666666666667 2026-09-23 21:53:17,093 INFO [train.py:1192] (0/2) Epoch 2, batch 800, loss[loss=0.5053, simple_loss=0.5094, pruned_loss=0.2466, over 24567.00 frames. ], tot_loss[loss=0.5979, simple_loss=0.5795, pruned_loss=0.3061, over 4740989.30 frames. ], batch size: 137, lr: 3.87e-02, grad_scale: 32.0 2026-09-23 21:53:18,204 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=5860.0, ans=0.22531250000000003 2026-09-23 21:53:41,968 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.936e+02 4.336e+02 5.358e+02 6.430e+02 1.287e+03, threshold=1.072e+03, percent-clipped=2.0 2026-09-23 21:53:44,093 INFO [train.py:1192] (0/2) Epoch 2, batch 850, loss[loss=0.6284, simple_loss=0.6146, pruned_loss=0.3192, over 24560.00 frames. ], tot_loss[loss=0.5908, simple_loss=0.5757, pruned_loss=0.3007, over 4762238.76 frames. ], batch size: 204, lr: 3.87e-02, grad_scale: 32.0 2026-09-23 21:53:55,718 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=6093.333333333333, ans=0.009544927536231883 2026-09-23 21:53:57,710 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=6093.333333333333, ans=0.21437499999999998 2026-09-23 21:54:02,209 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=6126.666666666667, ans=0.8112666666666667 2026-09-23 21:54:10,543 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=12.145 2026-09-23 21:54:10,814 INFO [train.py:1192] (0/2) Epoch 2, batch 900, loss[loss=0.483, simple_loss=0.4967, pruned_loss=0.2318, over 24557.00 frames. ], tot_loss[loss=0.5862, simple_loss=0.5733, pruned_loss=0.2973, over 4775381.14 frames. ], batch size: 137, lr: 3.86e-02, grad_scale: 32.0 2026-09-23 21:54:11,412 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=6193.333333333333, ans=0.009523188405797101 2026-09-23 21:54:16,555 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=6226.666666666667, ans=0.009515942028985508 2026-09-23 21:54:19,658 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=6226.666666666667, ans=0.6820666666666667 2026-09-23 21:54:27,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=6293.333333333333, ans=0.20500000000000002 2026-09-23 21:54:35,291 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=6326.666666666667, ans=0.2034375 2026-09-23 21:54:35,895 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.50 vs. limit=9.8725 2026-09-23 21:54:36,147 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.813e+02 3.863e+02 4.821e+02 6.210e+02 1.366e+03, threshold=9.642e+02, percent-clipped=4.0 2026-09-23 21:54:37,675 INFO [train.py:1192] (0/2) Epoch 2, batch 950, loss[loss=0.7602, simple_loss=0.6425, pruned_loss=0.4424, over 11586.00 frames. ], tot_loss[loss=0.5833, simple_loss=0.5705, pruned_loss=0.2961, over 4709488.41 frames. ], batch size: 333, lr: 3.86e-02, grad_scale: 16.0 2026-09-23 21:54:40,873 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=6360.0, ans=0.04016666666666667 2026-09-23 21:54:42,298 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-2.pt 2026-09-23 21:54:50,007 INFO [train.py:1192] (0/2) Epoch 3, batch 0, loss[loss=0.5127, simple_loss=0.5251, pruned_loss=0.2486, over 24559.00 frames. ], tot_loss[loss=0.5127, simple_loss=0.5251, pruned_loss=0.2486, over 24559.00 frames. ], batch size: 137, lr: 3.67e-02, grad_scale: 32.0 2026-09-23 21:54:50,007 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 21:55:01,602 INFO [train.py:1224] (0/2) Epoch 3, validation: loss=0.3463, simple_loss=0.4172, pruned_loss=0.1341, over 2564189.00 frames. 2026-09-23 21:55:01,602 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 21:55:01,840 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.57 vs. limit=9.895 2026-09-23 21:55:02,762 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.12 vs. limit=9.895 2026-09-23 21:55:07,606 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=6420.0, ans=0.0 2026-09-23 21:55:11,953 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=6453.333333333333, ans=0.1975 2026-09-23 21:55:12,471 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=6453.333333333333, ans=0.6741333333333334 2026-09-23 21:55:19,849 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 21:55:22,894 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.53 vs. limit=6.63 2026-09-23 21:55:23,278 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=6520.0, ans=0.19437500000000002 2026-09-23 21:55:27,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=6520.0, ans=0.09899494936611666 2026-09-23 21:55:28,190 INFO [train.py:1192] (0/2) Epoch 3, batch 50, loss[loss=0.4721, simple_loss=0.4899, pruned_loss=0.2264, over 24262.00 frames. ], tot_loss[loss=0.5809, simple_loss=0.5759, pruned_loss=0.2923, over 1075973.55 frames. ], batch size: 125, lr: 3.67e-02, grad_scale: 32.0 2026-09-23 21:55:48,716 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.250e+02 3.740e+02 4.895e+02 5.677e+02 1.442e+03, threshold=9.791e+02, percent-clipped=3.0 2026-09-23 21:55:54,571 INFO [train.py:1192] (0/2) Epoch 3, batch 100, loss[loss=0.4863, simple_loss=0.5076, pruned_loss=0.2325, over 24592.00 frames. ], tot_loss[loss=0.5742, simple_loss=0.574, pruned_loss=0.2868, over 1904029.68 frames. ], batch size: 154, lr: 3.66e-02, grad_scale: 32.0 2026-09-23 21:55:59,243 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=6753.333333333333, ans=0.03852777777777778 2026-09-23 21:55:59,248 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=6753.333333333333, ans=0.03852777777777778 2026-09-23 21:56:00,667 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=6753.333333333333, ans=0.18343749999999998 2026-09-23 21:56:19,019 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.83 vs. limit=12.64 2026-09-23 21:56:20,704 INFO [train.py:1192] (0/2) Epoch 3, batch 150, loss[loss=0.4782, simple_loss=0.4958, pruned_loss=0.2303, over 24269.00 frames. ], tot_loss[loss=0.5579, simple_loss=0.563, pruned_loss=0.2762, over 2557884.44 frames. ], batch size: 125, lr: 3.66e-02, grad_scale: 32.0 2026-09-23 21:56:22,262 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=6886.666666666667, ans=0.1771875 2026-09-23 21:56:30,112 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=6920.0, ans=0.17562499999999998 2026-09-23 21:56:41,069 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.211e+02 4.076e+02 4.435e+02 5.203e+02 9.887e+02, threshold=8.870e+02, percent-clipped=1.0 2026-09-23 21:56:46,643 INFO [train.py:1192] (0/2) Epoch 3, batch 200, loss[loss=0.6313, simple_loss=0.5972, pruned_loss=0.3327, over 21014.00 frames. ], tot_loss[loss=0.5512, simple_loss=0.5584, pruned_loss=0.2718, over 3054655.54 frames. ], batch size: 333, lr: 3.66e-02, grad_scale: 32.0 2026-09-23 21:56:47,684 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=7053.333333333333, ans=0.169375 2026-09-23 21:57:04,162 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=7153.333333333333, ans=0.22846666666666665 2026-09-23 21:57:07,529 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=7186.666666666667, ans=0.16312500000000002 2026-09-23 21:57:10,248 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.45 vs. limit=10.195 2026-09-23 21:57:13,124 INFO [train.py:1192] (0/2) Epoch 3, batch 250, loss[loss=0.5829, simple_loss=0.6012, pruned_loss=0.2823, over 24305.00 frames. ], tot_loss[loss=0.5473, simple_loss=0.5562, pruned_loss=0.269, over 3443538.86 frames. ], batch size: 234, lr: 3.65e-02, grad_scale: 32.0 2026-09-23 21:57:23,094 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=7286.666666666667, ans=0.1584375 2026-09-23 21:57:33,554 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.322e+02 4.274e+02 5.049e+02 6.099e+02 1.100e+03, threshold=1.010e+03, percent-clipped=5.0 2026-09-23 21:57:35,766 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=7353.333333333333, ans=0.15531250000000002 2026-09-23 21:57:39,833 INFO [train.py:1192] (0/2) Epoch 3, batch 300, loss[loss=0.5777, simple_loss=0.5938, pruned_loss=0.2808, over 24564.00 frames. ], tot_loss[loss=0.5418, simple_loss=0.5526, pruned_loss=0.2655, over 3750296.40 frames. ], batch size: 204, lr: 3.65e-02, grad_scale: 32.0 2026-09-23 21:57:41,840 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=7386.666666666667, ans=0.1761333333333333 2026-09-23 21:57:55,368 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=13.115 2026-09-23 21:57:56,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=7486.666666666667, ans=0.1490625 2026-09-23 21:58:03,099 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=7520.0, ans=0.14750000000000002 2026-09-23 21:58:05,597 INFO [train.py:1192] (0/2) Epoch 3, batch 350, loss[loss=0.4547, simple_loss=0.485, pruned_loss=0.2122, over 24557.00 frames. ], tot_loss[loss=0.5383, simple_loss=0.5512, pruned_loss=0.2626, over 3994147.49 frames. ], batch size: 137, lr: 3.65e-02, grad_scale: 32.0 2026-09-23 21:58:07,734 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=7553.333333333333, ans=0.009227536231884059 2026-09-23 21:58:08,902 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.52 vs. limit=10.3325 2026-09-23 21:58:15,345 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=7620.0, ans=0.6333 2026-09-23 21:58:25,052 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=7653.333333333333, ans=0.6321333333333334 2026-09-23 21:58:25,123 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys.whitening_limit, batch_count=7653.333333333333, ans=4.148 2026-09-23 21:58:25,886 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.871e+02 4.157e+02 4.789e+02 6.123e+02 1.192e+03, threshold=9.578e+02, percent-clipped=3.0 2026-09-23 21:58:31,667 INFO [train.py:1192] (0/2) Epoch 3, batch 400, loss[loss=0.521, simple_loss=0.5403, pruned_loss=0.2509, over 24574.00 frames. ], tot_loss[loss=0.5352, simple_loss=0.5487, pruned_loss=0.2608, over 4179131.99 frames. ], batch size: 170, lr: 3.64e-02, grad_scale: 32.0 2026-09-23 21:58:34,071 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=7720.0, ans=0.138125 2026-09-23 21:58:49,761 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=7820.0, ans=0.2218 2026-09-23 21:58:49,774 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=7820.0, ans=0.1334375 2026-09-23 21:58:50,192 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=7820.0, ans=0.1334375 2026-09-23 21:58:52,163 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.26 vs. limit=13.365 2026-09-23 21:58:57,804 INFO [train.py:1192] (0/2) Epoch 3, batch 450, loss[loss=0.5139, simple_loss=0.5463, pruned_loss=0.2408, over 24627.00 frames. ], tot_loss[loss=0.533, simple_loss=0.5477, pruned_loss=0.2591, over 4312561.84 frames. ], batch size: 175, lr: 3.64e-02, grad_scale: 32.0 2026-09-23 21:58:59,599 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=7886.666666666667, ans=0.04949747468305833 2026-09-23 21:59:04,248 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=7920.0, ans=0.12874999999999998 2026-09-23 21:59:06,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=7920.0, ans=0.03366666666666667 2026-09-23 21:59:17,841 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.96 vs. limit=13.49 2026-09-23 21:59:18,105 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.733e+02 3.882e+02 4.552e+02 5.545e+02 7.804e+02, threshold=9.105e+02, percent-clipped=0.0 2026-09-23 21:59:21,852 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=8020.0, ans=0.2198 2026-09-23 21:59:23,732 INFO [train.py:1192] (0/2) Epoch 3, batch 500, loss[loss=0.5382, simple_loss=0.5713, pruned_loss=0.2525, over 24494.00 frames. ], tot_loss[loss=0.5274, simple_loss=0.5442, pruned_loss=0.2553, over 4430065.06 frames. ], batch size: 218, lr: 3.64e-02, grad_scale: 32.0 2026-09-23 21:59:24,908 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=8053.333333333333, ans=0.03311111111111112 2026-09-23 21:59:32,136 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.49 vs. limit=10.5325 2026-09-23 21:59:46,601 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=8186.666666666667, ans=0.21813333333333335 2026-09-23 21:59:49,579 INFO [train.py:1192] (0/2) Epoch 3, batch 550, loss[loss=0.5698, simple_loss=0.5829, pruned_loss=0.2783, over 24270.00 frames. ], tot_loss[loss=0.5263, simple_loss=0.5441, pruned_loss=0.2543, over 4519365.59 frames. ], batch size: 257, lr: 3.63e-02, grad_scale: 32.0 2026-09-23 22:00:03,860 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:00:07,759 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=8320.0, ans=0.009060869565217391 2026-09-23 22:00:09,171 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=8320.0, ans=0.125 2026-09-23 22:00:10,144 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.877e+02 4.060e+02 4.625e+02 5.385e+02 1.045e+03, threshold=9.250e+02, percent-clipped=2.0 2026-09-23 22:00:16,071 INFO [train.py:1192] (0/2) Epoch 3, batch 600, loss[loss=0.5139, simple_loss=0.5489, pruned_loss=0.2394, over 24315.00 frames. ], tot_loss[loss=0.5233, simple_loss=0.5427, pruned_loss=0.2519, over 4585486.91 frames. ], batch size: 234, lr: 3.63e-02, grad_scale: 32.0 2026-09-23 22:00:24,511 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=8420.0, ans=0.03158333333333334 2026-09-23 22:00:30,966 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=8453.333333333334, ans=0.0 2026-09-23 22:00:41,379 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:00:42,215 INFO [train.py:1192] (0/2) Epoch 3, batch 650, loss[loss=0.4941, simple_loss=0.525, pruned_loss=0.2316, over 24565.00 frames. ], tot_loss[loss=0.5173, simple_loss=0.5391, pruned_loss=0.2477, over 4651008.74 frames. ], batch size: 162, lr: 3.63e-02, grad_scale: 32.0 2026-09-23 22:00:44,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=8553.333333333334, ans=0.16446666666666665 2026-09-23 22:00:50,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=8586.666666666666, ans=0.125 2026-09-23 22:00:57,295 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=8653.333333333334, ans=0.125 2026-09-23 22:01:02,476 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.648e+02 3.944e+02 4.523e+02 5.804e+02 9.829e+02, threshold=9.045e+02, percent-clipped=2.0 2026-09-23 22:01:04,941 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=8686.666666666666, ans=0.125 2026-09-23 22:01:08,251 INFO [train.py:1192] (0/2) Epoch 3, batch 700, loss[loss=0.4543, simple_loss=0.4962, pruned_loss=0.2062, over 24572.00 frames. ], tot_loss[loss=0.5147, simple_loss=0.538, pruned_loss=0.2457, over 4682537.34 frames. ], batch size: 154, lr: 3.62e-02, grad_scale: 32.0 2026-09-23 22:01:09,360 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=8720.0, ans=0.2128 2026-09-23 22:01:10,356 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:01:15,792 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=8753.333333333334, ans=0.125 2026-09-23 22:01:24,708 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=8820.0, ans=0.029916666666666668 2026-09-23 22:01:33,921 INFO [train.py:1192] (0/2) Epoch 3, batch 750, loss[loss=0.4852, simple_loss=0.5269, pruned_loss=0.2217, over 24548.00 frames. ], tot_loss[loss=0.5117, simple_loss=0.5358, pruned_loss=0.2438, over 4710134.16 frames. ], batch size: 170, lr: 3.62e-02, grad_scale: 32.0 2026-09-23 22:01:35,505 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=8886.666666666666, ans=0.125 2026-09-23 22:01:51,592 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.37 vs. limit=4.348 2026-09-23 22:01:54,439 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 3.161e+02 4.265e+02 5.051e+02 6.444e+02 1.055e+03, threshold=1.010e+03, percent-clipped=3.0 2026-09-23 22:01:58,306 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=9020.0, ans=0.2098 2026-09-23 22:01:59,743 INFO [train.py:1192] (0/2) Epoch 3, batch 800, loss[loss=0.4124, simple_loss=0.4606, pruned_loss=0.1821, over 24558.00 frames. ], tot_loss[loss=0.5083, simple_loss=0.5337, pruned_loss=0.2415, over 4735515.96 frames. ], batch size: 137, lr: 3.61e-02, grad_scale: 32.0 2026-09-23 22:02:19,981 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=5.80 vs. limit=10.932500000000001 2026-09-23 22:02:21,946 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.69 vs. limit=14.39 2026-09-23 22:02:25,517 INFO [train.py:1192] (0/2) Epoch 3, batch 850, loss[loss=0.5371, simple_loss=0.5584, pruned_loss=0.2579, over 24600.00 frames. ], tot_loss[loss=0.5046, simple_loss=0.5315, pruned_loss=0.2388, over 4758370.20 frames. ], batch size: 198, lr: 3.61e-02, grad_scale: 32.0 2026-09-23 22:02:29,886 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=9220.0, ans=0.008865217391304348 2026-09-23 22:02:33,587 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=9253.333333333334, ans=0.125 2026-09-23 22:02:40,550 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=9320.0, ans=0.20679999999999998 2026-09-23 22:02:45,546 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.677e+02 3.885e+02 4.421e+02 5.168e+02 9.776e+02, threshold=8.842e+02, percent-clipped=0.0 2026-09-23 22:02:49,042 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=9353.333333333334, ans=0.027694444444444445 2026-09-23 22:02:50,920 INFO [train.py:1192] (0/2) Epoch 3, batch 900, loss[loss=0.3995, simple_loss=0.4607, pruned_loss=0.1691, over 24570.00 frames. ], tot_loss[loss=0.5031, simple_loss=0.5308, pruned_loss=0.2376, over 4772541.15 frames. ], batch size: 137, lr: 3.61e-02, grad_scale: 32.0 2026-09-23 22:03:16,234 INFO [train.py:1192] (0/2) Epoch 3, batch 950, loss[loss=0.6919, simple_loss=0.6138, pruned_loss=0.385, over 11686.00 frames. ], tot_loss[loss=0.5042, simple_loss=0.5296, pruned_loss=0.2394, over 4712354.44 frames. ], batch size: 333, lr: 3.60e-02, grad_scale: 16.0 2026-09-23 22:03:18,916 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=9553.333333333334, ans=0.008792753623188406 2026-09-23 22:03:19,604 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=14.14 vs. limit=11.0825 2026-09-23 22:03:20,817 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-3.pt 2026-09-23 22:03:28,076 INFO [train.py:1192] (0/2) Epoch 4, batch 0, loss[loss=0.4594, simple_loss=0.492, pruned_loss=0.2134, over 24575.00 frames. ], tot_loss[loss=0.4594, simple_loss=0.492, pruned_loss=0.2134, over 24575.00 frames. ], batch size: 137, lr: 3.37e-02, grad_scale: 32.0 2026-09-23 22:03:28,076 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 22:03:39,679 INFO [train.py:1224] (0/2) Epoch 4, validation: loss=0.2977, simple_loss=0.3887, pruned_loss=0.1033, over 2564189.00 frames. 2026-09-23 22:03:39,679 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 22:03:47,916 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.26 vs. limit=4.442 2026-09-23 22:03:55,997 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.607e+02 3.936e+02 5.033e+02 6.810e+02 2.340e+03, threshold=1.007e+03, percent-clipped=8.0 2026-09-23 22:03:56,115 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=9680.0, ans=0.125 2026-09-23 22:04:01,638 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=9713.333333333334, ans=0.008757971014492754 2026-09-23 22:04:03,463 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=9713.333333333334, ans=0.125 2026-09-23 22:04:04,826 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=9713.333333333334, ans=0.125 2026-09-23 22:04:05,754 INFO [train.py:1192] (0/2) Epoch 4, batch 50, loss[loss=0.378, simple_loss=0.4334, pruned_loss=0.1613, over 24259.00 frames. ], tot_loss[loss=0.5185, simple_loss=0.542, pruned_loss=0.2475, over 1076452.75 frames. ], batch size: 125, lr: 3.36e-02, grad_scale: 32.0 2026-09-23 22:04:28,397 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=9880.0, ans=0.125 2026-09-23 22:04:30,630 INFO [train.py:1192] (0/2) Epoch 4, batch 100, loss[loss=0.4261, simple_loss=0.4767, pruned_loss=0.1878, over 24598.00 frames. ], tot_loss[loss=0.51, simple_loss=0.5388, pruned_loss=0.2406, over 1904522.17 frames. ], batch size: 154, lr: 3.36e-02, grad_scale: 32.0 2026-09-23 22:04:37,498 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:04:38,438 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=9946.666666666666, ans=0.008707246376811594 2026-09-23 22:04:46,887 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.831e+02 3.899e+02 4.639e+02 5.288e+02 7.699e+02, threshold=9.278e+02, percent-clipped=0.0 2026-09-23 22:04:48,348 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=10013.333333333334, ans=0.008692753623188406 2026-09-23 22:04:53,371 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=10046.666666666666, ans=0.02480555555555556 2026-09-23 22:04:53,393 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=10046.666666666666, ans=0.0 2026-09-23 22:04:56,342 INFO [train.py:1192] (0/2) Epoch 4, batch 150, loss[loss=0.3992, simple_loss=0.4407, pruned_loss=0.1789, over 24278.00 frames. ], tot_loss[loss=0.4978, simple_loss=0.5298, pruned_loss=0.2329, over 2558021.76 frames. ], batch size: 125, lr: 3.36e-02, grad_scale: 32.0 2026-09-23 22:04:57,101 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=6.48 vs. limit=8.032 2026-09-23 22:05:11,915 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=10180.0, ans=0.024250000000000004 2026-09-23 22:05:19,852 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=10213.333333333334, ans=0.0 2026-09-23 22:05:21,816 INFO [train.py:1192] (0/2) Epoch 4, batch 200, loss[loss=0.5601, simple_loss=0.5578, pruned_loss=0.2812, over 21142.00 frames. ], tot_loss[loss=0.4926, simple_loss=0.5259, pruned_loss=0.2296, over 3054196.73 frames. ], batch size: 333, lr: 3.35e-02, grad_scale: 32.0 2026-09-23 22:05:27,071 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=10280.0, ans=0.008634782608695652 2026-09-23 22:05:28,012 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=10280.0, ans=0.008634782608695652 2026-09-23 22:05:36,192 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=10313.333333333334, ans=0.19686666666666666 2026-09-23 22:05:38,187 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.870e+02 3.865e+02 4.525e+02 5.215e+02 8.851e+02, threshold=9.051e+02, percent-clipped=0.0 2026-09-23 22:05:43,163 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=10380.0, ans=0.125 2026-09-23 22:05:47,529 INFO [train.py:1192] (0/2) Epoch 4, batch 250, loss[loss=0.5583, simple_loss=0.5799, pruned_loss=0.2683, over 24309.00 frames. ], tot_loss[loss=0.4884, simple_loss=0.5233, pruned_loss=0.2267, over 3444089.33 frames. ], batch size: 234, lr: 3.35e-02, grad_scale: 32.0 2026-09-23 22:05:54,854 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=10446.666666666666, ans=0.125 2026-09-23 22:06:13,907 INFO [train.py:1192] (0/2) Epoch 4, batch 300, loss[loss=0.5455, simple_loss=0.5769, pruned_loss=0.2571, over 24537.00 frames. ], tot_loss[loss=0.4843, simple_loss=0.5204, pruned_loss=0.2241, over 3748304.79 frames. ], batch size: 204, lr: 3.34e-02, grad_scale: 32.0 2026-09-23 22:06:20,762 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=6.03 vs. limit=8.245333333333335 2026-09-23 22:06:22,134 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.70 vs. limit=11.48 2026-09-23 22:06:24,924 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=10646.666666666666, ans=0.125 2026-09-23 22:06:26,553 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=10646.666666666666, ans=0.02230555555555556 2026-09-23 22:06:26,567 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=10646.666666666666, ans=0.125 2026-09-23 22:06:29,738 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.647e+02 3.777e+02 4.403e+02 5.252e+02 8.106e+02, threshold=8.807e+02, percent-clipped=0.0 2026-09-23 22:06:32,712 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=10680.0, ans=0.125 2026-09-23 22:06:35,257 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.49 vs. limit=15.535 2026-09-23 22:06:38,930 INFO [train.py:1192] (0/2) Epoch 4, batch 350, loss[loss=0.4014, simple_loss=0.4492, pruned_loss=0.1768, over 24564.00 frames. ], tot_loss[loss=0.4833, simple_loss=0.5204, pruned_loss=0.2231, over 3991148.52 frames. ], batch size: 137, lr: 3.34e-02, grad_scale: 32.0 2026-09-23 22:06:39,005 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=10746.666666666666, ans=0.125 2026-09-23 22:06:40,578 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=10746.666666666666, ans=0.125 2026-09-23 22:06:47,332 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=10780.0, ans=0.021750000000000002 2026-09-23 22:06:56,529 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=10846.666666666666, ans=0.125 2026-09-23 22:06:57,103 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=10846.666666666666, ans=0.008511594202898551 2026-09-23 22:07:04,294 INFO [train.py:1192] (0/2) Epoch 4, batch 400, loss[loss=0.4897, simple_loss=0.524, pruned_loss=0.2277, over 24551.00 frames. ], tot_loss[loss=0.4791, simple_loss=0.5178, pruned_loss=0.2202, over 4174233.78 frames. ], batch size: 170, lr: 3.34e-02, grad_scale: 32.0 2026-09-23 22:07:13,685 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=10980.0, ans=0.125 2026-09-23 22:07:16,421 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=10980.0, ans=0.125 2026-09-23 22:07:20,156 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.811e+02 4.017e+02 4.684e+02 5.205e+02 7.719e+02, threshold=9.369e+02, percent-clipped=0.0 2026-09-23 22:07:29,420 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=11080.0, ans=0.0 2026-09-23 22:07:29,803 INFO [train.py:1192] (0/2) Epoch 4, batch 450, loss[loss=0.4971, simple_loss=0.5369, pruned_loss=0.2287, over 24618.00 frames. ], tot_loss[loss=0.4765, simple_loss=0.5162, pruned_loss=0.2184, over 4308644.63 frames. ], batch size: 175, lr: 3.33e-02, grad_scale: 32.0 2026-09-23 22:07:33,308 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=11080.0, ans=0.05 2026-09-23 22:07:38,056 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=11113.333333333334, ans=0.025 2026-09-23 22:07:44,889 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.64 vs. limit=4.677 2026-09-23 22:07:53,544 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=11213.333333333334, ans=0.0 2026-09-23 22:07:53,688 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.63 vs. limit=15.91 2026-09-23 22:07:54,473 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=11213.333333333334, ans=0.125 2026-09-23 22:07:55,434 INFO [train.py:1192] (0/2) Epoch 4, batch 500, loss[loss=0.4803, simple_loss=0.5334, pruned_loss=0.2135, over 24513.00 frames. ], tot_loss[loss=0.4731, simple_loss=0.5132, pruned_loss=0.2165, over 4426684.35 frames. ], batch size: 218, lr: 3.33e-02, grad_scale: 32.0 2026-09-23 22:08:09,878 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=11313.333333333334, ans=0.125 2026-09-23 22:08:11,950 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.881e+02 3.912e+02 4.426e+02 4.981e+02 8.897e+02, threshold=8.851e+02, percent-clipped=0.0 2026-09-23 22:08:13,106 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=11346.666666666666, ans=0.008402898550724638 2026-09-23 22:08:15,192 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=11346.666666666666, ans=0.18653333333333333 2026-09-23 22:08:16,245 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.16 vs. limit=11.7675 2026-09-23 22:08:20,977 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.93 vs. limit=10.706666666666667 2026-09-23 22:08:21,094 INFO [train.py:1192] (0/2) Epoch 4, batch 550, loss[loss=0.5419, simple_loss=0.5658, pruned_loss=0.259, over 24312.00 frames. ], tot_loss[loss=0.4721, simple_loss=0.5128, pruned_loss=0.2156, over 4516891.90 frames. ], batch size: 257, lr: 3.32e-02, grad_scale: 32.0 2026-09-23 22:08:29,018 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=11446.666666666666, ans=0.125 2026-09-23 22:08:46,463 INFO [train.py:1192] (0/2) Epoch 4, batch 600, loss[loss=0.5269, simple_loss=0.5584, pruned_loss=0.2477, over 24326.00 frames. ], tot_loss[loss=0.4703, simple_loss=0.5122, pruned_loss=0.2142, over 4584357.39 frames. ], batch size: 234, lr: 3.32e-02, grad_scale: 32.0 2026-09-23 22:08:47,775 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=11580.0, ans=0.8658 2026-09-23 22:08:48,838 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=11580.0, ans=0.125 2026-09-23 22:09:03,210 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.426e+02 3.617e+02 4.202e+02 4.992e+02 8.611e+02, threshold=8.403e+02, percent-clipped=0.0 2026-09-23 22:09:04,361 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.21 vs. limit=16.259999999999998 2026-09-23 22:09:12,191 INFO [train.py:1192] (0/2) Epoch 4, batch 650, loss[loss=0.4932, simple_loss=0.5266, pruned_loss=0.2299, over 24580.00 frames. ], tot_loss[loss=0.4676, simple_loss=0.5105, pruned_loss=0.2123, over 4649939.75 frames. ], batch size: 162, lr: 3.32e-02, grad_scale: 32.0 2026-09-23 22:09:21,369 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=11780.0, ans=0.017583333333333333 2026-09-23 22:09:25,149 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.23 vs. limit=11.93 2026-09-23 22:09:37,410 INFO [train.py:1192] (0/2) Epoch 4, batch 700, loss[loss=0.4228, simple_loss=0.4741, pruned_loss=0.1858, over 24562.00 frames. ], tot_loss[loss=0.4664, simple_loss=0.5103, pruned_loss=0.2112, over 4682130.47 frames. ], batch size: 154, lr: 3.31e-02, grad_scale: 32.0 2026-09-23 22:09:41,711 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=11913.333333333334, ans=0.017027777777777774 2026-09-23 22:09:50,556 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.58 vs. limit=11.9925 2026-09-23 22:09:52,598 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=12013.333333333334, ans=0.125 2026-09-23 22:09:53,890 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.737e+02 3.838e+02 4.537e+02 5.531e+02 8.302e+02, threshold=9.074e+02, percent-clipped=0.0 2026-09-23 22:09:53,944 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=12013.333333333334, ans=0.08887666666666667 2026-09-23 22:09:53,991 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=12013.333333333334, ans=0.17986666666666667 2026-09-23 22:10:00,324 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=12046.666666666666, ans=0.125 2026-09-23 22:10:03,374 INFO [train.py:1192] (0/2) Epoch 4, batch 750, loss[loss=0.4357, simple_loss=0.5, pruned_loss=0.1857, over 24569.00 frames. ], tot_loss[loss=0.4649, simple_loss=0.5089, pruned_loss=0.2105, over 4708956.84 frames. ], batch size: 170, lr: 3.31e-02, grad_scale: 32.0 2026-09-23 22:10:03,457 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=12080.0, ans=0.17919999999999997 2026-09-23 22:10:09,838 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=12113.333333333334, ans=0.07 2026-09-23 22:10:12,130 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=12113.333333333334, ans=0.125 2026-09-23 22:10:27,550 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=12213.333333333334, ans=0.125 2026-09-23 22:10:29,563 INFO [train.py:1192] (0/2) Epoch 4, batch 800, loss[loss=0.4254, simple_loss=0.4645, pruned_loss=0.1932, over 24540.00 frames. ], tot_loss[loss=0.4644, simple_loss=0.5083, pruned_loss=0.2103, over 4734387.09 frames. ], batch size: 137, lr: 3.30e-02, grad_scale: 32.0 2026-09-23 22:10:30,574 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=12246.666666666666, ans=0.4713666666666667 2026-09-23 22:10:37,001 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=12280.0, ans=0.0155 2026-09-23 22:10:45,945 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.897e+02 4.015e+02 4.569e+02 5.501e+02 1.043e+03, threshold=9.139e+02, percent-clipped=1.0 2026-09-23 22:10:48,114 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=12346.666666666666, ans=0.125 2026-09-23 22:10:55,550 INFO [train.py:1192] (0/2) Epoch 4, batch 850, loss[loss=0.5052, simple_loss=0.5499, pruned_loss=0.2303, over 24557.00 frames. ], tot_loss[loss=0.4631, simple_loss=0.5075, pruned_loss=0.2094, over 4757151.14 frames. ], batch size: 204, lr: 3.30e-02, grad_scale: 32.0 2026-09-23 22:10:55,638 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=12413.333333333334, ans=0.125 2026-09-23 22:11:03,466 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=12446.666666666666, ans=0.125 2026-09-23 22:11:15,037 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=12513.333333333334, ans=0.125 2026-09-23 22:11:21,290 INFO [train.py:1192] (0/2) Epoch 4, batch 900, loss[loss=0.3846, simple_loss=0.4464, pruned_loss=0.1614, over 24558.00 frames. ], tot_loss[loss=0.463, simple_loss=0.5074, pruned_loss=0.2093, over 4771185.11 frames. ], batch size: 137, lr: 3.29e-02, grad_scale: 32.0 2026-09-23 22:11:27,989 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=12613.333333333334, ans=0.014111111111111109 2026-09-23 22:11:29,574 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=12613.333333333334, ans=0.17386666666666667 2026-09-23 22:11:32,310 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=12.2425 2026-09-23 22:11:38,347 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.512e+02 3.730e+02 4.472e+02 5.508e+02 1.100e+03, threshold=8.944e+02, percent-clipped=1.0 2026-09-23 22:11:38,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=12680.0, ans=0.125 2026-09-23 22:11:41,185 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=20.34 vs. limit=12.254999999999999 2026-09-23 22:11:46,876 INFO [train.py:1192] (0/2) Epoch 4, batch 950, loss[loss=0.6126, simple_loss=0.5609, pruned_loss=0.3321, over 10895.00 frames. ], tot_loss[loss=0.4629, simple_loss=0.5056, pruned_loss=0.2101, over 4709962.16 frames. ], batch size: 333, lr: 3.29e-02, grad_scale: 32.0 2026-09-23 22:11:51,144 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-4.pt 2026-09-23 22:11:58,019 INFO [train.py:1192] (0/2) Epoch 5, batch 0, loss[loss=0.4228, simple_loss=0.4739, pruned_loss=0.1858, over 24566.00 frames. ], tot_loss[loss=0.4228, simple_loss=0.4739, pruned_loss=0.1858, over 24566.00 frames. ], batch size: 137, lr: 3.06e-02, grad_scale: 32.0 2026-09-23 22:11:58,019 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 22:12:09,818 INFO [train.py:1224] (0/2) Epoch 5, validation: loss=0.2774, simple_loss=0.3744, pruned_loss=0.09019, over 2564189.00 frames. 2026-09-23 22:12:09,819 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 22:12:22,791 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=12840.0, ans=0.125 2026-09-23 22:12:35,658 INFO [train.py:1192] (0/2) Epoch 5, batch 50, loss[loss=0.4006, simple_loss=0.4469, pruned_loss=0.1771, over 24244.00 frames. ], tot_loss[loss=0.4741, simple_loss=0.5156, pruned_loss=0.2163, over 1076402.51 frames. ], batch size: 125, lr: 3.06e-02, grad_scale: 32.0 2026-09-23 22:12:43,314 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=12973.333333333334, ans=0.17026666666666665 2026-09-23 22:12:48,346 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.520e+02 3.699e+02 4.389e+02 5.018e+02 1.310e+03, threshold=8.778e+02, percent-clipped=1.0 2026-09-23 22:12:53,784 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.97 vs. limit=4.9559999999999995 2026-09-23 22:13:01,218 INFO [train.py:1192] (0/2) Epoch 5, batch 100, loss[loss=0.4069, simple_loss=0.4729, pruned_loss=0.1704, over 24637.00 frames. ], tot_loss[loss=0.4686, simple_loss=0.5148, pruned_loss=0.2112, over 1904512.45 frames. ], batch size: 154, lr: 3.05e-02, grad_scale: 32.0 2026-09-23 22:13:01,870 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=13106.666666666666, ans=0.4412666666666667 2026-09-23 22:13:05,176 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=13106.666666666666, ans=0.16893333333333332 2026-09-23 22:13:05,242 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=13106.666666666666, ans=0.125 2026-09-23 22:13:07,685 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=13140.0, ans=0.1686 2026-09-23 22:13:11,641 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=13173.333333333334, ans=0.125 2026-09-23 22:13:13,192 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=18.53 vs. limit=17.380000000000003 2026-09-23 22:13:24,982 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:13:26,874 INFO [train.py:1192] (0/2) Epoch 5, batch 150, loss[loss=0.3848, simple_loss=0.4307, pruned_loss=0.1694, over 24257.00 frames. ], tot_loss[loss=0.4573, simple_loss=0.5061, pruned_loss=0.2043, over 2558194.23 frames. ], batch size: 125, lr: 3.05e-02, grad_scale: 32.0 2026-09-23 22:13:35,493 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-4000.pt 2026-09-23 22:13:35,849 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=13306.666666666666, ans=0.16693333333333335 2026-09-23 22:13:39,237 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.580e+02 3.450e+02 4.364e+02 5.078e+02 1.050e+03, threshold=8.729e+02, percent-clipped=2.0 2026-09-23 22:13:43,546 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=13373.333333333334, ans=0.125 2026-09-23 22:13:47,302 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=13406.666666666666, ans=0.007955072463768116 2026-09-23 22:13:48,432 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=13406.666666666666, ans=0.43076666666666674 2026-09-23 22:13:52,317 INFO [train.py:1192] (0/2) Epoch 5, batch 200, loss[loss=0.5305, simple_loss=0.5404, pruned_loss=0.2603, over 21052.00 frames. ], tot_loss[loss=0.4511, simple_loss=0.5016, pruned_loss=0.2003, over 3055652.17 frames. ], batch size: 333, lr: 3.05e-02, grad_scale: 32.0 2026-09-23 22:14:17,791 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=13573.333333333334, ans=11.786666666666667 2026-09-23 22:14:18,453 INFO [train.py:1192] (0/2) Epoch 5, batch 250, loss[loss=0.4964, simple_loss=0.5523, pruned_loss=0.2203, over 24303.00 frames. ], tot_loss[loss=0.4504, simple_loss=0.5009, pruned_loss=0.2, over 3445479.00 frames. ], batch size: 234, lr: 3.04e-02, grad_scale: 32.0 2026-09-23 22:14:27,382 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=13640.0, ans=10.0 2026-09-23 22:14:31,135 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.684e+02 3.685e+02 4.372e+02 5.234e+02 9.542e+02, threshold=8.745e+02, percent-clipped=1.0 2026-09-23 22:14:39,956 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=13740.0, ans=0.16260000000000002 2026-09-23 22:14:44,620 INFO [train.py:1192] (0/2) Epoch 5, batch 300, loss[loss=0.4387, simple_loss=0.5139, pruned_loss=0.1817, over 24551.00 frames. ], tot_loss[loss=0.4492, simple_loss=0.4996, pruned_loss=0.1994, over 3750015.91 frames. ], batch size: 204, lr: 3.04e-02, grad_scale: 32.0 2026-09-23 22:14:48,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=13773.333333333334, ans=0.125 2026-09-23 22:14:50,560 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=13806.666666666666, ans=0.007868115942028986 2026-09-23 22:15:00,579 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=13873.333333333334, ans=0.4144333333333333 2026-09-23 22:15:10,805 INFO [train.py:1192] (0/2) Epoch 5, batch 350, loss[loss=0.396, simple_loss=0.449, pruned_loss=0.1715, over 24543.00 frames. ], tot_loss[loss=0.4479, simple_loss=0.4992, pruned_loss=0.1984, over 3993793.03 frames. ], batch size: 137, lr: 3.03e-02, grad_scale: 32.0 2026-09-23 22:15:15,608 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=13973.333333333334, ans=0.125 2026-09-23 22:15:21,187 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=14006.666666666666, ans=0.008305555555555559 2026-09-23 22:15:23,484 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.338e+02 3.709e+02 4.180e+02 5.205e+02 8.413e+02, threshold=8.359e+02, percent-clipped=0.0 2026-09-23 22:15:31,014 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=14073.333333333334, ans=0.125 2026-09-23 22:15:31,651 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=6.53 vs. limit=12.7775 2026-09-23 22:15:33,558 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=14073.333333333334, ans=0.125 2026-09-23 22:15:37,106 INFO [train.py:1192] (0/2) Epoch 5, batch 400, loss[loss=0.4243, simple_loss=0.4893, pruned_loss=0.1796, over 24570.00 frames. ], tot_loss[loss=0.4454, simple_loss=0.4972, pruned_loss=0.1968, over 4177762.63 frames. ], batch size: 170, lr: 3.03e-02, grad_scale: 32.0 2026-09-23 22:15:48,219 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:15:53,490 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=14206.666666666666, ans=0.007781159420289855 2026-09-23 22:15:58,560 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=14240.0, ans=0.125 2026-09-23 22:16:03,114 INFO [train.py:1192] (0/2) Epoch 5, batch 450, loss[loss=0.4476, simple_loss=0.5075, pruned_loss=0.1939, over 24631.00 frames. ], tot_loss[loss=0.4435, simple_loss=0.4961, pruned_loss=0.1954, over 4311251.90 frames. ], batch size: 175, lr: 3.02e-02, grad_scale: 32.0 2026-09-23 22:16:03,219 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=14273.333333333334, ans=0.15726666666666667 2026-09-23 22:16:09,485 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=14306.666666666666, ans=0.007055555555555558 2026-09-23 22:16:10,274 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=14306.666666666666, ans=0.125 2026-09-23 22:16:11,450 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.92 vs. limit=12.153333333333332 2026-09-23 22:16:12,318 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=14306.666666666666, ans=0.39926666666666677 2026-09-23 22:16:12,819 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=14340.0, ans=0.15660000000000002 2026-09-23 22:16:15,594 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.349e+02 3.590e+02 4.169e+02 4.919e+02 7.184e+02, threshold=8.339e+02, percent-clipped=0.0 2026-09-23 22:16:15,695 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=14340.0, ans=0.025 2026-09-23 22:16:18,866 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=14373.333333333334, ans=0.125 2026-09-23 22:16:18,908 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=14373.333333333334, ans=0.025 2026-09-23 22:16:28,211 INFO [train.py:1192] (0/2) Epoch 5, batch 500, loss[loss=0.451, simple_loss=0.5167, pruned_loss=0.1926, over 24470.00 frames. ], tot_loss[loss=0.4409, simple_loss=0.4939, pruned_loss=0.1939, over 4428374.14 frames. ], batch size: 218, lr: 3.02e-02, grad_scale: 16.0 2026-09-23 22:16:40,761 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:16:42,046 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=14506.666666666666, ans=0.15493333333333334 2026-09-23 22:16:51,021 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=14573.333333333334, ans=0.00594444444444444 2026-09-23 22:16:52,463 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=14573.333333333334, ans=0.007701449275362319 2026-09-23 22:16:54,237 INFO [train.py:1192] (0/2) Epoch 5, batch 550, loss[loss=0.4576, simple_loss=0.516, pruned_loss=0.1995, over 24276.00 frames. ], tot_loss[loss=0.44, simple_loss=0.4937, pruned_loss=0.1931, over 4517741.03 frames. ], batch size: 257, lr: 3.02e-02, grad_scale: 16.0 2026-09-23 22:16:59,560 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=14640.0, ans=0.38760000000000006 2026-09-23 22:17:00,384 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=14640.0, ans=0.125 2026-09-23 22:17:05,215 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=14673.333333333334, ans=0.15326666666666666 2026-09-23 22:17:06,917 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.475e+02 3.784e+02 4.403e+02 5.598e+02 1.133e+03, threshold=8.806e+02, percent-clipped=4.0 2026-09-23 22:17:14,391 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=14740.0, ans=0.125 2026-09-23 22:17:19,681 INFO [train.py:1192] (0/2) Epoch 5, batch 600, loss[loss=0.4928, simple_loss=0.5415, pruned_loss=0.222, over 24332.00 frames. ], tot_loss[loss=0.4403, simple_loss=0.4941, pruned_loss=0.1932, over 4583640.68 frames. ], batch size: 234, lr: 3.01e-02, grad_scale: 16.0 2026-09-23 22:17:28,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=14806.666666666666, ans=0.004972222222222225 2026-09-23 22:17:32,921 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=14840.0, ans=0.125 2026-09-23 22:17:35,574 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=14873.333333333334, ans=0.42310000000000003 2026-09-23 22:17:45,413 INFO [train.py:1192] (0/2) Epoch 5, batch 650, loss[loss=0.4262, simple_loss=0.4889, pruned_loss=0.1817, over 24559.00 frames. ], tot_loss[loss=0.4362, simple_loss=0.4916, pruned_loss=0.1904, over 4649518.96 frames. ], batch size: 162, lr: 3.01e-02, grad_scale: 16.0 2026-09-23 22:17:47,685 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=14940.0, ans=0.125 2026-09-23 22:17:49,801 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=14940.0, ans=0.035 2026-09-23 22:17:58,443 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.307e+02 3.636e+02 4.065e+02 4.709e+02 8.553e+02, threshold=8.130e+02, percent-clipped=0.0 2026-09-23 22:18:04,029 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:18:05,322 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=15040.0, ans=0.125 2026-09-23 22:18:11,839 INFO [train.py:1192] (0/2) Epoch 5, batch 700, loss[loss=0.4271, simple_loss=0.4775, pruned_loss=0.1884, over 24588.00 frames. ], tot_loss[loss=0.4365, simple_loss=0.4922, pruned_loss=0.1904, over 4683162.03 frames. ], batch size: 154, lr: 3.00e-02, grad_scale: 16.0 2026-09-23 22:18:13,493 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=15106.666666666666, ans=0.007585507246376811 2026-09-23 22:18:17,954 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:18:26,105 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=17.41 vs. limit=18.880000000000003 2026-09-23 22:18:28,590 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=15206.666666666666, ans=0.025 2026-09-23 22:18:29,516 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=15206.666666666666, ans=0.14793333333333333 2026-09-23 22:18:37,906 INFO [train.py:1192] (0/2) Epoch 5, batch 750, loss[loss=0.4107, simple_loss=0.478, pruned_loss=0.1717, over 24550.00 frames. ], tot_loss[loss=0.4348, simple_loss=0.4905, pruned_loss=0.1895, over 4714045.03 frames. ], batch size: 170, lr: 3.00e-02, grad_scale: 16.0 2026-09-23 22:18:39,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=15273.333333333334, ans=0.125 2026-09-23 22:18:46,635 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=18.36 vs. limit=18.98 2026-09-23 22:18:48,814 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=15340.0, ans=0.125 2026-09-23 22:18:51,296 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.366e+02 3.748e+02 4.495e+02 5.532e+02 8.509e+02, threshold=8.989e+02, percent-clipped=3.0 2026-09-23 22:18:58,424 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=15406.666666666666, ans=0.14593333333333333 2026-09-23 22:19:01,701 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=15406.666666666666, ans=0.05 2026-09-23 22:19:04,053 INFO [train.py:1192] (0/2) Epoch 5, batch 800, loss[loss=0.3556, simple_loss=0.426, pruned_loss=0.1426, over 24556.00 frames. ], tot_loss[loss=0.4333, simple_loss=0.4895, pruned_loss=0.1886, over 4739398.39 frames. ], batch size: 137, lr: 2.99e-02, grad_scale: 32.0 2026-09-23 22:19:04,620 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=15440.0, ans=0.35960000000000003 2026-09-23 22:19:09,499 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.94 vs. limit=13.3025 2026-09-23 22:19:23,583 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=15540.0, ans=0.025 2026-09-23 22:19:26,573 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.95 vs. limit=13.34 2026-09-23 22:19:26,710 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.66 vs. limit=19.18 2026-09-23 22:19:29,796 INFO [train.py:1192] (0/2) Epoch 5, batch 850, loss[loss=0.4804, simple_loss=0.5293, pruned_loss=0.2157, over 24593.00 frames. ], tot_loss[loss=0.432, simple_loss=0.4886, pruned_loss=0.1877, over 4761752.23 frames. ], batch size: 198, lr: 2.99e-02, grad_scale: 32.0 2026-09-23 22:19:29,888 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=15606.666666666666, ans=0.0074768115942028986 2026-09-23 22:19:31,949 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.66 vs. limit=12.803333333333333 2026-09-23 22:19:37,937 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=15640.0, ans=0.125 2026-09-23 22:19:39,460 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=15640.0, ans=0.3526 2026-09-23 22:19:42,967 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.633e+02 3.828e+02 4.258e+02 5.330e+02 1.035e+03, threshold=8.516e+02, percent-clipped=1.0 2026-09-23 22:19:55,628 INFO [train.py:1192] (0/2) Epoch 5, batch 900, loss[loss=0.355, simple_loss=0.4319, pruned_loss=0.1391, over 24550.00 frames. ], tot_loss[loss=0.4305, simple_loss=0.4879, pruned_loss=0.1865, over 4775904.44 frames. ], batch size: 137, lr: 2.98e-02, grad_scale: 32.0 2026-09-23 22:20:02,938 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=15806.666666666666, ans=0.125 2026-09-23 22:20:21,870 INFO [train.py:1192] (0/2) Epoch 5, batch 950, loss[loss=0.5752, simple_loss=0.5437, pruned_loss=0.3034, over 11188.00 frames. ], tot_loss[loss=0.4324, simple_loss=0.4873, pruned_loss=0.1887, over 4716767.97 frames. ], batch size: 333, lr: 2.98e-02, grad_scale: 32.0 2026-09-23 22:20:22,523 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.89 vs. limit=5.391 2026-09-23 22:20:26,264 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-5.pt 2026-09-23 22:20:33,966 INFO [train.py:1192] (0/2) Epoch 6, batch 0, loss[loss=0.4108, simple_loss=0.4652, pruned_loss=0.1782, over 24558.00 frames. ], tot_loss[loss=0.4108, simple_loss=0.4652, pruned_loss=0.1782, over 24558.00 frames. ], batch size: 137, lr: 2.78e-02, grad_scale: 32.0 2026-09-23 22:20:33,966 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 22:20:46,090 INFO [train.py:1224] (0/2) Epoch 6, validation: loss=0.2624, simple_loss=0.364, pruned_loss=0.08034, over 2564189.00 frames. 2026-09-23 22:20:46,090 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 22:20:55,658 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.670e+02 3.768e+02 4.556e+02 5.587e+02 1.060e+03, threshold=9.113e+02, percent-clipped=3.0 2026-09-23 22:21:01,334 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=16066.666666666666, ans=13.525 2026-09-23 22:21:03,043 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=16066.666666666666, ans=0.125 2026-09-23 22:21:03,088 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=16066.666666666666, ans=13.033333333333333 2026-09-23 22:21:04,387 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.19 vs. limit=10.426666666666666 2026-09-23 22:21:12,021 INFO [train.py:1192] (0/2) Epoch 6, batch 50, loss[loss=0.3686, simple_loss=0.4297, pruned_loss=0.1538, over 24300.00 frames. ], tot_loss[loss=0.4429, simple_loss=0.497, pruned_loss=0.1943, over 1077482.15 frames. ], batch size: 125, lr: 2.78e-02, grad_scale: 32.0 2026-09-23 22:21:13,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=16133.333333333334, ans=0.0 2026-09-23 22:21:14,820 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=16133.333333333334, ans=0.125 2026-09-23 22:21:27,600 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=16233.333333333334, ans=0.07 2026-09-23 22:21:29,025 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=16233.333333333334, ans=0.0 2026-09-23 22:21:32,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=16266.666666666666, ans=0.0 2026-09-23 22:21:37,639 INFO [train.py:1192] (0/2) Epoch 6, batch 100, loss[loss=0.404, simple_loss=0.4642, pruned_loss=0.1719, over 24559.00 frames. ], tot_loss[loss=0.4398, simple_loss=0.4968, pruned_loss=0.1914, over 1905457.13 frames. ], batch size: 154, lr: 2.77e-02, grad_scale: 32.0 2026-09-23 22:21:46,923 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.660e+02 3.443e+02 4.030e+02 4.669e+02 8.575e+02, threshold=8.059e+02, percent-clipped=0.0 2026-09-23 22:22:03,770 INFO [train.py:1192] (0/2) Epoch 6, batch 150, loss[loss=0.3768, simple_loss=0.4345, pruned_loss=0.1596, over 24202.00 frames. ], tot_loss[loss=0.4289, simple_loss=0.4881, pruned_loss=0.1848, over 2558650.55 frames. ], batch size: 125, lr: 2.77e-02, grad_scale: 32.0 2026-09-23 22:22:03,876 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=16466.666666666668, ans=0.025 2026-09-23 22:22:11,514 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:22:17,407 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=16533.333333333332, ans=0.32133333333333347 2026-09-23 22:22:17,920 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=16533.333333333332, ans=0.125 2026-09-23 22:22:24,478 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=8.04 vs. limit=13.3 2026-09-23 22:22:29,018 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=16600.0, ans=0.125 2026-09-23 22:22:29,951 INFO [train.py:1192] (0/2) Epoch 6, batch 200, loss[loss=0.4832, simple_loss=0.5192, pruned_loss=0.2236, over 21223.00 frames. ], tot_loss[loss=0.4248, simple_loss=0.485, pruned_loss=0.1823, over 3055153.87 frames. ], batch size: 333, lr: 2.76e-02, grad_scale: 32.0 2026-09-23 22:22:39,186 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.471e+02 3.466e+02 4.197e+02 4.744e+02 7.854e+02, threshold=8.394e+02, percent-clipped=0.0 2026-09-23 22:22:44,255 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.49 vs. limit=5.505 2026-09-23 22:22:55,342 INFO [train.py:1192] (0/2) Epoch 6, batch 250, loss[loss=0.4673, simple_loss=0.531, pruned_loss=0.2018, over 24298.00 frames. ], tot_loss[loss=0.4217, simple_loss=0.4828, pruned_loss=0.1803, over 3443449.14 frames. ], batch size: 234, lr: 2.76e-02, grad_scale: 32.0 2026-09-23 22:22:56,376 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=16800.0, ans=0.025 2026-09-23 22:22:57,530 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.02 vs. limit=10.719999999999999 2026-09-23 22:22:58,802 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=16800.0, ans=0.0 2026-09-23 22:22:58,855 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=16800.0, ans=0.125 2026-09-23 22:23:04,571 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=16833.333333333332, ans=0.0 2026-09-23 22:23:20,785 INFO [train.py:1192] (0/2) Epoch 6, batch 300, loss[loss=0.429, simple_loss=0.5014, pruned_loss=0.1783, over 24520.00 frames. ], tot_loss[loss=0.4199, simple_loss=0.481, pruned_loss=0.1794, over 3748878.71 frames. ], batch size: 204, lr: 2.76e-02, grad_scale: 32.0 2026-09-23 22:23:23,860 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=16966.666666666668, ans=0.0 2026-09-23 22:23:24,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=16966.666666666668, ans=0.1303333333333333 2026-09-23 22:23:28,262 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.44 vs. limit=9.25 2026-09-23 22:23:30,250 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.608e+02 3.453e+02 3.959e+02 5.073e+02 7.733e+02, threshold=7.917e+02, percent-clipped=0.0 2026-09-23 22:23:46,443 INFO [train.py:1192] (0/2) Epoch 6, batch 350, loss[loss=0.3452, simple_loss=0.4124, pruned_loss=0.139, over 24588.00 frames. ], tot_loss[loss=0.4213, simple_loss=0.4823, pruned_loss=0.1802, over 3991751.21 frames. ], batch size: 137, lr: 2.75e-02, grad_scale: 32.0 2026-09-23 22:23:53,147 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=17166.666666666668, ans=0.05 2026-09-23 22:23:54,844 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=17166.666666666668, ans=13.9375 2026-09-23 22:23:56,090 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=17166.666666666668, ans=0.007137681159420289 2026-09-23 22:24:00,371 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=17200.0, ans=0.125 2026-09-23 22:24:12,997 INFO [train.py:1192] (0/2) Epoch 6, batch 400, loss[loss=0.4039, simple_loss=0.4675, pruned_loss=0.1702, over 24565.00 frames. ], tot_loss[loss=0.4197, simple_loss=0.4813, pruned_loss=0.1791, over 4178685.00 frames. ], batch size: 170, lr: 2.75e-02, grad_scale: 32.0 2026-09-23 22:24:22,046 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.579e+02 3.534e+02 4.312e+02 5.095e+02 8.398e+02, threshold=8.624e+02, percent-clipped=1.0 2026-09-23 22:24:22,243 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=14.0 2026-09-23 22:24:23,076 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=17366.666666666668, ans=0.125 2026-09-23 22:24:32,796 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=17433.333333333332, ans=0.125 2026-09-23 22:24:38,011 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=17466.666666666668, ans=0.125 2026-09-23 22:24:38,384 INFO [train.py:1192] (0/2) Epoch 6, batch 450, loss[loss=0.4504, simple_loss=0.5094, pruned_loss=0.1957, over 24610.00 frames. ], tot_loss[loss=0.4194, simple_loss=0.4814, pruned_loss=0.1787, over 4310876.72 frames. ], batch size: 175, lr: 2.74e-02, grad_scale: 32.0 2026-09-23 22:24:48,112 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=17533.333333333332, ans=0.125 2026-09-23 22:24:48,167 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.18 vs. limit=9.383333333333333 2026-09-23 22:24:50,847 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.95 vs. limit=14.075 2026-09-23 22:24:53,150 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=17.16 vs. limit=20.675 2026-09-23 22:25:00,530 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=18.80 vs. limit=20.7 2026-09-23 22:25:03,488 INFO [train.py:1192] (0/2) Epoch 6, batch 500, loss[loss=0.4763, simple_loss=0.5275, pruned_loss=0.2125, over 24510.00 frames. ], tot_loss[loss=0.4163, simple_loss=0.4785, pruned_loss=0.177, over 4428370.38 frames. ], batch size: 218, lr: 2.74e-02, grad_scale: 16.0 2026-09-23 22:25:04,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=17633.333333333332, ans=10.0 2026-09-23 22:25:13,501 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.61 vs. limit=11.066666666666666 2026-09-23 22:25:13,751 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.588e+02 3.510e+02 4.091e+02 4.746e+02 7.444e+02, threshold=8.182e+02, percent-clipped=0.0 2026-09-23 22:25:14,675 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=17700.0, ans=0.125 2026-09-23 22:25:22,353 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=17733.333333333332, ans=0.125 2026-09-23 22:25:23,250 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=17733.333333333332, ans=0.125 2026-09-23 22:25:29,814 INFO [train.py:1192] (0/2) Epoch 6, batch 550, loss[loss=0.4787, simple_loss=0.544, pruned_loss=0.2067, over 24247.00 frames. ], tot_loss[loss=0.4156, simple_loss=0.4786, pruned_loss=0.1763, over 4518160.11 frames. ], batch size: 257, lr: 2.73e-02, grad_scale: 16.0 2026-09-23 22:25:30,619 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.30 vs. limit=5.67 2026-09-23 22:25:54,086 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=17933.333333333332, ans=0.12066666666666667 2026-09-23 22:25:55,466 INFO [train.py:1192] (0/2) Epoch 6, batch 600, loss[loss=0.4459, simple_loss=0.526, pruned_loss=0.1829, over 24333.00 frames. ], tot_loss[loss=0.4158, simple_loss=0.479, pruned_loss=0.1763, over 4584645.51 frames. ], batch size: 234, lr: 2.73e-02, grad_scale: 16.0 2026-09-23 22:26:04,988 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=18000.0, ans=0.47000000000000003 2026-09-23 22:26:05,323 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.396e+02 3.556e+02 4.162e+02 5.274e+02 1.094e+03, threshold=8.324e+02, percent-clipped=2.0 2026-09-23 22:26:06,768 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=18033.333333333332, ans=0.125 2026-09-23 22:26:19,455 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=18100.0, ans=0.125 2026-09-23 22:26:20,444 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=18100.0, ans=0.125 2026-09-23 22:26:21,301 INFO [train.py:1192] (0/2) Epoch 6, batch 650, loss[loss=0.4118, simple_loss=0.4777, pruned_loss=0.173, over 24569.00 frames. ], tot_loss[loss=0.4154, simple_loss=0.4787, pruned_loss=0.1761, over 4650170.47 frames. ], batch size: 162, lr: 2.73e-02, grad_scale: 16.0 2026-09-23 22:26:25,086 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=18133.333333333332, ans=0.125 2026-09-23 22:26:32,930 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=18200.0, ans=0.263 2026-09-23 22:26:40,483 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.93 vs. limit=5.734999999999999 2026-09-23 22:26:47,622 INFO [train.py:1192] (0/2) Epoch 6, batch 700, loss[loss=0.42, simple_loss=0.4723, pruned_loss=0.1839, over 24586.00 frames. ], tot_loss[loss=0.4157, simple_loss=0.4791, pruned_loss=0.1762, over 4683049.72 frames. ], batch size: 154, lr: 2.72e-02, grad_scale: 16.0 2026-09-23 22:26:52,146 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=18300.0, ans=0.125 2026-09-23 22:26:56,451 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=18333.333333333332, ans=0.125 2026-09-23 22:26:57,313 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.618e+02 3.557e+02 4.369e+02 5.425e+02 8.837e+02, threshold=8.739e+02, percent-clipped=2.0 2026-09-23 22:27:02,369 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=18366.666666666668, ans=0.0 2026-09-23 22:27:07,309 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.00 vs. limit=9.6 2026-09-23 22:27:13,506 INFO [train.py:1192] (0/2) Epoch 6, batch 750, loss[loss=0.4101, simple_loss=0.4751, pruned_loss=0.1725, over 24555.00 frames. ], tot_loss[loss=0.414, simple_loss=0.4775, pruned_loss=0.1753, over 4711047.91 frames. ], batch size: 170, lr: 2.72e-02, grad_scale: 16.0 2026-09-23 22:27:33,817 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.62 vs. limit=14.4625 2026-09-23 22:27:35,751 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=18600.0, ans=0.11400000000000002 2026-09-23 22:27:39,810 INFO [train.py:1192] (0/2) Epoch 6, batch 800, loss[loss=0.3394, simple_loss=0.4141, pruned_loss=0.1324, over 24514.00 frames. ], tot_loss[loss=0.4122, simple_loss=0.4761, pruned_loss=0.1741, over 4735378.97 frames. ], batch size: 137, lr: 2.71e-02, grad_scale: 32.0 2026-09-23 22:27:50,262 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.559e+02 3.295e+02 3.854e+02 4.496e+02 1.051e+03, threshold=7.707e+02, percent-clipped=1.0 2026-09-23 22:27:54,214 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=18700.0, ans=0.125 2026-09-23 22:27:55,825 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=18733.333333333332, ans=0.2443333333333334 2026-09-23 22:27:59,098 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=18733.333333333332, ans=0.125 2026-09-23 22:28:00,022 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=18733.333333333332, ans=0.125 2026-09-23 22:28:06,957 INFO [train.py:1192] (0/2) Epoch 6, batch 850, loss[loss=0.4052, simple_loss=0.4829, pruned_loss=0.1637, over 24600.00 frames. ], tot_loss[loss=0.4113, simple_loss=0.4756, pruned_loss=0.1735, over 4758802.73 frames. ], batch size: 198, lr: 2.71e-02, grad_scale: 32.0 2026-09-23 22:28:23,002 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.03 vs. limit=5.835 2026-09-23 22:28:24,423 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=18900.0, ans=0.07 2026-09-23 22:28:33,046 INFO [train.py:1192] (0/2) Epoch 6, batch 900, loss[loss=0.3418, simple_loss=0.4211, pruned_loss=0.1312, over 24580.00 frames. ], tot_loss[loss=0.4113, simple_loss=0.4757, pruned_loss=0.1735, over 4772216.03 frames. ], batch size: 137, lr: 2.70e-02, grad_scale: 32.0 2026-09-23 22:28:34,434 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=4.21 vs. limit=14.6125 2026-09-23 22:28:36,854 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=18966.666666666668, ans=0.125 2026-09-23 22:28:42,804 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.12 vs. limit=14.625 2026-09-23 22:28:43,023 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.416e+02 3.503e+02 4.182e+02 4.940e+02 9.030e+02, threshold=8.364e+02, percent-clipped=1.0 2026-09-23 22:28:47,176 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.08 vs. limit=9.758333333333333 2026-09-23 22:28:55,775 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=19100.0, ans=0.125 2026-09-23 22:28:58,672 INFO [train.py:1192] (0/2) Epoch 6, batch 950, loss[loss=0.5645, simple_loss=0.5409, pruned_loss=0.294, over 11708.00 frames. ], tot_loss[loss=0.4142, simple_loss=0.4757, pruned_loss=0.1763, over 4712212.93 frames. ], batch size: 333, lr: 2.70e-02, grad_scale: 16.0 2026-09-23 22:29:00,670 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=19133.333333333332, ans=0.125 2026-09-23 22:29:03,092 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-6.pt 2026-09-23 22:29:10,806 INFO [train.py:1192] (0/2) Epoch 7, batch 0, loss[loss=0.3842, simple_loss=0.4518, pruned_loss=0.1583, over 24540.00 frames. ], tot_loss[loss=0.3842, simple_loss=0.4518, pruned_loss=0.1583, over 24540.00 frames. ], batch size: 137, lr: 2.53e-02, grad_scale: 32.0 2026-09-23 22:29:10,807 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 22:29:22,640 INFO [train.py:1224] (0/2) Epoch 7, validation: loss=0.2474, simple_loss=0.3531, pruned_loss=0.07088, over 2564189.00 frames. 2026-09-23 22:29:22,640 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 22:29:22,746 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=19160.0, ans=10.0 2026-09-23 22:29:22,748 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=19160.0, ans=0.07 2026-09-23 22:29:45,811 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=19293.333333333332, ans=0.0 2026-09-23 22:29:47,695 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=19326.666666666668, ans=0.006668115942028985 2026-09-23 22:29:48,120 INFO [train.py:1192] (0/2) Epoch 7, batch 50, loss[loss=0.3497, simple_loss=0.413, pruned_loss=0.1432, over 24226.00 frames. ], tot_loss[loss=0.4153, simple_loss=0.4808, pruned_loss=0.175, over 1074697.58 frames. ], batch size: 125, lr: 2.53e-02, grad_scale: 32.0 2026-09-23 22:29:53,664 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.228e+02 3.433e+02 4.092e+02 4.910e+02 9.076e+02, threshold=8.183e+02, percent-clipped=3.0 2026-09-23 22:30:13,929 INFO [train.py:1192] (0/2) Epoch 7, batch 100, loss[loss=0.3513, simple_loss=0.4329, pruned_loss=0.1349, over 24604.00 frames. ], tot_loss[loss=0.4176, simple_loss=0.4834, pruned_loss=0.1759, over 1905135.84 frames. ], batch size: 154, lr: 2.52e-02, grad_scale: 32.0 2026-09-23 22:30:33,938 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.49 vs. limit=14.796666666666665 2026-09-23 22:30:39,891 INFO [train.py:1192] (0/2) Epoch 7, batch 150, loss[loss=0.3349, simple_loss=0.3984, pruned_loss=0.1357, over 24259.00 frames. ], tot_loss[loss=0.4086, simple_loss=0.4759, pruned_loss=0.1707, over 2558966.81 frames. ], batch size: 125, lr: 2.52e-02, grad_scale: 32.0 2026-09-23 22:30:45,842 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.438e+02 3.557e+02 4.005e+02 4.673e+02 1.044e+03, threshold=8.010e+02, percent-clipped=1.0 2026-09-23 22:30:58,242 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=19760.0, ans=0.125 2026-09-23 22:31:00,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=19793.333333333332, ans=0.1020666666666667 2026-09-23 22:31:05,906 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=19826.666666666668, ans=0.0 2026-09-23 22:31:06,308 INFO [train.py:1192] (0/2) Epoch 7, batch 200, loss[loss=0.4668, simple_loss=0.5085, pruned_loss=0.2126, over 21138.00 frames. ], tot_loss[loss=0.4056, simple_loss=0.4732, pruned_loss=0.169, over 3054900.79 frames. ], batch size: 333, lr: 2.51e-02, grad_scale: 32.0 2026-09-23 22:31:14,984 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=19860.0, ans=0.07 2026-09-23 22:31:17,278 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=19893.333333333332, ans=0.125 2026-09-23 22:31:28,981 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=19960.0, ans=0.0065304347826086965 2026-09-23 22:31:29,085 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.29 vs. limit=5.994 2026-09-23 22:31:32,200 INFO [train.py:1192] (0/2) Epoch 7, batch 250, loss[loss=0.4302, simple_loss=0.5048, pruned_loss=0.1778, over 24302.00 frames. ], tot_loss[loss=0.4024, simple_loss=0.4708, pruned_loss=0.167, over 3442838.16 frames. ], batch size: 234, lr: 2.51e-02, grad_scale: 32.0 2026-09-23 22:31:32,312 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=19993.333333333332, ans=0.10006666666666669 2026-09-23 22:31:37,439 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.27 vs. limit=15.0 2026-09-23 22:31:38,386 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.126e+02 3.140e+02 3.704e+02 4.295e+02 7.836e+02, threshold=7.407e+02, percent-clipped=0.0 2026-09-23 22:31:38,492 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=20026.666666666668, ans=0.125 2026-09-23 22:31:40,051 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.61 vs. limit=6.0 2026-09-23 22:31:57,655 INFO [train.py:1192] (0/2) Epoch 7, batch 300, loss[loss=0.4207, simple_loss=0.4926, pruned_loss=0.1744, over 24521.00 frames. ], tot_loss[loss=0.401, simple_loss=0.4695, pruned_loss=0.1663, over 3748212.02 frames. ], batch size: 204, lr: 2.51e-02, grad_scale: 32.0 2026-09-23 22:32:01,620 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=20160.0, ans=0.05 2026-09-23 22:32:23,523 INFO [train.py:1192] (0/2) Epoch 7, batch 350, loss[loss=0.3467, simple_loss=0.4171, pruned_loss=0.1381, over 24565.00 frames. ], tot_loss[loss=0.4013, simple_loss=0.4702, pruned_loss=0.1662, over 3991823.63 frames. ], batch size: 137, lr: 2.50e-02, grad_scale: 32.0 2026-09-23 22:32:29,790 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.460e+02 3.574e+02 4.263e+02 5.284e+02 1.007e+03, threshold=8.526e+02, percent-clipped=4.0 2026-09-23 22:32:35,418 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=20393.333333333332, ans=0.0 2026-09-23 22:32:36,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=20393.333333333332, ans=0.125 2026-09-23 22:32:37,346 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:32:49,310 INFO [train.py:1192] (0/2) Epoch 7, batch 400, loss[loss=0.4331, simple_loss=0.4947, pruned_loss=0.1858, over 24540.00 frames. ], tot_loss[loss=0.399, simple_loss=0.4685, pruned_loss=0.1648, over 4175565.12 frames. ], batch size: 170, lr: 2.50e-02, grad_scale: 32.0 2026-09-23 22:32:52,354 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=20493.333333333332, ans=0.125 2026-09-23 22:33:07,705 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=20593.333333333332, ans=0.006392753623188406 2026-09-23 22:33:15,873 INFO [train.py:1192] (0/2) Epoch 7, batch 450, loss[loss=0.4307, simple_loss=0.4957, pruned_loss=0.1828, over 24623.00 frames. ], tot_loss[loss=0.3994, simple_loss=0.4686, pruned_loss=0.1651, over 4309122.08 frames. ], batch size: 175, lr: 2.49e-02, grad_scale: 16.0 2026-09-23 22:33:22,351 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.542e+02 3.393e+02 3.897e+02 4.560e+02 8.616e+02, threshold=7.793e+02, percent-clipped=1.0 2026-09-23 22:33:31,989 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=5.99 vs. limit=15.0 2026-09-23 22:33:41,212 INFO [train.py:1192] (0/2) Epoch 7, batch 500, loss[loss=0.4218, simple_loss=0.4958, pruned_loss=0.1739, over 24517.00 frames. ], tot_loss[loss=0.3973, simple_loss=0.4665, pruned_loss=0.164, over 4427358.49 frames. ], batch size: 218, lr: 2.49e-02, grad_scale: 16.0 2026-09-23 22:33:47,147 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=20860.0, ans=0.2 2026-09-23 22:33:48,924 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=20860.0, ans=0.04949747468305833 2026-09-23 22:33:49,743 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=20860.0, ans=0.125 2026-09-23 22:33:51,408 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=15.95 vs. limit=22.5 2026-09-23 22:33:53,100 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=20893.333333333332, ans=0.0 2026-09-23 22:34:03,118 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=20960.0, ans=0.125 2026-09-23 22:34:04,103 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=20960.0, ans=0.125 2026-09-23 22:34:06,701 INFO [train.py:1192] (0/2) Epoch 7, batch 550, loss[loss=0.4131, simple_loss=0.4905, pruned_loss=0.1679, over 24234.00 frames. ], tot_loss[loss=0.3956, simple_loss=0.4658, pruned_loss=0.1627, over 4517715.78 frames. ], batch size: 257, lr: 2.48e-02, grad_scale: 16.0 2026-09-23 22:34:08,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=20993.333333333332, ans=0.0 2026-09-23 22:34:12,701 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.497e+02 3.370e+02 3.713e+02 4.412e+02 6.548e+02, threshold=7.426e+02, percent-clipped=0.0 2026-09-23 22:34:19,271 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=21060.0, ans=0.006291304347826087 2026-09-23 22:34:32,411 INFO [train.py:1192] (0/2) Epoch 7, batch 600, loss[loss=0.4082, simple_loss=0.4904, pruned_loss=0.1631, over 24303.00 frames. ], tot_loss[loss=0.3961, simple_loss=0.4664, pruned_loss=0.1629, over 4584775.74 frames. ], batch size: 234, lr: 2.48e-02, grad_scale: 16.0 2026-09-23 22:34:37,312 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=21193.333333333332, ans=0.00626231884057971 2026-09-23 22:34:38,631 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=21193.333333333332, ans=0.0 2026-09-23 22:34:45,572 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=21226.666666666668, ans=0.04949747468305833 2026-09-23 22:34:51,103 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=21260.0, ans=0.1 2026-09-23 22:34:52,055 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=21293.333333333332, ans=0.1 2026-09-23 22:34:54,705 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:34:57,857 INFO [train.py:1192] (0/2) Epoch 7, batch 650, loss[loss=0.4093, simple_loss=0.4724, pruned_loss=0.1731, over 24562.00 frames. ], tot_loss[loss=0.3944, simple_loss=0.4653, pruned_loss=0.1618, over 4650372.71 frames. ], batch size: 162, lr: 2.48e-02, grad_scale: 16.0 2026-09-23 22:34:59,762 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=21326.666666666668, ans=0.125 2026-09-23 22:35:04,046 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.638e+02 3.513e+02 4.049e+02 4.785e+02 8.518e+02, threshold=8.098e+02, percent-clipped=6.0 2026-09-23 22:35:12,191 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=21393.333333333332, ans=0.006218840579710145 2026-09-23 22:35:23,370 INFO [train.py:1192] (0/2) Epoch 7, batch 700, loss[loss=0.3903, simple_loss=0.4588, pruned_loss=0.1608, over 24580.00 frames. ], tot_loss[loss=0.3946, simple_loss=0.4657, pruned_loss=0.1617, over 4682869.29 frames. ], batch size: 154, lr: 2.47e-02, grad_scale: 16.0 2026-09-23 22:35:49,343 INFO [train.py:1192] (0/2) Epoch 7, batch 750, loss[loss=0.3693, simple_loss=0.4564, pruned_loss=0.1411, over 24551.00 frames. ], tot_loss[loss=0.3933, simple_loss=0.4646, pruned_loss=0.161, over 4713537.99 frames. ], batch size: 170, lr: 2.47e-02, grad_scale: 16.0 2026-09-23 22:35:49,599 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.55 vs. limit=12.0 2026-09-23 22:35:50,434 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=21660.0, ans=0.125 2026-09-23 22:35:52,936 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=21660.0, ans=0.125 2026-09-23 22:35:56,130 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.721e+02 3.438e+02 3.905e+02 5.071e+02 8.170e+02, threshold=7.810e+02, percent-clipped=1.0 2026-09-23 22:35:58,515 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=21693.333333333332, ans=0.125 2026-09-23 22:36:06,299 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=21760.0, ans=0.006139130434782609 2026-09-23 22:36:09,906 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=21760.0, ans=0.125 2026-09-23 22:36:11,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=21793.333333333332, ans=0.125 2026-09-23 22:36:13,412 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=21793.333333333332, ans=0.125 2026-09-23 22:36:15,762 INFO [train.py:1192] (0/2) Epoch 7, batch 800, loss[loss=0.3137, simple_loss=0.3955, pruned_loss=0.1159, over 24562.00 frames. ], tot_loss[loss=0.393, simple_loss=0.4641, pruned_loss=0.1609, over 4737240.19 frames. ], batch size: 137, lr: 2.46e-02, grad_scale: 32.0 2026-09-23 22:36:21,960 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=21860.0, ans=0.015 2026-09-23 22:36:23,045 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=21860.0, ans=0.125 2026-09-23 22:36:30,221 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.93 vs. limit=6.0 2026-09-23 22:36:33,825 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=21926.666666666668, ans=0.0 2026-09-23 22:36:34,316 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=21926.666666666668, ans=0.006102898550724638 2026-09-23 22:36:40,575 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=21960.0, ans=0.125 2026-09-23 22:36:41,927 INFO [train.py:1192] (0/2) Epoch 7, batch 850, loss[loss=0.3843, simple_loss=0.4768, pruned_loss=0.1459, over 24568.00 frames. ], tot_loss[loss=0.3921, simple_loss=0.4634, pruned_loss=0.1604, over 4759614.18 frames. ], batch size: 204, lr: 2.46e-02, grad_scale: 32.0 2026-09-23 22:36:43,473 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.62 vs. limit=15.0 2026-09-23 22:36:49,250 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.337e+02 3.460e+02 3.975e+02 4.728e+02 8.722e+02, threshold=7.951e+02, percent-clipped=1.0 2026-09-23 22:36:51,497 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.21 vs. limit=12.0 2026-09-23 22:37:02,968 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=22126.666666666668, ans=0.125 2026-09-23 22:37:06,801 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=22126.666666666668, ans=0.025 2026-09-23 22:37:08,211 INFO [train.py:1192] (0/2) Epoch 7, batch 900, loss[loss=0.33, simple_loss=0.411, pruned_loss=0.1245, over 24541.00 frames. ], tot_loss[loss=0.3928, simple_loss=0.4641, pruned_loss=0.1608, over 4773274.31 frames. ], batch size: 137, lr: 2.46e-02, grad_scale: 32.0 2026-09-23 22:37:11,034 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=22160.0, ans=0.125 2026-09-23 22:37:25,883 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=22260.0, ans=0.1 2026-09-23 22:37:27,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=22260.0, ans=0.2 2026-09-23 22:37:31,366 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=22293.333333333332, ans=0.07 2026-09-23 22:37:33,934 INFO [train.py:1192] (0/2) Epoch 7, batch 950, loss[loss=0.5944, simple_loss=0.5653, pruned_loss=0.3118, over 11728.00 frames. ], tot_loss[loss=0.3949, simple_loss=0.4639, pruned_loss=0.163, over 4713855.46 frames. ], batch size: 334, lr: 2.45e-02, grad_scale: 32.0 2026-09-23 22:37:38,407 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-7.pt 2026-09-23 22:37:47,160 INFO [train.py:1192] (0/2) Epoch 8, batch 0, loss[loss=0.3542, simple_loss=0.4392, pruned_loss=0.1346, over 24575.00 frames. ], tot_loss[loss=0.3542, simple_loss=0.4392, pruned_loss=0.1346, over 24575.00 frames. ], batch size: 137, lr: 2.31e-02, grad_scale: 32.0 2026-09-23 22:37:47,160 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 22:37:50,764 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.6213, 3.1118, 3.5565, 3.1276], device='cuda:0') 2026-09-23 22:37:58,739 INFO [train.py:1224] (0/2) Epoch 8, validation: loss=0.24, simple_loss=0.3477, pruned_loss=0.06622, over 2564189.00 frames. 2026-09-23 22:37:58,739 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 22:38:01,379 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.723e+02 3.734e+02 4.392e+02 5.202e+02 8.156e+02, threshold=8.783e+02, percent-clipped=1.0 2026-09-23 22:38:08,914 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=22420.0, ans=0.1 2026-09-23 22:38:14,334 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.71 vs. limit=10.0 2026-09-23 22:38:24,725 INFO [train.py:1192] (0/2) Epoch 8, batch 50, loss[loss=0.3113, simple_loss=0.3912, pruned_loss=0.1157, over 24269.00 frames. ], tot_loss[loss=0.4074, simple_loss=0.4747, pruned_loss=0.1701, over 1077069.61 frames. ], batch size: 125, lr: 2.31e-02, grad_scale: 32.0 2026-09-23 22:38:24,850 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=22520.0, ans=0.0 2026-09-23 22:38:25,750 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=22520.0, ans=0.005973913043478261 2026-09-23 22:38:39,403 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=22586.666666666668, ans=0.125 2026-09-23 22:38:51,100 INFO [train.py:1192] (0/2) Epoch 8, batch 100, loss[loss=0.355, simple_loss=0.4335, pruned_loss=0.1382, over 24598.00 frames. ], tot_loss[loss=0.406, simple_loss=0.4754, pruned_loss=0.1682, over 1906436.40 frames. ], batch size: 154, lr: 2.30e-02, grad_scale: 32.0 2026-09-23 22:38:53,492 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.473e+02 3.371e+02 3.805e+02 4.450e+02 6.852e+02, threshold=7.610e+02, percent-clipped=0.0 2026-09-23 22:38:58,653 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=22720.0, ans=0.1 2026-09-23 22:39:12,149 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.50 vs. limit=22.5 2026-09-23 22:39:16,973 INFO [train.py:1192] (0/2) Epoch 8, batch 150, loss[loss=0.3563, simple_loss=0.4194, pruned_loss=0.1466, over 24308.00 frames. ], tot_loss[loss=0.3952, simple_loss=0.4673, pruned_loss=0.1616, over 2559515.74 frames. ], batch size: 125, lr: 2.30e-02, grad_scale: 32.0 2026-09-23 22:39:22,152 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.78 vs. limit=10.0 2026-09-23 22:39:31,275 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=22920.0, ans=0.0 2026-09-23 22:39:32,924 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=22953.333333333332, ans=0.125 2026-09-23 22:39:35,756 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.69 vs. limit=22.5 2026-09-23 22:39:38,225 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=22986.666666666668, ans=0.09899494936611666 2026-09-23 22:39:40,823 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.60 vs. limit=15.0 2026-09-23 22:39:43,721 INFO [train.py:1192] (0/2) Epoch 8, batch 200, loss[loss=0.4989, simple_loss=0.5288, pruned_loss=0.2345, over 21203.00 frames. ], tot_loss[loss=0.3921, simple_loss=0.4647, pruned_loss=0.1598, over 3056578.59 frames. ], batch size: 333, lr: 2.29e-02, grad_scale: 32.0 2026-09-23 22:39:44,300 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=23020.0, ans=0.1 2026-09-23 22:39:46,494 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.379e+02 3.268e+02 3.785e+02 4.404e+02 6.520e+02, threshold=7.571e+02, percent-clipped=0.0 2026-09-23 22:39:55,409 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=23086.666666666668, ans=0.125 2026-09-23 22:39:57,779 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=23086.666666666668, ans=0.07 2026-09-23 22:40:09,925 INFO [train.py:1192] (0/2) Epoch 8, batch 250, loss[loss=0.444, simple_loss=0.5154, pruned_loss=0.1863, over 24400.00 frames. ], tot_loss[loss=0.3903, simple_loss=0.4633, pruned_loss=0.1587, over 3445513.48 frames. ], batch size: 235, lr: 2.29e-02, grad_scale: 32.0 2026-09-23 22:40:19,344 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=23220.0, ans=6.0 2026-09-23 22:40:20,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=23253.333333333332, ans=0.125 2026-09-23 22:40:21,205 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.83 vs. limit=10.0 2026-09-23 22:40:31,142 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=23320.0, ans=0.125 2026-09-23 22:40:33,472 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=23320.0, ans=0.025 2026-09-23 22:40:35,626 INFO [train.py:1192] (0/2) Epoch 8, batch 300, loss[loss=0.4376, simple_loss=0.5033, pruned_loss=0.1859, over 24577.00 frames. ], tot_loss[loss=0.3877, simple_loss=0.4609, pruned_loss=0.1572, over 3749807.59 frames. ], batch size: 204, lr: 2.29e-02, grad_scale: 32.0 2026-09-23 22:40:38,382 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.584e+02 3.199e+02 3.783e+02 4.797e+02 7.354e+02, threshold=7.566e+02, percent-clipped=0.0 2026-09-23 22:40:47,176 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=23420.0, ans=0.0 2026-09-23 22:40:59,721 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=23486.666666666668, ans=0.125 2026-09-23 22:41:02,097 INFO [train.py:1192] (0/2) Epoch 8, batch 350, loss[loss=0.3314, simple_loss=0.4119, pruned_loss=0.1254, over 24573.00 frames. ], tot_loss[loss=0.3882, simple_loss=0.4618, pruned_loss=0.1573, over 3994364.74 frames. ], batch size: 137, lr: 2.28e-02, grad_scale: 32.0 2026-09-23 22:41:13,651 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=23586.666666666668, ans=0.125 2026-09-23 22:41:22,773 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=23653.333333333332, ans=0.2 2026-09-23 22:41:28,257 INFO [train.py:1192] (0/2) Epoch 8, batch 400, loss[loss=0.3921, simple_loss=0.4601, pruned_loss=0.1621, over 24549.00 frames. ], tot_loss[loss=0.386, simple_loss=0.46, pruned_loss=0.156, over 4178065.19 frames. ], batch size: 170, lr: 2.28e-02, grad_scale: 32.0 2026-09-23 22:41:30,684 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.797e+02 3.606e+02 4.328e+02 5.043e+02 8.307e+02, threshold=8.655e+02, percent-clipped=3.0 2026-09-23 22:41:32,310 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=23686.666666666668, ans=0.07 2026-09-23 22:41:53,883 INFO [train.py:1192] (0/2) Epoch 8, batch 450, loss[loss=0.4142, simple_loss=0.4871, pruned_loss=0.1706, over 24635.00 frames. ], tot_loss[loss=0.387, simple_loss=0.4606, pruned_loss=0.1567, over 4312531.62 frames. ], batch size: 175, lr: 2.28e-02, grad_scale: 32.0 2026-09-23 22:42:04,969 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.61 vs. limit=12.0 2026-09-23 22:42:05,345 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=23920.0, ans=0.2 2026-09-23 22:42:08,364 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.max_abs, batch_count=23920.0, ans=10.0 2026-09-23 22:42:13,231 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=23953.333333333332, ans=0.125 2026-09-23 22:42:19,488 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=23986.666666666668, ans=0.025 2026-09-23 22:42:20,491 INFO [train.py:1192] (0/2) Epoch 8, batch 500, loss[loss=0.4514, simple_loss=0.5178, pruned_loss=0.1925, over 24512.00 frames. ], tot_loss[loss=0.3845, simple_loss=0.4584, pruned_loss=0.1553, over 4430048.13 frames. ], batch size: 218, lr: 2.27e-02, grad_scale: 32.0 2026-09-23 22:42:22,786 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.389e+02 3.416e+02 3.901e+02 4.503e+02 6.166e+02, threshold=7.802e+02, percent-clipped=0.0 2026-09-23 22:42:29,780 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=24053.333333333332, ans=0.1 2026-09-23 22:42:31,512 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=15.0 2026-09-23 22:42:31,859 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=24086.666666666668, ans=0.05 2026-09-23 22:42:38,471 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.65 vs. limit=22.5 2026-09-23 22:42:45,789 INFO [train.py:1192] (0/2) Epoch 8, batch 550, loss[loss=0.4176, simple_loss=0.495, pruned_loss=0.1701, over 24290.00 frames. ], tot_loss[loss=0.384, simple_loss=0.4582, pruned_loss=0.1549, over 4519403.03 frames. ], batch size: 257, lr: 2.27e-02, grad_scale: 32.0 2026-09-23 22:42:47,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=24186.666666666668, ans=0.125 2026-09-23 22:42:52,553 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=24220.0, ans=0.0 2026-09-23 22:42:58,838 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.45 vs. limit=22.5 2026-09-23 22:43:12,072 INFO [train.py:1192] (0/2) Epoch 8, batch 600, loss[loss=0.3987, simple_loss=0.4877, pruned_loss=0.1549, over 24332.00 frames. ], tot_loss[loss=0.3836, simple_loss=0.4585, pruned_loss=0.1544, over 4584353.61 frames. ], batch size: 234, lr: 2.26e-02, grad_scale: 32.0 2026-09-23 22:43:14,633 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.462e+02 3.363e+02 3.921e+02 4.855e+02 7.268e+02, threshold=7.841e+02, percent-clipped=0.0 2026-09-23 22:43:24,064 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.35 vs. limit=15.0 2026-09-23 22:43:24,664 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=24420.0, ans=0.1 2026-09-23 22:43:27,651 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=24453.333333333332, ans=0.2 2026-09-23 22:43:33,499 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=24486.666666666668, ans=0.125 2026-09-23 22:43:38,131 INFO [train.py:1192] (0/2) Epoch 8, batch 650, loss[loss=0.3841, simple_loss=0.4603, pruned_loss=0.154, over 24548.00 frames. ], tot_loss[loss=0.3823, simple_loss=0.4575, pruned_loss=0.1536, over 4649950.17 frames. ], batch size: 162, lr: 2.26e-02, grad_scale: 32.0 2026-09-23 22:43:40,380 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.25 vs. limit=10.0 2026-09-23 22:43:45,999 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=24553.333333333332, ans=0.125 2026-09-23 22:44:03,081 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.16 vs. limit=10.0 2026-09-23 22:44:03,737 INFO [train.py:1192] (0/2) Epoch 8, batch 700, loss[loss=0.339, simple_loss=0.4206, pruned_loss=0.1287, over 24554.00 frames. ], tot_loss[loss=0.3823, simple_loss=0.4578, pruned_loss=0.1534, over 4681839.85 frames. ], batch size: 154, lr: 2.26e-02, grad_scale: 32.0 2026-09-23 22:44:06,125 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.679e+02 3.542e+02 4.120e+02 4.771e+02 6.786e+02, threshold=8.240e+02, percent-clipped=0.0 2026-09-23 22:44:15,183 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=24753.333333333332, ans=0.125 2026-09-23 22:44:15,648 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=24753.333333333332, ans=0.1 2026-09-23 22:44:21,210 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=24786.666666666668, ans=0.125 2026-09-23 22:44:22,221 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=24786.666666666668, ans=0.125 2026-09-23 22:44:29,780 INFO [train.py:1192] (0/2) Epoch 8, batch 750, loss[loss=0.3695, simple_loss=0.4492, pruned_loss=0.1449, over 24548.00 frames. ], tot_loss[loss=0.3807, simple_loss=0.4562, pruned_loss=0.1527, over 4710092.70 frames. ], batch size: 170, lr: 2.25e-02, grad_scale: 32.0 2026-09-23 22:44:29,892 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=24853.333333333332, ans=0.0 2026-09-23 22:44:31,965 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.24 vs. limit=10.0 2026-09-23 22:44:56,099 INFO [train.py:1192] (0/2) Epoch 8, batch 800, loss[loss=0.325, simple_loss=0.4052, pruned_loss=0.1224, over 24533.00 frames. ], tot_loss[loss=0.381, simple_loss=0.4562, pruned_loss=0.1529, over 4739451.62 frames. ], batch size: 137, lr: 2.25e-02, grad_scale: 32.0 2026-09-23 22:44:58,378 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.531e+02 3.448e+02 4.026e+02 4.905e+02 8.713e+02, threshold=8.051e+02, percent-clipped=1.0 2026-09-23 22:45:06,158 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=25086.666666666668, ans=0.1 2026-09-23 22:45:15,974 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=25153.333333333332, ans=0.1 2026-09-23 22:45:19,146 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=25153.333333333332, ans=0.125 2026-09-23 22:45:21,923 INFO [train.py:1192] (0/2) Epoch 8, batch 850, loss[loss=0.426, simple_loss=0.4984, pruned_loss=0.1768, over 24566.00 frames. ], tot_loss[loss=0.3794, simple_loss=0.4553, pruned_loss=0.1518, over 4762300.58 frames. ], batch size: 204, lr: 2.25e-02, grad_scale: 32.0 2026-09-23 22:45:27,972 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=25220.0, ans=0.035 2026-09-23 22:45:29,567 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:45:32,019 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=25253.333333333332, ans=0.2 2026-09-23 22:45:32,027 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=25253.333333333332, ans=0.0 2026-09-23 22:45:34,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=25253.333333333332, ans=0.125 2026-09-23 22:45:36,680 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=25253.333333333332, ans=10.0 2026-09-23 22:45:40,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=25286.666666666668, ans=0.125 2026-09-23 22:45:47,694 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=25353.333333333332, ans=0.005357971014492754 2026-09-23 22:45:48,073 INFO [train.py:1192] (0/2) Epoch 8, batch 900, loss[loss=0.3272, simple_loss=0.4126, pruned_loss=0.1208, over 24567.00 frames. ], tot_loss[loss=0.3804, simple_loss=0.4559, pruned_loss=0.1524, over 4774773.21 frames. ], batch size: 137, lr: 2.24e-02, grad_scale: 32.0 2026-09-23 22:45:48,658 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=25353.333333333332, ans=0.125 2026-09-23 22:45:50,754 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.644e+02 3.332e+02 3.860e+02 4.757e+02 7.777e+02, threshold=7.721e+02, percent-clipped=0.0 2026-09-23 22:45:53,020 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.15 vs. limit=15.0 2026-09-23 22:45:53,461 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=9.17 vs. limit=15.0 2026-09-23 22:45:57,693 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=25420.0, ans=0.005343478260869565 2026-09-23 22:46:04,642 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:46:06,611 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=25453.333333333332, ans=0.2 2026-09-23 22:46:13,398 INFO [train.py:1192] (0/2) Epoch 8, batch 950, loss[loss=0.5346, simple_loss=0.5297, pruned_loss=0.2697, over 11185.00 frames. ], tot_loss[loss=0.3815, simple_loss=0.455, pruned_loss=0.154, over 4710579.89 frames. ], batch size: 333, lr: 2.24e-02, grad_scale: 16.0 2026-09-23 22:46:17,928 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-8.pt 2026-09-23 22:46:25,543 INFO [train.py:1192] (0/2) Epoch 9, batch 0, loss[loss=0.3364, simple_loss=0.423, pruned_loss=0.1249, over 24570.00 frames. ], tot_loss[loss=0.3364, simple_loss=0.423, pruned_loss=0.1249, over 24570.00 frames. ], batch size: 137, lr: 2.12e-02, grad_scale: 32.0 2026-09-23 22:46:25,543 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 22:46:36,934 INFO [train.py:1224] (0/2) Epoch 9, validation: loss=0.2287, simple_loss=0.3401, pruned_loss=0.05867, over 2564189.00 frames. 2026-09-23 22:46:36,935 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 22:46:43,855 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.05 vs. limit=15.0 2026-09-23 22:46:46,144 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.76 vs. limit=15.0 2026-09-23 22:46:46,877 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=25613.333333333332, ans=0.0 2026-09-23 22:46:55,350 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=25646.666666666668, ans=0.125 2026-09-23 22:47:01,904 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.566e+02 3.336e+02 3.929e+02 5.395e+02 8.479e+02, threshold=7.858e+02, percent-clipped=1.0 2026-09-23 22:47:02,822 INFO [train.py:1192] (0/2) Epoch 9, batch 50, loss[loss=0.3356, simple_loss=0.41, pruned_loss=0.1306, over 24246.00 frames. ], tot_loss[loss=0.3931, simple_loss=0.4663, pruned_loss=0.16, over 1076041.23 frames. ], batch size: 125, lr: 2.11e-02, grad_scale: 32.0 2026-09-23 22:47:03,395 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=25713.333333333332, ans=0.0 2026-09-23 22:47:04,410 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=25713.333333333332, ans=0.0 2026-09-23 22:47:08,736 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=25746.666666666668, ans=0.125 2026-09-23 22:47:16,264 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=25780.0, ans=0.125 2026-09-23 22:47:17,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=25780.0, ans=0.1 2026-09-23 22:47:27,167 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=25846.666666666668, ans=0.0 2026-09-23 22:47:28,493 INFO [train.py:1192] (0/2) Epoch 9, batch 100, loss[loss=0.3572, simple_loss=0.4418, pruned_loss=0.1363, over 24601.00 frames. ], tot_loss[loss=0.3943, simple_loss=0.4686, pruned_loss=0.16, over 1903652.59 frames. ], batch size: 154, lr: 2.11e-02, grad_scale: 32.0 2026-09-23 22:47:31,972 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.62 vs. limit=12.0 2026-09-23 22:47:41,150 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=14.32 vs. limit=15.0 2026-09-23 22:47:50,176 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=26013.333333333332, ans=0.025 2026-09-23 22:47:53,081 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.515e+02 3.359e+02 3.717e+02 4.379e+02 7.313e+02, threshold=7.434e+02, percent-clipped=0.0 2026-09-23 22:47:54,228 INFO [train.py:1192] (0/2) Epoch 9, batch 150, loss[loss=0.2924, simple_loss=0.3739, pruned_loss=0.1055, over 24270.00 frames. ], tot_loss[loss=0.3849, simple_loss=0.4608, pruned_loss=0.1545, over 2557635.07 frames. ], batch size: 125, lr: 2.11e-02, grad_scale: 32.0 2026-09-23 22:48:02,950 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=26080.0, ans=0.125 2026-09-23 22:48:09,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=26113.333333333332, ans=0.1 2026-09-23 22:48:09,090 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=26113.333333333332, ans=0.005192753623188406 2026-09-23 22:48:12,015 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=26146.666666666668, ans=0.125 2026-09-23 22:48:14,656 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=26146.666666666668, ans=0.07 2026-09-23 22:48:17,605 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:48:21,177 INFO [train.py:1192] (0/2) Epoch 9, batch 200, loss[loss=0.4963, simple_loss=0.5257, pruned_loss=0.2335, over 21064.00 frames. ], tot_loss[loss=0.3814, simple_loss=0.4577, pruned_loss=0.1526, over 3054962.21 frames. ], batch size: 333, lr: 2.10e-02, grad_scale: 32.0 2026-09-23 22:48:46,579 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.433e+02 3.475e+02 4.252e+02 4.980e+02 8.416e+02, threshold=8.504e+02, percent-clipped=4.0 2026-09-23 22:48:47,546 INFO [train.py:1192] (0/2) Epoch 9, batch 250, loss[loss=0.4122, simple_loss=0.4951, pruned_loss=0.1646, over 24316.00 frames. ], tot_loss[loss=0.3809, simple_loss=0.4573, pruned_loss=0.1523, over 3444824.04 frames. ], batch size: 234, lr: 2.10e-02, grad_scale: 32.0 2026-09-23 22:48:55,560 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=26413.333333333332, ans=0.125 2026-09-23 22:49:00,311 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.92 vs. limit=10.0 2026-09-23 22:49:01,101 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=26446.666666666668, ans=0.0051202898550724634 2026-09-23 22:49:11,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=26513.333333333332, ans=0.125 2026-09-23 22:49:13,458 INFO [train.py:1192] (0/2) Epoch 9, batch 300, loss[loss=0.371, simple_loss=0.4662, pruned_loss=0.1379, over 24576.00 frames. ], tot_loss[loss=0.3785, simple_loss=0.4552, pruned_loss=0.1509, over 3750119.52 frames. ], batch size: 204, lr: 2.10e-02, grad_scale: 32.0 2026-09-23 22:49:30,166 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.66 vs. limit=15.0 2026-09-23 22:49:32,787 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-8000.pt 2026-09-23 22:49:34,053 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=26646.666666666668, ans=0.2 2026-09-23 22:49:39,518 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.408e+02 3.428e+02 3.972e+02 4.632e+02 6.980e+02, threshold=7.944e+02, percent-clipped=0.0 2026-09-23 22:49:39,640 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=26680.0, ans=0.07 2026-09-23 22:49:40,484 INFO [train.py:1192] (0/2) Epoch 9, batch 350, loss[loss=0.3221, simple_loss=0.3983, pruned_loss=0.123, over 24545.00 frames. ], tot_loss[loss=0.3781, simple_loss=0.4555, pruned_loss=0.1503, over 3993093.34 frames. ], batch size: 137, lr: 2.09e-02, grad_scale: 32.0 2026-09-23 22:49:42,049 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=26713.333333333332, ans=0.035 2026-09-23 22:49:49,921 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.22 vs. limit=15.0 2026-09-23 22:49:51,678 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=26780.0, ans=0.1 2026-09-23 22:49:56,284 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.82 vs. limit=15.0 2026-09-23 22:50:04,409 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=26846.666666666668, ans=0.2 2026-09-23 22:50:06,328 INFO [train.py:1192] (0/2) Epoch 9, batch 400, loss[loss=0.3923, simple_loss=0.4641, pruned_loss=0.1603, over 24562.00 frames. ], tot_loss[loss=0.3767, simple_loss=0.4541, pruned_loss=0.1497, over 4179036.74 frames. ], batch size: 170, lr: 2.09e-02, grad_scale: 32.0 2026-09-23 22:50:12,049 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.91 vs. limit=6.0 2026-09-23 22:50:28,018 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=27013.333333333332, ans=0.125 2026-09-23 22:50:31,474 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.095e+02 3.561e+02 4.305e+02 5.055e+02 8.003e+02, threshold=8.611e+02, percent-clipped=0.0 2026-09-23 22:50:32,501 INFO [train.py:1192] (0/2) Epoch 9, batch 450, loss[loss=0.3848, simple_loss=0.4622, pruned_loss=0.1537, over 24624.00 frames. ], tot_loss[loss=0.3768, simple_loss=0.4542, pruned_loss=0.1497, over 4312731.24 frames. ], batch size: 175, lr: 2.09e-02, grad_scale: 32.0 2026-09-23 22:50:45,158 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.12 vs. limit=10.0 2026-09-23 22:50:58,331 INFO [train.py:1192] (0/2) Epoch 9, batch 500, loss[loss=0.4133, simple_loss=0.4994, pruned_loss=0.1636, over 24512.00 frames. ], tot_loss[loss=0.3736, simple_loss=0.4515, pruned_loss=0.1478, over 4429915.29 frames. ], batch size: 218, lr: 2.08e-02, grad_scale: 32.0 2026-09-23 22:51:08,939 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=27280.0, ans=0.2 2026-09-23 22:51:14,769 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.84 vs. limit=15.0 2026-09-23 22:51:23,574 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.367e+02 3.132e+02 3.739e+02 4.190e+02 8.139e+02, threshold=7.479e+02, percent-clipped=0.0 2026-09-23 22:51:24,489 INFO [train.py:1192] (0/2) Epoch 9, batch 550, loss[loss=0.3914, simple_loss=0.4728, pruned_loss=0.155, over 24253.00 frames. ], tot_loss[loss=0.374, simple_loss=0.452, pruned_loss=0.1479, over 4520390.91 frames. ], batch size: 257, lr: 2.08e-02, grad_scale: 32.0 2026-09-23 22:51:49,886 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=27513.333333333332, ans=0.125 2026-09-23 22:51:50,841 INFO [train.py:1192] (0/2) Epoch 9, batch 600, loss[loss=0.4301, simple_loss=0.5106, pruned_loss=0.1748, over 24336.00 frames. ], tot_loss[loss=0.3741, simple_loss=0.4524, pruned_loss=0.148, over 4586646.89 frames. ], batch size: 234, lr: 2.08e-02, grad_scale: 32.0 2026-09-23 22:52:02,949 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.21 vs. limit=12.0 2026-09-23 22:52:03,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=27613.333333333332, ans=0.04949747468305833 2026-09-23 22:52:15,744 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.679e+02 3.351e+02 3.948e+02 4.727e+02 8.005e+02, threshold=7.895e+02, percent-clipped=1.0 2026-09-23 22:52:16,662 INFO [train.py:1192] (0/2) Epoch 9, batch 650, loss[loss=0.3665, simple_loss=0.4484, pruned_loss=0.1423, over 24560.00 frames. ], tot_loss[loss=0.3726, simple_loss=0.4513, pruned_loss=0.1469, over 4651897.54 frames. ], batch size: 162, lr: 2.07e-02, grad_scale: 32.0 2026-09-23 22:52:18,999 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=27713.333333333332, ans=0.125 2026-09-23 22:52:19,597 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=27713.333333333332, ans=0.0 2026-09-23 22:52:20,885 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.52 vs. limit=10.0 2026-09-23 22:52:28,694 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=27780.0, ans=0.125 2026-09-23 22:52:31,350 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=27780.0, ans=0.1 2026-09-23 22:52:34,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=27813.333333333332, ans=0.0 2026-09-23 22:52:43,731 INFO [train.py:1192] (0/2) Epoch 9, batch 700, loss[loss=0.365, simple_loss=0.4449, pruned_loss=0.1425, over 24592.00 frames. ], tot_loss[loss=0.3739, simple_loss=0.4527, pruned_loss=0.1476, over 4684472.17 frames. ], batch size: 154, lr: 2.07e-02, grad_scale: 32.0 2026-09-23 22:52:51,503 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.04 vs. limit=15.0 2026-09-23 22:52:58,802 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=27946.666666666668, ans=0.004794202898550724 2026-09-23 22:53:09,685 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.443e+02 3.501e+02 4.089e+02 5.092e+02 1.078e+03, threshold=8.179e+02, percent-clipped=2.0 2026-09-23 22:53:10,787 INFO [train.py:1192] (0/2) Epoch 9, batch 750, loss[loss=0.4065, simple_loss=0.4754, pruned_loss=0.1688, over 24547.00 frames. ], tot_loss[loss=0.3741, simple_loss=0.4521, pruned_loss=0.1481, over 4710844.60 frames. ], batch size: 170, lr: 2.07e-02, grad_scale: 32.0 2026-09-23 22:53:19,397 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=28080.0, ans=0.0 2026-09-23 22:53:22,089 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.89 vs. limit=22.5 2026-09-23 22:53:26,198 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=28146.666666666668, ans=0.025 2026-09-23 22:53:35,024 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=28180.0, ans=0.0 2026-09-23 22:53:36,312 INFO [train.py:1192] (0/2) Epoch 9, batch 800, loss[loss=0.2983, simple_loss=0.3901, pruned_loss=0.1032, over 24535.00 frames. ], tot_loss[loss=0.3727, simple_loss=0.4511, pruned_loss=0.1471, over 4740966.04 frames. ], batch size: 137, lr: 2.06e-02, grad_scale: 32.0 2026-09-23 22:53:53,524 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=28313.333333333332, ans=0.125 2026-09-23 22:54:01,805 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.663e+02 3.282e+02 3.877e+02 4.476e+02 6.858e+02, threshold=7.755e+02, percent-clipped=0.0 2026-09-23 22:54:03,007 INFO [train.py:1192] (0/2) Epoch 9, batch 850, loss[loss=0.3905, simple_loss=0.4788, pruned_loss=0.1511, over 24555.00 frames. ], tot_loss[loss=0.3716, simple_loss=0.4502, pruned_loss=0.1465, over 4762138.48 frames. ], batch size: 204, lr: 2.06e-02, grad_scale: 32.0 2026-09-23 22:54:12,434 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=28413.333333333332, ans=0.0 2026-09-23 22:54:29,023 INFO [train.py:1192] (0/2) Epoch 9, batch 900, loss[loss=0.2866, simple_loss=0.3826, pruned_loss=0.09531, over 24574.00 frames. ], tot_loss[loss=0.3711, simple_loss=0.4501, pruned_loss=0.1461, over 4774533.04 frames. ], batch size: 137, lr: 2.06e-02, grad_scale: 32.0 2026-09-23 22:54:33,766 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.04 vs. limit=15.0 2026-09-23 22:54:38,590 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=28613.333333333332, ans=0.125 2026-09-23 22:54:43,694 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=28646.666666666668, ans=0.125 2026-09-23 22:54:53,812 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.503e+02 3.409e+02 3.820e+02 4.682e+02 1.064e+03, threshold=7.640e+02, percent-clipped=2.0 2026-09-23 22:54:54,329 INFO [train.py:1192] (0/2) Epoch 9, batch 950, loss[loss=0.5356, simple_loss=0.5192, pruned_loss=0.276, over 11688.00 frames. ], tot_loss[loss=0.3735, simple_loss=0.45, pruned_loss=0.1485, over 4709932.69 frames. ], batch size: 333, lr: 2.05e-02, grad_scale: 16.0 2026-09-23 22:54:54,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=28713.333333333332, ans=0.004627536231884059 2026-09-23 22:54:58,904 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-9.pt 2026-09-23 22:55:05,757 INFO [train.py:1192] (0/2) Epoch 10, batch 0, loss[loss=0.3009, simple_loss=0.3957, pruned_loss=0.1031, over 24556.00 frames. ], tot_loss[loss=0.3009, simple_loss=0.3957, pruned_loss=0.1031, over 24556.00 frames. ], batch size: 137, lr: 1.95e-02, grad_scale: 32.0 2026-09-23 22:55:05,758 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 22:55:07,366 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([1.9049, 1.8796, 2.1643, 1.9704, 1.6425, 2.0123, 1.0540, 1.8432], device='cuda:0') 2026-09-23 22:55:17,321 INFO [train.py:1224] (0/2) Epoch 10, validation: loss=0.2259, simple_loss=0.3365, pruned_loss=0.05759, over 2564189.00 frames. 2026-09-23 22:55:17,322 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 22:55:24,533 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=28773.333333333332, ans=0.125 2026-09-23 22:55:24,957 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=28773.333333333332, ans=0.035 2026-09-23 22:55:42,657 INFO [train.py:1192] (0/2) Epoch 10, batch 50, loss[loss=0.2945, simple_loss=0.3793, pruned_loss=0.1048, over 24282.00 frames. ], tot_loss[loss=0.3836, simple_loss=0.4593, pruned_loss=0.154, over 1075852.39 frames. ], batch size: 125, lr: 1.95e-02, grad_scale: 32.0 2026-09-23 22:55:44,535 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=28906.666666666668, ans=0.125 2026-09-23 22:55:55,505 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=28973.333333333332, ans=0.2 2026-09-23 22:56:02,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=29006.666666666668, ans=0.2 2026-09-23 22:56:03,691 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.525e+02 3.405e+02 3.972e+02 4.546e+02 6.978e+02, threshold=7.944e+02, percent-clipped=0.0 2026-09-23 22:56:08,279 INFO [train.py:1192] (0/2) Epoch 10, batch 100, loss[loss=0.3504, simple_loss=0.4311, pruned_loss=0.1349, over 24619.00 frames. ], tot_loss[loss=0.3799, simple_loss=0.4588, pruned_loss=0.1505, over 1905813.52 frames. ], batch size: 154, lr: 1.94e-02, grad_scale: 32.0 2026-09-23 22:56:11,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=29073.333333333332, ans=0.0 2026-09-23 22:56:13,050 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.86 vs. limit=15.0 2026-09-23 22:56:20,536 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=29140.0, ans=0.1 2026-09-23 22:56:29,117 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=29206.666666666668, ans=10.0 2026-09-23 22:56:30,943 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=29206.666666666668, ans=0.125 2026-09-23 22:56:33,585 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.76 vs. limit=10.0 2026-09-23 22:56:34,219 INFO [train.py:1192] (0/2) Epoch 10, batch 150, loss[loss=0.2921, simple_loss=0.3759, pruned_loss=0.1041, over 24329.00 frames. ], tot_loss[loss=0.3715, simple_loss=0.4517, pruned_loss=0.1456, over 2559000.30 frames. ], batch size: 125, lr: 1.94e-02, grad_scale: 32.0 2026-09-23 22:56:35,441 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=29240.0, ans=0.1 2026-09-23 22:56:44,857 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=29306.666666666668, ans=0.0 2026-09-23 22:56:46,776 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=29306.666666666668, ans=0.5 2026-09-23 22:56:55,828 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.371e+02 3.224e+02 3.571e+02 4.187e+02 7.105e+02, threshold=7.141e+02, percent-clipped=0.0 2026-09-23 22:57:00,263 INFO [train.py:1192] (0/2) Epoch 10, batch 200, loss[loss=0.4323, simple_loss=0.4851, pruned_loss=0.1898, over 20903.00 frames. ], tot_loss[loss=0.3686, simple_loss=0.4495, pruned_loss=0.1438, over 3055284.74 frames. ], batch size: 334, lr: 1.94e-02, grad_scale: 32.0 2026-09-23 22:57:02,738 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=29406.666666666668, ans=0.125 2026-09-23 22:57:03,206 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=29406.666666666668, ans=0.0 2026-09-23 22:57:11,834 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=29473.333333333332, ans=0.1 2026-09-23 22:57:13,249 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=29473.333333333332, ans=0.07 2026-09-23 22:57:25,784 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=5.66 vs. limit=15.0 2026-09-23 22:57:26,439 INFO [train.py:1192] (0/2) Epoch 10, batch 250, loss[loss=0.4376, simple_loss=0.5083, pruned_loss=0.1834, over 24319.00 frames. ], tot_loss[loss=0.37, simple_loss=0.4501, pruned_loss=0.145, over 3444297.44 frames. ], batch size: 234, lr: 1.93e-02, grad_scale: 32.0 2026-09-23 22:57:29,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=29573.333333333332, ans=0.0 2026-09-23 22:57:30,360 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=29573.333333333332, ans=0.5 2026-09-23 22:57:35,388 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=20.61 vs. limit=22.5 2026-09-23 22:57:48,447 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.468e+02 3.253e+02 3.914e+02 4.692e+02 8.196e+02, threshold=7.827e+02, percent-clipped=1.0 2026-09-23 22:57:50,092 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.91 vs. limit=10.0 2026-09-23 22:57:50,919 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=29706.666666666668, ans=0.125 2026-09-23 22:57:52,403 INFO [train.py:1192] (0/2) Epoch 10, batch 300, loss[loss=0.3751, simple_loss=0.4591, pruned_loss=0.1455, over 24555.00 frames. ], tot_loss[loss=0.3688, simple_loss=0.4489, pruned_loss=0.1444, over 3753949.33 frames. ], batch size: 204, lr: 1.93e-02, grad_scale: 16.0 2026-09-23 22:58:05,789 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=29806.666666666668, ans=0.125 2026-09-23 22:58:13,109 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=29873.333333333332, ans=0.2 2026-09-23 22:58:18,452 INFO [train.py:1192] (0/2) Epoch 10, batch 350, loss[loss=0.3048, simple_loss=0.3866, pruned_loss=0.1115, over 24579.00 frames. ], tot_loss[loss=0.3682, simple_loss=0.4489, pruned_loss=0.1437, over 3995783.76 frames. ], batch size: 137, lr: 1.93e-02, grad_scale: 16.0 2026-09-23 22:58:32,658 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=29973.333333333332, ans=0.125 2026-09-23 22:58:34,260 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.87 vs. limit=22.5 2026-09-23 22:58:40,429 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=30040.0, ans=0.025 2026-09-23 22:58:40,740 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.483e+02 3.273e+02 3.815e+02 4.612e+02 7.991e+02, threshold=7.629e+02, percent-clipped=1.0 2026-09-23 22:58:43,306 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=30040.0, ans=0.0 2026-09-23 22:58:45,210 INFO [train.py:1192] (0/2) Epoch 10, batch 400, loss[loss=0.3734, simple_loss=0.4515, pruned_loss=0.1476, over 24565.00 frames. ], tot_loss[loss=0.3663, simple_loss=0.4473, pruned_loss=0.1427, over 4178777.57 frames. ], batch size: 170, lr: 1.93e-02, grad_scale: 32.0 2026-09-23 22:58:56,808 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 22:58:59,830 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=30140.0, ans=0.0 2026-09-23 22:59:10,908 INFO [train.py:1192] (0/2) Epoch 10, batch 450, loss[loss=0.3483, simple_loss=0.4428, pruned_loss=0.1269, over 24624.00 frames. ], tot_loss[loss=0.3651, simple_loss=0.4466, pruned_loss=0.1417, over 4311873.98 frames. ], batch size: 175, lr: 1.92e-02, grad_scale: 32.0 2026-09-23 22:59:12,986 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.99 vs. limit=10.0 2026-09-23 22:59:20,262 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=30273.333333333332, ans=0.125 2026-09-23 22:59:32,870 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.566e+02 3.169e+02 3.862e+02 4.320e+02 6.201e+02, threshold=7.724e+02, percent-clipped=0.0 2026-09-23 22:59:36,997 INFO [train.py:1192] (0/2) Epoch 10, batch 500, loss[loss=0.3795, simple_loss=0.4714, pruned_loss=0.1438, over 24494.00 frames. ], tot_loss[loss=0.3633, simple_loss=0.4448, pruned_loss=0.1409, over 4429609.60 frames. ], batch size: 218, lr: 1.92e-02, grad_scale: 32.0 2026-09-23 22:59:38,450 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=30406.666666666668, ans=0.2 2026-09-23 22:59:50,906 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.63 vs. limit=15.0 2026-09-23 22:59:55,201 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=30506.666666666668, ans=0.125 2026-09-23 22:59:59,857 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=30540.0, ans=0.07 2026-09-23 23:00:02,652 INFO [train.py:1192] (0/2) Epoch 10, batch 550, loss[loss=0.4212, simple_loss=0.5018, pruned_loss=0.1702, over 24252.00 frames. ], tot_loss[loss=0.3638, simple_loss=0.4454, pruned_loss=0.1411, over 4519098.40 frames. ], batch size: 257, lr: 1.92e-02, grad_scale: 32.0 2026-09-23 23:00:09,737 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.19 vs. limit=15.0 2026-09-23 23:00:11,049 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=30606.666666666668, ans=0.125 2026-09-23 23:00:13,159 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=30640.0, ans=0.004208695652173914 2026-09-23 23:00:24,965 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.548e+02 3.379e+02 3.947e+02 4.533e+02 6.325e+02, threshold=7.895e+02, percent-clipped=0.0 2026-09-23 23:00:25,192 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.19 vs. limit=6.0 2026-09-23 23:00:25,794 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.80 vs. limit=12.0 2026-09-23 23:00:28,903 INFO [train.py:1192] (0/2) Epoch 10, batch 600, loss[loss=0.4384, simple_loss=0.5139, pruned_loss=0.1814, over 24314.00 frames. ], tot_loss[loss=0.3663, simple_loss=0.4475, pruned_loss=0.1425, over 4586056.83 frames. ], batch size: 234, lr: 1.91e-02, grad_scale: 32.0 2026-09-23 23:00:34,351 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=30773.333333333332, ans=0.05 2026-09-23 23:00:51,860 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=30873.333333333332, ans=0.125 2026-09-23 23:00:54,806 INFO [train.py:1192] (0/2) Epoch 10, batch 650, loss[loss=0.3817, simple_loss=0.4572, pruned_loss=0.1531, over 24563.00 frames. ], tot_loss[loss=0.3637, simple_loss=0.4455, pruned_loss=0.141, over 4651203.86 frames. ], batch size: 162, lr: 1.91e-02, grad_scale: 32.0 2026-09-23 23:00:56,579 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.18 vs. limit=12.0 2026-09-23 23:01:01,076 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=30940.0, ans=0.125 2026-09-23 23:01:02,195 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.27 vs. limit=5.0 2026-09-23 23:01:08,080 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=30973.333333333332, ans=0.125 2026-09-23 23:01:16,592 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.511e+02 3.233e+02 3.609e+02 4.094e+02 7.383e+02, threshold=7.217e+02, percent-clipped=0.0 2026-09-23 23:01:20,541 INFO [train.py:1192] (0/2) Epoch 10, batch 700, loss[loss=0.3576, simple_loss=0.434, pruned_loss=0.1406, over 24583.00 frames. ], tot_loss[loss=0.3642, simple_loss=0.4463, pruned_loss=0.1411, over 4683039.70 frames. ], batch size: 154, lr: 1.91e-02, grad_scale: 32.0 2026-09-23 23:01:21,997 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=31073.333333333332, ans=0.125 2026-09-23 23:01:22,982 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=31073.333333333332, ans=0.125 2026-09-23 23:01:30,498 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=31140.0, ans=0.025 2026-09-23 23:01:31,961 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=31140.0, ans=0.125 2026-09-23 23:01:32,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=31140.0, ans=0.125 2026-09-23 23:01:39,087 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=31173.333333333332, ans=0.0 2026-09-23 23:01:40,695 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=31206.666666666668, ans=0.2 2026-09-23 23:01:44,512 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=31206.666666666668, ans=0.125 2026-09-23 23:01:46,766 INFO [train.py:1192] (0/2) Epoch 10, batch 750, loss[loss=0.393, simple_loss=0.4655, pruned_loss=0.1602, over 24568.00 frames. ], tot_loss[loss=0.3627, simple_loss=0.4448, pruned_loss=0.1402, over 4714808.46 frames. ], batch size: 170, lr: 1.90e-02, grad_scale: 32.0 2026-09-23 23:02:08,439 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.729e+02 3.541e+02 4.303e+02 5.012e+02 8.664e+02, threshold=8.605e+02, percent-clipped=3.0 2026-09-23 23:02:12,653 INFO [train.py:1192] (0/2) Epoch 10, batch 800, loss[loss=0.279, simple_loss=0.3729, pruned_loss=0.09257, over 24525.00 frames. ], tot_loss[loss=0.3625, simple_loss=0.4447, pruned_loss=0.1402, over 4738495.72 frames. ], batch size: 137, lr: 1.90e-02, grad_scale: 32.0 2026-09-23 23:02:28,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=31506.666666666668, ans=0.0 2026-09-23 23:02:35,187 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:02:38,833 INFO [train.py:1192] (0/2) Epoch 10, batch 850, loss[loss=0.3843, simple_loss=0.4797, pruned_loss=0.1444, over 24552.00 frames. ], tot_loss[loss=0.362, simple_loss=0.4445, pruned_loss=0.1398, over 4761833.50 frames. ], batch size: 204, lr: 1.90e-02, grad_scale: 16.0 2026-09-23 23:02:57,033 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=31673.333333333332, ans=0.2 2026-09-23 23:02:58,794 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=31706.666666666668, ans=0.2 2026-09-23 23:03:00,659 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.419e+02 3.449e+02 3.836e+02 4.528e+02 6.517e+02, threshold=7.672e+02, percent-clipped=0.0 2026-09-23 23:03:02,807 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.70 vs. limit=6.0 2026-09-23 23:03:03,197 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=31706.666666666668, ans=0.125 2026-09-23 23:03:04,646 INFO [train.py:1192] (0/2) Epoch 10, batch 900, loss[loss=0.3314, simple_loss=0.4063, pruned_loss=0.1283, over 24519.00 frames. ], tot_loss[loss=0.3628, simple_loss=0.445, pruned_loss=0.1403, over 4775500.60 frames. ], batch size: 137, lr: 1.89e-02, grad_scale: 16.0 2026-09-23 23:03:04,750 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=31740.0, ans=0.1 2026-09-23 23:03:18,477 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:03:18,989 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=31806.666666666668, ans=0.0 2026-09-23 23:03:29,074 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=31906.666666666668, ans=0.125 2026-09-23 23:03:29,379 INFO [train.py:1192] (0/2) Epoch 10, batch 950, loss[loss=0.4944, simple_loss=0.4936, pruned_loss=0.2476, over 11078.00 frames. ], tot_loss[loss=0.3639, simple_loss=0.4441, pruned_loss=0.1419, over 4716095.85 frames. ], batch size: 333, lr: 1.89e-02, grad_scale: 16.0 2026-09-23 23:03:33,945 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-10.pt 2026-09-23 23:03:41,400 INFO [train.py:1192] (0/2) Epoch 11, batch 0, loss[loss=0.3385, simple_loss=0.4261, pruned_loss=0.1255, over 24553.00 frames. ], tot_loss[loss=0.3385, simple_loss=0.4261, pruned_loss=0.1255, over 24553.00 frames. ], batch size: 137, lr: 1.81e-02, grad_scale: 32.0 2026-09-23 23:03:41,400 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 23:03:53,081 INFO [train.py:1224] (0/2) Epoch 11, validation: loss=0.2194, simple_loss=0.3319, pruned_loss=0.05341, over 2564189.00 frames. 2026-09-23 23:03:53,081 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 23:04:05,837 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=32000.0, ans=0.125 2026-09-23 23:04:07,161 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=32000.0, ans=0.125 2026-09-23 23:04:11,358 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.597e+02 3.569e+02 4.127e+02 4.841e+02 1.254e+03, threshold=8.254e+02, percent-clipped=4.0 2026-09-23 23:04:17,597 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.25 vs. limit=6.0 2026-09-23 23:04:18,729 INFO [train.py:1192] (0/2) Epoch 11, batch 50, loss[loss=0.3124, simple_loss=0.393, pruned_loss=0.1159, over 24280.00 frames. ], tot_loss[loss=0.3686, simple_loss=0.4499, pruned_loss=0.1437, over 1075069.71 frames. ], batch size: 125, lr: 1.80e-02, grad_scale: 32.0 2026-09-23 23:04:32,680 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=32166.666666666668, ans=0.2 2026-09-23 23:04:32,844 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.17 vs. limit=10.0 2026-09-23 23:04:34,985 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=32200.0, ans=0.0 2026-09-23 23:04:35,843 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:04:44,170 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.48 vs. limit=6.0 2026-09-23 23:04:44,362 INFO [train.py:1192] (0/2) Epoch 11, batch 100, loss[loss=0.3484, simple_loss=0.4273, pruned_loss=0.1347, over 24614.00 frames. ], tot_loss[loss=0.3708, simple_loss=0.4529, pruned_loss=0.1444, over 1904788.94 frames. ], batch size: 154, lr: 1.80e-02, grad_scale: 32.0 2026-09-23 23:04:51,870 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=32300.0, ans=0.003847826086956522 2026-09-23 23:05:02,347 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.429e+02 3.284e+02 3.638e+02 4.173e+02 7.698e+02, threshold=7.277e+02, percent-clipped=0.0 2026-09-23 23:05:09,768 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=32433.333333333332, ans=0.125 2026-09-23 23:05:10,202 INFO [train.py:1192] (0/2) Epoch 11, batch 150, loss[loss=0.3329, simple_loss=0.397, pruned_loss=0.1344, over 24284.00 frames. ], tot_loss[loss=0.3629, simple_loss=0.4462, pruned_loss=0.1398, over 2558189.29 frames. ], batch size: 125, lr: 1.80e-02, grad_scale: 32.0 2026-09-23 23:05:10,917 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.62 vs. limit=15.0 2026-09-23 23:05:13,066 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=32433.333333333332, ans=0.1 2026-09-23 23:05:17,835 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=32466.666666666668, ans=0.2 2026-09-23 23:05:21,010 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=32500.0, ans=0.2 2026-09-23 23:05:27,424 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=32533.333333333332, ans=0.125 2026-09-23 23:05:32,203 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=32566.666666666668, ans=0.125 2026-09-23 23:05:33,982 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=32566.666666666668, ans=0.125 2026-09-23 23:05:35,915 INFO [train.py:1192] (0/2) Epoch 11, batch 200, loss[loss=0.4691, simple_loss=0.5149, pruned_loss=0.2117, over 21047.00 frames. ], tot_loss[loss=0.3612, simple_loss=0.4448, pruned_loss=0.1388, over 3054946.27 frames. ], batch size: 333, lr: 1.79e-02, grad_scale: 32.0 2026-09-23 23:05:40,309 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=32600.0, ans=0.0 2026-09-23 23:05:52,872 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=32700.0, ans=0.125 2026-09-23 23:05:54,620 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.499e+02 3.124e+02 3.551e+02 4.295e+02 6.642e+02, threshold=7.103e+02, percent-clipped=0.0 2026-09-23 23:05:59,529 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.61 vs. limit=15.0 2026-09-23 23:05:59,954 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:06:02,482 INFO [train.py:1192] (0/2) Epoch 11, batch 250, loss[loss=0.4073, simple_loss=0.49, pruned_loss=0.1623, over 24290.00 frames. ], tot_loss[loss=0.3597, simple_loss=0.4434, pruned_loss=0.138, over 3444626.93 frames. ], batch size: 234, lr: 1.79e-02, grad_scale: 32.0 2026-09-23 23:06:04,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=32766.666666666668, ans=0.0 2026-09-23 23:06:21,667 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.39 vs. limit=8.0 2026-09-23 23:06:26,641 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=32900.0, ans=0.0 2026-09-23 23:06:28,100 INFO [train.py:1192] (0/2) Epoch 11, batch 300, loss[loss=0.3856, simple_loss=0.4764, pruned_loss=0.1474, over 24555.00 frames. ], tot_loss[loss=0.3588, simple_loss=0.4424, pruned_loss=0.1376, over 3749116.59 frames. ], batch size: 204, lr: 1.79e-02, grad_scale: 32.0 2026-09-23 23:06:28,773 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.35 vs. limit=15.0 2026-09-23 23:06:40,382 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.49 vs. limit=6.0 2026-09-23 23:06:41,951 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.58 vs. limit=22.5 2026-09-23 23:06:46,882 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.558e+02 3.403e+02 3.902e+02 4.638e+02 6.729e+02, threshold=7.803e+02, percent-clipped=0.0 2026-09-23 23:06:50,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=33066.666666666664, ans=0.1 2026-09-23 23:06:51,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=33066.666666666664, ans=0.125 2026-09-23 23:06:52,329 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=33066.666666666664, ans=0.125 2026-09-23 23:06:53,523 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.81 vs. limit=6.0 2026-09-23 23:06:54,292 INFO [train.py:1192] (0/2) Epoch 11, batch 350, loss[loss=0.3132, simple_loss=0.3957, pruned_loss=0.1154, over 24564.00 frames. ], tot_loss[loss=0.3604, simple_loss=0.444, pruned_loss=0.1383, over 3993007.47 frames. ], batch size: 137, lr: 1.78e-02, grad_scale: 32.0 2026-09-23 23:06:54,827 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=33100.0, ans=0.125 2026-09-23 23:06:58,879 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=33100.0, ans=0.125 2026-09-23 23:06:59,411 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=33133.333333333336, ans=0.125 2026-09-23 23:07:04,211 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.47 vs. limit=15.0 2026-09-23 23:07:12,330 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=33200.0, ans=0.125 2026-09-23 23:07:20,416 INFO [train.py:1192] (0/2) Epoch 11, batch 400, loss[loss=0.3447, simple_loss=0.4359, pruned_loss=0.1268, over 24571.00 frames. ], tot_loss[loss=0.3573, simple_loss=0.4417, pruned_loss=0.1364, over 4176866.02 frames. ], batch size: 170, lr: 1.78e-02, grad_scale: 32.0 2026-09-23 23:07:24,776 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=33266.666666666664, ans=0.0 2026-09-23 23:07:36,586 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.38 vs. limit=22.5 2026-09-23 23:07:39,253 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.379e+02 3.288e+02 3.676e+02 4.158e+02 6.374e+02, threshold=7.352e+02, percent-clipped=0.0 2026-09-23 23:07:40,801 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=33366.666666666664, ans=0.003615942028985508 2026-09-23 23:07:41,959 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.61 vs. limit=15.0 2026-09-23 23:07:47,027 INFO [train.py:1192] (0/2) Epoch 11, batch 450, loss[loss=0.3555, simple_loss=0.4535, pruned_loss=0.1287, over 24627.00 frames. ], tot_loss[loss=0.358, simple_loss=0.4423, pruned_loss=0.1369, over 4312242.64 frames. ], batch size: 175, lr: 1.78e-02, grad_scale: 32.0 2026-09-23 23:08:08,123 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.54 vs. limit=15.0 2026-09-23 23:08:13,244 INFO [train.py:1192] (0/2) Epoch 11, batch 500, loss[loss=0.3875, simple_loss=0.479, pruned_loss=0.148, over 24521.00 frames. ], tot_loss[loss=0.358, simple_loss=0.4415, pruned_loss=0.1372, over 4428740.19 frames. ], batch size: 218, lr: 1.78e-02, grad_scale: 32.0 2026-09-23 23:08:15,861 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=33600.0, ans=0.125 2026-09-23 23:08:16,814 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=33600.0, ans=0.125 2026-09-23 23:08:18,577 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=33633.333333333336, ans=0.0 2026-09-23 23:08:26,575 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=33666.666666666664, ans=0.1 2026-09-23 23:08:31,113 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=33700.0, ans=0.025 2026-09-23 23:08:31,385 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.556e+02 3.366e+02 3.820e+02 4.509e+02 7.829e+02, threshold=7.640e+02, percent-clipped=1.0 2026-09-23 23:08:39,055 INFO [train.py:1192] (0/2) Epoch 11, batch 550, loss[loss=0.4128, simple_loss=0.4935, pruned_loss=0.166, over 24250.00 frames. ], tot_loss[loss=0.3578, simple_loss=0.4416, pruned_loss=0.137, over 4519385.94 frames. ], batch size: 257, lr: 1.77e-02, grad_scale: 32.0 2026-09-23 23:08:39,570 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=33766.666666666664, ans=0.2 2026-09-23 23:08:44,933 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.11 vs. limit=10.0 2026-09-23 23:08:47,299 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=33800.0, ans=0.125 2026-09-23 23:08:50,043 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=33833.333333333336, ans=0.125 2026-09-23 23:08:52,625 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.07 vs. limit=15.0 2026-09-23 23:08:54,080 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.28 vs. limit=22.5 2026-09-23 23:08:58,584 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=33866.666666666664, ans=0.125 2026-09-23 23:09:05,010 INFO [train.py:1192] (0/2) Epoch 11, batch 600, loss[loss=0.4083, simple_loss=0.4954, pruned_loss=0.1606, over 24310.00 frames. ], tot_loss[loss=0.3577, simple_loss=0.4419, pruned_loss=0.1367, over 4586171.88 frames. ], batch size: 234, lr: 1.77e-02, grad_scale: 32.0 2026-09-23 23:09:06,667 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=2.83 vs. limit=15.0 2026-09-23 23:09:16,691 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=34000.0, ans=0.125 2026-09-23 23:09:23,195 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.536e+02 3.381e+02 3.797e+02 4.395e+02 7.584e+02, threshold=7.594e+02, percent-clipped=0.0 2026-09-23 23:09:25,719 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.81 vs. limit=15.0 2026-09-23 23:09:31,253 INFO [train.py:1192] (0/2) Epoch 11, batch 650, loss[loss=0.3332, simple_loss=0.4251, pruned_loss=0.1206, over 24560.00 frames. ], tot_loss[loss=0.3569, simple_loss=0.4412, pruned_loss=0.1363, over 4651120.26 frames. ], batch size: 162, lr: 1.77e-02, grad_scale: 32.0 2026-09-23 23:09:49,303 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=34200.0, ans=0.125 2026-09-23 23:09:57,006 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=34233.333333333336, ans=0.025 2026-09-23 23:09:57,860 INFO [train.py:1192] (0/2) Epoch 11, batch 700, loss[loss=0.3085, simple_loss=0.3996, pruned_loss=0.1087, over 24576.00 frames. ], tot_loss[loss=0.3574, simple_loss=0.4419, pruned_loss=0.1365, over 4682986.98 frames. ], batch size: 154, lr: 1.76e-02, grad_scale: 32.0 2026-09-23 23:10:05,969 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=34300.0, ans=0.125 2026-09-23 23:10:16,093 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.478e+02 3.428e+02 3.960e+02 4.700e+02 7.348e+02, threshold=7.919e+02, percent-clipped=0.0 2026-09-23 23:10:19,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=34400.0, ans=0.125 2026-09-23 23:10:23,798 INFO [train.py:1192] (0/2) Epoch 11, batch 750, loss[loss=0.3262, simple_loss=0.4264, pruned_loss=0.113, over 24548.00 frames. ], tot_loss[loss=0.3559, simple_loss=0.4404, pruned_loss=0.1357, over 4710195.86 frames. ], batch size: 170, lr: 1.76e-02, grad_scale: 32.0 2026-09-23 23:10:36,873 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.88 vs. limit=22.5 2026-09-23 23:10:42,178 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.72 vs. limit=12.0 2026-09-23 23:10:49,202 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=34566.666666666664, ans=0.125 2026-09-23 23:10:49,351 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.95 vs. limit=10.0 2026-09-23 23:10:50,130 INFO [train.py:1192] (0/2) Epoch 11, batch 800, loss[loss=0.2981, simple_loss=0.3882, pruned_loss=0.104, over 24541.00 frames. ], tot_loss[loss=0.3558, simple_loss=0.4402, pruned_loss=0.1358, over 4735726.75 frames. ], batch size: 137, lr: 1.76e-02, grad_scale: 32.0 2026-09-23 23:10:54,995 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=34633.333333333336, ans=0.2 2026-09-23 23:10:56,733 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=5.64 vs. limit=15.0 2026-09-23 23:10:57,578 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.58 vs. limit=10.0 2026-09-23 23:11:01,057 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=34666.666666666664, ans=10.0 2026-09-23 23:11:08,386 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.355e+02 3.357e+02 3.761e+02 4.428e+02 6.776e+02, threshold=7.523e+02, percent-clipped=0.0 2026-09-23 23:11:16,143 INFO [train.py:1192] (0/2) Epoch 11, batch 850, loss[loss=0.4149, simple_loss=0.5035, pruned_loss=0.1632, over 24531.00 frames. ], tot_loss[loss=0.3551, simple_loss=0.4398, pruned_loss=0.1352, over 4759373.79 frames. ], batch size: 204, lr: 1.76e-02, grad_scale: 32.0 2026-09-23 23:11:18,463 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.75 vs. limit=12.0 2026-09-23 23:11:28,950 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=34833.333333333336, ans=0.125 2026-09-23 23:11:35,055 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.26 vs. limit=15.0 2026-09-23 23:11:41,928 INFO [train.py:1192] (0/2) Epoch 11, batch 900, loss[loss=0.2829, simple_loss=0.3778, pruned_loss=0.09401, over 24564.00 frames. ], tot_loss[loss=0.3546, simple_loss=0.4395, pruned_loss=0.1348, over 4772286.86 frames. ], batch size: 137, lr: 1.75e-02, grad_scale: 32.0 2026-09-23 23:11:48,215 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=34966.666666666664, ans=0.1 2026-09-23 23:11:50,229 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=34966.666666666664, ans=0.0 2026-09-23 23:11:54,585 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=35000.0, ans=0.0 2026-09-23 23:11:59,318 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.406e+02 3.306e+02 3.818e+02 4.396e+02 7.651e+02, threshold=7.635e+02, percent-clipped=1.0 2026-09-23 23:11:59,929 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=35033.333333333336, ans=0.2 2026-09-23 23:12:05,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=35066.666666666664, ans=0.125 2026-09-23 23:12:06,815 INFO [train.py:1192] (0/2) Epoch 11, batch 950, loss[loss=0.4948, simple_loss=0.4965, pruned_loss=0.2466, over 11409.00 frames. ], tot_loss[loss=0.356, simple_loss=0.4389, pruned_loss=0.1366, over 4718248.53 frames. ], batch size: 333, lr: 1.75e-02, grad_scale: 16.0 2026-09-23 23:12:11,443 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-11.pt 2026-09-23 23:12:18,731 INFO [train.py:1192] (0/2) Epoch 12, batch 0, loss[loss=0.3482, simple_loss=0.4298, pruned_loss=0.1333, over 24591.00 frames. ], tot_loss[loss=0.3482, simple_loss=0.4298, pruned_loss=0.1333, over 24591.00 frames. ], batch size: 137, lr: 1.68e-02, grad_scale: 32.0 2026-09-23 23:12:18,731 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 23:12:30,449 INFO [train.py:1224] (0/2) Epoch 12, validation: loss=0.216, simple_loss=0.3291, pruned_loss=0.05141, over 2564189.00 frames. 2026-09-23 23:12:30,449 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 23:12:38,331 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=35160.0, ans=0.2 2026-09-23 23:12:55,023 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=35260.0, ans=0.125 2026-09-23 23:12:56,336 INFO [train.py:1192] (0/2) Epoch 12, batch 50, loss[loss=0.2758, simple_loss=0.3653, pruned_loss=0.09317, over 24268.00 frames. ], tot_loss[loss=0.3701, simple_loss=0.4513, pruned_loss=0.1445, over 1076641.84 frames. ], batch size: 125, lr: 1.67e-02, grad_scale: 32.0 2026-09-23 23:12:57,468 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.95 vs. limit=22.5 2026-09-23 23:13:01,372 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.58 vs. limit=15.0 2026-09-23 23:13:04,057 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=35326.666666666664, ans=0.125 2026-09-23 23:13:06,595 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.63 vs. limit=15.0 2026-09-23 23:13:07,493 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.54 vs. limit=22.5 2026-09-23 23:13:08,318 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=35360.0, ans=0.1 2026-09-23 23:13:10,574 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.513e+02 3.441e+02 3.981e+02 5.014e+02 8.366e+02, threshold=7.962e+02, percent-clipped=2.0 2026-09-23 23:13:21,588 INFO [train.py:1192] (0/2) Epoch 12, batch 100, loss[loss=0.3549, simple_loss=0.4386, pruned_loss=0.1356, over 24601.00 frames. ], tot_loss[loss=0.3674, simple_loss=0.4511, pruned_loss=0.1418, over 1905531.77 frames. ], batch size: 154, lr: 1.67e-02, grad_scale: 32.0 2026-09-23 23:13:26,227 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=35460.0, ans=0.125 2026-09-23 23:13:29,388 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:13:33,654 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=35526.666666666664, ans=0.125 2026-09-23 23:13:34,166 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=35526.666666666664, ans=0.1 2026-09-23 23:13:36,057 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.68 vs. limit=15.0 2026-09-23 23:13:47,087 INFO [train.py:1192] (0/2) Epoch 12, batch 150, loss[loss=0.3191, simple_loss=0.3998, pruned_loss=0.1192, over 24207.00 frames. ], tot_loss[loss=0.3584, simple_loss=0.4439, pruned_loss=0.1364, over 2559112.57 frames. ], batch size: 125, lr: 1.67e-02, grad_scale: 32.0 2026-09-23 23:13:48,011 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=35626.666666666664, ans=0.003124637681159421 2026-09-23 23:13:48,861 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=35626.666666666664, ans=0.125 2026-09-23 23:14:01,576 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.67 vs. limit=22.5 2026-09-23 23:14:01,840 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.479e+02 3.158e+02 3.739e+02 4.407e+02 7.030e+02, threshold=7.477e+02, percent-clipped=0.0 2026-09-23 23:14:01,940 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=35693.333333333336, ans=0.0031101449275362323 2026-09-23 23:14:03,039 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=35726.666666666664, ans=0.125 2026-09-23 23:14:12,499 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=35793.333333333336, ans=0.0 2026-09-23 23:14:13,303 INFO [train.py:1192] (0/2) Epoch 12, batch 200, loss[loss=0.4012, simple_loss=0.4649, pruned_loss=0.1687, over 21157.00 frames. ], tot_loss[loss=0.3544, simple_loss=0.4409, pruned_loss=0.134, over 3055914.30 frames. ], batch size: 333, lr: 1.67e-02, grad_scale: 32.0 2026-09-23 23:14:36,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=35926.666666666664, ans=0.0 2026-09-23 23:14:39,726 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=35960.0, ans=0.0 2026-09-23 23:14:40,067 INFO [train.py:1192] (0/2) Epoch 12, batch 250, loss[loss=0.3615, simple_loss=0.4593, pruned_loss=0.1318, over 24303.00 frames. ], tot_loss[loss=0.3539, simple_loss=0.4401, pruned_loss=0.1339, over 3445268.59 frames. ], batch size: 234, lr: 1.66e-02, grad_scale: 32.0 2026-09-23 23:14:42,305 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=35960.0, ans=0.1 2026-09-23 23:14:44,146 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=35960.0, ans=0.0030521739130434777 2026-09-23 23:14:52,662 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=36026.666666666664, ans=0.1 2026-09-23 23:14:54,015 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.274e+02 3.369e+02 3.869e+02 4.652e+02 8.346e+02, threshold=7.738e+02, percent-clipped=2.0 2026-09-23 23:14:55,399 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=36060.0, ans=0.0 2026-09-23 23:15:05,550 INFO [train.py:1192] (0/2) Epoch 12, batch 300, loss[loss=0.38, simple_loss=0.4688, pruned_loss=0.1456, over 24556.00 frames. ], tot_loss[loss=0.352, simple_loss=0.4381, pruned_loss=0.1329, over 3750591.03 frames. ], batch size: 204, lr: 1.66e-02, grad_scale: 32.0 2026-09-23 23:15:05,640 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=36126.666666666664, ans=0.0 2026-09-23 23:15:16,676 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=10.58 vs. limit=15.0 2026-09-23 23:15:17,113 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=20.78 vs. limit=22.5 2026-09-23 23:15:17,397 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=36193.333333333336, ans=0.125 2026-09-23 23:15:31,350 INFO [train.py:1192] (0/2) Epoch 12, batch 350, loss[loss=0.2939, simple_loss=0.386, pruned_loss=0.1009, over 24608.00 frames. ], tot_loss[loss=0.3528, simple_loss=0.4392, pruned_loss=0.1332, over 3993621.12 frames. ], batch size: 137, lr: 1.66e-02, grad_scale: 32.0 2026-09-23 23:15:43,225 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=36360.0, ans=0.125 2026-09-23 23:15:45,545 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.419e+02 3.376e+02 3.785e+02 4.280e+02 7.656e+02, threshold=7.571e+02, percent-clipped=0.0 2026-09-23 23:15:49,695 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=36393.333333333336, ans=0.0 2026-09-23 23:15:50,184 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=36393.333333333336, ans=0.0029579710144927536 2026-09-23 23:15:50,213 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=36393.333333333336, ans=0.125 2026-09-23 23:15:56,588 INFO [train.py:1192] (0/2) Epoch 12, batch 400, loss[loss=0.3595, simple_loss=0.444, pruned_loss=0.1376, over 24560.00 frames. ], tot_loss[loss=0.3501, simple_loss=0.4371, pruned_loss=0.1316, over 4180168.34 frames. ], batch size: 170, lr: 1.66e-02, grad_scale: 32.0 2026-09-23 23:15:57,710 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=36460.0, ans=0.125 2026-09-23 23:16:13,904 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=13.90 vs. limit=15.0 2026-09-23 23:16:15,070 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=36560.0, ans=0.0 2026-09-23 23:16:20,526 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=36593.333333333336, ans=0.125 2026-09-23 23:16:22,354 INFO [train.py:1192] (0/2) Epoch 12, batch 450, loss[loss=0.3786, simple_loss=0.4583, pruned_loss=0.1494, over 24615.00 frames. ], tot_loss[loss=0.3513, simple_loss=0.4378, pruned_loss=0.1324, over 4314089.13 frames. ], batch size: 175, lr: 1.65e-02, grad_scale: 32.0 2026-09-23 23:16:26,689 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=36626.666666666664, ans=15.0 2026-09-23 23:16:33,944 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=36693.333333333336, ans=0.125 2026-09-23 23:16:37,001 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.276e+02 3.307e+02 3.859e+02 4.571e+02 7.199e+02, threshold=7.718e+02, percent-clipped=0.0 2026-09-23 23:16:37,242 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.82 vs. limit=15.0 2026-09-23 23:16:45,032 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=36760.0, ans=0.125 2026-09-23 23:16:48,548 INFO [train.py:1192] (0/2) Epoch 12, batch 500, loss[loss=0.3852, simple_loss=0.4783, pruned_loss=0.1461, over 24524.00 frames. ], tot_loss[loss=0.35, simple_loss=0.4363, pruned_loss=0.1318, over 4431012.87 frames. ], batch size: 218, lr: 1.65e-02, grad_scale: 32.0 2026-09-23 23:16:48,797 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.77 vs. limit=15.0 2026-09-23 23:16:50,223 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=36793.333333333336, ans=0.2 2026-09-23 23:16:52,555 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=36793.333333333336, ans=0.0 2026-09-23 23:16:57,727 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=36826.666666666664, ans=0.125 2026-09-23 23:17:05,556 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=36893.333333333336, ans=0.125 2026-09-23 23:17:09,797 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=36926.666666666664, ans=0.0 2026-09-23 23:17:14,961 INFO [train.py:1192] (0/2) Epoch 12, batch 550, loss[loss=0.3724, simple_loss=0.4712, pruned_loss=0.1368, over 24273.00 frames. ], tot_loss[loss=0.3508, simple_loss=0.4371, pruned_loss=0.1322, over 4519717.92 frames. ], batch size: 257, lr: 1.65e-02, grad_scale: 32.0 2026-09-23 23:17:26,259 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=37026.666666666664, ans=0.125 2026-09-23 23:17:26,791 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=37026.666666666664, ans=0.05 2026-09-23 23:17:29,877 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.741e+02 3.365e+02 3.815e+02 4.468e+02 6.543e+02, threshold=7.630e+02, percent-clipped=0.0 2026-09-23 23:17:30,558 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=37060.0, ans=0.2 2026-09-23 23:17:34,399 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.67 vs. limit=15.0 2026-09-23 23:17:41,217 INFO [train.py:1192] (0/2) Epoch 12, batch 600, loss[loss=0.3665, simple_loss=0.4585, pruned_loss=0.1372, over 24331.00 frames. ], tot_loss[loss=0.3501, simple_loss=0.4372, pruned_loss=0.1315, over 4587000.73 frames. ], batch size: 234, lr: 1.65e-02, grad_scale: 32.0 2026-09-23 23:17:46,557 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=37160.0, ans=0.0 2026-09-23 23:18:03,279 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.64 vs. limit=10.0 2026-09-23 23:18:06,418 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=37260.0, ans=0.125 2026-09-23 23:18:07,215 INFO [train.py:1192] (0/2) Epoch 12, batch 650, loss[loss=0.3679, simple_loss=0.4471, pruned_loss=0.1443, over 24560.00 frames. ], tot_loss[loss=0.3491, simple_loss=0.4364, pruned_loss=0.1309, over 4651981.88 frames. ], batch size: 162, lr: 1.64e-02, grad_scale: 32.0 2026-09-23 23:18:13,700 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=37326.666666666664, ans=0.125 2026-09-23 23:18:17,069 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=37326.666666666664, ans=0.05 2026-09-23 23:18:22,116 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.397e+02 3.206e+02 3.726e+02 4.417e+02 8.115e+02, threshold=7.451e+02, percent-clipped=2.0 2026-09-23 23:18:22,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=37393.333333333336, ans=0.125 2026-09-23 23:18:33,709 INFO [train.py:1192] (0/2) Epoch 12, batch 700, loss[loss=0.3394, simple_loss=0.4256, pruned_loss=0.1266, over 24573.00 frames. ], tot_loss[loss=0.35, simple_loss=0.4374, pruned_loss=0.1313, over 4683608.96 frames. ], batch size: 154, lr: 1.64e-02, grad_scale: 32.0 2026-09-23 23:18:37,119 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=37460.0, ans=0.125 2026-09-23 23:18:44,503 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=37526.666666666664, ans=0.0 2026-09-23 23:18:59,038 INFO [train.py:1192] (0/2) Epoch 12, batch 750, loss[loss=0.3233, simple_loss=0.424, pruned_loss=0.1113, over 24568.00 frames. ], tot_loss[loss=0.3494, simple_loss=0.4365, pruned_loss=0.1311, over 4710893.57 frames. ], batch size: 170, lr: 1.64e-02, grad_scale: 32.0 2026-09-23 23:19:02,555 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=37626.666666666664, ans=0.125 2026-09-23 23:19:04,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=37660.0, ans=0.025 2026-09-23 23:19:13,350 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.77 vs. limit=22.5 2026-09-23 23:19:13,554 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.581e+02 3.554e+02 3.988e+02 5.020e+02 7.837e+02, threshold=7.976e+02, percent-clipped=1.0 2026-09-23 23:19:15,269 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=37726.666666666664, ans=0.1 2026-09-23 23:19:17,534 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=17.53 vs. limit=22.5 2026-09-23 23:19:21,565 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=37760.0, ans=0.125 2026-09-23 23:19:24,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=37760.0, ans=0.2 2026-09-23 23:19:25,089 INFO [train.py:1192] (0/2) Epoch 12, batch 800, loss[loss=0.2972, simple_loss=0.3835, pruned_loss=0.1055, over 24581.00 frames. ], tot_loss[loss=0.3482, simple_loss=0.4356, pruned_loss=0.1304, over 4739803.46 frames. ], batch size: 137, lr: 1.64e-02, grad_scale: 32.0 2026-09-23 23:19:39,719 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.25 vs. limit=12.0 2026-09-23 23:19:44,818 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=37893.333333333336, ans=0.125 2026-09-23 23:19:50,590 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.89 vs. limit=15.0 2026-09-23 23:19:51,289 INFO [train.py:1192] (0/2) Epoch 12, batch 850, loss[loss=0.3644, simple_loss=0.4624, pruned_loss=0.1332, over 24555.00 frames. ], tot_loss[loss=0.3481, simple_loss=0.4355, pruned_loss=0.1304, over 4762083.23 frames. ], batch size: 204, lr: 1.63e-02, grad_scale: 32.0 2026-09-23 23:19:51,400 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=37960.0, ans=0.1 2026-09-23 23:19:52,321 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=37960.0, ans=0.125 2026-09-23 23:19:55,852 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.10 vs. limit=22.5 2026-09-23 23:20:00,515 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=37993.333333333336, ans=0.125 2026-09-23 23:20:04,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=38026.666666666664, ans=0.2 2026-09-23 23:20:05,501 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=38026.666666666664, ans=0.0 2026-09-23 23:20:05,926 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.500e+02 3.277e+02 3.882e+02 4.474e+02 7.001e+02, threshold=7.765e+02, percent-clipped=0.0 2026-09-23 23:20:17,451 INFO [train.py:1192] (0/2) Epoch 12, batch 900, loss[loss=0.2993, simple_loss=0.394, pruned_loss=0.1023, over 24567.00 frames. ], tot_loss[loss=0.3485, simple_loss=0.436, pruned_loss=0.1305, over 4775854.91 frames. ], batch size: 137, lr: 1.63e-02, grad_scale: 32.0 2026-09-23 23:20:24,899 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=38160.0, ans=0.035 2026-09-23 23:20:29,904 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=38193.333333333336, ans=0.09899494936611666 2026-09-23 23:20:43,452 INFO [train.py:1192] (0/2) Epoch 12, batch 950, loss[loss=0.4602, simple_loss=0.4829, pruned_loss=0.2188, over 11781.00 frames. ], tot_loss[loss=0.3498, simple_loss=0.4351, pruned_loss=0.1322, over 4712548.77 frames. ], batch size: 333, lr: 1.63e-02, grad_scale: 16.0 2026-09-23 23:20:47,977 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-12.pt 2026-09-23 23:20:55,024 INFO [train.py:1192] (0/2) Epoch 13, batch 0, loss[loss=0.3147, simple_loss=0.4122, pruned_loss=0.1086, over 24579.00 frames. ], tot_loss[loss=0.3147, simple_loss=0.4122, pruned_loss=0.1086, over 24579.00 frames. ], batch size: 137, lr: 1.56e-02, grad_scale: 32.0 2026-09-23 23:20:55,025 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 23:21:06,513 INFO [train.py:1224] (0/2) Epoch 13, validation: loss=0.213, simple_loss=0.3279, pruned_loss=0.04906, over 2564189.00 frames. 2026-09-23 23:21:06,514 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 23:21:07,840 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=38320.0, ans=0.0 2026-09-23 23:21:08,888 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=38320.0, ans=0.1 2026-09-23 23:21:15,325 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=38353.333333333336, ans=0.125 2026-09-23 23:21:16,539 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.533e+02 3.165e+02 3.581e+02 4.158e+02 9.000e+02, threshold=7.163e+02, percent-clipped=2.0 2026-09-23 23:21:22,648 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=38420.0, ans=0.025 2026-09-23 23:21:29,709 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=38453.333333333336, ans=0.125 2026-09-23 23:21:32,016 INFO [train.py:1192] (0/2) Epoch 13, batch 50, loss[loss=0.3029, simple_loss=0.3865, pruned_loss=0.1097, over 24302.00 frames. ], tot_loss[loss=0.3612, simple_loss=0.4459, pruned_loss=0.1383, over 1077003.99 frames. ], batch size: 125, lr: 1.56e-02, grad_scale: 32.0 2026-09-23 23:21:41,701 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=17.23 vs. limit=22.5 2026-09-23 23:21:52,965 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=38620.0, ans=0.125 2026-09-23 23:21:57,489 INFO [train.py:1192] (0/2) Epoch 13, batch 100, loss[loss=0.3126, simple_loss=0.4056, pruned_loss=0.1098, over 24632.00 frames. ], tot_loss[loss=0.3581, simple_loss=0.4454, pruned_loss=0.1354, over 1905300.99 frames. ], batch size: 154, lr: 1.56e-02, grad_scale: 32.0 2026-09-23 23:21:57,577 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=38653.333333333336, ans=0.125 2026-09-23 23:21:57,615 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=38653.333333333336, ans=0.125 2026-09-23 23:22:00,816 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=7.05 vs. limit=15.0 2026-09-23 23:22:08,232 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.515e+02 3.237e+02 3.577e+02 4.076e+02 7.676e+02, threshold=7.155e+02, percent-clipped=1.0 2026-09-23 23:22:09,239 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=38720.0, ans=0.0024521739130434787 2026-09-23 23:22:10,751 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=38720.0, ans=0.125 2026-09-23 23:22:14,694 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=38753.333333333336, ans=0.04949747468305833 2026-09-23 23:22:15,928 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=38753.333333333336, ans=0.125 2026-09-23 23:22:21,919 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=38786.666666666664, ans=0.00243768115942029 2026-09-23 23:22:22,957 INFO [train.py:1192] (0/2) Epoch 13, batch 150, loss[loss=0.3314, simple_loss=0.4, pruned_loss=0.1314, over 24273.00 frames. ], tot_loss[loss=0.3529, simple_loss=0.4402, pruned_loss=0.1328, over 2558693.64 frames. ], batch size: 125, lr: 1.56e-02, grad_scale: 32.0 2026-09-23 23:22:24,939 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=38820.0, ans=0.002430434782608696 2026-09-23 23:22:41,112 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=38920.0, ans=0.125 2026-09-23 23:22:48,533 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=38953.333333333336, ans=0.125 2026-09-23 23:22:49,425 INFO [train.py:1192] (0/2) Epoch 13, batch 200, loss[loss=0.4004, simple_loss=0.4627, pruned_loss=0.1691, over 20992.00 frames. ], tot_loss[loss=0.3485, simple_loss=0.4367, pruned_loss=0.1301, over 3054427.14 frames. ], batch size: 333, lr: 1.56e-02, grad_scale: 32.0 2026-09-23 23:22:49,518 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=38986.666666666664, ans=0.002394202898550725 2026-09-23 23:22:53,797 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.06 vs. limit=10.0 2026-09-23 23:22:59,241 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=39020.0, ans=0.0 2026-09-23 23:23:00,612 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.461e+02 3.338e+02 3.856e+02 4.462e+02 9.088e+02, threshold=7.711e+02, percent-clipped=1.0 2026-09-23 23:23:01,330 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=39053.333333333336, ans=0.2 2026-09-23 23:23:05,611 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=39086.666666666664, ans=0.1 2026-09-23 23:23:06,453 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=39086.666666666664, ans=0.0 2026-09-23 23:23:12,028 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.98 vs. limit=12.0 2026-09-23 23:23:12,446 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=39120.0, ans=0.0023652173913043485 2026-09-23 23:23:16,061 INFO [train.py:1192] (0/2) Epoch 13, batch 250, loss[loss=0.374, simple_loss=0.4672, pruned_loss=0.1404, over 24313.00 frames. ], tot_loss[loss=0.347, simple_loss=0.4355, pruned_loss=0.1293, over 3443341.12 frames. ], batch size: 234, lr: 1.55e-02, grad_scale: 32.0 2026-09-23 23:23:28,297 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=39220.0, ans=0.125 2026-09-23 23:23:34,436 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.75 vs. limit=15.0 2026-09-23 23:23:41,848 INFO [train.py:1192] (0/2) Epoch 13, batch 300, loss[loss=0.4027, simple_loss=0.49, pruned_loss=0.1576, over 24557.00 frames. ], tot_loss[loss=0.3463, simple_loss=0.4346, pruned_loss=0.129, over 3749104.98 frames. ], batch size: 204, lr: 1.55e-02, grad_scale: 32.0 2026-09-23 23:23:52,604 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.522e+02 3.399e+02 3.954e+02 4.648e+02 7.202e+02, threshold=7.908e+02, percent-clipped=0.0 2026-09-23 23:23:59,028 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=39420.0, ans=0.125 2026-09-23 23:24:04,918 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=39453.333333333336, ans=0.0 2026-09-23 23:24:06,328 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=39453.333333333336, ans=0.1 2026-09-23 23:24:07,904 INFO [train.py:1192] (0/2) Epoch 13, batch 350, loss[loss=0.2562, simple_loss=0.3585, pruned_loss=0.077, over 24590.00 frames. ], tot_loss[loss=0.3475, simple_loss=0.4357, pruned_loss=0.1296, over 3993596.15 frames. ], batch size: 137, lr: 1.55e-02, grad_scale: 32.0 2026-09-23 23:24:11,746 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=39486.666666666664, ans=0.035 2026-09-23 23:24:18,182 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=39553.333333333336, ans=0.2 2026-09-23 23:24:20,588 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=39553.333333333336, ans=0.125 2026-09-23 23:24:24,876 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=39586.666666666664, ans=0.125 2026-09-23 23:24:34,311 INFO [train.py:1192] (0/2) Epoch 13, batch 400, loss[loss=0.3964, simple_loss=0.4661, pruned_loss=0.1634, over 24547.00 frames. ], tot_loss[loss=0.3458, simple_loss=0.4342, pruned_loss=0.1287, over 4180626.47 frames. ], batch size: 170, lr: 1.55e-02, grad_scale: 32.0 2026-09-23 23:24:43,856 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=39686.666666666664, ans=0.0 2026-09-23 23:24:45,094 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.476e+02 3.301e+02 3.730e+02 4.213e+02 6.408e+02, threshold=7.460e+02, percent-clipped=0.0 2026-09-23 23:24:52,135 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=39753.333333333336, ans=0.0 2026-09-23 23:24:58,553 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:24:59,859 INFO [train.py:1192] (0/2) Epoch 13, batch 450, loss[loss=0.3429, simple_loss=0.439, pruned_loss=0.1234, over 24635.00 frames. ], tot_loss[loss=0.3457, simple_loss=0.4343, pruned_loss=0.1285, over 4312926.47 frames. ], batch size: 175, lr: 1.54e-02, grad_scale: 32.0 2026-09-23 23:25:00,984 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=17.18 vs. limit=22.5 2026-09-23 23:25:01,324 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=39820.0, ans=0.07 2026-09-23 23:25:04,949 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=39853.333333333336, ans=0.0 2026-09-23 23:25:25,632 INFO [train.py:1192] (0/2) Epoch 13, batch 500, loss[loss=0.3903, simple_loss=0.4771, pruned_loss=0.1517, over 24492.00 frames. ], tot_loss[loss=0.3431, simple_loss=0.4319, pruned_loss=0.1272, over 4429821.50 frames. ], batch size: 218, lr: 1.54e-02, grad_scale: 32.0 2026-09-23 23:25:26,207 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.48 vs. limit=12.0 2026-09-23 23:25:27,080 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-12000.pt 2026-09-23 23:25:36,471 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=40053.333333333336, ans=0.125 2026-09-23 23:25:37,295 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.487e+02 3.141e+02 3.810e+02 4.692e+02 7.001e+02, threshold=7.621e+02, percent-clipped=0.0 2026-09-23 23:25:41,390 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=40086.666666666664, ans=0.0 2026-09-23 23:25:45,412 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=40086.666666666664, ans=15.0 2026-09-23 23:25:50,231 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=40120.0, ans=0.0 2026-09-23 23:25:51,571 INFO [train.py:1192] (0/2) Epoch 13, batch 550, loss[loss=0.3694, simple_loss=0.4684, pruned_loss=0.1352, over 24286.00 frames. ], tot_loss[loss=0.3427, simple_loss=0.4321, pruned_loss=0.1267, over 4519158.74 frames. ], batch size: 257, lr: 1.54e-02, grad_scale: 16.0 2026-09-23 23:25:51,672 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=40153.333333333336, ans=0.2 2026-09-23 23:26:05,000 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=40220.0, ans=0.0 2026-09-23 23:26:17,023 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=40286.666666666664, ans=0.125 2026-09-23 23:26:18,339 INFO [train.py:1192] (0/2) Epoch 13, batch 600, loss[loss=0.3838, simple_loss=0.479, pruned_loss=0.1443, over 24296.00 frames. ], tot_loss[loss=0.3442, simple_loss=0.4334, pruned_loss=0.1275, over 4584740.04 frames. ], batch size: 234, lr: 1.54e-02, grad_scale: 16.0 2026-09-23 23:26:18,442 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=40320.0, ans=0.2 2026-09-23 23:26:18,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=40320.0, ans=0.0 2026-09-23 23:26:27,481 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=5.28 vs. limit=15.0 2026-09-23 23:26:28,248 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=40386.666666666664, ans=0.04949747468305833 2026-09-23 23:26:29,963 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.559e+02 3.184e+02 3.728e+02 4.522e+02 8.525e+02, threshold=7.455e+02, percent-clipped=1.0 2026-09-23 23:26:39,533 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=40453.333333333336, ans=0.125 2026-09-23 23:26:43,247 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=40453.333333333336, ans=0.125 2026-09-23 23:26:44,153 INFO [train.py:1192] (0/2) Epoch 13, batch 650, loss[loss=0.3659, simple_loss=0.4465, pruned_loss=0.1426, over 24575.00 frames. ], tot_loss[loss=0.3431, simple_loss=0.4323, pruned_loss=0.127, over 4650281.62 frames. ], batch size: 162, lr: 1.53e-02, grad_scale: 16.0 2026-09-23 23:26:52,250 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=40520.0, ans=0.2 2026-09-23 23:26:52,556 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=15.80 vs. limit=22.5 2026-09-23 23:26:54,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=40553.333333333336, ans=0.1 2026-09-23 23:27:00,028 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=40586.666666666664, ans=0.07 2026-09-23 23:27:01,407 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=40586.666666666664, ans=0.1 2026-09-23 23:27:07,562 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=40620.0, ans=0.125 2026-09-23 23:27:08,530 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=40620.0, ans=0.125 2026-09-23 23:27:09,839 INFO [train.py:1192] (0/2) Epoch 13, batch 700, loss[loss=0.3555, simple_loss=0.4307, pruned_loss=0.1402, over 24575.00 frames. ], tot_loss[loss=0.344, simple_loss=0.4331, pruned_loss=0.1274, over 4683673.42 frames. ], batch size: 154, lr: 1.53e-02, grad_scale: 16.0 2026-09-23 23:27:10,809 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=40653.333333333336, ans=0.2 2026-09-23 23:27:15,566 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=40686.666666666664, ans=0.025 2026-09-23 23:27:19,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=40686.666666666664, ans=0.1 2026-09-23 23:27:20,806 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=40720.0, ans=0.1 2026-09-23 23:27:21,089 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.487e+02 3.495e+02 4.223e+02 5.427e+02 8.388e+02, threshold=8.446e+02, percent-clipped=3.0 2026-09-23 23:27:25,994 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=40753.333333333336, ans=0.125 2026-09-23 23:27:35,112 INFO [train.py:1192] (0/2) Epoch 13, batch 750, loss[loss=0.3297, simple_loss=0.4303, pruned_loss=0.1146, over 24562.00 frames. ], tot_loss[loss=0.3427, simple_loss=0.4318, pruned_loss=0.1268, over 4715003.68 frames. ], batch size: 170, lr: 1.53e-02, grad_scale: 16.0 2026-09-23 23:27:36,577 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=40820.0, ans=0.125 2026-09-23 23:27:38,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=40820.0, ans=0.0 2026-09-23 23:27:43,194 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=40853.333333333336, ans=0.0019884057971014495 2026-09-23 23:27:46,484 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=40886.666666666664, ans=0.0 2026-09-23 23:27:55,779 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=40953.333333333336, ans=0.125 2026-09-23 23:27:58,698 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=40953.333333333336, ans=0.125 2026-09-23 23:28:00,446 INFO [train.py:1192] (0/2) Epoch 13, batch 800, loss[loss=0.2695, simple_loss=0.3678, pruned_loss=0.0856, over 24534.00 frames. ], tot_loss[loss=0.3419, simple_loss=0.4311, pruned_loss=0.1264, over 4738993.40 frames. ], batch size: 137, lr: 1.53e-02, grad_scale: 32.0 2026-09-23 23:28:05,722 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=41020.0, ans=0.0019521739130434791 2026-09-23 23:28:12,117 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.613e+02 3.339e+02 3.897e+02 4.682e+02 7.943e+02, threshold=7.793e+02, percent-clipped=0.0 2026-09-23 23:28:26,626 INFO [train.py:1192] (0/2) Epoch 13, batch 850, loss[loss=0.3372, simple_loss=0.439, pruned_loss=0.1177, over 24599.00 frames. ], tot_loss[loss=0.3412, simple_loss=0.4308, pruned_loss=0.1258, over 4760583.24 frames. ], batch size: 198, lr: 1.53e-02, grad_scale: 32.0 2026-09-23 23:28:30,580 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=41153.333333333336, ans=0.0 2026-09-23 23:28:34,353 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=41186.666666666664, ans=0.1 2026-09-23 23:28:36,533 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.29 vs. limit=15.0 2026-09-23 23:28:38,370 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=41220.0, ans=0.125 2026-09-23 23:28:42,777 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.14 vs. limit=12.0 2026-09-23 23:28:48,969 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=41286.666666666664, ans=0.125 2026-09-23 23:28:52,873 INFO [train.py:1192] (0/2) Epoch 13, batch 900, loss[loss=0.3006, simple_loss=0.3944, pruned_loss=0.1035, over 24564.00 frames. ], tot_loss[loss=0.3423, simple_loss=0.4316, pruned_loss=0.1265, over 4774233.98 frames. ], batch size: 137, lr: 1.52e-02, grad_scale: 32.0 2026-09-23 23:29:02,210 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=41353.333333333336, ans=0.1 2026-09-23 23:29:03,879 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.91 vs. limit=6.0 2026-09-23 23:29:04,176 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.460e+02 3.273e+02 3.594e+02 4.174e+02 7.421e+02, threshold=7.188e+02, percent-clipped=0.0 2026-09-23 23:29:05,240 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=41386.666666666664, ans=0.125 2026-09-23 23:29:05,352 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.48 vs. limit=6.0 2026-09-23 23:29:10,442 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.71 vs. limit=15.0 2026-09-23 23:29:16,864 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=41453.333333333336, ans=0.2 2026-09-23 23:29:18,118 INFO [train.py:1192] (0/2) Epoch 13, batch 950, loss[loss=0.4656, simple_loss=0.4937, pruned_loss=0.2187, over 11601.00 frames. ], tot_loss[loss=0.3436, simple_loss=0.4309, pruned_loss=0.1281, over 4718988.88 frames. ], batch size: 333, lr: 1.52e-02, grad_scale: 32.0 2026-09-23 23:29:20,166 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=41486.666666666664, ans=0.125 2026-09-23 23:29:20,309 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=21.08 vs. limit=15.0 2026-09-23 23:29:22,599 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-13.pt 2026-09-23 23:29:29,337 INFO [train.py:1192] (0/2) Epoch 14, batch 0, loss[loss=0.2944, simple_loss=0.3912, pruned_loss=0.09882, over 24563.00 frames. ], tot_loss[loss=0.2944, simple_loss=0.3912, pruned_loss=0.09882, over 24563.00 frames. ], batch size: 137, lr: 1.46e-02, grad_scale: 32.0 2026-09-23 23:29:29,337 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 23:29:39,030 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.8170, 2.2286, 1.7077, 2.6587], device='cuda:0') 2026-09-23 23:29:40,950 INFO [train.py:1224] (0/2) Epoch 14, validation: loss=0.2077, simple_loss=0.3229, pruned_loss=0.04624, over 2564189.00 frames. 2026-09-23 23:29:40,950 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 23:29:42,006 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=41513.333333333336, ans=0.125 2026-09-23 23:29:53,017 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.68 vs. limit=15.0 2026-09-23 23:29:57,407 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.68 vs. limit=15.0 2026-09-23 23:29:58,747 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=41613.333333333336, ans=0.05 2026-09-23 23:30:02,678 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=41646.666666666664, ans=0.1 2026-09-23 23:30:04,535 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=41646.666666666664, ans=0.125 2026-09-23 23:30:06,381 INFO [train.py:1192] (0/2) Epoch 14, batch 50, loss[loss=0.2751, simple_loss=0.3648, pruned_loss=0.09272, over 24264.00 frames. ], tot_loss[loss=0.3521, simple_loss=0.4396, pruned_loss=0.1322, over 1075188.71 frames. ], batch size: 125, lr: 1.46e-02, grad_scale: 32.0 2026-09-23 23:30:13,341 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.567e+02 3.394e+02 3.972e+02 4.555e+02 8.560e+02, threshold=7.945e+02, percent-clipped=1.0 2026-09-23 23:30:16,364 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.03 vs. limit=22.5 2026-09-23 23:30:24,083 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=41780.0, ans=0.2 2026-09-23 23:30:25,171 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=41780.0, ans=0.0 2026-09-23 23:30:30,264 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=41813.333333333336, ans=0.125 2026-09-23 23:30:31,892 INFO [train.py:1192] (0/2) Epoch 14, batch 100, loss[loss=0.2998, simple_loss=0.4028, pruned_loss=0.09838, over 24621.00 frames. ], tot_loss[loss=0.3549, simple_loss=0.443, pruned_loss=0.1333, over 1903658.35 frames. ], batch size: 154, lr: 1.46e-02, grad_scale: 32.0 2026-09-23 23:30:47,732 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.79 vs. limit=15.0 2026-09-23 23:30:50,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=41946.666666666664, ans=0.125 2026-09-23 23:30:57,600 INFO [train.py:1192] (0/2) Epoch 14, batch 150, loss[loss=0.3101, simple_loss=0.3912, pruned_loss=0.1145, over 24274.00 frames. ], tot_loss[loss=0.3445, simple_loss=0.4345, pruned_loss=0.1273, over 2557547.46 frames. ], batch size: 125, lr: 1.46e-02, grad_scale: 32.0 2026-09-23 23:30:57,812 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.33 vs. limit=6.0 2026-09-23 23:31:02,229 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=42046.666666666664, ans=0.125 2026-09-23 23:31:04,512 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.486e+02 3.227e+02 3.665e+02 4.456e+02 7.037e+02, threshold=7.330e+02, percent-clipped=0.0 2026-09-23 23:31:15,309 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=2.74 vs. limit=15.0 2026-09-23 23:31:16,458 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.01 vs. limit=15.0 2026-09-23 23:31:21,849 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.79 vs. limit=15.0 2026-09-23 23:31:23,168 INFO [train.py:1192] (0/2) Epoch 14, batch 200, loss[loss=0.3972, simple_loss=0.4678, pruned_loss=0.1633, over 21089.00 frames. ], tot_loss[loss=0.3416, simple_loss=0.4319, pruned_loss=0.1256, over 3055333.25 frames. ], batch size: 333, lr: 1.46e-02, grad_scale: 32.0 2026-09-23 23:31:39,980 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=42280.0, ans=0.025 2026-09-23 23:31:42,041 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.50 vs. limit=22.5 2026-09-23 23:31:47,748 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=42313.333333333336, ans=0.125 2026-09-23 23:31:48,780 INFO [train.py:1192] (0/2) Epoch 14, batch 250, loss[loss=0.4056, simple_loss=0.4956, pruned_loss=0.1577, over 24313.00 frames. ], tot_loss[loss=0.342, simple_loss=0.4321, pruned_loss=0.1259, over 3443094.98 frames. ], batch size: 234, lr: 1.45e-02, grad_scale: 32.0 2026-09-23 23:31:55,688 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.358e+02 3.519e+02 3.925e+02 4.523e+02 7.075e+02, threshold=7.850e+02, percent-clipped=0.0 2026-09-23 23:31:56,010 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.15 vs. limit=15.0 2026-09-23 23:32:01,370 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.83 vs. limit=15.0 2026-09-23 23:32:01,877 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=42413.333333333336, ans=0.125 2026-09-23 23:32:04,322 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=42446.666666666664, ans=0.125 2026-09-23 23:32:14,258 INFO [train.py:1192] (0/2) Epoch 14, batch 300, loss[loss=0.3282, simple_loss=0.4369, pruned_loss=0.1098, over 24507.00 frames. ], tot_loss[loss=0.3402, simple_loss=0.4305, pruned_loss=0.125, over 3748366.60 frames. ], batch size: 204, lr: 1.45e-02, grad_scale: 32.0 2026-09-23 23:32:16,207 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=42513.333333333336, ans=0.125 2026-09-23 23:32:18,668 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=42546.666666666664, ans=0.1 2026-09-23 23:32:24,403 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.64 vs. limit=10.0 2026-09-23 23:32:26,172 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=42580.0, ans=0.00161304347826087 2026-09-23 23:32:40,074 INFO [train.py:1192] (0/2) Epoch 14, batch 350, loss[loss=0.2896, simple_loss=0.3824, pruned_loss=0.09845, over 24572.00 frames. ], tot_loss[loss=0.3412, simple_loss=0.4316, pruned_loss=0.1254, over 3989468.90 frames. ], batch size: 137, lr: 1.45e-02, grad_scale: 32.0 2026-09-23 23:32:44,670 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=42713.333333333336, ans=0.125 2026-09-23 23:32:47,135 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.578e+02 3.219e+02 3.708e+02 4.100e+02 5.899e+02, threshold=7.416e+02, percent-clipped=0.0 2026-09-23 23:32:50,185 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.08 vs. limit=15.0 2026-09-23 23:32:53,780 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=42746.666666666664, ans=0.125 2026-09-23 23:33:01,544 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=42813.333333333336, ans=0.125 2026-09-23 23:33:03,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=42813.333333333336, ans=0.0 2026-09-23 23:33:06,221 INFO [train.py:1192] (0/2) Epoch 14, batch 400, loss[loss=0.3252, simple_loss=0.4178, pruned_loss=0.1163, over 24549.00 frames. ], tot_loss[loss=0.3391, simple_loss=0.4297, pruned_loss=0.1242, over 4176538.34 frames. ], batch size: 170, lr: 1.45e-02, grad_scale: 32.0 2026-09-23 23:33:24,484 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=9.31 vs. limit=15.0 2026-09-23 23:33:32,716 INFO [train.py:1192] (0/2) Epoch 14, batch 450, loss[loss=0.3595, simple_loss=0.4497, pruned_loss=0.1346, over 24637.00 frames. ], tot_loss[loss=0.3406, simple_loss=0.4309, pruned_loss=0.1251, over 4310226.92 frames. ], batch size: 175, lr: 1.45e-02, grad_scale: 32.0 2026-09-23 23:33:33,345 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.82 vs. limit=22.5 2026-09-23 23:33:38,945 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=43046.666666666664, ans=0.0 2026-09-23 23:33:39,875 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.447e+02 3.281e+02 3.675e+02 4.156e+02 6.613e+02, threshold=7.350e+02, percent-clipped=0.0 2026-09-23 23:33:39,999 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=43046.666666666664, ans=0.125 2026-09-23 23:33:41,124 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=43046.666666666664, ans=0.125 2026-09-23 23:33:50,086 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=43113.333333333336, ans=0.1 2026-09-23 23:33:57,219 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=43146.666666666664, ans=0.2 2026-09-23 23:33:58,585 INFO [train.py:1192] (0/2) Epoch 14, batch 500, loss[loss=0.359, simple_loss=0.4544, pruned_loss=0.1319, over 24488.00 frames. ], tot_loss[loss=0.3395, simple_loss=0.4295, pruned_loss=0.1247, over 4428101.82 frames. ], batch size: 218, lr: 1.44e-02, grad_scale: 32.0 2026-09-23 23:34:02,094 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=43180.0, ans=0.05 2026-09-23 23:34:17,703 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=43280.0, ans=0.1 2026-09-23 23:34:22,890 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=43313.333333333336, ans=0.125 2026-09-23 23:34:24,665 INFO [train.py:1192] (0/2) Epoch 14, batch 550, loss[loss=0.3984, simple_loss=0.4796, pruned_loss=0.1586, over 24286.00 frames. ], tot_loss[loss=0.3401, simple_loss=0.4301, pruned_loss=0.1251, over 4517766.29 frames. ], batch size: 257, lr: 1.44e-02, grad_scale: 32.0 2026-09-23 23:34:25,567 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=43346.666666666664, ans=0.2 2026-09-23 23:34:32,374 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.484e+02 3.155e+02 3.570e+02 4.011e+02 7.207e+02, threshold=7.140e+02, percent-clipped=0.0 2026-09-23 23:34:40,231 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.33 vs. limit=6.0 2026-09-23 23:34:41,987 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=43446.666666666664, ans=0.1 2026-09-23 23:34:47,694 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=43480.0, ans=0.1 2026-09-23 23:34:50,849 INFO [train.py:1192] (0/2) Epoch 14, batch 600, loss[loss=0.385, simple_loss=0.4758, pruned_loss=0.1471, over 24334.00 frames. ], tot_loss[loss=0.3411, simple_loss=0.4312, pruned_loss=0.1255, over 4584870.09 frames. ], batch size: 234, lr: 1.44e-02, grad_scale: 32.0 2026-09-23 23:34:53,241 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=43513.333333333336, ans=0.0 2026-09-23 23:35:09,636 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=43613.333333333336, ans=0.125 2026-09-23 23:35:17,058 INFO [train.py:1192] (0/2) Epoch 14, batch 650, loss[loss=0.3617, simple_loss=0.4432, pruned_loss=0.1401, over 24569.00 frames. ], tot_loss[loss=0.34, simple_loss=0.4303, pruned_loss=0.1248, over 4650082.27 frames. ], batch size: 162, lr: 1.44e-02, grad_scale: 32.0 2026-09-23 23:35:19,074 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=43680.0, ans=0.125 2026-09-23 23:35:24,144 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.655e+02 3.233e+02 3.641e+02 4.223e+02 7.455e+02, threshold=7.283e+02, percent-clipped=1.0 2026-09-23 23:35:31,645 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=43746.666666666664, ans=0.1 2026-09-23 23:35:33,195 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=43780.0, ans=0.2 2026-09-23 23:35:37,918 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=43813.333333333336, ans=0.125 2026-09-23 23:35:43,297 INFO [train.py:1192] (0/2) Epoch 14, batch 700, loss[loss=0.3219, simple_loss=0.4113, pruned_loss=0.1163, over 24566.00 frames. ], tot_loss[loss=0.3413, simple_loss=0.4316, pruned_loss=0.1255, over 4682361.06 frames. ], batch size: 154, lr: 1.44e-02, grad_scale: 32.0 2026-09-23 23:35:47,625 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=43846.666666666664, ans=0.125 2026-09-23 23:35:53,902 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=43913.333333333336, ans=0.1 2026-09-23 23:36:02,241 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=43946.666666666664, ans=0.125 2026-09-23 23:36:02,617 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=43946.666666666664, ans=0.1 2026-09-23 23:36:09,620 INFO [train.py:1192] (0/2) Epoch 14, batch 750, loss[loss=0.3519, simple_loss=0.4402, pruned_loss=0.1318, over 24561.00 frames. ], tot_loss[loss=0.34, simple_loss=0.4301, pruned_loss=0.125, over 4709565.06 frames. ], batch size: 170, lr: 1.43e-02, grad_scale: 32.0 2026-09-23 23:36:16,552 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.458e+02 3.410e+02 3.887e+02 4.924e+02 8.810e+02, threshold=7.774e+02, percent-clipped=3.0 2026-09-23 23:36:22,018 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=44080.0, ans=0.2 2026-09-23 23:36:22,912 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=44080.0, ans=0.5 2026-09-23 23:36:30,027 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=44146.666666666664, ans=0.2 2026-09-23 23:36:33,613 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=44146.666666666664, ans=0.025 2026-09-23 23:36:34,675 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.32 vs. limit=12.0 2026-09-23 23:36:35,425 INFO [train.py:1192] (0/2) Epoch 14, batch 800, loss[loss=0.2623, simple_loss=0.3625, pruned_loss=0.08108, over 24560.00 frames. ], tot_loss[loss=0.3386, simple_loss=0.4288, pruned_loss=0.1241, over 4735778.19 frames. ], batch size: 137, lr: 1.43e-02, grad_scale: 32.0 2026-09-23 23:36:54,318 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.32 vs. limit=22.5 2026-09-23 23:36:55,793 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=44313.333333333336, ans=0.0 2026-09-23 23:37:01,134 INFO [train.py:1192] (0/2) Epoch 14, batch 850, loss[loss=0.3562, simple_loss=0.4498, pruned_loss=0.1313, over 24579.00 frames. ], tot_loss[loss=0.338, simple_loss=0.4286, pruned_loss=0.1237, over 4758308.91 frames. ], batch size: 198, lr: 1.43e-02, grad_scale: 32.0 2026-09-23 23:37:08,655 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.320e+02 3.426e+02 4.029e+02 4.509e+02 7.575e+02, threshold=8.058e+02, percent-clipped=0.0 2026-09-23 23:37:12,054 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=44413.333333333336, ans=0.2 2026-09-23 23:37:13,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=44413.333333333336, ans=0.1 2026-09-23 23:37:16,938 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=44446.666666666664, ans=0.125 2026-09-23 23:37:20,311 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=44446.666666666664, ans=0.0 2026-09-23 23:37:27,244 INFO [train.py:1192] (0/2) Epoch 14, batch 900, loss[loss=0.2717, simple_loss=0.3736, pruned_loss=0.08494, over 24576.00 frames. ], tot_loss[loss=0.3378, simple_loss=0.4286, pruned_loss=0.1235, over 4773137.27 frames. ], batch size: 137, lr: 1.43e-02, grad_scale: 32.0 2026-09-23 23:37:31,686 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.93 vs. limit=6.0 2026-09-23 23:37:39,634 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=44580.0, ans=0.125 2026-09-23 23:37:42,362 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=44613.333333333336, ans=10.0 2026-09-23 23:37:46,378 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=44613.333333333336, ans=0.125 2026-09-23 23:37:52,366 INFO [train.py:1192] (0/2) Epoch 14, batch 950, loss[loss=0.5175, simple_loss=0.5083, pruned_loss=0.2634, over 11217.00 frames. ], tot_loss[loss=0.3386, simple_loss=0.4274, pruned_loss=0.1249, over 4716537.28 frames. ], batch size: 333, lr: 1.43e-02, grad_scale: 32.0 2026-09-23 23:37:52,465 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=44680.0, ans=0.125 2026-09-23 23:37:52,588 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.44 vs. limit=15.0 2026-09-23 23:37:56,574 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-14.pt 2026-09-23 23:38:03,846 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.57 vs. limit=15.0 2026-09-23 23:38:04,065 INFO [train.py:1192] (0/2) Epoch 15, batch 0, loss[loss=0.2957, simple_loss=0.3964, pruned_loss=0.09749, over 24551.00 frames. ], tot_loss[loss=0.2957, simple_loss=0.3964, pruned_loss=0.09749, over 24551.00 frames. ], batch size: 137, lr: 1.38e-02, grad_scale: 32.0 2026-09-23 23:38:04,065 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 23:38:05,814 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.1721, 2.2203, 1.7762, 2.7282], device='cuda:0') 2026-09-23 23:38:07,305 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.8566, 2.5648, 3.4404, 1.6022], device='cuda:0') 2026-09-23 23:38:15,993 INFO [train.py:1224] (0/2) Epoch 15, validation: loss=0.2061, simple_loss=0.3207, pruned_loss=0.04577, over 2564189.00 frames. 2026-09-23 23:38:15,993 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 23:38:19,721 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.501e+02 3.380e+02 3.957e+02 4.664e+02 1.051e+03, threshold=7.915e+02, percent-clipped=1.0 2026-09-23 23:38:29,016 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=44773.333333333336, ans=0.0 2026-09-23 23:38:31,350 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=44806.666666666664, ans=0.125 2026-09-23 23:38:35,282 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=44806.666666666664, ans=0.2 2026-09-23 23:38:39,510 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=44840.0, ans=0.1 2026-09-23 23:38:41,714 INFO [train.py:1192] (0/2) Epoch 15, batch 50, loss[loss=0.3047, simple_loss=0.3871, pruned_loss=0.1112, over 24252.00 frames. ], tot_loss[loss=0.344, simple_loss=0.4342, pruned_loss=0.1269, over 1075864.90 frames. ], batch size: 125, lr: 1.37e-02, grad_scale: 32.0 2026-09-23 23:38:46,241 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=44906.666666666664, ans=0.025 2026-09-23 23:38:46,772 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=44906.666666666664, ans=0.1 2026-09-23 23:38:58,874 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=44973.333333333336, ans=0.0010927536231884055 2026-09-23 23:39:03,755 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=45006.666666666664, ans=0.125 2026-09-23 23:39:07,645 INFO [train.py:1192] (0/2) Epoch 15, batch 100, loss[loss=0.3518, simple_loss=0.431, pruned_loss=0.1362, over 24618.00 frames. ], tot_loss[loss=0.3482, simple_loss=0.4389, pruned_loss=0.1287, over 1904451.25 frames. ], batch size: 154, lr: 1.37e-02, grad_scale: 32.0 2026-09-23 23:39:10,964 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.413e+02 3.189e+02 3.581e+02 3.932e+02 5.918e+02, threshold=7.163e+02, percent-clipped=0.0 2026-09-23 23:39:27,984 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=45173.333333333336, ans=0.2 2026-09-23 23:39:31,614 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=45173.333333333336, ans=0.125 2026-09-23 23:39:34,012 INFO [train.py:1192] (0/2) Epoch 15, batch 150, loss[loss=0.267, simple_loss=0.363, pruned_loss=0.08556, over 24265.00 frames. ], tot_loss[loss=0.3406, simple_loss=0.4325, pruned_loss=0.1244, over 2558247.69 frames. ], batch size: 125, lr: 1.37e-02, grad_scale: 32.0 2026-09-23 23:39:34,529 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=45206.666666666664, ans=0.125 2026-09-23 23:39:37,659 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:39:42,231 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.65 vs. limit=15.0 2026-09-23 23:39:43,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=45273.333333333336, ans=0.2 2026-09-23 23:39:46,304 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=45273.333333333336, ans=0.125 2026-09-23 23:39:59,901 INFO [train.py:1192] (0/2) Epoch 15, batch 200, loss[loss=0.4339, simple_loss=0.4933, pruned_loss=0.1872, over 21121.00 frames. ], tot_loss[loss=0.338, simple_loss=0.43, pruned_loss=0.123, over 3053187.09 frames. ], batch size: 333, lr: 1.37e-02, grad_scale: 32.0 2026-09-23 23:40:03,854 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.661e+02 3.367e+02 3.964e+02 4.660e+02 6.467e+02, threshold=7.928e+02, percent-clipped=0.0 2026-09-23 23:40:05,577 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.30 vs. limit=22.5 2026-09-23 23:40:14,321 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.56 vs. limit=22.5 2026-09-23 23:40:24,016 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=45506.666666666664, ans=0.0 2026-09-23 23:40:25,177 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.34 vs. limit=15.0 2026-09-23 23:40:25,871 INFO [train.py:1192] (0/2) Epoch 15, batch 250, loss[loss=0.358, simple_loss=0.4569, pruned_loss=0.1295, over 24303.00 frames. ], tot_loss[loss=0.3361, simple_loss=0.4286, pruned_loss=0.1218, over 3442687.71 frames. ], batch size: 234, lr: 1.37e-02, grad_scale: 32.0 2026-09-23 23:40:32,268 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=45573.333333333336, ans=0.95 2026-09-23 23:40:44,513 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:40:52,021 INFO [train.py:1192] (0/2) Epoch 15, batch 300, loss[loss=0.3498, simple_loss=0.4587, pruned_loss=0.1204, over 24575.00 frames. ], tot_loss[loss=0.3335, simple_loss=0.4262, pruned_loss=0.1205, over 3748630.85 frames. ], batch size: 204, lr: 1.36e-02, grad_scale: 32.0 2026-09-23 23:40:55,733 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.700e+02 3.209e+02 3.542e+02 4.096e+02 7.328e+02, threshold=7.083e+02, percent-clipped=0.0 2026-09-23 23:40:55,843 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=45706.666666666664, ans=0.1 2026-09-23 23:41:18,351 INFO [train.py:1192] (0/2) Epoch 15, batch 350, loss[loss=0.2596, simple_loss=0.359, pruned_loss=0.08009, over 24562.00 frames. ], tot_loss[loss=0.3336, simple_loss=0.4265, pruned_loss=0.1204, over 3991801.79 frames. ], batch size: 137, lr: 1.36e-02, grad_scale: 32.0 2026-09-23 23:41:28,448 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=45940.0, ans=0.125 2026-09-23 23:41:33,558 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=45973.333333333336, ans=0.125 2026-09-23 23:41:42,810 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.33 vs. limit=15.0 2026-09-23 23:41:43,158 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=46006.666666666664, ans=0.125 2026-09-23 23:41:44,561 INFO [train.py:1192] (0/2) Epoch 15, batch 400, loss[loss=0.3406, simple_loss=0.4322, pruned_loss=0.1245, over 24557.00 frames. ], tot_loss[loss=0.334, simple_loss=0.4266, pruned_loss=0.1207, over 4178814.27 frames. ], batch size: 170, lr: 1.36e-02, grad_scale: 32.0 2026-09-23 23:41:46,571 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=46040.0, ans=0.2 2026-09-23 23:41:47,914 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.656e+02 3.547e+02 3.870e+02 4.473e+02 8.428e+02, threshold=7.740e+02, percent-clipped=1.0 2026-09-23 23:41:49,993 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=46073.333333333336, ans=0.0 2026-09-23 23:41:53,713 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=46073.333333333336, ans=0.2 2026-09-23 23:42:03,200 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=46140.0, ans=0.07 2026-09-23 23:42:10,753 INFO [train.py:1192] (0/2) Epoch 15, batch 450, loss[loss=0.3221, simple_loss=0.4189, pruned_loss=0.1126, over 24643.00 frames. ], tot_loss[loss=0.3356, simple_loss=0.4275, pruned_loss=0.1218, over 4311784.19 frames. ], batch size: 175, lr: 1.36e-02, grad_scale: 32.0 2026-09-23 23:42:12,190 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:42:13,298 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=46206.666666666664, ans=0.0 2026-09-23 23:42:36,256 INFO [train.py:1192] (0/2) Epoch 15, batch 500, loss[loss=0.3717, simple_loss=0.4642, pruned_loss=0.1396, over 24490.00 frames. ], tot_loss[loss=0.3334, simple_loss=0.4256, pruned_loss=0.1206, over 4429176.37 frames. ], batch size: 218, lr: 1.36e-02, grad_scale: 32.0 2026-09-23 23:42:38,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=46373.333333333336, ans=0.125 2026-09-23 23:42:39,842 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.477e+02 3.140e+02 3.597e+02 4.164e+02 8.514e+02, threshold=7.194e+02, percent-clipped=1.0 2026-09-23 23:42:46,100 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=46440.0, ans=0.0007739130434782603 2026-09-23 23:42:46,110 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=46440.0, ans=0.0 2026-09-23 23:43:02,401 INFO [train.py:1192] (0/2) Epoch 15, batch 550, loss[loss=0.4127, simple_loss=0.4979, pruned_loss=0.1637, over 24270.00 frames. ], tot_loss[loss=0.3336, simple_loss=0.426, pruned_loss=0.1206, over 4519923.04 frames. ], batch size: 257, lr: 1.36e-02, grad_scale: 32.0 2026-09-23 23:43:08,404 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=46573.333333333336, ans=0.125 2026-09-23 23:43:24,078 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=46673.333333333336, ans=0.125 2026-09-23 23:43:24,564 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=46673.333333333336, ans=0.125 2026-09-23 23:43:28,121 INFO [train.py:1192] (0/2) Epoch 15, batch 600, loss[loss=0.3415, simple_loss=0.4463, pruned_loss=0.1183, over 24353.00 frames. ], tot_loss[loss=0.3337, simple_loss=0.4264, pruned_loss=0.1205, over 4586892.78 frames. ], batch size: 234, lr: 1.35e-02, grad_scale: 32.0 2026-09-23 23:43:31,839 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.550e+02 3.078e+02 3.546e+02 4.123e+02 7.010e+02, threshold=7.092e+02, percent-clipped=0.0 2026-09-23 23:43:32,811 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=46740.0, ans=0.1 2026-09-23 23:43:44,612 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=46806.666666666664, ans=0.0 2026-09-23 23:43:52,701 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.62 vs. limit=15.0 2026-09-23 23:43:53,374 INFO [train.py:1192] (0/2) Epoch 15, batch 650, loss[loss=0.3291, simple_loss=0.4225, pruned_loss=0.1179, over 24574.00 frames. ], tot_loss[loss=0.3318, simple_loss=0.4251, pruned_loss=0.1193, over 4652067.94 frames. ], batch size: 162, lr: 1.35e-02, grad_scale: 32.0 2026-09-23 23:43:55,275 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=46873.333333333336, ans=0.125 2026-09-23 23:44:10,814 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.39 vs. limit=10.0 2026-09-23 23:44:11,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=46973.333333333336, ans=0.125 2026-09-23 23:44:19,589 INFO [train.py:1192] (0/2) Epoch 15, batch 700, loss[loss=0.3044, simple_loss=0.3975, pruned_loss=0.1057, over 24579.00 frames. ], tot_loss[loss=0.3327, simple_loss=0.4259, pruned_loss=0.1197, over 4684397.48 frames. ], batch size: 154, lr: 1.35e-02, grad_scale: 32.0 2026-09-23 23:44:20,500 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=47040.0, ans=0.125 2026-09-23 23:44:23,381 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.653e+02 3.383e+02 3.864e+02 4.776e+02 7.601e+02, threshold=7.728e+02, percent-clipped=2.0 2026-09-23 23:44:24,425 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=47073.333333333336, ans=0.2 2026-09-23 23:44:25,982 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=47073.333333333336, ans=0.2 2026-09-23 23:44:26,811 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.18 vs. limit=22.5 2026-09-23 23:44:31,295 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=47106.666666666664, ans=0.125 2026-09-23 23:44:45,983 INFO [train.py:1192] (0/2) Epoch 15, batch 750, loss[loss=0.3533, simple_loss=0.4456, pruned_loss=0.1305, over 24557.00 frames. ], tot_loss[loss=0.3325, simple_loss=0.4253, pruned_loss=0.1198, over 4711508.26 frames. ], batch size: 170, lr: 1.35e-02, grad_scale: 32.0 2026-09-23 23:44:46,548 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=47206.666666666664, ans=0.0 2026-09-23 23:44:48,703 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=47206.666666666664, ans=0.95 2026-09-23 23:44:56,640 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=47273.333333333336, ans=0.2 2026-09-23 23:45:05,207 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.42 vs. limit=12.0 2026-09-23 23:45:11,018 INFO [train.py:1192] (0/2) Epoch 15, batch 800, loss[loss=0.2951, simple_loss=0.3851, pruned_loss=0.1026, over 24525.00 frames. ], tot_loss[loss=0.3309, simple_loss=0.4241, pruned_loss=0.1188, over 4736584.73 frames. ], batch size: 137, lr: 1.35e-02, grad_scale: 32.0 2026-09-23 23:45:11,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=47373.333333333336, ans=0.125 2026-09-23 23:45:14,710 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.721e+02 3.390e+02 3.796e+02 4.598e+02 6.702e+02, threshold=7.592e+02, percent-clipped=0.0 2026-09-23 23:45:19,817 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=47406.666666666664, ans=0.125 2026-09-23 23:45:22,833 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=47440.0, ans=0.025 2026-09-23 23:45:27,636 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.97 vs. limit=12.0 2026-09-23 23:45:37,138 INFO [train.py:1192] (0/2) Epoch 15, batch 850, loss[loss=0.3643, simple_loss=0.4583, pruned_loss=0.1351, over 24591.00 frames. ], tot_loss[loss=0.3314, simple_loss=0.4244, pruned_loss=0.1192, over 4759838.72 frames. ], batch size: 198, lr: 1.34e-02, grad_scale: 32.0 2026-09-23 23:45:38,816 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=47540.0, ans=0.1 2026-09-23 23:45:40,667 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.68 vs. limit=6.0 2026-09-23 23:45:45,570 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=47573.333333333336, ans=0.0 2026-09-23 23:45:45,883 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.34 vs. limit=15.0 2026-09-23 23:45:50,662 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=47606.666666666664, ans=0.025 2026-09-23 23:45:56,610 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=47640.0, ans=0.125 2026-09-23 23:45:59,251 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=47673.333333333336, ans=10.0 2026-09-23 23:46:00,752 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=47673.333333333336, ans=0.5 2026-09-23 23:46:03,087 INFO [train.py:1192] (0/2) Epoch 15, batch 900, loss[loss=0.2896, simple_loss=0.3847, pruned_loss=0.09725, over 24555.00 frames. ], tot_loss[loss=0.3314, simple_loss=0.4246, pruned_loss=0.1191, over 4773591.49 frames. ], batch size: 137, lr: 1.34e-02, grad_scale: 32.0 2026-09-23 23:46:06,710 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.630e+02 3.321e+02 3.769e+02 4.391e+02 7.411e+02, threshold=7.539e+02, percent-clipped=0.0 2026-09-23 23:46:07,121 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.06 vs. limit=6.0 2026-09-23 23:46:07,553 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=47706.666666666664, ans=0.0004985507246376818 2026-09-23 23:46:17,047 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=47773.333333333336, ans=0.125 2026-09-23 23:46:22,805 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=47806.666666666664, ans=0.125 2026-09-23 23:46:28,313 INFO [train.py:1192] (0/2) Epoch 15, batch 950, loss[loss=0.4984, simple_loss=0.5124, pruned_loss=0.2421, over 11423.00 frames. ], tot_loss[loss=0.3327, simple_loss=0.424, pruned_loss=0.1207, over 4716883.63 frames. ], batch size: 334, lr: 1.34e-02, grad_scale: 32.0 2026-09-23 23:46:32,750 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-15.pt 2026-09-23 23:46:40,276 INFO [train.py:1192] (0/2) Epoch 16, batch 0, loss[loss=0.2748, simple_loss=0.377, pruned_loss=0.08636, over 24574.00 frames. ], tot_loss[loss=0.2748, simple_loss=0.377, pruned_loss=0.08636, over 24574.00 frames. ], batch size: 137, lr: 1.30e-02, grad_scale: 32.0 2026-09-23 23:46:40,276 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 23:46:51,434 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.0046, 3.6156, 3.9026, 3.5798], device='cuda:0') 2026-09-23 23:46:52,128 INFO [train.py:1224] (0/2) Epoch 16, validation: loss=0.2069, simple_loss=0.3218, pruned_loss=0.04601, over 2564189.00 frames. 2026-09-23 23:46:52,128 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 23:46:52,228 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=47900.0, ans=0.07 2026-09-23 23:46:57,771 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=47933.333333333336, ans=0.125 2026-09-23 23:47:06,296 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.28 vs. limit=22.5 2026-09-23 23:47:17,777 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.506e+02 3.512e+02 4.009e+02 4.590e+02 7.436e+02, threshold=8.017e+02, percent-clipped=0.0 2026-09-23 23:47:18,323 INFO [train.py:1192] (0/2) Epoch 16, batch 50, loss[loss=0.2991, simple_loss=0.3895, pruned_loss=0.1043, over 24214.00 frames. ], tot_loss[loss=0.3444, simple_loss=0.4348, pruned_loss=0.127, over 1077402.94 frames. ], batch size: 125, lr: 1.30e-02, grad_scale: 32.0 2026-09-23 23:47:23,877 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=48100.0, ans=0.125 2026-09-23 23:47:26,447 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=48100.0, ans=0.2 2026-09-23 23:47:45,059 INFO [train.py:1192] (0/2) Epoch 16, batch 100, loss[loss=0.3195, simple_loss=0.4103, pruned_loss=0.1143, over 24600.00 frames. ], tot_loss[loss=0.3465, simple_loss=0.4372, pruned_loss=0.1279, over 1906423.99 frames. ], batch size: 154, lr: 1.29e-02, grad_scale: 32.0 2026-09-23 23:47:55,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=48300.0, ans=0.125 2026-09-23 23:48:00,688 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.09 vs. limit=22.5 2026-09-23 23:48:01,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=48333.333333333336, ans=0.125 2026-09-23 23:48:04,156 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.53 vs. limit=22.5 2026-09-23 23:48:10,500 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.527e+02 3.264e+02 3.645e+02 3.998e+02 6.612e+02, threshold=7.291e+02, percent-clipped=0.0 2026-09-23 23:48:10,713 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.73 vs. limit=15.0 2026-09-23 23:48:10,889 INFO [train.py:1192] (0/2) Epoch 16, batch 150, loss[loss=0.2577, simple_loss=0.3554, pruned_loss=0.08002, over 24301.00 frames. ], tot_loss[loss=0.3362, simple_loss=0.4293, pruned_loss=0.1216, over 2559709.21 frames. ], batch size: 125, lr: 1.29e-02, grad_scale: 32.0 2026-09-23 23:48:12,665 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.79 vs. limit=22.5 2026-09-23 23:48:18,573 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=48433.333333333336, ans=0.2 2026-09-23 23:48:23,120 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=48466.666666666664, ans=0.125 2026-09-23 23:48:36,699 INFO [train.py:1192] (0/2) Epoch 16, batch 200, loss[loss=0.4274, simple_loss=0.4871, pruned_loss=0.1838, over 21193.00 frames. ], tot_loss[loss=0.3317, simple_loss=0.4257, pruned_loss=0.1188, over 3055591.77 frames. ], batch size: 333, lr: 1.29e-02, grad_scale: 32.0 2026-09-23 23:48:36,900 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.43 vs. limit=12.0 2026-09-23 23:48:44,152 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.75 vs. limit=12.0 2026-09-23 23:48:54,120 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.90 vs. limit=6.0 2026-09-23 23:48:54,615 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.40 vs. limit=22.5 2026-09-23 23:48:56,793 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.85 vs. limit=15.0 2026-09-23 23:49:02,278 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=48700.0, ans=0.125 2026-09-23 23:49:02,677 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.320e+02 3.341e+02 3.785e+02 4.483e+02 7.624e+02, threshold=7.571e+02, percent-clipped=1.0 2026-09-23 23:49:03,195 INFO [train.py:1192] (0/2) Epoch 16, batch 250, loss[loss=0.3958, simple_loss=0.4815, pruned_loss=0.1551, over 24322.00 frames. ], tot_loss[loss=0.3317, simple_loss=0.4253, pruned_loss=0.119, over 3445747.83 frames. ], batch size: 234, lr: 1.29e-02, grad_scale: 32.0 2026-09-23 23:49:05,301 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=48733.333333333336, ans=0.125 2026-09-23 23:49:06,773 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=48733.333333333336, ans=0.125 2026-09-23 23:49:09,999 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=48766.666666666664, ans=0.125 2026-09-23 23:49:15,453 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=48800.0, ans=0.0 2026-09-23 23:49:27,695 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=48866.666666666664, ans=0.125 2026-09-23 23:49:28,663 INFO [train.py:1192] (0/2) Epoch 16, batch 300, loss[loss=0.3339, simple_loss=0.4507, pruned_loss=0.1086, over 24570.00 frames. ], tot_loss[loss=0.3288, simple_loss=0.4229, pruned_loss=0.1174, over 3750683.51 frames. ], batch size: 204, lr: 1.29e-02, grad_scale: 32.0 2026-09-23 23:49:31,221 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.87 vs. limit=15.0 2026-09-23 23:49:37,253 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=48933.333333333336, ans=0.1 2026-09-23 23:49:46,882 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=49000.0, ans=0.1 2026-09-23 23:49:47,853 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=49000.0, ans=0.07 2026-09-23 23:49:48,445 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.26 vs. limit=15.0 2026-09-23 23:49:50,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=49033.333333333336, ans=0.125 2026-09-23 23:49:50,672 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.95 vs. limit=15.0 2026-09-23 23:49:53,858 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.703e+02 3.205e+02 3.601e+02 4.111e+02 5.860e+02, threshold=7.201e+02, percent-clipped=0.0 2026-09-23 23:49:54,410 INFO [train.py:1192] (0/2) Epoch 16, batch 350, loss[loss=0.2683, simple_loss=0.3692, pruned_loss=0.08373, over 24566.00 frames. ], tot_loss[loss=0.3303, simple_loss=0.4246, pruned_loss=0.118, over 3993274.12 frames. ], batch size: 137, lr: 1.28e-02, grad_scale: 32.0 2026-09-23 23:49:58,163 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=49066.666666666664, ans=0.1 2026-09-23 23:50:01,698 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=49100.0, ans=0.0 2026-09-23 23:50:08,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=49133.333333333336, ans=0.07 2026-09-23 23:50:19,956 INFO [train.py:1192] (0/2) Epoch 16, batch 400, loss[loss=0.2935, simple_loss=0.4045, pruned_loss=0.09126, over 24556.00 frames. ], tot_loss[loss=0.3287, simple_loss=0.4231, pruned_loss=0.1172, over 4176730.21 frames. ], batch size: 170, lr: 1.28e-02, grad_scale: 32.0 2026-09-23 23:50:22,784 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.80 vs. limit=15.0 2026-09-23 23:50:41,421 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=49366.666666666664, ans=0.125 2026-09-23 23:50:45,306 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.507e+02 3.240e+02 3.670e+02 4.199e+02 6.759e+02, threshold=7.340e+02, percent-clipped=0.0 2026-09-23 23:50:45,829 INFO [train.py:1192] (0/2) Epoch 16, batch 450, loss[loss=0.3468, simple_loss=0.4348, pruned_loss=0.1294, over 24605.00 frames. ], tot_loss[loss=0.3292, simple_loss=0.4234, pruned_loss=0.1175, over 4308841.62 frames. ], batch size: 175, lr: 1.28e-02, grad_scale: 32.0 2026-09-23 23:50:54,045 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.00 vs. limit=15.0 2026-09-23 23:51:03,598 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=49500.0, ans=0.125 2026-09-23 23:51:11,446 INFO [train.py:1192] (0/2) Epoch 16, batch 500, loss[loss=0.3392, simple_loss=0.4406, pruned_loss=0.1189, over 24512.00 frames. ], tot_loss[loss=0.3275, simple_loss=0.4214, pruned_loss=0.1168, over 4426852.71 frames. ], batch size: 218, lr: 1.28e-02, grad_scale: 32.0 2026-09-23 23:51:27,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=49666.666666666664, ans=0.0 2026-09-23 23:51:36,977 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.627e+02 3.089e+02 3.648e+02 4.070e+02 6.215e+02, threshold=7.297e+02, percent-clipped=0.0 2026-09-23 23:51:37,496 INFO [train.py:1192] (0/2) Epoch 16, batch 550, loss[loss=0.4057, simple_loss=0.4954, pruned_loss=0.158, over 24299.00 frames. ], tot_loss[loss=0.329, simple_loss=0.4226, pruned_loss=0.1177, over 4516935.98 frames. ], batch size: 257, lr: 1.28e-02, grad_scale: 32.0 2026-09-23 23:51:45,453 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.85 vs. limit=15.0 2026-09-23 23:51:52,917 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.75 vs. limit=15.0 2026-09-23 23:52:03,873 INFO [train.py:1192] (0/2) Epoch 16, batch 600, loss[loss=0.3675, simple_loss=0.4714, pruned_loss=0.1318, over 24425.00 frames. ], tot_loss[loss=0.3299, simple_loss=0.4239, pruned_loss=0.1179, over 4585164.30 frames. ], batch size: 235, lr: 1.28e-02, grad_scale: 32.0 2026-09-23 23:52:10,708 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=49933.333333333336, ans=0.0 2026-09-23 23:52:29,268 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.504e+02 3.407e+02 4.001e+02 4.729e+02 7.317e+02, threshold=8.001e+02, percent-clipped=1.0 2026-09-23 23:52:29,790 INFO [train.py:1192] (0/2) Epoch 16, batch 650, loss[loss=0.3585, simple_loss=0.4427, pruned_loss=0.1372, over 24554.00 frames. ], tot_loss[loss=0.3289, simple_loss=0.423, pruned_loss=0.1174, over 4650686.24 frames. ], batch size: 162, lr: 1.27e-02, grad_scale: 32.0 2026-09-23 23:52:30,392 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=50066.666666666664, ans=15.0 2026-09-23 23:52:45,925 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=50166.666666666664, ans=0.025 2026-09-23 23:52:51,530 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=50200.0, ans=0.1 2026-09-23 23:52:52,112 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.03 vs. limit=15.0 2026-09-23 23:52:55,355 INFO [train.py:1192] (0/2) Epoch 16, batch 700, loss[loss=0.2875, simple_loss=0.3896, pruned_loss=0.09277, over 24573.00 frames. ], tot_loss[loss=0.3274, simple_loss=0.4225, pruned_loss=0.1161, over 4684485.22 frames. ], batch size: 154, lr: 1.27e-02, grad_scale: 32.0 2026-09-23 23:53:01,550 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=50266.666666666664, ans=0.125 2026-09-23 23:53:02,475 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=50266.666666666664, ans=0.2 2026-09-23 23:53:21,226 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.364e+02 3.437e+02 3.892e+02 4.404e+02 6.968e+02, threshold=7.784e+02, percent-clipped=0.0 2026-09-23 23:53:21,674 INFO [train.py:1192] (0/2) Epoch 16, batch 750, loss[loss=0.3019, simple_loss=0.4056, pruned_loss=0.0991, over 24559.00 frames. ], tot_loss[loss=0.3263, simple_loss=0.4211, pruned_loss=0.1158, over 4711837.59 frames. ], batch size: 170, lr: 1.27e-02, grad_scale: 32.0 2026-09-23 23:53:25,926 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.00 vs. limit=10.0 2026-09-23 23:53:31,132 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=50433.333333333336, ans=10.0 2026-09-23 23:53:37,848 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=50500.0, ans=0.125 2026-09-23 23:53:38,392 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=50500.0, ans=0.125 2026-09-23 23:53:39,906 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.78 vs. limit=22.5 2026-09-23 23:53:41,676 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=50500.0, ans=0.1 2026-09-23 23:53:44,534 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=50533.333333333336, ans=0.125 2026-09-23 23:53:45,526 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=50533.333333333336, ans=0.2 2026-09-23 23:53:48,137 INFO [train.py:1192] (0/2) Epoch 16, batch 800, loss[loss=0.2601, simple_loss=0.3624, pruned_loss=0.07888, over 24530.00 frames. ], tot_loss[loss=0.3264, simple_loss=0.4211, pruned_loss=0.1159, over 4736429.61 frames. ], batch size: 137, lr: 1.27e-02, grad_scale: 32.0 2026-09-23 23:53:53,445 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=50600.0, ans=0.0 2026-09-23 23:53:54,908 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=50600.0, ans=0.0 2026-09-23 23:54:04,980 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=50666.666666666664, ans=0.0 2026-09-23 23:54:13,052 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=50700.0, ans=0.125 2026-09-23 23:54:13,382 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.330e+02 3.262e+02 3.651e+02 4.190e+02 5.928e+02, threshold=7.301e+02, percent-clipped=0.0 2026-09-23 23:54:13,924 INFO [train.py:1192] (0/2) Epoch 16, batch 850, loss[loss=0.3207, simple_loss=0.4209, pruned_loss=0.1102, over 24594.00 frames. ], tot_loss[loss=0.3252, simple_loss=0.4202, pruned_loss=0.1151, over 4758892.93 frames. ], batch size: 198, lr: 1.27e-02, grad_scale: 32.0 2026-09-23 23:54:35,821 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=50866.666666666664, ans=0.1 2026-09-23 23:54:39,972 INFO [train.py:1192] (0/2) Epoch 16, batch 900, loss[loss=0.2703, simple_loss=0.3737, pruned_loss=0.0834, over 24547.00 frames. ], tot_loss[loss=0.3261, simple_loss=0.421, pruned_loss=0.1156, over 4772926.33 frames. ], batch size: 137, lr: 1.27e-02, grad_scale: 32.0 2026-09-23 23:54:53,161 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=50966.666666666664, ans=0.125 2026-09-23 23:54:53,753 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=50966.666666666664, ans=15.0 2026-09-23 23:55:01,462 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.48 vs. limit=15.0 2026-09-23 23:55:04,778 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.543e+02 3.475e+02 3.921e+02 4.492e+02 7.001e+02, threshold=7.841e+02, percent-clipped=0.0 2026-09-23 23:55:05,290 INFO [train.py:1192] (0/2) Epoch 16, batch 950, loss[loss=0.5016, simple_loss=0.5062, pruned_loss=0.2485, over 11321.00 frames. ], tot_loss[loss=0.3287, simple_loss=0.4211, pruned_loss=0.1181, over 4714970.03 frames. ], batch size: 334, lr: 1.26e-02, grad_scale: 32.0 2026-09-23 23:55:09,715 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-16.pt 2026-09-23 23:55:15,795 INFO [train.py:1192] (0/2) Epoch 17, batch 0, loss[loss=0.2719, simple_loss=0.377, pruned_loss=0.0834, over 24579.00 frames. ], tot_loss[loss=0.2719, simple_loss=0.377, pruned_loss=0.0834, over 24579.00 frames. ], batch size: 137, lr: 1.23e-02, grad_scale: 32.0 2026-09-23 23:55:15,796 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-23 23:55:17,260 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.7424, 2.5948, 2.2063, 3.2406], device='cuda:0') 2026-09-23 23:55:22,102 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([1.2273, 1.2528, 1.3971, 1.1672, 1.2535, 0.9996, 1.0755, 1.0943], device='cuda:0') 2026-09-23 23:55:22,457 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.8665, 2.5273, 3.4996, 1.5362], device='cuda:0') 2026-09-23 23:55:27,425 INFO [train.py:1224] (0/2) Epoch 17, validation: loss=0.2007, simple_loss=0.3174, pruned_loss=0.04202, over 2564189.00 frames. 2026-09-23 23:55:27,429 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-23 23:55:35,186 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=51126.666666666664, ans=0.2 2026-09-23 23:55:36,757 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=51126.666666666664, ans=0.125 2026-09-23 23:55:38,142 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=51160.0, ans=0.05 2026-09-23 23:55:38,183 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:55:40,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=51160.0, ans=0.2 2026-09-23 23:55:45,726 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.56 vs. limit=22.5 2026-09-23 23:55:50,334 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=51226.666666666664, ans=0.1 2026-09-23 23:55:52,507 INFO [train.py:1192] (0/2) Epoch 17, batch 50, loss[loss=0.2619, simple_loss=0.3567, pruned_loss=0.08354, over 24232.00 frames. ], tot_loss[loss=0.3335, simple_loss=0.4275, pruned_loss=0.1198, over 1075720.72 frames. ], batch size: 125, lr: 1.22e-02, grad_scale: 32.0 2026-09-23 23:55:52,589 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=51260.0, ans=0.2 2026-09-23 23:55:56,052 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:56:08,112 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=51360.0, ans=0.0 2026-09-23 23:56:13,092 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:56:13,383 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.772e+02 3.266e+02 3.742e+02 4.253e+02 6.443e+02, threshold=7.483e+02, percent-clipped=0.0 2026-09-23 23:56:14,772 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=51393.333333333336, ans=0.1 2026-09-23 23:56:17,526 INFO [train.py:1192] (0/2) Epoch 17, batch 100, loss[loss=0.2911, simple_loss=0.3946, pruned_loss=0.09382, over 24583.00 frames. ], tot_loss[loss=0.3366, simple_loss=0.4313, pruned_loss=0.121, over 1903569.09 frames. ], batch size: 154, lr: 1.22e-02, grad_scale: 64.0 2026-09-23 23:56:22,739 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=14.58 vs. limit=22.5 2026-09-23 23:56:27,941 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=51493.333333333336, ans=0.125 2026-09-23 23:56:28,912 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=51493.333333333336, ans=0.1 2026-09-23 23:56:43,128 INFO [train.py:1192] (0/2) Epoch 17, batch 150, loss[loss=0.287, simple_loss=0.3764, pruned_loss=0.09878, over 24277.00 frames. ], tot_loss[loss=0.3321, simple_loss=0.4266, pruned_loss=0.1188, over 2557361.36 frames. ], batch size: 125, lr: 1.22e-02, grad_scale: 64.0 2026-09-23 23:57:01,806 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.61 vs. limit=15.0 2026-09-23 23:57:04,108 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.461e+02 3.307e+02 3.706e+02 4.440e+02 7.679e+02, threshold=7.412e+02, percent-clipped=1.0 2026-09-23 23:57:09,035 INFO [train.py:1192] (0/2) Epoch 17, batch 200, loss[loss=0.3941, simple_loss=0.4641, pruned_loss=0.1621, over 21060.00 frames. ], tot_loss[loss=0.3295, simple_loss=0.4242, pruned_loss=0.1174, over 3054087.29 frames. ], batch size: 333, lr: 1.22e-02, grad_scale: 32.0 2026-09-23 23:57:17,809 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.76 vs. limit=12.0 2026-09-23 23:57:20,406 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=51826.666666666664, ans=0.125 2026-09-23 23:57:31,124 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=51893.333333333336, ans=0.025 2026-09-23 23:57:33,795 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=51893.333333333336, ans=0.125 2026-09-23 23:57:34,610 INFO [train.py:1192] (0/2) Epoch 17, batch 250, loss[loss=0.3601, simple_loss=0.4582, pruned_loss=0.1311, over 24314.00 frames. ], tot_loss[loss=0.3275, simple_loss=0.4225, pruned_loss=0.1163, over 3442914.63 frames. ], batch size: 234, lr: 1.22e-02, grad_scale: 32.0 2026-09-23 23:57:35,716 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=51926.666666666664, ans=0.0 2026-09-23 23:57:39,301 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=51960.0, ans=0.1 2026-09-23 23:57:39,827 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=51960.0, ans=0.125 2026-09-23 23:57:47,480 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=51993.333333333336, ans=0.1 2026-09-23 23:57:47,940 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=51993.333333333336, ans=0.125 2026-09-23 23:57:51,229 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=52026.666666666664, ans=0.125 2026-09-23 23:57:52,327 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=52026.666666666664, ans=0.0 2026-09-23 23:57:53,315 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=52026.666666666664, ans=0.125 2026-09-23 23:57:53,775 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=52026.666666666664, ans=0.0 2026-09-23 23:57:56,534 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.479e+02 3.375e+02 3.794e+02 4.631e+02 7.260e+02, threshold=7.588e+02, percent-clipped=0.0 2026-09-23 23:58:00,329 INFO [train.py:1192] (0/2) Epoch 17, batch 300, loss[loss=0.3681, simple_loss=0.4627, pruned_loss=0.1368, over 24520.00 frames. ], tot_loss[loss=0.3274, simple_loss=0.4218, pruned_loss=0.1165, over 3749263.26 frames. ], batch size: 204, lr: 1.22e-02, grad_scale: 16.0 2026-09-23 23:58:08,068 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.46 vs. limit=15.0 2026-09-23 23:58:09,724 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=52126.666666666664, ans=0.0 2026-09-23 23:58:14,738 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-23 23:58:23,980 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=52226.666666666664, ans=0.125 2026-09-23 23:58:25,632 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.33 vs. limit=15.0 2026-09-23 23:58:25,921 INFO [train.py:1192] (0/2) Epoch 17, batch 350, loss[loss=0.2634, simple_loss=0.3658, pruned_loss=0.08045, over 24590.00 frames. ], tot_loss[loss=0.3267, simple_loss=0.422, pruned_loss=0.1157, over 3992908.41 frames. ], batch size: 137, lr: 1.21e-02, grad_scale: 16.0 2026-09-23 23:58:28,003 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=52260.0, ans=0.05 2026-09-23 23:58:37,347 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=52326.666666666664, ans=0.2 2026-09-23 23:58:47,571 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.561e+02 3.171e+02 3.579e+02 4.200e+02 7.417e+02, threshold=7.158e+02, percent-clipped=0.0 2026-09-23 23:58:50,855 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=52426.666666666664, ans=0.125 2026-09-23 23:58:51,345 INFO [train.py:1192] (0/2) Epoch 17, batch 400, loss[loss=0.3528, simple_loss=0.4431, pruned_loss=0.1313, over 24573.00 frames. ], tot_loss[loss=0.325, simple_loss=0.4206, pruned_loss=0.1147, over 4180907.70 frames. ], batch size: 170, lr: 1.21e-02, grad_scale: 32.0 2026-09-23 23:58:57,962 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=52460.0, ans=0.0 2026-09-23 23:59:00,771 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=52460.0, ans=0.125 2026-09-23 23:59:04,859 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.13 vs. limit=15.0 2026-09-23 23:59:16,997 INFO [train.py:1192] (0/2) Epoch 17, batch 450, loss[loss=0.3283, simple_loss=0.4283, pruned_loss=0.1142, over 24610.00 frames. ], tot_loss[loss=0.3248, simple_loss=0.4205, pruned_loss=0.1146, over 4314209.37 frames. ], batch size: 175, lr: 1.21e-02, grad_scale: 32.0 2026-09-23 23:59:21,940 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=52626.666666666664, ans=0.125 2026-09-23 23:59:22,942 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=52626.666666666664, ans=0.1 2026-09-23 23:59:22,967 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=52626.666666666664, ans=0.125 2026-09-23 23:59:27,287 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=52660.0, ans=0.2 2026-09-23 23:59:32,042 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=52693.333333333336, ans=0.0 2026-09-23 23:59:38,265 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.585e+02 3.216e+02 3.919e+02 4.471e+02 7.457e+02, threshold=7.837e+02, percent-clipped=1.0 2026-09-23 23:59:41,862 INFO [train.py:1192] (0/2) Epoch 17, batch 500, loss[loss=0.357, simple_loss=0.4599, pruned_loss=0.127, over 24528.00 frames. ], tot_loss[loss=0.3223, simple_loss=0.4182, pruned_loss=0.1132, over 4430810.35 frames. ], batch size: 218, lr: 1.21e-02, grad_scale: 32.0 2026-09-23 23:59:42,041 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.79 vs. limit=15.0 2026-09-23 23:59:47,219 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=52793.333333333336, ans=0.0 2026-09-23 23:59:51,407 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=52826.666666666664, ans=0.2 2026-09-23 23:59:55,686 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.75 vs. limit=6.0 2026-09-23 23:59:59,431 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=52860.0, ans=0.125 2026-09-24 00:00:07,638 INFO [train.py:1192] (0/2) Epoch 17, batch 550, loss[loss=0.3439, simple_loss=0.4491, pruned_loss=0.1193, over 24202.00 frames. ], tot_loss[loss=0.3227, simple_loss=0.4188, pruned_loss=0.1133, over 4519896.55 frames. ], batch size: 257, lr: 1.21e-02, grad_scale: 32.0 2026-09-24 00:00:10,834 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=52926.666666666664, ans=0.1 2026-09-24 00:00:13,235 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=52960.0, ans=0.0 2026-09-24 00:00:20,221 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=52993.333333333336, ans=0.0 2026-09-24 00:00:24,817 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=53026.666666666664, ans=0.0 2026-09-24 00:00:29,552 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.579e+02 3.286e+02 3.817e+02 4.388e+02 6.403e+02, threshold=7.633e+02, percent-clipped=0.0 2026-09-24 00:00:31,543 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=53060.0, ans=0.05 2026-09-24 00:00:33,275 INFO [train.py:1192] (0/2) Epoch 17, batch 600, loss[loss=0.3414, simple_loss=0.4462, pruned_loss=0.1182, over 24330.00 frames. ], tot_loss[loss=0.3231, simple_loss=0.4194, pruned_loss=0.1134, over 4585021.82 frames. ], batch size: 234, lr: 1.21e-02, grad_scale: 32.0 2026-09-24 00:00:46,544 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=53160.0, ans=0.0 2026-09-24 00:00:58,508 INFO [train.py:1192] (0/2) Epoch 17, batch 650, loss[loss=0.3353, simple_loss=0.4275, pruned_loss=0.1216, over 24570.00 frames. ], tot_loss[loss=0.3219, simple_loss=0.4183, pruned_loss=0.1127, over 4650509.11 frames. ], batch size: 162, lr: 1.21e-02, grad_scale: 32.0 2026-09-24 00:01:02,285 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.04 vs. limit=15.0 2026-09-24 00:01:04,923 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=53293.333333333336, ans=0.2 2026-09-24 00:01:09,180 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-16000.pt 2026-09-24 00:01:16,197 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=53360.0, ans=0.04949747468305833 2026-09-24 00:01:20,177 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.620e+02 3.299e+02 3.782e+02 4.517e+02 7.737e+02, threshold=7.565e+02, percent-clipped=1.0 2026-09-24 00:01:20,838 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=53393.333333333336, ans=0.0 2026-09-24 00:01:22,310 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.32 vs. limit=22.5 2026-09-24 00:01:23,593 INFO [train.py:1192] (0/2) Epoch 17, batch 700, loss[loss=0.3317, simple_loss=0.417, pruned_loss=0.1232, over 24565.00 frames. ], tot_loss[loss=0.3219, simple_loss=0.4187, pruned_loss=0.1126, over 4682870.08 frames. ], batch size: 154, lr: 1.20e-02, grad_scale: 32.0 2026-09-24 00:01:29,465 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=53460.0, ans=0.1 2026-09-24 00:01:32,473 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=53460.0, ans=0.0 2026-09-24 00:01:37,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=53493.333333333336, ans=0.025 2026-09-24 00:01:44,064 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=53526.666666666664, ans=0.0 2026-09-24 00:01:48,523 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=53560.0, ans=0.1 2026-09-24 00:01:49,880 INFO [train.py:1192] (0/2) Epoch 17, batch 750, loss[loss=0.2972, simple_loss=0.408, pruned_loss=0.09317, over 24564.00 frames. ], tot_loss[loss=0.3213, simple_loss=0.4179, pruned_loss=0.1123, over 4710467.70 frames. ], batch size: 170, lr: 1.20e-02, grad_scale: 32.0 2026-09-24 00:01:52,060 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=53593.333333333336, ans=0.2 2026-09-24 00:01:53,932 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=53593.333333333336, ans=0.125 2026-09-24 00:02:03,229 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.99 vs. limit=15.0 2026-09-24 00:02:06,596 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=53693.333333333336, ans=0.5 2026-09-24 00:02:08,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=53693.333333333336, ans=0.125 2026-09-24 00:02:09,439 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=53693.333333333336, ans=0.125 2026-09-24 00:02:11,792 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.751e+02 3.366e+02 3.846e+02 4.642e+02 7.031e+02, threshold=7.693e+02, percent-clipped=0.0 2026-09-24 00:02:15,308 INFO [train.py:1192] (0/2) Epoch 17, batch 800, loss[loss=0.2786, simple_loss=0.3737, pruned_loss=0.09173, over 24545.00 frames. ], tot_loss[loss=0.3219, simple_loss=0.4181, pruned_loss=0.1128, over 4739728.90 frames. ], batch size: 137, lr: 1.20e-02, grad_scale: 32.0 2026-09-24 00:02:16,434 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=53760.0, ans=0.0 2026-09-24 00:02:21,302 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=53793.333333333336, ans=0.0 2026-09-24 00:02:32,587 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=53860.0, ans=0.125 2026-09-24 00:02:36,087 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=53893.333333333336, ans=0.1 2026-09-24 00:02:41,006 INFO [train.py:1192] (0/2) Epoch 17, batch 850, loss[loss=0.3516, simple_loss=0.4548, pruned_loss=0.1241, over 24547.00 frames. ], tot_loss[loss=0.3224, simple_loss=0.4185, pruned_loss=0.1131, over 4762035.84 frames. ], batch size: 204, lr: 1.20e-02, grad_scale: 32.0 2026-09-24 00:02:49,996 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=53960.0, ans=0.125 2026-09-24 00:02:56,347 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=54026.666666666664, ans=0.125 2026-09-24 00:03:03,068 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=54060.0, ans=0.025 2026-09-24 00:03:03,503 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.516e+02 3.169e+02 3.598e+02 4.008e+02 6.242e+02, threshold=7.196e+02, percent-clipped=0.0 2026-09-24 00:03:04,150 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=54060.0, ans=0.025 2026-09-24 00:03:07,091 INFO [train.py:1192] (0/2) Epoch 17, batch 900, loss[loss=0.2958, simple_loss=0.3872, pruned_loss=0.1022, over 24565.00 frames. ], tot_loss[loss=0.3237, simple_loss=0.4196, pruned_loss=0.1139, over 4774802.56 frames. ], batch size: 137, lr: 1.20e-02, grad_scale: 32.0 2026-09-24 00:03:20,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=54160.0, ans=0.1 2026-09-24 00:03:27,710 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=54226.666666666664, ans=0.04949747468305833 2026-09-24 00:03:31,696 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=54226.666666666664, ans=0.05 2026-09-24 00:03:32,562 INFO [train.py:1192] (0/2) Epoch 17, batch 950, loss[loss=0.4747, simple_loss=0.4907, pruned_loss=0.2294, over 11556.00 frames. ], tot_loss[loss=0.3251, simple_loss=0.4191, pruned_loss=0.1156, over 4710584.08 frames. ], batch size: 333, lr: 1.20e-02, grad_scale: 32.0 2026-09-24 00:03:33,347 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=54260.0, ans=0.0 2026-09-24 00:03:37,080 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-17.pt 2026-09-24 00:03:43,754 INFO [train.py:1192] (0/2) Epoch 18, batch 0, loss[loss=0.2915, simple_loss=0.3933, pruned_loss=0.09481, over 24556.00 frames. ], tot_loss[loss=0.2915, simple_loss=0.3933, pruned_loss=0.09481, over 24556.00 frames. ], batch size: 137, lr: 1.16e-02, grad_scale: 32.0 2026-09-24 00:03:43,754 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 00:03:49,500 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.0078, 2.5470, 2.3832, 2.0295], device='cuda:0') 2026-09-24 00:03:55,532 INFO [train.py:1224] (0/2) Epoch 18, validation: loss=0.1978, simple_loss=0.314, pruned_loss=0.04077, over 2564189.00 frames. 2026-09-24 00:03:55,533 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 00:04:02,620 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=54320.0, ans=0.05 2026-09-24 00:04:12,903 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:04:13,260 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=54386.666666666664, ans=0.125 2026-09-24 00:04:13,550 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.645e+02 3.595e+02 4.226e+02 5.155e+02 1.027e+03, threshold=8.453e+02, percent-clipped=2.0 2026-09-24 00:04:21,178 INFO [train.py:1192] (0/2) Epoch 18, batch 50, loss[loss=0.3157, simple_loss=0.3967, pruned_loss=0.1174, over 24251.00 frames. ], tot_loss[loss=0.3298, simple_loss=0.4254, pruned_loss=0.1171, over 1076720.56 frames. ], batch size: 125, lr: 1.16e-02, grad_scale: 32.0 2026-09-24 00:04:44,532 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.16 vs. limit=15.0 2026-09-24 00:04:46,231 INFO [train.py:1192] (0/2) Epoch 18, batch 100, loss[loss=0.3208, simple_loss=0.4131, pruned_loss=0.1142, over 24590.00 frames. ], tot_loss[loss=0.3333, simple_loss=0.429, pruned_loss=0.1188, over 1904075.55 frames. ], batch size: 154, lr: 1.16e-02, grad_scale: 32.0 2026-09-24 00:04:50,105 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=54620.0, ans=0.0 2026-09-24 00:04:56,736 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.23 vs. limit=12.0 2026-09-24 00:04:59,326 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.42 vs. limit=15.0 2026-09-24 00:05:00,097 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=54686.666666666664, ans=0.125 2026-09-24 00:05:02,893 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=54720.0, ans=0.2 2026-09-24 00:05:04,427 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.666e+02 3.279e+02 3.662e+02 4.041e+02 5.656e+02, threshold=7.323e+02, percent-clipped=0.0 2026-09-24 00:05:07,543 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.19 vs. limit=15.0 2026-09-24 00:05:11,796 INFO [train.py:1192] (0/2) Epoch 18, batch 150, loss[loss=0.2786, simple_loss=0.3657, pruned_loss=0.09578, over 24237.00 frames. ], tot_loss[loss=0.3261, simple_loss=0.4225, pruned_loss=0.1149, over 2558230.85 frames. ], batch size: 125, lr: 1.16e-02, grad_scale: 32.0 2026-09-24 00:05:20,874 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=54820.0, ans=0.025 2026-09-24 00:05:27,043 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.31 vs. limit=15.0 2026-09-24 00:05:32,960 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=54920.0, ans=0.0 2026-09-24 00:05:37,759 INFO [train.py:1192] (0/2) Epoch 18, batch 200, loss[loss=0.3718, simple_loss=0.452, pruned_loss=0.1458, over 21175.00 frames. ], tot_loss[loss=0.324, simple_loss=0.4206, pruned_loss=0.1137, over 3055629.31 frames. ], batch size: 333, lr: 1.16e-02, grad_scale: 32.0 2026-09-24 00:05:45,884 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=54986.666666666664, ans=0.015 2026-09-24 00:05:48,295 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=55020.0, ans=0.125 2026-09-24 00:05:52,524 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=55053.333333333336, ans=0.07 2026-09-24 00:05:54,501 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=55053.333333333336, ans=0.09899494936611666 2026-09-24 00:05:55,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=55053.333333333336, ans=0.125 2026-09-24 00:05:56,031 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.514e+02 3.426e+02 3.972e+02 4.847e+02 1.093e+03, threshold=7.944e+02, percent-clipped=3.0 2026-09-24 00:05:59,448 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=55086.666666666664, ans=0.125 2026-09-24 00:06:03,498 INFO [train.py:1192] (0/2) Epoch 18, batch 250, loss[loss=0.3377, simple_loss=0.444, pruned_loss=0.1157, over 24325.00 frames. ], tot_loss[loss=0.3242, simple_loss=0.4204, pruned_loss=0.114, over 3444264.15 frames. ], batch size: 234, lr: 1.15e-02, grad_scale: 32.0 2026-09-24 00:06:10,767 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=55153.333333333336, ans=0.125 2026-09-24 00:06:18,873 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=55220.0, ans=0.125 2026-09-24 00:06:28,931 INFO [train.py:1192] (0/2) Epoch 18, batch 300, loss[loss=0.3302, simple_loss=0.4341, pruned_loss=0.1131, over 24529.00 frames. ], tot_loss[loss=0.3212, simple_loss=0.4179, pruned_loss=0.1123, over 3748893.78 frames. ], batch size: 204, lr: 1.15e-02, grad_scale: 32.0 2026-09-24 00:06:29,047 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=55286.666666666664, ans=0.2 2026-09-24 00:06:30,716 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.51 vs. limit=15.0 2026-09-24 00:06:31,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=55286.666666666664, ans=0.0 2026-09-24 00:06:32,925 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:06:35,405 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=55320.0, ans=0.0 2026-09-24 00:06:39,358 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=55353.333333333336, ans=0.0 2026-09-24 00:06:47,192 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.468e+02 3.228e+02 3.688e+02 4.089e+02 5.893e+02, threshold=7.377e+02, percent-clipped=0.0 2026-09-24 00:06:48,406 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.71 vs. limit=6.0 2026-09-24 00:06:49,364 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=55420.0, ans=0.2 2026-09-24 00:06:51,234 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=55420.0, ans=0.0 2026-09-24 00:06:54,148 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.28 vs. limit=15.0 2026-09-24 00:06:54,406 INFO [train.py:1192] (0/2) Epoch 18, batch 350, loss[loss=0.2618, simple_loss=0.3609, pruned_loss=0.08134, over 24531.00 frames. ], tot_loss[loss=0.3209, simple_loss=0.4181, pruned_loss=0.1118, over 3992418.42 frames. ], batch size: 137, lr: 1.15e-02, grad_scale: 32.0 2026-09-24 00:07:03,459 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=55486.666666666664, ans=0.0 2026-09-24 00:07:04,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=55486.666666666664, ans=0.0 2026-09-24 00:07:19,801 INFO [train.py:1192] (0/2) Epoch 18, batch 400, loss[loss=0.317, simple_loss=0.4105, pruned_loss=0.1118, over 24565.00 frames. ], tot_loss[loss=0.3196, simple_loss=0.4168, pruned_loss=0.1112, over 4178304.89 frames. ], batch size: 170, lr: 1.15e-02, grad_scale: 32.0 2026-09-24 00:07:26,719 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.25 vs. limit=12.0 2026-09-24 00:07:27,035 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=55653.333333333336, ans=0.125 2026-09-24 00:07:27,054 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=55653.333333333336, ans=0.1 2026-09-24 00:07:27,157 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.11 vs. limit=15.0 2026-09-24 00:07:29,294 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=55653.333333333336, ans=0.1 2026-09-24 00:07:36,573 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=55720.0, ans=0.2 2026-09-24 00:07:38,706 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.576e+02 3.216e+02 3.635e+02 4.179e+02 6.381e+02, threshold=7.269e+02, percent-clipped=0.0 2026-09-24 00:07:43,580 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=55753.333333333336, ans=0.125 2026-09-24 00:07:45,999 INFO [train.py:1192] (0/2) Epoch 18, batch 450, loss[loss=0.3412, simple_loss=0.4407, pruned_loss=0.1208, over 24651.00 frames. ], tot_loss[loss=0.3215, simple_loss=0.4183, pruned_loss=0.1124, over 4312234.98 frames. ], batch size: 175, lr: 1.15e-02, grad_scale: 32.0 2026-09-24 00:07:50,199 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.04 vs. limit=15.0 2026-09-24 00:08:00,341 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=55853.333333333336, ans=0.125 2026-09-24 00:08:02,405 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=55886.666666666664, ans=0.0 2026-09-24 00:08:07,139 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.36 vs. limit=15.0 2026-09-24 00:08:10,723 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.69 vs. limit=15.0 2026-09-24 00:08:11,418 INFO [train.py:1192] (0/2) Epoch 18, batch 500, loss[loss=0.3501, simple_loss=0.4425, pruned_loss=0.1288, over 24510.00 frames. ], tot_loss[loss=0.3204, simple_loss=0.417, pruned_loss=0.1119, over 4429697.57 frames. ], batch size: 218, lr: 1.15e-02, grad_scale: 32.0 2026-09-24 00:08:11,965 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=55953.333333333336, ans=0.1 2026-09-24 00:08:12,051 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.55 vs. limit=15.0 2026-09-24 00:08:18,340 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=55986.666666666664, ans=0.0 2026-09-24 00:08:26,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=56053.333333333336, ans=0.125 2026-09-24 00:08:29,843 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.316e+02 3.128e+02 3.590e+02 4.213e+02 6.818e+02, threshold=7.180e+02, percent-clipped=0.0 2026-09-24 00:08:37,075 INFO [train.py:1192] (0/2) Epoch 18, batch 550, loss[loss=0.3418, simple_loss=0.4419, pruned_loss=0.1209, over 24249.00 frames. ], tot_loss[loss=0.32, simple_loss=0.417, pruned_loss=0.1115, over 4519418.36 frames. ], batch size: 257, lr: 1.15e-02, grad_scale: 32.0 2026-09-24 00:08:38,193 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=56120.0, ans=0.1 2026-09-24 00:08:38,394 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.59 vs. limit=15.0 2026-09-24 00:08:43,386 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=15.07 vs. limit=15.0 2026-09-24 00:08:44,071 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=23.23 vs. limit=22.5 2026-09-24 00:08:48,235 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.21 vs. limit=22.5 2026-09-24 00:08:51,430 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=56186.666666666664, ans=0.1 2026-09-24 00:08:51,442 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=56186.666666666664, ans=0.125 2026-09-24 00:08:53,355 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.56 vs. limit=15.0 2026-09-24 00:08:53,633 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.31 vs. limit=22.5 2026-09-24 00:08:58,295 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=56253.333333333336, ans=0.2 2026-09-24 00:08:58,834 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=56253.333333333336, ans=0.125 2026-09-24 00:08:59,919 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=2.70 vs. limit=15.0 2026-09-24 00:09:02,616 INFO [train.py:1192] (0/2) Epoch 18, batch 600, loss[loss=0.3409, simple_loss=0.4481, pruned_loss=0.1168, over 24341.00 frames. ], tot_loss[loss=0.3199, simple_loss=0.4173, pruned_loss=0.1113, over 4585328.79 frames. ], batch size: 234, lr: 1.14e-02, grad_scale: 32.0 2026-09-24 00:09:11,739 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.48 vs. limit=15.0 2026-09-24 00:09:13,018 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.28 vs. limit=22.5 2026-09-24 00:09:14,971 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=2.83 vs. limit=15.0 2026-09-24 00:09:20,507 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.559e+02 3.331e+02 3.759e+02 4.324e+02 6.098e+02, threshold=7.519e+02, percent-clipped=0.0 2026-09-24 00:09:28,006 INFO [train.py:1192] (0/2) Epoch 18, batch 650, loss[loss=0.3233, simple_loss=0.4211, pruned_loss=0.1128, over 24556.00 frames. ], tot_loss[loss=0.3182, simple_loss=0.4162, pruned_loss=0.1102, over 4650672.27 frames. ], batch size: 162, lr: 1.14e-02, grad_scale: 32.0 2026-09-24 00:09:30,452 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=56453.333333333336, ans=0.125 2026-09-24 00:09:38,642 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=56520.0, ans=15.0 2026-09-24 00:09:41,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=56520.0, ans=0.125 2026-09-24 00:09:49,049 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=56586.666666666664, ans=0.04949747468305833 2026-09-24 00:09:53,162 INFO [train.py:1192] (0/2) Epoch 18, batch 700, loss[loss=0.2946, simple_loss=0.3964, pruned_loss=0.09634, over 24585.00 frames. ], tot_loss[loss=0.3176, simple_loss=0.4161, pruned_loss=0.1096, over 4684148.74 frames. ], batch size: 154, lr: 1.14e-02, grad_scale: 32.0 2026-09-24 00:09:55,953 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=56620.0, ans=0.125 2026-09-24 00:09:57,321 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=56620.0, ans=0.0 2026-09-24 00:10:00,150 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.71 vs. limit=22.5 2026-09-24 00:10:00,985 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=56653.333333333336, ans=0.025 2026-09-24 00:10:06,407 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=56686.666666666664, ans=0.1 2026-09-24 00:10:11,061 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.563e+02 3.294e+02 3.823e+02 4.482e+02 7.746e+02, threshold=7.647e+02, percent-clipped=1.0 2026-09-24 00:10:18,467 INFO [train.py:1192] (0/2) Epoch 18, batch 750, loss[loss=0.3045, simple_loss=0.4146, pruned_loss=0.09722, over 24545.00 frames. ], tot_loss[loss=0.3164, simple_loss=0.4146, pruned_loss=0.1091, over 4711624.70 frames. ], batch size: 170, lr: 1.14e-02, grad_scale: 32.0 2026-09-24 00:10:22,710 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=56786.666666666664, ans=0.05 2026-09-24 00:10:44,026 INFO [train.py:1192] (0/2) Epoch 18, batch 800, loss[loss=0.2557, simple_loss=0.3585, pruned_loss=0.07645, over 24571.00 frames. ], tot_loss[loss=0.3168, simple_loss=0.4148, pruned_loss=0.1094, over 4736931.97 frames. ], batch size: 137, lr: 1.14e-02, grad_scale: 32.0 2026-09-24 00:10:49,185 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.78 vs. limit=10.0 2026-09-24 00:10:58,879 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.94 vs. limit=15.0 2026-09-24 00:11:03,091 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.619e+02 3.257e+02 3.850e+02 4.421e+02 6.931e+02, threshold=7.700e+02, percent-clipped=0.0 2026-09-24 00:11:04,124 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=57053.333333333336, ans=0.035 2026-09-24 00:11:05,508 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=13.72 vs. limit=15.0 2026-09-24 00:11:10,035 INFO [train.py:1192] (0/2) Epoch 18, batch 850, loss[loss=0.3639, simple_loss=0.4554, pruned_loss=0.1362, over 24578.00 frames. ], tot_loss[loss=0.3176, simple_loss=0.4151, pruned_loss=0.11, over 4759380.12 frames. ], batch size: 198, lr: 1.14e-02, grad_scale: 32.0 2026-09-24 00:11:11,147 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=57120.0, ans=0.0 2026-09-24 00:11:14,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=57120.0, ans=0.125 2026-09-24 00:11:35,700 INFO [train.py:1192] (0/2) Epoch 18, batch 900, loss[loss=0.2593, simple_loss=0.3629, pruned_loss=0.07785, over 24581.00 frames. ], tot_loss[loss=0.3177, simple_loss=0.4154, pruned_loss=0.11, over 4773686.62 frames. ], batch size: 137, lr: 1.14e-02, grad_scale: 32.0 2026-09-24 00:11:40,092 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.51 vs. limit=15.0 2026-09-24 00:11:41,616 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=57320.0, ans=0.125 2026-09-24 00:11:48,207 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.76 vs. limit=15.0 2026-09-24 00:11:54,002 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.493e+02 3.195e+02 3.716e+02 4.460e+02 7.730e+02, threshold=7.431e+02, percent-clipped=0.0 2026-09-24 00:12:01,095 INFO [train.py:1192] (0/2) Epoch 18, batch 950, loss[loss=0.4476, simple_loss=0.4657, pruned_loss=0.2147, over 11056.00 frames. ], tot_loss[loss=0.3198, simple_loss=0.4151, pruned_loss=0.1122, over 4710309.33 frames. ], batch size: 334, lr: 1.13e-02, grad_scale: 32.0 2026-09-24 00:12:04,323 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=19.90 vs. limit=15.0 2026-09-24 00:12:05,485 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-18.pt 2026-09-24 00:12:12,683 INFO [train.py:1192] (0/2) Epoch 19, batch 0, loss[loss=0.2982, simple_loss=0.3947, pruned_loss=0.1009, over 24575.00 frames. ], tot_loss[loss=0.2982, simple_loss=0.3947, pruned_loss=0.1009, over 24575.00 frames. ], batch size: 137, lr: 1.10e-02, grad_scale: 32.0 2026-09-24 00:12:12,683 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 00:12:24,365 INFO [train.py:1224] (0/2) Epoch 19, validation: loss=0.1973, simple_loss=0.3132, pruned_loss=0.04069, over 2564189.00 frames. 2026-09-24 00:12:24,365 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 00:12:28,215 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=57480.0, ans=0.125 2026-09-24 00:12:31,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=57513.333333333336, ans=0.0 2026-09-24 00:12:35,055 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.97 vs. limit=15.0 2026-09-24 00:12:36,051 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=57546.666666666664, ans=0.2 2026-09-24 00:12:46,614 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=57613.333333333336, ans=0.2 2026-09-24 00:12:50,323 INFO [train.py:1192] (0/2) Epoch 19, batch 50, loss[loss=0.2509, simple_loss=0.351, pruned_loss=0.07537, over 24246.00 frames. ], tot_loss[loss=0.3294, simple_loss=0.4248, pruned_loss=0.117, over 1075107.94 frames. ], batch size: 125, lr: 1.10e-02, grad_scale: 32.0 2026-09-24 00:12:51,043 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.39 vs. limit=15.0 2026-09-24 00:12:52,142 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.52 vs. limit=12.0 2026-09-24 00:12:58,667 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=57680.0, ans=0.04949747468305833 2026-09-24 00:13:01,761 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=57713.333333333336, ans=0.125 2026-09-24 00:13:02,222 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=57713.333333333336, ans=0.0 2026-09-24 00:13:04,398 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.526e+02 3.413e+02 3.881e+02 4.761e+02 8.226e+02, threshold=7.762e+02, percent-clipped=3.0 2026-09-24 00:13:07,586 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=57746.666666666664, ans=0.1 2026-09-24 00:13:15,640 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=57813.333333333336, ans=10.0 2026-09-24 00:13:16,074 INFO [train.py:1192] (0/2) Epoch 19, batch 100, loss[loss=0.3103, simple_loss=0.4077, pruned_loss=0.1065, over 24615.00 frames. ], tot_loss[loss=0.3308, simple_loss=0.4272, pruned_loss=0.1172, over 1903336.19 frames. ], batch size: 154, lr: 1.10e-02, grad_scale: 32.0 2026-09-24 00:13:16,743 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=57813.333333333336, ans=0.1 2026-09-24 00:13:27,495 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.72 vs. limit=6.0 2026-09-24 00:13:36,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=57946.666666666664, ans=0.025 2026-09-24 00:13:39,991 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=57946.666666666664, ans=0.0 2026-09-24 00:13:41,319 INFO [train.py:1192] (0/2) Epoch 19, batch 150, loss[loss=0.2558, simple_loss=0.3501, pruned_loss=0.08073, over 24268.00 frames. ], tot_loss[loss=0.3243, simple_loss=0.4216, pruned_loss=0.1135, over 2557113.31 frames. ], batch size: 125, lr: 1.10e-02, grad_scale: 32.0 2026-09-24 00:13:43,283 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=57980.0, ans=0.2 2026-09-24 00:13:55,669 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.658e+02 3.215e+02 3.678e+02 4.342e+02 6.275e+02, threshold=7.355e+02, percent-clipped=0.0 2026-09-24 00:14:06,875 INFO [train.py:1192] (0/2) Epoch 19, batch 200, loss[loss=0.4114, simple_loss=0.4793, pruned_loss=0.1717, over 21192.00 frames. ], tot_loss[loss=0.3228, simple_loss=0.42, pruned_loss=0.1128, over 3054467.93 frames. ], batch size: 333, lr: 1.10e-02, grad_scale: 32.0 2026-09-24 00:14:17,202 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=58213.333333333336, ans=0.0 2026-09-24 00:14:26,586 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:14:26,641 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.06 vs. limit=15.0 2026-09-24 00:14:27,337 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=58280.0, ans=0.125 2026-09-24 00:14:32,156 INFO [train.py:1192] (0/2) Epoch 19, batch 250, loss[loss=0.3514, simple_loss=0.4543, pruned_loss=0.1242, over 24292.00 frames. ], tot_loss[loss=0.3205, simple_loss=0.4181, pruned_loss=0.1114, over 3444870.67 frames. ], batch size: 234, lr: 1.10e-02, grad_scale: 32.0 2026-09-24 00:14:36,311 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=58313.333333333336, ans=0.125 2026-09-24 00:14:40,896 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=58346.666666666664, ans=0.125 2026-09-24 00:14:46,446 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.462e+02 3.293e+02 3.643e+02 4.390e+02 7.831e+02, threshold=7.286e+02, percent-clipped=2.0 2026-09-24 00:14:52,192 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=58446.666666666664, ans=0.125 2026-09-24 00:14:57,671 INFO [train.py:1192] (0/2) Epoch 19, batch 300, loss[loss=0.3584, simple_loss=0.4576, pruned_loss=0.1296, over 24569.00 frames. ], tot_loss[loss=0.3191, simple_loss=0.4167, pruned_loss=0.1108, over 3750294.24 frames. ], batch size: 204, lr: 1.10e-02, grad_scale: 32.0 2026-09-24 00:15:10,521 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=58546.666666666664, ans=0.125 2026-09-24 00:15:23,421 INFO [train.py:1192] (0/2) Epoch 19, batch 350, loss[loss=0.2636, simple_loss=0.3608, pruned_loss=0.0832, over 24585.00 frames. ], tot_loss[loss=0.3193, simple_loss=0.4172, pruned_loss=0.1107, over 3993657.56 frames. ], batch size: 137, lr: 1.09e-02, grad_scale: 32.0 2026-09-24 00:15:34,726 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=58713.333333333336, ans=0.0 2026-09-24 00:15:35,669 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=58713.333333333336, ans=0.2 2026-09-24 00:15:36,088 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=58713.333333333336, ans=0.0 2026-09-24 00:15:37,277 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.461e+02 3.163e+02 3.527e+02 4.072e+02 5.266e+02, threshold=7.054e+02, percent-clipped=0.0 2026-09-24 00:15:37,403 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=58713.333333333336, ans=0.125 2026-09-24 00:15:38,358 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=58746.666666666664, ans=0.1 2026-09-24 00:15:38,892 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=58746.666666666664, ans=0.125 2026-09-24 00:15:42,803 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=58780.0, ans=0.025 2026-09-24 00:15:44,223 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=58780.0, ans=0.025 2026-09-24 00:15:48,730 INFO [train.py:1192] (0/2) Epoch 19, batch 400, loss[loss=0.2841, simple_loss=0.3935, pruned_loss=0.08734, over 24564.00 frames. ], tot_loss[loss=0.3171, simple_loss=0.4155, pruned_loss=0.1093, over 4178152.87 frames. ], batch size: 170, lr: 1.09e-02, grad_scale: 32.0 2026-09-24 00:15:50,890 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=15.40 vs. limit=22.5 2026-09-24 00:16:06,484 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=58913.333333333336, ans=0.0 2026-09-24 00:16:11,418 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=58946.666666666664, ans=0.125 2026-09-24 00:16:11,951 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=58946.666666666664, ans=0.0 2026-09-24 00:16:13,772 INFO [train.py:1192] (0/2) Epoch 19, batch 450, loss[loss=0.3027, simple_loss=0.4093, pruned_loss=0.09802, over 24615.00 frames. ], tot_loss[loss=0.3174, simple_loss=0.4158, pruned_loss=0.1095, over 4308359.05 frames. ], batch size: 175, lr: 1.09e-02, grad_scale: 32.0 2026-09-24 00:16:14,310 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=58980.0, ans=0.125 2026-09-24 00:16:28,057 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.424e+02 3.210e+02 3.696e+02 4.183e+02 6.561e+02, threshold=7.393e+02, percent-clipped=0.0 2026-09-24 00:16:30,742 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=59080.0, ans=0.0 2026-09-24 00:16:35,932 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=59113.333333333336, ans=0.0 2026-09-24 00:16:39,371 INFO [train.py:1192] (0/2) Epoch 19, batch 500, loss[loss=0.3494, simple_loss=0.452, pruned_loss=0.1234, over 24505.00 frames. ], tot_loss[loss=0.3163, simple_loss=0.4145, pruned_loss=0.109, over 4426305.89 frames. ], batch size: 218, lr: 1.09e-02, grad_scale: 32.0 2026-09-24 00:16:40,037 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=59146.666666666664, ans=0.2 2026-09-24 00:16:46,051 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=59180.0, ans=0.025 2026-09-24 00:16:47,096 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=59180.0, ans=0.125 2026-09-24 00:16:47,665 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=59180.0, ans=0.125 2026-09-24 00:17:05,196 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.47 vs. limit=15.0 2026-09-24 00:17:05,371 INFO [train.py:1192] (0/2) Epoch 19, batch 550, loss[loss=0.3755, simple_loss=0.4701, pruned_loss=0.1404, over 24251.00 frames. ], tot_loss[loss=0.3171, simple_loss=0.4153, pruned_loss=0.1094, over 4516240.56 frames. ], batch size: 257, lr: 1.09e-02, grad_scale: 32.0 2026-09-24 00:17:12,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=59346.666666666664, ans=0.1 2026-09-24 00:17:13,647 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=59346.666666666664, ans=0.125 2026-09-24 00:17:14,207 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=59346.666666666664, ans=0.0 2026-09-24 00:17:17,119 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=59380.0, ans=0.125 2026-09-24 00:17:19,411 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.650e+02 3.204e+02 3.511e+02 3.985e+02 7.550e+02, threshold=7.022e+02, percent-clipped=1.0 2026-09-24 00:17:31,097 INFO [train.py:1192] (0/2) Epoch 19, batch 600, loss[loss=0.3763, simple_loss=0.4708, pruned_loss=0.1409, over 24339.00 frames. ], tot_loss[loss=0.317, simple_loss=0.4154, pruned_loss=0.1093, over 4582301.93 frames. ], batch size: 234, lr: 1.09e-02, grad_scale: 32.0 2026-09-24 00:17:31,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=59480.0, ans=0.125 2026-09-24 00:17:32,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=59480.0, ans=0.125 2026-09-24 00:17:49,181 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=59580.0, ans=0.125 2026-09-24 00:17:55,682 INFO [train.py:1192] (0/2) Epoch 19, batch 650, loss[loss=0.2951, simple_loss=0.3969, pruned_loss=0.09665, over 24577.00 frames. ], tot_loss[loss=0.3151, simple_loss=0.414, pruned_loss=0.1081, over 4648487.65 frames. ], batch size: 162, lr: 1.09e-02, grad_scale: 32.0 2026-09-24 00:17:58,613 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=59646.666666666664, ans=0.0 2026-09-24 00:18:01,119 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=59680.0, ans=0.0 2026-09-24 00:18:05,366 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=59680.0, ans=0.0 2026-09-24 00:18:11,014 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.719e+02 3.292e+02 3.623e+02 4.647e+02 7.311e+02, threshold=7.246e+02, percent-clipped=1.0 2026-09-24 00:18:14,868 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=59746.666666666664, ans=0.1 2026-09-24 00:18:15,972 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=59746.666666666664, ans=0.125 2026-09-24 00:18:22,468 INFO [train.py:1192] (0/2) Epoch 19, batch 700, loss[loss=0.2919, simple_loss=0.3938, pruned_loss=0.09499, over 24575.00 frames. ], tot_loss[loss=0.316, simple_loss=0.415, pruned_loss=0.1085, over 4681780.81 frames. ], batch size: 154, lr: 1.09e-02, grad_scale: 32.0 2026-09-24 00:18:28,262 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=59846.666666666664, ans=0.05 2026-09-24 00:18:29,343 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=59846.666666666664, ans=0.0 2026-09-24 00:18:41,206 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=59913.333333333336, ans=0.125 2026-09-24 00:18:41,719 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=59913.333333333336, ans=0.125 2026-09-24 00:18:44,324 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=59946.666666666664, ans=0.0 2026-09-24 00:18:48,879 INFO [train.py:1192] (0/2) Epoch 19, batch 750, loss[loss=0.2932, simple_loss=0.3987, pruned_loss=0.09385, over 24559.00 frames. ], tot_loss[loss=0.3151, simple_loss=0.4137, pruned_loss=0.1083, over 4710347.56 frames. ], batch size: 170, lr: 1.08e-02, grad_scale: 32.0 2026-09-24 00:18:51,883 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=59980.0, ans=0.0 2026-09-24 00:18:51,889 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=59980.0, ans=0.0 2026-09-24 00:19:01,112 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=60046.666666666664, ans=0.125 2026-09-24 00:19:03,165 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.802e+02 3.406e+02 3.960e+02 4.663e+02 7.283e+02, threshold=7.919e+02, percent-clipped=1.0 2026-09-24 00:19:07,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=60080.0, ans=0.1 2026-09-24 00:19:07,587 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=60080.0, ans=0.0 2026-09-24 00:19:10,995 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=60113.333333333336, ans=0.09899494936611666 2026-09-24 00:19:14,354 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:19:14,822 INFO [train.py:1192] (0/2) Epoch 19, batch 800, loss[loss=0.2643, simple_loss=0.3639, pruned_loss=0.08238, over 24569.00 frames. ], tot_loss[loss=0.3144, simple_loss=0.413, pruned_loss=0.1079, over 4736555.56 frames. ], batch size: 137, lr: 1.08e-02, grad_scale: 32.0 2026-09-24 00:19:15,531 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.81 vs. limit=15.0 2026-09-24 00:19:20,776 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=60180.0, ans=0.07 2026-09-24 00:19:21,991 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.18 vs. limit=15.0 2026-09-24 00:19:30,357 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.88 vs. limit=15.0 2026-09-24 00:19:39,302 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=60313.333333333336, ans=0.125 2026-09-24 00:19:39,705 INFO [train.py:1192] (0/2) Epoch 19, batch 850, loss[loss=0.3439, simple_loss=0.4403, pruned_loss=0.1238, over 24608.00 frames. ], tot_loss[loss=0.313, simple_loss=0.4118, pruned_loss=0.1071, over 4759249.21 frames. ], batch size: 198, lr: 1.08e-02, grad_scale: 32.0 2026-09-24 00:19:48,624 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=60346.666666666664, ans=0.0 2026-09-24 00:19:54,000 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.449e+02 3.136e+02 3.636e+02 4.203e+02 7.119e+02, threshold=7.271e+02, percent-clipped=0.0 2026-09-24 00:19:54,579 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=60413.333333333336, ans=0.125 2026-09-24 00:20:05,070 INFO [train.py:1192] (0/2) Epoch 19, batch 900, loss[loss=0.2751, simple_loss=0.3783, pruned_loss=0.08594, over 24544.00 frames. ], tot_loss[loss=0.3139, simple_loss=0.4128, pruned_loss=0.1075, over 4773427.40 frames. ], batch size: 137, lr: 1.08e-02, grad_scale: 32.0 2026-09-24 00:20:06,229 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=60480.0, ans=0.0 2026-09-24 00:20:14,779 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.43 vs. limit=10.0 2026-09-24 00:20:29,905 INFO [train.py:1192] (0/2) Epoch 19, batch 950, loss[loss=0.4248, simple_loss=0.4579, pruned_loss=0.1958, over 11593.00 frames. ], tot_loss[loss=0.3149, simple_loss=0.4119, pruned_loss=0.109, over 4714189.65 frames. ], batch size: 333, lr: 1.08e-02, grad_scale: 16.0 2026-09-24 00:20:31,441 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=60646.666666666664, ans=0.125 2026-09-24 00:20:34,065 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-19.pt 2026-09-24 00:20:41,046 INFO [train.py:1192] (0/2) Epoch 20, batch 0, loss[loss=0.269, simple_loss=0.3757, pruned_loss=0.08111, over 24585.00 frames. ], tot_loss[loss=0.269, simple_loss=0.3757, pruned_loss=0.08111, over 24585.00 frames. ], batch size: 137, lr: 1.05e-02, grad_scale: 32.0 2026-09-24 00:20:41,046 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 00:20:51,873 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.8793, 4.3094, 4.6279, 4.2738], device='cuda:0') 2026-09-24 00:20:52,690 INFO [train.py:1224] (0/2) Epoch 20, validation: loss=0.1968, simple_loss=0.3133, pruned_loss=0.04012, over 2564189.00 frames. 2026-09-24 00:20:52,691 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 00:20:56,169 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=60673.333333333336, ans=0.0 2026-09-24 00:20:58,743 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=60706.666666666664, ans=0.0 2026-09-24 00:20:59,989 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=60706.666666666664, ans=0.125 2026-09-24 00:21:03,573 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.422e+02 3.285e+02 3.850e+02 4.927e+02 9.160e+02, threshold=7.699e+02, percent-clipped=3.0 2026-09-24 00:21:18,284 INFO [train.py:1192] (0/2) Epoch 20, batch 50, loss[loss=0.2452, simple_loss=0.3411, pruned_loss=0.07464, over 24255.00 frames. ], tot_loss[loss=0.3298, simple_loss=0.4257, pruned_loss=0.117, over 1076167.43 frames. ], batch size: 125, lr: 1.05e-02, grad_scale: 32.0 2026-09-24 00:21:20,867 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=60840.0, ans=0.1 2026-09-24 00:21:22,995 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=60873.333333333336, ans=0.0 2026-09-24 00:21:32,315 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=60906.666666666664, ans=0.125 2026-09-24 00:21:38,544 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.22 vs. limit=10.0 2026-09-24 00:21:44,211 INFO [train.py:1192] (0/2) Epoch 20, batch 100, loss[loss=0.3049, simple_loss=0.3964, pruned_loss=0.1067, over 24609.00 frames. ], tot_loss[loss=0.3298, simple_loss=0.427, pruned_loss=0.1163, over 1904391.27 frames. ], batch size: 154, lr: 1.05e-02, grad_scale: 32.0 2026-09-24 00:21:47,032 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=61006.666666666664, ans=0.0 2026-09-24 00:21:50,827 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.44 vs. limit=6.0 2026-09-24 00:21:54,566 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.627e+02 3.176e+02 3.578e+02 4.126e+02 5.402e+02, threshold=7.156e+02, percent-clipped=0.0 2026-09-24 00:22:00,122 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=61106.666666666664, ans=0.025 2026-09-24 00:22:07,029 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=61140.0, ans=15.0 2026-09-24 00:22:09,580 INFO [train.py:1192] (0/2) Epoch 20, batch 150, loss[loss=0.2676, simple_loss=0.3605, pruned_loss=0.08731, over 24289.00 frames. ], tot_loss[loss=0.3212, simple_loss=0.4194, pruned_loss=0.1115, over 2557563.79 frames. ], batch size: 125, lr: 1.05e-02, grad_scale: 32.0 2026-09-24 00:22:11,044 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=61173.333333333336, ans=0.1 2026-09-24 00:22:17,824 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.58 vs. limit=22.5 2026-09-24 00:22:18,334 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.41 vs. limit=15.0 2026-09-24 00:22:24,171 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=61240.0, ans=0.125 2026-09-24 00:22:32,622 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=61306.666666666664, ans=0.125 2026-09-24 00:22:33,328 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.16 vs. limit=6.0 2026-09-24 00:22:34,440 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=61306.666666666664, ans=0.1 2026-09-24 00:22:34,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=61340.0, ans=0.07 2026-09-24 00:22:35,275 INFO [train.py:1192] (0/2) Epoch 20, batch 200, loss[loss=0.3443, simple_loss=0.4321, pruned_loss=0.1283, over 21176.00 frames. ], tot_loss[loss=0.3174, simple_loss=0.416, pruned_loss=0.1094, over 3054405.05 frames. ], batch size: 333, lr: 1.05e-02, grad_scale: 32.0 2026-09-24 00:22:35,837 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=61340.0, ans=0.2 2026-09-24 00:22:46,212 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.384e+02 3.179e+02 3.621e+02 4.117e+02 8.279e+02, threshold=7.242e+02, percent-clipped=1.0 2026-09-24 00:22:54,551 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=61440.0, ans=0.1 2026-09-24 00:23:00,778 INFO [train.py:1192] (0/2) Epoch 20, batch 250, loss[loss=0.3165, simple_loss=0.4305, pruned_loss=0.1013, over 24320.00 frames. ], tot_loss[loss=0.3165, simple_loss=0.4153, pruned_loss=0.1088, over 3443058.55 frames. ], batch size: 234, lr: 1.04e-02, grad_scale: 32.0 2026-09-24 00:23:07,626 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=61540.0, ans=0.125 2026-09-24 00:23:08,762 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=61540.0, ans=0.125 2026-09-24 00:23:20,509 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=61640.0, ans=0.025 2026-09-24 00:23:21,139 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:23:26,129 INFO [train.py:1192] (0/2) Epoch 20, batch 300, loss[loss=0.3362, simple_loss=0.446, pruned_loss=0.1132, over 24557.00 frames. ], tot_loss[loss=0.3149, simple_loss=0.4136, pruned_loss=0.1081, over 3749189.07 frames. ], batch size: 204, lr: 1.04e-02, grad_scale: 32.0 2026-09-24 00:23:26,213 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=61673.333333333336, ans=0.1 2026-09-24 00:23:26,744 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=61673.333333333336, ans=0.0 2026-09-24 00:23:36,738 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.703e+02 3.287e+02 3.773e+02 4.449e+02 6.240e+02, threshold=7.545e+02, percent-clipped=0.0 2026-09-24 00:23:40,651 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.20 vs. limit=15.0 2026-09-24 00:23:43,625 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.81 vs. limit=22.5 2026-09-24 00:23:48,688 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=61806.666666666664, ans=0.1 2026-09-24 00:23:50,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=61806.666666666664, ans=0.2 2026-09-24 00:23:51,737 INFO [train.py:1192] (0/2) Epoch 20, batch 350, loss[loss=0.2783, simple_loss=0.3751, pruned_loss=0.09076, over 24585.00 frames. ], tot_loss[loss=0.3159, simple_loss=0.415, pruned_loss=0.1084, over 3993065.43 frames. ], batch size: 137, lr: 1.04e-02, grad_scale: 16.0 2026-09-24 00:23:52,830 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=61840.0, ans=0.125 2026-09-24 00:24:00,079 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=61873.333333333336, ans=0.0 2026-09-24 00:24:00,886 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=61873.333333333336, ans=0.0 2026-09-24 00:24:06,452 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=61906.666666666664, ans=0.0 2026-09-24 00:24:08,234 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=61940.0, ans=0.0 2026-09-24 00:24:12,673 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=61973.333333333336, ans=0.125 2026-09-24 00:24:17,834 INFO [train.py:1192] (0/2) Epoch 20, batch 400, loss[loss=0.3095, simple_loss=0.4131, pruned_loss=0.1029, over 24573.00 frames. ], tot_loss[loss=0.315, simple_loss=0.4141, pruned_loss=0.1079, over 4179510.19 frames. ], batch size: 170, lr: 1.04e-02, grad_scale: 32.0 2026-09-24 00:24:23,309 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=62040.0, ans=0.04949747468305833 2026-09-24 00:24:27,625 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.80 vs. limit=15.0 2026-09-24 00:24:29,298 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.592e+02 3.221e+02 3.696e+02 4.247e+02 5.702e+02, threshold=7.392e+02, percent-clipped=0.0 2026-09-24 00:24:29,429 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:24:33,628 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.06 vs. limit=15.0 2026-09-24 00:24:37,079 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=62106.666666666664, ans=0.1 2026-09-24 00:24:43,914 INFO [train.py:1192] (0/2) Epoch 20, batch 450, loss[loss=0.3508, simple_loss=0.4366, pruned_loss=0.1325, over 24630.00 frames. ], tot_loss[loss=0.3153, simple_loss=0.4143, pruned_loss=0.1081, over 4313675.35 frames. ], batch size: 175, lr: 1.04e-02, grad_scale: 32.0 2026-09-24 00:24:48,068 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=62173.333333333336, ans=0.015 2026-09-24 00:24:58,848 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.32 vs. limit=22.5 2026-09-24 00:25:01,734 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.08 vs. limit=12.0 2026-09-24 00:25:05,717 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=62306.666666666664, ans=0.125 2026-09-24 00:25:07,943 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=62306.666666666664, ans=0.125 2026-09-24 00:25:08,450 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=62306.666666666664, ans=0.125 2026-09-24 00:25:09,135 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=62306.666666666664, ans=0.0 2026-09-24 00:25:10,150 INFO [train.py:1192] (0/2) Epoch 20, batch 500, loss[loss=0.3679, simple_loss=0.4585, pruned_loss=0.1387, over 24505.00 frames. ], tot_loss[loss=0.3142, simple_loss=0.4129, pruned_loss=0.1077, over 4430235.43 frames. ], batch size: 218, lr: 1.04e-02, grad_scale: 32.0 2026-09-24 00:25:21,097 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.458e+02 3.194e+02 3.580e+02 4.057e+02 7.296e+02, threshold=7.161e+02, percent-clipped=0.0 2026-09-24 00:25:30,135 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=62473.333333333336, ans=0.125 2026-09-24 00:25:30,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=62473.333333333336, ans=0.0 2026-09-24 00:25:31,305 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.40 vs. limit=15.0 2026-09-24 00:25:34,477 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=62473.333333333336, ans=0.2 2026-09-24 00:25:35,317 INFO [train.py:1192] (0/2) Epoch 20, batch 550, loss[loss=0.3505, simple_loss=0.4511, pruned_loss=0.125, over 24272.00 frames. ], tot_loss[loss=0.3148, simple_loss=0.4136, pruned_loss=0.108, over 4520512.69 frames. ], batch size: 257, lr: 1.04e-02, grad_scale: 32.0 2026-09-24 00:25:36,911 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=62506.666666666664, ans=0.125 2026-09-24 00:25:38,587 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.15 vs. limit=15.0 2026-09-24 00:25:45,219 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=62573.333333333336, ans=0.125 2026-09-24 00:25:49,236 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=62573.333333333336, ans=0.0 2026-09-24 00:25:53,590 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=62606.666666666664, ans=0.025 2026-09-24 00:25:55,821 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=62640.0, ans=0.0 2026-09-24 00:26:01,526 INFO [train.py:1192] (0/2) Epoch 20, batch 600, loss[loss=0.3324, simple_loss=0.4425, pruned_loss=0.1112, over 24349.00 frames. ], tot_loss[loss=0.3147, simple_loss=0.4142, pruned_loss=0.1076, over 4585298.52 frames. ], batch size: 234, lr: 1.04e-02, grad_scale: 32.0 2026-09-24 00:26:02,117 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=62673.333333333336, ans=0.0 2026-09-24 00:26:02,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=62673.333333333336, ans=0.125 2026-09-24 00:26:03,178 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=62673.333333333336, ans=0.125 2026-09-24 00:26:07,750 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=62706.666666666664, ans=0.2 2026-09-24 00:26:11,798 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=62740.0, ans=0.125 2026-09-24 00:26:12,607 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.425e+02 3.205e+02 3.582e+02 4.002e+02 6.555e+02, threshold=7.163e+02, percent-clipped=0.0 2026-09-24 00:26:16,507 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_ff2.min_abs, batch_count=62773.333333333336, ans=0.1 2026-09-24 00:26:27,156 INFO [train.py:1192] (0/2) Epoch 20, batch 650, loss[loss=0.3196, simple_loss=0.4178, pruned_loss=0.1107, over 24559.00 frames. ], tot_loss[loss=0.3131, simple_loss=0.4129, pruned_loss=0.1066, over 4650655.19 frames. ], batch size: 162, lr: 1.04e-02, grad_scale: 32.0 2026-09-24 00:26:27,259 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=62840.0, ans=0.125 2026-09-24 00:26:53,214 INFO [train.py:1192] (0/2) Epoch 20, batch 700, loss[loss=0.2883, simple_loss=0.3915, pruned_loss=0.09261, over 24548.00 frames. ], tot_loss[loss=0.314, simple_loss=0.4138, pruned_loss=0.1071, over 4684181.00 frames. ], batch size: 154, lr: 1.03e-02, grad_scale: 32.0 2026-09-24 00:26:53,832 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=63006.666666666664, ans=0.0 2026-09-24 00:27:04,396 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.341e+02 3.283e+02 3.811e+02 4.565e+02 8.019e+02, threshold=7.622e+02, percent-clipped=1.0 2026-09-24 00:27:18,634 INFO [train.py:1192] (0/2) Epoch 20, batch 750, loss[loss=0.3034, simple_loss=0.4113, pruned_loss=0.09779, over 24566.00 frames. ], tot_loss[loss=0.3125, simple_loss=0.4122, pruned_loss=0.1064, over 4714683.36 frames. ], batch size: 170, lr: 1.03e-02, grad_scale: 32.0 2026-09-24 00:27:21,765 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.50 vs. limit=22.5 2026-09-24 00:27:22,353 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=63173.333333333336, ans=0.1 2026-09-24 00:27:25,540 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=8.68 vs. limit=15.0 2026-09-24 00:27:39,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=63306.666666666664, ans=0.0 2026-09-24 00:27:41,145 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=63306.666666666664, ans=0.125 2026-09-24 00:27:44,855 INFO [train.py:1192] (0/2) Epoch 20, batch 800, loss[loss=0.2423, simple_loss=0.352, pruned_loss=0.06626, over 24570.00 frames. ], tot_loss[loss=0.3122, simple_loss=0.4117, pruned_loss=0.1063, over 4739067.77 frames. ], batch size: 137, lr: 1.03e-02, grad_scale: 32.0 2026-09-24 00:27:49,480 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=63340.0, ans=0.2 2026-09-24 00:27:56,442 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.500e+02 3.155e+02 3.540e+02 4.118e+02 5.592e+02, threshold=7.080e+02, percent-clipped=0.0 2026-09-24 00:27:58,826 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=63406.666666666664, ans=0.0 2026-09-24 00:28:04,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=63440.0, ans=0.07 2026-09-24 00:28:11,224 INFO [train.py:1192] (0/2) Epoch 20, batch 850, loss[loss=0.3262, simple_loss=0.4365, pruned_loss=0.1079, over 24529.00 frames. ], tot_loss[loss=0.312, simple_loss=0.4116, pruned_loss=0.1062, over 4762102.73 frames. ], batch size: 204, lr: 1.03e-02, grad_scale: 32.0 2026-09-24 00:28:13,262 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=63506.666666666664, ans=0.0 2026-09-24 00:28:23,705 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=63573.333333333336, ans=0.125 2026-09-24 00:28:29,985 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=63606.666666666664, ans=0.125 2026-09-24 00:28:37,217 INFO [train.py:1192] (0/2) Epoch 20, batch 900, loss[loss=0.2569, simple_loss=0.3643, pruned_loss=0.07478, over 24532.00 frames. ], tot_loss[loss=0.3126, simple_loss=0.4123, pruned_loss=0.1065, over 4775370.59 frames. ], batch size: 137, lr: 1.03e-02, grad_scale: 32.0 2026-09-24 00:28:48,704 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.595e+02 3.360e+02 3.782e+02 4.282e+02 5.847e+02, threshold=7.564e+02, percent-clipped=0.0 2026-09-24 00:28:53,678 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=63773.333333333336, ans=0.1 2026-09-24 00:28:59,363 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:29:01,510 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=13.08 vs. limit=15.0 2026-09-24 00:29:02,177 INFO [train.py:1192] (0/2) Epoch 20, batch 950, loss[loss=0.4121, simple_loss=0.4564, pruned_loss=0.1838, over 11644.00 frames. ], tot_loss[loss=0.3143, simple_loss=0.412, pruned_loss=0.1083, over 4719893.98 frames. ], batch size: 333, lr: 1.03e-02, grad_scale: 32.0 2026-09-24 00:29:03,528 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=63840.0, ans=0.1 2026-09-24 00:29:03,538 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=63840.0, ans=0.1 2026-09-24 00:29:06,361 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-20.pt 2026-09-24 00:29:13,683 INFO [train.py:1192] (0/2) Epoch 21, batch 0, loss[loss=0.2768, simple_loss=0.3805, pruned_loss=0.08656, over 24574.00 frames. ], tot_loss[loss=0.2768, simple_loss=0.3805, pruned_loss=0.08656, over 24574.00 frames. ], batch size: 137, lr: 1.00e-02, grad_scale: 32.0 2026-09-24 00:29:13,683 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 00:29:25,320 INFO [train.py:1224] (0/2) Epoch 21, validation: loss=0.195, simple_loss=0.3111, pruned_loss=0.03952, over 2564189.00 frames. 2026-09-24 00:29:25,321 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 00:29:32,263 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=63900.0, ans=0.0 2026-09-24 00:29:38,250 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=63933.333333333336, ans=0.125 2026-09-24 00:29:41,999 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.21 vs. limit=15.0 2026-09-24 00:29:46,164 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=64000.0, ans=0.0 2026-09-24 00:29:51,390 INFO [train.py:1192] (0/2) Epoch 21, batch 50, loss[loss=0.2713, simple_loss=0.3685, pruned_loss=0.08703, over 24219.00 frames. ], tot_loss[loss=0.3264, simple_loss=0.4231, pruned_loss=0.1149, over 1076409.98 frames. ], batch size: 125, lr: 1.00e-02, grad_scale: 32.0 2026-09-24 00:29:54,270 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=64033.333333333336, ans=0.5 2026-09-24 00:29:58,675 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.549e+02 3.392e+02 4.011e+02 4.620e+02 8.651e+02, threshold=8.021e+02, percent-clipped=2.0 2026-09-24 00:30:00,349 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=64066.666666666664, ans=0.0 2026-09-24 00:30:01,585 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.91 vs. limit=6.0 2026-09-24 00:30:15,050 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=64166.666666666664, ans=0.125 2026-09-24 00:30:16,380 INFO [train.py:1192] (0/2) Epoch 21, batch 100, loss[loss=0.2881, simple_loss=0.3865, pruned_loss=0.09485, over 24605.00 frames. ], tot_loss[loss=0.3264, simple_loss=0.425, pruned_loss=0.1139, over 1905098.39 frames. ], batch size: 154, lr: 1.00e-02, grad_scale: 32.0 2026-09-24 00:30:17,484 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=64200.0, ans=0.0 2026-09-24 00:30:41,964 INFO [train.py:1192] (0/2) Epoch 21, batch 150, loss[loss=0.261, simple_loss=0.3587, pruned_loss=0.08169, over 24262.00 frames. ], tot_loss[loss=0.3193, simple_loss=0.4189, pruned_loss=0.1099, over 2559557.44 frames. ], batch size: 125, lr: 9.99e-03, grad_scale: 32.0 2026-09-24 00:30:49,040 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=64400.0, ans=0.125 2026-09-24 00:30:49,823 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.497e+02 3.235e+02 3.700e+02 4.139e+02 6.411e+02, threshold=7.399e+02, percent-clipped=0.0 2026-09-24 00:30:54,473 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=64433.333333333336, ans=0.125 2026-09-24 00:31:07,602 INFO [train.py:1192] (0/2) Epoch 21, batch 200, loss[loss=0.3791, simple_loss=0.4559, pruned_loss=0.1512, over 21030.00 frames. ], tot_loss[loss=0.3136, simple_loss=0.4142, pruned_loss=0.1065, over 3055733.29 frames. ], batch size: 333, lr: 9.98e-03, grad_scale: 32.0 2026-09-24 00:31:16,211 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=64566.666666666664, ans=0.04949747468305833 2026-09-24 00:31:17,054 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=64566.666666666664, ans=0.0 2026-09-24 00:31:25,417 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=64633.333333333336, ans=0.0 2026-09-24 00:31:29,479 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=64666.666666666664, ans=0.125 2026-09-24 00:31:33,205 INFO [train.py:1192] (0/2) Epoch 21, batch 250, loss[loss=0.3738, simple_loss=0.4687, pruned_loss=0.1395, over 24301.00 frames. ], tot_loss[loss=0.3124, simple_loss=0.4129, pruned_loss=0.1059, over 3443532.09 frames. ], batch size: 234, lr: 9.97e-03, grad_scale: 32.0 2026-09-24 00:31:37,211 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=64700.0, ans=0.2 2026-09-24 00:31:40,028 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=64733.333333333336, ans=0.125 2026-09-24 00:31:40,030 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=64733.333333333336, ans=0.125 2026-09-24 00:31:40,886 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.329e+02 3.379e+02 3.844e+02 4.387e+02 9.015e+02, threshold=7.688e+02, percent-clipped=1.0 2026-09-24 00:31:49,819 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=64800.0, ans=0.125 2026-09-24 00:31:58,374 INFO [train.py:1192] (0/2) Epoch 21, batch 300, loss[loss=0.281, simple_loss=0.3987, pruned_loss=0.08163, over 24551.00 frames. ], tot_loss[loss=0.3117, simple_loss=0.4117, pruned_loss=0.1059, over 3749893.49 frames. ], batch size: 204, lr: 9.96e-03, grad_scale: 32.0 2026-09-24 00:32:04,252 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=64900.0, ans=0.0 2026-09-24 00:32:10,378 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.96 vs. limit=15.0 2026-09-24 00:32:10,731 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=64933.333333333336, ans=0.1 2026-09-24 00:32:11,698 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=64933.333333333336, ans=0.0 2026-09-24 00:32:14,160 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=64966.666666666664, ans=0.07 2026-09-24 00:32:15,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_positive, batch_count=64966.666666666664, ans=0.05 2026-09-24 00:32:21,169 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=65000.0, ans=0.2 2026-09-24 00:32:23,531 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:32:23,988 INFO [train.py:1192] (0/2) Epoch 21, batch 350, loss[loss=0.2582, simple_loss=0.3582, pruned_loss=0.07911, over 24586.00 frames. ], tot_loss[loss=0.3122, simple_loss=0.4125, pruned_loss=0.1059, over 3993095.50 frames. ], batch size: 137, lr: 9.95e-03, grad_scale: 32.0 2026-09-24 00:32:28,633 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=65066.666666666664, ans=0.125 2026-09-24 00:32:31,862 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.496e+02 3.120e+02 3.532e+02 3.996e+02 6.407e+02, threshold=7.064e+02, percent-clipped=0.0 2026-09-24 00:32:41,023 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=65133.333333333336, ans=0.125 2026-09-24 00:32:49,199 INFO [train.py:1192] (0/2) Epoch 21, batch 400, loss[loss=0.3344, simple_loss=0.4312, pruned_loss=0.1188, over 24571.00 frames. ], tot_loss[loss=0.3118, simple_loss=0.4121, pruned_loss=0.1058, over 4176776.61 frames. ], batch size: 170, lr: 9.94e-03, grad_scale: 32.0 2026-09-24 00:32:50,407 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=65200.0, ans=0.07 2026-09-24 00:32:57,678 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=65233.333333333336, ans=0.04949747468305833 2026-09-24 00:32:59,063 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=65266.666666666664, ans=0.1 2026-09-24 00:32:59,565 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.37 vs. limit=22.5 2026-09-24 00:33:00,059 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.87 vs. limit=22.5 2026-09-24 00:33:01,328 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=65266.666666666664, ans=0.125 2026-09-24 00:33:03,671 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=65300.0, ans=0.125 2026-09-24 00:33:12,949 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=65333.333333333336, ans=0.125 2026-09-24 00:33:14,388 INFO [train.py:1192] (0/2) Epoch 21, batch 450, loss[loss=0.3045, simple_loss=0.4095, pruned_loss=0.0998, over 24611.00 frames. ], tot_loss[loss=0.3101, simple_loss=0.4111, pruned_loss=0.1045, over 4308323.04 frames. ], batch size: 175, lr: 9.93e-03, grad_scale: 32.0 2026-09-24 00:33:15,001 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=65366.666666666664, ans=0.125 2026-09-24 00:33:22,298 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.589e+02 3.354e+02 3.803e+02 4.582e+02 8.143e+02, threshold=7.607e+02, percent-clipped=1.0 2026-09-24 00:33:37,261 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=65500.0, ans=0.2 2026-09-24 00:33:39,969 INFO [train.py:1192] (0/2) Epoch 21, batch 500, loss[loss=0.3349, simple_loss=0.4394, pruned_loss=0.1152, over 24529.00 frames. ], tot_loss[loss=0.3087, simple_loss=0.4093, pruned_loss=0.1041, over 4427556.14 frames. ], batch size: 218, lr: 9.91e-03, grad_scale: 32.0 2026-09-24 00:33:41,382 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=65533.333333333336, ans=0.125 2026-09-24 00:33:42,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=65533.333333333336, ans=0.2 2026-09-24 00:33:57,852 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=65633.33333333333, ans=0.0 2026-09-24 00:34:06,300 INFO [train.py:1192] (0/2) Epoch 21, batch 550, loss[loss=0.3457, simple_loss=0.4518, pruned_loss=0.1199, over 24239.00 frames. ], tot_loss[loss=0.3083, simple_loss=0.4093, pruned_loss=0.1037, over 4517251.07 frames. ], batch size: 257, lr: 9.90e-03, grad_scale: 32.0 2026-09-24 00:34:12,575 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=65733.33333333333, ans=0.125 2026-09-24 00:34:13,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=65733.33333333333, ans=0.2 2026-09-24 00:34:13,335 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.501e+02 3.059e+02 3.593e+02 4.039e+02 5.830e+02, threshold=7.187e+02, percent-clipped=0.0 2026-09-24 00:34:13,904 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=65733.33333333333, ans=0.125 2026-09-24 00:34:31,576 INFO [train.py:1192] (0/2) Epoch 21, batch 600, loss[loss=0.3493, simple_loss=0.4576, pruned_loss=0.1205, over 24339.00 frames. ], tot_loss[loss=0.3087, simple_loss=0.4099, pruned_loss=0.1037, over 4583920.16 frames. ], batch size: 234, lr: 9.89e-03, grad_scale: 32.0 2026-09-24 00:34:44,518 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=65933.33333333333, ans=0.0 2026-09-24 00:34:52,925 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=66000.0, ans=0.125 2026-09-24 00:34:56,140 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=66000.0, ans=0.1 2026-09-24 00:34:57,471 INFO [train.py:1192] (0/2) Epoch 21, batch 650, loss[loss=0.3085, simple_loss=0.4092, pruned_loss=0.1039, over 24561.00 frames. ], tot_loss[loss=0.3083, simple_loss=0.4093, pruned_loss=0.1036, over 4649393.09 frames. ], batch size: 162, lr: 9.88e-03, grad_scale: 32.0 2026-09-24 00:34:57,566 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=66033.33333333333, ans=0.125 2026-09-24 00:34:58,053 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=66033.33333333333, ans=0.125 2026-09-24 00:35:00,848 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=66033.33333333333, ans=0.0 2026-09-24 00:35:04,494 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.662e+02 3.257e+02 3.614e+02 4.275e+02 7.702e+02, threshold=7.227e+02, percent-clipped=2.0 2026-09-24 00:35:23,290 INFO [train.py:1192] (0/2) Epoch 21, batch 700, loss[loss=0.2919, simple_loss=0.3922, pruned_loss=0.09579, over 24579.00 frames. ], tot_loss[loss=0.3091, simple_loss=0.4101, pruned_loss=0.104, over 4681816.78 frames. ], batch size: 154, lr: 9.87e-03, grad_scale: 32.0 2026-09-24 00:35:23,494 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.73 vs. limit=15.0 2026-09-24 00:35:24,421 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=66200.0, ans=0.125 2026-09-24 00:35:32,082 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=66233.33333333333, ans=0.125 2026-09-24 00:35:48,274 INFO [train.py:1192] (0/2) Epoch 21, batch 750, loss[loss=0.3042, simple_loss=0.4105, pruned_loss=0.09897, over 24556.00 frames. ], tot_loss[loss=0.3083, simple_loss=0.4091, pruned_loss=0.1037, over 4709976.71 frames. ], batch size: 170, lr: 9.86e-03, grad_scale: 32.0 2026-09-24 00:35:56,350 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.596e+02 3.197e+02 3.654e+02 4.498e+02 6.742e+02, threshold=7.308e+02, percent-clipped=0.0 2026-09-24 00:36:14,114 INFO [train.py:1192] (0/2) Epoch 21, batch 800, loss[loss=0.2479, simple_loss=0.3584, pruned_loss=0.06872, over 24546.00 frames. ], tot_loss[loss=0.3077, simple_loss=0.4087, pruned_loss=0.1034, over 4736156.55 frames. ], batch size: 137, lr: 9.85e-03, grad_scale: 32.0 2026-09-24 00:36:19,751 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=66566.66666666667, ans=0.0 2026-09-24 00:36:20,249 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=66566.66666666667, ans=0.125 2026-09-24 00:36:28,321 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=66600.0, ans=0.1 2026-09-24 00:36:32,937 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=66633.33333333333, ans=0.0 2026-09-24 00:36:34,262 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-20000.pt 2026-09-24 00:36:38,556 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:36:39,054 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=66666.66666666667, ans=0.125 2026-09-24 00:36:39,843 INFO [train.py:1192] (0/2) Epoch 21, batch 850, loss[loss=0.3209, simple_loss=0.4308, pruned_loss=0.1054, over 24605.00 frames. ], tot_loss[loss=0.3065, simple_loss=0.4079, pruned_loss=0.1026, over 4760212.55 frames. ], batch size: 198, lr: 9.84e-03, grad_scale: 32.0 2026-09-24 00:36:47,761 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.669e+02 3.302e+02 3.726e+02 4.097e+02 5.783e+02, threshold=7.453e+02, percent-clipped=0.0 2026-09-24 00:36:56,804 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:37:00,789 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=66833.33333333333, ans=0.0 2026-09-24 00:37:02,761 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=66833.33333333333, ans=0.125 2026-09-24 00:37:05,598 INFO [train.py:1192] (0/2) Epoch 21, batch 900, loss[loss=0.2743, simple_loss=0.377, pruned_loss=0.08584, over 24556.00 frames. ], tot_loss[loss=0.3073, simple_loss=0.4086, pruned_loss=0.103, over 4773789.02 frames. ], batch size: 137, lr: 9.83e-03, grad_scale: 16.0 2026-09-24 00:37:09,957 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.50 vs. limit=12.0 2026-09-24 00:37:10,374 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=66900.0, ans=0.1 2026-09-24 00:37:10,882 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=66900.0, ans=0.125 2026-09-24 00:37:19,070 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=66933.33333333333, ans=0.125 2026-09-24 00:37:30,990 INFO [train.py:1192] (0/2) Epoch 21, batch 950, loss[loss=0.4407, simple_loss=0.4842, pruned_loss=0.1986, over 11554.00 frames. ], tot_loss[loss=0.3098, simple_loss=0.4088, pruned_loss=0.1054, over 4716795.40 frames. ], batch size: 334, lr: 9.82e-03, grad_scale: 16.0 2026-09-24 00:37:35,613 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-21.pt 2026-09-24 00:37:43,086 INFO [train.py:1192] (0/2) Epoch 22, batch 0, loss[loss=0.2665, simple_loss=0.3714, pruned_loss=0.08084, over 24590.00 frames. ], tot_loss[loss=0.2665, simple_loss=0.3714, pruned_loss=0.08084, over 24590.00 frames. ], batch size: 137, lr: 9.59e-03, grad_scale: 32.0 2026-09-24 00:37:43,087 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 00:37:47,257 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.8962, 3.2564, 3.1340, 1.8351], device='cuda:0') 2026-09-24 00:37:54,394 INFO [train.py:1224] (0/2) Epoch 22, validation: loss=0.1913, simple_loss=0.3083, pruned_loss=0.03717, over 2564189.00 frames. 2026-09-24 00:37:54,395 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 00:37:54,905 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=67060.0, ans=0.125 2026-09-24 00:37:58,356 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.587e+02 3.465e+02 4.030e+02 4.667e+02 7.455e+02, threshold=8.059e+02, percent-clipped=1.0 2026-09-24 00:38:01,331 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.72 vs. limit=15.0 2026-09-24 00:38:05,636 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=67126.66666666667, ans=0.125 2026-09-24 00:38:14,390 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=67193.33333333333, ans=0.025 2026-09-24 00:38:19,197 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=67193.33333333333, ans=0.125 2026-09-24 00:38:19,927 INFO [train.py:1192] (0/2) Epoch 22, batch 50, loss[loss=0.254, simple_loss=0.354, pruned_loss=0.07696, over 24233.00 frames. ], tot_loss[loss=0.3144, simple_loss=0.4149, pruned_loss=0.107, over 1075686.23 frames. ], batch size: 125, lr: 9.57e-03, grad_scale: 32.0 2026-09-24 00:38:22,396 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=67226.66666666667, ans=0.2 2026-09-24 00:38:22,880 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=67226.66666666667, ans=0.0 2026-09-24 00:38:32,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=67293.33333333333, ans=0.0 2026-09-24 00:38:36,927 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=67326.66666666667, ans=0.125 2026-09-24 00:38:41,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=67360.0, ans=0.0 2026-09-24 00:38:43,954 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.18 vs. limit=15.0 2026-09-24 00:38:45,037 INFO [train.py:1192] (0/2) Epoch 22, batch 100, loss[loss=0.2889, simple_loss=0.3886, pruned_loss=0.09461, over 24604.00 frames. ], tot_loss[loss=0.3181, simple_loss=0.419, pruned_loss=0.1086, over 1904839.40 frames. ], batch size: 154, lr: 9.56e-03, grad_scale: 32.0 2026-09-24 00:38:49,283 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.513e+02 3.095e+02 3.490e+02 4.082e+02 5.584e+02, threshold=6.980e+02, percent-clipped=0.0 2026-09-24 00:38:58,028 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.78 vs. limit=12.0 2026-09-24 00:38:58,920 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.56 vs. limit=15.0 2026-09-24 00:39:01,477 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.54 vs. limit=15.0 2026-09-24 00:39:06,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=67526.66666666667, ans=0.125 2026-09-24 00:39:07,057 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=67526.66666666667, ans=0.1 2026-09-24 00:39:11,403 INFO [train.py:1192] (0/2) Epoch 22, batch 150, loss[loss=0.2352, simple_loss=0.3409, pruned_loss=0.06473, over 24258.00 frames. ], tot_loss[loss=0.3112, simple_loss=0.413, pruned_loss=0.1047, over 2558612.09 frames. ], batch size: 125, lr: 9.55e-03, grad_scale: 32.0 2026-09-24 00:39:11,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=67560.0, ans=0.125 2026-09-24 00:39:12,553 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.62 vs. limit=22.5 2026-09-24 00:39:22,624 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.11 vs. limit=5.0 2026-09-24 00:39:29,525 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=67660.0, ans=0.125 2026-09-24 00:39:34,610 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.64 vs. limit=6.0 2026-09-24 00:39:34,969 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=67693.33333333333, ans=0.2 2026-09-24 00:39:37,254 INFO [train.py:1192] (0/2) Epoch 22, batch 200, loss[loss=0.3781, simple_loss=0.4573, pruned_loss=0.1494, over 21113.00 frames. ], tot_loss[loss=0.3104, simple_loss=0.4119, pruned_loss=0.1045, over 3055465.06 frames. ], batch size: 333, lr: 9.54e-03, grad_scale: 32.0 2026-09-24 00:39:37,359 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=67726.66666666667, ans=0.125 2026-09-24 00:39:41,517 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.529e+02 3.283e+02 3.694e+02 4.375e+02 6.321e+02, threshold=7.388e+02, percent-clipped=0.0 2026-09-24 00:39:42,049 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=67760.0, ans=0.125 2026-09-24 00:39:50,239 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.61 vs. limit=12.0 2026-09-24 00:39:58,310 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=67860.0, ans=0.0 2026-09-24 00:40:03,095 INFO [train.py:1192] (0/2) Epoch 22, batch 250, loss[loss=0.3471, simple_loss=0.4531, pruned_loss=0.1206, over 24305.00 frames. ], tot_loss[loss=0.311, simple_loss=0.4121, pruned_loss=0.105, over 3444917.38 frames. ], batch size: 234, lr: 9.53e-03, grad_scale: 16.0 2026-09-24 00:40:11,662 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=67926.66666666667, ans=0.125 2026-09-24 00:40:15,533 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=67960.0, ans=0.0 2026-09-24 00:40:18,232 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=67993.33333333333, ans=0.0 2026-09-24 00:40:19,731 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=67993.33333333333, ans=0.125 2026-09-24 00:40:22,841 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=67993.33333333333, ans=0.125 2026-09-24 00:40:26,775 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=68026.66666666667, ans=0.0 2026-09-24 00:40:28,798 INFO [train.py:1192] (0/2) Epoch 22, batch 300, loss[loss=0.3519, simple_loss=0.4537, pruned_loss=0.125, over 24535.00 frames. ], tot_loss[loss=0.3094, simple_loss=0.4102, pruned_loss=0.1043, over 3750168.70 frames. ], batch size: 204, lr: 9.52e-03, grad_scale: 16.0 2026-09-24 00:40:33,113 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.694e+02 3.361e+02 3.840e+02 4.584e+02 9.476e+02, threshold=7.681e+02, percent-clipped=4.0 2026-09-24 00:40:35,605 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.97 vs. limit=22.5 2026-09-24 00:40:38,380 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:40:49,005 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=68193.33333333333, ans=0.125 2026-09-24 00:40:49,481 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=68193.33333333333, ans=0.1 2026-09-24 00:40:50,546 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:40:54,232 INFO [train.py:1192] (0/2) Epoch 22, batch 350, loss[loss=0.2792, simple_loss=0.3735, pruned_loss=0.09243, over 24585.00 frames. ], tot_loss[loss=0.3098, simple_loss=0.4108, pruned_loss=0.1043, over 3989793.92 frames. ], batch size: 137, lr: 9.51e-03, grad_scale: 16.0 2026-09-24 00:40:54,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=68226.66666666667, ans=0.05 2026-09-24 00:40:57,633 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=68226.66666666667, ans=0.125 2026-09-24 00:41:08,202 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=68293.33333333333, ans=0.0 2026-09-24 00:41:19,454 INFO [train.py:1192] (0/2) Epoch 22, batch 400, loss[loss=0.3075, simple_loss=0.4057, pruned_loss=0.1047, over 24578.00 frames. ], tot_loss[loss=0.3085, simple_loss=0.4097, pruned_loss=0.1036, over 4175899.40 frames. ], batch size: 170, lr: 9.50e-03, grad_scale: 32.0 2026-09-24 00:41:23,734 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.589e+02 3.105e+02 3.438e+02 4.062e+02 5.782e+02, threshold=6.875e+02, percent-clipped=0.0 2026-09-24 00:41:26,149 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=68426.66666666667, ans=0.0 2026-09-24 00:41:42,248 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=68526.66666666667, ans=0.125 2026-09-24 00:41:42,785 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=68526.66666666667, ans=0.2 2026-09-24 00:41:44,302 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=68526.66666666667, ans=0.125 2026-09-24 00:41:45,176 INFO [train.py:1192] (0/2) Epoch 22, batch 450, loss[loss=0.3011, simple_loss=0.4065, pruned_loss=0.09783, over 24626.00 frames. ], tot_loss[loss=0.3101, simple_loss=0.4109, pruned_loss=0.1046, over 4307365.47 frames. ], batch size: 175, lr: 9.49e-03, grad_scale: 32.0 2026-09-24 00:41:47,802 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=68560.0, ans=0.0 2026-09-24 00:41:48,639 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=68560.0, ans=0.2 2026-09-24 00:41:49,380 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.90 vs. limit=6.0 2026-09-24 00:41:49,674 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=68593.33333333333, ans=0.0 2026-09-24 00:41:52,011 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.63 vs. limit=22.5 2026-09-24 00:42:00,357 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=68660.0, ans=0.125 2026-09-24 00:42:08,236 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=68693.33333333333, ans=0.025 2026-09-24 00:42:10,386 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.30 vs. limit=12.0 2026-09-24 00:42:10,647 INFO [train.py:1192] (0/2) Epoch 22, batch 500, loss[loss=0.3748, simple_loss=0.4717, pruned_loss=0.139, over 24498.00 frames. ], tot_loss[loss=0.3098, simple_loss=0.4101, pruned_loss=0.1048, over 4425551.70 frames. ], batch size: 218, lr: 9.48e-03, grad_scale: 32.0 2026-09-24 00:42:13,879 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.80 vs. limit=22.5 2026-09-24 00:42:14,307 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=68726.66666666667, ans=0.0 2026-09-24 00:42:15,217 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.545e+02 3.417e+02 3.981e+02 4.780e+02 8.506e+02, threshold=7.962e+02, percent-clipped=3.0 2026-09-24 00:42:21,658 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=68793.33333333333, ans=0.125 2026-09-24 00:42:28,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=68826.66666666667, ans=0.025 2026-09-24 00:42:32,072 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.67 vs. limit=8.0 2026-09-24 00:42:35,483 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.27 vs. limit=6.0 2026-09-24 00:42:36,643 INFO [train.py:1192] (0/2) Epoch 22, batch 550, loss[loss=0.3697, simple_loss=0.4691, pruned_loss=0.1351, over 24248.00 frames. ], tot_loss[loss=0.3096, simple_loss=0.4102, pruned_loss=0.1045, over 4515749.38 frames. ], batch size: 257, lr: 9.47e-03, grad_scale: 32.0 2026-09-24 00:42:39,448 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=68893.33333333333, ans=0.025 2026-09-24 00:42:44,445 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=68926.66666666667, ans=0.1 2026-09-24 00:42:50,462 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.05 vs. limit=15.0 2026-09-24 00:42:57,498 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=69026.66666666667, ans=0.1 2026-09-24 00:42:58,039 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=69026.66666666667, ans=0.125 2026-09-24 00:43:02,560 INFO [train.py:1192] (0/2) Epoch 22, batch 600, loss[loss=0.3552, simple_loss=0.4622, pruned_loss=0.1241, over 24343.00 frames. ], tot_loss[loss=0.3095, simple_loss=0.4105, pruned_loss=0.1043, over 4581942.96 frames. ], batch size: 234, lr: 9.46e-03, grad_scale: 32.0 2026-09-24 00:43:07,409 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.420e+02 3.284e+02 3.687e+02 4.180e+02 5.343e+02, threshold=7.374e+02, percent-clipped=0.0 2026-09-24 00:43:16,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=69126.66666666667, ans=0.125 2026-09-24 00:43:16,107 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=69126.66666666667, ans=0.2 2026-09-24 00:43:18,673 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=69160.0, ans=0.2 2026-09-24 00:43:19,639 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=69160.0, ans=0.04949747468305833 2026-09-24 00:43:24,965 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=69193.33333333333, ans=0.125 2026-09-24 00:43:26,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=69193.33333333333, ans=0.2 2026-09-24 00:43:27,344 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=69193.33333333333, ans=0.0 2026-09-24 00:43:28,439 INFO [train.py:1192] (0/2) Epoch 22, batch 650, loss[loss=0.2841, simple_loss=0.3901, pruned_loss=0.089, over 24555.00 frames. ], tot_loss[loss=0.3068, simple_loss=0.4085, pruned_loss=0.1026, over 4647850.01 frames. ], batch size: 162, lr: 9.45e-03, grad_scale: 32.0 2026-09-24 00:43:28,961 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=69226.66666666667, ans=0.125 2026-09-24 00:43:45,368 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.92 vs. limit=15.0 2026-09-24 00:43:54,194 INFO [train.py:1192] (0/2) Epoch 22, batch 700, loss[loss=0.3162, simple_loss=0.412, pruned_loss=0.1102, over 24576.00 frames. ], tot_loss[loss=0.3063, simple_loss=0.4088, pruned_loss=0.1019, over 4681458.59 frames. ], batch size: 154, lr: 9.44e-03, grad_scale: 32.0 2026-09-24 00:43:58,621 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.790e+02 3.448e+02 3.947e+02 4.620e+02 7.877e+02, threshold=7.894e+02, percent-clipped=2.0 2026-09-24 00:44:02,172 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.91 vs. limit=15.0 2026-09-24 00:44:07,615 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=69460.0, ans=0.125 2026-09-24 00:44:19,070 INFO [train.py:1192] (0/2) Epoch 22, batch 750, loss[loss=0.3124, simple_loss=0.414, pruned_loss=0.1054, over 24556.00 frames. ], tot_loss[loss=0.3051, simple_loss=0.4074, pruned_loss=0.1013, over 4713575.37 frames. ], batch size: 170, lr: 9.43e-03, grad_scale: 32.0 2026-09-24 00:44:24,710 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.26 vs. limit=15.0 2026-09-24 00:44:26,031 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=69593.33333333333, ans=0.025 2026-09-24 00:44:31,290 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=69626.66666666667, ans=0.0 2026-09-24 00:44:40,140 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=69693.33333333333, ans=0.1 2026-09-24 00:44:44,679 INFO [train.py:1192] (0/2) Epoch 22, batch 800, loss[loss=0.2748, simple_loss=0.3733, pruned_loss=0.08814, over 24538.00 frames. ], tot_loss[loss=0.3046, simple_loss=0.407, pruned_loss=0.1011, over 4738632.14 frames. ], batch size: 137, lr: 9.42e-03, grad_scale: 32.0 2026-09-24 00:44:48,944 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.402e+02 3.206e+02 3.625e+02 4.112e+02 5.774e+02, threshold=7.249e+02, percent-clipped=0.0 2026-09-24 00:44:49,059 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:44:53,783 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.14 vs. limit=10.0 2026-09-24 00:44:54,840 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.10 vs. limit=6.0 2026-09-24 00:44:55,031 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=69793.33333333333, ans=0.0 2026-09-24 00:44:57,287 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=69793.33333333333, ans=0.2 2026-09-24 00:44:58,874 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=69793.33333333333, ans=0.07 2026-09-24 00:44:59,971 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=69826.66666666667, ans=0.125 2026-09-24 00:45:00,423 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=69826.66666666667, ans=0.025 2026-09-24 00:45:01,462 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=69826.66666666667, ans=0.025 2026-09-24 00:45:10,112 INFO [train.py:1192] (0/2) Epoch 22, batch 850, loss[loss=0.301, simple_loss=0.4125, pruned_loss=0.09475, over 24594.00 frames. ], tot_loss[loss=0.3049, simple_loss=0.4071, pruned_loss=0.1013, over 4760981.47 frames. ], batch size: 198, lr: 9.41e-03, grad_scale: 32.0 2026-09-24 00:45:22,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=69960.0, ans=0.1 2026-09-24 00:45:31,803 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.86 vs. limit=10.0 2026-09-24 00:45:36,327 INFO [train.py:1192] (0/2) Epoch 22, batch 900, loss[loss=0.2642, simple_loss=0.3714, pruned_loss=0.07849, over 24536.00 frames. ], tot_loss[loss=0.3059, simple_loss=0.4079, pruned_loss=0.102, over 4773914.75 frames. ], batch size: 137, lr: 9.40e-03, grad_scale: 32.0 2026-09-24 00:45:40,946 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.552e+02 3.184e+02 3.761e+02 4.280e+02 6.226e+02, threshold=7.523e+02, percent-clipped=0.0 2026-09-24 00:45:42,397 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=70093.33333333333, ans=0.125 2026-09-24 00:46:01,430 INFO [train.py:1192] (0/2) Epoch 22, batch 950, loss[loss=0.4279, simple_loss=0.4691, pruned_loss=0.1934, over 10785.00 frames. ], tot_loss[loss=0.307, simple_loss=0.4072, pruned_loss=0.1034, over 4712528.91 frames. ], batch size: 333, lr: 9.39e-03, grad_scale: 32.0 2026-09-24 00:46:05,763 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-22.pt 2026-09-24 00:46:12,763 INFO [train.py:1192] (0/2) Epoch 23, batch 0, loss[loss=0.2882, simple_loss=0.3928, pruned_loss=0.09178, over 24543.00 frames. ], tot_loss[loss=0.2882, simple_loss=0.3928, pruned_loss=0.09178, over 24543.00 frames. ], batch size: 137, lr: 9.18e-03, grad_scale: 32.0 2026-09-24 00:46:12,763 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 00:46:24,383 INFO [train.py:1224] (0/2) Epoch 23, validation: loss=0.1918, simple_loss=0.3086, pruned_loss=0.0375, over 2564189.00 frames. 2026-09-24 00:46:24,383 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 00:46:31,752 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=70286.66666666667, ans=0.1 2026-09-24 00:46:35,627 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=70320.0, ans=0.125 2026-09-24 00:46:46,302 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=70386.66666666667, ans=0.2 2026-09-24 00:46:46,504 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.27 vs. limit=15.0 2026-09-24 00:46:49,881 INFO [train.py:1192] (0/2) Epoch 23, batch 50, loss[loss=0.2456, simple_loss=0.3441, pruned_loss=0.07349, over 24284.00 frames. ], tot_loss[loss=0.3165, simple_loss=0.4164, pruned_loss=0.1083, over 1076127.51 frames. ], batch size: 125, lr: 9.17e-03, grad_scale: 32.0 2026-09-24 00:46:50,291 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.535e+02 3.301e+02 3.835e+02 4.680e+02 7.694e+02, threshold=7.671e+02, percent-clipped=1.0 2026-09-24 00:47:09,069 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=70520.0, ans=0.025 2026-09-24 00:47:15,829 INFO [train.py:1192] (0/2) Epoch 23, batch 100, loss[loss=0.3054, simple_loss=0.4035, pruned_loss=0.1037, over 24610.00 frames. ], tot_loss[loss=0.3187, simple_loss=0.4192, pruned_loss=0.1091, over 1904652.01 frames. ], batch size: 154, lr: 9.16e-03, grad_scale: 32.0 2026-09-24 00:47:29,269 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=70653.33333333333, ans=0.125 2026-09-24 00:47:34,892 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.16 vs. limit=15.0 2026-09-24 00:47:41,017 INFO [train.py:1192] (0/2) Epoch 23, batch 150, loss[loss=0.2735, simple_loss=0.3692, pruned_loss=0.08888, over 24266.00 frames. ], tot_loss[loss=0.3119, simple_loss=0.4134, pruned_loss=0.1052, over 2557733.51 frames. ], batch size: 125, lr: 9.15e-03, grad_scale: 32.0 2026-09-24 00:47:41,480 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.365e+02 3.149e+02 3.599e+02 4.194e+02 5.146e+02, threshold=7.197e+02, percent-clipped=0.0 2026-09-24 00:47:42,641 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.98 vs. limit=22.5 2026-09-24 00:47:57,817 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=70853.33333333333, ans=0.035 2026-09-24 00:48:01,062 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=70886.66666666667, ans=0.1 2026-09-24 00:48:06,369 INFO [train.py:1192] (0/2) Epoch 23, batch 200, loss[loss=0.3983, simple_loss=0.4677, pruned_loss=0.1645, over 21116.00 frames. ], tot_loss[loss=0.3108, simple_loss=0.412, pruned_loss=0.1048, over 3054516.19 frames. ], batch size: 333, lr: 9.14e-03, grad_scale: 32.0 2026-09-24 00:48:22,270 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.35 vs. limit=15.0 2026-09-24 00:48:27,408 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=71053.33333333333, ans=0.0 2026-09-24 00:48:29,524 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:48:31,588 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=7.85 vs. limit=15.0 2026-09-24 00:48:31,769 INFO [train.py:1192] (0/2) Epoch 23, batch 250, loss[loss=0.3375, simple_loss=0.4444, pruned_loss=0.1153, over 24304.00 frames. ], tot_loss[loss=0.3079, simple_loss=0.4099, pruned_loss=0.103, over 3444377.72 frames. ], batch size: 234, lr: 9.13e-03, grad_scale: 32.0 2026-09-24 00:48:32,325 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.652e+02 3.248e+02 3.854e+02 4.393e+02 1.028e+03, threshold=7.707e+02, percent-clipped=3.0 2026-09-24 00:48:34,132 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=71086.66666666667, ans=0.0 2026-09-24 00:48:37,988 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=71120.0, ans=0.0 2026-09-24 00:48:39,836 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=71120.0, ans=0.125 2026-09-24 00:48:49,601 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=71186.66666666667, ans=0.0 2026-09-24 00:48:52,611 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=71220.0, ans=0.07 2026-09-24 00:48:57,977 INFO [train.py:1192] (0/2) Epoch 23, batch 300, loss[loss=0.3524, simple_loss=0.4486, pruned_loss=0.128, over 24530.00 frames. ], tot_loss[loss=0.307, simple_loss=0.4087, pruned_loss=0.1026, over 3753273.79 frames. ], batch size: 204, lr: 9.12e-03, grad_scale: 32.0 2026-09-24 00:49:00,694 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:49:00,798 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=22.5 2026-09-24 00:49:18,918 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=71386.66666666667, ans=0.125 2026-09-24 00:49:23,871 INFO [train.py:1192] (0/2) Epoch 23, batch 350, loss[loss=0.2592, simple_loss=0.3621, pruned_loss=0.07811, over 24561.00 frames. ], tot_loss[loss=0.3075, simple_loss=0.4096, pruned_loss=0.1027, over 3998129.84 frames. ], batch size: 137, lr: 9.11e-03, grad_scale: 32.0 2026-09-24 00:49:24,398 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.415e+02 3.149e+02 3.538e+02 4.133e+02 6.571e+02, threshold=7.075e+02, percent-clipped=0.0 2026-09-24 00:49:25,966 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=71420.0, ans=0.125 2026-09-24 00:49:30,424 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.00 vs. limit=12.0 2026-09-24 00:49:32,146 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=71453.33333333333, ans=0.125 2026-09-24 00:49:37,326 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=71486.66666666667, ans=0.1 2026-09-24 00:49:49,060 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=71586.66666666667, ans=0.0 2026-09-24 00:49:49,539 INFO [train.py:1192] (0/2) Epoch 23, batch 400, loss[loss=0.3335, simple_loss=0.4287, pruned_loss=0.1192, over 24559.00 frames. ], tot_loss[loss=0.3056, simple_loss=0.4079, pruned_loss=0.1017, over 4182562.42 frames. ], batch size: 170, lr: 9.10e-03, grad_scale: 32.0 2026-09-24 00:49:52,027 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=71586.66666666667, ans=0.1 2026-09-24 00:50:02,709 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=71653.33333333333, ans=10.0 2026-09-24 00:50:15,352 INFO [train.py:1192] (0/2) Epoch 23, batch 450, loss[loss=0.2949, simple_loss=0.412, pruned_loss=0.08885, over 24635.00 frames. ], tot_loss[loss=0.306, simple_loss=0.4083, pruned_loss=0.1019, over 4315798.44 frames. ], batch size: 175, lr: 9.09e-03, grad_scale: 32.0 2026-09-24 00:50:15,792 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.435e+02 3.309e+02 3.663e+02 4.121e+02 6.849e+02, threshold=7.325e+02, percent-clipped=0.0 2026-09-24 00:50:24,389 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=71786.66666666667, ans=0.2 2026-09-24 00:50:36,955 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.22 vs. limit=15.0 2026-09-24 00:50:41,172 INFO [train.py:1192] (0/2) Epoch 23, batch 500, loss[loss=0.3347, simple_loss=0.4436, pruned_loss=0.1129, over 24512.00 frames. ], tot_loss[loss=0.3054, simple_loss=0.4072, pruned_loss=0.1018, over 4432504.48 frames. ], batch size: 218, lr: 9.08e-03, grad_scale: 32.0 2026-09-24 00:50:41,722 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=71920.0, ans=0.0 2026-09-24 00:50:54,778 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.37 vs. limit=15.0 2026-09-24 00:50:55,150 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=71986.66666666667, ans=0.0 2026-09-24 00:51:07,065 INFO [train.py:1192] (0/2) Epoch 23, batch 550, loss[loss=0.3309, simple_loss=0.4425, pruned_loss=0.1096, over 24251.00 frames. ], tot_loss[loss=0.3048, simple_loss=0.4071, pruned_loss=0.1012, over 4521373.69 frames. ], batch size: 257, lr: 9.08e-03, grad_scale: 32.0 2026-09-24 00:51:07,605 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.625e+02 3.353e+02 3.894e+02 4.521e+02 6.252e+02, threshold=7.789e+02, percent-clipped=0.0 2026-09-24 00:51:08,311 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=72086.66666666667, ans=0.0 2026-09-24 00:51:20,512 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=72153.33333333333, ans=0.125 2026-09-24 00:51:21,543 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=72153.33333333333, ans=0.0 2026-09-24 00:51:22,729 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=72186.66666666667, ans=0.0 2026-09-24 00:51:25,932 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=72186.66666666667, ans=0.125 2026-09-24 00:51:32,633 INFO [train.py:1192] (0/2) Epoch 23, batch 600, loss[loss=0.3288, simple_loss=0.4386, pruned_loss=0.1095, over 24311.00 frames. ], tot_loss[loss=0.3049, simple_loss=0.4074, pruned_loss=0.1012, over 4586839.62 frames. ], batch size: 234, lr: 9.07e-03, grad_scale: 32.0 2026-09-24 00:51:36,083 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=72253.33333333333, ans=0.125 2026-09-24 00:51:48,689 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:51:52,912 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=72386.66666666667, ans=0.125 2026-09-24 00:51:53,808 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.23 vs. limit=22.5 2026-09-24 00:51:57,954 INFO [train.py:1192] (0/2) Epoch 23, batch 650, loss[loss=0.3029, simple_loss=0.4044, pruned_loss=0.1007, over 24568.00 frames. ], tot_loss[loss=0.3033, simple_loss=0.4062, pruned_loss=0.1003, over 4651888.39 frames. ], batch size: 162, lr: 9.06e-03, grad_scale: 32.0 2026-09-24 00:51:58,344 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.562e+02 3.127e+02 3.494e+02 4.174e+02 5.653e+02, threshold=6.988e+02, percent-clipped=0.0 2026-09-24 00:52:07,079 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=72453.33333333333, ans=0.2 2026-09-24 00:52:10,047 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=72486.66666666667, ans=0.1 2026-09-24 00:52:24,009 INFO [train.py:1192] (0/2) Epoch 23, batch 700, loss[loss=0.2886, simple_loss=0.3889, pruned_loss=0.09412, over 24557.00 frames. ], tot_loss[loss=0.3042, simple_loss=0.4071, pruned_loss=0.1006, over 4685304.55 frames. ], batch size: 154, lr: 9.05e-03, grad_scale: 32.0 2026-09-24 00:52:25,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=72586.66666666667, ans=0.125 2026-09-24 00:52:26,098 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=72586.66666666667, ans=0.125 2026-09-24 00:52:43,562 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=72686.66666666667, ans=0.125 2026-09-24 00:52:44,895 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=72720.0, ans=0.1 2026-09-24 00:52:46,757 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=72720.0, ans=0.0 2026-09-24 00:52:49,652 INFO [train.py:1192] (0/2) Epoch 23, batch 750, loss[loss=0.2903, simple_loss=0.4033, pruned_loss=0.08862, over 24551.00 frames. ], tot_loss[loss=0.3032, simple_loss=0.4059, pruned_loss=0.1002, over 4713253.93 frames. ], batch size: 170, lr: 9.04e-03, grad_scale: 32.0 2026-09-24 00:52:49,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=72753.33333333333, ans=0.125 2026-09-24 00:52:50,106 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.747e+02 3.315e+02 3.683e+02 4.146e+02 6.702e+02, threshold=7.366e+02, percent-clipped=0.0 2026-09-24 00:52:53,709 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=72753.33333333333, ans=0.025 2026-09-24 00:53:00,665 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=72820.0, ans=0.0 2026-09-24 00:53:04,118 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=72820.0, ans=0.2 2026-09-24 00:53:15,296 INFO [train.py:1192] (0/2) Epoch 23, batch 800, loss[loss=0.2674, simple_loss=0.3704, pruned_loss=0.08223, over 24550.00 frames. ], tot_loss[loss=0.3025, simple_loss=0.4055, pruned_loss=0.09978, over 4742033.95 frames. ], batch size: 137, lr: 9.03e-03, grad_scale: 32.0 2026-09-24 00:53:24,612 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=72953.33333333333, ans=0.0 2026-09-24 00:53:31,681 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=73020.0, ans=0.0 2026-09-24 00:53:32,754 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=73020.0, ans=0.1 2026-09-24 00:53:40,873 INFO [train.py:1192] (0/2) Epoch 23, batch 850, loss[loss=0.3414, simple_loss=0.4407, pruned_loss=0.121, over 24576.00 frames. ], tot_loss[loss=0.3025, simple_loss=0.4054, pruned_loss=0.09982, over 4762662.56 frames. ], batch size: 204, lr: 9.02e-03, grad_scale: 32.0 2026-09-24 00:53:41,370 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.638e+02 3.291e+02 3.719e+02 4.187e+02 5.970e+02, threshold=7.437e+02, percent-clipped=0.0 2026-09-24 00:53:59,182 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=73186.66666666667, ans=0.125 2026-09-24 00:54:02,923 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.36 vs. limit=15.0 2026-09-24 00:54:07,476 INFO [train.py:1192] (0/2) Epoch 23, batch 900, loss[loss=0.252, simple_loss=0.362, pruned_loss=0.07098, over 24529.00 frames. ], tot_loss[loss=0.3037, simple_loss=0.4064, pruned_loss=0.1005, over 4775609.82 frames. ], batch size: 137, lr: 9.01e-03, grad_scale: 32.0 2026-09-24 00:54:08,466 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=73253.33333333333, ans=0.1 2026-09-24 00:54:14,112 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=73286.66666666667, ans=0.0 2026-09-24 00:54:18,593 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=73320.0, ans=0.025 2026-09-24 00:54:32,535 INFO [train.py:1192] (0/2) Epoch 23, batch 950, loss[loss=0.3985, simple_loss=0.4429, pruned_loss=0.1771, over 11033.00 frames. ], tot_loss[loss=0.3049, simple_loss=0.4059, pruned_loss=0.102, over 4712115.28 frames. ], batch size: 333, lr: 9.00e-03, grad_scale: 32.0 2026-09-24 00:54:33,096 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.255e+02 3.474e+02 3.945e+02 4.433e+02 7.088e+02, threshold=7.889e+02, percent-clipped=0.0 2026-09-24 00:54:35,054 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=73420.0, ans=0.2 2026-09-24 00:54:35,497 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=73420.0, ans=0.0 2026-09-24 00:54:35,847 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=15.43 vs. limit=15.0 2026-09-24 00:54:37,012 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-23.pt 2026-09-24 00:54:43,064 INFO [train.py:1192] (0/2) Epoch 24, batch 0, loss[loss=0.2753, simple_loss=0.3818, pruned_loss=0.0844, over 24554.00 frames. ], tot_loss[loss=0.2753, simple_loss=0.3818, pruned_loss=0.0844, over 24554.00 frames. ], batch size: 137, lr: 8.81e-03, grad_scale: 32.0 2026-09-24 00:54:43,064 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 00:54:46,116 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.2235, 2.4963, 3.7629, 1.5986], device='cuda:0') 2026-09-24 00:54:50,475 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.6777, 2.8376, 2.3725, 3.6542], device='cuda:0') 2026-09-24 00:54:54,495 INFO [train.py:1224] (0/2) Epoch 24, validation: loss=0.1877, simple_loss=0.3051, pruned_loss=0.03517, over 2564189.00 frames. 2026-09-24 00:54:54,495 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 00:54:56,107 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=73446.66666666667, ans=0.1 2026-09-24 00:54:57,577 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=73446.66666666667, ans=0.025 2026-09-24 00:55:03,578 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.78 vs. limit=6.0 2026-09-24 00:55:05,846 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:55:14,271 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=73546.66666666667, ans=0.125 2026-09-24 00:55:18,669 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=73580.0, ans=0.125 2026-09-24 00:55:20,090 INFO [train.py:1192] (0/2) Epoch 24, batch 50, loss[loss=0.2794, simple_loss=0.3758, pruned_loss=0.09151, over 24281.00 frames. ], tot_loss[loss=0.3128, simple_loss=0.4137, pruned_loss=0.1059, over 1076234.58 frames. ], batch size: 125, lr: 8.80e-03, grad_scale: 32.0 2026-09-24 00:55:20,170 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=73613.33333333333, ans=0.1 2026-09-24 00:55:26,274 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.76 vs. limit=22.5 2026-09-24 00:55:31,868 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=73680.0, ans=0.2 2026-09-24 00:55:32,405 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=73680.0, ans=0.1 2026-09-24 00:55:39,750 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=73713.33333333333, ans=0.2 2026-09-24 00:55:42,124 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.601e+02 3.263e+02 3.527e+02 3.981e+02 7.523e+02, threshold=7.053e+02, percent-clipped=0.0 2026-09-24 00:55:45,662 INFO [train.py:1192] (0/2) Epoch 24, batch 100, loss[loss=0.3081, simple_loss=0.4041, pruned_loss=0.1061, over 24606.00 frames. ], tot_loss[loss=0.3146, simple_loss=0.4164, pruned_loss=0.1064, over 1904531.46 frames. ], batch size: 154, lr: 8.79e-03, grad_scale: 32.0 2026-09-24 00:55:50,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=73813.33333333333, ans=0.1 2026-09-24 00:55:58,704 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.58 vs. limit=22.5 2026-09-24 00:55:59,409 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=73846.66666666667, ans=0.05 2026-09-24 00:56:05,319 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.41 vs. limit=15.0 2026-09-24 00:56:07,732 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=73913.33333333333, ans=0.125 2026-09-24 00:56:11,246 INFO [train.py:1192] (0/2) Epoch 24, batch 150, loss[loss=0.2403, simple_loss=0.3418, pruned_loss=0.06943, over 24308.00 frames. ], tot_loss[loss=0.3083, simple_loss=0.4108, pruned_loss=0.1029, over 2558820.79 frames. ], batch size: 125, lr: 8.78e-03, grad_scale: 32.0 2026-09-24 00:56:15,321 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=73946.66666666667, ans=0.1 2026-09-24 00:56:15,326 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=73946.66666666667, ans=0.1 2026-09-24 00:56:17,280 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=73980.0, ans=0.0 2026-09-24 00:56:20,790 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=74013.33333333333, ans=0.025 2026-09-24 00:56:24,990 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=74013.33333333333, ans=0.2 2026-09-24 00:56:31,629 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=74080.0, ans=0.125 2026-09-24 00:56:33,203 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.517e+02 3.199e+02 3.618e+02 4.148e+02 6.081e+02, threshold=7.236e+02, percent-clipped=0.0 2026-09-24 00:56:36,649 INFO [train.py:1192] (0/2) Epoch 24, batch 200, loss[loss=0.3875, simple_loss=0.4576, pruned_loss=0.1587, over 21060.00 frames. ], tot_loss[loss=0.3061, simple_loss=0.4086, pruned_loss=0.1017, over 3055014.41 frames. ], batch size: 333, lr: 8.77e-03, grad_scale: 32.0 2026-09-24 00:56:38,446 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.69 vs. limit=15.0 2026-09-24 00:56:42,452 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=74146.66666666667, ans=0.0 2026-09-24 00:56:46,941 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=74180.0, ans=0.0 2026-09-24 00:56:54,178 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=74213.33333333333, ans=0.125 2026-09-24 00:56:54,765 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=74213.33333333333, ans=0.125 2026-09-24 00:56:57,958 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=74246.66666666667, ans=0.05 2026-09-24 00:57:01,297 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=74246.66666666667, ans=0.125 2026-09-24 00:57:01,389 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.87 vs. limit=15.0 2026-09-24 00:57:02,135 INFO [train.py:1192] (0/2) Epoch 24, batch 250, loss[loss=0.327, simple_loss=0.4415, pruned_loss=0.1062, over 24326.00 frames. ], tot_loss[loss=0.3042, simple_loss=0.4073, pruned_loss=0.1005, over 3443482.68 frames. ], batch size: 234, lr: 8.76e-03, grad_scale: 32.0 2026-09-24 00:57:02,228 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=74280.0, ans=0.0 2026-09-24 00:57:10,033 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=74313.33333333333, ans=0.125 2026-09-24 00:57:15,393 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=74346.66666666667, ans=0.1 2026-09-24 00:57:23,987 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.559e+02 3.355e+02 3.739e+02 4.396e+02 1.004e+03, threshold=7.479e+02, percent-clipped=1.0 2026-09-24 00:57:27,460 INFO [train.py:1192] (0/2) Epoch 24, batch 300, loss[loss=0.3064, simple_loss=0.4249, pruned_loss=0.09399, over 24531.00 frames. ], tot_loss[loss=0.304, simple_loss=0.4064, pruned_loss=0.1008, over 3749652.29 frames. ], batch size: 204, lr: 8.75e-03, grad_scale: 64.0 2026-09-24 00:57:28,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=74446.66666666667, ans=0.125 2026-09-24 00:57:30,750 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=74446.66666666667, ans=0.125 2026-09-24 00:57:44,356 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.56 vs. limit=22.5 2026-09-24 00:57:49,213 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=74580.0, ans=0.125 2026-09-24 00:57:52,825 INFO [train.py:1192] (0/2) Epoch 24, batch 350, loss[loss=0.2549, simple_loss=0.3536, pruned_loss=0.07812, over 24554.00 frames. ], tot_loss[loss=0.3034, simple_loss=0.4064, pruned_loss=0.1002, over 3991788.13 frames. ], batch size: 137, lr: 8.74e-03, grad_scale: 32.0 2026-09-24 00:57:54,707 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.70 vs. limit=15.0 2026-09-24 00:58:07,754 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=74680.0, ans=22.5 2026-09-24 00:58:08,323 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=74713.33333333333, ans=0.125 2026-09-24 00:58:15,648 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.641e+02 3.311e+02 3.647e+02 4.189e+02 5.156e+02, threshold=7.294e+02, percent-clipped=0.0 2026-09-24 00:58:18,607 INFO [train.py:1192] (0/2) Epoch 24, batch 400, loss[loss=0.2973, simple_loss=0.4091, pruned_loss=0.09278, over 24551.00 frames. ], tot_loss[loss=0.3018, simple_loss=0.4051, pruned_loss=0.09927, over 4177683.68 frames. ], batch size: 170, lr: 8.74e-03, grad_scale: 32.0 2026-09-24 00:58:19,232 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 00:58:26,709 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.48 vs. limit=15.0 2026-09-24 00:58:29,914 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=74846.66666666667, ans=0.0 2026-09-24 00:58:35,509 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=74880.0, ans=0.125 2026-09-24 00:58:38,772 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=74880.0, ans=0.0 2026-09-24 00:58:39,634 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=74913.33333333333, ans=0.125 2026-09-24 00:58:44,654 INFO [train.py:1192] (0/2) Epoch 24, batch 450, loss[loss=0.3297, simple_loss=0.4289, pruned_loss=0.1152, over 24649.00 frames. ], tot_loss[loss=0.3026, simple_loss=0.4058, pruned_loss=0.09975, over 4312080.05 frames. ], batch size: 175, lr: 8.73e-03, grad_scale: 32.0 2026-09-24 00:58:51,731 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=74980.0, ans=0.0 2026-09-24 00:58:56,972 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=75013.33333333333, ans=0.125 2026-09-24 00:59:06,523 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.377e+02 3.242e+02 3.715e+02 4.285e+02 7.156e+02, threshold=7.431e+02, percent-clipped=0.0 2026-09-24 00:59:09,120 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.90 vs. limit=10.0 2026-09-24 00:59:09,838 INFO [train.py:1192] (0/2) Epoch 24, batch 500, loss[loss=0.3183, simple_loss=0.4246, pruned_loss=0.106, over 24527.00 frames. ], tot_loss[loss=0.3017, simple_loss=0.4045, pruned_loss=0.09947, over 4429082.68 frames. ], batch size: 218, lr: 8.72e-03, grad_scale: 32.0 2026-09-24 00:59:10,417 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=75113.33333333333, ans=0.125 2026-09-24 00:59:34,565 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=8.46 vs. limit=15.0 2026-09-24 00:59:34,963 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=75246.66666666667, ans=0.0 2026-09-24 00:59:35,797 INFO [train.py:1192] (0/2) Epoch 24, batch 550, loss[loss=0.3006, simple_loss=0.4154, pruned_loss=0.09293, over 24284.00 frames. ], tot_loss[loss=0.3022, simple_loss=0.405, pruned_loss=0.0997, over 4518918.60 frames. ], batch size: 257, lr: 8.71e-03, grad_scale: 32.0 2026-09-24 00:59:39,106 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=75280.0, ans=0.2 2026-09-24 00:59:40,064 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=75280.0, ans=0.1 2026-09-24 00:59:41,988 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=75313.33333333333, ans=0.0 2026-09-24 00:59:49,712 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.23 vs. limit=12.0 2026-09-24 00:59:49,996 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=75346.66666666667, ans=0.125 2026-09-24 00:59:53,056 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=75380.0, ans=0.2 2026-09-24 00:59:58,732 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.456e+02 3.257e+02 3.612e+02 4.275e+02 6.072e+02, threshold=7.224e+02, percent-clipped=0.0 2026-09-24 01:00:01,509 INFO [train.py:1192] (0/2) Epoch 24, batch 600, loss[loss=0.3126, simple_loss=0.4345, pruned_loss=0.09535, over 24302.00 frames. ], tot_loss[loss=0.3028, simple_loss=0.4057, pruned_loss=0.0999, over 4585286.92 frames. ], batch size: 234, lr: 8.70e-03, grad_scale: 32.0 2026-09-24 01:00:07,367 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=75480.0, ans=0.125 2026-09-24 01:00:09,891 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=75480.0, ans=0.1 2026-09-24 01:00:13,002 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.11 vs. limit=15.0 2026-09-24 01:00:22,026 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=75580.0, ans=0.2 2026-09-24 01:00:22,800 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=7.01 vs. limit=10.0 2026-09-24 01:00:27,023 INFO [train.py:1192] (0/2) Epoch 24, batch 650, loss[loss=0.2914, simple_loss=0.3995, pruned_loss=0.0916, over 24551.00 frames. ], tot_loss[loss=0.3009, simple_loss=0.4043, pruned_loss=0.0987, over 4650879.63 frames. ], batch size: 162, lr: 8.69e-03, grad_scale: 32.0 2026-09-24 01:00:28,223 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=75613.33333333333, ans=0.1 2026-09-24 01:00:28,469 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.46 vs. limit=15.0 2026-09-24 01:00:38,866 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=75680.0, ans=0.07 2026-09-24 01:00:42,318 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=75713.33333333333, ans=0.025 2026-09-24 01:00:42,339 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=75713.33333333333, ans=0.125 2026-09-24 01:00:42,849 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=75713.33333333333, ans=0.025 2026-09-24 01:00:43,108 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.68 vs. limit=12.0 2026-09-24 01:00:50,075 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.734e+02 3.308e+02 3.642e+02 4.382e+02 5.994e+02, threshold=7.284e+02, percent-clipped=0.0 2026-09-24 01:00:50,673 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=75746.66666666667, ans=0.015 2026-09-24 01:00:53,193 INFO [train.py:1192] (0/2) Epoch 24, batch 700, loss[loss=0.2754, simple_loss=0.3791, pruned_loss=0.08584, over 24571.00 frames. ], tot_loss[loss=0.3003, simple_loss=0.4043, pruned_loss=0.09813, over 4682953.98 frames. ], batch size: 154, lr: 8.68e-03, grad_scale: 32.0 2026-09-24 01:00:54,764 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=75780.0, ans=0.125 2026-09-24 01:00:55,692 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=75780.0, ans=0.05 2026-09-24 01:01:12,842 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=75880.0, ans=0.015 2026-09-24 01:01:12,897 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=75880.0, ans=0.1 2026-09-24 01:01:18,585 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=75913.33333333333, ans=0.025 2026-09-24 01:01:19,355 INFO [train.py:1192] (0/2) Epoch 24, batch 750, loss[loss=0.3071, simple_loss=0.4178, pruned_loss=0.09823, over 24564.00 frames. ], tot_loss[loss=0.3002, simple_loss=0.4038, pruned_loss=0.09828, over 4710022.44 frames. ], batch size: 170, lr: 8.68e-03, grad_scale: 32.0 2026-09-24 01:01:28,154 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=75980.0, ans=0.125 2026-09-24 01:01:33,594 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.55 vs. limit=10.0 2026-09-24 01:01:42,462 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.581e+02 3.355e+02 3.755e+02 4.221e+02 7.670e+02, threshold=7.510e+02, percent-clipped=2.0 2026-09-24 01:01:45,983 INFO [train.py:1192] (0/2) Epoch 24, batch 800, loss[loss=0.2646, simple_loss=0.3629, pruned_loss=0.08315, over 24543.00 frames. ], tot_loss[loss=0.2998, simple_loss=0.4035, pruned_loss=0.09809, over 4735250.65 frames. ], batch size: 137, lr: 8.67e-03, grad_scale: 32.0 2026-09-24 01:01:47,173 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.21 vs. limit=12.0 2026-09-24 01:01:49,382 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=76113.33333333333, ans=0.125 2026-09-24 01:01:50,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=76146.66666666667, ans=0.025 2026-09-24 01:01:54,742 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=76146.66666666667, ans=0.125 2026-09-24 01:01:58,578 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=76180.0, ans=15.0 2026-09-24 01:01:59,336 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=76180.0, ans=0.0 2026-09-24 01:02:11,027 INFO [train.py:1192] (0/2) Epoch 24, batch 850, loss[loss=0.3052, simple_loss=0.4177, pruned_loss=0.09633, over 24601.00 frames. ], tot_loss[loss=0.2998, simple_loss=0.4033, pruned_loss=0.09817, over 4757823.78 frames. ], batch size: 198, lr: 8.66e-03, grad_scale: 32.0 2026-09-24 01:02:15,257 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=76280.0, ans=0.1 2026-09-24 01:02:27,735 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=76380.0, ans=0.125 2026-09-24 01:02:33,951 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.514e+02 3.189e+02 3.492e+02 4.274e+02 6.534e+02, threshold=6.983e+02, percent-clipped=0.0 2026-09-24 01:02:34,946 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=76413.33333333333, ans=0.125 2026-09-24 01:02:35,259 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.75 vs. limit=15.0 2026-09-24 01:02:37,264 INFO [train.py:1192] (0/2) Epoch 24, batch 900, loss[loss=0.2553, simple_loss=0.365, pruned_loss=0.07275, over 24544.00 frames. ], tot_loss[loss=0.3011, simple_loss=0.4043, pruned_loss=0.0989, over 4771952.09 frames. ], batch size: 137, lr: 8.65e-03, grad_scale: 32.0 2026-09-24 01:02:38,078 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.96 vs. limit=22.5 2026-09-24 01:02:40,298 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=76446.66666666667, ans=0.125 2026-09-24 01:02:42,493 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.79 vs. limit=22.5 2026-09-24 01:02:44,941 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.40 vs. limit=15.0 2026-09-24 01:02:49,210 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=76513.33333333333, ans=0.125 2026-09-24 01:02:54,319 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=76546.66666666667, ans=0.125 2026-09-24 01:02:57,844 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=76580.0, ans=0.125 2026-09-24 01:03:02,344 INFO [train.py:1192] (0/2) Epoch 24, batch 950, loss[loss=0.4482, simple_loss=0.4657, pruned_loss=0.2153, over 11214.00 frames. ], tot_loss[loss=0.3024, simple_loss=0.4038, pruned_loss=0.1005, over 4709339.26 frames. ], batch size: 333, lr: 8.64e-03, grad_scale: 32.0 2026-09-24 01:03:04,385 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=20.55 vs. limit=22.5 2026-09-24 01:03:04,791 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=76613.33333333333, ans=0.07 2026-09-24 01:03:06,698 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-24.pt 2026-09-24 01:03:14,039 INFO [train.py:1192] (0/2) Epoch 25, batch 0, loss[loss=0.2674, simple_loss=0.3758, pruned_loss=0.07948, over 24556.00 frames. ], tot_loss[loss=0.2674, simple_loss=0.3758, pruned_loss=0.07948, over 24556.00 frames. ], batch size: 137, lr: 8.46e-03, grad_scale: 32.0 2026-09-24 01:03:14,039 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 01:03:25,462 INFO [train.py:1224] (0/2) Epoch 25, validation: loss=0.1888, simple_loss=0.3056, pruned_loss=0.03594, over 2564189.00 frames. 2026-09-24 01:03:25,462 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 01:03:26,063 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=76640.0, ans=0.2 2026-09-24 01:03:36,719 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=76706.66666666667, ans=0.1 2026-09-24 01:03:41,748 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=8.00 vs. limit=12.0 2026-09-24 01:03:43,896 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.648e+02 3.558e+02 3.928e+02 4.511e+02 6.708e+02, threshold=7.857e+02, percent-clipped=0.0 2026-09-24 01:03:50,047 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=76773.33333333333, ans=0.1 2026-09-24 01:03:50,126 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.55 vs. limit=15.0 2026-09-24 01:03:50,444 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=76806.66666666667, ans=0.1 2026-09-24 01:03:50,899 INFO [train.py:1192] (0/2) Epoch 25, batch 50, loss[loss=0.2964, simple_loss=0.3783, pruned_loss=0.1072, over 24265.00 frames. ], tot_loss[loss=0.3175, simple_loss=0.4173, pruned_loss=0.1089, over 1076372.29 frames. ], batch size: 125, lr: 8.45e-03, grad_scale: 32.0 2026-09-24 01:03:51,008 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=76806.66666666667, ans=0.2 2026-09-24 01:03:52,199 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.93 vs. limit=15.0 2026-09-24 01:04:00,921 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=76873.33333333333, ans=0.125 2026-09-24 01:04:08,180 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=76906.66666666667, ans=0.2 2026-09-24 01:04:16,385 INFO [train.py:1192] (0/2) Epoch 25, batch 100, loss[loss=0.2859, simple_loss=0.3876, pruned_loss=0.09211, over 24589.00 frames. ], tot_loss[loss=0.3151, simple_loss=0.4174, pruned_loss=0.1064, over 1903461.99 frames. ], batch size: 154, lr: 8.45e-03, grad_scale: 32.0 2026-09-24 01:04:33,858 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=77073.33333333333, ans=10.0 2026-09-24 01:04:34,991 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.604e+02 3.288e+02 3.807e+02 4.554e+02 6.437e+02, threshold=7.614e+02, percent-clipped=0.0 2026-09-24 01:04:41,942 INFO [train.py:1192] (0/2) Epoch 25, batch 150, loss[loss=0.2479, simple_loss=0.3519, pruned_loss=0.07192, over 24302.00 frames. ], tot_loss[loss=0.309, simple_loss=0.4115, pruned_loss=0.1032, over 2558118.13 frames. ], batch size: 125, lr: 8.44e-03, grad_scale: 32.0 2026-09-24 01:04:47,543 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=77173.33333333333, ans=0.125 2026-09-24 01:04:49,889 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=77173.33333333333, ans=0.1 2026-09-24 01:04:53,017 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=77206.66666666667, ans=0.025 2026-09-24 01:04:59,168 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=77240.0, ans=0.1 2026-09-24 01:05:03,598 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=77273.33333333333, ans=0.0 2026-09-24 01:05:07,705 INFO [train.py:1192] (0/2) Epoch 25, batch 200, loss[loss=0.3891, simple_loss=0.4675, pruned_loss=0.1554, over 20930.00 frames. ], tot_loss[loss=0.3048, simple_loss=0.4082, pruned_loss=0.1007, over 3055371.68 frames. ], batch size: 333, lr: 8.43e-03, grad_scale: 32.0 2026-09-24 01:05:14,719 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=77340.0, ans=0.0 2026-09-24 01:05:25,545 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=77406.66666666667, ans=0.0 2026-09-24 01:05:26,376 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.524e+02 3.418e+02 3.833e+02 4.710e+02 1.132e+03, threshold=7.665e+02, percent-clipped=3.0 2026-09-24 01:05:33,652 INFO [train.py:1192] (0/2) Epoch 25, batch 250, loss[loss=0.3149, simple_loss=0.4295, pruned_loss=0.1002, over 24316.00 frames. ], tot_loss[loss=0.3032, simple_loss=0.4067, pruned_loss=0.09984, over 3445903.59 frames. ], batch size: 234, lr: 8.42e-03, grad_scale: 32.0 2026-09-24 01:05:33,750 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=77473.33333333333, ans=0.125 2026-09-24 01:05:37,411 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=77473.33333333333, ans=0.0 2026-09-24 01:05:44,847 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=77540.0, ans=10.0 2026-09-24 01:05:56,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=77606.66666666667, ans=0.125 2026-09-24 01:05:58,780 INFO [train.py:1192] (0/2) Epoch 25, batch 300, loss[loss=0.3379, simple_loss=0.4439, pruned_loss=0.116, over 24542.00 frames. ], tot_loss[loss=0.302, simple_loss=0.4052, pruned_loss=0.09936, over 3751014.97 frames. ], batch size: 204, lr: 8.41e-03, grad_scale: 32.0 2026-09-24 01:06:00,804 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=77640.0, ans=0.0 2026-09-24 01:06:09,608 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.98 vs. limit=22.5 2026-09-24 01:06:15,665 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=77740.0, ans=0.0 2026-09-24 01:06:17,485 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.405e+02 3.316e+02 3.736e+02 4.252e+02 6.211e+02, threshold=7.471e+02, percent-clipped=0.0 2026-09-24 01:06:17,533 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=77740.0, ans=0.125 2026-09-24 01:06:22,351 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=77773.33333333333, ans=0.0 2026-09-24 01:06:24,744 INFO [train.py:1192] (0/2) Epoch 25, batch 350, loss[loss=0.2886, simple_loss=0.3789, pruned_loss=0.09917, over 24587.00 frames. ], tot_loss[loss=0.302, simple_loss=0.4057, pruned_loss=0.09916, over 3993974.16 frames. ], batch size: 137, lr: 8.40e-03, grad_scale: 32.0 2026-09-24 01:06:30,292 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.92 vs. limit=15.0 2026-09-24 01:06:38,557 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=77873.33333333333, ans=0.125 2026-09-24 01:06:47,539 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=77940.0, ans=0.0 2026-09-24 01:06:50,457 INFO [train.py:1192] (0/2) Epoch 25, batch 400, loss[loss=0.3186, simple_loss=0.4173, pruned_loss=0.11, over 24578.00 frames. ], tot_loss[loss=0.3005, simple_loss=0.4044, pruned_loss=0.09828, over 4181998.65 frames. ], batch size: 170, lr: 8.40e-03, grad_scale: 32.0 2026-09-24 01:06:56,618 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.66 vs. limit=15.0 2026-09-24 01:06:58,507 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.22 vs. limit=15.0 2026-09-24 01:07:03,835 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.99 vs. limit=22.5 2026-09-24 01:07:09,270 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.561e+02 3.215e+02 3.447e+02 4.158e+02 5.856e+02, threshold=6.895e+02, percent-clipped=0.0 2026-09-24 01:07:13,514 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.36 vs. limit=22.5 2026-09-24 01:07:14,546 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.77 vs. limit=22.5 2026-09-24 01:07:16,275 INFO [train.py:1192] (0/2) Epoch 25, batch 450, loss[loss=0.3263, simple_loss=0.4304, pruned_loss=0.1111, over 24617.00 frames. ], tot_loss[loss=0.3007, simple_loss=0.4047, pruned_loss=0.09837, over 4311198.22 frames. ], batch size: 175, lr: 8.39e-03, grad_scale: 32.0 2026-09-24 01:07:22,260 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=78173.33333333333, ans=0.025 2026-09-24 01:07:22,859 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.46 vs. limit=15.0 2026-09-24 01:07:29,496 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=78206.66666666667, ans=0.125 2026-09-24 01:07:31,854 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=78240.0, ans=0.125 2026-09-24 01:07:42,169 INFO [train.py:1192] (0/2) Epoch 25, batch 500, loss[loss=0.3213, simple_loss=0.4381, pruned_loss=0.1022, over 24501.00 frames. ], tot_loss[loss=0.3002, simple_loss=0.4037, pruned_loss=0.09831, over 4428424.99 frames. ], batch size: 218, lr: 8.38e-03, grad_scale: 32.0 2026-09-24 01:07:48,527 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.77 vs. limit=15.0 2026-09-24 01:08:00,941 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.588e+02 3.246e+02 3.675e+02 4.099e+02 7.002e+02, threshold=7.349e+02, percent-clipped=1.0 2026-09-24 01:08:05,033 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=78440.0, ans=0.1 2026-09-24 01:08:05,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=78440.0, ans=0.0 2026-09-24 01:08:08,370 INFO [train.py:1192] (0/2) Epoch 25, batch 550, loss[loss=0.3168, simple_loss=0.4293, pruned_loss=0.1022, over 24304.00 frames. ], tot_loss[loss=0.3007, simple_loss=0.4044, pruned_loss=0.09851, over 4518140.84 frames. ], batch size: 257, lr: 8.37e-03, grad_scale: 32.0 2026-09-24 01:08:08,855 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=78473.33333333333, ans=0.2 2026-09-24 01:08:10,924 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=78473.33333333333, ans=0.125 2026-09-24 01:08:12,542 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=78473.33333333333, ans=0.04949747468305833 2026-09-24 01:08:16,118 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=78506.66666666667, ans=0.1 2026-09-24 01:08:18,904 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=78540.0, ans=0.125 2026-09-24 01:08:27,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=78573.33333333333, ans=0.0 2026-09-24 01:08:27,351 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=78573.33333333333, ans=0.0 2026-09-24 01:08:29,403 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=78606.66666666667, ans=0.125 2026-09-24 01:08:31,715 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=78606.66666666667, ans=0.1 2026-09-24 01:08:32,178 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=78606.66666666667, ans=0.125 2026-09-24 01:08:34,154 INFO [train.py:1192] (0/2) Epoch 25, batch 600, loss[loss=0.306, simple_loss=0.425, pruned_loss=0.0935, over 24434.00 frames. ], tot_loss[loss=0.3011, simple_loss=0.4049, pruned_loss=0.0986, over 4585963.73 frames. ], batch size: 235, lr: 8.36e-03, grad_scale: 32.0 2026-09-24 01:08:46,168 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=78706.66666666667, ans=0.0 2026-09-24 01:08:52,197 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.725e+02 3.335e+02 3.829e+02 4.379e+02 5.837e+02, threshold=7.659e+02, percent-clipped=0.0 2026-09-24 01:08:52,403 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.38 vs. limit=12.0 2026-09-24 01:08:57,081 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=78773.33333333333, ans=0.125 2026-09-24 01:08:59,360 INFO [train.py:1192] (0/2) Epoch 25, batch 650, loss[loss=0.3057, simple_loss=0.4105, pruned_loss=0.1004, over 24552.00 frames. ], tot_loss[loss=0.299, simple_loss=0.4034, pruned_loss=0.09725, over 4650998.93 frames. ], batch size: 162, lr: 8.36e-03, grad_scale: 32.0 2026-09-24 01:08:59,999 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=78806.66666666667, ans=0.035 2026-09-24 01:09:18,349 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=78906.66666666667, ans=0.125 2026-09-24 01:09:19,949 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=78940.0, ans=0.0 2026-09-24 01:09:24,577 INFO [train.py:1192] (0/2) Epoch 25, batch 700, loss[loss=0.2785, simple_loss=0.3831, pruned_loss=0.08695, over 24606.00 frames. ], tot_loss[loss=0.2993, simple_loss=0.4038, pruned_loss=0.09739, over 4684983.69 frames. ], batch size: 154, lr: 8.35e-03, grad_scale: 32.0 2026-09-24 01:09:43,037 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.22 vs. limit=22.5 2026-09-24 01:09:43,174 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.541e+02 3.236e+02 3.745e+02 4.278e+02 6.318e+02, threshold=7.490e+02, percent-clipped=0.0 2026-09-24 01:09:50,974 INFO [train.py:1192] (0/2) Epoch 25, batch 750, loss[loss=0.2919, simple_loss=0.4074, pruned_loss=0.08821, over 24564.00 frames. ], tot_loss[loss=0.2997, simple_loss=0.4036, pruned_loss=0.09789, over 4712052.80 frames. ], batch size: 170, lr: 8.34e-03, grad_scale: 32.0 2026-09-24 01:09:58,356 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=79173.33333333333, ans=0.125 2026-09-24 01:10:00,413 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=79173.33333333333, ans=0.125 2026-09-24 01:10:07,066 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.26 vs. limit=22.5 2026-09-24 01:10:16,673 INFO [train.py:1192] (0/2) Epoch 25, batch 800, loss[loss=0.259, simple_loss=0.3585, pruned_loss=0.07971, over 24513.00 frames. ], tot_loss[loss=0.2984, simple_loss=0.4026, pruned_loss=0.09712, over 4740877.14 frames. ], batch size: 137, lr: 8.33e-03, grad_scale: 32.0 2026-09-24 01:10:35,666 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.734e+02 3.368e+02 3.828e+02 4.294e+02 6.886e+02, threshold=7.657e+02, percent-clipped=0.0 2026-09-24 01:10:38,622 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=79440.0, ans=0.0 2026-09-24 01:10:42,446 INFO [train.py:1192] (0/2) Epoch 25, batch 850, loss[loss=0.2972, simple_loss=0.4164, pruned_loss=0.08901, over 24531.00 frames. ], tot_loss[loss=0.2976, simple_loss=0.402, pruned_loss=0.09655, over 4762096.98 frames. ], batch size: 204, lr: 8.32e-03, grad_scale: 32.0 2026-09-24 01:10:48,461 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=79506.66666666667, ans=0.2 2026-09-24 01:10:49,721 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=12.54 vs. limit=15.0 2026-09-24 01:10:53,849 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=79540.0, ans=0.025 2026-09-24 01:10:57,550 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=79573.33333333333, ans=0.0 2026-09-24 01:11:00,832 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=79573.33333333333, ans=0.125 2026-09-24 01:11:08,248 INFO [train.py:1192] (0/2) Epoch 25, batch 900, loss[loss=0.251, simple_loss=0.3577, pruned_loss=0.07211, over 24572.00 frames. ], tot_loss[loss=0.2972, simple_loss=0.402, pruned_loss=0.09622, over 4775341.07 frames. ], batch size: 137, lr: 8.32e-03, grad_scale: 32.0 2026-09-24 01:11:15,896 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0 2026-09-24 01:11:25,607 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.551e+02 3.146e+02 3.608e+02 4.225e+02 6.295e+02, threshold=7.217e+02, percent-clipped=0.0 2026-09-24 01:11:29,915 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:11:31,707 INFO [train.py:1192] (0/2) Epoch 25, batch 950, loss[loss=0.4309, simple_loss=0.4634, pruned_loss=0.1992, over 11270.00 frames. ], tot_loss[loss=0.2969, simple_loss=0.4005, pruned_loss=0.09669, over 4708893.84 frames. ], batch size: 334, lr: 8.31e-03, grad_scale: 32.0 2026-09-24 01:11:36,425 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-25.pt 2026-09-24 01:11:42,184 INFO [train.py:1192] (0/2) Epoch 26, batch 0, loss[loss=0.2542, simple_loss=0.3644, pruned_loss=0.07204, over 24570.00 frames. ], tot_loss[loss=0.2542, simple_loss=0.3644, pruned_loss=0.07204, over 24570.00 frames. ], batch size: 137, lr: 8.14e-03, grad_scale: 32.0 2026-09-24 01:11:42,184 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 01:11:53,555 INFO [train.py:1224] (0/2) Epoch 26, validation: loss=0.186, simple_loss=0.3043, pruned_loss=0.03381, over 2564189.00 frames. 2026-09-24 01:11:53,555 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 01:11:59,625 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=79866.66666666667, ans=0.025 2026-09-24 01:12:00,649 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=79866.66666666667, ans=0.0 2026-09-24 01:12:01,738 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=79866.66666666667, ans=0.0 2026-09-24 01:12:03,492 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.04 vs. limit=22.5 2026-09-24 01:12:04,396 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=79900.0, ans=0.0 2026-09-24 01:12:19,767 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-24000.pt 2026-09-24 01:12:20,453 INFO [train.py:1192] (0/2) Epoch 26, batch 50, loss[loss=0.2526, simple_loss=0.3571, pruned_loss=0.07409, over 24289.00 frames. ], tot_loss[loss=0.3144, simple_loss=0.4152, pruned_loss=0.1068, over 1076315.64 frames. ], batch size: 125, lr: 8.13e-03, grad_scale: 32.0 2026-09-24 01:12:20,940 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=80000.0, ans=0.0 2026-09-24 01:12:28,423 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=80033.33333333333, ans=0.0 2026-09-24 01:12:35,342 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.648e+02 3.559e+02 3.997e+02 5.046e+02 7.744e+02, threshold=7.995e+02, percent-clipped=3.0 2026-09-24 01:12:43,047 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=80133.33333333333, ans=0.125 2026-09-24 01:12:45,731 INFO [train.py:1192] (0/2) Epoch 26, batch 100, loss[loss=0.3028, simple_loss=0.4002, pruned_loss=0.1027, over 24613.00 frames. ], tot_loss[loss=0.3136, simple_loss=0.4163, pruned_loss=0.1054, over 1905455.28 frames. ], batch size: 154, lr: 8.13e-03, grad_scale: 32.0 2026-09-24 01:12:48,532 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=80166.66666666667, ans=0.0 2026-09-24 01:13:05,428 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.15 vs. limit=15.0 2026-09-24 01:13:11,258 INFO [train.py:1192] (0/2) Epoch 26, batch 150, loss[loss=0.2394, simple_loss=0.347, pruned_loss=0.06584, over 24266.00 frames. ], tot_loss[loss=0.3048, simple_loss=0.4088, pruned_loss=0.1005, over 2557789.65 frames. ], batch size: 125, lr: 8.12e-03, grad_scale: 32.0 2026-09-24 01:13:13,475 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=80333.33333333333, ans=0.0 2026-09-24 01:13:24,383 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=80400.0, ans=0.0 2026-09-24 01:13:25,726 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.558e+02 3.240e+02 3.503e+02 3.960e+02 5.414e+02, threshold=7.006e+02, percent-clipped=0.0 2026-09-24 01:13:36,621 INFO [train.py:1192] (0/2) Epoch 26, batch 200, loss[loss=0.3405, simple_loss=0.4325, pruned_loss=0.1243, over 20982.00 frames. ], tot_loss[loss=0.3016, simple_loss=0.4058, pruned_loss=0.09869, over 3053833.03 frames. ], batch size: 333, lr: 8.11e-03, grad_scale: 32.0 2026-09-24 01:13:36,733 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=80500.0, ans=0.1 2026-09-24 01:13:38,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=80500.0, ans=0.0 2026-09-24 01:13:42,062 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=80533.33333333333, ans=0.125 2026-09-24 01:13:52,717 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=80600.0, ans=0.1 2026-09-24 01:13:55,123 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=80600.0, ans=0.0 2026-09-24 01:14:01,691 INFO [train.py:1192] (0/2) Epoch 26, batch 250, loss[loss=0.3211, simple_loss=0.4362, pruned_loss=0.103, over 24316.00 frames. ], tot_loss[loss=0.2992, simple_loss=0.404, pruned_loss=0.0972, over 3443345.34 frames. ], batch size: 234, lr: 8.10e-03, grad_scale: 32.0 2026-09-24 01:14:16,436 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.260e+02 3.537e+02 4.089e+02 5.149e+02 1.030e+03, threshold=8.177e+02, percent-clipped=7.0 2026-09-24 01:14:23,560 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=80800.0, ans=0.125 2026-09-24 01:14:26,835 INFO [train.py:1192] (0/2) Epoch 26, batch 300, loss[loss=0.3078, simple_loss=0.426, pruned_loss=0.09475, over 24548.00 frames. ], tot_loss[loss=0.297, simple_loss=0.4019, pruned_loss=0.09604, over 3748484.07 frames. ], batch size: 204, lr: 8.10e-03, grad_scale: 32.0 2026-09-24 01:14:34,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=80866.66666666667, ans=0.1 2026-09-24 01:14:44,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=80933.33333333333, ans=0.125 2026-09-24 01:14:52,308 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=81000.0, ans=0.0 2026-09-24 01:14:52,679 INFO [train.py:1192] (0/2) Epoch 26, batch 350, loss[loss=0.2348, simple_loss=0.3432, pruned_loss=0.06319, over 24549.00 frames. ], tot_loss[loss=0.2991, simple_loss=0.4039, pruned_loss=0.09711, over 3992255.72 frames. ], batch size: 137, lr: 8.09e-03, grad_scale: 32.0 2026-09-24 01:14:59,684 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.61 vs. limit=15.0 2026-09-24 01:15:06,925 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.03 vs. limit=15.0 2026-09-24 01:15:07,604 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.513e+02 3.230e+02 3.645e+02 3.996e+02 6.167e+02, threshold=7.290e+02, percent-clipped=0.0 2026-09-24 01:15:18,629 INFO [train.py:1192] (0/2) Epoch 26, batch 400, loss[loss=0.3132, simple_loss=0.4167, pruned_loss=0.1049, over 24550.00 frames. ], tot_loss[loss=0.2982, simple_loss=0.4031, pruned_loss=0.09662, over 4172602.82 frames. ], batch size: 170, lr: 8.08e-03, grad_scale: 32.0 2026-09-24 01:15:30,415 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.95 vs. limit=22.5 2026-09-24 01:15:31,994 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.80 vs. limit=22.5 2026-09-24 01:15:35,704 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=13.05 vs. limit=15.0 2026-09-24 01:15:36,815 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=81266.66666666667, ans=0.2 2026-09-24 01:15:38,537 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=81266.66666666667, ans=0.1 2026-09-24 01:15:41,688 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=81300.0, ans=0.1 2026-09-24 01:15:43,126 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=81300.0, ans=0.1 2026-09-24 01:15:43,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=81300.0, ans=0.0 2026-09-24 01:15:44,951 INFO [train.py:1192] (0/2) Epoch 26, batch 450, loss[loss=0.2735, simple_loss=0.3969, pruned_loss=0.07504, over 24612.00 frames. ], tot_loss[loss=0.2989, simple_loss=0.4038, pruned_loss=0.09697, over 4307785.81 frames. ], batch size: 175, lr: 8.07e-03, grad_scale: 32.0 2026-09-24 01:15:47,053 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=81333.33333333333, ans=0.125 2026-09-24 01:15:47,275 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=81333.33333333333, ans=15.0 2026-09-24 01:15:59,654 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.632e+02 3.397e+02 3.913e+02 4.554e+02 6.749e+02, threshold=7.825e+02, percent-clipped=0.0 2026-09-24 01:16:10,897 INFO [train.py:1192] (0/2) Epoch 26, batch 500, loss[loss=0.3488, simple_loss=0.4516, pruned_loss=0.123, over 24520.00 frames. ], tot_loss[loss=0.298, simple_loss=0.4024, pruned_loss=0.09679, over 4426527.70 frames. ], batch size: 218, lr: 8.07e-03, grad_scale: 32.0 2026-09-24 01:16:16,149 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=81533.33333333333, ans=0.0 2026-09-24 01:16:18,024 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=81533.33333333333, ans=0.125 2026-09-24 01:16:24,512 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=81566.66666666667, ans=0.125 2026-09-24 01:16:34,210 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=81633.33333333333, ans=0.0 2026-09-24 01:16:34,721 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=81633.33333333333, ans=0.125 2026-09-24 01:16:35,698 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=81666.66666666667, ans=0.0 2026-09-24 01:16:36,116 INFO [train.py:1192] (0/2) Epoch 26, batch 550, loss[loss=0.3375, simple_loss=0.4434, pruned_loss=0.1158, over 24258.00 frames. ], tot_loss[loss=0.2977, simple_loss=0.4021, pruned_loss=0.0966, over 4516783.02 frames. ], batch size: 257, lr: 8.06e-03, grad_scale: 32.0 2026-09-24 01:16:51,113 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.714e+02 3.189e+02 3.611e+02 4.230e+02 5.792e+02, threshold=7.223e+02, percent-clipped=0.0 2026-09-24 01:16:56,558 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_abs, batch_count=81800.0, ans=0.5 2026-09-24 01:17:01,246 INFO [train.py:1192] (0/2) Epoch 26, batch 600, loss[loss=0.2981, simple_loss=0.4146, pruned_loss=0.09078, over 24340.00 frames. ], tot_loss[loss=0.2976, simple_loss=0.4023, pruned_loss=0.09642, over 4584279.48 frames. ], batch size: 234, lr: 8.05e-03, grad_scale: 32.0 2026-09-24 01:17:05,895 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.91 vs. limit=22.5 2026-09-24 01:17:08,900 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=81866.66666666667, ans=0.05 2026-09-24 01:17:13,235 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:17:19,190 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.56 vs. limit=22.5 2026-09-24 01:17:24,181 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=81966.66666666667, ans=0.1 2026-09-24 01:17:26,423 INFO [train.py:1192] (0/2) Epoch 26, batch 650, loss[loss=0.3024, simple_loss=0.4057, pruned_loss=0.09957, over 24562.00 frames. ], tot_loss[loss=0.2959, simple_loss=0.401, pruned_loss=0.09538, over 4649921.82 frames. ], batch size: 162, lr: 8.04e-03, grad_scale: 32.0 2026-09-24 01:17:28,153 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=82000.0, ans=0.025 2026-09-24 01:17:41,515 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.485e+02 3.234e+02 3.746e+02 4.393e+02 6.257e+02, threshold=7.492e+02, percent-clipped=0.0 2026-09-24 01:17:52,308 INFO [train.py:1192] (0/2) Epoch 26, batch 700, loss[loss=0.2813, simple_loss=0.3823, pruned_loss=0.09021, over 24566.00 frames. ], tot_loss[loss=0.2961, simple_loss=0.4016, pruned_loss=0.09529, over 4683955.51 frames. ], batch size: 154, lr: 8.04e-03, grad_scale: 32.0 2026-09-24 01:18:17,428 INFO [train.py:1192] (0/2) Epoch 26, batch 750, loss[loss=0.2732, simple_loss=0.3899, pruned_loss=0.07824, over 24532.00 frames. ], tot_loss[loss=0.2958, simple_loss=0.4008, pruned_loss=0.09538, over 4710743.28 frames. ], batch size: 170, lr: 8.03e-03, grad_scale: 32.0 2026-09-24 01:18:32,209 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.741e+02 3.352e+02 3.754e+02 4.234e+02 6.090e+02, threshold=7.507e+02, percent-clipped=0.0 2026-09-24 01:18:33,305 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=82433.33333333333, ans=0.125 2026-09-24 01:18:39,958 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=82466.66666666667, ans=0.0 2026-09-24 01:18:41,240 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=82466.66666666667, ans=0.05 2026-09-24 01:18:43,222 INFO [train.py:1192] (0/2) Epoch 26, batch 800, loss[loss=0.248, simple_loss=0.3573, pruned_loss=0.06933, over 24543.00 frames. ], tot_loss[loss=0.2962, simple_loss=0.4011, pruned_loss=0.09567, over 4736897.56 frames. ], batch size: 137, lr: 8.02e-03, grad_scale: 32.0 2026-09-24 01:18:44,291 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=82500.0, ans=0.1 2026-09-24 01:19:01,109 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=82600.0, ans=0.025 2026-09-24 01:19:05,420 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=82633.33333333333, ans=0.04949747468305833 2026-09-24 01:19:09,073 INFO [train.py:1192] (0/2) Epoch 26, batch 850, loss[loss=0.3242, simple_loss=0.4297, pruned_loss=0.1093, over 24610.00 frames. ], tot_loss[loss=0.2965, simple_loss=0.4012, pruned_loss=0.0959, over 4759045.78 frames. ], batch size: 198, lr: 8.01e-03, grad_scale: 32.0 2026-09-24 01:19:18,280 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=82733.33333333333, ans=0.125 2026-09-24 01:19:23,299 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.687e+02 3.286e+02 3.795e+02 4.177e+02 6.657e+02, threshold=7.591e+02, percent-clipped=0.0 2026-09-24 01:19:28,092 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=82766.66666666667, ans=0.0 2026-09-24 01:19:29,943 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=82800.0, ans=0.1 2026-09-24 01:19:31,279 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.03 vs. limit=15.0 2026-09-24 01:19:34,167 INFO [train.py:1192] (0/2) Epoch 26, batch 900, loss[loss=0.2501, simple_loss=0.364, pruned_loss=0.06807, over 24569.00 frames. ], tot_loss[loss=0.2967, simple_loss=0.4016, pruned_loss=0.0959, over 4773882.98 frames. ], batch size: 137, lr: 8.01e-03, grad_scale: 32.0 2026-09-24 01:19:50,425 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=82933.33333333333, ans=0.0 2026-09-24 01:19:52,241 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:19:56,346 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=82966.66666666667, ans=0.125 2026-09-24 01:19:59,488 INFO [train.py:1192] (0/2) Epoch 26, batch 950, loss[loss=0.4329, simple_loss=0.4684, pruned_loss=0.1987, over 11427.00 frames. ], tot_loss[loss=0.2976, simple_loss=0.4008, pruned_loss=0.09721, over 4711199.50 frames. ], batch size: 334, lr: 8.00e-03, grad_scale: 32.0 2026-09-24 01:19:59,552 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=83000.0, ans=0.1 2026-09-24 01:20:01,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=83000.0, ans=0.125 2026-09-24 01:20:02,526 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=83000.0, ans=0.125 2026-09-24 01:20:03,845 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-26.pt 2026-09-24 01:20:09,649 INFO [train.py:1192] (0/2) Epoch 27, batch 0, loss[loss=0.2573, simple_loss=0.364, pruned_loss=0.07528, over 24570.00 frames. ], tot_loss[loss=0.2573, simple_loss=0.364, pruned_loss=0.07528, over 24570.00 frames. ], batch size: 137, lr: 7.84e-03, grad_scale: 32.0 2026-09-24 01:20:09,649 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 01:20:12,714 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.8260, 3.3052, 3.1489, 1.7487], device='cuda:0') 2026-09-24 01:20:20,459 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([1.4372, 1.4832, 1.7305, 1.4143], device='cuda:0') 2026-09-24 01:20:21,081 INFO [train.py:1224] (0/2) Epoch 27, validation: loss=0.1849, simple_loss=0.3023, pruned_loss=0.03371, over 2564189.00 frames. 2026-09-24 01:20:21,082 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 01:20:21,658 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=83026.66666666667, ans=0.0 2026-09-24 01:20:22,073 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=83026.66666666667, ans=0.0 2026-09-24 01:20:22,590 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=83026.66666666667, ans=0.125 2026-09-24 01:20:31,628 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.571e+02 3.432e+02 4.011e+02 4.668e+02 7.874e+02, threshold=8.023e+02, percent-clipped=1.0 2026-09-24 01:20:39,129 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=83126.66666666667, ans=0.125 2026-09-24 01:20:39,529 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=83126.66666666667, ans=0.125 2026-09-24 01:20:41,707 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=83160.0, ans=0.125 2026-09-24 01:20:44,107 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=83160.0, ans=0.2 2026-09-24 01:20:46,495 INFO [train.py:1192] (0/2) Epoch 27, batch 50, loss[loss=0.2501, simple_loss=0.3521, pruned_loss=0.07402, over 24252.00 frames. ], tot_loss[loss=0.3067, simple_loss=0.4095, pruned_loss=0.1019, over 1076645.90 frames. ], batch size: 125, lr: 7.84e-03, grad_scale: 32.0 2026-09-24 01:20:47,097 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=83193.33333333333, ans=0.0 2026-09-24 01:20:52,848 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=83226.66666666667, ans=0.1 2026-09-24 01:20:55,523 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.47 vs. limit=15.0 2026-09-24 01:20:56,828 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=83260.0, ans=0.95 2026-09-24 01:20:58,846 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.90 vs. limit=10.0 2026-09-24 01:21:01,468 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=83260.0, ans=0.125 2026-09-24 01:21:04,132 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=83293.33333333333, ans=0.125 2026-09-24 01:21:11,458 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=83326.66666666667, ans=0.125 2026-09-24 01:21:12,260 INFO [train.py:1192] (0/2) Epoch 27, batch 100, loss[loss=0.268, simple_loss=0.3761, pruned_loss=0.07995, over 24587.00 frames. ], tot_loss[loss=0.3073, simple_loss=0.4116, pruned_loss=0.1015, over 1905117.57 frames. ], batch size: 154, lr: 7.83e-03, grad_scale: 32.0 2026-09-24 01:21:17,253 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=83393.33333333333, ans=0.125 2026-09-24 01:21:22,899 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.627e+02 3.379e+02 3.839e+02 4.300e+02 6.857e+02, threshold=7.678e+02, percent-clipped=0.0 2026-09-24 01:21:25,378 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.78 vs. limit=15.0 2026-09-24 01:21:37,903 INFO [train.py:1192] (0/2) Epoch 27, batch 150, loss[loss=0.2645, simple_loss=0.3594, pruned_loss=0.08478, over 24278.00 frames. ], tot_loss[loss=0.3011, simple_loss=0.406, pruned_loss=0.09805, over 2559397.01 frames. ], batch size: 125, lr: 7.82e-03, grad_scale: 32.0 2026-09-24 01:21:38,030 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=83526.66666666667, ans=0.0 2026-09-24 01:21:51,187 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=83593.33333333333, ans=0.0 2026-09-24 01:22:03,902 INFO [train.py:1192] (0/2) Epoch 27, batch 200, loss[loss=0.378, simple_loss=0.4552, pruned_loss=0.1504, over 21079.00 frames. ], tot_loss[loss=0.2981, simple_loss=0.4034, pruned_loss=0.09642, over 3057447.58 frames. ], batch size: 333, lr: 7.82e-03, grad_scale: 32.0 2026-09-24 01:22:12,219 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=83726.66666666667, ans=0.0 2026-09-24 01:22:14,488 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.467e+02 3.221e+02 3.661e+02 4.138e+02 9.713e+02, threshold=7.323e+02, percent-clipped=3.0 2026-09-24 01:22:29,061 INFO [train.py:1192] (0/2) Epoch 27, batch 250, loss[loss=0.3166, simple_loss=0.4331, pruned_loss=0.1001, over 24295.00 frames. ], tot_loss[loss=0.297, simple_loss=0.4024, pruned_loss=0.09577, over 3446406.28 frames. ], batch size: 234, lr: 7.81e-03, grad_scale: 32.0 2026-09-24 01:22:44,063 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=83960.0, ans=0.125 2026-09-24 01:22:48,487 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=83960.0, ans=0.0 2026-09-24 01:22:54,052 INFO [train.py:1192] (0/2) Epoch 27, batch 300, loss[loss=0.3224, simple_loss=0.4315, pruned_loss=0.1067, over 24563.00 frames. ], tot_loss[loss=0.2957, simple_loss=0.401, pruned_loss=0.0952, over 3751004.44 frames. ], batch size: 204, lr: 7.80e-03, grad_scale: 32.0 2026-09-24 01:22:55,632 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.64 vs. limit=15.0 2026-09-24 01:22:58,769 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.15 vs. limit=15.0 2026-09-24 01:23:04,241 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.635e+02 3.238e+02 3.753e+02 4.253e+02 6.492e+02, threshold=7.505e+02, percent-clipped=0.0 2026-09-24 01:23:04,322 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=84093.33333333333, ans=0.0 2026-09-24 01:23:11,195 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=84126.66666666667, ans=0.2 2026-09-24 01:23:18,844 INFO [train.py:1192] (0/2) Epoch 27, batch 350, loss[loss=0.267, simple_loss=0.3663, pruned_loss=0.08388, over 24628.00 frames. ], tot_loss[loss=0.2958, simple_loss=0.4014, pruned_loss=0.09503, over 3993777.56 frames. ], batch size: 137, lr: 7.79e-03, grad_scale: 32.0 2026-09-24 01:23:22,843 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=84193.33333333333, ans=0.125 2026-09-24 01:23:24,268 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=84226.66666666667, ans=0.2 2026-09-24 01:23:27,278 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.46 vs. limit=15.0 2026-09-24 01:23:27,757 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=1.986e-02 2026-09-24 01:23:28,816 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=84260.0, ans=0.125 2026-09-24 01:23:29,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=84260.0, ans=0.125 2026-09-24 01:23:44,383 INFO [train.py:1192] (0/2) Epoch 27, batch 400, loss[loss=0.3136, simple_loss=0.422, pruned_loss=0.1026, over 24573.00 frames. ], tot_loss[loss=0.2952, simple_loss=0.4009, pruned_loss=0.0947, over 4180980.98 frames. ], batch size: 170, lr: 7.79e-03, grad_scale: 32.0 2026-09-24 01:23:55,152 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.594e+02 3.209e+02 3.534e+02 4.019e+02 5.305e+02, threshold=7.068e+02, percent-clipped=0.0 2026-09-24 01:24:02,443 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=84460.0, ans=0.0 2026-09-24 01:24:09,696 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:24:09,709 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=84493.33333333333, ans=0.125 2026-09-24 01:24:10,372 INFO [train.py:1192] (0/2) Epoch 27, batch 450, loss[loss=0.2966, simple_loss=0.403, pruned_loss=0.0951, over 24622.00 frames. ], tot_loss[loss=0.2945, simple_loss=0.4006, pruned_loss=0.09423, over 4314442.19 frames. ], batch size: 175, lr: 7.78e-03, grad_scale: 32.0 2026-09-24 01:24:10,491 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=84526.66666666667, ans=0.04949747468305833 2026-09-24 01:24:19,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=84560.0, ans=0.1 2026-09-24 01:24:31,465 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=84660.0, ans=0.125 2026-09-24 01:24:32,690 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.50 vs. limit=22.5 2026-09-24 01:24:35,855 INFO [train.py:1192] (0/2) Epoch 27, batch 500, loss[loss=0.3349, simple_loss=0.4429, pruned_loss=0.1135, over 24530.00 frames. ], tot_loss[loss=0.2929, simple_loss=0.3987, pruned_loss=0.09355, over 4431201.41 frames. ], batch size: 218, lr: 7.77e-03, grad_scale: 32.0 2026-09-24 01:24:46,630 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.511e+02 3.213e+02 3.588e+02 4.289e+02 6.663e+02, threshold=7.177e+02, percent-clipped=0.0 2026-09-24 01:24:51,102 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=84793.33333333333, ans=0.025 2026-09-24 01:25:00,232 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=84826.66666666667, ans=0.1 2026-09-24 01:25:01,612 INFO [train.py:1192] (0/2) Epoch 27, batch 550, loss[loss=0.3052, simple_loss=0.4197, pruned_loss=0.09539, over 24319.00 frames. ], tot_loss[loss=0.2937, simple_loss=0.3994, pruned_loss=0.094, over 4520961.28 frames. ], batch size: 257, lr: 7.77e-03, grad_scale: 32.0 2026-09-24 01:25:05,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=84860.0, ans=0.04949747468305833 2026-09-24 01:25:16,178 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=84960.0, ans=0.1 2026-09-24 01:25:25,715 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.51 vs. limit=22.5 2026-09-24 01:25:27,066 INFO [train.py:1192] (0/2) Epoch 27, batch 600, loss[loss=0.3106, simple_loss=0.4271, pruned_loss=0.09704, over 24318.00 frames. ], tot_loss[loss=0.2938, simple_loss=0.3996, pruned_loss=0.09397, over 4586292.36 frames. ], batch size: 234, lr: 7.76e-03, grad_scale: 32.0 2026-09-24 01:25:37,947 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=85093.33333333333, ans=0.125 2026-09-24 01:25:38,198 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.522e+02 3.192e+02 3.627e+02 4.252e+02 5.650e+02, threshold=7.253e+02, percent-clipped=0.0 2026-09-24 01:25:49,430 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.63 vs. limit=15.0 2026-09-24 01:25:53,051 INFO [train.py:1192] (0/2) Epoch 27, batch 650, loss[loss=0.3075, simple_loss=0.4096, pruned_loss=0.1027, over 24566.00 frames. ], tot_loss[loss=0.293, simple_loss=0.3991, pruned_loss=0.09339, over 4651082.17 frames. ], batch size: 162, lr: 7.75e-03, grad_scale: 32.0 2026-09-24 01:25:59,884 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=85226.66666666667, ans=0.125 2026-09-24 01:26:00,403 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=85226.66666666667, ans=0.125 2026-09-24 01:26:03,090 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=85260.0, ans=0.1 2026-09-24 01:26:04,903 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=85260.0, ans=0.0 2026-09-24 01:26:06,484 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=85260.0, ans=0.04949747468305833 2026-09-24 01:26:13,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=85326.66666666667, ans=0.0 2026-09-24 01:26:18,802 INFO [train.py:1192] (0/2) Epoch 27, batch 700, loss[loss=0.2673, simple_loss=0.3737, pruned_loss=0.08047, over 24563.00 frames. ], tot_loss[loss=0.2937, simple_loss=0.4, pruned_loss=0.09364, over 4683066.31 frames. ], batch size: 154, lr: 7.75e-03, grad_scale: 32.0 2026-09-24 01:26:23,046 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=85360.0, ans=0.125 2026-09-24 01:26:24,590 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=85393.33333333333, ans=0.125 2026-09-24 01:26:25,656 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=85393.33333333333, ans=0.125 2026-09-24 01:26:27,561 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=2.96 vs. limit=15.0 2026-09-24 01:26:29,688 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.715e+02 3.445e+02 3.970e+02 4.680e+02 6.665e+02, threshold=7.940e+02, percent-clipped=0.0 2026-09-24 01:26:35,452 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=85460.0, ans=0.2 2026-09-24 01:26:36,017 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=85460.0, ans=0.1 2026-09-24 01:26:44,381 INFO [train.py:1192] (0/2) Epoch 27, batch 750, loss[loss=0.2805, simple_loss=0.3955, pruned_loss=0.08271, over 24571.00 frames. ], tot_loss[loss=0.2927, simple_loss=0.3989, pruned_loss=0.0932, over 4709864.81 frames. ], batch size: 170, lr: 7.74e-03, grad_scale: 32.0 2026-09-24 01:26:57,932 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=85593.33333333333, ans=0.2 2026-09-24 01:27:03,780 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=85626.66666666667, ans=0.1 2026-09-24 01:27:05,141 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=85660.0, ans=0.07 2026-09-24 01:27:07,328 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=85660.0, ans=0.0 2026-09-24 01:27:07,368 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:27:08,731 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:27:09,815 INFO [train.py:1192] (0/2) Epoch 27, batch 800, loss[loss=0.2781, simple_loss=0.3783, pruned_loss=0.08895, over 24543.00 frames. ], tot_loss[loss=0.2919, simple_loss=0.3983, pruned_loss=0.09277, over 4735658.22 frames. ], batch size: 137, lr: 7.73e-03, grad_scale: 32.0 2026-09-24 01:27:20,251 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.589e+02 3.155e+02 3.540e+02 3.989e+02 5.498e+02, threshold=7.080e+02, percent-clipped=0.0 2026-09-24 01:27:23,083 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=85760.0, ans=0.0 2026-09-24 01:27:24,111 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=85760.0, ans=0.125 2026-09-24 01:27:35,542 INFO [train.py:1192] (0/2) Epoch 27, batch 850, loss[loss=0.2954, simple_loss=0.4119, pruned_loss=0.08941, over 24609.00 frames. ], tot_loss[loss=0.292, simple_loss=0.3982, pruned_loss=0.09287, over 4758979.92 frames. ], batch size: 198, lr: 7.72e-03, grad_scale: 32.0 2026-09-24 01:27:38,678 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=85860.0, ans=0.125 2026-09-24 01:27:44,012 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=85893.33333333333, ans=0.125 2026-09-24 01:27:53,125 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=85960.0, ans=0.0 2026-09-24 01:28:01,253 INFO [train.py:1192] (0/2) Epoch 27, batch 900, loss[loss=0.2387, simple_loss=0.3512, pruned_loss=0.06306, over 24568.00 frames. ], tot_loss[loss=0.2927, simple_loss=0.3986, pruned_loss=0.0934, over 4772519.12 frames. ], batch size: 137, lr: 7.72e-03, grad_scale: 64.0 2026-09-24 01:28:05,839 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.59 vs. limit=15.0 2026-09-24 01:28:10,003 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=86060.0, ans=0.025 2026-09-24 01:28:11,879 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.467e+02 3.251e+02 3.662e+02 4.258e+02 5.909e+02, threshold=7.324e+02, percent-clipped=0.0 2026-09-24 01:28:19,554 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=86126.66666666667, ans=0.125 2026-09-24 01:28:20,544 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=86160.0, ans=0.2 2026-09-24 01:28:25,958 INFO [train.py:1192] (0/2) Epoch 27, batch 950, loss[loss=0.4018, simple_loss=0.4503, pruned_loss=0.1767, over 12059.00 frames. ], tot_loss[loss=0.2935, simple_loss=0.3978, pruned_loss=0.09462, over 4709904.70 frames. ], batch size: 333, lr: 7.71e-03, grad_scale: 32.0 2026-09-24 01:28:30,619 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-27.pt 2026-09-24 01:28:37,269 INFO [train.py:1192] (0/2) Epoch 28, batch 0, loss[loss=0.2324, simple_loss=0.3488, pruned_loss=0.05806, over 24556.00 frames. ], tot_loss[loss=0.2324, simple_loss=0.3488, pruned_loss=0.05806, over 24556.00 frames. ], batch size: 137, lr: 7.57e-03, grad_scale: 32.0 2026-09-24 01:28:37,270 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 01:28:49,020 INFO [train.py:1224] (0/2) Epoch 28, validation: loss=0.1841, simple_loss=0.3017, pruned_loss=0.03321, over 2564189.00 frames. 2026-09-24 01:28:49,020 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 01:28:51,689 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.74 vs. limit=12.0 2026-09-24 01:28:58,263 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=86253.33333333333, ans=0.2 2026-09-24 01:29:04,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=86320.0, ans=0.125 2026-09-24 01:29:14,149 INFO [train.py:1192] (0/2) Epoch 28, batch 50, loss[loss=0.2569, simple_loss=0.3517, pruned_loss=0.08102, over 24251.00 frames. ], tot_loss[loss=0.3046, simple_loss=0.4087, pruned_loss=0.1003, over 1076930.09 frames. ], batch size: 125, lr: 7.56e-03, grad_scale: 32.0 2026-09-24 01:29:21,927 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.593e+02 3.506e+02 4.112e+02 4.821e+02 1.243e+03, threshold=8.224e+02, percent-clipped=5.0 2026-09-24 01:29:39,781 INFO [train.py:1192] (0/2) Epoch 28, batch 100, loss[loss=0.2957, simple_loss=0.3958, pruned_loss=0.09782, over 24609.00 frames. ], tot_loss[loss=0.3062, simple_loss=0.4114, pruned_loss=0.1005, over 1906255.77 frames. ], batch size: 154, lr: 7.55e-03, grad_scale: 32.0 2026-09-24 01:29:46,969 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=86586.66666666667, ans=0.0 2026-09-24 01:30:00,154 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.17 vs. limit=10.0 2026-09-24 01:30:05,331 INFO [train.py:1192] (0/2) Epoch 28, batch 150, loss[loss=0.2579, simple_loss=0.3542, pruned_loss=0.08074, over 24303.00 frames. ], tot_loss[loss=0.2992, simple_loss=0.4052, pruned_loss=0.0966, over 2558930.08 frames. ], batch size: 125, lr: 7.55e-03, grad_scale: 32.0 2026-09-24 01:30:06,713 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=86720.0, ans=0.125 2026-09-24 01:30:06,744 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=86720.0, ans=0.0 2026-09-24 01:30:13,162 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.392e+02 3.154e+02 3.543e+02 4.109e+02 5.817e+02, threshold=7.086e+02, percent-clipped=0.0 2026-09-24 01:30:14,720 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=86753.33333333333, ans=0.125 2026-09-24 01:30:18,339 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=86786.66666666667, ans=0.125 2026-09-24 01:30:18,348 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=86786.66666666667, ans=0.125 2026-09-24 01:30:26,347 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:30:30,645 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=86853.33333333333, ans=0.125 2026-09-24 01:30:31,512 INFO [train.py:1192] (0/2) Epoch 28, batch 200, loss[loss=0.3526, simple_loss=0.4402, pruned_loss=0.1325, over 21175.00 frames. ], tot_loss[loss=0.2966, simple_loss=0.4029, pruned_loss=0.0952, over 3057094.71 frames. ], batch size: 333, lr: 7.54e-03, grad_scale: 32.0 2026-09-24 01:30:40,427 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=86920.0, ans=0.0 2026-09-24 01:30:49,389 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=86986.66666666667, ans=0.0 2026-09-24 01:30:57,021 INFO [train.py:1192] (0/2) Epoch 28, batch 250, loss[loss=0.3198, simple_loss=0.4282, pruned_loss=0.1057, over 24308.00 frames. ], tot_loss[loss=0.2962, simple_loss=0.4021, pruned_loss=0.09517, over 3447294.72 frames. ], batch size: 234, lr: 7.53e-03, grad_scale: 32.0 2026-09-24 01:30:58,074 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=87053.33333333333, ans=0.0 2026-09-24 01:31:00,709 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:31:04,851 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.536e+02 3.459e+02 4.032e+02 4.652e+02 9.855e+02, threshold=8.064e+02, percent-clipped=4.0 2026-09-24 01:31:06,472 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.96 vs. limit=15.0 2026-09-24 01:31:20,453 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=87186.66666666667, ans=0.0 2026-09-24 01:31:22,329 INFO [train.py:1192] (0/2) Epoch 28, batch 300, loss[loss=0.314, simple_loss=0.4234, pruned_loss=0.1023, over 24541.00 frames. ], tot_loss[loss=0.2955, simple_loss=0.401, pruned_loss=0.09499, over 3751569.42 frames. ], batch size: 204, lr: 7.53e-03, grad_scale: 32.0 2026-09-24 01:31:30,850 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.29 vs. limit=12.0 2026-09-24 01:31:35,353 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=10.73 vs. limit=15.0 2026-09-24 01:31:43,124 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.26 vs. limit=12.0 2026-09-24 01:31:45,340 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=87353.33333333333, ans=0.025 2026-09-24 01:31:48,339 INFO [train.py:1192] (0/2) Epoch 28, batch 350, loss[loss=0.2682, simple_loss=0.3681, pruned_loss=0.08414, over 24559.00 frames. ], tot_loss[loss=0.2963, simple_loss=0.402, pruned_loss=0.09534, over 3990778.35 frames. ], batch size: 137, lr: 7.52e-03, grad_scale: 32.0 2026-09-24 01:31:56,461 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.584e+02 3.212e+02 3.605e+02 4.264e+02 7.770e+02, threshold=7.210e+02, percent-clipped=0.0 2026-09-24 01:32:03,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=87453.33333333333, ans=0.025 2026-09-24 01:32:04,779 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=18.37 vs. limit=22.5 2026-09-24 01:32:12,673 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.49 vs. limit=22.5 2026-09-24 01:32:14,292 INFO [train.py:1192] (0/2) Epoch 28, batch 400, loss[loss=0.308, simple_loss=0.4063, pruned_loss=0.1049, over 24564.00 frames. ], tot_loss[loss=0.2951, simple_loss=0.4008, pruned_loss=0.09468, over 4177555.95 frames. ], batch size: 170, lr: 7.51e-03, grad_scale: 32.0 2026-09-24 01:32:14,836 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=87553.33333333333, ans=0.125 2026-09-24 01:32:31,430 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=87653.33333333333, ans=0.2 2026-09-24 01:32:39,727 INFO [train.py:1192] (0/2) Epoch 28, batch 450, loss[loss=0.3169, simple_loss=0.4255, pruned_loss=0.1041, over 24618.00 frames. ], tot_loss[loss=0.2965, simple_loss=0.402, pruned_loss=0.09545, over 4311112.41 frames. ], batch size: 175, lr: 7.51e-03, grad_scale: 32.0 2026-09-24 01:32:47,431 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.492e+02 3.188e+02 3.687e+02 4.228e+02 7.096e+02, threshold=7.375e+02, percent-clipped=0.0 2026-09-24 01:32:56,933 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=87820.0, ans=0.2 2026-09-24 01:33:04,835 INFO [train.py:1192] (0/2) Epoch 28, batch 500, loss[loss=0.3374, simple_loss=0.4414, pruned_loss=0.1167, over 24518.00 frames. ], tot_loss[loss=0.2935, simple_loss=0.3994, pruned_loss=0.09383, over 4428225.88 frames. ], batch size: 218, lr: 7.50e-03, grad_scale: 32.0 2026-09-24 01:33:08,636 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.57 vs. limit=6.0 2026-09-24 01:33:18,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=87953.33333333333, ans=0.125 2026-09-24 01:33:20,857 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.25 vs. limit=10.0 2026-09-24 01:33:27,031 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=88020.0, ans=0.125 2026-09-24 01:33:27,564 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=88020.0, ans=0.125 2026-09-24 01:33:28,985 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.55 vs. limit=12.0 2026-09-24 01:33:30,701 INFO [train.py:1192] (0/2) Epoch 28, batch 550, loss[loss=0.3201, simple_loss=0.4353, pruned_loss=0.1025, over 24284.00 frames. ], tot_loss[loss=0.2936, simple_loss=0.3996, pruned_loss=0.09379, over 4517598.85 frames. ], batch size: 257, lr: 7.50e-03, grad_scale: 32.0 2026-09-24 01:33:35,747 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=88086.66666666667, ans=0.125 2026-09-24 01:33:38,380 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.413e+02 3.323e+02 3.632e+02 4.082e+02 5.460e+02, threshold=7.264e+02, percent-clipped=0.0 2026-09-24 01:33:45,089 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=88120.0, ans=0.125 2026-09-24 01:33:49,730 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=88153.33333333333, ans=0.0 2026-09-24 01:33:56,029 INFO [train.py:1192] (0/2) Epoch 28, batch 600, loss[loss=0.3299, simple_loss=0.4478, pruned_loss=0.106, over 24416.00 frames. ], tot_loss[loss=0.2938, simple_loss=0.4001, pruned_loss=0.09372, over 4585091.71 frames. ], batch size: 235, lr: 7.49e-03, grad_scale: 16.0 2026-09-24 01:33:56,586 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=88220.0, ans=0.0 2026-09-24 01:34:04,852 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=88253.33333333333, ans=0.2 2026-09-24 01:34:05,366 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=88253.33333333333, ans=0.0 2026-09-24 01:34:09,273 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=88286.66666666667, ans=0.125 2026-09-24 01:34:16,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=88353.33333333333, ans=0.1 2026-09-24 01:34:21,078 INFO [train.py:1192] (0/2) Epoch 28, batch 650, loss[loss=0.3112, simple_loss=0.4127, pruned_loss=0.1049, over 24557.00 frames. ], tot_loss[loss=0.2925, simple_loss=0.3992, pruned_loss=0.09292, over 4650364.03 frames. ], batch size: 162, lr: 7.48e-03, grad_scale: 16.0 2026-09-24 01:34:21,164 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=88386.66666666667, ans=0.1 2026-09-24 01:34:24,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=88386.66666666667, ans=0.1 2026-09-24 01:34:27,891 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=88420.0, ans=0.0 2026-09-24 01:34:29,319 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=88420.0, ans=0.0 2026-09-24 01:34:29,709 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.498e+02 3.177e+02 3.682e+02 4.351e+02 7.098e+02, threshold=7.363e+02, percent-clipped=0.0 2026-09-24 01:34:35,350 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=88453.33333333333, ans=0.125 2026-09-24 01:34:41,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=88520.0, ans=0.0 2026-09-24 01:34:41,830 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=88520.0, ans=0.02 2026-09-24 01:34:47,653 INFO [train.py:1192] (0/2) Epoch 28, batch 700, loss[loss=0.2986, simple_loss=0.3961, pruned_loss=0.1006, over 24561.00 frames. ], tot_loss[loss=0.2938, simple_loss=0.4002, pruned_loss=0.09374, over 4683451.37 frames. ], batch size: 154, lr: 7.48e-03, grad_scale: 16.0 2026-09-24 01:34:59,404 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=88620.0, ans=0.125 2026-09-24 01:34:59,496 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.10 vs. limit=15.0 2026-09-24 01:35:10,293 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.38 vs. limit=15.0 2026-09-24 01:35:13,519 INFO [train.py:1192] (0/2) Epoch 28, batch 750, loss[loss=0.303, simple_loss=0.4089, pruned_loss=0.09854, over 24556.00 frames. ], tot_loss[loss=0.2929, simple_loss=0.399, pruned_loss=0.09341, over 4710945.33 frames. ], batch size: 170, lr: 7.47e-03, grad_scale: 16.0 2026-09-24 01:35:14,057 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=88720.0, ans=0.0 2026-09-24 01:35:14,060 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=88720.0, ans=0.0 2026-09-24 01:35:20,878 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=88753.33333333333, ans=0.125 2026-09-24 01:35:21,670 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.551e+02 3.294e+02 3.656e+02 4.145e+02 6.425e+02, threshold=7.312e+02, percent-clipped=0.0 2026-09-24 01:35:34,366 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=7.22 vs. limit=12.0 2026-09-24 01:35:36,628 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=88853.33333333333, ans=0.125 2026-09-24 01:35:37,180 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.91 vs. limit=15.0 2026-09-24 01:35:38,608 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.65 vs. limit=15.0 2026-09-24 01:35:39,244 INFO [train.py:1192] (0/2) Epoch 28, batch 800, loss[loss=0.2393, simple_loss=0.3479, pruned_loss=0.06534, over 24534.00 frames. ], tot_loss[loss=0.2923, simple_loss=0.3986, pruned_loss=0.09298, over 4739536.45 frames. ], batch size: 137, lr: 7.46e-03, grad_scale: 32.0 2026-09-24 01:35:43,462 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=88886.66666666667, ans=0.025 2026-09-24 01:35:45,632 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.27 vs. limit=15.0 2026-09-24 01:36:04,633 INFO [train.py:1192] (0/2) Epoch 28, batch 850, loss[loss=0.3092, simple_loss=0.4206, pruned_loss=0.09893, over 24533.00 frames. ], tot_loss[loss=0.2916, simple_loss=0.3981, pruned_loss=0.0925, over 4760901.74 frames. ], batch size: 204, lr: 7.46e-03, grad_scale: 32.0 2026-09-24 01:36:07,480 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=89053.33333333333, ans=0.2 2026-09-24 01:36:11,932 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.83 vs. limit=15.0 2026-09-24 01:36:12,820 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.573e+02 3.192e+02 3.623e+02 4.155e+02 5.810e+02, threshold=7.245e+02, percent-clipped=0.0 2026-09-24 01:36:21,711 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=89153.33333333333, ans=10.0 2026-09-24 01:36:23,383 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.39 vs. limit=22.5 2026-09-24 01:36:30,326 INFO [train.py:1192] (0/2) Epoch 28, batch 900, loss[loss=0.2493, simple_loss=0.3591, pruned_loss=0.06968, over 24549.00 frames. ], tot_loss[loss=0.2914, simple_loss=0.3981, pruned_loss=0.09232, over 4774153.92 frames. ], batch size: 137, lr: 7.45e-03, grad_scale: 32.0 2026-09-24 01:36:31,868 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=89220.0, ans=0.125 2026-09-24 01:36:45,141 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.22 vs. limit=12.0 2026-09-24 01:36:54,547 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=89386.66666666667, ans=10.0 2026-09-24 01:36:54,899 INFO [train.py:1192] (0/2) Epoch 28, batch 950, loss[loss=0.4428, simple_loss=0.4803, pruned_loss=0.2027, over 11856.00 frames. ], tot_loss[loss=0.2927, simple_loss=0.3974, pruned_loss=0.09394, over 4709848.02 frames. ], batch size: 333, lr: 7.44e-03, grad_scale: 32.0 2026-09-24 01:36:59,368 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-28.pt 2026-09-24 01:37:06,575 INFO [train.py:1192] (0/2) Epoch 29, batch 0, loss[loss=0.237, simple_loss=0.3517, pruned_loss=0.06116, over 24571.00 frames. ], tot_loss[loss=0.237, simple_loss=0.3517, pruned_loss=0.06116, over 24571.00 frames. ], batch size: 137, lr: 7.31e-03, grad_scale: 32.0 2026-09-24 01:37:06,576 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 01:37:18,074 INFO [train.py:1224] (0/2) Epoch 29, validation: loss=0.1827, simple_loss=0.3014, pruned_loss=0.03193, over 2564189.00 frames. 2026-09-24 01:37:18,074 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 01:37:22,093 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.384e+02 3.217e+02 3.666e+02 4.169e+02 7.339e+02, threshold=7.333e+02, percent-clipped=0.0 2026-09-24 01:37:24,647 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=89446.66666666667, ans=0.125 2026-09-24 01:37:39,943 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.20 vs. limit=15.0 2026-09-24 01:37:43,814 INFO [train.py:1192] (0/2) Epoch 29, batch 50, loss[loss=0.2482, simple_loss=0.3433, pruned_loss=0.07658, over 24256.00 frames. ], tot_loss[loss=0.3035, simple_loss=0.4075, pruned_loss=0.09973, over 1076460.44 frames. ], batch size: 125, lr: 7.30e-03, grad_scale: 32.0 2026-09-24 01:37:48,325 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=89613.33333333333, ans=0.0 2026-09-24 01:37:49,294 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=89613.33333333333, ans=0.125 2026-09-24 01:37:58,161 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=89646.66666666667, ans=0.1 2026-09-24 01:38:00,221 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=89680.0, ans=0.125 2026-09-24 01:38:02,014 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=89680.0, ans=0.0 2026-09-24 01:38:06,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=89713.33333333333, ans=0.125 2026-09-24 01:38:06,374 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=89713.33333333333, ans=0.125 2026-09-24 01:38:09,296 INFO [train.py:1192] (0/2) Epoch 29, batch 100, loss[loss=0.3044, simple_loss=0.3996, pruned_loss=0.1046, over 24593.00 frames. ], tot_loss[loss=0.3038, simple_loss=0.4093, pruned_loss=0.09911, over 1905277.89 frames. ], batch size: 154, lr: 7.30e-03, grad_scale: 32.0 2026-09-24 01:38:13,479 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.584e+02 3.381e+02 3.673e+02 4.125e+02 5.808e+02, threshold=7.347e+02, percent-clipped=1.0 2026-09-24 01:38:17,922 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=89780.0, ans=0.0 2026-09-24 01:38:18,424 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=89780.0, ans=0.125 2026-09-24 01:38:19,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=89813.33333333333, ans=0.125 2026-09-24 01:38:21,608 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=89813.33333333333, ans=0.1 2026-09-24 01:38:34,729 INFO [train.py:1192] (0/2) Epoch 29, batch 150, loss[loss=0.2455, simple_loss=0.3491, pruned_loss=0.07096, over 24290.00 frames. ], tot_loss[loss=0.2981, simple_loss=0.4042, pruned_loss=0.09598, over 2558747.30 frames. ], batch size: 125, lr: 7.29e-03, grad_scale: 32.0 2026-09-24 01:38:55,978 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=90046.66666666667, ans=0.125 2026-09-24 01:38:59,673 INFO [train.py:1192] (0/2) Epoch 29, batch 200, loss[loss=0.3491, simple_loss=0.439, pruned_loss=0.1296, over 21166.00 frames. ], tot_loss[loss=0.2942, simple_loss=0.4007, pruned_loss=0.09381, over 3055898.84 frames. ], batch size: 333, lr: 7.28e-03, grad_scale: 32.0 2026-09-24 01:39:03,728 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.419e+02 3.215e+02 3.663e+02 4.520e+02 7.059e+02, threshold=7.326e+02, percent-clipped=0.0 2026-09-24 01:39:14,201 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=90146.66666666667, ans=0.0 2026-09-24 01:39:17,331 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.37 vs. limit=12.0 2026-09-24 01:39:25,693 INFO [train.py:1192] (0/2) Epoch 29, batch 250, loss[loss=0.3559, simple_loss=0.4536, pruned_loss=0.1291, over 24315.00 frames. ], tot_loss[loss=0.2948, simple_loss=0.4007, pruned_loss=0.09445, over 3444045.84 frames. ], batch size: 234, lr: 7.28e-03, grad_scale: 32.0 2026-09-24 01:39:28,761 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.96 vs. limit=15.0 2026-09-24 01:39:31,969 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=90280.0, ans=0.0 2026-09-24 01:39:50,963 INFO [train.py:1192] (0/2) Epoch 29, batch 300, loss[loss=0.3046, simple_loss=0.4252, pruned_loss=0.092, over 24555.00 frames. ], tot_loss[loss=0.2929, simple_loss=0.3989, pruned_loss=0.09342, over 3748838.12 frames. ], batch size: 204, lr: 7.27e-03, grad_scale: 32.0 2026-09-24 01:39:53,467 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=90413.33333333333, ans=0.2 2026-09-24 01:39:55,378 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.798e+02 3.448e+02 3.952e+02 4.554e+02 9.567e+02, threshold=7.905e+02, percent-clipped=3.0 2026-09-24 01:39:57,394 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.83 vs. limit=12.0 2026-09-24 01:39:57,905 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.36 vs. limit=22.5 2026-09-24 01:39:59,019 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=14.48 vs. limit=22.5 2026-09-24 01:40:00,575 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=90446.66666666667, ans=0.125 2026-09-24 01:40:00,971 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=90480.0, ans=0.2 2026-09-24 01:40:04,024 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.20 vs. limit=15.0 2026-09-24 01:40:04,431 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=90480.0, ans=0.0 2026-09-24 01:40:05,790 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=90480.0, ans=0.95 2026-09-24 01:40:08,444 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:40:10,325 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=90513.33333333333, ans=0.125 2026-09-24 01:40:10,337 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=90513.33333333333, ans=0.125 2026-09-24 01:40:11,984 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=90546.66666666667, ans=0.1 2026-09-24 01:40:13,816 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=16.13 vs. limit=22.5 2026-09-24 01:40:16,977 INFO [train.py:1192] (0/2) Epoch 29, batch 350, loss[loss=0.2685, simple_loss=0.3644, pruned_loss=0.08633, over 24573.00 frames. ], tot_loss[loss=0.2938, simple_loss=0.4003, pruned_loss=0.09372, over 3992870.86 frames. ], batch size: 137, lr: 7.27e-03, grad_scale: 32.0 2026-09-24 01:40:24,413 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=90613.33333333333, ans=0.025 2026-09-24 01:40:25,510 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=90613.33333333333, ans=0.0 2026-09-24 01:40:27,468 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.19 vs. limit=12.0 2026-09-24 01:40:33,432 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=90680.0, ans=6.0 2026-09-24 01:40:42,511 INFO [train.py:1192] (0/2) Epoch 29, batch 400, loss[loss=0.2977, simple_loss=0.4, pruned_loss=0.09769, over 24553.00 frames. ], tot_loss[loss=0.2923, simple_loss=0.399, pruned_loss=0.09276, over 4177792.14 frames. ], batch size: 170, lr: 7.26e-03, grad_scale: 32.0 2026-09-24 01:40:43,522 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=90746.66666666667, ans=0.125 2026-09-24 01:40:44,054 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=15.0 2026-09-24 01:40:46,578 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.611e+02 3.076e+02 3.524e+02 4.096e+02 7.653e+02, threshold=7.049e+02, percent-clipped=0.0 2026-09-24 01:41:07,796 INFO [train.py:1192] (0/2) Epoch 29, batch 450, loss[loss=0.3102, simple_loss=0.4164, pruned_loss=0.102, over 24624.00 frames. ], tot_loss[loss=0.2931, simple_loss=0.3997, pruned_loss=0.09325, over 4307916.33 frames. ], batch size: 175, lr: 7.25e-03, grad_scale: 32.0 2026-09-24 01:41:15,190 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=90946.66666666667, ans=0.125 2026-09-24 01:41:16,680 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=90946.66666666667, ans=0.025 2026-09-24 01:41:19,639 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=90980.0, ans=0.125 2026-09-24 01:41:20,895 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.47 vs. limit=10.0 2026-09-24 01:41:26,217 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=91013.33333333333, ans=0.1 2026-09-24 01:41:30,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=91046.66666666667, ans=0.0 2026-09-24 01:41:33,392 INFO [train.py:1192] (0/2) Epoch 29, batch 500, loss[loss=0.2787, simple_loss=0.4048, pruned_loss=0.07633, over 24511.00 frames. ], tot_loss[loss=0.2923, simple_loss=0.3984, pruned_loss=0.09308, over 4425735.66 frames. ], batch size: 218, lr: 7.25e-03, grad_scale: 32.0 2026-09-24 01:41:37,532 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.610e+02 3.257e+02 3.691e+02 4.343e+02 9.379e+02, threshold=7.382e+02, percent-clipped=2.0 2026-09-24 01:41:37,975 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=91080.0, ans=0.125 2026-09-24 01:41:38,374 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=91113.33333333333, ans=0.05 2026-09-24 01:41:43,329 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=91146.66666666667, ans=0.125 2026-09-24 01:41:59,581 INFO [train.py:1192] (0/2) Epoch 29, batch 550, loss[loss=0.3041, simple_loss=0.4202, pruned_loss=0.094, over 24270.00 frames. ], tot_loss[loss=0.292, simple_loss=0.3983, pruned_loss=0.09289, over 4515876.88 frames. ], batch size: 257, lr: 7.24e-03, grad_scale: 32.0 2026-09-24 01:42:07,478 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=91280.0, ans=0.1 2026-09-24 01:42:07,971 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=91280.0, ans=0.125 2026-09-24 01:42:13,266 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=91313.33333333333, ans=0.125 2026-09-24 01:42:20,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=91380.0, ans=0.1 2026-09-24 01:42:25,277 INFO [train.py:1192] (0/2) Epoch 29, batch 600, loss[loss=0.3438, simple_loss=0.4503, pruned_loss=0.1186, over 24319.00 frames. ], tot_loss[loss=0.2925, simple_loss=0.3989, pruned_loss=0.09305, over 4581778.07 frames. ], batch size: 234, lr: 7.24e-03, grad_scale: 32.0 2026-09-24 01:42:27,764 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=91413.33333333333, ans=0.125 2026-09-24 01:42:29,066 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.662e+02 3.191e+02 3.540e+02 4.132e+02 5.235e+02, threshold=7.080e+02, percent-clipped=0.0 2026-09-24 01:42:38,002 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.76 vs. limit=22.5 2026-09-24 01:42:50,896 INFO [train.py:1192] (0/2) Epoch 29, batch 650, loss[loss=0.3248, simple_loss=0.4213, pruned_loss=0.1142, over 24555.00 frames. ], tot_loss[loss=0.2915, simple_loss=0.3982, pruned_loss=0.09236, over 4648150.89 frames. ], batch size: 162, lr: 7.23e-03, grad_scale: 32.0 2026-09-24 01:43:11,765 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=91713.33333333333, ans=0.0 2026-09-24 01:43:12,255 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=91713.33333333333, ans=0.2 2026-09-24 01:43:15,600 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=91713.33333333333, ans=0.2 2026-09-24 01:43:16,998 INFO [train.py:1192] (0/2) Epoch 29, batch 700, loss[loss=0.2538, simple_loss=0.3652, pruned_loss=0.0712, over 24569.00 frames. ], tot_loss[loss=0.2919, simple_loss=0.3989, pruned_loss=0.09244, over 4681106.88 frames. ], batch size: 154, lr: 7.22e-03, grad_scale: 32.0 2026-09-24 01:43:20,796 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.643e+02 3.317e+02 3.742e+02 4.248e+02 7.075e+02, threshold=7.485e+02, percent-clipped=0.0 2026-09-24 01:43:42,384 INFO [train.py:1192] (0/2) Epoch 29, batch 750, loss[loss=0.3003, simple_loss=0.4098, pruned_loss=0.09535, over 24570.00 frames. ], tot_loss[loss=0.2903, simple_loss=0.3971, pruned_loss=0.09169, over 4712135.01 frames. ], batch size: 170, lr: 7.22e-03, grad_scale: 32.0 2026-09-24 01:43:53,738 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=91980.0, ans=0.125 2026-09-24 01:43:58,640 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=92013.33333333333, ans=0.125 2026-09-24 01:44:00,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=92013.33333333333, ans=0.0 2026-09-24 01:44:04,070 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=92046.66666666667, ans=0.125 2026-09-24 01:44:04,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=92046.66666666667, ans=10.0 2026-09-24 01:44:07,244 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:44:07,589 INFO [train.py:1192] (0/2) Epoch 29, batch 800, loss[loss=0.255, simple_loss=0.3592, pruned_loss=0.07541, over 24511.00 frames. ], tot_loss[loss=0.2897, simple_loss=0.3966, pruned_loss=0.09137, over 4736656.34 frames. ], batch size: 137, lr: 7.21e-03, grad_scale: 32.0 2026-09-24 01:44:07,990 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=3.86 vs. limit=12.0 2026-09-24 01:44:11,512 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.222e+02 3.268e+02 3.638e+02 4.164e+02 6.382e+02, threshold=7.275e+02, percent-clipped=0.0 2026-09-24 01:44:12,931 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=92113.33333333333, ans=0.0 2026-09-24 01:44:26,043 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=92180.0, ans=0.1 2026-09-24 01:44:26,049 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=92180.0, ans=0.0 2026-09-24 01:44:29,220 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=92213.33333333333, ans=0.2 2026-09-24 01:44:29,357 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.55 vs. limit=6.0 2026-09-24 01:44:32,731 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.85 vs. limit=15.0 2026-09-24 01:44:33,418 INFO [train.py:1192] (0/2) Epoch 29, batch 850, loss[loss=0.3148, simple_loss=0.426, pruned_loss=0.1018, over 24541.00 frames. ], tot_loss[loss=0.2896, simple_loss=0.3966, pruned_loss=0.09126, over 4758901.47 frames. ], batch size: 204, lr: 7.20e-03, grad_scale: 32.0 2026-09-24 01:44:36,670 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=92246.66666666667, ans=0.1 2026-09-24 01:44:39,603 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=92280.0, ans=0.125 2026-09-24 01:44:42,898 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=92280.0, ans=10.0 2026-09-24 01:44:52,809 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=92346.66666666667, ans=0.125 2026-09-24 01:44:53,234 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=92380.0, ans=0.2 2026-09-24 01:44:55,078 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=92380.0, ans=0.1 2026-09-24 01:44:58,929 INFO [train.py:1192] (0/2) Epoch 29, batch 900, loss[loss=0.2552, simple_loss=0.3605, pruned_loss=0.07496, over 24574.00 frames. ], tot_loss[loss=0.29, simple_loss=0.397, pruned_loss=0.09152, over 4772774.40 frames. ], batch size: 137, lr: 7.20e-03, grad_scale: 32.0 2026-09-24 01:45:02,994 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.552e+02 3.260e+02 3.615e+02 4.126e+02 6.303e+02, threshold=7.230e+02, percent-clipped=0.0 2026-09-24 01:45:05,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=92446.66666666667, ans=0.1 2026-09-24 01:45:23,359 INFO [train.py:1192] (0/2) Epoch 29, batch 950, loss[loss=0.4325, simple_loss=0.4669, pruned_loss=0.1991, over 11480.00 frames. ], tot_loss[loss=0.2911, simple_loss=0.3962, pruned_loss=0.093, over 4717852.32 frames. ], batch size: 333, lr: 7.19e-03, grad_scale: 32.0 2026-09-24 01:45:23,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=92580.0, ans=0.125 2026-09-24 01:45:27,920 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-29.pt 2026-09-24 01:45:34,453 INFO [train.py:1192] (0/2) Epoch 30, batch 0, loss[loss=0.2331, simple_loss=0.3516, pruned_loss=0.05729, over 24589.00 frames. ], tot_loss[loss=0.2331, simple_loss=0.3516, pruned_loss=0.05729, over 24589.00 frames. ], batch size: 137, lr: 7.07e-03, grad_scale: 32.0 2026-09-24 01:45:34,453 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 01:45:45,791 INFO [train.py:1224] (0/2) Epoch 30, validation: loss=0.1842, simple_loss=0.3023, pruned_loss=0.03305, over 2564189.00 frames. 2026-09-24 01:45:45,791 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 01:45:54,233 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=92640.0, ans=0.0 2026-09-24 01:46:11,798 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.221e+02 3.410e+02 3.907e+02 4.808e+02 7.268e+02, threshold=7.815e+02, percent-clipped=1.0 2026-09-24 01:46:11,804 INFO [train.py:1192] (0/2) Epoch 30, batch 50, loss[loss=0.2408, simple_loss=0.3418, pruned_loss=0.06986, over 24276.00 frames. ], tot_loss[loss=0.2947, simple_loss=0.4019, pruned_loss=0.09377, over 1075441.78 frames. ], batch size: 125, lr: 7.06e-03, grad_scale: 32.0 2026-09-24 01:46:14,329 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.82 vs. limit=15.0 2026-09-24 01:46:19,638 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=92806.66666666667, ans=0.0 2026-09-24 01:46:26,806 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=92840.0, ans=0.1 2026-09-24 01:46:33,591 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=92906.66666666667, ans=0.0 2026-09-24 01:46:37,597 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=92940.0, ans=0.1 2026-09-24 01:46:37,970 INFO [train.py:1192] (0/2) Epoch 30, batch 100, loss[loss=0.3087, simple_loss=0.4023, pruned_loss=0.1075, over 24597.00 frames. ], tot_loss[loss=0.3007, simple_loss=0.4072, pruned_loss=0.09708, over 1902919.00 frames. ], batch size: 154, lr: 7.06e-03, grad_scale: 32.0 2026-09-24 01:46:44,462 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=92973.33333333333, ans=0.1 2026-09-24 01:46:56,160 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:46:56,182 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:46:56,194 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=93040.0, ans=0.95 2026-09-24 01:46:59,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=93073.33333333333, ans=0.2 2026-09-24 01:47:02,897 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.616e+02 3.261e+02 3.575e+02 4.013e+02 5.707e+02, threshold=7.151e+02, percent-clipped=0.0 2026-09-24 01:47:02,903 INFO [train.py:1192] (0/2) Epoch 30, batch 150, loss[loss=0.2256, simple_loss=0.3316, pruned_loss=0.05984, over 24266.00 frames. ], tot_loss[loss=0.293, simple_loss=0.4005, pruned_loss=0.09278, over 2557029.91 frames. ], batch size: 125, lr: 7.05e-03, grad_scale: 32.0 2026-09-24 01:47:28,970 INFO [train.py:1192] (0/2) Epoch 30, batch 200, loss[loss=0.3564, simple_loss=0.4449, pruned_loss=0.134, over 21093.00 frames. ], tot_loss[loss=0.2907, simple_loss=0.3983, pruned_loss=0.0915, over 3053809.12 frames. ], batch size: 333, lr: 7.04e-03, grad_scale: 32.0 2026-09-24 01:47:30,086 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=93273.33333333333, ans=0.125 2026-09-24 01:47:37,552 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-28000.pt 2026-09-24 01:47:39,256 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=93340.0, ans=0.1 2026-09-24 01:47:39,808 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=93340.0, ans=0.07 2026-09-24 01:47:41,704 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=93340.0, ans=0.125 2026-09-24 01:47:48,748 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=93373.33333333333, ans=0.1 2026-09-24 01:47:50,616 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=93406.66666666667, ans=0.125 2026-09-24 01:47:51,112 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.21 vs. limit=15.0 2026-09-24 01:47:51,490 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=93406.66666666667, ans=0.0 2026-09-24 01:47:54,224 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.683e+02 3.373e+02 3.806e+02 4.796e+02 1.018e+03, threshold=7.612e+02, percent-clipped=3.0 2026-09-24 01:47:54,230 INFO [train.py:1192] (0/2) Epoch 30, batch 250, loss[loss=0.3056, simple_loss=0.4284, pruned_loss=0.09144, over 24290.00 frames. ], tot_loss[loss=0.2912, simple_loss=0.3988, pruned_loss=0.09182, over 3442725.83 frames. ], batch size: 234, lr: 7.04e-03, grad_scale: 32.0 2026-09-24 01:47:54,700 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=93440.0, ans=0.125 2026-09-24 01:47:56,178 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=93440.0, ans=0.0 2026-09-24 01:47:56,201 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=93440.0, ans=0.0 2026-09-24 01:48:06,010 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=93506.66666666667, ans=0.125 2026-09-24 01:48:13,125 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=93540.0, ans=0.125 2026-09-24 01:48:15,041 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.32 vs. limit=15.0 2026-09-24 01:48:15,427 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=93573.33333333333, ans=0.0 2026-09-24 01:48:19,621 INFO [train.py:1192] (0/2) Epoch 30, batch 300, loss[loss=0.2925, simple_loss=0.4095, pruned_loss=0.0877, over 24545.00 frames. ], tot_loss[loss=0.2901, simple_loss=0.3975, pruned_loss=0.09138, over 3748654.56 frames. ], batch size: 204, lr: 7.03e-03, grad_scale: 32.0 2026-09-24 01:48:24,744 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=93640.0, ans=0.1 2026-09-24 01:48:26,753 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=93640.0, ans=0.1 2026-09-24 01:48:27,663 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=93640.0, ans=0.0 2026-09-24 01:48:29,552 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=93673.33333333333, ans=0.125 2026-09-24 01:48:42,553 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=93740.0, ans=0.125 2026-09-24 01:48:45,146 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.431e+02 3.204e+02 3.555e+02 3.928e+02 5.395e+02, threshold=7.109e+02, percent-clipped=0.0 2026-09-24 01:48:45,153 INFO [train.py:1192] (0/2) Epoch 30, batch 350, loss[loss=0.2386, simple_loss=0.3473, pruned_loss=0.06491, over 24565.00 frames. ], tot_loss[loss=0.2903, simple_loss=0.398, pruned_loss=0.09133, over 3991906.33 frames. ], batch size: 137, lr: 7.03e-03, grad_scale: 32.0 2026-09-24 01:48:57,930 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.max_abs, batch_count=93840.0, ans=10.0 2026-09-24 01:49:05,309 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=93873.33333333333, ans=0.125 2026-09-24 01:49:08,808 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=93906.66666666667, ans=0.0 2026-09-24 01:49:08,843 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=93906.66666666667, ans=0.2 2026-09-24 01:49:11,432 INFO [train.py:1192] (0/2) Epoch 30, batch 400, loss[loss=0.3131, simple_loss=0.413, pruned_loss=0.1066, over 24565.00 frames. ], tot_loss[loss=0.2905, simple_loss=0.3979, pruned_loss=0.09158, over 4174226.81 frames. ], batch size: 170, lr: 7.02e-03, grad_scale: 32.0 2026-09-24 01:49:24,943 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=94006.66666666667, ans=0.0 2026-09-24 01:49:28,297 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.98 vs. limit=10.0 2026-09-24 01:49:37,087 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.395e+02 3.288e+02 3.765e+02 4.465e+02 8.248e+02, threshold=7.529e+02, percent-clipped=1.0 2026-09-24 01:49:37,093 INFO [train.py:1192] (0/2) Epoch 30, batch 450, loss[loss=0.2938, simple_loss=0.4048, pruned_loss=0.09135, over 24616.00 frames. ], tot_loss[loss=0.291, simple_loss=0.3983, pruned_loss=0.0919, over 4308789.72 frames. ], batch size: 175, lr: 7.02e-03, grad_scale: 32.0 2026-09-24 01:49:45,296 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=94140.0, ans=0.2 2026-09-24 01:49:47,261 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:49:48,753 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=94173.33333333333, ans=0.125 2026-09-24 01:49:49,365 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.80 vs. limit=15.0 2026-09-24 01:49:52,703 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=94206.66666666667, ans=0.0 2026-09-24 01:49:54,256 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.34 vs. limit=22.5 2026-09-24 01:50:02,834 INFO [train.py:1192] (0/2) Epoch 30, batch 500, loss[loss=0.2981, simple_loss=0.4169, pruned_loss=0.0896, over 24511.00 frames. ], tot_loss[loss=0.2901, simple_loss=0.397, pruned_loss=0.09163, over 4426902.06 frames. ], batch size: 218, lr: 7.01e-03, grad_scale: 32.0 2026-09-24 01:50:02,920 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=94273.33333333333, ans=0.125 2026-09-24 01:50:04,023 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.27 vs. limit=15.0 2026-09-24 01:50:10,066 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=94306.66666666667, ans=0.125 2026-09-24 01:50:11,445 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=94306.66666666667, ans=0.125 2026-09-24 01:50:27,286 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.484e+02 3.186e+02 3.398e+02 3.899e+02 5.797e+02, threshold=6.796e+02, percent-clipped=0.0 2026-09-24 01:50:27,292 INFO [train.py:1192] (0/2) Epoch 30, batch 550, loss[loss=0.3009, simple_loss=0.4184, pruned_loss=0.0917, over 24286.00 frames. ], tot_loss[loss=0.2892, simple_loss=0.3966, pruned_loss=0.09084, over 4516387.99 frames. ], batch size: 257, lr: 7.00e-03, grad_scale: 32.0 2026-09-24 01:50:48,043 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=94573.33333333333, ans=0.1 2026-09-24 01:50:48,183 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.84 vs. limit=22.5 2026-09-24 01:50:50,502 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=94573.33333333333, ans=0.0 2026-09-24 01:50:51,582 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.55 vs. limit=15.0 2026-09-24 01:50:53,389 INFO [train.py:1192] (0/2) Epoch 30, batch 600, loss[loss=0.3095, simple_loss=0.4308, pruned_loss=0.09409, over 24371.00 frames. ], tot_loss[loss=0.2896, simple_loss=0.3973, pruned_loss=0.09096, over 4583645.56 frames. ], batch size: 234, lr: 7.00e-03, grad_scale: 32.0 2026-09-24 01:51:05,853 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=94673.33333333333, ans=0.0 2026-09-24 01:51:06,283 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=94673.33333333333, ans=0.0 2026-09-24 01:51:10,487 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=94706.66666666667, ans=0.0 2026-09-24 01:51:10,926 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=94706.66666666667, ans=0.1 2026-09-24 01:51:15,741 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=94740.0, ans=0.1 2026-09-24 01:51:17,591 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=94740.0, ans=0.0 2026-09-24 01:51:18,950 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.444e+02 3.185e+02 3.671e+02 4.261e+02 6.024e+02, threshold=7.341e+02, percent-clipped=0.0 2026-09-24 01:51:18,956 INFO [train.py:1192] (0/2) Epoch 30, batch 650, loss[loss=0.2708, simple_loss=0.3844, pruned_loss=0.07862, over 24561.00 frames. ], tot_loss[loss=0.2884, simple_loss=0.3962, pruned_loss=0.09027, over 4649745.45 frames. ], batch size: 162, lr: 6.99e-03, grad_scale: 32.0 2026-09-24 01:51:23,157 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=94773.33333333333, ans=0.0 2026-09-24 01:51:25,664 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten.whitening_limit, batch_count=94806.66666666667, ans=15.0 2026-09-24 01:51:27,908 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=94806.66666666667, ans=0.0 2026-09-24 01:51:45,002 INFO [train.py:1192] (0/2) Epoch 30, batch 700, loss[loss=0.2722, simple_loss=0.38, pruned_loss=0.0822, over 24556.00 frames. ], tot_loss[loss=0.2893, simple_loss=0.3972, pruned_loss=0.0907, over 4682735.52 frames. ], batch size: 154, lr: 6.99e-03, grad_scale: 64.0 2026-09-24 01:51:46,920 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=94940.0, ans=0.5 2026-09-24 01:51:47,603 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.40 vs. limit=15.0 2026-09-24 01:51:48,382 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=94940.0, ans=0.125 2026-09-24 01:51:58,451 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:52:02,189 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=95040.0, ans=0.125 2026-09-24 01:52:11,072 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.585e+02 3.195e+02 3.649e+02 4.277e+02 7.083e+02, threshold=7.297e+02, percent-clipped=0.0 2026-09-24 01:52:11,079 INFO [train.py:1192] (0/2) Epoch 30, batch 750, loss[loss=0.2727, simple_loss=0.3862, pruned_loss=0.07964, over 24550.00 frames. ], tot_loss[loss=0.2892, simple_loss=0.3966, pruned_loss=0.09089, over 4709822.43 frames. ], batch size: 170, lr: 6.98e-03, grad_scale: 64.0 2026-09-24 01:52:17,566 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=95140.0, ans=0.0 2026-09-24 01:52:24,574 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=95173.33333333333, ans=0.0 2026-09-24 01:52:26,284 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=10.36 vs. limit=15.0 2026-09-24 01:52:36,686 INFO [train.py:1192] (0/2) Epoch 30, batch 800, loss[loss=0.2629, simple_loss=0.3613, pruned_loss=0.08224, over 24547.00 frames. ], tot_loss[loss=0.2884, simple_loss=0.3958, pruned_loss=0.0905, over 4734953.47 frames. ], batch size: 137, lr: 6.98e-03, grad_scale: 64.0 2026-09-24 01:52:41,291 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=95306.66666666667, ans=0.2 2026-09-24 01:52:50,147 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=95340.0, ans=0.07 2026-09-24 01:52:53,484 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=95373.33333333333, ans=0.125 2026-09-24 01:52:54,008 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=95373.33333333333, ans=0.125 2026-09-24 01:53:02,362 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.841e+02 3.325e+02 3.701e+02 4.282e+02 6.167e+02, threshold=7.403e+02, percent-clipped=0.0 2026-09-24 01:53:02,368 INFO [train.py:1192] (0/2) Epoch 30, batch 850, loss[loss=0.318, simple_loss=0.423, pruned_loss=0.1065, over 24589.00 frames. ], tot_loss[loss=0.2876, simple_loss=0.3952, pruned_loss=0.09, over 4758173.07 frames. ], batch size: 198, lr: 6.97e-03, grad_scale: 64.0 2026-09-24 01:53:03,049 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=95440.0, ans=0.0 2026-09-24 01:53:05,024 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.72 vs. limit=15.0 2026-09-24 01:53:12,500 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=14.82 vs. limit=22.5 2026-09-24 01:53:22,961 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=95573.33333333333, ans=0.125 2026-09-24 01:53:25,079 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=95573.33333333333, ans=0.025 2026-09-24 01:53:27,963 INFO [train.py:1192] (0/2) Epoch 30, batch 900, loss[loss=0.235, simple_loss=0.3505, pruned_loss=0.05972, over 24564.00 frames. ], tot_loss[loss=0.2882, simple_loss=0.3959, pruned_loss=0.09029, over 4773277.42 frames. ], batch size: 137, lr: 6.96e-03, grad_scale: 64.0 2026-09-24 01:53:40,333 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.03 vs. limit=10.0 2026-09-24 01:53:40,651 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:53:41,137 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=95673.33333333333, ans=0.0 2026-09-24 01:53:45,926 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.68 vs. limit=6.0 2026-09-24 01:53:47,661 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=95740.0, ans=0.2 2026-09-24 01:53:52,830 INFO [train.py:1192] (0/2) Epoch 30, batch 950, loss[loss=0.426, simple_loss=0.464, pruned_loss=0.194, over 11147.00 frames. ], tot_loss[loss=0.2887, simple_loss=0.3949, pruned_loss=0.09121, over 4713469.33 frames. ], batch size: 333, lr: 6.96e-03, grad_scale: 32.0 2026-09-24 01:53:53,059 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=26.81 vs. limit=22.5 2026-09-24 01:53:53,319 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.530e+02 3.392e+02 3.907e+02 4.492e+02 6.002e+02, threshold=7.815e+02, percent-clipped=0.0 2026-09-24 01:53:57,202 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-30.pt 2026-09-24 01:54:02,965 INFO [train.py:1192] (0/2) Epoch 31, batch 0, loss[loss=0.2501, simple_loss=0.3614, pruned_loss=0.06939, over 24576.00 frames. ], tot_loss[loss=0.2501, simple_loss=0.3614, pruned_loss=0.06939, over 24576.00 frames. ], batch size: 137, lr: 6.84e-03, grad_scale: 32.0 2026-09-24 01:54:02,970 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 01:54:11,595 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.7291, 3.1780, 3.0990, 1.7035], device='cuda:0') 2026-09-24 01:54:14,453 INFO [train.py:1224] (0/2) Epoch 31, validation: loss=0.1821, simple_loss=0.3003, pruned_loss=0.03191, over 2564189.00 frames. 2026-09-24 01:54:14,454 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 01:54:15,896 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=95800.0, ans=0.0 2026-09-24 01:54:18,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=95800.0, ans=0.1 2026-09-24 01:54:34,748 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=95933.33333333333, ans=0.0 2026-09-24 01:54:39,603 INFO [train.py:1192] (0/2) Epoch 31, batch 50, loss[loss=0.2574, simple_loss=0.3551, pruned_loss=0.07982, over 24333.00 frames. ], tot_loss[loss=0.2919, simple_loss=0.3989, pruned_loss=0.09246, over 1076524.43 frames. ], batch size: 125, lr: 6.84e-03, grad_scale: 32.0 2026-09-24 01:54:39,689 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=95966.66666666667, ans=0.2 2026-09-24 01:55:01,005 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=96100.0, ans=0.125 2026-09-24 01:55:01,165 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.81 vs. limit=15.0 2026-09-24 01:55:01,758 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.596e+02 3.558e+02 3.996e+02 4.712e+02 6.980e+02, threshold=7.991e+02, percent-clipped=0.0 2026-09-24 01:55:05,469 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=96133.33333333333, ans=0.0 2026-09-24 01:55:05,867 INFO [train.py:1192] (0/2) Epoch 31, batch 100, loss[loss=0.2871, simple_loss=0.3881, pruned_loss=0.093, over 24589.00 frames. ], tot_loss[loss=0.2979, simple_loss=0.4051, pruned_loss=0.09532, over 1904350.55 frames. ], batch size: 154, lr: 6.83e-03, grad_scale: 32.0 2026-09-24 01:55:08,349 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=96133.33333333333, ans=0.125 2026-09-24 01:55:09,245 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=96133.33333333333, ans=0.125 2026-09-24 01:55:11,852 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:55:15,343 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=5.13 vs. limit=15.0 2026-09-24 01:55:27,558 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.01 vs. limit=12.0 2026-09-24 01:55:27,993 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=96266.66666666667, ans=0.125 2026-09-24 01:55:29,459 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=96266.66666666667, ans=0.0 2026-09-24 01:55:30,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=96266.66666666667, ans=0.125 2026-09-24 01:55:31,291 INFO [train.py:1192] (0/2) Epoch 31, batch 150, loss[loss=0.2518, simple_loss=0.3541, pruned_loss=0.07476, over 24224.00 frames. ], tot_loss[loss=0.293, simple_loss=0.4004, pruned_loss=0.0928, over 2557931.67 frames. ], batch size: 125, lr: 6.83e-03, grad_scale: 32.0 2026-09-24 01:55:34,060 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=96300.0, ans=0.0 2026-09-24 01:55:39,068 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=96333.33333333333, ans=0.025 2026-09-24 01:55:40,602 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.58 vs. limit=10.0 2026-09-24 01:55:45,557 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=96366.66666666667, ans=0.035 2026-09-24 01:55:50,872 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=96400.0, ans=0.125 2026-09-24 01:55:52,457 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=96433.33333333333, ans=0.125 2026-09-24 01:55:53,304 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.532e+02 3.348e+02 3.902e+02 4.462e+02 6.331e+02, threshold=7.805e+02, percent-clipped=0.0 2026-09-24 01:55:56,660 INFO [train.py:1192] (0/2) Epoch 31, batch 200, loss[loss=0.3456, simple_loss=0.4355, pruned_loss=0.1279, over 21153.00 frames. ], tot_loss[loss=0.2914, simple_loss=0.3993, pruned_loss=0.09179, over 3054335.46 frames. ], batch size: 333, lr: 6.82e-03, grad_scale: 32.0 2026-09-24 01:56:07,959 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=96533.33333333333, ans=0.1 2026-09-24 01:56:22,026 INFO [train.py:1192] (0/2) Epoch 31, batch 250, loss[loss=0.3228, simple_loss=0.439, pruned_loss=0.1033, over 24304.00 frames. ], tot_loss[loss=0.2901, simple_loss=0.3981, pruned_loss=0.09102, over 3443232.45 frames. ], batch size: 234, lr: 6.81e-03, grad_scale: 32.0 2026-09-24 01:56:25,989 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=96633.33333333333, ans=0.1 2026-09-24 01:56:30,047 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=96666.66666666667, ans=0.125 2026-09-24 01:56:36,105 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=96700.0, ans=0.0 2026-09-24 01:56:40,144 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=96733.33333333333, ans=0.125 2026-09-24 01:56:40,655 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=96733.33333333333, ans=0.125 2026-09-24 01:56:42,196 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=96766.66666666667, ans=0.125 2026-09-24 01:56:42,734 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=96766.66666666667, ans=0.0 2026-09-24 01:56:44,173 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.501e+02 3.365e+02 3.961e+02 4.949e+02 8.830e+02, threshold=7.921e+02, percent-clipped=2.0 2026-09-24 01:56:47,466 INFO [train.py:1192] (0/2) Epoch 31, batch 300, loss[loss=0.2875, simple_loss=0.4013, pruned_loss=0.08683, over 24553.00 frames. ], tot_loss[loss=0.2896, simple_loss=0.3968, pruned_loss=0.09116, over 3749062.34 frames. ], batch size: 204, lr: 6.81e-03, grad_scale: 32.0 2026-09-24 01:56:54,478 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=96833.33333333333, ans=0.0 2026-09-24 01:56:56,338 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=96833.33333333333, ans=0.0 2026-09-24 01:56:57,493 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.89 vs. limit=6.0 2026-09-24 01:57:09,177 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=96933.33333333333, ans=0.125 2026-09-24 01:57:10,799 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.08 vs. limit=15.0 2026-09-24 01:57:13,217 INFO [train.py:1192] (0/2) Epoch 31, batch 350, loss[loss=0.2425, simple_loss=0.3458, pruned_loss=0.06958, over 24572.00 frames. ], tot_loss[loss=0.2901, simple_loss=0.3976, pruned_loss=0.09131, over 3992098.53 frames. ], batch size: 137, lr: 6.80e-03, grad_scale: 32.0 2026-09-24 01:57:22,045 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.57 vs. limit=15.0 2026-09-24 01:57:27,072 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=97033.33333333333, ans=0.025 2026-09-24 01:57:35,377 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.524e+02 3.164e+02 3.531e+02 3.999e+02 7.327e+02, threshold=7.063e+02, percent-clipped=0.0 2026-09-24 01:57:37,638 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=97100.0, ans=0.0 2026-09-24 01:57:38,933 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=97133.33333333333, ans=0.1 2026-09-24 01:57:39,409 INFO [train.py:1192] (0/2) Epoch 31, batch 400, loss[loss=0.2932, simple_loss=0.4031, pruned_loss=0.09159, over 24574.00 frames. ], tot_loss[loss=0.2884, simple_loss=0.3963, pruned_loss=0.09026, over 4178573.41 frames. ], batch size: 170, lr: 6.80e-03, grad_scale: 32.0 2026-09-24 01:57:46,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=97166.66666666667, ans=0.025 2026-09-24 01:57:47,244 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=97166.66666666667, ans=0.1 2026-09-24 01:57:56,607 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 01:58:05,082 INFO [train.py:1192] (0/2) Epoch 31, batch 450, loss[loss=0.2744, simple_loss=0.3943, pruned_loss=0.07724, over 24620.00 frames. ], tot_loss[loss=0.2891, simple_loss=0.3968, pruned_loss=0.09067, over 4310807.55 frames. ], batch size: 175, lr: 6.79e-03, grad_scale: 32.0 2026-09-24 01:58:05,198 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=97300.0, ans=0.2 2026-09-24 01:58:14,017 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=97333.33333333333, ans=0.0 2026-09-24 01:58:26,151 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=97433.33333333333, ans=0.1 2026-09-24 01:58:27,872 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.438e+02 3.243e+02 3.714e+02 4.161e+02 6.137e+02, threshold=7.427e+02, percent-clipped=0.0 2026-09-24 01:58:27,978 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=97433.33333333333, ans=0.125 2026-09-24 01:58:31,105 INFO [train.py:1192] (0/2) Epoch 31, batch 500, loss[loss=0.3011, simple_loss=0.4225, pruned_loss=0.08981, over 24504.00 frames. ], tot_loss[loss=0.2883, simple_loss=0.3956, pruned_loss=0.09052, over 4428758.99 frames. ], batch size: 218, lr: 6.79e-03, grad_scale: 32.0 2026-09-24 01:58:56,340 INFO [train.py:1192] (0/2) Epoch 31, batch 550, loss[loss=0.3105, simple_loss=0.4242, pruned_loss=0.09838, over 24276.00 frames. ], tot_loss[loss=0.2884, simple_loss=0.3958, pruned_loss=0.09047, over 4518385.04 frames. ], batch size: 257, lr: 6.78e-03, grad_scale: 32.0 2026-09-24 01:59:00,658 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=97633.33333333333, ans=0.125 2026-09-24 01:59:02,339 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=97666.66666666667, ans=0.025 2026-09-24 01:59:03,664 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=97666.66666666667, ans=0.125 2026-09-24 01:59:06,799 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.62 vs. limit=15.0 2026-09-24 01:59:18,285 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.697e+02 3.182e+02 3.595e+02 4.025e+02 7.869e+02, threshold=7.190e+02, percent-clipped=1.0 2026-09-24 01:59:19,455 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=97766.66666666667, ans=0.125 2026-09-24 01:59:20,901 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=97766.66666666667, ans=0.125 2026-09-24 01:59:21,848 INFO [train.py:1192] (0/2) Epoch 31, batch 600, loss[loss=0.3328, simple_loss=0.4416, pruned_loss=0.112, over 24333.00 frames. ], tot_loss[loss=0.2885, simple_loss=0.3963, pruned_loss=0.09038, over 4584848.96 frames. ], batch size: 234, lr: 6.78e-03, grad_scale: 32.0 2026-09-24 01:59:25,793 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.09 vs. limit=12.0 2026-09-24 01:59:26,701 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.87 vs. limit=15.0 2026-09-24 01:59:37,872 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=97900.0, ans=0.04949747468305833 2026-09-24 01:59:39,581 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=97900.0, ans=0.0 2026-09-24 01:59:45,512 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=97933.33333333333, ans=15.0 2026-09-24 01:59:45,958 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=5.134e-02 2026-09-24 01:59:47,755 INFO [train.py:1192] (0/2) Epoch 31, batch 650, loss[loss=0.2684, simple_loss=0.3795, pruned_loss=0.07868, over 24587.00 frames. ], tot_loss[loss=0.2876, simple_loss=0.3955, pruned_loss=0.08983, over 4650393.66 frames. ], batch size: 162, lr: 6.77e-03, grad_scale: 32.0 2026-09-24 01:59:54,228 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.26 vs. limit=15.0 2026-09-24 01:59:57,101 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=98000.0, ans=0.125 2026-09-24 02:00:01,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=98033.33333333333, ans=0.95 2026-09-24 02:00:03,775 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=98066.66666666667, ans=0.125 2026-09-24 02:00:09,695 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.509e+02 3.300e+02 3.827e+02 4.639e+02 6.903e+02, threshold=7.654e+02, percent-clipped=0.0 2026-09-24 02:00:12,837 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=98133.33333333333, ans=0.1 2026-09-24 02:00:13,295 INFO [train.py:1192] (0/2) Epoch 31, batch 700, loss[loss=0.2866, simple_loss=0.3906, pruned_loss=0.09135, over 24559.00 frames. ], tot_loss[loss=0.2879, simple_loss=0.3961, pruned_loss=0.08981, over 4682289.55 frames. ], batch size: 154, lr: 6.77e-03, grad_scale: 32.0 2026-09-24 02:00:39,513 INFO [train.py:1192] (0/2) Epoch 31, batch 750, loss[loss=0.3008, simple_loss=0.409, pruned_loss=0.09631, over 24571.00 frames. ], tot_loss[loss=0.2868, simple_loss=0.3948, pruned_loss=0.08937, over 4713533.40 frames. ], batch size: 170, lr: 6.76e-03, grad_scale: 32.0 2026-09-24 02:00:41,976 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=98300.0, ans=0.125 2026-09-24 02:00:43,531 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=98300.0, ans=0.1 2026-09-24 02:00:50,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=98366.66666666667, ans=0.0 2026-09-24 02:01:01,955 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.732e+02 3.346e+02 3.709e+02 4.178e+02 5.646e+02, threshold=7.418e+02, percent-clipped=0.0 2026-09-24 02:01:02,707 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.08 vs. limit=15.0 2026-09-24 02:01:05,410 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=18.51 vs. limit=22.5 2026-09-24 02:01:05,569 INFO [train.py:1192] (0/2) Epoch 31, batch 800, loss[loss=0.2505, simple_loss=0.357, pruned_loss=0.07198, over 24567.00 frames. ], tot_loss[loss=0.2871, simple_loss=0.3947, pruned_loss=0.08977, over 4737890.29 frames. ], batch size: 137, lr: 6.75e-03, grad_scale: 32.0 2026-09-24 02:01:05,663 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=98466.66666666667, ans=0.125 2026-09-24 02:01:17,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=98533.33333333333, ans=0.0 2026-09-24 02:01:18,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=98533.33333333333, ans=0.1 2026-09-24 02:01:21,683 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=98566.66666666667, ans=0.07 2026-09-24 02:01:23,063 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=98566.66666666667, ans=0.1 2026-09-24 02:01:29,797 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=98600.0, ans=0.125 2026-09-24 02:01:31,109 INFO [train.py:1192] (0/2) Epoch 31, batch 850, loss[loss=0.3026, simple_loss=0.4252, pruned_loss=0.09004, over 24536.00 frames. ], tot_loss[loss=0.2872, simple_loss=0.3949, pruned_loss=0.08973, over 4760707.32 frames. ], batch size: 204, lr: 6.75e-03, grad_scale: 32.0 2026-09-24 02:01:38,186 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.21 vs. limit=15.0 2026-09-24 02:01:41,623 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=98700.0, ans=0.0 2026-09-24 02:01:49,605 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.41 vs. limit=12.0 2026-09-24 02:01:50,572 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.25 vs. limit=12.0 2026-09-24 02:01:54,184 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.796e+02 3.307e+02 3.789e+02 4.588e+02 6.875e+02, threshold=7.578e+02, percent-clipped=0.0 2026-09-24 02:01:54,964 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.42 vs. limit=15.0 2026-09-24 02:01:57,774 INFO [train.py:1192] (0/2) Epoch 31, batch 900, loss[loss=0.2243, simple_loss=0.3392, pruned_loss=0.05468, over 24557.00 frames. ], tot_loss[loss=0.2875, simple_loss=0.3952, pruned_loss=0.08989, over 4774788.20 frames. ], batch size: 137, lr: 6.74e-03, grad_scale: 32.0 2026-09-24 02:01:57,877 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=98800.0, ans=0.025 2026-09-24 02:02:01,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=98800.0, ans=0.125 2026-09-24 02:02:20,907 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=98933.33333333333, ans=0.05 2026-09-24 02:02:23,254 INFO [train.py:1192] (0/2) Epoch 31, batch 950, loss[loss=0.4352, simple_loss=0.464, pruned_loss=0.2032, over 11635.00 frames. ], tot_loss[loss=0.2877, simple_loss=0.3938, pruned_loss=0.09081, over 4704663.99 frames. ], batch size: 334, lr: 6.74e-03, grad_scale: 32.0 2026-09-24 02:02:27,832 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-31.pt 2026-09-24 02:02:34,771 INFO [train.py:1192] (0/2) Epoch 32, batch 0, loss[loss=0.2352, simple_loss=0.3523, pruned_loss=0.0591, over 24556.00 frames. ], tot_loss[loss=0.2352, simple_loss=0.3523, pruned_loss=0.0591, over 24556.00 frames. ], batch size: 137, lr: 6.63e-03, grad_scale: 32.0 2026-09-24 02:02:34,771 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 02:02:46,482 INFO [train.py:1224] (0/2) Epoch 32, validation: loss=0.1822, simple_loss=0.3006, pruned_loss=0.03189, over 2564189.00 frames. 2026-09-24 02:02:46,483 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 02:02:53,414 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=99026.66666666667, ans=0.0 2026-09-24 02:03:03,299 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=99093.33333333333, ans=0.1 2026-09-24 02:03:03,758 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=99093.33333333333, ans=0.125 2026-09-24 02:03:04,818 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.330e+02 3.362e+02 3.762e+02 4.535e+02 8.123e+02, threshold=7.524e+02, percent-clipped=1.0 2026-09-24 02:03:06,001 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=99093.33333333333, ans=0.1 2026-09-24 02:03:10,543 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=99126.66666666667, ans=0.1 2026-09-24 02:03:11,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=99126.66666666667, ans=0.125 2026-09-24 02:03:13,300 INFO [train.py:1192] (0/2) Epoch 32, batch 50, loss[loss=0.2649, simple_loss=0.3627, pruned_loss=0.08354, over 24270.00 frames. ], tot_loss[loss=0.2961, simple_loss=0.4029, pruned_loss=0.09468, over 1077233.18 frames. ], batch size: 125, lr: 6.62e-03, grad_scale: 32.0 2026-09-24 02:03:14,335 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=99160.0, ans=0.0 2026-09-24 02:03:15,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=99160.0, ans=0.2 2026-09-24 02:03:16,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=99160.0, ans=0.0 2026-09-24 02:03:35,110 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=99293.33333333333, ans=0.125 2026-09-24 02:03:39,436 INFO [train.py:1192] (0/2) Epoch 32, batch 100, loss[loss=0.2723, simple_loss=0.3731, pruned_loss=0.08575, over 24594.00 frames. ], tot_loss[loss=0.2994, simple_loss=0.4067, pruned_loss=0.09604, over 1905178.22 frames. ], batch size: 154, lr: 6.62e-03, grad_scale: 32.0 2026-09-24 02:03:45,940 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=99360.0, ans=10.0 2026-09-24 02:03:49,777 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=99393.33333333333, ans=0.2 2026-09-24 02:03:51,476 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=5.04 vs. limit=15.0 2026-09-24 02:03:57,495 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.503e+02 3.354e+02 3.724e+02 4.176e+02 6.331e+02, threshold=7.449e+02, percent-clipped=0.0 2026-09-24 02:03:59,466 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:04:01,535 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=99460.0, ans=0.125 2026-09-24 02:04:04,689 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=99493.33333333333, ans=0.04949747468305833 2026-09-24 02:04:05,057 INFO [train.py:1192] (0/2) Epoch 32, batch 150, loss[loss=0.2444, simple_loss=0.3496, pruned_loss=0.0696, over 24303.00 frames. ], tot_loss[loss=0.2918, simple_loss=0.4003, pruned_loss=0.09168, over 2558820.32 frames. ], batch size: 125, lr: 6.61e-03, grad_scale: 32.0 2026-09-24 02:04:07,409 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=99493.33333333333, ans=0.0 2026-09-24 02:04:08,477 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=99493.33333333333, ans=0.0 2026-09-24 02:04:20,820 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=99593.33333333333, ans=0.125 2026-09-24 02:04:20,950 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.49 vs. limit=22.5 2026-09-24 02:04:21,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=99593.33333333333, ans=0.0 2026-09-24 02:04:23,264 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=99593.33333333333, ans=0.1 2026-09-24 02:04:24,250 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=99593.33333333333, ans=0.025 2026-09-24 02:04:24,748 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=99626.66666666667, ans=0.125 2026-09-24 02:04:24,754 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=99626.66666666667, ans=0.1 2026-09-24 02:04:30,776 INFO [train.py:1192] (0/2) Epoch 32, batch 200, loss[loss=0.3499, simple_loss=0.4401, pruned_loss=0.1298, over 21076.00 frames. ], tot_loss[loss=0.2896, simple_loss=0.3983, pruned_loss=0.0905, over 3056600.51 frames. ], batch size: 333, lr: 6.61e-03, grad_scale: 32.0 2026-09-24 02:04:35,931 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=99693.33333333333, ans=0.125 2026-09-24 02:04:37,763 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=99693.33333333333, ans=0.125 2026-09-24 02:04:47,948 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=99760.0, ans=0.125 2026-09-24 02:04:48,285 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.312e+02 3.441e+02 4.026e+02 4.721e+02 9.759e+02, threshold=8.053e+02, percent-clipped=2.0 2026-09-24 02:04:49,387 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=99760.0, ans=0.125 2026-09-24 02:04:56,164 INFO [train.py:1192] (0/2) Epoch 32, batch 250, loss[loss=0.3383, simple_loss=0.443, pruned_loss=0.1168, over 24295.00 frames. ], tot_loss[loss=0.2883, simple_loss=0.3971, pruned_loss=0.08977, over 3444958.06 frames. ], batch size: 234, lr: 6.60e-03, grad_scale: 32.0 2026-09-24 02:04:58,306 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff3.min_abs, batch_count=99826.66666666667, ans=0.2 2026-09-24 02:05:07,591 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=99893.33333333333, ans=0.1 2026-09-24 02:05:16,314 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.31 vs. limit=12.0 2026-09-24 02:05:18,378 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=99960.0, ans=0.125 2026-09-24 02:05:21,920 INFO [train.py:1192] (0/2) Epoch 32, batch 300, loss[loss=0.3052, simple_loss=0.4145, pruned_loss=0.09798, over 24539.00 frames. ], tot_loss[loss=0.2867, simple_loss=0.3952, pruned_loss=0.08908, over 3749436.84 frames. ], batch size: 204, lr: 6.60e-03, grad_scale: 32.0 2026-09-24 02:05:25,549 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=99993.33333333333, ans=0.025 2026-09-24 02:05:30,936 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=100026.66666666667, ans=0.125 2026-09-24 02:05:40,274 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.472e+02 3.268e+02 3.539e+02 4.255e+02 5.982e+02, threshold=7.078e+02, percent-clipped=0.0 2026-09-24 02:05:47,879 INFO [train.py:1192] (0/2) Epoch 32, batch 350, loss[loss=0.2504, simple_loss=0.3583, pruned_loss=0.07122, over 24534.00 frames. ], tot_loss[loss=0.2882, simple_loss=0.3967, pruned_loss=0.08982, over 3989094.53 frames. ], batch size: 137, lr: 6.59e-03, grad_scale: 32.0 2026-09-24 02:05:52,300 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=100160.0, ans=0.125 2026-09-24 02:05:56,676 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.29 vs. limit=15.0 2026-09-24 02:06:02,044 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=100226.66666666667, ans=0.0 2026-09-24 02:06:03,220 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=4.92 vs. limit=15.0 2026-09-24 02:06:06,755 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=100260.0, ans=0.0 2026-09-24 02:06:06,767 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=100260.0, ans=0.0 2026-09-24 02:06:12,944 INFO [train.py:1192] (0/2) Epoch 32, batch 400, loss[loss=0.2729, simple_loss=0.3855, pruned_loss=0.0801, over 24583.00 frames. ], tot_loss[loss=0.2863, simple_loss=0.3951, pruned_loss=0.08871, over 4173576.84 frames. ], batch size: 170, lr: 6.59e-03, grad_scale: 32.0 2026-09-24 02:06:21,573 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:06:29,689 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=100426.66666666667, ans=0.125 2026-09-24 02:06:31,447 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.511e+02 3.151e+02 3.623e+02 4.376e+02 7.654e+02, threshold=7.247e+02, percent-clipped=1.0 2026-09-24 02:06:31,568 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=100426.66666666667, ans=0.0 2026-09-24 02:06:38,940 INFO [train.py:1192] (0/2) Epoch 32, batch 450, loss[loss=0.302, simple_loss=0.4154, pruned_loss=0.09426, over 24633.00 frames. ], tot_loss[loss=0.2869, simple_loss=0.3955, pruned_loss=0.0891, over 4308473.69 frames. ], batch size: 175, lr: 6.58e-03, grad_scale: 32.0 2026-09-24 02:06:47,184 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=100526.66666666667, ans=0.125 2026-09-24 02:06:59,163 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=100626.66666666667, ans=0.125 2026-09-24 02:07:04,781 INFO [train.py:1192] (0/2) Epoch 32, batch 500, loss[loss=0.3484, simple_loss=0.4533, pruned_loss=0.1218, over 24507.00 frames. ], tot_loss[loss=0.285, simple_loss=0.3937, pruned_loss=0.08818, over 4426391.24 frames. ], batch size: 218, lr: 6.58e-03, grad_scale: 32.0 2026-09-24 02:07:04,888 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=100660.0, ans=0.2 2026-09-24 02:07:06,345 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=100660.0, ans=0.2 2026-09-24 02:07:12,966 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=100693.33333333333, ans=0.2 2026-09-24 02:07:20,150 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=100760.0, ans=0.1 2026-09-24 02:07:23,026 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.464e+02 3.154e+02 3.498e+02 3.927e+02 5.241e+02, threshold=6.996e+02, percent-clipped=0.0 2026-09-24 02:07:30,716 INFO [train.py:1192] (0/2) Epoch 32, batch 550, loss[loss=0.3026, simple_loss=0.4165, pruned_loss=0.09435, over 24273.00 frames. ], tot_loss[loss=0.2854, simple_loss=0.3942, pruned_loss=0.08827, over 4516301.25 frames. ], batch size: 257, lr: 6.57e-03, grad_scale: 32.0 2026-09-24 02:07:39,997 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=100860.0, ans=0.125 2026-09-24 02:07:41,983 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.79 vs. limit=15.0 2026-09-24 02:07:56,913 INFO [train.py:1192] (0/2) Epoch 32, batch 600, loss[loss=0.3022, simple_loss=0.4176, pruned_loss=0.0934, over 24327.00 frames. ], tot_loss[loss=0.2864, simple_loss=0.3952, pruned_loss=0.08883, over 4583366.83 frames. ], batch size: 234, lr: 6.57e-03, grad_scale: 32.0 2026-09-24 02:08:06,941 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.55 vs. limit=22.5 2026-09-24 02:08:14,663 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=101093.33333333333, ans=0.0 2026-09-24 02:08:14,995 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.576e+02 3.196e+02 3.540e+02 4.133e+02 6.002e+02, threshold=7.080e+02, percent-clipped=0.0 2026-09-24 02:08:22,482 INFO [train.py:1192] (0/2) Epoch 32, batch 650, loss[loss=0.293, simple_loss=0.3968, pruned_loss=0.09462, over 24562.00 frames. ], tot_loss[loss=0.2848, simple_loss=0.394, pruned_loss=0.08778, over 4649303.38 frames. ], batch size: 162, lr: 6.56e-03, grad_scale: 32.0 2026-09-24 02:08:32,749 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.16 vs. limit=15.0 2026-09-24 02:08:36,145 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=101226.66666666667, ans=0.125 2026-09-24 02:08:41,087 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=101260.0, ans=0.125 2026-09-24 02:08:48,679 INFO [train.py:1192] (0/2) Epoch 32, batch 700, loss[loss=0.2716, simple_loss=0.3728, pruned_loss=0.0852, over 24591.00 frames. ], tot_loss[loss=0.2859, simple_loss=0.3952, pruned_loss=0.08835, over 4681537.88 frames. ], batch size: 154, lr: 6.56e-03, grad_scale: 32.0 2026-09-24 02:09:07,204 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.596e+02 3.226e+02 3.648e+02 4.091e+02 6.979e+02, threshold=7.297e+02, percent-clipped=0.0 2026-09-24 02:09:08,585 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=101426.66666666667, ans=0.0 2026-09-24 02:09:11,254 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.36 vs. limit=15.0 2026-09-24 02:09:14,935 INFO [train.py:1192] (0/2) Epoch 32, batch 750, loss[loss=0.2926, simple_loss=0.4024, pruned_loss=0.09138, over 24553.00 frames. ], tot_loss[loss=0.2857, simple_loss=0.3942, pruned_loss=0.08856, over 4708922.74 frames. ], batch size: 170, lr: 6.55e-03, grad_scale: 32.0 2026-09-24 02:09:17,608 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=101493.33333333333, ans=0.1 2026-09-24 02:09:20,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=101526.66666666667, ans=0.05 2026-09-24 02:09:22,673 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=101526.66666666667, ans=0.125 2026-09-24 02:09:22,687 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=101526.66666666667, ans=0.1 2026-09-24 02:09:26,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=101560.0, ans=0.025 2026-09-24 02:09:26,581 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=101560.0, ans=0.125 2026-09-24 02:09:30,485 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=101593.33333333333, ans=0.0 2026-09-24 02:09:40,624 INFO [train.py:1192] (0/2) Epoch 32, batch 800, loss[loss=0.2468, simple_loss=0.356, pruned_loss=0.06878, over 24528.00 frames. ], tot_loss[loss=0.2848, simple_loss=0.3936, pruned_loss=0.08799, over 4734504.60 frames. ], batch size: 137, lr: 6.55e-03, grad_scale: 32.0 2026-09-24 02:09:47,088 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=101693.33333333333, ans=0.125 2026-09-24 02:09:48,969 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=7.08 vs. limit=12.0 2026-09-24 02:09:52,649 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=101726.66666666667, ans=0.0 2026-09-24 02:09:54,542 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=101726.66666666667, ans=0.125 2026-09-24 02:09:56,424 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.66 vs. limit=6.0 2026-09-24 02:09:58,468 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.569e+02 3.280e+02 3.615e+02 4.129e+02 5.868e+02, threshold=7.230e+02, percent-clipped=0.0 2026-09-24 02:10:06,170 INFO [train.py:1192] (0/2) Epoch 32, batch 850, loss[loss=0.3198, simple_loss=0.433, pruned_loss=0.1033, over 24537.00 frames. ], tot_loss[loss=0.2846, simple_loss=0.3934, pruned_loss=0.08796, over 4757648.50 frames. ], batch size: 204, lr: 6.54e-03, grad_scale: 32.0 2026-09-24 02:10:10,456 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=101826.66666666667, ans=0.025 2026-09-24 02:10:18,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=101893.33333333333, ans=0.0 2026-09-24 02:10:28,598 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=101960.0, ans=0.1 2026-09-24 02:10:32,281 INFO [train.py:1192] (0/2) Epoch 32, batch 900, loss[loss=0.2175, simple_loss=0.3378, pruned_loss=0.04864, over 24568.00 frames. ], tot_loss[loss=0.2853, simple_loss=0.394, pruned_loss=0.08834, over 4771475.45 frames. ], batch size: 137, lr: 6.54e-03, grad_scale: 32.0 2026-09-24 02:10:45,806 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.48 vs. limit=22.5 2026-09-24 02:10:47,153 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=102093.33333333333, ans=0.2 2026-09-24 02:10:48,637 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=102093.33333333333, ans=0.125 2026-09-24 02:10:49,908 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.338e+02 3.264e+02 3.534e+02 4.046e+02 5.970e+02, threshold=7.068e+02, percent-clipped=0.0 2026-09-24 02:10:50,936 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=102093.33333333333, ans=0.125 2026-09-24 02:10:55,100 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=102126.66666666667, ans=0.0 2026-09-24 02:10:57,047 INFO [train.py:1192] (0/2) Epoch 32, batch 950, loss[loss=0.3888, simple_loss=0.4431, pruned_loss=0.1673, over 10902.00 frames. ], tot_loss[loss=0.2857, simple_loss=0.393, pruned_loss=0.08919, over 4714945.59 frames. ], batch size: 334, lr: 6.53e-03, grad_scale: 32.0 2026-09-24 02:10:57,114 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=102160.0, ans=0.025 2026-09-24 02:10:58,784 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=102160.0, ans=0.1 2026-09-24 02:11:01,580 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-32.pt 2026-09-24 02:11:08,241 INFO [train.py:1192] (0/2) Epoch 33, batch 0, loss[loss=0.2649, simple_loss=0.3705, pruned_loss=0.07966, over 24557.00 frames. ], tot_loss[loss=0.2649, simple_loss=0.3705, pruned_loss=0.07966, over 24557.00 frames. ], batch size: 137, lr: 6.43e-03, grad_scale: 32.0 2026-09-24 02:11:08,242 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 02:11:17,247 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([3.1984, 3.0297, 2.7018, 3.9229], device='cuda:0') 2026-09-24 02:11:19,601 INFO [train.py:1224] (0/2) Epoch 33, validation: loss=0.1799, simple_loss=0.2984, pruned_loss=0.03069, over 2564189.00 frames. 2026-09-24 02:11:19,601 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 02:11:30,688 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=102253.33333333333, ans=0.125 2026-09-24 02:11:32,563 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=102253.33333333333, ans=0.1 2026-09-24 02:11:32,581 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=102253.33333333333, ans=0.125 2026-09-24 02:11:38,113 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=102286.66666666667, ans=0.2 2026-09-24 02:11:40,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=102320.0, ans=0.035 2026-09-24 02:11:42,367 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.90 vs. limit=22.5 2026-09-24 02:11:46,277 INFO [train.py:1192] (0/2) Epoch 33, batch 50, loss[loss=0.2244, simple_loss=0.3343, pruned_loss=0.05725, over 24293.00 frames. ], tot_loss[loss=0.2927, simple_loss=0.3995, pruned_loss=0.09301, over 1075172.38 frames. ], batch size: 125, lr: 6.42e-03, grad_scale: 32.0 2026-09-24 02:11:47,905 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=102353.33333333333, ans=0.0 2026-09-24 02:11:50,505 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.71 vs. limit=15.0 2026-09-24 02:11:56,955 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=102420.0, ans=0.125 2026-09-24 02:11:59,262 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=102420.0, ans=0.125 2026-09-24 02:12:00,097 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.782e+02 3.355e+02 3.885e+02 4.255e+02 6.960e+02, threshold=7.770e+02, percent-clipped=0.0 2026-09-24 02:12:10,331 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=102486.66666666667, ans=0.125 2026-09-24 02:12:12,167 INFO [train.py:1192] (0/2) Epoch 33, batch 100, loss[loss=0.2476, simple_loss=0.3628, pruned_loss=0.06622, over 24595.00 frames. ], tot_loss[loss=0.2933, simple_loss=0.4017, pruned_loss=0.09243, over 1904035.46 frames. ], batch size: 154, lr: 6.42e-03, grad_scale: 64.0 2026-09-24 02:12:18,749 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=8.57 vs. limit=15.0 2026-09-24 02:12:26,261 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.46 vs. limit=22.5 2026-09-24 02:12:32,032 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=102620.0, ans=0.0 2026-09-24 02:12:33,012 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=102653.33333333333, ans=0.0 2026-09-24 02:12:38,094 INFO [train.py:1192] (0/2) Epoch 33, batch 150, loss[loss=0.231, simple_loss=0.3369, pruned_loss=0.06254, over 24264.00 frames. ], tot_loss[loss=0.2897, simple_loss=0.3983, pruned_loss=0.09053, over 2557674.02 frames. ], batch size: 125, lr: 6.41e-03, grad_scale: 64.0 2026-09-24 02:12:42,213 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=102686.66666666667, ans=0.125 2026-09-24 02:12:51,631 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.186e+02 3.285e+02 3.723e+02 4.193e+02 6.149e+02, threshold=7.446e+02, percent-clipped=0.0 2026-09-24 02:12:53,755 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=102786.66666666667, ans=0.0 2026-09-24 02:13:03,054 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=102853.33333333333, ans=0.0 2026-09-24 02:13:03,347 INFO [train.py:1192] (0/2) Epoch 33, batch 200, loss[loss=0.317, simple_loss=0.4192, pruned_loss=0.1074, over 20960.00 frames. ], tot_loss[loss=0.2862, simple_loss=0.3955, pruned_loss=0.08845, over 3054133.78 frames. ], batch size: 333, lr: 6.41e-03, grad_scale: 64.0 2026-09-24 02:13:05,948 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.74 vs. limit=15.0 2026-09-24 02:13:21,616 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=102953.33333333333, ans=0.1 2026-09-24 02:13:29,433 INFO [train.py:1192] (0/2) Epoch 33, batch 250, loss[loss=0.3033, simple_loss=0.4172, pruned_loss=0.09471, over 24319.00 frames. ], tot_loss[loss=0.2855, simple_loss=0.3946, pruned_loss=0.08823, over 3442697.24 frames. ], batch size: 234, lr: 6.40e-03, grad_scale: 32.0 2026-09-24 02:13:40,887 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=103086.66666666667, ans=0.125 2026-09-24 02:13:44,468 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.787e+02 3.379e+02 3.953e+02 4.528e+02 8.699e+02, threshold=7.906e+02, percent-clipped=3.0 2026-09-24 02:13:55,535 INFO [train.py:1192] (0/2) Epoch 33, batch 300, loss[loss=0.2874, simple_loss=0.4005, pruned_loss=0.08717, over 24553.00 frames. ], tot_loss[loss=0.285, simple_loss=0.3934, pruned_loss=0.0883, over 3748096.39 frames. ], batch size: 204, lr: 6.40e-03, grad_scale: 32.0 2026-09-24 02:14:08,780 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.66 vs. limit=15.0 2026-09-24 02:14:13,493 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=103286.66666666667, ans=0.125 2026-09-24 02:14:17,940 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=103320.0, ans=0.125 2026-09-24 02:14:21,253 INFO [train.py:1192] (0/2) Epoch 33, batch 350, loss[loss=0.2542, simple_loss=0.3533, pruned_loss=0.07755, over 24570.00 frames. ], tot_loss[loss=0.2861, simple_loss=0.3948, pruned_loss=0.08863, over 3991964.98 frames. ], batch size: 137, lr: 6.40e-03, grad_scale: 32.0 2026-09-24 02:14:23,429 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=103353.33333333333, ans=0.125 2026-09-24 02:14:26,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=103386.66666666667, ans=0.0 2026-09-24 02:14:35,548 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.10 vs. limit=12.0 2026-09-24 02:14:35,895 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.614e+02 3.333e+02 3.657e+02 4.243e+02 7.835e+02, threshold=7.314e+02, percent-clipped=0.0 2026-09-24 02:14:40,631 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=103453.33333333333, ans=0.025 2026-09-24 02:14:47,370 INFO [train.py:1192] (0/2) Epoch 33, batch 400, loss[loss=0.3127, simple_loss=0.4104, pruned_loss=0.1074, over 24562.00 frames. ], tot_loss[loss=0.285, simple_loss=0.3939, pruned_loss=0.08809, over 4176094.53 frames. ], batch size: 170, lr: 6.39e-03, grad_scale: 32.0 2026-09-24 02:14:50,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=103520.0, ans=0.2 2026-09-24 02:14:54,022 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=103553.33333333333, ans=0.125 2026-09-24 02:15:05,063 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.01 vs. limit=15.0 2026-09-24 02:15:11,655 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=103653.33333333333, ans=0.125 2026-09-24 02:15:13,092 INFO [train.py:1192] (0/2) Epoch 33, batch 450, loss[loss=0.3111, simple_loss=0.4177, pruned_loss=0.1023, over 24628.00 frames. ], tot_loss[loss=0.2849, simple_loss=0.3941, pruned_loss=0.08784, over 4307194.90 frames. ], batch size: 175, lr: 6.39e-03, grad_scale: 32.0 2026-09-24 02:15:27,676 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.570e+02 3.135e+02 3.583e+02 4.105e+02 5.710e+02, threshold=7.165e+02, percent-clipped=0.0 2026-09-24 02:15:30,986 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.60 vs. limit=15.0 2026-09-24 02:15:36,518 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=103820.0, ans=0.0 2026-09-24 02:15:39,003 INFO [train.py:1192] (0/2) Epoch 33, batch 500, loss[loss=0.3173, simple_loss=0.4286, pruned_loss=0.1029, over 24498.00 frames. ], tot_loss[loss=0.2844, simple_loss=0.3931, pruned_loss=0.08785, over 4425368.56 frames. ], batch size: 218, lr: 6.38e-03, grad_scale: 32.0 2026-09-24 02:15:39,971 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=103853.33333333333, ans=0.125 2026-09-24 02:15:45,180 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=103886.66666666667, ans=0.05 2026-09-24 02:15:51,488 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=103920.0, ans=0.125 2026-09-24 02:15:54,382 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=103953.33333333333, ans=0.125 2026-09-24 02:16:01,133 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.81 vs. limit=15.0 2026-09-24 02:16:04,928 INFO [train.py:1192] (0/2) Epoch 33, batch 550, loss[loss=0.3162, simple_loss=0.4291, pruned_loss=0.1016, over 24258.00 frames. ], tot_loss[loss=0.2846, simple_loss=0.3934, pruned_loss=0.08786, over 4515566.66 frames. ], batch size: 257, lr: 6.38e-03, grad_scale: 32.0 2026-09-24 02:16:05,039 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=104020.0, ans=0.1 2026-09-24 02:16:07,345 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=104020.0, ans=0.125 2026-09-24 02:16:16,734 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=104086.66666666667, ans=0.0 2026-09-24 02:16:18,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=104086.66666666667, ans=0.1 2026-09-24 02:16:19,231 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.31 vs. limit=15.0 2026-09-24 02:16:19,434 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.554e+02 3.273e+02 3.583e+02 4.011e+02 7.398e+02, threshold=7.166e+02, percent-clipped=1.0 2026-09-24 02:16:21,993 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:16:23,410 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=104120.0, ans=0.125 2026-09-24 02:16:30,480 INFO [train.py:1192] (0/2) Epoch 33, batch 600, loss[loss=0.3456, simple_loss=0.4503, pruned_loss=0.1205, over 24343.00 frames. ], tot_loss[loss=0.285, simple_loss=0.394, pruned_loss=0.088, over 4583186.83 frames. ], batch size: 234, lr: 6.37e-03, grad_scale: 32.0 2026-09-24 02:16:38,588 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=104220.0, ans=0.05 2026-09-24 02:16:50,682 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.38 vs. limit=15.0 2026-09-24 02:16:51,930 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=104320.0, ans=0.05 2026-09-24 02:16:56,634 INFO [train.py:1192] (0/2) Epoch 33, batch 650, loss[loss=0.2867, simple_loss=0.3923, pruned_loss=0.09052, over 24575.00 frames. ], tot_loss[loss=0.284, simple_loss=0.3931, pruned_loss=0.08742, over 4649051.29 frames. ], batch size: 162, lr: 6.37e-03, grad_scale: 32.0 2026-09-24 02:17:07,547 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=104420.0, ans=0.0 2026-09-24 02:17:11,054 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.600e+02 3.293e+02 3.637e+02 4.365e+02 6.291e+02, threshold=7.274e+02, percent-clipped=0.0 2026-09-24 02:17:20,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=104486.66666666667, ans=0.125 2026-09-24 02:17:22,478 INFO [train.py:1192] (0/2) Epoch 33, batch 700, loss[loss=0.2581, simple_loss=0.3648, pruned_loss=0.07571, over 24591.00 frames. ], tot_loss[loss=0.2846, simple_loss=0.3939, pruned_loss=0.08761, over 4681237.93 frames. ], batch size: 154, lr: 6.36e-03, grad_scale: 32.0 2026-09-24 02:17:30,099 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=104553.33333333333, ans=0.2 2026-09-24 02:17:30,681 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=104553.33333333333, ans=0.125 2026-09-24 02:17:42,485 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=104620.0, ans=0.0 2026-09-24 02:17:43,589 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=104653.33333333333, ans=0.2 2026-09-24 02:17:48,758 INFO [train.py:1192] (0/2) Epoch 33, batch 750, loss[loss=0.2885, simple_loss=0.4002, pruned_loss=0.08837, over 24567.00 frames. ], tot_loss[loss=0.2834, simple_loss=0.3928, pruned_loss=0.08705, over 4709019.04 frames. ], batch size: 170, lr: 6.36e-03, grad_scale: 32.0 2026-09-24 02:17:57,810 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.70 vs. limit=15.0 2026-09-24 02:18:03,151 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.772e+02 3.276e+02 3.703e+02 4.111e+02 6.270e+02, threshold=7.406e+02, percent-clipped=0.0 2026-09-24 02:18:14,586 INFO [train.py:1192] (0/2) Epoch 33, batch 800, loss[loss=0.2175, simple_loss=0.3286, pruned_loss=0.05323, over 24556.00 frames. ], tot_loss[loss=0.2835, simple_loss=0.3926, pruned_loss=0.08717, over 4734772.58 frames. ], batch size: 137, lr: 6.35e-03, grad_scale: 32.0 2026-09-24 02:18:14,695 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=104853.33333333333, ans=0.2 2026-09-24 02:18:22,420 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=104886.66666666667, ans=0.125 2026-09-24 02:18:36,297 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=104986.66666666667, ans=0.0 2026-09-24 02:18:38,400 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=104986.66666666667, ans=0.125 2026-09-24 02:18:40,257 INFO [train.py:1192] (0/2) Epoch 33, batch 850, loss[loss=0.2659, simple_loss=0.3896, pruned_loss=0.07114, over 24594.00 frames. ], tot_loss[loss=0.2825, simple_loss=0.3919, pruned_loss=0.0865, over 4757492.16 frames. ], batch size: 198, lr: 6.35e-03, grad_scale: 16.0 2026-09-24 02:18:44,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=105020.0, ans=0.125 2026-09-24 02:18:55,062 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.682e+02 3.209e+02 3.625e+02 3.953e+02 5.601e+02, threshold=7.250e+02, percent-clipped=0.0 2026-09-24 02:18:55,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=105120.0, ans=0.0 2026-09-24 02:19:04,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=105153.33333333333, ans=0.1 2026-09-24 02:19:05,507 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=105186.66666666667, ans=0.125 2026-09-24 02:19:05,522 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=105186.66666666667, ans=0.1 2026-09-24 02:19:05,854 INFO [train.py:1192] (0/2) Epoch 33, batch 900, loss[loss=0.2345, simple_loss=0.3467, pruned_loss=0.06115, over 24546.00 frames. ], tot_loss[loss=0.2828, simple_loss=0.3924, pruned_loss=0.08665, over 4771281.98 frames. ], batch size: 137, lr: 6.34e-03, grad_scale: 16.0 2026-09-24 02:19:13,268 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=105220.0, ans=0.0 2026-09-24 02:19:15,101 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=105220.0, ans=0.04949747468305833 2026-09-24 02:19:18,406 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=105253.33333333333, ans=0.125 2026-09-24 02:19:27,835 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=105320.0, ans=0.025 2026-09-24 02:19:30,551 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=105353.33333333333, ans=0.125 2026-09-24 02:19:30,976 INFO [train.py:1192] (0/2) Epoch 33, batch 950, loss[loss=0.4416, simple_loss=0.47, pruned_loss=0.2066, over 10988.00 frames. ], tot_loss[loss=0.2827, simple_loss=0.3907, pruned_loss=0.08732, over 4717086.58 frames. ], batch size: 334, lr: 6.34e-03, grad_scale: 16.0 2026-09-24 02:19:32,613 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=105353.33333333333, ans=0.2 2026-09-24 02:19:34,117 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=105353.33333333333, ans=0.1 2026-09-24 02:19:35,743 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-33.pt 2026-09-24 02:20:14,634 INFO [train.py:1192] (0/2) Epoch 34, batch 0, loss[loss=0.2244, simple_loss=0.34, pruned_loss=0.05438, over 24578.00 frames. ], tot_loss[loss=0.2244, simple_loss=0.34, pruned_loss=0.05438, over 24578.00 frames. ], batch size: 137, lr: 6.24e-03, grad_scale: 32.0 2026-09-24 02:20:14,634 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 02:20:16,706 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.8485, 2.5030, 2.4927, 1.9749, 2.7523, 2.4103, 2.6467, 1.9745], device='cuda:0') 2026-09-24 02:20:26,324 INFO [train.py:1224] (0/2) Epoch 34, validation: loss=0.1791, simple_loss=0.2978, pruned_loss=0.03024, over 2564189.00 frames. 2026-09-24 02:20:26,324 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 02:20:31,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=105413.33333333333, ans=0.1 2026-09-24 02:20:37,071 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.525e+02 3.365e+02 3.705e+02 4.305e+02 8.296e+02, threshold=7.410e+02, percent-clipped=2.0 2026-09-24 02:20:52,267 INFO [train.py:1192] (0/2) Epoch 34, batch 50, loss[loss=0.2621, simple_loss=0.3568, pruned_loss=0.08369, over 24232.00 frames. ], tot_loss[loss=0.2966, simple_loss=0.4023, pruned_loss=0.09542, over 1076985.99 frames. ], batch size: 125, lr: 6.24e-03, grad_scale: 32.0 2026-09-24 02:20:59,928 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=105580.0, ans=0.125 2026-09-24 02:21:08,545 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=105646.66666666667, ans=0.0 2026-09-24 02:21:12,664 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=105680.0, ans=0.0 2026-09-24 02:21:16,654 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=105680.0, ans=0.125 2026-09-24 02:21:18,025 INFO [train.py:1192] (0/2) Epoch 34, batch 100, loss[loss=0.2713, simple_loss=0.3813, pruned_loss=0.08068, over 24611.00 frames. ], tot_loss[loss=0.2967, simple_loss=0.4053, pruned_loss=0.09402, over 1905354.26 frames. ], batch size: 154, lr: 6.23e-03, grad_scale: 32.0 2026-09-24 02:21:18,125 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=105713.33333333333, ans=0.0 2026-09-24 02:21:23,724 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=105746.66666666667, ans=0.0 2026-09-24 02:21:26,326 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.40 vs. limit=15.0 2026-09-24 02:21:28,614 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.684e+02 3.438e+02 3.805e+02 4.609e+02 7.277e+02, threshold=7.610e+02, percent-clipped=0.0 2026-09-24 02:21:32,815 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=105813.33333333333, ans=0.125 2026-09-24 02:21:34,277 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=105813.33333333333, ans=0.125 2026-09-24 02:21:35,794 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=105813.33333333333, ans=0.125 2026-09-24 02:21:36,669 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=105813.33333333333, ans=0.0 2026-09-24 02:21:38,188 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=105846.66666666667, ans=0.2 2026-09-24 02:21:39,140 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=105846.66666666667, ans=0.125 2026-09-24 02:21:43,487 INFO [train.py:1192] (0/2) Epoch 34, batch 150, loss[loss=0.2532, simple_loss=0.3514, pruned_loss=0.07748, over 24242.00 frames. ], tot_loss[loss=0.2901, simple_loss=0.3994, pruned_loss=0.09036, over 2558887.47 frames. ], batch size: 125, lr: 6.23e-03, grad_scale: 32.0 2026-09-24 02:21:44,771 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.61 vs. limit=12.0 2026-09-24 02:21:45,620 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=105880.0, ans=0.125 2026-09-24 02:22:08,732 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.48 vs. limit=15.0 2026-09-24 02:22:09,522 INFO [train.py:1192] (0/2) Epoch 34, batch 200, loss[loss=0.3527, simple_loss=0.4406, pruned_loss=0.1324, over 21032.00 frames. ], tot_loss[loss=0.2871, simple_loss=0.3965, pruned_loss=0.08879, over 3055709.29 frames. ], batch size: 333, lr: 6.22e-03, grad_scale: 32.0 2026-09-24 02:22:09,790 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.28 vs. limit=15.0 2026-09-24 02:22:12,239 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.53 vs. limit=15.0 2026-09-24 02:22:13,468 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=106046.66666666667, ans=0.2 2026-09-24 02:22:20,345 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.605e+02 3.582e+02 4.182e+02 5.101e+02 7.909e+02, threshold=8.363e+02, percent-clipped=1.0 2026-09-24 02:22:22,048 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=106113.33333333333, ans=0.1 2026-09-24 02:22:22,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=106113.33333333333, ans=0.0 2026-09-24 02:22:33,300 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=10.94 vs. limit=15.0 2026-09-24 02:22:35,454 INFO [train.py:1192] (0/2) Epoch 34, batch 250, loss[loss=0.3221, simple_loss=0.4391, pruned_loss=0.1026, over 24294.00 frames. ], tot_loss[loss=0.2864, simple_loss=0.3958, pruned_loss=0.08853, over 3444203.39 frames. ], batch size: 234, lr: 6.22e-03, grad_scale: 32.0 2026-09-24 02:22:42,235 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=106246.66666666667, ans=0.0 2026-09-24 02:22:45,804 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=106280.0, ans=0.125 2026-09-24 02:22:46,877 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.74 vs. limit=15.0 2026-09-24 02:22:50,120 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=106280.0, ans=0.125 2026-09-24 02:22:51,974 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=106313.33333333333, ans=0.125 2026-09-24 02:23:01,171 INFO [train.py:1192] (0/2) Epoch 34, batch 300, loss[loss=0.3097, simple_loss=0.4201, pruned_loss=0.09963, over 24557.00 frames. ], tot_loss[loss=0.2853, simple_loss=0.3942, pruned_loss=0.08816, over 3749440.92 frames. ], batch size: 204, lr: 6.21e-03, grad_scale: 32.0 2026-09-24 02:23:10,389 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=106413.33333333333, ans=0.0 2026-09-24 02:23:12,089 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.676e+02 3.333e+02 3.803e+02 4.108e+02 6.453e+02, threshold=7.607e+02, percent-clipped=0.0 2026-09-24 02:23:12,712 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=106446.66666666667, ans=0.1 2026-09-24 02:23:17,246 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.15 vs. limit=6.0 2026-09-24 02:23:23,970 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=106513.33333333333, ans=0.0 2026-09-24 02:23:26,636 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=106546.66666666667, ans=0.0 2026-09-24 02:23:26,967 INFO [train.py:1192] (0/2) Epoch 34, batch 350, loss[loss=0.2252, simple_loss=0.336, pruned_loss=0.05724, over 24587.00 frames. ], tot_loss[loss=0.2855, simple_loss=0.3947, pruned_loss=0.08817, over 3992693.22 frames. ], batch size: 137, lr: 6.21e-03, grad_scale: 32.0 2026-09-24 02:23:35,876 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=106580.0, ans=0.125 2026-09-24 02:23:42,951 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=106646.66666666667, ans=0.125 2026-09-24 02:23:45,189 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.53 vs. limit=15.0 2026-09-24 02:23:45,479 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-32000.pt 2026-09-24 02:23:45,819 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=106646.66666666667, ans=0.125 2026-09-24 02:23:48,784 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.60 vs. limit=15.0 2026-09-24 02:23:53,629 INFO [train.py:1192] (0/2) Epoch 34, batch 400, loss[loss=0.288, simple_loss=0.3941, pruned_loss=0.09096, over 24555.00 frames. ], tot_loss[loss=0.2844, simple_loss=0.3935, pruned_loss=0.08761, over 4178690.22 frames. ], batch size: 170, lr: 6.20e-03, grad_scale: 32.0 2026-09-24 02:23:58,213 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=106746.66666666667, ans=0.125 2026-09-24 02:23:58,231 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=106746.66666666667, ans=0.125 2026-09-24 02:24:04,410 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.466e+02 3.230e+02 3.623e+02 4.018e+02 6.105e+02, threshold=7.247e+02, percent-clipped=0.0 2026-09-24 02:24:19,490 INFO [train.py:1192] (0/2) Epoch 34, batch 450, loss[loss=0.2804, simple_loss=0.3973, pruned_loss=0.0818, over 24632.00 frames. ], tot_loss[loss=0.2844, simple_loss=0.3938, pruned_loss=0.08754, over 4311906.71 frames. ], batch size: 175, lr: 6.20e-03, grad_scale: 32.0 2026-09-24 02:24:20,527 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=106880.0, ans=0.0 2026-09-24 02:24:23,302 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:24:23,758 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=106913.33333333333, ans=0.125 2026-09-24 02:24:34,553 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=106980.0, ans=0.1 2026-09-24 02:24:38,508 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=106980.0, ans=0.125 2026-09-24 02:24:39,609 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0 2026-09-24 02:24:39,612 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.12 vs. limit=22.5 2026-09-24 02:24:43,298 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=107013.33333333333, ans=0.125 2026-09-24 02:24:44,674 INFO [train.py:1192] (0/2) Epoch 34, batch 500, loss[loss=0.3067, simple_loss=0.4214, pruned_loss=0.09602, over 24489.00 frames. ], tot_loss[loss=0.283, simple_loss=0.3921, pruned_loss=0.08694, over 4429032.34 frames. ], batch size: 218, lr: 6.19e-03, grad_scale: 32.0 2026-09-24 02:24:44,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=107046.66666666667, ans=10.0 2026-09-24 02:24:46,811 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=107046.66666666667, ans=0.125 2026-09-24 02:24:48,890 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=107046.66666666667, ans=0.2 2026-09-24 02:24:55,873 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.612e+02 3.167e+02 3.567e+02 4.304e+02 6.742e+02, threshold=7.134e+02, percent-clipped=0.0 2026-09-24 02:25:10,541 INFO [train.py:1192] (0/2) Epoch 34, batch 550, loss[loss=0.2955, simple_loss=0.417, pruned_loss=0.08701, over 24219.00 frames. ], tot_loss[loss=0.283, simple_loss=0.3924, pruned_loss=0.08678, over 4518208.39 frames. ], batch size: 257, lr: 6.19e-03, grad_scale: 32.0 2026-09-24 02:25:21,250 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=107280.0, ans=0.125 2026-09-24 02:25:34,830 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=107346.66666666667, ans=0.0 2026-09-24 02:25:36,126 INFO [train.py:1192] (0/2) Epoch 34, batch 600, loss[loss=0.2617, simple_loss=0.3939, pruned_loss=0.06476, over 24331.00 frames. ], tot_loss[loss=0.2832, simple_loss=0.393, pruned_loss=0.0867, over 4585584.17 frames. ], batch size: 234, lr: 6.19e-03, grad_scale: 32.0 2026-09-24 02:25:43,857 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.40 vs. limit=15.0 2026-09-24 02:25:46,841 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.437e+02 3.136e+02 3.565e+02 4.273e+02 7.446e+02, threshold=7.130e+02, percent-clipped=2.0 2026-09-24 02:25:49,132 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.76 vs. limit=15.0 2026-09-24 02:25:52,117 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=107480.0, ans=0.125 2026-09-24 02:26:01,688 INFO [train.py:1192] (0/2) Epoch 34, batch 650, loss[loss=0.2707, simple_loss=0.3833, pruned_loss=0.07906, over 24557.00 frames. ], tot_loss[loss=0.2819, simple_loss=0.392, pruned_loss=0.08591, over 4651046.08 frames. ], batch size: 162, lr: 6.18e-03, grad_scale: 32.0 2026-09-24 02:26:01,792 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=107546.66666666667, ans=0.2 2026-09-24 02:26:08,869 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=107580.0, ans=0.0 2026-09-24 02:26:10,325 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=107580.0, ans=0.5 2026-09-24 02:26:15,143 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=107613.33333333333, ans=0.125 2026-09-24 02:26:15,634 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=107613.33333333333, ans=0.125 2026-09-24 02:26:16,605 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.60 vs. limit=10.0 2026-09-24 02:26:20,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=107646.66666666667, ans=0.125 2026-09-24 02:26:24,334 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.10 vs. limit=15.0 2026-09-24 02:26:26,588 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.89 vs. limit=15.0 2026-09-24 02:26:27,331 INFO [train.py:1192] (0/2) Epoch 34, batch 700, loss[loss=0.2649, simple_loss=0.3743, pruned_loss=0.07776, over 24571.00 frames. ], tot_loss[loss=0.282, simple_loss=0.3924, pruned_loss=0.08587, over 4683823.52 frames. ], batch size: 154, lr: 6.18e-03, grad_scale: 32.0 2026-09-24 02:26:28,744 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=107713.33333333333, ans=0.1 2026-09-24 02:26:29,235 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=107713.33333333333, ans=0.125 2026-09-24 02:26:29,261 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=107713.33333333333, ans=0.125 2026-09-24 02:26:38,486 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.549e+02 3.203e+02 3.581e+02 4.318e+02 6.316e+02, threshold=7.163e+02, percent-clipped=0.0 2026-09-24 02:26:39,549 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=107780.0, ans=0.1 2026-09-24 02:26:47,455 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.36 vs. limit=15.0 2026-09-24 02:26:52,546 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=107846.66666666667, ans=0.125 2026-09-24 02:26:53,935 INFO [train.py:1192] (0/2) Epoch 34, batch 750, loss[loss=0.2677, simple_loss=0.3862, pruned_loss=0.07463, over 24578.00 frames. ], tot_loss[loss=0.2823, simple_loss=0.392, pruned_loss=0.0863, over 4710795.38 frames. ], batch size: 170, lr: 6.17e-03, grad_scale: 32.0 2026-09-24 02:26:55,348 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:26:56,359 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=107880.0, ans=0.125 2026-09-24 02:26:57,815 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=107880.0, ans=15.0 2026-09-24 02:26:59,027 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=107913.33333333333, ans=0.025 2026-09-24 02:27:07,519 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=107946.66666666667, ans=0.2 2026-09-24 02:27:19,624 INFO [train.py:1192] (0/2) Epoch 34, batch 800, loss[loss=0.2304, simple_loss=0.3427, pruned_loss=0.05906, over 24534.00 frames. ], tot_loss[loss=0.2809, simple_loss=0.391, pruned_loss=0.08542, over 4740263.58 frames. ], batch size: 137, lr: 6.17e-03, grad_scale: 32.0 2026-09-24 02:27:20,668 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=108046.66666666667, ans=0.125 2026-09-24 02:27:27,807 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0 2026-09-24 02:27:29,927 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.511e+02 3.141e+02 3.618e+02 4.122e+02 6.477e+02, threshold=7.235e+02, percent-clipped=0.0 2026-09-24 02:27:33,154 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=108113.33333333333, ans=0.125 2026-09-24 02:27:35,830 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=108146.66666666667, ans=15.0 2026-09-24 02:27:37,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=108146.66666666667, ans=0.0 2026-09-24 02:27:37,735 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=108146.66666666667, ans=0.0 2026-09-24 02:27:41,431 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=9.28 vs. limit=15.0 2026-09-24 02:27:45,346 INFO [train.py:1192] (0/2) Epoch 34, batch 850, loss[loss=0.3145, simple_loss=0.4271, pruned_loss=0.101, over 24549.00 frames. ], tot_loss[loss=0.28, simple_loss=0.3902, pruned_loss=0.08491, over 4761684.72 frames. ], batch size: 204, lr: 6.16e-03, grad_scale: 32.0 2026-09-24 02:28:02,491 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=108313.33333333333, ans=0.125 2026-09-24 02:28:04,670 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.36 vs. limit=15.0 2026-09-24 02:28:11,293 INFO [train.py:1192] (0/2) Epoch 34, batch 900, loss[loss=0.2216, simple_loss=0.3363, pruned_loss=0.05345, over 24534.00 frames. ], tot_loss[loss=0.281, simple_loss=0.3909, pruned_loss=0.08552, over 4774544.73 frames. ], batch size: 137, lr: 6.16e-03, grad_scale: 32.0 2026-09-24 02:28:16,264 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=108413.33333333333, ans=0.125 2026-09-24 02:28:22,342 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.549e+02 3.205e+02 3.680e+02 4.328e+02 6.277e+02, threshold=7.359e+02, percent-clipped=0.0 2026-09-24 02:28:25,315 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=108446.66666666667, ans=0.0 2026-09-24 02:28:36,637 INFO [train.py:1192] (0/2) Epoch 34, batch 950, loss[loss=0.3628, simple_loss=0.4226, pruned_loss=0.1515, over 11684.00 frames. ], tot_loss[loss=0.2816, simple_loss=0.39, pruned_loss=0.08659, over 4709047.75 frames. ], batch size: 333, lr: 6.15e-03, grad_scale: 32.0 2026-09-24 02:28:38,993 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=108546.66666666667, ans=0.0 2026-09-24 02:28:40,776 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-34.pt 2026-09-24 02:28:47,828 INFO [train.py:1192] (0/2) Epoch 35, batch 0, loss[loss=0.231, simple_loss=0.3462, pruned_loss=0.05795, over 24573.00 frames. ], tot_loss[loss=0.231, simple_loss=0.3462, pruned_loss=0.05795, over 24573.00 frames. ], batch size: 137, lr: 6.06e-03, grad_scale: 32.0 2026-09-24 02:28:47,828 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 02:28:57,570 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.0822, 3.7598, 3.5527, 3.0971], device='cuda:0') 2026-09-24 02:28:59,564 INFO [train.py:1224] (0/2) Epoch 35, validation: loss=0.1781, simple_loss=0.2962, pruned_loss=0.02996, over 2564189.00 frames. 2026-09-24 02:28:59,564 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 02:29:15,652 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=108673.33333333333, ans=0.1 2026-09-24 02:29:20,037 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=108706.66666666667, ans=0.2 2026-09-24 02:29:24,973 INFO [train.py:1192] (0/2) Epoch 35, batch 50, loss[loss=0.2424, simple_loss=0.3451, pruned_loss=0.06983, over 24235.00 frames. ], tot_loss[loss=0.2879, simple_loss=0.3969, pruned_loss=0.08946, over 1076453.21 frames. ], batch size: 125, lr: 6.06e-03, grad_scale: 32.0 2026-09-24 02:29:32,002 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.435e+02 3.745e+02 4.015e+02 4.535e+02 7.894e+02, threshold=8.030e+02, percent-clipped=1.0 2026-09-24 02:29:38,311 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=108806.66666666667, ans=0.125 2026-09-24 02:29:49,851 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.07 vs. limit=10.0 2026-09-24 02:29:50,622 INFO [train.py:1192] (0/2) Epoch 35, batch 100, loss[loss=0.2937, simple_loss=0.3914, pruned_loss=0.09803, over 24601.00 frames. ], tot_loss[loss=0.2913, simple_loss=0.4006, pruned_loss=0.09104, over 1905844.16 frames. ], batch size: 154, lr: 6.05e-03, grad_scale: 32.0 2026-09-24 02:30:01,025 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=108973.33333333333, ans=0.0 2026-09-24 02:30:02,246 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.08 vs. limit=15.0 2026-09-24 02:30:02,960 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=108973.33333333333, ans=0.0 2026-09-24 02:30:05,652 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:30:10,985 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=109040.0, ans=0.125 2026-09-24 02:30:16,606 INFO [train.py:1192] (0/2) Epoch 35, batch 150, loss[loss=0.2508, simple_loss=0.3447, pruned_loss=0.07843, over 24239.00 frames. ], tot_loss[loss=0.2861, simple_loss=0.3958, pruned_loss=0.08823, over 2558783.61 frames. ], batch size: 125, lr: 6.05e-03, grad_scale: 32.0 2026-09-24 02:30:22,533 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=109106.66666666667, ans=0.125 2026-09-24 02:30:22,741 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=5.26 vs. limit=15.0 2026-09-24 02:30:23,450 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.624e+02 3.181e+02 3.537e+02 3.997e+02 6.502e+02, threshold=7.075e+02, percent-clipped=0.0 2026-09-24 02:30:26,496 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=109140.0, ans=0.125 2026-09-24 02:30:28,372 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=109140.0, ans=0.5 2026-09-24 02:30:31,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=109140.0, ans=0.1 2026-09-24 02:30:38,959 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=109206.66666666667, ans=0.125 2026-09-24 02:30:42,069 INFO [train.py:1192] (0/2) Epoch 35, batch 200, loss[loss=0.3562, simple_loss=0.4448, pruned_loss=0.1338, over 21169.00 frames. ], tot_loss[loss=0.2847, simple_loss=0.3946, pruned_loss=0.0874, over 3056763.77 frames. ], batch size: 333, lr: 6.05e-03, grad_scale: 32.0 2026-09-24 02:30:43,259 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=109240.0, ans=0.125 2026-09-24 02:30:47,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=109273.33333333333, ans=0.0 2026-09-24 02:30:47,561 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=109273.33333333333, ans=0.125 2026-09-24 02:30:48,159 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=109273.33333333333, ans=0.125 2026-09-24 02:30:54,068 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=109306.66666666667, ans=0.125 2026-09-24 02:30:55,746 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=12.32 vs. limit=15.0 2026-09-24 02:30:56,014 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=109306.66666666667, ans=0.125 2026-09-24 02:31:07,728 INFO [train.py:1192] (0/2) Epoch 35, batch 250, loss[loss=0.302, simple_loss=0.4242, pruned_loss=0.08987, over 24295.00 frames. ], tot_loss[loss=0.2837, simple_loss=0.3935, pruned_loss=0.08692, over 3444884.74 frames. ], batch size: 234, lr: 6.04e-03, grad_scale: 32.0 2026-09-24 02:31:09,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=109406.66666666667, ans=0.0 2026-09-24 02:31:12,441 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.84 vs. limit=22.5 2026-09-24 02:31:15,039 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.659e+02 3.448e+02 3.807e+02 4.569e+02 7.598e+02, threshold=7.614e+02, percent-clipped=2.0 2026-09-24 02:31:15,976 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=109440.0, ans=0.025 2026-09-24 02:31:16,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=109440.0, ans=0.0 2026-09-24 02:31:24,052 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=109506.66666666667, ans=0.125 2026-09-24 02:31:29,205 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=109540.0, ans=0.0 2026-09-24 02:31:29,263 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=109540.0, ans=15.0 2026-09-24 02:31:30,489 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=109540.0, ans=0.05 2026-09-24 02:31:31,616 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.whiten.whitening_limit, batch_count=109540.0, ans=15.0 2026-09-24 02:31:33,590 INFO [train.py:1192] (0/2) Epoch 35, batch 300, loss[loss=0.2957, simple_loss=0.4157, pruned_loss=0.08789, over 24581.00 frames. ], tot_loss[loss=0.2833, simple_loss=0.3927, pruned_loss=0.08695, over 3750420.20 frames. ], batch size: 204, lr: 6.04e-03, grad_scale: 32.0 2026-09-24 02:31:36,676 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=109573.33333333333, ans=0.125 2026-09-24 02:31:45,466 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=109640.0, ans=0.1 2026-09-24 02:31:47,062 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=109640.0, ans=0.125 2026-09-24 02:31:49,021 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=3.82 vs. limit=12.0 2026-09-24 02:31:52,764 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=109673.33333333333, ans=0.0 2026-09-24 02:31:53,263 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=109673.33333333333, ans=0.2 2026-09-24 02:31:56,176 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.63 vs. limit=12.0 2026-09-24 02:31:59,373 INFO [train.py:1192] (0/2) Epoch 35, batch 350, loss[loss=0.2447, simple_loss=0.3514, pruned_loss=0.06902, over 24562.00 frames. ], tot_loss[loss=0.2831, simple_loss=0.393, pruned_loss=0.0866, over 3993634.66 frames. ], batch size: 137, lr: 6.03e-03, grad_scale: 32.0 2026-09-24 02:32:03,151 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=109740.0, ans=0.2 2026-09-24 02:32:04,148 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=109773.33333333333, ans=0.125 2026-09-24 02:32:06,046 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=109773.33333333333, ans=0.1 2026-09-24 02:32:06,442 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.601e+02 3.296e+02 3.647e+02 4.243e+02 6.887e+02, threshold=7.294e+02, percent-clipped=0.0 2026-09-24 02:32:12,346 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=109806.66666666667, ans=0.0 2026-09-24 02:32:13,602 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.89 vs. limit=6.0 2026-09-24 02:32:20,775 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=109873.33333333333, ans=0.125 2026-09-24 02:32:21,316 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=109873.33333333333, ans=0.125 2026-09-24 02:32:25,362 INFO [train.py:1192] (0/2) Epoch 35, batch 400, loss[loss=0.2883, simple_loss=0.3977, pruned_loss=0.08945, over 24573.00 frames. ], tot_loss[loss=0.2815, simple_loss=0.3915, pruned_loss=0.08577, over 4176750.55 frames. ], batch size: 170, lr: 6.03e-03, grad_scale: 32.0 2026-09-24 02:32:40,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=109973.33333333333, ans=0.0 2026-09-24 02:32:49,778 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.99 vs. limit=15.0 2026-09-24 02:32:50,537 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=12.12 vs. limit=15.0 2026-09-24 02:32:51,505 INFO [train.py:1192] (0/2) Epoch 35, batch 450, loss[loss=0.2811, simple_loss=0.3958, pruned_loss=0.08323, over 24619.00 frames. ], tot_loss[loss=0.2827, simple_loss=0.3924, pruned_loss=0.08651, over 4307124.24 frames. ], batch size: 175, lr: 6.02e-03, grad_scale: 32.0 2026-09-24 02:32:55,578 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=110073.33333333333, ans=0.125 2026-09-24 02:32:58,170 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=110106.66666666667, ans=0.2 2026-09-24 02:32:58,610 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.579e+02 3.247e+02 3.647e+02 4.122e+02 5.699e+02, threshold=7.294e+02, percent-clipped=0.0 2026-09-24 02:33:03,261 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=110140.0, ans=0.0 2026-09-24 02:33:11,378 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=110173.33333333333, ans=0.125 2026-09-24 02:33:12,379 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=110206.66666666667, ans=0.0 2026-09-24 02:33:17,574 INFO [train.py:1192] (0/2) Epoch 35, batch 500, loss[loss=0.3018, simple_loss=0.4214, pruned_loss=0.09108, over 24492.00 frames. ], tot_loss[loss=0.2814, simple_loss=0.3909, pruned_loss=0.0859, over 4425778.07 frames. ], batch size: 218, lr: 6.02e-03, grad_scale: 32.0 2026-09-24 02:33:23,728 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=110273.33333333333, ans=0.125 2026-09-24 02:33:26,623 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=110273.33333333333, ans=0.125 2026-09-24 02:33:34,546 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=110340.0, ans=0.0 2026-09-24 02:33:35,527 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=110340.0, ans=0.2 2026-09-24 02:33:36,579 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=110340.0, ans=0.1 2026-09-24 02:33:42,137 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=110373.33333333333, ans=22.5 2026-09-24 02:33:43,413 INFO [train.py:1192] (0/2) Epoch 35, batch 550, loss[loss=0.2835, simple_loss=0.4031, pruned_loss=0.08195, over 24263.00 frames. ], tot_loss[loss=0.281, simple_loss=0.391, pruned_loss=0.08551, over 4515799.66 frames. ], batch size: 257, lr: 6.02e-03, grad_scale: 32.0 2026-09-24 02:33:50,688 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.628e+02 3.226e+02 3.467e+02 4.015e+02 5.376e+02, threshold=6.935e+02, percent-clipped=0.0 2026-09-24 02:33:54,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=110473.33333333333, ans=0.2 2026-09-24 02:33:59,680 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=110506.66666666667, ans=0.0 2026-09-24 02:34:08,867 INFO [train.py:1192] (0/2) Epoch 35, batch 600, loss[loss=0.3461, simple_loss=0.4516, pruned_loss=0.1203, over 24340.00 frames. ], tot_loss[loss=0.2814, simple_loss=0.3915, pruned_loss=0.08568, over 4583524.85 frames. ], batch size: 234, lr: 6.01e-03, grad_scale: 32.0 2026-09-24 02:34:18,341 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=110606.66666666667, ans=0.0 2026-09-24 02:34:25,753 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=110673.33333333333, ans=0.1 2026-09-24 02:34:25,799 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=110673.33333333333, ans=0.125 2026-09-24 02:34:35,031 INFO [train.py:1192] (0/2) Epoch 35, batch 650, loss[loss=0.2771, simple_loss=0.3878, pruned_loss=0.08318, over 24563.00 frames. ], tot_loss[loss=0.2806, simple_loss=0.3906, pruned_loss=0.08526, over 4649280.94 frames. ], batch size: 162, lr: 6.01e-03, grad_scale: 32.0 2026-09-24 02:34:39,494 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=110740.0, ans=0.0 2026-09-24 02:34:42,654 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.489e+02 3.190e+02 3.547e+02 4.027e+02 6.396e+02, threshold=7.094e+02, percent-clipped=0.0 2026-09-24 02:34:44,230 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=110773.33333333333, ans=0.0 2026-09-24 02:34:52,672 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=110840.0, ans=0.125 2026-09-24 02:34:55,076 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=110840.0, ans=0.0 2026-09-24 02:34:58,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=110873.33333333333, ans=0.1 2026-09-24 02:35:01,166 INFO [train.py:1192] (0/2) Epoch 35, batch 700, loss[loss=0.2624, simple_loss=0.3695, pruned_loss=0.07763, over 24564.00 frames. ], tot_loss[loss=0.2818, simple_loss=0.392, pruned_loss=0.08583, over 4683359.24 frames. ], batch size: 154, lr: 6.00e-03, grad_scale: 32.0 2026-09-24 02:35:08,157 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.70 vs. limit=22.5 2026-09-24 02:35:09,609 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=110940.0, ans=0.125 2026-09-24 02:35:09,609 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=110940.0, ans=0.125 2026-09-24 02:35:11,758 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=110973.33333333333, ans=0.125 2026-09-24 02:35:23,039 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=111040.0, ans=0.0 2026-09-24 02:35:25,993 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=111040.0, ans=0.2 2026-09-24 02:35:26,926 INFO [train.py:1192] (0/2) Epoch 35, batch 750, loss[loss=0.286, simple_loss=0.4027, pruned_loss=0.08462, over 24552.00 frames. ], tot_loss[loss=0.2805, simple_loss=0.3905, pruned_loss=0.08526, over 4710457.93 frames. ], batch size: 170, lr: 6.00e-03, grad_scale: 32.0 2026-09-24 02:35:33,397 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=111106.66666666667, ans=0.0 2026-09-24 02:35:34,285 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.475e+02 3.230e+02 3.760e+02 4.436e+02 7.049e+02, threshold=7.521e+02, percent-clipped=0.0 2026-09-24 02:35:37,719 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=111140.0, ans=0.125 2026-09-24 02:35:40,637 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:35:52,727 INFO [train.py:1192] (0/2) Epoch 35, batch 800, loss[loss=0.2716, simple_loss=0.3707, pruned_loss=0.08628, over 24533.00 frames. ], tot_loss[loss=0.2799, simple_loss=0.39, pruned_loss=0.08491, over 4735064.50 frames. ], batch size: 137, lr: 5.99e-03, grad_scale: 32.0 2026-09-24 02:36:15,005 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=111373.33333333333, ans=0.125 2026-09-24 02:36:18,018 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=111373.33333333333, ans=0.025 2026-09-24 02:36:19,260 INFO [train.py:1192] (0/2) Epoch 35, batch 850, loss[loss=0.3075, simple_loss=0.4152, pruned_loss=0.09988, over 24613.00 frames. ], tot_loss[loss=0.2804, simple_loss=0.3904, pruned_loss=0.08518, over 4757680.46 frames. ], batch size: 198, lr: 5.99e-03, grad_scale: 32.0 2026-09-24 02:36:26,218 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.635e+02 3.255e+02 3.671e+02 4.302e+02 6.594e+02, threshold=7.341e+02, percent-clipped=0.0 2026-09-24 02:36:26,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=111440.0, ans=0.025 2026-09-24 02:36:44,657 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=111573.33333333333, ans=0.125 2026-09-24 02:36:44,998 INFO [train.py:1192] (0/2) Epoch 35, batch 900, loss[loss=0.2293, simple_loss=0.3453, pruned_loss=0.05667, over 24562.00 frames. ], tot_loss[loss=0.2802, simple_loss=0.3905, pruned_loss=0.08494, over 4772022.76 frames. ], batch size: 137, lr: 5.99e-03, grad_scale: 32.0 2026-09-24 02:36:47,157 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.21 vs. limit=15.0 2026-09-24 02:36:56,244 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=111640.0, ans=0.125 2026-09-24 02:37:06,038 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=111706.66666666667, ans=0.125 2026-09-24 02:37:10,270 INFO [train.py:1192] (0/2) Epoch 35, batch 950, loss[loss=0.4143, simple_loss=0.4596, pruned_loss=0.1845, over 11479.00 frames. ], tot_loss[loss=0.2814, simple_loss=0.3899, pruned_loss=0.08642, over 4711474.38 frames. ], batch size: 333, lr: 5.98e-03, grad_scale: 32.0 2026-09-24 02:37:12,943 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=111740.0, ans=0.0 2026-09-24 02:37:14,769 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-35.pt 2026-09-24 02:37:22,234 INFO [train.py:1192] (0/2) Epoch 36, batch 0, loss[loss=0.2544, simple_loss=0.3667, pruned_loss=0.07105, over 24569.00 frames. ], tot_loss[loss=0.2544, simple_loss=0.3667, pruned_loss=0.07105, over 24569.00 frames. ], batch size: 137, lr: 5.90e-03, grad_scale: 32.0 2026-09-24 02:37:22,234 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 02:37:25,108 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([3.8932, 2.9773, 2.6384, 2.2307], device='cuda:0') 2026-09-24 02:37:27,231 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.7262, 4.2568, 4.1941, 4.5534], device='cuda:0') 2026-09-24 02:37:34,101 INFO [train.py:1224] (0/2) Epoch 36, validation: loss=0.1793, simple_loss=0.2976, pruned_loss=0.03051, over 2564189.00 frames. 2026-09-24 02:37:34,101 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 02:37:37,073 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.624e+02 3.274e+02 3.763e+02 4.231e+02 6.255e+02, threshold=7.525e+02, percent-clipped=0.0 2026-09-24 02:37:51,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=111866.66666666667, ans=0.125 2026-09-24 02:38:00,198 INFO [train.py:1192] (0/2) Epoch 36, batch 50, loss[loss=0.2461, simple_loss=0.3482, pruned_loss=0.07206, over 24247.00 frames. ], tot_loss[loss=0.2882, simple_loss=0.3974, pruned_loss=0.08947, over 1075641.70 frames. ], batch size: 125, lr: 5.89e-03, grad_scale: 32.0 2026-09-24 02:38:00,537 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=4.96 vs. limit=15.0 2026-09-24 02:38:00,819 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=111933.33333333333, ans=0.0 2026-09-24 02:38:25,794 INFO [train.py:1192] (0/2) Epoch 36, batch 100, loss[loss=0.284, simple_loss=0.3845, pruned_loss=0.09172, over 24612.00 frames. ], tot_loss[loss=0.2907, simple_loss=0.4006, pruned_loss=0.09043, over 1905072.85 frames. ], batch size: 154, lr: 5.89e-03, grad_scale: 32.0 2026-09-24 02:38:28,994 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.701e+02 3.326e+02 3.642e+02 4.179e+02 6.797e+02, threshold=7.283e+02, percent-clipped=0.0 2026-09-24 02:38:30,707 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.43 vs. limit=12.0 2026-09-24 02:38:34,559 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=112133.33333333333, ans=0.025 2026-09-24 02:38:35,622 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=112166.66666666667, ans=0.0 2026-09-24 02:38:49,371 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=112233.33333333333, ans=0.125 2026-09-24 02:38:51,329 INFO [train.py:1192] (0/2) Epoch 36, batch 150, loss[loss=0.236, simple_loss=0.3413, pruned_loss=0.06536, over 24282.00 frames. ], tot_loss[loss=0.2836, simple_loss=0.3944, pruned_loss=0.08642, over 2559142.35 frames. ], batch size: 125, lr: 5.88e-03, grad_scale: 32.0 2026-09-24 02:38:52,682 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:38:57,729 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=112300.0, ans=0.0 2026-09-24 02:38:58,251 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=112300.0, ans=0.95 2026-09-24 02:38:58,509 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=5.94 vs. limit=15.0 2026-09-24 02:39:17,600 INFO [train.py:1192] (0/2) Epoch 36, batch 200, loss[loss=0.3458, simple_loss=0.4352, pruned_loss=0.1282, over 21153.00 frames. ], tot_loss[loss=0.2811, simple_loss=0.3922, pruned_loss=0.08503, over 3056446.54 frames. ], batch size: 333, lr: 5.88e-03, grad_scale: 32.0 2026-09-24 02:39:19,743 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.74 vs. limit=22.5 2026-09-24 02:39:20,634 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.387e+02 3.387e+02 3.908e+02 4.634e+02 6.934e+02, threshold=7.816e+02, percent-clipped=0.0 2026-09-24 02:39:26,871 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=112466.66666666667, ans=0.125 2026-09-24 02:39:29,516 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=112500.0, ans=0.125 2026-09-24 02:39:43,149 INFO [train.py:1192] (0/2) Epoch 36, batch 250, loss[loss=0.3072, simple_loss=0.4251, pruned_loss=0.09464, over 24301.00 frames. ], tot_loss[loss=0.2806, simple_loss=0.3915, pruned_loss=0.08486, over 3446043.70 frames. ], batch size: 234, lr: 5.87e-03, grad_scale: 32.0 2026-09-24 02:39:44,726 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=112600.0, ans=0.0 2026-09-24 02:39:48,396 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=112633.33333333333, ans=0.125 2026-09-24 02:39:54,343 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.21 vs. limit=6.0 2026-09-24 02:39:55,204 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=112666.66666666667, ans=0.1 2026-09-24 02:39:59,328 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=112700.0, ans=0.2 2026-09-24 02:40:08,404 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:40:08,811 INFO [train.py:1192] (0/2) Epoch 36, batch 300, loss[loss=0.3083, simple_loss=0.4205, pruned_loss=0.098, over 24519.00 frames. ], tot_loss[loss=0.2801, simple_loss=0.3904, pruned_loss=0.08489, over 3751417.78 frames. ], batch size: 204, lr: 5.87e-03, grad_scale: 32.0 2026-09-24 02:40:11,404 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=112766.66666666667, ans=0.125 2026-09-24 02:40:11,840 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.729e+02 3.343e+02 3.734e+02 4.325e+02 7.519e+02, threshold=7.468e+02, percent-clipped=0.0 2026-09-24 02:40:20,460 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=112833.33333333333, ans=0.1 2026-09-24 02:40:25,228 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=112866.66666666667, ans=0.1 2026-09-24 02:40:34,542 INFO [train.py:1192] (0/2) Epoch 36, batch 350, loss[loss=0.2383, simple_loss=0.35, pruned_loss=0.06325, over 24602.00 frames. ], tot_loss[loss=0.2819, simple_loss=0.3923, pruned_loss=0.08577, over 3991090.01 frames. ], batch size: 137, lr: 5.87e-03, grad_scale: 32.0 2026-09-24 02:40:35,592 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=112933.33333333333, ans=0.2 2026-09-24 02:40:38,321 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.92 vs. limit=15.0 2026-09-24 02:40:54,379 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=113066.66666666667, ans=0.025 2026-09-24 02:41:00,255 INFO [train.py:1192] (0/2) Epoch 36, batch 400, loss[loss=0.2779, simple_loss=0.3955, pruned_loss=0.08011, over 24561.00 frames. ], tot_loss[loss=0.2806, simple_loss=0.3912, pruned_loss=0.08497, over 4178545.25 frames. ], batch size: 170, lr: 5.86e-03, grad_scale: 32.0 2026-09-24 02:41:03,093 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.772e+02 3.358e+02 3.839e+02 4.514e+02 6.504e+02, threshold=7.677e+02, percent-clipped=0.0 2026-09-24 02:41:08,177 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=113133.33333333333, ans=0.95 2026-09-24 02:41:08,525 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=113133.33333333333, ans=0.125 2026-09-24 02:41:12,193 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.39 vs. limit=22.5 2026-09-24 02:41:14,664 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=113200.0, ans=0.0 2026-09-24 02:41:16,067 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=113200.0, ans=0.125 2026-09-24 02:41:25,733 INFO [train.py:1192] (0/2) Epoch 36, batch 450, loss[loss=0.2801, simple_loss=0.3961, pruned_loss=0.08207, over 24616.00 frames. ], tot_loss[loss=0.2811, simple_loss=0.3918, pruned_loss=0.08521, over 4312534.18 frames. ], batch size: 175, lr: 5.86e-03, grad_scale: 32.0 2026-09-24 02:41:33,106 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:41:34,064 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=113300.0, ans=0.0 2026-09-24 02:41:40,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=113366.66666666667, ans=0.2 2026-09-24 02:41:51,180 INFO [train.py:1192] (0/2) Epoch 36, batch 500, loss[loss=0.2858, simple_loss=0.4064, pruned_loss=0.08255, over 24496.00 frames. ], tot_loss[loss=0.2796, simple_loss=0.39, pruned_loss=0.08457, over 4429341.80 frames. ], batch size: 218, lr: 5.85e-03, grad_scale: 32.0 2026-09-24 02:41:51,297 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=113433.33333333333, ans=0.1 2026-09-24 02:41:53,044 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:41:54,349 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.330e+02 3.193e+02 3.643e+02 4.133e+02 7.090e+02, threshold=7.286e+02, percent-clipped=0.0 2026-09-24 02:41:54,482 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=113433.33333333333, ans=0.2 2026-09-24 02:42:04,022 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.92 vs. limit=22.5 2026-09-24 02:42:07,401 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=113533.33333333333, ans=0.1 2026-09-24 02:42:11,636 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=113566.66666666667, ans=0.04949747468305833 2026-09-24 02:42:12,626 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:42:17,050 INFO [train.py:1192] (0/2) Epoch 36, batch 550, loss[loss=0.2798, simple_loss=0.4038, pruned_loss=0.07786, over 24243.00 frames. ], tot_loss[loss=0.2806, simple_loss=0.3911, pruned_loss=0.08509, over 4518912.12 frames. ], batch size: 257, lr: 5.85e-03, grad_scale: 32.0 2026-09-24 02:42:21,041 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=113600.0, ans=0.2 2026-09-24 02:42:36,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten.whitening_limit, batch_count=113700.0, ans=15.0 2026-09-24 02:42:37,236 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:42:41,600 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=113733.33333333333, ans=0.1 2026-09-24 02:42:43,195 INFO [train.py:1192] (0/2) Epoch 36, batch 600, loss[loss=0.3344, simple_loss=0.4483, pruned_loss=0.1103, over 24335.00 frames. ], tot_loss[loss=0.2809, simple_loss=0.3916, pruned_loss=0.08511, over 4585946.73 frames. ], batch size: 234, lr: 5.85e-03, grad_scale: 32.0 2026-09-24 02:42:46,142 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.574e+02 3.225e+02 3.585e+02 4.048e+02 6.274e+02, threshold=7.170e+02, percent-clipped=0.0 2026-09-24 02:43:01,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1.whitening_limit, batch_count=113866.66666666667, ans=10.0 2026-09-24 02:43:08,521 INFO [train.py:1192] (0/2) Epoch 36, batch 650, loss[loss=0.2659, simple_loss=0.3806, pruned_loss=0.07564, over 24556.00 frames. ], tot_loss[loss=0.2797, simple_loss=0.3906, pruned_loss=0.08439, over 4651315.36 frames. ], batch size: 162, lr: 5.84e-03, grad_scale: 32.0 2026-09-24 02:43:09,938 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=113933.33333333333, ans=0.125 2026-09-24 02:43:33,760 INFO [train.py:1192] (0/2) Epoch 36, batch 700, loss[loss=0.2714, simple_loss=0.3766, pruned_loss=0.08311, over 24598.00 frames. ], tot_loss[loss=0.2799, simple_loss=0.391, pruned_loss=0.08445, over 4684233.89 frames. ], batch size: 154, lr: 5.84e-03, grad_scale: 32.0 2026-09-24 02:43:34,987 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.59 vs. limit=22.5 2026-09-24 02:43:36,916 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.532e+02 3.172e+02 3.783e+02 4.286e+02 7.223e+02, threshold=7.566e+02, percent-clipped=1.0 2026-09-24 02:43:45,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=114166.66666666667, ans=0.0 2026-09-24 02:43:56,810 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=114233.33333333333, ans=0.5 2026-09-24 02:44:00,127 INFO [train.py:1192] (0/2) Epoch 36, batch 750, loss[loss=0.2579, simple_loss=0.3834, pruned_loss=0.06618, over 24585.00 frames. ], tot_loss[loss=0.2793, simple_loss=0.39, pruned_loss=0.08427, over 4711180.20 frames. ], batch size: 170, lr: 5.83e-03, grad_scale: 32.0 2026-09-24 02:44:00,257 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=114266.66666666667, ans=0.125 2026-09-24 02:44:11,217 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=114333.33333333333, ans=0.2 2026-09-24 02:44:20,244 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=114366.66666666667, ans=0.125 2026-09-24 02:44:23,351 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=114400.0, ans=0.5 2026-09-24 02:44:26,931 INFO [train.py:1192] (0/2) Epoch 36, batch 800, loss[loss=0.2419, simple_loss=0.3503, pruned_loss=0.06678, over 24542.00 frames. ], tot_loss[loss=0.2791, simple_loss=0.3897, pruned_loss=0.08425, over 4740162.03 frames. ], batch size: 137, lr: 5.83e-03, grad_scale: 32.0 2026-09-24 02:44:30,530 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.694e+02 3.229e+02 3.756e+02 4.190e+02 6.028e+02, threshold=7.512e+02, percent-clipped=0.0 2026-09-24 02:44:35,273 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=114466.66666666667, ans=0.125 2026-09-24 02:44:41,899 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=114500.0, ans=0.125 2026-09-24 02:44:49,547 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=114566.66666666667, ans=0.1 2026-09-24 02:44:53,278 INFO [train.py:1192] (0/2) Epoch 36, batch 850, loss[loss=0.3037, simple_loss=0.4215, pruned_loss=0.09296, over 24553.00 frames. ], tot_loss[loss=0.2791, simple_loss=0.3897, pruned_loss=0.08427, over 4763285.61 frames. ], batch size: 204, lr: 5.83e-03, grad_scale: 32.0 2026-09-24 02:44:54,015 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.56 vs. limit=15.0 2026-09-24 02:44:57,160 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=114600.0, ans=0.0 2026-09-24 02:45:00,811 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=114633.33333333333, ans=0.125 2026-09-24 02:45:01,470 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.02 vs. limit=15.0 2026-09-24 02:45:04,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=114666.66666666667, ans=0.125 2026-09-24 02:45:09,105 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.99 vs. limit=10.0 2026-09-24 02:45:19,370 INFO [train.py:1192] (0/2) Epoch 36, batch 900, loss[loss=0.2246, simple_loss=0.3415, pruned_loss=0.05383, over 24539.00 frames. ], tot_loss[loss=0.2793, simple_loss=0.3898, pruned_loss=0.08439, over 4776549.12 frames. ], batch size: 137, lr: 5.82e-03, grad_scale: 32.0 2026-09-24 02:45:22,318 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.378e+02 3.202e+02 3.609e+02 4.015e+02 5.684e+02, threshold=7.217e+02, percent-clipped=0.0 2026-09-24 02:45:41,791 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=114900.0, ans=0.1 2026-09-24 02:45:44,362 INFO [train.py:1192] (0/2) Epoch 36, batch 950, loss[loss=0.4045, simple_loss=0.4506, pruned_loss=0.1792, over 11242.00 frames. ], tot_loss[loss=0.2803, simple_loss=0.3892, pruned_loss=0.08569, over 4709328.47 frames. ], batch size: 333, lr: 5.82e-03, grad_scale: 32.0 2026-09-24 02:45:48,672 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-36.pt 2026-09-24 02:45:54,779 INFO [train.py:1192] (0/2) Epoch 37, batch 0, loss[loss=0.2186, simple_loss=0.3373, pruned_loss=0.04993, over 24569.00 frames. ], tot_loss[loss=0.2186, simple_loss=0.3373, pruned_loss=0.04993, over 24569.00 frames. ], batch size: 137, lr: 5.74e-03, grad_scale: 32.0 2026-09-24 02:45:54,779 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 02:46:06,791 INFO [train.py:1224] (0/2) Epoch 37, validation: loss=0.1782, simple_loss=0.2965, pruned_loss=0.0299, over 2564189.00 frames. 2026-09-24 02:46:06,791 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 02:46:07,527 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.54 vs. limit=15.0 2026-09-24 02:46:09,240 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=114960.0, ans=0.0 2026-09-24 02:46:16,346 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=115026.66666666667, ans=0.0 2026-09-24 02:46:21,346 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=115060.0, ans=0.125 2026-09-24 02:46:22,923 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=115060.0, ans=0.07 2026-09-24 02:46:26,014 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=115093.33333333333, ans=0.1 2026-09-24 02:46:31,319 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.478e+02 3.261e+02 3.711e+02 4.242e+02 6.808e+02, threshold=7.422e+02, percent-clipped=0.0 2026-09-24 02:46:32,434 INFO [train.py:1192] (0/2) Epoch 37, batch 50, loss[loss=0.249, simple_loss=0.3512, pruned_loss=0.07338, over 24252.00 frames. ], tot_loss[loss=0.2846, simple_loss=0.3941, pruned_loss=0.08757, over 1077120.88 frames. ], batch size: 125, lr: 5.73e-03, grad_scale: 32.0 2026-09-24 02:46:38,399 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=115160.0, ans=0.0 2026-09-24 02:46:47,943 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=115226.66666666667, ans=0.0 2026-09-24 02:46:53,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=115260.0, ans=0.125 2026-09-24 02:46:54,486 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:46:58,407 INFO [train.py:1192] (0/2) Epoch 37, batch 100, loss[loss=0.2635, simple_loss=0.3753, pruned_loss=0.07585, over 24605.00 frames. ], tot_loss[loss=0.2897, simple_loss=0.3998, pruned_loss=0.08979, over 1905479.04 frames. ], batch size: 154, lr: 5.73e-03, grad_scale: 64.0 2026-09-24 02:47:12,855 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=115360.0, ans=0.0 2026-09-24 02:47:21,508 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.34 vs. limit=15.0 2026-09-24 02:47:23,774 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.684e+02 3.309e+02 3.683e+02 4.350e+02 6.360e+02, threshold=7.366e+02, percent-clipped=0.0 2026-09-24 02:47:24,713 INFO [train.py:1192] (0/2) Epoch 37, batch 150, loss[loss=0.2183, simple_loss=0.3243, pruned_loss=0.05616, over 24247.00 frames. ], tot_loss[loss=0.2846, simple_loss=0.3949, pruned_loss=0.08715, over 2558615.76 frames. ], batch size: 125, lr: 5.72e-03, grad_scale: 64.0 2026-09-24 02:47:50,714 INFO [train.py:1192] (0/2) Epoch 37, batch 200, loss[loss=0.3217, simple_loss=0.4195, pruned_loss=0.112, over 21196.00 frames. ], tot_loss[loss=0.2824, simple_loss=0.393, pruned_loss=0.08586, over 3054751.91 frames. ], batch size: 333, lr: 5.72e-03, grad_scale: 32.0 2026-09-24 02:47:56,614 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=115660.0, ans=15.0 2026-09-24 02:48:16,229 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.586e+02 3.511e+02 4.062e+02 4.877e+02 7.585e+02, threshold=8.125e+02, percent-clipped=2.0 2026-09-24 02:48:16,746 INFO [train.py:1192] (0/2) Epoch 37, batch 250, loss[loss=0.3203, simple_loss=0.4328, pruned_loss=0.1039, over 24313.00 frames. ], tot_loss[loss=0.2814, simple_loss=0.392, pruned_loss=0.08537, over 3444232.71 frames. ], batch size: 234, lr: 5.72e-03, grad_scale: 32.0 2026-09-24 02:48:22,725 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=115826.66666666667, ans=0.1 2026-09-24 02:48:30,551 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.18 vs. limit=22.5 2026-09-24 02:48:42,695 INFO [train.py:1192] (0/2) Epoch 37, batch 300, loss[loss=0.2851, simple_loss=0.4055, pruned_loss=0.08231, over 24539.00 frames. ], tot_loss[loss=0.2801, simple_loss=0.3906, pruned_loss=0.08478, over 3749651.47 frames. ], batch size: 204, lr: 5.71e-03, grad_scale: 32.0 2026-09-24 02:48:45,574 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.22 vs. limit=15.0 2026-09-24 02:48:47,397 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.44 vs. limit=12.0 2026-09-24 02:48:58,868 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.06 vs. limit=10.0 2026-09-24 02:49:00,807 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=8.04 vs. limit=12.0 2026-09-24 02:49:01,728 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=116060.0, ans=0.1 2026-09-24 02:49:02,210 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=116060.0, ans=0.0 2026-09-24 02:49:03,995 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=116093.33333333333, ans=0.2 2026-09-24 02:49:08,120 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.612e+02 3.098e+02 3.455e+02 3.844e+02 5.865e+02, threshold=6.911e+02, percent-clipped=0.0 2026-09-24 02:49:08,679 INFO [train.py:1192] (0/2) Epoch 37, batch 350, loss[loss=0.227, simple_loss=0.3371, pruned_loss=0.05847, over 24595.00 frames. ], tot_loss[loss=0.2803, simple_loss=0.3914, pruned_loss=0.08457, over 3993008.07 frames. ], batch size: 137, lr: 5.71e-03, grad_scale: 32.0 2026-09-24 02:49:09,489 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.39 vs. limit=6.0 2026-09-24 02:49:21,724 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=116193.33333333333, ans=0.125 2026-09-24 02:49:22,781 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=116193.33333333333, ans=0.2 2026-09-24 02:49:29,570 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=7.93 vs. limit=15.0 2026-09-24 02:49:32,143 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=116260.0, ans=0.0 2026-09-24 02:49:34,869 INFO [train.py:1192] (0/2) Epoch 37, batch 400, loss[loss=0.2833, simple_loss=0.3935, pruned_loss=0.08654, over 24580.00 frames. ], tot_loss[loss=0.2795, simple_loss=0.3906, pruned_loss=0.08414, over 4177125.74 frames. ], batch size: 170, lr: 5.70e-03, grad_scale: 32.0 2026-09-24 02:49:34,983 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=116293.33333333333, ans=0.025 2026-09-24 02:49:40,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=116326.66666666667, ans=0.125 2026-09-24 02:49:40,066 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=116326.66666666667, ans=0.125 2026-09-24 02:49:54,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=116393.33333333333, ans=0.125 2026-09-24 02:50:00,923 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.704e+02 3.336e+02 3.681e+02 4.206e+02 6.815e+02, threshold=7.363e+02, percent-clipped=0.0 2026-09-24 02:50:01,360 INFO [train.py:1192] (0/2) Epoch 37, batch 450, loss[loss=0.2796, simple_loss=0.3955, pruned_loss=0.08183, over 24640.00 frames. ], tot_loss[loss=0.2799, simple_loss=0.391, pruned_loss=0.08438, over 4311348.95 frames. ], batch size: 175, lr: 5.70e-03, grad_scale: 32.0 2026-09-24 02:50:04,019 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=116460.0, ans=0.125 2026-09-24 02:50:04,916 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=116460.0, ans=0.0 2026-09-24 02:50:14,705 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=116526.66666666667, ans=0.0 2026-09-24 02:50:15,287 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.34 vs. limit=12.0 2026-09-24 02:50:21,757 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=116560.0, ans=0.125 2026-09-24 02:50:27,701 INFO [train.py:1192] (0/2) Epoch 37, batch 500, loss[loss=0.3263, simple_loss=0.4398, pruned_loss=0.1064, over 24535.00 frames. ], tot_loss[loss=0.2785, simple_loss=0.3893, pruned_loss=0.08381, over 4428991.81 frames. ], batch size: 218, lr: 5.70e-03, grad_scale: 32.0 2026-09-24 02:50:29,830 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=116626.66666666667, ans=0.125 2026-09-24 02:50:33,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=116660.0, ans=0.0 2026-09-24 02:50:38,388 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.94 vs. limit=15.0 2026-09-24 02:50:43,574 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=116726.66666666667, ans=0.0 2026-09-24 02:50:46,565 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=116726.66666666667, ans=0.125 2026-09-24 02:50:53,468 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.663e+02 3.097e+02 3.515e+02 3.895e+02 6.020e+02, threshold=7.030e+02, percent-clipped=0.0 2026-09-24 02:50:53,929 INFO [train.py:1192] (0/2) Epoch 37, batch 550, loss[loss=0.3004, simple_loss=0.4176, pruned_loss=0.09161, over 24286.00 frames. ], tot_loss[loss=0.2795, simple_loss=0.3903, pruned_loss=0.08434, over 4518449.00 frames. ], batch size: 257, lr: 5.69e-03, grad_scale: 32.0 2026-09-24 02:50:56,342 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=116793.33333333333, ans=0.1 2026-09-24 02:51:07,184 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.35 vs. limit=15.0 2026-09-24 02:51:08,478 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=116860.0, ans=0.125 2026-09-24 02:51:19,902 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.42 vs. limit=6.0 2026-09-24 02:51:20,203 INFO [train.py:1192] (0/2) Epoch 37, batch 600, loss[loss=0.3407, simple_loss=0.4511, pruned_loss=0.1151, over 24341.00 frames. ], tot_loss[loss=0.2801, simple_loss=0.391, pruned_loss=0.08461, over 4585924.10 frames. ], batch size: 234, lr: 5.69e-03, grad_scale: 32.0 2026-09-24 02:51:29,583 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.20 vs. limit=15.0 2026-09-24 02:51:36,141 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.24 vs. limit=6.0 2026-09-24 02:51:39,578 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=117060.0, ans=0.2 2026-09-24 02:51:45,715 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.564e+02 3.262e+02 3.741e+02 4.492e+02 7.596e+02, threshold=7.483e+02, percent-clipped=1.0 2026-09-24 02:51:45,820 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=117126.66666666667, ans=0.125 2026-09-24 02:51:46,285 INFO [train.py:1192] (0/2) Epoch 37, batch 650, loss[loss=0.2762, simple_loss=0.3885, pruned_loss=0.08197, over 24573.00 frames. ], tot_loss[loss=0.2793, simple_loss=0.3901, pruned_loss=0.08422, over 4651201.36 frames. ], batch size: 162, lr: 5.68e-03, grad_scale: 32.0 2026-09-24 02:51:49,304 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=117126.66666666667, ans=0.125 2026-09-24 02:52:03,282 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=117226.66666666667, ans=0.125 2026-09-24 02:52:12,544 INFO [train.py:1192] (0/2) Epoch 37, batch 700, loss[loss=0.2509, simple_loss=0.3616, pruned_loss=0.07013, over 24579.00 frames. ], tot_loss[loss=0.2793, simple_loss=0.3906, pruned_loss=0.08397, over 4685329.43 frames. ], batch size: 154, lr: 5.68e-03, grad_scale: 32.0 2026-09-24 02:52:15,943 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.32 vs. limit=15.0 2026-09-24 02:52:17,719 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=117326.66666666667, ans=0.125 2026-09-24 02:52:23,571 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=117360.0, ans=0.125 2026-09-24 02:52:24,729 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.10 vs. limit=15.0 2026-09-24 02:52:28,573 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=117393.33333333333, ans=0.0 2026-09-24 02:52:33,869 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=117426.66666666667, ans=0.2 2026-09-24 02:52:37,365 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.586e+02 3.247e+02 3.506e+02 4.061e+02 6.119e+02, threshold=7.013e+02, percent-clipped=0.0 2026-09-24 02:52:37,868 INFO [train.py:1192] (0/2) Epoch 37, batch 750, loss[loss=0.2807, simple_loss=0.3959, pruned_loss=0.08274, over 24560.00 frames. ], tot_loss[loss=0.2782, simple_loss=0.3892, pruned_loss=0.08359, over 4712051.61 frames. ], batch size: 170, lr: 5.68e-03, grad_scale: 32.0 2026-09-24 02:52:41,347 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=117460.0, ans=0.025 2026-09-24 02:52:47,742 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.39 vs. limit=15.0 2026-09-24 02:53:03,672 INFO [train.py:1192] (0/2) Epoch 37, batch 800, loss[loss=0.2435, simple_loss=0.3484, pruned_loss=0.06926, over 24563.00 frames. ], tot_loss[loss=0.2772, simple_loss=0.3884, pruned_loss=0.08295, over 4742209.19 frames. ], batch size: 137, lr: 5.67e-03, grad_scale: 32.0 2026-09-24 02:53:15,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=117693.33333333333, ans=0.125 2026-09-24 02:53:23,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=117726.66666666667, ans=0.125 2026-09-24 02:53:24,602 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.02 vs. limit=22.5 2026-09-24 02:53:26,042 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:53:29,073 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.564e+02 3.186e+02 3.597e+02 4.291e+02 6.249e+02, threshold=7.194e+02, percent-clipped=0.0 2026-09-24 02:53:29,505 INFO [train.py:1192] (0/2) Epoch 37, batch 850, loss[loss=0.2839, simple_loss=0.4073, pruned_loss=0.08024, over 24529.00 frames. ], tot_loss[loss=0.2766, simple_loss=0.388, pruned_loss=0.08257, over 4763353.65 frames. ], batch size: 204, lr: 5.67e-03, grad_scale: 32.0 2026-09-24 02:53:36,936 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:53:38,442 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.10 vs. limit=15.0 2026-09-24 02:53:40,395 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.52 vs. limit=22.5 2026-09-24 02:53:55,224 INFO [train.py:1192] (0/2) Epoch 37, batch 900, loss[loss=0.2502, simple_loss=0.36, pruned_loss=0.07014, over 24562.00 frames. ], tot_loss[loss=0.2775, simple_loss=0.3887, pruned_loss=0.0832, over 4775680.84 frames. ], batch size: 137, lr: 5.67e-03, grad_scale: 32.0 2026-09-24 02:53:58,558 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=7.63 vs. limit=15.0 2026-09-24 02:53:59,394 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.67 vs. limit=12.0 2026-09-24 02:54:07,071 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=8.51 vs. limit=12.0 2026-09-24 02:54:07,412 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=118026.66666666667, ans=0.025 2026-09-24 02:54:12,918 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.51 vs. limit=15.0 2026-09-24 02:54:14,754 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=118060.0, ans=0.2 2026-09-24 02:54:16,772 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.03 vs. limit=15.0 2026-09-24 02:54:19,038 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=118093.33333333333, ans=0.125 2026-09-24 02:54:20,878 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.586e+02 3.397e+02 3.779e+02 4.305e+02 8.230e+02, threshold=7.558e+02, percent-clipped=2.0 2026-09-24 02:54:20,884 INFO [train.py:1192] (0/2) Epoch 37, batch 950, loss[loss=0.3675, simple_loss=0.4347, pruned_loss=0.1501, over 11182.00 frames. ], tot_loss[loss=0.2792, simple_loss=0.3884, pruned_loss=0.085, over 4708754.85 frames. ], batch size: 333, lr: 5.66e-03, grad_scale: 16.0 2026-09-24 02:54:25,213 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-37.pt 2026-09-24 02:54:31,165 INFO [train.py:1192] (0/2) Epoch 38, batch 0, loss[loss=0.2204, simple_loss=0.3418, pruned_loss=0.0495, over 24587.00 frames. ], tot_loss[loss=0.2204, simple_loss=0.3418, pruned_loss=0.0495, over 24587.00 frames. ], batch size: 137, lr: 5.58e-03, grad_scale: 32.0 2026-09-24 02:54:31,165 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 02:54:33,838 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([1.4368, 1.5453, 1.7620, 1.5464, 1.4033, 1.7161, 0.9391, 1.1692], device='cuda:0') 2026-09-24 02:54:42,986 INFO [train.py:1224] (0/2) Epoch 38, validation: loss=0.179, simple_loss=0.2974, pruned_loss=0.0303, over 2564189.00 frames. 2026-09-24 02:54:42,986 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 02:54:43,157 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.06 vs. limit=15.0 2026-09-24 02:54:52,793 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=118220.0, ans=0.2 2026-09-24 02:54:53,853 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.00 vs. limit=10.0 2026-09-24 02:54:58,044 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.39 vs. limit=10.0 2026-09-24 02:55:09,019 INFO [train.py:1192] (0/2) Epoch 38, batch 50, loss[loss=0.2261, simple_loss=0.3297, pruned_loss=0.06121, over 24271.00 frames. ], tot_loss[loss=0.2888, simple_loss=0.3974, pruned_loss=0.09005, over 1076123.26 frames. ], batch size: 125, lr: 5.58e-03, grad_scale: 32.0 2026-09-24 02:55:30,410 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.646e+02 3.388e+02 3.790e+02 4.470e+02 6.487e+02, threshold=7.579e+02, percent-clipped=0.0 2026-09-24 02:55:34,353 INFO [train.py:1192] (0/2) Epoch 38, batch 100, loss[loss=0.2591, simple_loss=0.3676, pruned_loss=0.07531, over 24632.00 frames. ], tot_loss[loss=0.2877, simple_loss=0.3984, pruned_loss=0.08852, over 1904404.92 frames. ], batch size: 154, lr: 5.58e-03, grad_scale: 32.0 2026-09-24 02:55:34,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=118486.66666666667, ans=0.125 2026-09-24 02:55:47,783 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=118553.33333333333, ans=0.0 2026-09-24 02:55:49,370 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=118586.66666666667, ans=0.125 2026-09-24 02:55:50,841 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=118586.66666666667, ans=0.125 2026-09-24 02:55:53,195 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.90 vs. limit=15.0 2026-09-24 02:55:54,978 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.50 vs. limit=6.0 2026-09-24 02:56:00,439 INFO [train.py:1192] (0/2) Epoch 38, batch 150, loss[loss=0.2755, simple_loss=0.3679, pruned_loss=0.09159, over 24282.00 frames. ], tot_loss[loss=0.2846, simple_loss=0.3952, pruned_loss=0.087, over 2558362.70 frames. ], batch size: 125, lr: 5.57e-03, grad_scale: 32.0 2026-09-24 02:56:08,114 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=118686.66666666667, ans=10.0 2026-09-24 02:56:21,948 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.483e+02 3.407e+02 3.926e+02 4.423e+02 6.071e+02, threshold=7.852e+02, percent-clipped=0.0 2026-09-24 02:56:25,873 INFO [train.py:1192] (0/2) Epoch 38, batch 200, loss[loss=0.3308, simple_loss=0.4265, pruned_loss=0.1175, over 21007.00 frames. ], tot_loss[loss=0.2819, simple_loss=0.3927, pruned_loss=0.08557, over 3054497.78 frames. ], batch size: 333, lr: 5.57e-03, grad_scale: 32.0 2026-09-24 02:56:28,315 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=118820.0, ans=0.125 2026-09-24 02:56:33,330 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=118853.33333333333, ans=0.125 2026-09-24 02:56:35,117 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=118853.33333333333, ans=0.1 2026-09-24 02:56:38,049 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=118886.66666666667, ans=0.125 2026-09-24 02:56:39,946 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.54 vs. limit=15.0 2026-09-24 02:56:42,077 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=118920.0, ans=0.0 2026-09-24 02:56:43,057 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:56:47,789 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=118953.33333333333, ans=0.125 2026-09-24 02:56:50,209 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=118953.33333333333, ans=0.125 2026-09-24 02:56:51,196 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=118986.66666666667, ans=0.125 2026-09-24 02:56:51,599 INFO [train.py:1192] (0/2) Epoch 38, batch 250, loss[loss=0.2948, simple_loss=0.4168, pruned_loss=0.08641, over 24314.00 frames. ], tot_loss[loss=0.281, simple_loss=0.3916, pruned_loss=0.08515, over 3443385.09 frames. ], batch size: 234, lr: 5.57e-03, grad_scale: 32.0 2026-09-24 02:57:13,771 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.465e+02 3.346e+02 3.899e+02 4.555e+02 7.441e+02, threshold=7.798e+02, percent-clipped=0.0 2026-09-24 02:57:17,819 INFO [train.py:1192] (0/2) Epoch 38, batch 300, loss[loss=0.3084, simple_loss=0.4241, pruned_loss=0.09635, over 24529.00 frames. ], tot_loss[loss=0.2787, simple_loss=0.3896, pruned_loss=0.0839, over 3749025.96 frames. ], batch size: 204, lr: 5.56e-03, grad_scale: 32.0 2026-09-24 02:57:33,114 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.92 vs. limit=15.0 2026-09-24 02:57:43,540 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=119320.0, ans=0.125 2026-09-24 02:57:43,983 INFO [train.py:1192] (0/2) Epoch 38, batch 350, loss[loss=0.2677, simple_loss=0.3679, pruned_loss=0.08375, over 24552.00 frames. ], tot_loss[loss=0.2794, simple_loss=0.3907, pruned_loss=0.08402, over 3992700.35 frames. ], batch size: 137, lr: 5.56e-03, grad_scale: 32.0 2026-09-24 02:57:47,114 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=119320.0, ans=0.125 2026-09-24 02:57:49,592 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=119353.33333333333, ans=0.2 2026-09-24 02:57:50,048 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 02:57:53,725 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=119353.33333333333, ans=0.0 2026-09-24 02:58:02,718 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=119420.0, ans=0.025 2026-09-24 02:58:05,497 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.732e+02 3.234e+02 3.550e+02 4.047e+02 7.185e+02, threshold=7.100e+02, percent-clipped=0.0 2026-09-24 02:58:09,908 INFO [train.py:1192] (0/2) Epoch 38, batch 400, loss[loss=0.2664, simple_loss=0.3805, pruned_loss=0.07611, over 24575.00 frames. ], tot_loss[loss=0.2781, simple_loss=0.3894, pruned_loss=0.0834, over 4176988.93 frames. ], batch size: 170, lr: 5.55e-03, grad_scale: 32.0 2026-09-24 02:58:13,501 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=119486.66666666667, ans=0.125 2026-09-24 02:58:25,427 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=119586.66666666667, ans=0.125 2026-09-24 02:58:31,241 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=119620.0, ans=0.2 2026-09-24 02:58:35,617 INFO [train.py:1192] (0/2) Epoch 38, batch 450, loss[loss=0.2847, simple_loss=0.3988, pruned_loss=0.08535, over 24649.00 frames. ], tot_loss[loss=0.2791, simple_loss=0.3902, pruned_loss=0.08403, over 4310194.60 frames. ], batch size: 175, lr: 5.55e-03, grad_scale: 32.0 2026-09-24 02:58:38,087 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.19 vs. limit=15.0 2026-09-24 02:58:38,470 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=119653.33333333333, ans=0.1 2026-09-24 02:58:47,462 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=119720.0, ans=0.0 2026-09-24 02:58:51,858 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=119753.33333333333, ans=0.025 2026-09-24 02:58:52,400 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=119753.33333333333, ans=0.0 2026-09-24 02:58:57,085 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.476e+02 3.254e+02 3.716e+02 4.079e+02 6.258e+02, threshold=7.433e+02, percent-clipped=0.0 2026-09-24 02:59:01,215 INFO [train.py:1192] (0/2) Epoch 38, batch 500, loss[loss=0.3136, simple_loss=0.4289, pruned_loss=0.09914, over 24494.00 frames. ], tot_loss[loss=0.2774, simple_loss=0.3883, pruned_loss=0.08326, over 4427616.57 frames. ], batch size: 218, lr: 5.55e-03, grad_scale: 32.0 2026-09-24 02:59:16,932 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=119920.0, ans=0.1 2026-09-24 02:59:17,866 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=119920.0, ans=0.2 2026-09-24 02:59:26,468 INFO [train.py:1192] (0/2) Epoch 38, batch 550, loss[loss=0.3042, simple_loss=0.4246, pruned_loss=0.09187, over 24273.00 frames. ], tot_loss[loss=0.2775, simple_loss=0.3887, pruned_loss=0.08311, over 4517047.85 frames. ], batch size: 257, lr: 5.54e-03, grad_scale: 32.0 2026-09-24 02:59:28,114 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-36000.pt 2026-09-24 02:59:34,619 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=120020.0, ans=0.0 2026-09-24 02:59:36,993 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.86 vs. limit=6.0 2026-09-24 02:59:37,624 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=120053.33333333333, ans=0.125 2026-09-24 02:59:48,236 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=120120.0, ans=0.025 2026-09-24 02:59:48,622 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.442e+02 3.197e+02 3.513e+02 3.957e+02 5.244e+02, threshold=7.025e+02, percent-clipped=0.0 2026-09-24 02:59:48,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=120120.0, ans=0.125 2026-09-24 02:59:52,895 INFO [train.py:1192] (0/2) Epoch 38, batch 600, loss[loss=0.2779, simple_loss=0.4073, pruned_loss=0.07423, over 24418.00 frames. ], tot_loss[loss=0.2784, simple_loss=0.3898, pruned_loss=0.08351, over 4584114.25 frames. ], batch size: 235, lr: 5.54e-03, grad_scale: 32.0 2026-09-24 03:00:01,705 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=120186.66666666667, ans=0.0 2026-09-24 03:00:01,825 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.19 vs. limit=10.0 2026-09-24 03:00:03,063 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=120220.0, ans=0.0 2026-09-24 03:00:05,944 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.14 vs. limit=10.0 2026-09-24 03:00:07,778 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=120220.0, ans=10.0 2026-09-24 03:00:13,626 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=120286.66666666667, ans=0.125 2026-09-24 03:00:18,492 INFO [train.py:1192] (0/2) Epoch 38, batch 650, loss[loss=0.2936, simple_loss=0.3964, pruned_loss=0.09541, over 24578.00 frames. ], tot_loss[loss=0.2777, simple_loss=0.3891, pruned_loss=0.08311, over 4649515.69 frames. ], batch size: 162, lr: 5.54e-03, grad_scale: 32.0 2026-09-24 03:00:22,199 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=120320.0, ans=0.2 2026-09-24 03:00:32,646 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=120386.66666666667, ans=0.125 2026-09-24 03:00:40,702 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.639e+02 3.335e+02 3.771e+02 4.337e+02 7.675e+02, threshold=7.542e+02, percent-clipped=1.0 2026-09-24 03:00:44,700 INFO [train.py:1192] (0/2) Epoch 38, batch 700, loss[loss=0.2584, simple_loss=0.3673, pruned_loss=0.07475, over 24565.00 frames. ], tot_loss[loss=0.2781, simple_loss=0.3897, pruned_loss=0.08324, over 4682372.80 frames. ], batch size: 154, lr: 5.53e-03, grad_scale: 32.0 2026-09-24 03:00:48,688 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.15 vs. limit=22.5 2026-09-24 03:00:48,955 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=120486.66666666667, ans=0.1 2026-09-24 03:00:56,234 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=120553.33333333333, ans=0.125 2026-09-24 03:00:57,169 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=120553.33333333333, ans=0.125 2026-09-24 03:01:05,158 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=120620.0, ans=0.025 2026-09-24 03:01:10,247 INFO [train.py:1192] (0/2) Epoch 38, batch 750, loss[loss=0.2763, simple_loss=0.3925, pruned_loss=0.08005, over 24560.00 frames. ], tot_loss[loss=0.2766, simple_loss=0.3883, pruned_loss=0.08249, over 4713718.66 frames. ], batch size: 170, lr: 5.53e-03, grad_scale: 32.0 2026-09-24 03:01:11,315 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=120653.33333333333, ans=0.1 2026-09-24 03:01:13,984 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=120653.33333333333, ans=0.05 2026-09-24 03:01:28,775 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=120753.33333333333, ans=22.5 2026-09-24 03:01:29,179 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=120753.33333333333, ans=0.125 2026-09-24 03:01:32,091 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.547e+02 3.305e+02 3.709e+02 4.019e+02 5.996e+02, threshold=7.418e+02, percent-clipped=0.0 2026-09-24 03:01:36,221 INFO [train.py:1192] (0/2) Epoch 38, batch 800, loss[loss=0.2712, simple_loss=0.3718, pruned_loss=0.08529, over 24543.00 frames. ], tot_loss[loss=0.2761, simple_loss=0.3878, pruned_loss=0.08223, over 4738475.42 frames. ], batch size: 137, lr: 5.53e-03, grad_scale: 32.0 2026-09-24 03:02:01,831 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=120986.66666666667, ans=0.09899494936611666 2026-09-24 03:02:02,175 INFO [train.py:1192] (0/2) Epoch 38, batch 850, loss[loss=0.2765, simple_loss=0.4002, pruned_loss=0.07642, over 24599.00 frames. ], tot_loss[loss=0.2756, simple_loss=0.3873, pruned_loss=0.08195, over 4761219.42 frames. ], batch size: 198, lr: 5.52e-03, grad_scale: 32.0 2026-09-24 03:02:10,974 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff3.min_abs, batch_count=121020.0, ans=0.2 2026-09-24 03:02:22,315 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=121086.66666666667, ans=0.025 2026-09-24 03:02:24,244 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.820e+02 3.178e+02 3.659e+02 4.130e+02 5.422e+02, threshold=7.319e+02, percent-clipped=0.0 2026-09-24 03:02:28,152 INFO [train.py:1192] (0/2) Epoch 38, batch 900, loss[loss=0.2223, simple_loss=0.3371, pruned_loss=0.05375, over 24561.00 frames. ], tot_loss[loss=0.2766, simple_loss=0.3881, pruned_loss=0.0825, over 4774457.87 frames. ], batch size: 137, lr: 5.52e-03, grad_scale: 32.0 2026-09-24 03:02:32,773 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=121186.66666666667, ans=0.0 2026-09-24 03:02:38,069 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=121220.0, ans=0.125 2026-09-24 03:02:53,699 INFO [train.py:1192] (0/2) Epoch 38, batch 950, loss[loss=0.3532, simple_loss=0.4178, pruned_loss=0.1443, over 11487.00 frames. ], tot_loss[loss=0.277, simple_loss=0.3871, pruned_loss=0.08345, over 4710477.96 frames. ], batch size: 333, lr: 5.51e-03, grad_scale: 32.0 2026-09-24 03:02:58,027 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-38.pt 2026-09-24 03:03:03,569 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=121346.66666666667, ans=0.0 2026-09-24 03:03:03,988 INFO [train.py:1192] (0/2) Epoch 39, batch 0, loss[loss=0.2396, simple_loss=0.3539, pruned_loss=0.0626, over 24566.00 frames. ], tot_loss[loss=0.2396, simple_loss=0.3539, pruned_loss=0.0626, over 24566.00 frames. ], batch size: 137, lr: 5.44e-03, grad_scale: 32.0 2026-09-24 03:03:03,989 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 03:03:14,330 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.6323, 2.7564, 3.2283, 2.7685, 2.4801, 3.1835, 1.9206, 2.5453], device='cuda:0') 2026-09-24 03:03:15,758 INFO [train.py:1224] (0/2) Epoch 39, validation: loss=0.177, simple_loss=0.2956, pruned_loss=0.02925, over 2564189.00 frames. 2026-09-24 03:03:15,758 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 03:03:16,826 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.29 vs. limit=15.0 2026-09-24 03:03:33,307 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.743e+02 3.319e+02 3.713e+02 4.164e+02 7.088e+02, threshold=7.426e+02, percent-clipped=0.0 2026-09-24 03:03:38,817 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=7.85 vs. limit=15.0 2026-09-24 03:03:40,681 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.55 vs. limit=12.0 2026-09-24 03:03:40,995 INFO [train.py:1192] (0/2) Epoch 39, batch 50, loss[loss=0.2164, simple_loss=0.3271, pruned_loss=0.05283, over 24221.00 frames. ], tot_loss[loss=0.2829, simple_loss=0.3934, pruned_loss=0.08616, over 1075362.38 frames. ], batch size: 125, lr: 5.44e-03, grad_scale: 32.0 2026-09-24 03:03:41,573 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=121513.33333333333, ans=0.025 2026-09-24 03:03:46,743 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=121546.66666666667, ans=0.125 2026-09-24 03:03:53,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=121580.0, ans=0.2 2026-09-24 03:04:07,128 INFO [train.py:1192] (0/2) Epoch 39, batch 100, loss[loss=0.2601, simple_loss=0.3778, pruned_loss=0.07115, over 24585.00 frames. ], tot_loss[loss=0.286, simple_loss=0.3975, pruned_loss=0.08723, over 1904403.34 frames. ], batch size: 154, lr: 5.43e-03, grad_scale: 32.0 2026-09-24 03:04:24,350 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.533e+02 3.287e+02 3.705e+02 4.110e+02 5.372e+02, threshold=7.409e+02, percent-clipped=0.0 2026-09-24 03:04:25,906 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=121780.0, ans=0.125 2026-09-24 03:04:26,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=121780.0, ans=0.125 2026-09-24 03:04:31,402 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:04:31,438 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=121813.33333333333, ans=0.0 2026-09-24 03:04:32,697 INFO [train.py:1192] (0/2) Epoch 39, batch 150, loss[loss=0.2459, simple_loss=0.3476, pruned_loss=0.07209, over 24268.00 frames. ], tot_loss[loss=0.281, simple_loss=0.3926, pruned_loss=0.08467, over 2558046.47 frames. ], batch size: 125, lr: 5.43e-03, grad_scale: 32.0 2026-09-24 03:04:49,554 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=16.38 vs. limit=22.5 2026-09-24 03:04:52,252 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=121980.0, ans=0.125 2026-09-24 03:04:58,142 INFO [train.py:1192] (0/2) Epoch 39, batch 200, loss[loss=0.3074, simple_loss=0.41, pruned_loss=0.1024, over 21089.00 frames. ], tot_loss[loss=0.2783, simple_loss=0.3902, pruned_loss=0.08318, over 3054696.14 frames. ], batch size: 333, lr: 5.43e-03, grad_scale: 32.0 2026-09-24 03:05:02,178 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=122013.33333333333, ans=0.2 2026-09-24 03:05:08,599 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=122080.0, ans=0.125 2026-09-24 03:05:12,275 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=6.19 vs. limit=15.0 2026-09-24 03:05:15,481 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.822e+02 3.464e+02 3.893e+02 4.645e+02 6.817e+02, threshold=7.787e+02, percent-clipped=0.0 2026-09-24 03:05:24,147 INFO [train.py:1192] (0/2) Epoch 39, batch 250, loss[loss=0.3108, simple_loss=0.4287, pruned_loss=0.09647, over 24308.00 frames. ], tot_loss[loss=0.2783, simple_loss=0.3897, pruned_loss=0.08348, over 3444463.44 frames. ], batch size: 234, lr: 5.42e-03, grad_scale: 32.0 2026-09-24 03:05:28,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=122180.0, ans=0.125 2026-09-24 03:05:35,723 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.49 vs. limit=12.0 2026-09-24 03:05:45,196 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.54 vs. limit=15.0 2026-09-24 03:05:49,186 INFO [train.py:1192] (0/2) Epoch 39, batch 300, loss[loss=0.2924, simple_loss=0.4155, pruned_loss=0.08469, over 24521.00 frames. ], tot_loss[loss=0.2764, simple_loss=0.388, pruned_loss=0.08244, over 3750657.98 frames. ], batch size: 204, lr: 5.42e-03, grad_scale: 32.0 2026-09-24 03:05:49,407 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.73 vs. limit=15.0 2026-09-24 03:05:57,860 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=122380.0, ans=0.125 2026-09-24 03:06:04,409 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=122446.66666666667, ans=0.0 2026-09-24 03:06:06,880 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.529e+02 3.291e+02 3.545e+02 3.950e+02 6.010e+02, threshold=7.091e+02, percent-clipped=0.0 2026-09-24 03:06:08,840 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=122446.66666666667, ans=0.125 2026-09-24 03:06:11,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=122480.0, ans=0.125 2026-09-24 03:06:11,131 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.87 vs. limit=22.5 2026-09-24 03:06:14,914 INFO [train.py:1192] (0/2) Epoch 39, batch 350, loss[loss=0.2352, simple_loss=0.3436, pruned_loss=0.0634, over 24568.00 frames. ], tot_loss[loss=0.2769, simple_loss=0.3888, pruned_loss=0.08244, over 3994484.34 frames. ], batch size: 137, lr: 5.42e-03, grad_scale: 32.0 2026-09-24 03:06:15,474 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=122513.33333333333, ans=0.0 2026-09-24 03:06:18,274 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=122513.33333333333, ans=0.125 2026-09-24 03:06:21,372 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=122546.66666666667, ans=0.5 2026-09-24 03:06:40,565 INFO [train.py:1192] (0/2) Epoch 39, batch 400, loss[loss=0.2887, simple_loss=0.3949, pruned_loss=0.09121, over 24548.00 frames. ], tot_loss[loss=0.2765, simple_loss=0.3883, pruned_loss=0.08234, over 4177159.20 frames. ], batch size: 170, lr: 5.41e-03, grad_scale: 32.0 2026-09-24 03:06:57,820 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.331e+02 3.207e+02 3.695e+02 4.107e+02 6.392e+02, threshold=7.391e+02, percent-clipped=0.0 2026-09-24 03:06:58,375 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=122780.0, ans=0.125 2026-09-24 03:06:59,415 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:07:01,181 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=122813.33333333333, ans=0.1 2026-09-24 03:07:02,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=122813.33333333333, ans=0.125 2026-09-24 03:07:05,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=122846.66666666667, ans=0.2 2026-09-24 03:07:06,106 INFO [train.py:1192] (0/2) Epoch 39, batch 450, loss[loss=0.2981, simple_loss=0.4055, pruned_loss=0.09531, over 24628.00 frames. ], tot_loss[loss=0.2763, simple_loss=0.3882, pruned_loss=0.08221, over 4311958.26 frames. ], batch size: 175, lr: 5.41e-03, grad_scale: 32.0 2026-09-24 03:07:09,726 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=122846.66666666667, ans=0.1 2026-09-24 03:07:14,585 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=122880.0, ans=0.2 2026-09-24 03:07:16,030 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=122913.33333333333, ans=0.2 2026-09-24 03:07:31,322 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=123013.33333333333, ans=0.025 2026-09-24 03:07:31,712 INFO [train.py:1192] (0/2) Epoch 39, batch 500, loss[loss=0.3237, simple_loss=0.4387, pruned_loss=0.1044, over 24525.00 frames. ], tot_loss[loss=0.276, simple_loss=0.3873, pruned_loss=0.08229, over 4429878.21 frames. ], batch size: 218, lr: 5.41e-03, grad_scale: 32.0 2026-09-24 03:07:35,782 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=123013.33333333333, ans=0.2 2026-09-24 03:07:49,209 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.615e+02 3.265e+02 3.601e+02 4.061e+02 6.133e+02, threshold=7.203e+02, percent-clipped=0.0 2026-09-24 03:07:52,076 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.78 vs. limit=12.0 2026-09-24 03:07:57,452 INFO [train.py:1192] (0/2) Epoch 39, batch 550, loss[loss=0.292, simple_loss=0.4135, pruned_loss=0.0852, over 24262.00 frames. ], tot_loss[loss=0.2765, simple_loss=0.3881, pruned_loss=0.08248, over 4518968.19 frames. ], batch size: 257, lr: 5.40e-03, grad_scale: 32.0 2026-09-24 03:08:03,518 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.04 vs. limit=15.0 2026-09-24 03:08:03,547 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.24 vs. limit=10.0 2026-09-24 03:08:11,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=123246.66666666667, ans=0.125 2026-09-24 03:08:11,509 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=123246.66666666667, ans=0.125 2026-09-24 03:08:18,490 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=123313.33333333333, ans=0.0 2026-09-24 03:08:22,966 INFO [train.py:1192] (0/2) Epoch 39, batch 600, loss[loss=0.3115, simple_loss=0.4334, pruned_loss=0.09481, over 24325.00 frames. ], tot_loss[loss=0.2767, simple_loss=0.3885, pruned_loss=0.08243, over 4584060.41 frames. ], batch size: 234, lr: 5.40e-03, grad_scale: 32.0 2026-09-24 03:08:23,533 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=123346.66666666667, ans=0.1 2026-09-24 03:08:40,113 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.514e+02 3.211e+02 3.556e+02 3.999e+02 6.519e+02, threshold=7.112e+02, percent-clipped=0.0 2026-09-24 03:08:42,714 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.89 vs. limit=22.5 2026-09-24 03:08:48,404 INFO [train.py:1192] (0/2) Epoch 39, batch 650, loss[loss=0.2615, simple_loss=0.3765, pruned_loss=0.07324, over 24567.00 frames. ], tot_loss[loss=0.2754, simple_loss=0.3876, pruned_loss=0.08161, over 4649613.21 frames. ], batch size: 162, lr: 5.40e-03, grad_scale: 32.0 2026-09-24 03:08:48,936 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=123513.33333333333, ans=0.125 2026-09-24 03:08:56,303 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.06 vs. limit=22.5 2026-09-24 03:09:06,055 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=123613.33333333333, ans=0.0 2026-09-24 03:09:11,668 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=123646.66666666667, ans=0.07 2026-09-24 03:09:14,146 INFO [train.py:1192] (0/2) Epoch 39, batch 700, loss[loss=0.2681, simple_loss=0.3727, pruned_loss=0.0818, over 24556.00 frames. ], tot_loss[loss=0.2756, simple_loss=0.3878, pruned_loss=0.08173, over 4683799.77 frames. ], batch size: 154, lr: 5.39e-03, grad_scale: 32.0 2026-09-24 03:09:19,890 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=123713.33333333333, ans=0.0 2026-09-24 03:09:32,402 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.634e+02 3.315e+02 3.820e+02 4.368e+02 6.372e+02, threshold=7.640e+02, percent-clipped=0.0 2026-09-24 03:09:39,928 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=123846.66666666667, ans=0.1 2026-09-24 03:09:40,290 INFO [train.py:1192] (0/2) Epoch 39, batch 750, loss[loss=0.2999, simple_loss=0.4091, pruned_loss=0.09538, over 24573.00 frames. ], tot_loss[loss=0.2748, simple_loss=0.3866, pruned_loss=0.08149, over 4709918.03 frames. ], batch size: 170, lr: 5.39e-03, grad_scale: 32.0 2026-09-24 03:09:54,988 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=123913.33333333333, ans=0.2 2026-09-24 03:10:00,095 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=123946.66666666667, ans=0.125 2026-09-24 03:10:00,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=123946.66666666667, ans=0.0 2026-09-24 03:10:03,502 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=123980.0, ans=0.125 2026-09-24 03:10:04,826 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.82 vs. limit=6.0 2026-09-24 03:10:06,298 INFO [train.py:1192] (0/2) Epoch 39, batch 800, loss[loss=0.2189, simple_loss=0.3344, pruned_loss=0.05168, over 24541.00 frames. ], tot_loss[loss=0.2744, simple_loss=0.3863, pruned_loss=0.08125, over 4735808.88 frames. ], batch size: 137, lr: 5.39e-03, grad_scale: 32.0 2026-09-24 03:10:06,502 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.42 vs. limit=22.5 2026-09-24 03:10:13,623 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.51 vs. limit=22.5 2026-09-24 03:10:18,845 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=124080.0, ans=0.2 2026-09-24 03:10:19,933 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.09 vs. limit=15.0 2026-09-24 03:10:24,160 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.717e+02 3.163e+02 3.551e+02 4.126e+02 5.357e+02, threshold=7.103e+02, percent-clipped=0.0 2026-09-24 03:10:31,386 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=124146.66666666667, ans=0.95 2026-09-24 03:10:32,370 INFO [train.py:1192] (0/2) Epoch 39, batch 850, loss[loss=0.3268, simple_loss=0.4387, pruned_loss=0.1075, over 24604.00 frames. ], tot_loss[loss=0.2739, simple_loss=0.3861, pruned_loss=0.0809, over 4758793.32 frames. ], batch size: 198, lr: 5.38e-03, grad_scale: 32.0 2026-09-24 03:10:34,528 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=124180.0, ans=0.0 2026-09-24 03:10:47,302 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=124246.66666666667, ans=0.0 2026-09-24 03:10:47,303 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=124246.66666666667, ans=0.125 2026-09-24 03:10:53,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=124313.33333333333, ans=0.0 2026-09-24 03:10:54,161 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=124313.33333333333, ans=0.1 2026-09-24 03:10:58,698 INFO [train.py:1192] (0/2) Epoch 39, batch 900, loss[loss=0.2373, simple_loss=0.3503, pruned_loss=0.06219, over 24539.00 frames. ], tot_loss[loss=0.2746, simple_loss=0.3867, pruned_loss=0.0813, over 4772956.17 frames. ], batch size: 137, lr: 5.38e-03, grad_scale: 32.0 2026-09-24 03:11:15,669 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.434e+02 3.204e+02 3.697e+02 4.181e+02 5.797e+02, threshold=7.393e+02, percent-clipped=0.0 2026-09-24 03:11:16,267 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=124446.66666666667, ans=0.025 2026-09-24 03:11:17,642 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=124446.66666666667, ans=0.125 2026-09-24 03:11:21,591 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=124480.0, ans=0.0 2026-09-24 03:11:24,010 INFO [train.py:1192] (0/2) Epoch 39, batch 950, loss[loss=0.4297, simple_loss=0.4656, pruned_loss=0.1969, over 11313.00 frames. ], tot_loss[loss=0.2748, simple_loss=0.3852, pruned_loss=0.08218, over 4714715.68 frames. ], batch size: 334, lr: 5.37e-03, grad_scale: 32.0 2026-09-24 03:11:24,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=124513.33333333333, ans=0.125 2026-09-24 03:11:28,357 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-39.pt 2026-09-24 03:11:35,755 INFO [train.py:1192] (0/2) Epoch 40, batch 0, loss[loss=0.22, simple_loss=0.3424, pruned_loss=0.04883, over 24564.00 frames. ], tot_loss[loss=0.22, simple_loss=0.3424, pruned_loss=0.04883, over 24564.00 frames. ], batch size: 137, lr: 5.31e-03, grad_scale: 32.0 2026-09-24 03:11:35,755 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 03:11:38,903 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.8504, 3.1540, 3.0557, 1.8343], device='cuda:0') 2026-09-24 03:11:47,648 INFO [train.py:1224] (0/2) Epoch 40, validation: loss=0.176, simple_loss=0.2949, pruned_loss=0.0286, over 2564189.00 frames. 2026-09-24 03:11:47,648 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 03:11:51,110 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.16 vs. limit=22.5 2026-09-24 03:11:53,123 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.81 vs. limit=22.5 2026-09-24 03:11:53,734 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=124573.33333333333, ans=0.0 2026-09-24 03:11:57,797 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=124606.66666666667, ans=0.0 2026-09-24 03:12:08,495 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.68 vs. limit=15.0 2026-09-24 03:12:12,965 INFO [train.py:1192] (0/2) Epoch 40, batch 50, loss[loss=0.2464, simple_loss=0.3511, pruned_loss=0.07091, over 24254.00 frames. ], tot_loss[loss=0.2815, simple_loss=0.393, pruned_loss=0.08502, over 1076601.79 frames. ], batch size: 125, lr: 5.30e-03, grad_scale: 32.0 2026-09-24 03:12:16,055 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=124706.66666666667, ans=0.125 2026-09-24 03:12:22,725 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.87 vs. limit=15.0 2026-09-24 03:12:26,217 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.799e+02 3.387e+02 3.810e+02 4.289e+02 6.309e+02, threshold=7.620e+02, percent-clipped=0.0 2026-09-24 03:12:27,364 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=124773.33333333333, ans=0.5 2026-09-24 03:12:38,644 INFO [train.py:1192] (0/2) Epoch 40, batch 100, loss[loss=0.2476, simple_loss=0.3614, pruned_loss=0.06687, over 24619.00 frames. ], tot_loss[loss=0.2844, simple_loss=0.3964, pruned_loss=0.08619, over 1904574.07 frames. ], batch size: 154, lr: 5.30e-03, grad_scale: 64.0 2026-09-24 03:13:04,708 INFO [train.py:1192] (0/2) Epoch 40, batch 150, loss[loss=0.2374, simple_loss=0.3417, pruned_loss=0.06656, over 24261.00 frames. ], tot_loss[loss=0.2793, simple_loss=0.3917, pruned_loss=0.08349, over 2558924.05 frames. ], batch size: 125, lr: 5.30e-03, grad_scale: 64.0 2026-09-24 03:13:09,266 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=125073.33333333333, ans=0.0 2026-09-24 03:13:18,084 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.382e+02 3.300e+02 3.693e+02 4.054e+02 6.775e+02, threshold=7.385e+02, percent-clipped=0.0 2026-09-24 03:13:30,571 INFO [train.py:1192] (0/2) Epoch 40, batch 200, loss[loss=0.3278, simple_loss=0.425, pruned_loss=0.1153, over 21002.00 frames. ], tot_loss[loss=0.2771, simple_loss=0.3897, pruned_loss=0.08222, over 3055411.82 frames. ], batch size: 333, lr: 5.29e-03, grad_scale: 64.0 2026-09-24 03:13:31,986 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=125206.66666666667, ans=0.025 2026-09-24 03:13:34,301 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=125206.66666666667, ans=0.1 2026-09-24 03:13:36,164 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.43 vs. limit=12.0 2026-09-24 03:13:40,545 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=6.96 vs. limit=15.0 2026-09-24 03:13:48,126 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=125306.66666666667, ans=0.125 2026-09-24 03:13:51,977 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=125340.0, ans=0.0 2026-09-24 03:13:56,672 INFO [train.py:1192] (0/2) Epoch 40, batch 250, loss[loss=0.295, simple_loss=0.4176, pruned_loss=0.08619, over 24313.00 frames. ], tot_loss[loss=0.2764, simple_loss=0.3886, pruned_loss=0.08213, over 3444664.81 frames. ], batch size: 234, lr: 5.29e-03, grad_scale: 64.0 2026-09-24 03:14:03,483 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.79 vs. limit=15.0 2026-09-24 03:14:10,309 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.416e+02 3.427e+02 3.987e+02 4.778e+02 6.902e+02, threshold=7.973e+02, percent-clipped=0.0 2026-09-24 03:14:17,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=125506.66666666667, ans=0.0 2026-09-24 03:14:22,778 INFO [train.py:1192] (0/2) Epoch 40, batch 300, loss[loss=0.2889, simple_loss=0.4028, pruned_loss=0.08749, over 24547.00 frames. ], tot_loss[loss=0.2761, simple_loss=0.3878, pruned_loss=0.08219, over 3750229.30 frames. ], batch size: 204, lr: 5.28e-03, grad_scale: 64.0 2026-09-24 03:14:40,874 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=125640.0, ans=0.09899494936611666 2026-09-24 03:14:42,788 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=125673.33333333333, ans=0.025 2026-09-24 03:14:45,734 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.93 vs. limit=15.0 2026-09-24 03:14:48,293 INFO [train.py:1192] (0/2) Epoch 40, batch 350, loss[loss=0.2177, simple_loss=0.3327, pruned_loss=0.05142, over 24593.00 frames. ], tot_loss[loss=0.2773, simple_loss=0.3891, pruned_loss=0.08272, over 3989754.06 frames. ], batch size: 137, lr: 5.28e-03, grad_scale: 64.0 2026-09-24 03:14:57,001 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=125740.0, ans=0.015 2026-09-24 03:15:01,562 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.650e+02 3.199e+02 3.646e+02 4.083e+02 5.507e+02, threshold=7.292e+02, percent-clipped=0.0 2026-09-24 03:15:02,653 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=125773.33333333333, ans=0.0 2026-09-24 03:15:03,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=125806.66666666667, ans=0.0 2026-09-24 03:15:06,954 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=125806.66666666667, ans=0.125 2026-09-24 03:15:08,409 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=125840.0, ans=0.0 2026-09-24 03:15:09,801 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=125840.0, ans=0.0 2026-09-24 03:15:10,258 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=125840.0, ans=0.125 2026-09-24 03:15:13,997 INFO [train.py:1192] (0/2) Epoch 40, batch 400, loss[loss=0.251, simple_loss=0.366, pruned_loss=0.06805, over 24553.00 frames. ], tot_loss[loss=0.2763, simple_loss=0.3882, pruned_loss=0.08216, over 4178135.05 frames. ], batch size: 170, lr: 5.28e-03, grad_scale: 64.0 2026-09-24 03:15:14,690 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.12 vs. limit=6.0 2026-09-24 03:15:18,716 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.27 vs. limit=12.0 2026-09-24 03:15:29,833 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=125973.33333333333, ans=0.125 2026-09-24 03:15:36,092 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=126006.66666666667, ans=0.125 2026-09-24 03:15:39,668 INFO [train.py:1192] (0/2) Epoch 40, batch 450, loss[loss=0.2848, simple_loss=0.4041, pruned_loss=0.08282, over 24625.00 frames. ], tot_loss[loss=0.2769, simple_loss=0.3888, pruned_loss=0.08252, over 4310826.30 frames. ], batch size: 175, lr: 5.27e-03, grad_scale: 64.0 2026-09-24 03:15:47,221 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=126073.33333333333, ans=0.125 2026-09-24 03:15:53,261 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.616e+02 3.315e+02 3.764e+02 4.234e+02 7.619e+02, threshold=7.527e+02, percent-clipped=1.0 2026-09-24 03:15:53,553 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.95 vs. limit=22.5 2026-09-24 03:15:58,000 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=126140.0, ans=0.125 2026-09-24 03:16:03,504 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.07 vs. limit=22.5 2026-09-24 03:16:06,365 INFO [train.py:1192] (0/2) Epoch 40, batch 500, loss[loss=0.2725, simple_loss=0.402, pruned_loss=0.07153, over 24481.00 frames. ], tot_loss[loss=0.2764, simple_loss=0.3878, pruned_loss=0.08245, over 4427851.00 frames. ], batch size: 218, lr: 5.27e-03, grad_scale: 64.0 2026-09-24 03:16:06,447 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=126206.66666666667, ans=0.035 2026-09-24 03:16:07,503 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:16:12,038 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=126240.0, ans=0.125 2026-09-24 03:16:14,713 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.11 vs. limit=15.0 2026-09-24 03:16:21,300 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=126273.33333333333, ans=0.0 2026-09-24 03:16:32,542 INFO [train.py:1192] (0/2) Epoch 40, batch 550, loss[loss=0.2946, simple_loss=0.4083, pruned_loss=0.09041, over 24261.00 frames. ], tot_loss[loss=0.276, simple_loss=0.3877, pruned_loss=0.08217, over 4517067.29 frames. ], batch size: 257, lr: 5.27e-03, grad_scale: 64.0 2026-09-24 03:16:32,757 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.84 vs. limit=10.0 2026-09-24 03:16:45,767 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.472e+02 3.255e+02 3.676e+02 4.301e+02 8.242e+02, threshold=7.351e+02, percent-clipped=1.0 2026-09-24 03:16:58,500 INFO [train.py:1192] (0/2) Epoch 40, batch 600, loss[loss=0.3168, simple_loss=0.4297, pruned_loss=0.102, over 24364.00 frames. ], tot_loss[loss=0.2768, simple_loss=0.3888, pruned_loss=0.08241, over 4584269.67 frames. ], batch size: 234, lr: 5.26e-03, grad_scale: 64.0 2026-09-24 03:17:01,501 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.33 vs. limit=8.0 2026-09-24 03:17:01,673 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=126540.0, ans=0.125 2026-09-24 03:17:07,690 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=126573.33333333333, ans=0.125 2026-09-24 03:17:17,911 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=126640.0, ans=0.125 2026-09-24 03:17:18,472 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=126673.33333333333, ans=0.125 2026-09-24 03:17:22,584 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=126673.33333333333, ans=0.0 2026-09-24 03:17:23,589 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=126706.66666666667, ans=0.0 2026-09-24 03:17:23,963 INFO [train.py:1192] (0/2) Epoch 40, batch 650, loss[loss=0.2629, simple_loss=0.3791, pruned_loss=0.0734, over 24562.00 frames. ], tot_loss[loss=0.2765, simple_loss=0.3884, pruned_loss=0.08228, over 4649476.73 frames. ], batch size: 162, lr: 5.26e-03, grad_scale: 64.0 2026-09-24 03:17:27,537 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=12.06 vs. limit=22.5 2026-09-24 03:17:31,998 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=126740.0, ans=0.125 2026-09-24 03:17:32,946 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=126740.0, ans=0.0 2026-09-24 03:17:37,282 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.662e+02 3.206e+02 3.623e+02 4.248e+02 7.318e+02, threshold=7.246e+02, percent-clipped=0.0 2026-09-24 03:17:37,383 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=126773.33333333333, ans=0.0 2026-09-24 03:17:37,900 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=126773.33333333333, ans=0.1 2026-09-24 03:17:47,726 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=126840.0, ans=0.125 2026-09-24 03:17:48,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=126840.0, ans=0.1 2026-09-24 03:17:49,654 INFO [train.py:1192] (0/2) Epoch 40, batch 700, loss[loss=0.2604, simple_loss=0.3689, pruned_loss=0.07591, over 24566.00 frames. ], tot_loss[loss=0.277, simple_loss=0.389, pruned_loss=0.08244, over 4681358.64 frames. ], batch size: 154, lr: 5.26e-03, grad_scale: 64.0 2026-09-24 03:17:51,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=126873.33333333333, ans=0.025 2026-09-24 03:18:04,039 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.44 vs. limit=15.0 2026-09-24 03:18:12,804 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=127006.66666666667, ans=0.125 2026-09-24 03:18:14,410 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=127006.66666666667, ans=0.2 2026-09-24 03:18:15,795 INFO [train.py:1192] (0/2) Epoch 40, batch 750, loss[loss=0.2595, simple_loss=0.3838, pruned_loss=0.06762, over 24560.00 frames. ], tot_loss[loss=0.2758, simple_loss=0.3877, pruned_loss=0.08198, over 4712717.07 frames. ], batch size: 170, lr: 5.25e-03, grad_scale: 64.0 2026-09-24 03:18:16,596 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=127040.0, ans=0.125 2026-09-24 03:18:22,772 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=127073.33333333333, ans=0.125 2026-09-24 03:18:29,577 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.517e+02 3.310e+02 3.828e+02 4.163e+02 6.054e+02, threshold=7.657e+02, percent-clipped=0.0 2026-09-24 03:18:31,168 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=127140.0, ans=0.0 2026-09-24 03:18:37,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=127173.33333333333, ans=0.1 2026-09-24 03:18:41,726 INFO [train.py:1192] (0/2) Epoch 40, batch 800, loss[loss=0.2266, simple_loss=0.336, pruned_loss=0.05861, over 24594.00 frames. ], tot_loss[loss=0.2751, simple_loss=0.387, pruned_loss=0.08165, over 4738375.65 frames. ], batch size: 137, lr: 5.25e-03, grad_scale: 64.0 2026-09-24 03:18:53,014 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=127273.33333333333, ans=0.1 2026-09-24 03:18:55,749 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=127273.33333333333, ans=0.125 2026-09-24 03:18:59,020 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:19:01,052 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=127306.66666666667, ans=0.2 2026-09-24 03:19:01,180 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=11.18 vs. limit=15.0 2026-09-24 03:19:04,389 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=127340.0, ans=0.125 2026-09-24 03:19:05,391 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=127340.0, ans=0.125 2026-09-24 03:19:07,091 INFO [train.py:1192] (0/2) Epoch 40, batch 850, loss[loss=0.2966, simple_loss=0.411, pruned_loss=0.09106, over 24589.00 frames. ], tot_loss[loss=0.2742, simple_loss=0.3861, pruned_loss=0.08116, over 4760840.63 frames. ], batch size: 198, lr: 5.25e-03, grad_scale: 64.0 2026-09-24 03:19:17,368 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=127440.0, ans=0.125 2026-09-24 03:19:20,520 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.749e+02 3.370e+02 3.782e+02 4.373e+02 5.808e+02, threshold=7.564e+02, percent-clipped=0.0 2026-09-24 03:19:29,310 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.25 vs. limit=15.0 2026-09-24 03:19:32,624 INFO [train.py:1192] (0/2) Epoch 40, batch 900, loss[loss=0.2196, simple_loss=0.3403, pruned_loss=0.04951, over 24555.00 frames. ], tot_loss[loss=0.2747, simple_loss=0.3866, pruned_loss=0.08136, over 4774044.07 frames. ], batch size: 137, lr: 5.24e-03, grad_scale: 64.0 2026-09-24 03:19:33,650 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=127540.0, ans=0.2 2026-09-24 03:19:43,430 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=127606.66666666667, ans=0.1 2026-09-24 03:19:58,310 INFO [train.py:1192] (0/2) Epoch 40, batch 950, loss[loss=0.3578, simple_loss=0.421, pruned_loss=0.1473, over 11266.00 frames. ], tot_loss[loss=0.275, simple_loss=0.3854, pruned_loss=0.08224, over 4715598.45 frames. ], batch size: 334, lr: 5.24e-03, grad_scale: 32.0 2026-09-24 03:20:02,419 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-40.pt 2026-09-24 03:20:09,889 INFO [train.py:1192] (0/2) Epoch 41, batch 0, loss[loss=0.2436, simple_loss=0.3597, pruned_loss=0.06373, over 24573.00 frames. ], tot_loss[loss=0.2436, simple_loss=0.3597, pruned_loss=0.06373, over 24573.00 frames. ], batch size: 137, lr: 5.18e-03, grad_scale: 32.0 2026-09-24 03:20:09,889 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 03:20:21,760 INFO [train.py:1224] (0/2) Epoch 41, validation: loss=0.1763, simple_loss=0.295, pruned_loss=0.02877, over 2564189.00 frames. 2026-09-24 03:20:21,761 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 03:20:30,716 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=127766.66666666667, ans=0.125 2026-09-24 03:20:31,164 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.577e+02 3.354e+02 3.712e+02 4.319e+02 6.875e+02, threshold=7.424e+02, percent-clipped=0.0 2026-09-24 03:20:35,542 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=127800.0, ans=0.125 2026-09-24 03:20:46,349 INFO [train.py:1192] (0/2) Epoch 41, batch 50, loss[loss=0.2159, simple_loss=0.3246, pruned_loss=0.05361, over 24300.00 frames. ], tot_loss[loss=0.2788, simple_loss=0.3913, pruned_loss=0.08318, over 1075686.10 frames. ], batch size: 125, lr: 5.17e-03, grad_scale: 32.0 2026-09-24 03:21:08,742 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.39 vs. limit=22.5 2026-09-24 03:21:10,182 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=128033.33333333333, ans=0.0 2026-09-24 03:21:11,879 INFO [train.py:1192] (0/2) Epoch 41, batch 100, loss[loss=0.2531, simple_loss=0.3691, pruned_loss=0.06857, over 24611.00 frames. ], tot_loss[loss=0.2848, simple_loss=0.3969, pruned_loss=0.0863, over 1904516.35 frames. ], batch size: 154, lr: 5.17e-03, grad_scale: 32.0 2026-09-24 03:21:16,630 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.13 vs. limit=15.0 2026-09-24 03:21:21,532 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.622e+02 3.322e+02 3.653e+02 4.087e+02 6.031e+02, threshold=7.305e+02, percent-clipped=0.0 2026-09-24 03:21:23,712 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=12.0 2026-09-24 03:21:36,475 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=128200.0, ans=0.1 2026-09-24 03:21:36,879 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=128233.33333333333, ans=0.125 2026-09-24 03:21:37,300 INFO [train.py:1192] (0/2) Epoch 41, batch 150, loss[loss=0.2011, simple_loss=0.3148, pruned_loss=0.04372, over 24238.00 frames. ], tot_loss[loss=0.2786, simple_loss=0.3911, pruned_loss=0.08305, over 2557982.39 frames. ], batch size: 125, lr: 5.17e-03, grad_scale: 32.0 2026-09-24 03:21:40,715 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=128233.33333333333, ans=0.125 2026-09-24 03:21:41,634 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=128233.33333333333, ans=0.125 2026-09-24 03:21:52,578 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.47 vs. limit=15.0 2026-09-24 03:21:52,788 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.49 vs. limit=6.0 2026-09-24 03:21:56,252 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.06 vs. limit=15.0 2026-09-24 03:21:59,296 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=128366.66666666667, ans=0.125 2026-09-24 03:22:03,508 INFO [train.py:1192] (0/2) Epoch 41, batch 200, loss[loss=0.3024, simple_loss=0.411, pruned_loss=0.09693, over 21151.00 frames. ], tot_loss[loss=0.2763, simple_loss=0.3889, pruned_loss=0.08185, over 3055256.51 frames. ], batch size: 333, lr: 5.16e-03, grad_scale: 32.0 2026-09-24 03:22:09,952 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=128433.33333333333, ans=0.125 2026-09-24 03:22:13,110 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.657e+02 3.570e+02 3.881e+02 4.626e+02 6.095e+02, threshold=7.762e+02, percent-clipped=0.0 2026-09-24 03:22:14,243 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=128466.66666666667, ans=0.1 2026-09-24 03:22:18,419 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=128500.0, ans=0.1 2026-09-24 03:22:29,463 INFO [train.py:1192] (0/2) Epoch 41, batch 250, loss[loss=0.3166, simple_loss=0.4337, pruned_loss=0.09974, over 24298.00 frames. ], tot_loss[loss=0.2762, simple_loss=0.3887, pruned_loss=0.08188, over 3444795.13 frames. ], batch size: 234, lr: 5.16e-03, grad_scale: 32.0 2026-09-24 03:22:31,966 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=128566.66666666667, ans=0.125 2026-09-24 03:22:34,349 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=128600.0, ans=0.0 2026-09-24 03:22:38,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=128600.0, ans=0.0 2026-09-24 03:22:42,190 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=128633.33333333333, ans=0.125 2026-09-24 03:22:43,851 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.95 vs. limit=22.5 2026-09-24 03:22:47,151 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=128666.66666666667, ans=0.04949747468305833 2026-09-24 03:22:50,944 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.19 vs. limit=15.0 2026-09-24 03:22:55,051 INFO [train.py:1192] (0/2) Epoch 41, batch 300, loss[loss=0.2766, simple_loss=0.3908, pruned_loss=0.08116, over 24549.00 frames. ], tot_loss[loss=0.2752, simple_loss=0.3874, pruned_loss=0.0815, over 3749453.79 frames. ], batch size: 204, lr: 5.16e-03, grad_scale: 32.0 2026-09-24 03:23:01,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=128766.66666666667, ans=0.0 2026-09-24 03:23:05,199 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.682e+02 3.400e+02 3.675e+02 4.084e+02 5.232e+02, threshold=7.351e+02, percent-clipped=0.0 2026-09-24 03:23:06,678 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=128800.0, ans=0.2 2026-09-24 03:23:07,212 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=128800.0, ans=0.125 2026-09-24 03:23:07,286 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.09 vs. limit=15.0 2026-09-24 03:23:08,081 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=128800.0, ans=0.1 2026-09-24 03:23:21,014 INFO [train.py:1192] (0/2) Epoch 41, batch 350, loss[loss=0.2229, simple_loss=0.334, pruned_loss=0.05593, over 24551.00 frames. ], tot_loss[loss=0.2763, simple_loss=0.3887, pruned_loss=0.08199, over 3989364.07 frames. ], batch size: 137, lr: 5.15e-03, grad_scale: 32.0 2026-09-24 03:23:26,271 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=128933.33333333333, ans=0.0 2026-09-24 03:23:27,584 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.76 vs. limit=15.0 2026-09-24 03:23:33,350 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.24 vs. limit=22.5 2026-09-24 03:23:46,954 INFO [train.py:1192] (0/2) Epoch 41, batch 400, loss[loss=0.2739, simple_loss=0.3902, pruned_loss=0.07887, over 24573.00 frames. ], tot_loss[loss=0.276, simple_loss=0.3883, pruned_loss=0.08189, over 4177127.05 frames. ], batch size: 170, lr: 5.15e-03, grad_scale: 32.0 2026-09-24 03:23:47,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=129066.66666666667, ans=0.125 2026-09-24 03:23:51,797 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.62 vs. limit=22.5 2026-09-24 03:23:56,706 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=129100.0, ans=0.2 2026-09-24 03:23:56,978 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.582e+02 3.301e+02 3.573e+02 3.910e+02 6.479e+02, threshold=7.145e+02, percent-clipped=0.0 2026-09-24 03:24:10,362 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=129200.0, ans=0.1 2026-09-24 03:24:11,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=129200.0, ans=0.1 2026-09-24 03:24:12,982 INFO [train.py:1192] (0/2) Epoch 41, batch 450, loss[loss=0.27, simple_loss=0.393, pruned_loss=0.07352, over 24611.00 frames. ], tot_loss[loss=0.2762, simple_loss=0.3885, pruned_loss=0.08192, over 4312386.33 frames. ], batch size: 175, lr: 5.15e-03, grad_scale: 32.0 2026-09-24 03:24:26,137 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=129300.0, ans=0.025 2026-09-24 03:24:30,080 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=129333.33333333333, ans=0.0 2026-09-24 03:24:33,889 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=129366.66666666667, ans=0.0 2026-09-24 03:24:38,324 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.94 vs. limit=15.0 2026-09-24 03:24:39,063 INFO [train.py:1192] (0/2) Epoch 41, batch 500, loss[loss=0.2835, simple_loss=0.4109, pruned_loss=0.07801, over 24517.00 frames. ], tot_loss[loss=0.2746, simple_loss=0.3866, pruned_loss=0.08127, over 4429870.50 frames. ], batch size: 218, lr: 5.14e-03, grad_scale: 32.0 2026-09-24 03:24:40,177 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.81 vs. limit=10.0 2026-09-24 03:24:48,631 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.368e+02 3.221e+02 3.628e+02 4.202e+02 5.845e+02, threshold=7.256e+02, percent-clipped=0.0 2026-09-24 03:24:52,786 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=129466.66666666667, ans=0.0 2026-09-24 03:24:55,857 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=129500.0, ans=0.125 2026-09-24 03:24:58,249 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.42 vs. limit=15.0 2026-09-24 03:25:00,733 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.67 vs. limit=22.5 2026-09-24 03:25:01,104 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=129533.33333333333, ans=0.0 2026-09-24 03:25:05,015 INFO [train.py:1192] (0/2) Epoch 41, batch 550, loss[loss=0.2913, simple_loss=0.417, pruned_loss=0.08282, over 24314.00 frames. ], tot_loss[loss=0.2745, simple_loss=0.3867, pruned_loss=0.08116, over 4519106.17 frames. ], batch size: 257, lr: 5.14e-03, grad_scale: 32.0 2026-09-24 03:25:07,676 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.64 vs. limit=15.0 2026-09-24 03:25:08,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=129566.66666666667, ans=0.125 2026-09-24 03:25:08,096 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=129566.66666666667, ans=0.2 2026-09-24 03:25:20,894 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=129666.66666666667, ans=0.2 2026-09-24 03:25:28,370 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=129700.0, ans=0.1 2026-09-24 03:25:30,675 INFO [train.py:1192] (0/2) Epoch 41, batch 600, loss[loss=0.2773, simple_loss=0.4009, pruned_loss=0.07688, over 24342.00 frames. ], tot_loss[loss=0.2741, simple_loss=0.3868, pruned_loss=0.08067, over 4585156.83 frames. ], batch size: 234, lr: 5.14e-03, grad_scale: 32.0 2026-09-24 03:25:38,071 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer_ff3.min_abs, batch_count=129766.66666666667, ans=0.2 2026-09-24 03:25:40,286 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.644e+02 3.363e+02 3.684e+02 4.152e+02 7.573e+02, threshold=7.368e+02, percent-clipped=1.0 2026-09-24 03:25:50,308 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=129866.66666666667, ans=0.125 2026-09-24 03:25:55,733 INFO [train.py:1192] (0/2) Epoch 41, batch 650, loss[loss=0.272, simple_loss=0.3844, pruned_loss=0.07984, over 24569.00 frames. ], tot_loss[loss=0.2727, simple_loss=0.3856, pruned_loss=0.07987, over 4650215.27 frames. ], batch size: 162, lr: 5.13e-03, grad_scale: 32.0 2026-09-24 03:26:05,366 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=129933.33333333333, ans=0.2 2026-09-24 03:26:08,489 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=129966.66666666667, ans=0.0 2026-09-24 03:26:08,494 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=129966.66666666667, ans=0.1 2026-09-24 03:26:17,938 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=130033.33333333333, ans=0.0 2026-09-24 03:26:18,963 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=130033.33333333333, ans=0.2 2026-09-24 03:26:20,392 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=130033.33333333333, ans=0.2 2026-09-24 03:26:20,774 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=130033.33333333333, ans=0.125 2026-09-24 03:26:21,602 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.74 vs. limit=10.0 2026-09-24 03:26:21,744 INFO [train.py:1192] (0/2) Epoch 41, batch 700, loss[loss=0.2493, simple_loss=0.3605, pruned_loss=0.06901, over 24561.00 frames. ], tot_loss[loss=0.2728, simple_loss=0.3861, pruned_loss=0.07977, over 4683100.74 frames. ], batch size: 154, lr: 5.13e-03, grad_scale: 32.0 2026-09-24 03:26:29,357 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=130100.0, ans=0.2 2026-09-24 03:26:31,723 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.677e+02 3.266e+02 3.752e+02 4.247e+02 5.971e+02, threshold=7.505e+02, percent-clipped=0.0 2026-09-24 03:26:34,529 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=130133.33333333333, ans=0.0 2026-09-24 03:26:46,819 INFO [train.py:1192] (0/2) Epoch 41, batch 750, loss[loss=0.2779, simple_loss=0.3967, pruned_loss=0.0795, over 24558.00 frames. ], tot_loss[loss=0.2717, simple_loss=0.3846, pruned_loss=0.07936, over 4710846.52 frames. ], batch size: 170, lr: 5.13e-03, grad_scale: 32.0 2026-09-24 03:26:46,929 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=130233.33333333333, ans=0.0 2026-09-24 03:27:00,342 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=130300.0, ans=0.1 2026-09-24 03:27:03,614 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=130333.33333333333, ans=0.125 2026-09-24 03:27:12,707 INFO [train.py:1192] (0/2) Epoch 41, batch 800, loss[loss=0.2355, simple_loss=0.3427, pruned_loss=0.06413, over 24563.00 frames. ], tot_loss[loss=0.2721, simple_loss=0.3848, pruned_loss=0.07971, over 4735875.58 frames. ], batch size: 137, lr: 5.12e-03, grad_scale: 32.0 2026-09-24 03:27:22,723 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.416e+02 3.285e+02 3.718e+02 4.236e+02 6.146e+02, threshold=7.436e+02, percent-clipped=0.0 2026-09-24 03:27:23,378 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=130466.66666666667, ans=0.0 2026-09-24 03:27:25,403 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=130466.66666666667, ans=0.025 2026-09-24 03:27:29,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=130500.0, ans=0.125 2026-09-24 03:27:31,323 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.90 vs. limit=22.5 2026-09-24 03:27:38,053 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=130533.33333333333, ans=0.2 2026-09-24 03:27:38,900 INFO [train.py:1192] (0/2) Epoch 41, batch 850, loss[loss=0.2721, simple_loss=0.3977, pruned_loss=0.07329, over 24588.00 frames. ], tot_loss[loss=0.2718, simple_loss=0.3845, pruned_loss=0.07948, over 4758472.68 frames. ], batch size: 198, lr: 5.12e-03, grad_scale: 32.0 2026-09-24 03:27:56,976 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.58 vs. limit=22.5 2026-09-24 03:27:57,707 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=130666.66666666667, ans=0.125 2026-09-24 03:27:59,003 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.82 vs. limit=15.0 2026-09-24 03:28:02,366 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=130700.0, ans=0.025 2026-09-24 03:28:05,005 INFO [train.py:1192] (0/2) Epoch 41, batch 900, loss[loss=0.2314, simple_loss=0.3418, pruned_loss=0.0605, over 24551.00 frames. ], tot_loss[loss=0.2727, simple_loss=0.3854, pruned_loss=0.07999, over 4772535.43 frames. ], batch size: 137, lr: 5.12e-03, grad_scale: 32.0 2026-09-24 03:28:05,614 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=130733.33333333333, ans=0.125 2026-09-24 03:28:13,769 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=130766.66666666667, ans=0.0 2026-09-24 03:28:14,671 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.270e+02 3.061e+02 3.538e+02 4.190e+02 5.569e+02, threshold=7.076e+02, percent-clipped=0.0 2026-09-24 03:28:15,320 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=130800.0, ans=0.025 2026-09-24 03:28:17,044 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=130800.0, ans=0.0 2026-09-24 03:28:29,003 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=130866.66666666667, ans=0.2 2026-09-24 03:28:30,318 INFO [train.py:1192] (0/2) Epoch 41, batch 950, loss[loss=0.3982, simple_loss=0.4509, pruned_loss=0.1727, over 11827.00 frames. ], tot_loss[loss=0.2739, simple_loss=0.3847, pruned_loss=0.08154, over 4713059.02 frames. ], batch size: 333, lr: 5.11e-03, grad_scale: 32.0 2026-09-24 03:28:34,938 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-41.pt 2026-09-24 03:28:41,255 INFO [train.py:1192] (0/2) Epoch 42, batch 0, loss[loss=0.2333, simple_loss=0.3516, pruned_loss=0.05752, over 24591.00 frames. ], tot_loss[loss=0.2333, simple_loss=0.3516, pruned_loss=0.05752, over 24591.00 frames. ], batch size: 137, lr: 5.05e-03, grad_scale: 32.0 2026-09-24 03:28:41,255 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 03:28:53,098 INFO [train.py:1224] (0/2) Epoch 42, validation: loss=0.1768, simple_loss=0.2955, pruned_loss=0.02906, over 2564189.00 frames. 2026-09-24 03:28:53,098 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 03:28:55,894 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=130926.66666666667, ans=0.125 2026-09-24 03:28:56,906 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=130926.66666666667, ans=0.125 2026-09-24 03:28:59,935 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=130960.0, ans=0.125 2026-09-24 03:29:02,259 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=130960.0, ans=0.0 2026-09-24 03:29:03,990 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=130993.33333333333, ans=0.95 2026-09-24 03:29:04,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=130993.33333333333, ans=0.07 2026-09-24 03:29:12,713 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=22.5 2026-09-24 03:29:19,484 INFO [train.py:1192] (0/2) Epoch 42, batch 50, loss[loss=0.2664, simple_loss=0.3641, pruned_loss=0.08436, over 24286.00 frames. ], tot_loss[loss=0.2798, simple_loss=0.3916, pruned_loss=0.08402, over 1074653.40 frames. ], batch size: 125, lr: 5.05e-03, grad_scale: 32.0 2026-09-24 03:29:24,802 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.770e+02 3.478e+02 3.914e+02 4.628e+02 6.581e+02, threshold=7.828e+02, percent-clipped=0.0 2026-09-24 03:29:33,201 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=131160.0, ans=0.2 2026-09-24 03:29:38,073 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=131193.33333333334, ans=0.0 2026-09-24 03:29:39,118 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=131226.66666666666, ans=0.125 2026-09-24 03:29:41,937 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.26 vs. limit=6.0 2026-09-24 03:29:42,291 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=131226.66666666666, ans=0.05 2026-09-24 03:29:45,044 INFO [train.py:1192] (0/2) Epoch 42, batch 100, loss[loss=0.2779, simple_loss=0.3803, pruned_loss=0.08777, over 24634.00 frames. ], tot_loss[loss=0.2829, simple_loss=0.3952, pruned_loss=0.08531, over 1903112.66 frames. ], batch size: 154, lr: 5.05e-03, grad_scale: 32.0 2026-09-24 03:29:50,024 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=131293.33333333334, ans=0.035 2026-09-24 03:29:58,757 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=131326.66666666666, ans=0.0 2026-09-24 03:30:02,215 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=131360.0, ans=0.0 2026-09-24 03:30:10,432 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.80 vs. limit=22.5 2026-09-24 03:30:10,621 INFO [train.py:1192] (0/2) Epoch 42, batch 150, loss[loss=0.2538, simple_loss=0.3558, pruned_loss=0.07592, over 24236.00 frames. ], tot_loss[loss=0.277, simple_loss=0.3899, pruned_loss=0.08208, over 2557093.48 frames. ], batch size: 125, lr: 5.04e-03, grad_scale: 32.0 2026-09-24 03:30:16,132 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.645e+02 3.318e+02 3.586e+02 4.260e+02 6.638e+02, threshold=7.171e+02, percent-clipped=0.0 2026-09-24 03:30:35,883 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=15.0 2026-09-24 03:30:36,151 INFO [train.py:1192] (0/2) Epoch 42, batch 200, loss[loss=0.3493, simple_loss=0.44, pruned_loss=0.1293, over 21222.00 frames. ], tot_loss[loss=0.2762, simple_loss=0.389, pruned_loss=0.08171, over 3054166.71 frames. ], batch size: 333, lr: 5.04e-03, grad_scale: 32.0 2026-09-24 03:30:47,260 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten.whitening_limit, batch_count=131660.0, ans=22.5 2026-09-24 03:30:49,113 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.43 vs. limit=10.0 2026-09-24 03:30:55,477 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=131693.33333333334, ans=0.2 2026-09-24 03:30:56,918 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=131726.66666666666, ans=0.5 2026-09-24 03:30:59,594 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=131726.66666666666, ans=0.0 2026-09-24 03:31:01,943 INFO [train.py:1192] (0/2) Epoch 42, batch 250, loss[loss=0.3087, simple_loss=0.43, pruned_loss=0.09368, over 24308.00 frames. ], tot_loss[loss=0.2754, simple_loss=0.3881, pruned_loss=0.08137, over 3443678.60 frames. ], batch size: 234, lr: 5.04e-03, grad_scale: 32.0 2026-09-24 03:31:02,897 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.35 vs. limit=15.0 2026-09-24 03:31:07,513 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.589e+02 3.539e+02 4.041e+02 4.530e+02 7.446e+02, threshold=8.082e+02, percent-clipped=1.0 2026-09-24 03:31:27,551 INFO [train.py:1192] (0/2) Epoch 42, batch 300, loss[loss=0.2856, simple_loss=0.4139, pruned_loss=0.07867, over 24546.00 frames. ], tot_loss[loss=0.2747, simple_loss=0.3869, pruned_loss=0.08119, over 3748448.32 frames. ], batch size: 204, lr: 5.03e-03, grad_scale: 32.0 2026-09-24 03:31:33,082 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=131960.0, ans=0.1 2026-09-24 03:31:34,007 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=131960.0, ans=0.035 2026-09-24 03:31:34,032 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=131960.0, ans=0.125 2026-09-24 03:31:35,722 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.64 vs. limit=15.0 2026-09-24 03:31:47,066 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=132026.66666666666, ans=0.2 2026-09-24 03:31:48,036 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=132060.0, ans=0.0 2026-09-24 03:31:53,356 INFO [train.py:1192] (0/2) Epoch 42, batch 350, loss[loss=0.2375, simple_loss=0.3478, pruned_loss=0.06363, over 24558.00 frames. ], tot_loss[loss=0.2755, simple_loss=0.3881, pruned_loss=0.08143, over 3992882.33 frames. ], batch size: 137, lr: 5.03e-03, grad_scale: 32.0 2026-09-24 03:31:55,888 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.44 vs. limit=10.0 2026-09-24 03:31:58,552 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.397e+02 3.160e+02 3.538e+02 3.995e+02 6.562e+02, threshold=7.075e+02, percent-clipped=0.0 2026-09-24 03:32:10,056 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=132193.33333333334, ans=0.125 2026-09-24 03:32:19,030 INFO [train.py:1192] (0/2) Epoch 42, batch 400, loss[loss=0.2667, simple_loss=0.3834, pruned_loss=0.07498, over 24586.00 frames. ], tot_loss[loss=0.2746, simple_loss=0.3871, pruned_loss=0.08104, over 4178974.47 frames. ], batch size: 170, lr: 5.03e-03, grad_scale: 32.0 2026-09-24 03:32:19,656 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=132260.0, ans=0.0 2026-09-24 03:32:34,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=132360.0, ans=0.2 2026-09-24 03:32:42,188 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=132393.33333333334, ans=0.125 2026-09-24 03:32:42,205 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=132393.33333333334, ans=0.025 2026-09-24 03:32:44,927 INFO [train.py:1192] (0/2) Epoch 42, batch 450, loss[loss=0.2818, simple_loss=0.3991, pruned_loss=0.0823, over 24641.00 frames. ], tot_loss[loss=0.2745, simple_loss=0.387, pruned_loss=0.08101, over 4308743.19 frames. ], batch size: 175, lr: 5.02e-03, grad_scale: 32.0 2026-09-24 03:32:47,184 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.42 vs. limit=15.0 2026-09-24 03:32:50,705 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.418e+02 3.287e+02 3.646e+02 4.125e+02 6.218e+02, threshold=7.292e+02, percent-clipped=0.0 2026-09-24 03:32:51,836 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.76 vs. limit=22.5 2026-09-24 03:32:54,162 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=132460.0, ans=0.125 2026-09-24 03:32:56,064 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=4.49 vs. limit=15.0 2026-09-24 03:33:02,457 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=132526.66666666666, ans=0.05 2026-09-24 03:33:10,610 INFO [train.py:1192] (0/2) Epoch 42, batch 500, loss[loss=0.2714, simple_loss=0.4005, pruned_loss=0.07118, over 24507.00 frames. ], tot_loss[loss=0.2742, simple_loss=0.386, pruned_loss=0.08117, over 4426532.76 frames. ], batch size: 218, lr: 5.02e-03, grad_scale: 32.0 2026-09-24 03:33:17,372 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=132626.66666666666, ans=0.125 2026-09-24 03:33:22,604 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=132660.0, ans=0.125 2026-09-24 03:33:22,622 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=132660.0, ans=0.0 2026-09-24 03:33:36,588 INFO [train.py:1192] (0/2) Epoch 42, batch 550, loss[loss=0.3017, simple_loss=0.4222, pruned_loss=0.09056, over 24270.00 frames. ], tot_loss[loss=0.2753, simple_loss=0.3872, pruned_loss=0.08169, over 4516682.10 frames. ], batch size: 257, lr: 5.02e-03, grad_scale: 32.0 2026-09-24 03:33:39,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=132760.0, ans=0.125 2026-09-24 03:33:42,137 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:33:42,505 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.595e+02 3.112e+02 3.537e+02 3.986e+02 5.703e+02, threshold=7.073e+02, percent-clipped=0.0 2026-09-24 03:33:43,666 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=132793.33333333334, ans=0.05 2026-09-24 03:33:44,083 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=132793.33333333334, ans=0.0 2026-09-24 03:33:44,090 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=132793.33333333334, ans=0.025 2026-09-24 03:33:45,462 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=132793.33333333334, ans=0.125 2026-09-24 03:33:47,966 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=132826.66666666666, ans=0.0 2026-09-24 03:33:57,350 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.70 vs. limit=15.0 2026-09-24 03:33:59,236 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=132893.33333333334, ans=0.125 2026-09-24 03:34:02,757 INFO [train.py:1192] (0/2) Epoch 42, batch 600, loss[loss=0.2992, simple_loss=0.4233, pruned_loss=0.0875, over 24312.00 frames. ], tot_loss[loss=0.2755, simple_loss=0.3878, pruned_loss=0.0816, over 4583283.46 frames. ], batch size: 234, lr: 5.02e-03, grad_scale: 32.0 2026-09-24 03:34:06,118 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:34:06,792 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.59 vs. limit=22.5 2026-09-24 03:34:18,295 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=133026.66666666666, ans=0.0 2026-09-24 03:34:19,172 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=133026.66666666666, ans=0.025 2026-09-24 03:34:23,052 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=133060.0, ans=0.1 2026-09-24 03:34:25,876 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=133060.0, ans=0.1 2026-09-24 03:34:27,692 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=133093.33333333334, ans=0.125 2026-09-24 03:34:28,074 INFO [train.py:1192] (0/2) Epoch 42, batch 650, loss[loss=0.2761, simple_loss=0.3928, pruned_loss=0.07977, over 24564.00 frames. ], tot_loss[loss=0.2732, simple_loss=0.386, pruned_loss=0.08023, over 4649227.84 frames. ], batch size: 162, lr: 5.01e-03, grad_scale: 32.0 2026-09-24 03:34:32,741 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=133126.66666666666, ans=0.1 2026-09-24 03:34:33,629 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.432e+02 3.326e+02 3.659e+02 3.994e+02 6.136e+02, threshold=7.318e+02, percent-clipped=0.0 2026-09-24 03:34:39,911 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=6.90 vs. limit=15.0 2026-09-24 03:34:43,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=133193.33333333334, ans=0.125 2026-09-24 03:34:45,183 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=133193.33333333334, ans=0.125 2026-09-24 03:34:45,216 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=133193.33333333334, ans=10.0 2026-09-24 03:34:45,862 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.12 vs. limit=15.0 2026-09-24 03:34:53,865 INFO [train.py:1192] (0/2) Epoch 42, batch 700, loss[loss=0.2419, simple_loss=0.3567, pruned_loss=0.06352, over 24565.00 frames. ], tot_loss[loss=0.2732, simple_loss=0.3863, pruned_loss=0.08006, over 4682721.15 frames. ], batch size: 154, lr: 5.01e-03, grad_scale: 32.0 2026-09-24 03:34:55,996 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=133260.0, ans=0.0 2026-09-24 03:35:04,596 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-40000.pt 2026-09-24 03:35:05,469 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=6.43 vs. limit=12.0 2026-09-24 03:35:06,005 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.72 vs. limit=15.0 2026-09-24 03:35:18,590 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=133393.33333333334, ans=0.125 2026-09-24 03:35:19,594 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_abs, batch_count=133426.66666666666, ans=0.5 2026-09-24 03:35:19,963 INFO [train.py:1192] (0/2) Epoch 42, batch 750, loss[loss=0.284, simple_loss=0.3975, pruned_loss=0.08522, over 24561.00 frames. ], tot_loss[loss=0.2727, simple_loss=0.3854, pruned_loss=0.08, over 4710771.74 frames. ], batch size: 170, lr: 5.01e-03, grad_scale: 32.0 2026-09-24 03:35:25,862 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.650e+02 3.266e+02 3.751e+02 4.333e+02 6.246e+02, threshold=7.502e+02, percent-clipped=0.0 2026-09-24 03:35:27,231 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=133460.0, ans=0.2 2026-09-24 03:35:33,045 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=133493.33333333334, ans=0.125 2026-09-24 03:35:37,947 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=133526.66666666666, ans=0.125 2026-09-24 03:35:45,904 INFO [train.py:1192] (0/2) Epoch 42, batch 800, loss[loss=0.2124, simple_loss=0.3296, pruned_loss=0.04762, over 24554.00 frames. ], tot_loss[loss=0.273, simple_loss=0.3855, pruned_loss=0.08023, over 4735159.12 frames. ], batch size: 137, lr: 5.00e-03, grad_scale: 32.0 2026-09-24 03:35:56,879 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=133660.0, ans=0.04949747468305833 2026-09-24 03:36:11,451 INFO [train.py:1192] (0/2) Epoch 42, batch 850, loss[loss=0.2931, simple_loss=0.4119, pruned_loss=0.08711, over 24596.00 frames. ], tot_loss[loss=0.2731, simple_loss=0.3856, pruned_loss=0.08028, over 4757880.96 frames. ], batch size: 198, lr: 5.00e-03, grad_scale: 32.0 2026-09-24 03:36:13,602 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=133760.0, ans=0.125 2026-09-24 03:36:17,257 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.579e+02 3.197e+02 3.653e+02 4.244e+02 6.667e+02, threshold=7.306e+02, percent-clipped=0.0 2026-09-24 03:36:19,352 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.90 vs. limit=15.0 2026-09-24 03:36:28,099 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=133860.0, ans=0.1 2026-09-24 03:36:33,777 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.63 vs. limit=22.5 2026-09-24 03:36:37,492 INFO [train.py:1192] (0/2) Epoch 42, batch 900, loss[loss=0.2339, simple_loss=0.3486, pruned_loss=0.05957, over 24540.00 frames. ], tot_loss[loss=0.2734, simple_loss=0.3859, pruned_loss=0.08047, over 4771813.51 frames. ], batch size: 137, lr: 5.00e-03, grad_scale: 32.0 2026-09-24 03:36:48,942 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.48 vs. limit=15.0 2026-09-24 03:36:51,809 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.56 vs. limit=10.0 2026-09-24 03:36:58,021 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=134060.0, ans=0.0 2026-09-24 03:37:03,541 INFO [train.py:1192] (0/2) Epoch 42, batch 950, loss[loss=0.3434, simple_loss=0.4154, pruned_loss=0.1358, over 11297.00 frames. ], tot_loss[loss=0.2744, simple_loss=0.3853, pruned_loss=0.08176, over 4713738.98 frames. ], batch size: 334, lr: 4.99e-03, grad_scale: 32.0 2026-09-24 03:37:07,892 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-42.pt 2026-09-24 03:37:12,894 INFO [train.py:1192] (0/2) Epoch 43, batch 0, loss[loss=0.2205, simple_loss=0.3395, pruned_loss=0.05073, over 24563.00 frames. ], tot_loss[loss=0.2205, simple_loss=0.3395, pruned_loss=0.05073, over 24563.00 frames. ], batch size: 137, lr: 4.93e-03, grad_scale: 32.0 2026-09-24 03:37:12,894 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 03:37:17,560 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([1.8969, 2.5710, 2.2812, 2.0090], device='cuda:0') 2026-09-24 03:37:19,923 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.6059, 2.1413, 2.5960, 1.3904], device='cuda:0') 2026-09-24 03:37:24,630 INFO [train.py:1224] (0/2) Epoch 43, validation: loss=0.1758, simple_loss=0.2945, pruned_loss=0.02853, over 2564189.00 frames. 2026-09-24 03:37:24,630 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 03:37:24,740 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=134120.0, ans=0.04949747468305833 2026-09-24 03:37:26,129 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.548e+02 3.430e+02 3.868e+02 4.424e+02 7.560e+02, threshold=7.736e+02, percent-clipped=1.0 2026-09-24 03:37:30,467 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=134153.33333333334, ans=0.1 2026-09-24 03:37:49,519 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=134253.33333333334, ans=0.2 2026-09-24 03:37:50,468 INFO [train.py:1192] (0/2) Epoch 43, batch 50, loss[loss=0.2237, simple_loss=0.3291, pruned_loss=0.05919, over 24282.00 frames. ], tot_loss[loss=0.2788, simple_loss=0.3908, pruned_loss=0.08333, over 1077082.25 frames. ], batch size: 125, lr: 4.93e-03, grad_scale: 64.0 2026-09-24 03:37:56,154 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=134320.0, ans=0.0 2026-09-24 03:38:00,991 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=134353.33333333334, ans=0.2 2026-09-24 03:38:16,964 INFO [train.py:1192] (0/2) Epoch 43, batch 100, loss[loss=0.2757, simple_loss=0.3837, pruned_loss=0.08381, over 24626.00 frames. ], tot_loss[loss=0.2841, simple_loss=0.3961, pruned_loss=0.08602, over 1906377.52 frames. ], batch size: 154, lr: 4.93e-03, grad_scale: 64.0 2026-09-24 03:38:17,086 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=134453.33333333334, ans=0.0 2026-09-24 03:38:17,092 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=134453.33333333334, ans=0.2 2026-09-24 03:38:18,350 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.587e+02 3.349e+02 3.785e+02 4.307e+02 6.139e+02, threshold=7.569e+02, percent-clipped=0.0 2026-09-24 03:38:20,025 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.21 vs. limit=15.0 2026-09-24 03:38:21,644 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.57 vs. limit=10.0 2026-09-24 03:38:32,345 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=134553.33333333334, ans=0.1 2026-09-24 03:38:37,113 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=134586.66666666666, ans=0.0 2026-09-24 03:38:38,299 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=10.59 vs. limit=15.0 2026-09-24 03:38:42,697 INFO [train.py:1192] (0/2) Epoch 43, batch 150, loss[loss=0.2111, simple_loss=0.3183, pruned_loss=0.05195, over 24265.00 frames. ], tot_loss[loss=0.2765, simple_loss=0.3897, pruned_loss=0.08168, over 2559239.21 frames. ], batch size: 125, lr: 4.93e-03, grad_scale: 64.0 2026-09-24 03:38:48,092 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=134653.33333333334, ans=0.0 2026-09-24 03:38:59,309 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=4.65 vs. limit=15.0 2026-09-24 03:39:02,029 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=134720.0, ans=0.125 2026-09-24 03:39:08,303 INFO [train.py:1192] (0/2) Epoch 43, batch 200, loss[loss=0.3144, simple_loss=0.412, pruned_loss=0.1084, over 21040.00 frames. ], tot_loss[loss=0.2744, simple_loss=0.3878, pruned_loss=0.08053, over 3056276.78 frames. ], batch size: 333, lr: 4.92e-03, grad_scale: 64.0 2026-09-24 03:39:09,612 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.539e+02 3.385e+02 3.870e+02 4.574e+02 6.427e+02, threshold=7.741e+02, percent-clipped=0.0 2026-09-24 03:39:09,812 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.90 vs. limit=15.0 2026-09-24 03:39:17,929 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.61 vs. limit=22.5 2026-09-24 03:39:23,923 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=134886.66666666666, ans=0.125 2026-09-24 03:39:27,831 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:39:34,456 INFO [train.py:1192] (0/2) Epoch 43, batch 250, loss[loss=0.3225, simple_loss=0.4315, pruned_loss=0.1068, over 24298.00 frames. ], tot_loss[loss=0.2741, simple_loss=0.3873, pruned_loss=0.08045, over 3445150.90 frames. ], batch size: 234, lr: 4.92e-03, grad_scale: 32.0 2026-09-24 03:39:36,257 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.14 vs. limit=15.0 2026-09-24 03:39:37,481 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=134953.33333333334, ans=0.0 2026-09-24 03:39:41,853 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=134986.66666666666, ans=0.0 2026-09-24 03:39:45,822 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=13.59 vs. limit=15.0 2026-09-24 03:39:53,987 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=135053.33333333334, ans=0.0 2026-09-24 03:39:59,857 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=135120.0, ans=0.0 2026-09-24 03:40:00,258 INFO [train.py:1192] (0/2) Epoch 43, batch 300, loss[loss=0.2799, simple_loss=0.3999, pruned_loss=0.07995, over 24525.00 frames. ], tot_loss[loss=0.2729, simple_loss=0.3859, pruned_loss=0.07998, over 3750016.37 frames. ], batch size: 204, lr: 4.92e-03, grad_scale: 32.0 2026-09-24 03:40:02,363 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.466e+02 3.257e+02 3.581e+02 4.184e+02 6.737e+02, threshold=7.162e+02, percent-clipped=0.0 2026-09-24 03:40:02,993 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=135120.0, ans=0.1 2026-09-24 03:40:03,536 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=135120.0, ans=0.125 2026-09-24 03:40:04,345 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=135120.0, ans=0.2 2026-09-24 03:40:11,112 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.38 vs. limit=15.0 2026-09-24 03:40:11,918 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=135186.66666666666, ans=0.0 2026-09-24 03:40:17,393 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=135220.0, ans=0.0 2026-09-24 03:40:25,980 INFO [train.py:1192] (0/2) Epoch 43, batch 350, loss[loss=0.2239, simple_loss=0.3328, pruned_loss=0.05749, over 24569.00 frames. ], tot_loss[loss=0.2733, simple_loss=0.3867, pruned_loss=0.07995, over 3993859.57 frames. ], batch size: 137, lr: 4.91e-03, grad_scale: 32.0 2026-09-24 03:40:26,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=135286.66666666666, ans=0.2 2026-09-24 03:40:36,738 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=135353.33333333334, ans=0.125 2026-09-24 03:40:46,180 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=135420.0, ans=0.1 2026-09-24 03:40:47,797 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.71 vs. limit=15.0 2026-09-24 03:40:52,034 INFO [train.py:1192] (0/2) Epoch 43, batch 400, loss[loss=0.2787, simple_loss=0.394, pruned_loss=0.08174, over 24548.00 frames. ], tot_loss[loss=0.2724, simple_loss=0.3859, pruned_loss=0.07947, over 4176025.75 frames. ], batch size: 170, lr: 4.91e-03, grad_scale: 32.0 2026-09-24 03:40:53,740 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.813e+02 3.350e+02 3.684e+02 4.267e+02 5.929e+02, threshold=7.368e+02, percent-clipped=0.0 2026-09-24 03:40:56,988 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=135486.66666666666, ans=0.125 2026-09-24 03:40:57,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=135486.66666666666, ans=0.125 2026-09-24 03:40:58,879 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=135486.66666666666, ans=0.125 2026-09-24 03:41:07,188 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=135553.33333333334, ans=0.0 2026-09-24 03:41:09,744 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.06 vs. limit=15.0 2026-09-24 03:41:10,494 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=135553.33333333334, ans=0.0 2026-09-24 03:41:11,027 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=135553.33333333334, ans=0.125 2026-09-24 03:41:15,729 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=135586.66666666666, ans=0.0 2026-09-24 03:41:18,021 INFO [train.py:1192] (0/2) Epoch 43, batch 450, loss[loss=0.2768, simple_loss=0.3962, pruned_loss=0.07867, over 24636.00 frames. ], tot_loss[loss=0.2731, simple_loss=0.3867, pruned_loss=0.07981, over 4307811.88 frames. ], batch size: 175, lr: 4.91e-03, grad_scale: 32.0 2026-09-24 03:41:18,146 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=135620.0, ans=0.0 2026-09-24 03:41:22,777 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=135653.33333333334, ans=0.2 2026-09-24 03:41:26,321 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=135653.33333333334, ans=0.0 2026-09-24 03:41:29,362 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=135686.66666666666, ans=0.07 2026-09-24 03:41:33,325 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=135720.0, ans=0.2 2026-09-24 03:41:39,687 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=7.68 vs. limit=12.0 2026-09-24 03:41:43,625 INFO [train.py:1192] (0/2) Epoch 43, batch 500, loss[loss=0.3175, simple_loss=0.4353, pruned_loss=0.09984, over 24491.00 frames. ], tot_loss[loss=0.2725, simple_loss=0.3854, pruned_loss=0.07984, over 4425681.11 frames. ], batch size: 218, lr: 4.90e-03, grad_scale: 32.0 2026-09-24 03:41:45,632 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.508e+02 3.227e+02 3.679e+02 4.100e+02 5.797e+02, threshold=7.357e+02, percent-clipped=0.0 2026-09-24 03:41:47,693 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=135786.66666666666, ans=0.0 2026-09-24 03:42:09,392 INFO [train.py:1192] (0/2) Epoch 43, batch 550, loss[loss=0.2842, simple_loss=0.4086, pruned_loss=0.07988, over 24279.00 frames. ], tot_loss[loss=0.2733, simple_loss=0.3863, pruned_loss=0.0802, over 4516223.05 frames. ], batch size: 257, lr: 4.90e-03, grad_scale: 32.0 2026-09-24 03:42:10,688 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=135953.33333333334, ans=0.125 2026-09-24 03:42:19,243 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=136020.0, ans=0.125 2026-09-24 03:42:24,087 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=136053.33333333334, ans=0.0 2026-09-24 03:42:26,649 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:42:34,375 INFO [train.py:1192] (0/2) Epoch 43, batch 600, loss[loss=0.3001, simple_loss=0.4183, pruned_loss=0.09093, over 24323.00 frames. ], tot_loss[loss=0.273, simple_loss=0.3861, pruned_loss=0.07993, over 4583770.50 frames. ], batch size: 234, lr: 4.90e-03, grad_scale: 32.0 2026-09-24 03:42:36,395 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.535e+02 3.174e+02 3.490e+02 3.947e+02 5.244e+02, threshold=6.979e+02, percent-clipped=0.0 2026-09-24 03:42:42,836 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=136153.33333333334, ans=0.2 2026-09-24 03:42:44,784 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=136186.66666666666, ans=0.2 2026-09-24 03:42:59,733 INFO [train.py:1192] (0/2) Epoch 43, batch 650, loss[loss=0.2888, simple_loss=0.3936, pruned_loss=0.09202, over 24568.00 frames. ], tot_loss[loss=0.2716, simple_loss=0.385, pruned_loss=0.07914, over 4649377.56 frames. ], batch size: 162, lr: 4.90e-03, grad_scale: 32.0 2026-09-24 03:42:59,837 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=136286.66666666666, ans=0.0 2026-09-24 03:43:03,358 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=136286.66666666666, ans=0.2 2026-09-24 03:43:10,121 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=136353.33333333334, ans=0.125 2026-09-24 03:43:17,584 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=136386.66666666666, ans=0.1 2026-09-24 03:43:25,135 INFO [train.py:1192] (0/2) Epoch 43, batch 700, loss[loss=0.2397, simple_loss=0.3541, pruned_loss=0.06265, over 24572.00 frames. ], tot_loss[loss=0.2712, simple_loss=0.3849, pruned_loss=0.07874, over 4682611.41 frames. ], batch size: 154, lr: 4.89e-03, grad_scale: 32.0 2026-09-24 03:43:27,638 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.557e+02 3.247e+02 3.744e+02 4.318e+02 6.508e+02, threshold=7.488e+02, percent-clipped=0.0 2026-09-24 03:43:29,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=136453.33333333334, ans=0.125 2026-09-24 03:43:36,297 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=136520.0, ans=0.125 2026-09-24 03:43:39,086 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=136520.0, ans=0.5 2026-09-24 03:43:46,364 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=136586.66666666666, ans=0.1 2026-09-24 03:43:51,161 INFO [train.py:1192] (0/2) Epoch 43, batch 750, loss[loss=0.2729, simple_loss=0.3915, pruned_loss=0.07714, over 24574.00 frames. ], tot_loss[loss=0.2702, simple_loss=0.3838, pruned_loss=0.07833, over 4709806.91 frames. ], batch size: 170, lr: 4.89e-03, grad_scale: 32.0 2026-09-24 03:44:00,043 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=136653.33333333334, ans=0.09899494936611666 2026-09-24 03:44:08,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=136720.0, ans=0.125 2026-09-24 03:44:15,305 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.80 vs. limit=12.0 2026-09-24 03:44:16,153 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=136753.33333333334, ans=0.0 2026-09-24 03:44:17,093 INFO [train.py:1192] (0/2) Epoch 43, batch 800, loss[loss=0.2409, simple_loss=0.3519, pruned_loss=0.06498, over 24523.00 frames. ], tot_loss[loss=0.2712, simple_loss=0.3843, pruned_loss=0.07901, over 4735183.77 frames. ], batch size: 137, lr: 4.89e-03, grad_scale: 32.0 2026-09-24 03:44:18,947 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.667e+02 3.416e+02 3.835e+02 4.566e+02 6.616e+02, threshold=7.670e+02, percent-clipped=0.0 2026-09-24 03:44:20,967 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=136786.66666666666, ans=0.125 2026-09-24 03:44:31,581 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=136853.33333333334, ans=0.125 2026-09-24 03:44:42,631 INFO [train.py:1192] (0/2) Epoch 43, batch 850, loss[loss=0.2878, simple_loss=0.4007, pruned_loss=0.0874, over 24620.00 frames. ], tot_loss[loss=0.272, simple_loss=0.3849, pruned_loss=0.07958, over 4758099.42 frames. ], batch size: 198, lr: 4.88e-03, grad_scale: 32.0 2026-09-24 03:44:57,805 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.10 vs. limit=6.0 2026-09-24 03:45:00,554 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=137053.33333333334, ans=0.125 2026-09-24 03:45:08,784 INFO [train.py:1192] (0/2) Epoch 43, batch 900, loss[loss=0.2337, simple_loss=0.346, pruned_loss=0.06072, over 24560.00 frames. ], tot_loss[loss=0.2723, simple_loss=0.3852, pruned_loss=0.07974, over 4772508.25 frames. ], batch size: 137, lr: 4.88e-03, grad_scale: 32.0 2026-09-24 03:45:09,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=137120.0, ans=0.1 2026-09-24 03:45:10,758 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.669e+02 3.203e+02 3.662e+02 4.207e+02 6.397e+02, threshold=7.324e+02, percent-clipped=0.0 2026-09-24 03:45:17,364 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=137153.33333333334, ans=0.125 2026-09-24 03:45:21,333 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.80 vs. limit=15.0 2026-09-24 03:45:24,098 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=137220.0, ans=0.125 2026-09-24 03:45:26,518 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=137220.0, ans=0.0 2026-09-24 03:45:29,146 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.14 vs. limit=22.5 2026-09-24 03:45:33,121 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=137253.33333333334, ans=0.1 2026-09-24 03:45:33,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=137253.33333333334, ans=0.125 2026-09-24 03:45:34,050 INFO [train.py:1192] (0/2) Epoch 43, batch 950, loss[loss=0.3769, simple_loss=0.437, pruned_loss=0.1584, over 11334.00 frames. ], tot_loss[loss=0.2727, simple_loss=0.3839, pruned_loss=0.08072, over 4713354.66 frames. ], batch size: 334, lr: 4.88e-03, grad_scale: 32.0 2026-09-24 03:45:38,185 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-43.pt 2026-09-24 03:45:44,797 INFO [train.py:1192] (0/2) Epoch 44, batch 0, loss[loss=0.2133, simple_loss=0.336, pruned_loss=0.04534, over 24565.00 frames. ], tot_loss[loss=0.2133, simple_loss=0.336, pruned_loss=0.04534, over 24565.00 frames. ], batch size: 137, lr: 4.82e-03, grad_scale: 32.0 2026-09-24 03:45:44,797 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 03:45:56,519 INFO [train.py:1224] (0/2) Epoch 44, validation: loss=0.1759, simple_loss=0.2947, pruned_loss=0.02857, over 2564189.00 frames. 2026-09-24 03:45:56,519 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 03:46:10,214 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.24 vs. limit=15.0 2026-09-24 03:46:20,387 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.625e+02 3.475e+02 3.849e+02 4.265e+02 7.177e+02, threshold=7.699e+02, percent-clipped=0.0 2026-09-24 03:46:22,255 INFO [train.py:1192] (0/2) Epoch 44, batch 50, loss[loss=0.2187, simple_loss=0.3246, pruned_loss=0.05634, over 24230.00 frames. ], tot_loss[loss=0.2773, simple_loss=0.3895, pruned_loss=0.08262, over 1075431.85 frames. ], batch size: 125, lr: 4.82e-03, grad_scale: 32.0 2026-09-24 03:46:28,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=137513.33333333334, ans=0.0 2026-09-24 03:46:30,616 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=137513.33333333334, ans=0.07 2026-09-24 03:46:33,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=137546.66666666666, ans=0.0 2026-09-24 03:46:35,277 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=137546.66666666666, ans=0.2 2026-09-24 03:46:41,922 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=137580.0, ans=0.025 2026-09-24 03:46:42,617 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.37 vs. limit=6.0 2026-09-24 03:46:47,910 INFO [train.py:1192] (0/2) Epoch 44, batch 100, loss[loss=0.2481, simple_loss=0.3695, pruned_loss=0.06339, over 24596.00 frames. ], tot_loss[loss=0.2809, simple_loss=0.3937, pruned_loss=0.08412, over 1904019.67 frames. ], batch size: 154, lr: 4.82e-03, grad_scale: 32.0 2026-09-24 03:46:48,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=137646.66666666666, ans=0.025 2026-09-24 03:46:50,957 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=137646.66666666666, ans=0.0 2026-09-24 03:46:52,459 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=137680.0, ans=0.125 2026-09-24 03:46:53,442 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=137680.0, ans=0.125 2026-09-24 03:46:54,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=137680.0, ans=0.1 2026-09-24 03:47:04,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=137746.66666666666, ans=0.125 2026-09-24 03:47:11,645 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.478e+02 3.527e+02 3.794e+02 4.347e+02 7.117e+02, threshold=7.588e+02, percent-clipped=0.0 2026-09-24 03:47:13,986 INFO [train.py:1192] (0/2) Epoch 44, batch 150, loss[loss=0.227, simple_loss=0.3308, pruned_loss=0.06164, over 24265.00 frames. ], tot_loss[loss=0.2778, simple_loss=0.3904, pruned_loss=0.08259, over 2557737.84 frames. ], batch size: 125, lr: 4.81e-03, grad_scale: 32.0 2026-09-24 03:47:22,278 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=137846.66666666666, ans=0.125 2026-09-24 03:47:31,950 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=137913.33333333334, ans=0.0 2026-09-24 03:47:39,699 INFO [train.py:1192] (0/2) Epoch 44, batch 200, loss[loss=0.3012, simple_loss=0.4055, pruned_loss=0.0985, over 21078.00 frames. ], tot_loss[loss=0.2746, simple_loss=0.3877, pruned_loss=0.08079, over 3055018.05 frames. ], batch size: 333, lr: 4.81e-03, grad_scale: 32.0 2026-09-24 03:47:44,122 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=138013.33333333334, ans=0.025 2026-09-24 03:47:48,295 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=138013.33333333334, ans=0.125 2026-09-24 03:47:50,580 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=138046.66666666666, ans=0.1 2026-09-24 03:47:50,583 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=138046.66666666666, ans=0.125 2026-09-24 03:47:51,092 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=138046.66666666666, ans=0.0 2026-09-24 03:48:03,206 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.576e+02 3.480e+02 3.849e+02 4.558e+02 7.319e+02, threshold=7.698e+02, percent-clipped=0.0 2026-09-24 03:48:05,161 INFO [train.py:1192] (0/2) Epoch 44, batch 250, loss[loss=0.316, simple_loss=0.4373, pruned_loss=0.0973, over 24397.00 frames. ], tot_loss[loss=0.2735, simple_loss=0.3865, pruned_loss=0.08021, over 3444351.96 frames. ], batch size: 235, lr: 4.81e-03, grad_scale: 32.0 2026-09-24 03:48:06,621 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=138146.66666666666, ans=0.1 2026-09-24 03:48:15,664 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.07 vs. limit=15.0 2026-09-24 03:48:30,553 INFO [train.py:1192] (0/2) Epoch 44, batch 300, loss[loss=0.2664, simple_loss=0.3933, pruned_loss=0.06978, over 24552.00 frames. ], tot_loss[loss=0.2726, simple_loss=0.3854, pruned_loss=0.07988, over 3750751.77 frames. ], batch size: 204, lr: 4.80e-03, grad_scale: 32.0 2026-09-24 03:48:31,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=138313.33333333334, ans=0.125 2026-09-24 03:48:33,296 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.65 vs. limit=10.0 2026-09-24 03:48:43,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=138380.0, ans=0.2 2026-09-24 03:48:47,681 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.49 vs. limit=22.5 2026-09-24 03:48:53,674 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.587e+02 3.116e+02 3.558e+02 3.992e+02 7.971e+02, threshold=7.116e+02, percent-clipped=1.0 2026-09-24 03:48:55,527 INFO [train.py:1192] (0/2) Epoch 44, batch 350, loss[loss=0.2386, simple_loss=0.3432, pruned_loss=0.06695, over 24568.00 frames. ], tot_loss[loss=0.2736, simple_loss=0.3869, pruned_loss=0.08018, over 3989932.82 frames. ], batch size: 137, lr: 4.80e-03, grad_scale: 32.0 2026-09-24 03:49:02,098 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=138513.33333333334, ans=0.2 2026-09-24 03:49:08,746 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:49:20,846 INFO [train.py:1192] (0/2) Epoch 44, batch 400, loss[loss=0.2725, simple_loss=0.3851, pruned_loss=0.07994, over 24579.00 frames. ], tot_loss[loss=0.2728, simple_loss=0.386, pruned_loss=0.07983, over 4174417.52 frames. ], batch size: 170, lr: 4.80e-03, grad_scale: 32.0 2026-09-24 03:49:23,113 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.min_positive, batch_count=138646.66666666666, ans=0.025 2026-09-24 03:49:30,846 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=138713.33333333334, ans=0.125 2026-09-24 03:49:37,242 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.33 vs. limit=15.0 2026-09-24 03:49:44,346 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.485e+02 3.290e+02 3.759e+02 4.377e+02 6.505e+02, threshold=7.518e+02, percent-clipped=0.0 2026-09-24 03:49:46,475 INFO [train.py:1192] (0/2) Epoch 44, batch 450, loss[loss=0.2695, simple_loss=0.3902, pruned_loss=0.07446, over 24631.00 frames. ], tot_loss[loss=0.273, simple_loss=0.3861, pruned_loss=0.07996, over 4309549.28 frames. ], batch size: 175, lr: 4.80e-03, grad_scale: 32.0 2026-09-24 03:49:47,245 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.33 vs. limit=15.0 2026-09-24 03:49:52,620 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.34 vs. limit=15.0 2026-09-24 03:49:53,984 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=138846.66666666666, ans=0.1 2026-09-24 03:49:55,392 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=138846.66666666666, ans=0.0 2026-09-24 03:50:01,631 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=138913.33333333334, ans=0.2 2026-09-24 03:50:04,023 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=138913.33333333334, ans=0.025 2026-09-24 03:50:06,875 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=138946.66666666666, ans=0.0 2026-09-24 03:50:11,672 INFO [train.py:1192] (0/2) Epoch 44, batch 500, loss[loss=0.2949, simple_loss=0.4192, pruned_loss=0.08528, over 24529.00 frames. ], tot_loss[loss=0.2707, simple_loss=0.3839, pruned_loss=0.07873, over 4427597.22 frames. ], batch size: 218, lr: 4.79e-03, grad_scale: 32.0 2026-09-24 03:50:19,705 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=139013.33333333334, ans=0.125 2026-09-24 03:50:26,607 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=139080.0, ans=0.125 2026-09-24 03:50:31,968 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=139113.33333333334, ans=0.1 2026-09-24 03:50:35,231 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.334e+02 3.160e+02 3.480e+02 3.855e+02 6.130e+02, threshold=6.959e+02, percent-clipped=0.0 2026-09-24 03:50:37,416 INFO [train.py:1192] (0/2) Epoch 44, batch 550, loss[loss=0.2897, simple_loss=0.4091, pruned_loss=0.08521, over 24244.00 frames. ], tot_loss[loss=0.2714, simple_loss=0.3847, pruned_loss=0.07901, over 4517880.78 frames. ], batch size: 257, lr: 4.79e-03, grad_scale: 32.0 2026-09-24 03:50:48,917 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=6.24 vs. limit=15.0 2026-09-24 03:50:49,225 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=139213.33333333334, ans=0.125 2026-09-24 03:51:00,153 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=139280.0, ans=0.125 2026-09-24 03:51:03,146 INFO [train.py:1192] (0/2) Epoch 44, batch 600, loss[loss=0.2972, simple_loss=0.4206, pruned_loss=0.08686, over 24436.00 frames. ], tot_loss[loss=0.2721, simple_loss=0.3856, pruned_loss=0.07933, over 4584814.98 frames. ], batch size: 235, lr: 4.79e-03, grad_scale: 32.0 2026-09-24 03:51:03,440 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.46 vs. limit=22.5 2026-09-24 03:51:03,787 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=139313.33333333334, ans=0.125 2026-09-24 03:51:10,764 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.31 vs. limit=12.0 2026-09-24 03:51:11,057 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:51:11,986 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=139346.66666666666, ans=0.0 2026-09-24 03:51:12,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=139346.66666666666, ans=0.1 2026-09-24 03:51:26,653 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.702e+02 3.328e+02 3.826e+02 4.343e+02 7.892e+02, threshold=7.652e+02, percent-clipped=1.0 2026-09-24 03:51:28,608 INFO [train.py:1192] (0/2) Epoch 44, batch 650, loss[loss=0.2712, simple_loss=0.385, pruned_loss=0.0787, over 24564.00 frames. ], tot_loss[loss=0.2719, simple_loss=0.3852, pruned_loss=0.07928, over 4650246.27 frames. ], batch size: 162, lr: 4.79e-03, grad_scale: 32.0 2026-09-24 03:51:32,784 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=139480.0, ans=0.125 2026-09-24 03:51:35,997 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=139513.33333333334, ans=0.025 2026-09-24 03:51:48,610 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.61 vs. limit=10.0 2026-09-24 03:51:54,230 INFO [train.py:1192] (0/2) Epoch 44, batch 700, loss[loss=0.2567, simple_loss=0.3623, pruned_loss=0.07551, over 24586.00 frames. ], tot_loss[loss=0.2721, simple_loss=0.3856, pruned_loss=0.0793, over 4684398.78 frames. ], batch size: 154, lr: 4.78e-03, grad_scale: 32.0 2026-09-24 03:51:55,575 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=139646.66666666666, ans=0.0 2026-09-24 03:51:56,982 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.max_positive, batch_count=139646.66666666666, ans=0.95 2026-09-24 03:51:58,374 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.02 vs. limit=22.5 2026-09-24 03:52:17,791 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.781e+02 3.272e+02 3.702e+02 4.229e+02 6.362e+02, threshold=7.404e+02, percent-clipped=0.0 2026-09-24 03:52:19,837 INFO [train.py:1192] (0/2) Epoch 44, batch 750, loss[loss=0.2907, simple_loss=0.4004, pruned_loss=0.0905, over 24565.00 frames. ], tot_loss[loss=0.2712, simple_loss=0.3844, pruned_loss=0.07895, over 4711598.41 frames. ], batch size: 170, lr: 4.78e-03, grad_scale: 32.0 2026-09-24 03:52:20,897 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=139813.33333333334, ans=0.2 2026-09-24 03:52:23,180 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=139813.33333333334, ans=0.5 2026-09-24 03:52:32,866 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=139880.0, ans=0.1 2026-09-24 03:52:44,614 INFO [train.py:1192] (0/2) Epoch 44, batch 800, loss[loss=0.2458, simple_loss=0.3533, pruned_loss=0.06914, over 24586.00 frames. ], tot_loss[loss=0.2705, simple_loss=0.3839, pruned_loss=0.07857, over 4736905.66 frames. ], batch size: 137, lr: 4.78e-03, grad_scale: 32.0 2026-09-24 03:52:47,492 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.88 vs. limit=22.5 2026-09-24 03:52:48,121 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=139980.0, ans=0.125 2026-09-24 03:53:04,332 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=140080.0, ans=0.125 2026-09-24 03:53:06,795 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.13 vs. limit=15.0 2026-09-24 03:53:08,110 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.700e+02 3.287e+02 3.727e+02 4.320e+02 6.374e+02, threshold=7.454e+02, percent-clipped=0.0 2026-09-24 03:53:10,435 INFO [train.py:1192] (0/2) Epoch 44, batch 850, loss[loss=0.2798, simple_loss=0.3987, pruned_loss=0.08047, over 24594.00 frames. ], tot_loss[loss=0.27, simple_loss=0.3835, pruned_loss=0.07823, over 4759161.55 frames. ], batch size: 198, lr: 4.77e-03, grad_scale: 32.0 2026-09-24 03:53:11,004 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=140146.66666666666, ans=0.025 2026-09-24 03:53:15,239 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=140180.0, ans=0.0 2026-09-24 03:53:18,062 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=140180.0, ans=0.025 2026-09-24 03:53:20,939 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=140213.33333333334, ans=0.2 2026-09-24 03:53:29,053 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=140246.66666666666, ans=0.0 2026-09-24 03:53:31,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=140280.0, ans=0.1 2026-09-24 03:53:36,114 INFO [train.py:1192] (0/2) Epoch 44, batch 900, loss[loss=0.2369, simple_loss=0.3508, pruned_loss=0.06148, over 24527.00 frames. ], tot_loss[loss=0.2711, simple_loss=0.3845, pruned_loss=0.07883, over 4773504.39 frames. ], batch size: 137, lr: 4.77e-03, grad_scale: 32.0 2026-09-24 03:53:39,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=140313.33333333334, ans=0.0 2026-09-24 03:53:46,077 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.05 vs. limit=22.5 2026-09-24 03:53:46,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=140380.0, ans=0.0 2026-09-24 03:53:51,737 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=140413.33333333334, ans=0.125 2026-09-24 03:53:57,846 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:53:59,151 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.855e+02 3.359e+02 3.856e+02 4.342e+02 7.075e+02, threshold=7.713e+02, percent-clipped=0.0 2026-09-24 03:54:01,407 INFO [train.py:1192] (0/2) Epoch 44, batch 950, loss[loss=0.387, simple_loss=0.4463, pruned_loss=0.1638, over 11011.00 frames. ], tot_loss[loss=0.2714, simple_loss=0.3834, pruned_loss=0.07971, over 4711515.40 frames. ], batch size: 334, lr: 4.77e-03, grad_scale: 32.0 2026-09-24 03:54:03,283 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:54:03,719 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=140480.0, ans=0.1 2026-09-24 03:54:05,643 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-44.pt 2026-09-24 03:54:11,293 INFO [train.py:1192] (0/2) Epoch 45, batch 0, loss[loss=0.2171, simple_loss=0.3391, pruned_loss=0.04755, over 24569.00 frames. ], tot_loss[loss=0.2171, simple_loss=0.3391, pruned_loss=0.04755, over 24569.00 frames. ], batch size: 137, lr: 4.71e-03, grad_scale: 32.0 2026-09-24 03:54:11,293 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 03:54:22,647 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.6036, 3.0742, 4.2604, 2.1013], device='cuda:0') 2026-09-24 03:54:23,062 INFO [train.py:1224] (0/2) Epoch 45, validation: loss=0.175, simple_loss=0.2937, pruned_loss=0.02817, over 2564189.00 frames. 2026-09-24 03:54:23,062 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14082MB 2026-09-24 03:54:25,118 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 03:54:25,556 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=140506.66666666666, ans=0.125 2026-09-24 03:54:28,478 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=140540.0, ans=0.1 2026-09-24 03:54:35,649 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.01 vs. limit=12.0 2026-09-24 03:54:43,234 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=140640.0, ans=0.0 2026-09-24 03:54:45,059 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=140640.0, ans=0.0 2026-09-24 03:54:45,629 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=140640.0, ans=0.125 2026-09-24 03:54:47,233 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=140640.0, ans=0.0 2026-09-24 03:54:48,564 INFO [train.py:1192] (0/2) Epoch 45, batch 50, loss[loss=0.239, simple_loss=0.3411, pruned_loss=0.06845, over 24307.00 frames. ], tot_loss[loss=0.2775, simple_loss=0.3912, pruned_loss=0.0819, over 1074722.50 frames. ], batch size: 125, lr: 4.71e-03, grad_scale: 32.0 2026-09-24 03:54:52,442 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=140673.33333333334, ans=0.0 2026-09-24 03:54:58,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=140740.0, ans=0.125 2026-09-24 03:55:03,129 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=4.12 vs. limit=5.0 2026-09-24 03:55:06,453 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=140773.33333333334, ans=0.2 2026-09-24 03:55:08,144 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.716e+02 3.456e+02 3.884e+02 4.368e+02 7.152e+02, threshold=7.768e+02, percent-clipped=0.0 2026-09-24 03:55:13,709 INFO [train.py:1192] (0/2) Epoch 45, batch 100, loss[loss=0.2665, simple_loss=0.3793, pruned_loss=0.07688, over 24630.00 frames. ], tot_loss[loss=0.281, simple_loss=0.3948, pruned_loss=0.08363, over 1904011.16 frames. ], batch size: 154, lr: 4.71e-03, grad_scale: 32.0 2026-09-24 03:55:25,364 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=140906.66666666666, ans=0.125 2026-09-24 03:55:34,388 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.10 vs. limit=6.0 2026-09-24 03:55:38,940 INFO [train.py:1192] (0/2) Epoch 45, batch 150, loss[loss=0.2227, simple_loss=0.3325, pruned_loss=0.05649, over 24266.00 frames. ], tot_loss[loss=0.2753, simple_loss=0.389, pruned_loss=0.08076, over 2558133.42 frames. ], batch size: 125, lr: 4.71e-03, grad_scale: 32.0 2026-09-24 03:55:47,675 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=141040.0, ans=0.125 2026-09-24 03:55:47,676 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=141040.0, ans=0.125 2026-09-24 03:55:48,235 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=141040.0, ans=0.125 2026-09-24 03:55:50,615 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=141073.33333333334, ans=0.125 2026-09-24 03:55:54,749 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=141106.66666666666, ans=0.2 2026-09-24 03:55:54,928 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.93 vs. limit=15.0 2026-09-24 03:55:56,414 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.82 vs. limit=15.0 2026-09-24 03:55:58,176 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.550e+02 3.376e+02 3.915e+02 4.425e+02 7.469e+02, threshold=7.829e+02, percent-clipped=0.0 2026-09-24 03:56:00,689 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.22 vs. limit=22.5 2026-09-24 03:56:02,485 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=141140.0, ans=0.125 2026-09-24 03:56:04,582 INFO [train.py:1192] (0/2) Epoch 45, batch 200, loss[loss=0.3139, simple_loss=0.4178, pruned_loss=0.105, over 21130.00 frames. ], tot_loss[loss=0.2727, simple_loss=0.3866, pruned_loss=0.0794, over 3054696.59 frames. ], batch size: 333, lr: 4.70e-03, grad_scale: 32.0 2026-09-24 03:56:18,788 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.12 vs. limit=6.0 2026-09-24 03:56:20,492 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.38 vs. limit=15.0 2026-09-24 03:56:21,395 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=141273.33333333334, ans=0.125 2026-09-24 03:56:21,954 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.49 vs. limit=15.0 2026-09-24 03:56:30,216 INFO [train.py:1192] (0/2) Epoch 45, batch 250, loss[loss=0.3122, simple_loss=0.4336, pruned_loss=0.09538, over 24318.00 frames. ], tot_loss[loss=0.2724, simple_loss=0.386, pruned_loss=0.07938, over 3442708.40 frames. ], batch size: 234, lr: 4.70e-03, grad_scale: 32.0 2026-09-24 03:56:35,200 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=141373.33333333334, ans=0.2 2026-09-24 03:56:43,021 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.84 vs. limit=22.5 2026-09-24 03:56:49,000 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.566e+02 3.289e+02 3.843e+02 4.322e+02 7.099e+02, threshold=7.687e+02, percent-clipped=0.0 2026-09-24 03:56:55,383 INFO [train.py:1192] (0/2) Epoch 45, batch 300, loss[loss=0.306, simple_loss=0.422, pruned_loss=0.09499, over 24537.00 frames. ], tot_loss[loss=0.2712, simple_loss=0.3845, pruned_loss=0.07893, over 3749012.79 frames. ], batch size: 204, lr: 4.70e-03, grad_scale: 32.0 2026-09-24 03:57:00,994 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=141540.0, ans=0.0 2026-09-24 03:57:06,236 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=13.30 vs. limit=22.5 2026-09-24 03:57:13,927 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=141606.66666666666, ans=0.125 2026-09-24 03:57:20,304 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=141673.33333333334, ans=0.2 2026-09-24 03:57:20,658 INFO [train.py:1192] (0/2) Epoch 45, batch 350, loss[loss=0.2087, simple_loss=0.331, pruned_loss=0.04316, over 24574.00 frames. ], tot_loss[loss=0.2721, simple_loss=0.3856, pruned_loss=0.07929, over 3990763.72 frames. ], batch size: 137, lr: 4.70e-03, grad_scale: 64.0 2026-09-24 03:57:25,257 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=141706.66666666666, ans=0.0 2026-09-24 03:57:33,834 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=141740.0, ans=0.0 2026-09-24 03:57:40,561 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.533e+02 3.226e+02 3.567e+02 4.126e+02 5.906e+02, threshold=7.134e+02, percent-clipped=0.0 2026-09-24 03:57:41,203 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=141806.66666666666, ans=0.04949747468305833 2026-09-24 03:57:42,122 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=141806.66666666666, ans=0.07 2026-09-24 03:57:44,354 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=141806.66666666666, ans=0.125 2026-09-24 03:57:46,218 INFO [train.py:1192] (0/2) Epoch 45, batch 400, loss[loss=0.2825, simple_loss=0.3942, pruned_loss=0.08538, over 24585.00 frames. ], tot_loss[loss=0.2712, simple_loss=0.3848, pruned_loss=0.07879, over 4174474.45 frames. ], batch size: 170, lr: 4.69e-03, grad_scale: 32.0 2026-09-24 03:57:49,795 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=141840.0, ans=0.125 2026-09-24 03:57:56,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=141906.66666666666, ans=0.0 2026-09-24 03:58:08,348 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=141973.33333333334, ans=0.125 2026-09-24 03:58:08,799 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=141973.33333333334, ans=0.125 2026-09-24 03:58:10,699 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=141973.33333333334, ans=0.025 2026-09-24 03:58:11,168 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=142006.66666666666, ans=0.0 2026-09-24 03:58:11,586 INFO [train.py:1192] (0/2) Epoch 45, batch 450, loss[loss=0.2868, simple_loss=0.3986, pruned_loss=0.08748, over 24628.00 frames. ], tot_loss[loss=0.2709, simple_loss=0.3847, pruned_loss=0.07852, over 4309929.73 frames. ], batch size: 175, lr: 4.69e-03, grad_scale: 32.0 2026-09-24 03:58:15,100 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=142006.66666666666, ans=0.125 2026-09-24 03:58:16,022 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=142006.66666666666, ans=0.025 2026-09-24 03:58:20,976 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.36 vs. limit=15.0 2026-09-24 03:58:28,015 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=142106.66666666666, ans=0.125 2026-09-24 03:58:29,603 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=142106.66666666666, ans=0.2 2026-09-24 03:58:31,283 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=142106.66666666666, ans=0.1 2026-09-24 03:58:31,734 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.402e+02 3.304e+02 3.778e+02 4.404e+02 6.390e+02, threshold=7.556e+02, percent-clipped=0.0 2026-09-24 03:58:32,304 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=142140.0, ans=0.0 2026-09-24 03:58:36,034 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=142140.0, ans=0.125 2026-09-24 03:58:37,488 INFO [train.py:1192] (0/2) Epoch 45, batch 500, loss[loss=0.2923, simple_loss=0.423, pruned_loss=0.08082, over 24498.00 frames. ], tot_loss[loss=0.2702, simple_loss=0.3837, pruned_loss=0.07839, over 4427589.74 frames. ], batch size: 218, lr: 4.69e-03, grad_scale: 32.0 2026-09-24 03:58:53,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=142273.33333333334, ans=0.025 2026-09-24 03:58:56,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=142273.33333333334, ans=0.0 2026-09-24 03:58:58,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=142306.66666666666, ans=0.0 2026-09-24 03:59:03,181 INFO [train.py:1192] (0/2) Epoch 45, batch 550, loss[loss=0.3254, simple_loss=0.4431, pruned_loss=0.1039, over 24276.00 frames. ], tot_loss[loss=0.2713, simple_loss=0.3846, pruned_loss=0.079, over 4517555.41 frames. ], batch size: 257, lr: 4.68e-03, grad_scale: 32.0 2026-09-24 03:59:09,543 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.91 vs. limit=15.0 2026-09-24 03:59:23,080 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.702e+02 3.255e+02 3.555e+02 3.956e+02 6.275e+02, threshold=7.109e+02, percent-clipped=0.0 2026-09-24 03:59:26,650 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=142473.33333333334, ans=0.125 2026-09-24 03:59:29,342 INFO [train.py:1192] (0/2) Epoch 45, batch 600, loss[loss=0.269, simple_loss=0.3988, pruned_loss=0.06961, over 24377.00 frames. ], tot_loss[loss=0.2719, simple_loss=0.3854, pruned_loss=0.07924, over 4584508.71 frames. ], batch size: 234, lr: 4.68e-03, grad_scale: 32.0 2026-09-24 03:59:43,021 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=142573.33333333334, ans=0.125 2026-09-24 03:59:54,107 INFO [train.py:1192] (0/2) Epoch 45, batch 650, loss[loss=0.2567, simple_loss=0.3756, pruned_loss=0.06892, over 24560.00 frames. ], tot_loss[loss=0.2702, simple_loss=0.3841, pruned_loss=0.07819, over 4650066.18 frames. ], batch size: 162, lr: 4.68e-03, grad_scale: 32.0 2026-09-24 04:00:13,790 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.583e+02 3.274e+02 3.633e+02 4.554e+02 7.149e+02, threshold=7.266e+02, percent-clipped=1.0 2026-09-24 04:00:18,982 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=142840.0, ans=0.125 2026-09-24 04:00:19,430 INFO [train.py:1192] (0/2) Epoch 45, batch 700, loss[loss=0.2627, simple_loss=0.3742, pruned_loss=0.07557, over 24556.00 frames. ], tot_loss[loss=0.27, simple_loss=0.3843, pruned_loss=0.07786, over 4682123.02 frames. ], batch size: 154, lr: 4.68e-03, grad_scale: 16.0 2026-09-24 04:00:29,838 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=142906.66666666666, ans=0.0 2026-09-24 04:00:30,781 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=142906.66666666666, ans=0.2 2026-09-24 04:00:39,307 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=142940.0, ans=0.125 2026-09-24 04:00:39,422 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=7.49 vs. limit=12.0 2026-09-24 04:00:41,573 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=142973.33333333334, ans=0.5 2026-09-24 04:00:45,206 INFO [train.py:1192] (0/2) Epoch 45, batch 750, loss[loss=0.2605, simple_loss=0.3837, pruned_loss=0.06869, over 24552.00 frames. ], tot_loss[loss=0.2693, simple_loss=0.3832, pruned_loss=0.07774, over 4709317.01 frames. ], batch size: 170, lr: 4.67e-03, grad_scale: 16.0 2026-09-24 04:00:54,974 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=143073.33333333334, ans=0.125 2026-09-24 04:00:55,119 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.77 vs. limit=22.5 2026-09-24 04:00:55,512 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=143073.33333333334, ans=0.0 2026-09-24 04:00:59,730 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.94 vs. limit=22.5 2026-09-24 04:01:00,837 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=143106.66666666666, ans=0.2 2026-09-24 04:01:05,938 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.482e+02 3.353e+02 3.726e+02 4.298e+02 5.582e+02, threshold=7.452e+02, percent-clipped=0.0 2026-09-24 04:01:10,796 INFO [train.py:1192] (0/2) Epoch 45, batch 800, loss[loss=0.2246, simple_loss=0.3385, pruned_loss=0.0553, over 24551.00 frames. ], tot_loss[loss=0.2692, simple_loss=0.3829, pruned_loss=0.07774, over 4734815.37 frames. ], batch size: 137, lr: 4.67e-03, grad_scale: 32.0 2026-09-24 04:01:16,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=143206.66666666666, ans=0.125 2026-09-24 04:01:24,373 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer_ff2.min_abs, batch_count=143240.0, ans=0.1 2026-09-24 04:01:25,835 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=12.56 vs. limit=15.0 2026-09-24 04:01:28,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=143273.33333333334, ans=0.0 2026-09-24 04:01:35,906 INFO [train.py:1192] (0/2) Epoch 45, batch 850, loss[loss=0.2813, simple_loss=0.396, pruned_loss=0.08327, over 24597.00 frames. ], tot_loss[loss=0.2684, simple_loss=0.3823, pruned_loss=0.0772, over 4758120.80 frames. ], batch size: 198, lr: 4.67e-03, grad_scale: 32.0 2026-09-24 04:01:40,872 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=143373.33333333334, ans=0.1 2026-09-24 04:01:51,805 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.51 vs. limit=12.0 2026-09-24 04:01:56,442 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.722e+02 3.280e+02 3.837e+02 4.261e+02 5.745e+02, threshold=7.675e+02, percent-clipped=0.0 2026-09-24 04:02:01,672 INFO [train.py:1192] (0/2) Epoch 45, batch 900, loss[loss=0.2313, simple_loss=0.3427, pruned_loss=0.06001, over 24556.00 frames. ], tot_loss[loss=0.2695, simple_loss=0.3832, pruned_loss=0.07792, over 4771846.52 frames. ], batch size: 137, lr: 4.67e-03, grad_scale: 32.0 2026-09-24 04:02:06,276 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=143540.0, ans=0.05 2026-09-24 04:02:12,491 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=143573.33333333334, ans=0.025 2026-09-24 04:02:18,191 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=143606.66666666666, ans=0.125 2026-09-24 04:02:26,398 INFO [train.py:1192] (0/2) Epoch 45, batch 950, loss[loss=0.3888, simple_loss=0.4443, pruned_loss=0.1667, over 11399.00 frames. ], tot_loss[loss=0.2703, simple_loss=0.3824, pruned_loss=0.07915, over 4714117.26 frames. ], batch size: 333, lr: 4.66e-03, grad_scale: 32.0 2026-09-24 04:02:30,663 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-45.pt 2026-09-24 04:02:37,994 INFO [train.py:1192] (0/2) Epoch 46, batch 0, loss[loss=0.2404, simple_loss=0.351, pruned_loss=0.06485, over 24571.00 frames. ], tot_loss[loss=0.2404, simple_loss=0.351, pruned_loss=0.06485, over 24571.00 frames. ], batch size: 137, lr: 4.61e-03, grad_scale: 32.0 2026-09-24 04:02:37,994 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 04:02:47,484 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.4903, 2.8607, 3.0880, 2.4462, 3.2425, 2.9436, 3.1633, 2.4901], device='cuda:0') 2026-09-24 04:02:49,714 INFO [train.py:1224] (0/2) Epoch 46, validation: loss=0.1736, simple_loss=0.2924, pruned_loss=0.02741, over 2564189.00 frames. 2026-09-24 04:02:49,714 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 04:02:52,106 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=143700.0, ans=0.125 2026-09-24 04:02:56,136 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.46 vs. limit=15.0 2026-09-24 04:03:02,811 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.74 vs. limit=15.0 2026-09-24 04:03:04,099 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=143766.66666666666, ans=0.125 2026-09-24 04:03:05,941 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.612e+02 3.424e+02 3.959e+02 4.444e+02 9.934e+02, threshold=7.917e+02, percent-clipped=1.0 2026-09-24 04:03:13,542 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=143833.33333333334, ans=0.0 2026-09-24 04:03:15,316 INFO [train.py:1192] (0/2) Epoch 46, batch 50, loss[loss=0.2298, simple_loss=0.3379, pruned_loss=0.06091, over 24228.00 frames. ], tot_loss[loss=0.2768, simple_loss=0.3893, pruned_loss=0.08213, over 1075570.12 frames. ], batch size: 125, lr: 4.61e-03, grad_scale: 32.0 2026-09-24 04:03:30,273 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=7.40 vs. limit=15.0 2026-09-24 04:03:36,354 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.68 vs. limit=15.0 2026-09-24 04:03:40,814 INFO [train.py:1192] (0/2) Epoch 46, batch 100, loss[loss=0.2516, simple_loss=0.3621, pruned_loss=0.07053, over 24612.00 frames. ], tot_loss[loss=0.2804, simple_loss=0.3936, pruned_loss=0.08363, over 1904316.33 frames. ], batch size: 154, lr: 4.61e-03, grad_scale: 32.0 2026-09-24 04:03:45,513 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=144066.66666666666, ans=0.125 2026-09-24 04:03:46,055 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=144066.66666666666, ans=0.125 2026-09-24 04:03:53,896 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=144100.0, ans=0.125 2026-09-24 04:03:57,492 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.558e+02 3.314e+02 3.738e+02 4.223e+02 7.597e+02, threshold=7.476e+02, percent-clipped=0.0 2026-09-24 04:03:59,165 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=144133.33333333334, ans=0.035 2026-09-24 04:04:06,755 INFO [train.py:1192] (0/2) Epoch 46, batch 150, loss[loss=0.2221, simple_loss=0.3311, pruned_loss=0.05649, over 24266.00 frames. ], tot_loss[loss=0.275, simple_loss=0.3885, pruned_loss=0.0808, over 2558023.71 frames. ], batch size: 125, lr: 4.60e-03, grad_scale: 32.0 2026-09-24 04:04:16,572 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=144266.66666666666, ans=0.125 2026-09-24 04:04:29,876 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=144333.33333333334, ans=0.5 2026-09-24 04:04:30,922 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=144333.33333333334, ans=0.0 2026-09-24 04:04:32,396 INFO [train.py:1192] (0/2) Epoch 46, batch 200, loss[loss=0.3061, simple_loss=0.4087, pruned_loss=0.1017, over 21225.00 frames. ], tot_loss[loss=0.274, simple_loss=0.3874, pruned_loss=0.08027, over 3055949.66 frames. ], batch size: 333, lr: 4.60e-03, grad_scale: 32.0 2026-09-24 04:04:38,318 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=144400.0, ans=0.125 2026-09-24 04:04:44,585 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=144433.33333333334, ans=0.125 2026-09-24 04:04:49,082 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.682e+02 3.354e+02 3.903e+02 4.472e+02 8.657e+02, threshold=7.805e+02, percent-clipped=1.0 2026-09-24 04:04:56,022 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.89 vs. limit=15.0 2026-09-24 04:04:57,441 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=144500.0, ans=0.125 2026-09-24 04:04:58,389 INFO [train.py:1192] (0/2) Epoch 46, batch 250, loss[loss=0.3165, simple_loss=0.4355, pruned_loss=0.09877, over 24311.00 frames. ], tot_loss[loss=0.2738, simple_loss=0.387, pruned_loss=0.08036, over 3446244.71 frames. ], batch size: 234, lr: 4.60e-03, grad_scale: 32.0 2026-09-24 04:05:00,921 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.91 vs. limit=15.0 2026-09-24 04:05:16,704 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=144633.33333333334, ans=0.125 2026-09-24 04:05:23,978 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=144700.0, ans=0.95 2026-09-24 04:05:24,421 INFO [train.py:1192] (0/2) Epoch 46, batch 300, loss[loss=0.2882, simple_loss=0.4112, pruned_loss=0.08254, over 24532.00 frames. ], tot_loss[loss=0.2725, simple_loss=0.3856, pruned_loss=0.0797, over 3752019.99 frames. ], batch size: 204, lr: 4.60e-03, grad_scale: 32.0 2026-09-24 04:05:30,767 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=144733.33333333334, ans=0.0 2026-09-24 04:05:34,144 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.15 vs. limit=10.0 2026-09-24 04:05:37,964 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.56 vs. limit=22.5 2026-09-24 04:05:40,634 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.777e+02 3.310e+02 3.640e+02 4.094e+02 5.743e+02, threshold=7.281e+02, percent-clipped=0.0 2026-09-24 04:05:42,165 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=144800.0, ans=0.5 2026-09-24 04:05:44,604 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=144833.33333333334, ans=0.125 2026-09-24 04:05:49,883 INFO [train.py:1192] (0/2) Epoch 46, batch 350, loss[loss=0.2433, simple_loss=0.3485, pruned_loss=0.06899, over 24577.00 frames. ], tot_loss[loss=0.2731, simple_loss=0.3863, pruned_loss=0.08001, over 3995211.61 frames. ], batch size: 137, lr: 4.59e-03, grad_scale: 32.0 2026-09-24 04:05:51,029 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=144866.66666666666, ans=0.0 2026-09-24 04:05:52,369 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=144866.66666666666, ans=0.2 2026-09-24 04:05:54,841 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=144900.0, ans=0.0 2026-09-24 04:05:55,420 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=144900.0, ans=0.0 2026-09-24 04:06:02,740 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten.whitening_limit, batch_count=144933.33333333334, ans=15.0 2026-09-24 04:06:09,823 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=145000.0, ans=0.1 2026-09-24 04:06:15,695 INFO [train.py:1192] (0/2) Epoch 46, batch 400, loss[loss=0.2999, simple_loss=0.4081, pruned_loss=0.09583, over 24559.00 frames. ], tot_loss[loss=0.2718, simple_loss=0.3853, pruned_loss=0.07919, over 4178001.73 frames. ], batch size: 170, lr: 4.59e-03, grad_scale: 32.0 2026-09-24 04:06:15,784 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=145033.33333333334, ans=0.125 2026-09-24 04:06:17,308 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=145033.33333333334, ans=0.125 2026-09-24 04:06:18,823 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=145033.33333333334, ans=0.125 2026-09-24 04:06:31,864 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.676e+02 3.294e+02 3.676e+02 4.081e+02 6.453e+02, threshold=7.352e+02, percent-clipped=0.0 2026-09-24 04:06:33,046 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=145133.33333333334, ans=0.125 2026-09-24 04:06:36,850 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=145166.66666666666, ans=0.0 2026-09-24 04:06:40,692 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=145166.66666666666, ans=0.125 2026-09-24 04:06:41,589 INFO [train.py:1192] (0/2) Epoch 46, batch 450, loss[loss=0.2784, simple_loss=0.3982, pruned_loss=0.07929, over 24630.00 frames. ], tot_loss[loss=0.2727, simple_loss=0.3858, pruned_loss=0.07983, over 4311905.14 frames. ], batch size: 175, lr: 4.59e-03, grad_scale: 32.0 2026-09-24 04:06:50,574 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=145266.66666666666, ans=0.1 2026-09-24 04:06:51,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=145266.66666666666, ans=0.125 2026-09-24 04:07:01,211 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=145333.33333333334, ans=0.04949747468305833 2026-09-24 04:07:01,732 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.01 vs. limit=15.0 2026-09-24 04:07:04,222 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=145333.33333333334, ans=0.1 2026-09-24 04:07:06,922 INFO [train.py:1192] (0/2) Epoch 46, batch 500, loss[loss=0.2982, simple_loss=0.4195, pruned_loss=0.08839, over 24515.00 frames. ], tot_loss[loss=0.2709, simple_loss=0.3838, pruned_loss=0.07895, over 4429175.13 frames. ], batch size: 218, lr: 4.59e-03, grad_scale: 32.0 2026-09-24 04:07:09,202 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=11.78 vs. limit=15.0 2026-09-24 04:07:15,511 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=145400.0, ans=0.1 2026-09-24 04:07:16,448 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=145400.0, ans=0.2 2026-09-24 04:07:23,927 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.597e+02 3.252e+02 3.691e+02 4.584e+02 6.780e+02, threshold=7.382e+02, percent-clipped=0.0 2026-09-24 04:07:32,673 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=145533.33333333334, ans=0.2 2026-09-24 04:07:33,064 INFO [train.py:1192] (0/2) Epoch 46, batch 550, loss[loss=0.2761, simple_loss=0.3994, pruned_loss=0.07635, over 24251.00 frames. ], tot_loss[loss=0.2708, simple_loss=0.3841, pruned_loss=0.07879, over 4518625.43 frames. ], batch size: 257, lr: 4.58e-03, grad_scale: 32.0 2026-09-24 04:07:35,322 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=145533.33333333334, ans=0.125 2026-09-24 04:07:47,627 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=145600.0, ans=0.125 2026-09-24 04:07:48,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=145633.33333333334, ans=0.0 2026-09-24 04:07:51,659 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=145633.33333333334, ans=0.1 2026-09-24 04:07:58,899 INFO [train.py:1192] (0/2) Epoch 46, batch 600, loss[loss=0.2796, simple_loss=0.4076, pruned_loss=0.07584, over 24390.00 frames. ], tot_loss[loss=0.2711, simple_loss=0.3847, pruned_loss=0.07872, over 4585046.73 frames. ], batch size: 235, lr: 4.58e-03, grad_scale: 32.0 2026-09-24 04:08:05,682 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=145733.33333333334, ans=0.09899494936611666 2026-09-24 04:08:15,072 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.576e+02 3.178e+02 3.584e+02 4.061e+02 7.593e+02, threshold=7.168e+02, percent-clipped=1.0 2026-09-24 04:08:20,420 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=145833.33333333334, ans=0.5 2026-09-24 04:08:24,004 INFO [train.py:1192] (0/2) Epoch 46, batch 650, loss[loss=0.2709, simple_loss=0.3837, pruned_loss=0.07906, over 24570.00 frames. ], tot_loss[loss=0.2698, simple_loss=0.3837, pruned_loss=0.07796, over 4650204.30 frames. ], batch size: 162, lr: 4.58e-03, grad_scale: 32.0 2026-09-24 04:08:29,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=145900.0, ans=0.1 2026-09-24 04:08:34,322 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=145933.33333333334, ans=0.04949747468305833 2026-09-24 04:08:46,199 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.72 vs. limit=22.5 2026-09-24 04:08:50,129 INFO [train.py:1192] (0/2) Epoch 46, batch 700, loss[loss=0.2593, simple_loss=0.3671, pruned_loss=0.07577, over 24550.00 frames. ], tot_loss[loss=0.2707, simple_loss=0.3847, pruned_loss=0.07835, over 4684146.94 frames. ], batch size: 154, lr: 4.58e-03, grad_scale: 32.0 2026-09-24 04:09:02,102 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=146100.0, ans=0.125 2026-09-24 04:09:05,602 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=146133.33333333334, ans=0.125 2026-09-24 04:09:06,411 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.700e+02 3.386e+02 3.749e+02 4.522e+02 6.408e+02, threshold=7.499e+02, percent-clipped=0.0 2026-09-24 04:09:09,540 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=146133.33333333334, ans=0.125 2026-09-24 04:09:10,072 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=146166.66666666666, ans=0.125 2026-09-24 04:09:11,167 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=146166.66666666666, ans=0.125 2026-09-24 04:09:14,473 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=146166.66666666666, ans=0.0 2026-09-24 04:09:15,466 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=146200.0, ans=0.125 2026-09-24 04:09:15,855 INFO [train.py:1192] (0/2) Epoch 46, batch 750, loss[loss=0.2763, simple_loss=0.391, pruned_loss=0.08079, over 24566.00 frames. ], tot_loss[loss=0.2698, simple_loss=0.3835, pruned_loss=0.07804, over 4711244.94 frames. ], batch size: 170, lr: 4.57e-03, grad_scale: 32.0 2026-09-24 04:09:20,984 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.11 vs. limit=6.0 2026-09-24 04:09:28,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=146266.66666666666, ans=0.125 2026-09-24 04:09:35,214 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2.whitening_limit, batch_count=146300.0, ans=15.0 2026-09-24 04:09:36,453 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=146333.33333333334, ans=0.1 2026-09-24 04:09:37,537 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.whiten.whitening_limit, batch_count=146333.33333333334, ans=12.0 2026-09-24 04:09:41,938 INFO [train.py:1192] (0/2) Epoch 46, batch 800, loss[loss=0.2529, simple_loss=0.3647, pruned_loss=0.07055, over 24535.00 frames. ], tot_loss[loss=0.2688, simple_loss=0.3826, pruned_loss=0.07746, over 4740321.16 frames. ], batch size: 137, lr: 4.57e-03, grad_scale: 32.0 2026-09-24 04:09:50,952 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.12 vs. limit=15.0 2026-09-24 04:09:52,341 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.36 vs. limit=22.5 2026-09-24 04:09:53,189 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=146433.33333333334, ans=0.2 2026-09-24 04:09:55,628 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.49 vs. limit=12.0 2026-09-24 04:09:58,205 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.506e+02 3.341e+02 3.701e+02 4.189e+02 6.774e+02, threshold=7.403e+02, percent-clipped=0.0 2026-09-24 04:10:03,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=146500.0, ans=0.025 2026-09-24 04:10:04,055 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.78 vs. limit=22.5 2026-09-24 04:10:06,081 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.68 vs. limit=15.0 2026-09-24 04:10:07,166 INFO [train.py:1192] (0/2) Epoch 46, batch 850, loss[loss=0.3061, simple_loss=0.4201, pruned_loss=0.0961, over 24530.00 frames. ], tot_loss[loss=0.2683, simple_loss=0.3822, pruned_loss=0.07715, over 4763207.13 frames. ], batch size: 204, lr: 4.57e-03, grad_scale: 32.0 2026-09-24 04:10:27,158 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-44000.pt 2026-09-24 04:10:28,879 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=146666.66666666666, ans=0.1 2026-09-24 04:10:31,888 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=146666.66666666666, ans=0.1 2026-09-24 04:10:32,767 INFO [train.py:1192] (0/2) Epoch 46, batch 900, loss[loss=0.2517, simple_loss=0.3654, pruned_loss=0.06903, over 24575.00 frames. ], tot_loss[loss=0.2689, simple_loss=0.3829, pruned_loss=0.07745, over 4777026.03 frames. ], batch size: 137, lr: 4.57e-03, grad_scale: 32.0 2026-09-24 04:10:33,089 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=4.08 vs. limit=12.0 2026-09-24 04:10:34,285 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=146700.0, ans=0.0 2026-09-24 04:10:34,753 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=146700.0, ans=0.125 2026-09-24 04:10:39,375 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=146733.33333333334, ans=0.09899494936611666 2026-09-24 04:10:49,531 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.635e+02 3.272e+02 3.689e+02 4.102e+02 7.758e+02, threshold=7.378e+02, percent-clipped=1.0 2026-09-24 04:10:50,371 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=6.72 vs. limit=15.0 2026-09-24 04:10:58,350 INFO [train.py:1192] (0/2) Epoch 46, batch 950, loss[loss=0.346, simple_loss=0.4218, pruned_loss=0.1351, over 11480.00 frames. ], tot_loss[loss=0.2695, simple_loss=0.3818, pruned_loss=0.07863, over 4715395.92 frames. ], batch size: 333, lr: 4.56e-03, grad_scale: 32.0 2026-09-24 04:11:02,569 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-46.pt 2026-09-24 04:11:09,911 INFO [train.py:1192] (0/2) Epoch 47, batch 0, loss[loss=0.2197, simple_loss=0.3397, pruned_loss=0.0498, over 24565.00 frames. ], tot_loss[loss=0.2197, simple_loss=0.3397, pruned_loss=0.0498, over 24565.00 frames. ], batch size: 137, lr: 4.51e-03, grad_scale: 32.0 2026-09-24 04:11:09,912 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 04:11:12,214 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([1.9622, 1.6541, 2.6835, 1.6921], device='cuda:0') 2026-09-24 04:11:18,801 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.1216, 2.4835, 2.8527, 2.1414, 3.0339, 2.6521, 2.8447, 2.2069], device='cuda:0') 2026-09-24 04:11:21,605 INFO [train.py:1224] (0/2) Epoch 47, validation: loss=0.1733, simple_loss=0.2922, pruned_loss=0.02718, over 2564189.00 frames. 2026-09-24 04:11:21,605 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 04:11:21,869 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.52 vs. limit=15.0 2026-09-24 04:11:27,597 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=146926.66666666666, ans=0.015 2026-09-24 04:11:42,912 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=147026.66666666666, ans=0.125 2026-09-24 04:11:42,971 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=147026.66666666666, ans=0.1 2026-09-24 04:11:46,841 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=147060.0, ans=0.125 2026-09-24 04:11:47,142 INFO [train.py:1192] (0/2) Epoch 47, batch 50, loss[loss=0.211, simple_loss=0.324, pruned_loss=0.04898, over 24263.00 frames. ], tot_loss[loss=0.2761, simple_loss=0.3886, pruned_loss=0.08183, over 1076768.36 frames. ], batch size: 125, lr: 4.51e-03, grad_scale: 32.0 2026-09-24 04:11:59,574 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.653e+02 3.466e+02 3.856e+02 4.341e+02 7.987e+02, threshold=7.712e+02, percent-clipped=1.0 2026-09-24 04:12:01,694 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=147126.66666666666, ans=0.125 2026-09-24 04:12:08,946 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=147193.33333333334, ans=0.125 2026-09-24 04:12:12,481 INFO [train.py:1192] (0/2) Epoch 47, batch 100, loss[loss=0.2464, simple_loss=0.3639, pruned_loss=0.0645, over 24613.00 frames. ], tot_loss[loss=0.2782, simple_loss=0.3923, pruned_loss=0.08208, over 1904804.98 frames. ], batch size: 154, lr: 4.51e-03, grad_scale: 32.0 2026-09-24 04:12:13,441 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=147226.66666666666, ans=0.125 2026-09-24 04:12:14,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=147226.66666666666, ans=0.125 2026-09-24 04:12:23,620 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:12:32,258 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.83 vs. limit=15.0 2026-09-24 04:12:37,598 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.01 vs. limit=22.5 2026-09-24 04:12:37,840 INFO [train.py:1192] (0/2) Epoch 47, batch 150, loss[loss=0.2231, simple_loss=0.3353, pruned_loss=0.05541, over 24252.00 frames. ], tot_loss[loss=0.2731, simple_loss=0.3873, pruned_loss=0.07947, over 2559166.95 frames. ], batch size: 125, lr: 4.51e-03, grad_scale: 32.0 2026-09-24 04:12:44,102 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=147426.66666666666, ans=0.0 2026-09-24 04:12:48,226 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=147460.0, ans=0.0 2026-09-24 04:12:50,036 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.765e+02 3.439e+02 3.895e+02 4.306e+02 6.653e+02, threshold=7.789e+02, percent-clipped=0.0 2026-09-24 04:12:52,429 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=147493.33333333334, ans=0.125 2026-09-24 04:13:03,211 INFO [train.py:1192] (0/2) Epoch 47, batch 200, loss[loss=0.3254, simple_loss=0.4182, pruned_loss=0.1163, over 21123.00 frames. ], tot_loss[loss=0.2705, simple_loss=0.3848, pruned_loss=0.07815, over 3056011.79 frames. ], batch size: 333, lr: 4.50e-03, grad_scale: 32.0 2026-09-24 04:13:11,985 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=147593.33333333334, ans=0.2 2026-09-24 04:13:12,146 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.21 vs. limit=15.0 2026-09-24 04:13:16,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=147626.66666666666, ans=0.0 2026-09-24 04:13:17,587 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.63 vs. limit=15.0 2026-09-24 04:13:28,086 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=147726.66666666666, ans=0.125 2026-09-24 04:13:28,518 INFO [train.py:1192] (0/2) Epoch 47, batch 250, loss[loss=0.3262, simple_loss=0.4389, pruned_loss=0.1067, over 24300.00 frames. ], tot_loss[loss=0.2698, simple_loss=0.3842, pruned_loss=0.07774, over 3445589.97 frames. ], batch size: 234, lr: 4.50e-03, grad_scale: 32.0 2026-09-24 04:13:29,071 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=147726.66666666666, ans=0.125 2026-09-24 04:13:29,092 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=147726.66666666666, ans=0.025 2026-09-24 04:13:40,674 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.617e+02 3.262e+02 3.790e+02 4.435e+02 6.838e+02, threshold=7.580e+02, percent-clipped=0.0 2026-09-24 04:13:53,304 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=147893.33333333334, ans=0.0 2026-09-24 04:13:53,711 INFO [train.py:1192] (0/2) Epoch 47, batch 300, loss[loss=0.3264, simple_loss=0.4345, pruned_loss=0.1091, over 24544.00 frames. ], tot_loss[loss=0.2688, simple_loss=0.3829, pruned_loss=0.0774, over 3749946.86 frames. ], batch size: 204, lr: 4.50e-03, grad_scale: 32.0 2026-09-24 04:13:55,003 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=147893.33333333334, ans=0.0 2026-09-24 04:14:00,460 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.min_positive, batch_count=147926.66666666666, ans=0.05 2026-09-24 04:14:02,390 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.22 vs. limit=22.5 2026-09-24 04:14:04,715 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=147960.0, ans=0.125 2026-09-24 04:14:05,946 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:14:16,699 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=148026.66666666666, ans=0.125 2026-09-24 04:14:19,257 INFO [train.py:1192] (0/2) Epoch 47, batch 350, loss[loss=0.2434, simple_loss=0.3536, pruned_loss=0.06658, over 24585.00 frames. ], tot_loss[loss=0.2703, simple_loss=0.3841, pruned_loss=0.07824, over 3993246.95 frames. ], batch size: 137, lr: 4.50e-03, grad_scale: 32.0 2026-09-24 04:14:25,183 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=148093.33333333334, ans=0.1 2026-09-24 04:14:26,641 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=148093.33333333334, ans=0.125 2026-09-24 04:14:31,313 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.466e+02 3.317e+02 3.744e+02 4.508e+02 7.087e+02, threshold=7.487e+02, percent-clipped=0.0 2026-09-24 04:14:35,930 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=148160.0, ans=6.0 2026-09-24 04:14:39,113 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=148193.33333333334, ans=0.025 2026-09-24 04:14:44,388 INFO [train.py:1192] (0/2) Epoch 47, batch 400, loss[loss=0.2621, simple_loss=0.3798, pruned_loss=0.07217, over 24568.00 frames. ], tot_loss[loss=0.2691, simple_loss=0.3832, pruned_loss=0.07751, over 4176980.17 frames. ], batch size: 170, lr: 4.49e-03, grad_scale: 32.0 2026-09-24 04:14:50,878 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.50 vs. limit=22.5 2026-09-24 04:14:54,268 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:14:58,017 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=148293.33333333334, ans=0.125 2026-09-24 04:15:09,636 INFO [train.py:1192] (0/2) Epoch 47, batch 450, loss[loss=0.2679, simple_loss=0.3888, pruned_loss=0.07356, over 24622.00 frames. ], tot_loss[loss=0.2686, simple_loss=0.383, pruned_loss=0.07711, over 4310127.06 frames. ], batch size: 175, lr: 4.49e-03, grad_scale: 32.0 2026-09-24 04:15:09,732 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=148393.33333333334, ans=0.0 2026-09-24 04:15:11,182 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=148393.33333333334, ans=0.125 2026-09-24 04:15:11,331 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.66 vs. limit=15.0 2026-09-24 04:15:13,298 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=148393.33333333334, ans=0.0 2026-09-24 04:15:21,773 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.549e+02 3.364e+02 3.852e+02 4.437e+02 6.702e+02, threshold=7.704e+02, percent-clipped=0.0 2026-09-24 04:15:25,724 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=148493.33333333334, ans=0.0 2026-09-24 04:15:27,040 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=148493.33333333334, ans=0.125 2026-09-24 04:15:30,465 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=148526.66666666666, ans=0.125 2026-09-24 04:15:35,027 INFO [train.py:1192] (0/2) Epoch 47, batch 500, loss[loss=0.2797, simple_loss=0.4052, pruned_loss=0.07713, over 24525.00 frames. ], tot_loss[loss=0.2682, simple_loss=0.3821, pruned_loss=0.07715, over 4427900.45 frames. ], batch size: 218, lr: 4.49e-03, grad_scale: 32.0 2026-09-24 04:15:40,434 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=148593.33333333334, ans=0.95 2026-09-24 04:15:40,445 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=148593.33333333334, ans=0.0 2026-09-24 04:15:45,222 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.46 vs. limit=6.0 2026-09-24 04:15:46,356 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=148626.66666666666, ans=0.0 2026-09-24 04:15:49,369 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=148626.66666666666, ans=0.0 2026-09-24 04:15:53,983 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=148660.0, ans=0.0 2026-09-24 04:15:58,877 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=148693.33333333334, ans=0.0 2026-09-24 04:15:59,396 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=148693.33333333334, ans=0.125 2026-09-24 04:16:00,748 INFO [train.py:1192] (0/2) Epoch 47, batch 550, loss[loss=0.3074, simple_loss=0.4242, pruned_loss=0.0953, over 24284.00 frames. ], tot_loss[loss=0.2689, simple_loss=0.3828, pruned_loss=0.07751, over 4517445.94 frames. ], batch size: 257, lr: 4.49e-03, grad_scale: 32.0 2026-09-24 04:16:08,418 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=148760.0, ans=0.2 2026-09-24 04:16:08,842 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=148760.0, ans=0.125 2026-09-24 04:16:12,039 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=148793.33333333334, ans=0.1 2026-09-24 04:16:12,774 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.497e+02 3.250e+02 3.585e+02 3.951e+02 5.774e+02, threshold=7.171e+02, percent-clipped=0.0 2026-09-24 04:16:16,260 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=148826.66666666666, ans=0.0 2026-09-24 04:16:17,733 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=148826.66666666666, ans=0.0 2026-09-24 04:16:23,313 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=148860.0, ans=0.125 2026-09-24 04:16:25,617 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=148893.33333333334, ans=0.125 2026-09-24 04:16:25,891 INFO [train.py:1192] (0/2) Epoch 47, batch 600, loss[loss=0.2638, simple_loss=0.3936, pruned_loss=0.06706, over 24440.00 frames. ], tot_loss[loss=0.2687, simple_loss=0.383, pruned_loss=0.0772, over 4584612.73 frames. ], batch size: 235, lr: 4.48e-03, grad_scale: 32.0 2026-09-24 04:16:28,731 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=148893.33333333334, ans=0.1 2026-09-24 04:16:30,769 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=148926.66666666666, ans=0.025 2026-09-24 04:16:35,235 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=4.95 vs. limit=15.0 2026-09-24 04:16:38,101 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=148960.0, ans=0.0 2026-09-24 04:16:40,813 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=148960.0, ans=0.125 2026-09-24 04:16:50,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=149026.66666666666, ans=0.125 2026-09-24 04:16:51,792 INFO [train.py:1192] (0/2) Epoch 47, batch 650, loss[loss=0.2577, simple_loss=0.3779, pruned_loss=0.06879, over 24562.00 frames. ], tot_loss[loss=0.2686, simple_loss=0.3828, pruned_loss=0.0772, over 4650113.03 frames. ], batch size: 162, lr: 4.48e-03, grad_scale: 32.0 2026-09-24 04:17:03,999 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.790e+02 3.432e+02 3.795e+02 4.283e+02 7.994e+02, threshold=7.590e+02, percent-clipped=1.0 2026-09-24 04:17:06,224 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=149126.66666666666, ans=0.025 2026-09-24 04:17:06,764 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=149126.66666666666, ans=0.025 2026-09-24 04:17:11,166 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=149160.0, ans=0.025 2026-09-24 04:17:12,784 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=149193.33333333334, ans=0.125 2026-09-24 04:17:18,159 INFO [train.py:1192] (0/2) Epoch 47, batch 700, loss[loss=0.2656, simple_loss=0.3708, pruned_loss=0.08016, over 24562.00 frames. ], tot_loss[loss=0.2687, simple_loss=0.3833, pruned_loss=0.07707, over 4681434.99 frames. ], batch size: 154, lr: 4.48e-03, grad_scale: 32.0 2026-09-24 04:17:35,142 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=149326.66666666666, ans=0.0 2026-09-24 04:17:39,956 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=149360.0, ans=0.125 2026-09-24 04:17:41,148 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.86 vs. limit=15.0 2026-09-24 04:17:42,287 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=149360.0, ans=0.2 2026-09-24 04:17:43,631 INFO [train.py:1192] (0/2) Epoch 47, batch 750, loss[loss=0.2877, simple_loss=0.3999, pruned_loss=0.08774, over 24536.00 frames. ], tot_loss[loss=0.2671, simple_loss=0.3816, pruned_loss=0.0763, over 4709001.33 frames. ], batch size: 170, lr: 4.48e-03, grad_scale: 32.0 2026-09-24 04:17:43,731 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=149393.33333333334, ans=0.0 2026-09-24 04:17:46,857 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=149393.33333333334, ans=0.1 2026-09-24 04:17:56,196 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.544e+02 3.424e+02 3.890e+02 4.427e+02 7.116e+02, threshold=7.781e+02, percent-clipped=0.0 2026-09-24 04:18:04,492 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=149526.66666666666, ans=0.0 2026-09-24 04:18:06,803 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.28 vs. limit=12.0 2026-09-24 04:18:09,201 INFO [train.py:1192] (0/2) Epoch 47, batch 800, loss[loss=0.239, simple_loss=0.3491, pruned_loss=0.06443, over 24558.00 frames. ], tot_loss[loss=0.2668, simple_loss=0.3813, pruned_loss=0.07612, over 4739136.74 frames. ], batch size: 137, lr: 4.47e-03, grad_scale: 64.0 2026-09-24 04:18:14,949 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=149593.33333333334, ans=0.125 2026-09-24 04:18:19,586 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.93 vs. limit=12.0 2026-09-24 04:18:20,608 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=149626.66666666666, ans=0.1 2026-09-24 04:18:32,946 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.85 vs. limit=10.0 2026-09-24 04:18:35,065 INFO [train.py:1192] (0/2) Epoch 47, batch 850, loss[loss=0.2838, simple_loss=0.405, pruned_loss=0.08133, over 24526.00 frames. ], tot_loss[loss=0.2669, simple_loss=0.3813, pruned_loss=0.07623, over 4760680.33 frames. ], batch size: 204, lr: 4.47e-03, grad_scale: 64.0 2026-09-24 04:18:36,633 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=149726.66666666666, ans=0.0 2026-09-24 04:18:39,040 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=149726.66666666666, ans=0.2 2026-09-24 04:18:45,934 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=149793.33333333334, ans=0.125 2026-09-24 04:18:47,893 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.652e+02 3.362e+02 3.668e+02 4.158e+02 7.639e+02, threshold=7.335e+02, percent-clipped=0.0 2026-09-24 04:18:48,510 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.10 vs. limit=15.0 2026-09-24 04:18:51,458 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=149826.66666666666, ans=0.125 2026-09-24 04:18:53,162 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=149826.66666666666, ans=0.125 2026-09-24 04:19:01,472 INFO [train.py:1192] (0/2) Epoch 47, batch 900, loss[loss=0.2149, simple_loss=0.3385, pruned_loss=0.0456, over 24550.00 frames. ], tot_loss[loss=0.268, simple_loss=0.3824, pruned_loss=0.07683, over 4774026.88 frames. ], batch size: 137, lr: 4.47e-03, grad_scale: 64.0 2026-09-24 04:19:11,989 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:19:12,514 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=149960.0, ans=0.125 2026-09-24 04:19:15,203 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=17.20 vs. limit=22.5 2026-09-24 04:19:21,094 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=149993.33333333334, ans=0.2 2026-09-24 04:19:22,357 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=150026.66666666666, ans=0.125 2026-09-24 04:19:23,464 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.59 vs. limit=12.0 2026-09-24 04:19:26,901 INFO [train.py:1192] (0/2) Epoch 47, batch 950, loss[loss=0.3488, simple_loss=0.4148, pruned_loss=0.1414, over 11342.00 frames. ], tot_loss[loss=0.2683, simple_loss=0.3811, pruned_loss=0.07777, over 4710338.07 frames. ], batch size: 333, lr: 4.47e-03, grad_scale: 32.0 2026-09-24 04:19:31,277 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-47.pt 2026-09-24 04:19:38,545 INFO [train.py:1192] (0/2) Epoch 48, batch 0, loss[loss=0.218, simple_loss=0.3403, pruned_loss=0.04784, over 24630.00 frames. ], tot_loss[loss=0.218, simple_loss=0.3403, pruned_loss=0.04784, over 24630.00 frames. ], batch size: 137, lr: 4.42e-03, grad_scale: 32.0 2026-09-24 04:19:38,546 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 04:19:50,025 INFO [train.py:1224] (0/2) Epoch 48, validation: loss=0.1731, simple_loss=0.2919, pruned_loss=0.02714, over 2564189.00 frames. 2026-09-24 04:19:50,025 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 04:19:58,771 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.755e+02 3.407e+02 3.828e+02 4.280e+02 6.595e+02, threshold=7.656e+02, percent-clipped=0.0 2026-09-24 04:20:12,548 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.86 vs. limit=15.0 2026-09-24 04:20:13,536 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=150220.0, ans=0.0 2026-09-24 04:20:15,532 INFO [train.py:1192] (0/2) Epoch 48, batch 50, loss[loss=0.2282, simple_loss=0.3305, pruned_loss=0.063, over 24312.00 frames. ], tot_loss[loss=0.2713, simple_loss=0.3851, pruned_loss=0.07877, over 1075804.61 frames. ], batch size: 125, lr: 4.42e-03, grad_scale: 32.0 2026-09-24 04:20:26,351 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=150320.0, ans=0.0 2026-09-24 04:20:31,480 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=150353.33333333334, ans=0.125 2026-09-24 04:20:37,371 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1.whitening_limit, batch_count=150386.66666666666, ans=10.0 2026-09-24 04:20:39,726 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=150386.66666666666, ans=0.125 2026-09-24 04:20:41,304 INFO [train.py:1192] (0/2) Epoch 48, batch 100, loss[loss=0.2463, simple_loss=0.3658, pruned_loss=0.06337, over 24602.00 frames. ], tot_loss[loss=0.2757, simple_loss=0.3905, pruned_loss=0.08039, over 1904066.58 frames. ], batch size: 154, lr: 4.41e-03, grad_scale: 32.0 2026-09-24 04:20:49,515 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.639e+02 3.321e+02 3.619e+02 4.205e+02 5.847e+02, threshold=7.239e+02, percent-clipped=0.0 2026-09-24 04:20:56,327 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.20 vs. limit=15.0 2026-09-24 04:21:02,094 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=150553.33333333334, ans=0.125 2026-09-24 04:21:03,520 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=150553.33333333334, ans=0.025 2026-09-24 04:21:04,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=150553.33333333334, ans=0.125 2026-09-24 04:21:05,083 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=150553.33333333334, ans=0.125 2026-09-24 04:21:06,297 INFO [train.py:1192] (0/2) Epoch 48, batch 150, loss[loss=0.2113, simple_loss=0.3241, pruned_loss=0.04928, over 24307.00 frames. ], tot_loss[loss=0.271, simple_loss=0.3863, pruned_loss=0.07786, over 2557889.03 frames. ], batch size: 125, lr: 4.41e-03, grad_scale: 32.0 2026-09-24 04:21:06,816 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=150586.66666666666, ans=0.5 2026-09-24 04:21:16,207 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=150653.33333333334, ans=0.2 2026-09-24 04:21:16,598 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.52 vs. limit=6.0 2026-09-24 04:21:17,884 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=150653.33333333334, ans=0.0 2026-09-24 04:21:19,362 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=150653.33333333334, ans=0.125 2026-09-24 04:21:19,369 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=150653.33333333334, ans=0.125 2026-09-24 04:21:23,986 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=150686.66666666666, ans=0.025 2026-09-24 04:21:31,937 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:21:32,297 INFO [train.py:1192] (0/2) Epoch 48, batch 200, loss[loss=0.3019, simple_loss=0.4051, pruned_loss=0.09931, over 21110.00 frames. ], tot_loss[loss=0.2692, simple_loss=0.3842, pruned_loss=0.07703, over 3053663.85 frames. ], batch size: 333, lr: 4.41e-03, grad_scale: 32.0 2026-09-24 04:21:33,344 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=150753.33333333334, ans=0.1 2026-09-24 04:21:40,828 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.610e+02 3.325e+02 3.930e+02 4.573e+02 7.027e+02, threshold=7.861e+02, percent-clipped=0.0 2026-09-24 04:21:40,949 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=150786.66666666666, ans=0.05 2026-09-24 04:21:45,291 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=150820.0, ans=0.125 2026-09-24 04:21:57,570 INFO [train.py:1192] (0/2) Epoch 48, batch 250, loss[loss=0.2971, simple_loss=0.4179, pruned_loss=0.08817, over 24305.00 frames. ], tot_loss[loss=0.2688, simple_loss=0.3837, pruned_loss=0.07697, over 3442396.03 frames. ], batch size: 234, lr: 4.41e-03, grad_scale: 32.0 2026-09-24 04:22:02,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=150953.33333333334, ans=0.1 2026-09-24 04:22:09,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=150986.66666666666, ans=0.125 2026-09-24 04:22:22,759 INFO [train.py:1192] (0/2) Epoch 48, batch 300, loss[loss=0.2926, simple_loss=0.411, pruned_loss=0.08707, over 24538.00 frames. ], tot_loss[loss=0.268, simple_loss=0.3825, pruned_loss=0.07676, over 3747975.33 frames. ], batch size: 204, lr: 4.40e-03, grad_scale: 32.0 2026-09-24 04:22:31,874 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.428e+02 3.339e+02 3.787e+02 4.147e+02 5.561e+02, threshold=7.574e+02, percent-clipped=0.0 2026-09-24 04:22:39,215 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=151186.66666666666, ans=0.125 2026-09-24 04:22:44,855 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=151220.0, ans=0.0 2026-09-24 04:22:44,861 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=151220.0, ans=0.0 2026-09-24 04:22:48,414 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=151253.33333333334, ans=0.0 2026-09-24 04:22:48,688 INFO [train.py:1192] (0/2) Epoch 48, batch 350, loss[loss=0.2148, simple_loss=0.3291, pruned_loss=0.05021, over 24580.00 frames. ], tot_loss[loss=0.2695, simple_loss=0.384, pruned_loss=0.07752, over 3991133.09 frames. ], batch size: 137, lr: 4.40e-03, grad_scale: 32.0 2026-09-24 04:22:53,441 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=151286.66666666666, ans=0.1 2026-09-24 04:23:03,026 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.58 vs. limit=15.0 2026-09-24 04:23:03,364 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=151353.33333333334, ans=0.125 2026-09-24 04:23:08,206 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=151386.66666666666, ans=0.1 2026-09-24 04:23:10,374 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=151386.66666666666, ans=0.2 2026-09-24 04:23:13,578 INFO [train.py:1192] (0/2) Epoch 48, batch 400, loss[loss=0.2965, simple_loss=0.4008, pruned_loss=0.09613, over 24556.00 frames. ], tot_loss[loss=0.2676, simple_loss=0.3822, pruned_loss=0.07651, over 4178288.40 frames. ], batch size: 170, lr: 4.40e-03, grad_scale: 32.0 2026-09-24 04:23:15,137 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.82 vs. limit=6.0 2026-09-24 04:23:17,928 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=151420.0, ans=0.125 2026-09-24 04:23:20,206 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.12 vs. limit=15.0 2026-09-24 04:23:22,736 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.837e+02 3.335e+02 3.676e+02 4.305e+02 6.049e+02, threshold=7.352e+02, percent-clipped=0.0 2026-09-24 04:23:39,845 INFO [train.py:1192] (0/2) Epoch 48, batch 450, loss[loss=0.2891, simple_loss=0.4043, pruned_loss=0.08689, over 24617.00 frames. ], tot_loss[loss=0.2684, simple_loss=0.3829, pruned_loss=0.07702, over 4309507.05 frames. ], batch size: 175, lr: 4.40e-03, grad_scale: 32.0 2026-09-24 04:23:58,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=151686.66666666666, ans=0.1 2026-09-24 04:24:00,995 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=151720.0, ans=0.09899494936611666 2026-09-24 04:24:04,826 INFO [train.py:1192] (0/2) Epoch 48, batch 500, loss[loss=0.2719, simple_loss=0.4022, pruned_loss=0.07078, over 24483.00 frames. ], tot_loss[loss=0.2669, simple_loss=0.3812, pruned_loss=0.07629, over 4428006.81 frames. ], batch size: 218, lr: 4.39e-03, grad_scale: 32.0 2026-09-24 04:24:11,690 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=151786.66666666666, ans=0.125 2026-09-24 04:24:13,349 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.540e+02 3.286e+02 3.673e+02 4.043e+02 7.538e+02, threshold=7.347e+02, percent-clipped=1.0 2026-09-24 04:24:20,404 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=151853.33333333334, ans=0.0 2026-09-24 04:24:23,752 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=5.77 vs. limit=15.0 2026-09-24 04:24:23,979 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=151853.33333333334, ans=0.125 2026-09-24 04:24:26,727 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=151886.66666666666, ans=0.0 2026-09-24 04:24:30,563 INFO [train.py:1192] (0/2) Epoch 48, batch 550, loss[loss=0.2862, simple_loss=0.4085, pruned_loss=0.08193, over 24228.00 frames. ], tot_loss[loss=0.2671, simple_loss=0.3816, pruned_loss=0.07629, over 4517601.27 frames. ], batch size: 257, lr: 4.39e-03, grad_scale: 32.0 2026-09-24 04:24:32,172 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.23 vs. limit=10.0 2026-09-24 04:24:37,462 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=151953.33333333334, ans=0.0 2026-09-24 04:24:45,657 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.75 vs. limit=6.0 2026-09-24 04:24:52,994 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=152053.33333333334, ans=0.125 2026-09-24 04:24:55,518 INFO [train.py:1192] (0/2) Epoch 48, batch 600, loss[loss=0.2526, simple_loss=0.3848, pruned_loss=0.06019, over 24325.00 frames. ], tot_loss[loss=0.2675, simple_loss=0.3821, pruned_loss=0.07648, over 4584937.63 frames. ], batch size: 234, lr: 4.39e-03, grad_scale: 32.0 2026-09-24 04:24:59,768 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=152086.66666666666, ans=0.0 2026-09-24 04:25:03,862 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.649e+02 3.267e+02 3.673e+02 4.105e+02 6.858e+02, threshold=7.345e+02, percent-clipped=0.0 2026-09-24 04:25:04,406 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=152120.0, ans=0.125 2026-09-24 04:25:20,876 INFO [train.py:1192] (0/2) Epoch 48, batch 650, loss[loss=0.2915, simple_loss=0.3996, pruned_loss=0.09172, over 24568.00 frames. ], tot_loss[loss=0.2674, simple_loss=0.3819, pruned_loss=0.07641, over 4650263.27 frames. ], batch size: 162, lr: 4.39e-03, grad_scale: 32.0 2026-09-24 04:25:24,603 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=152253.33333333334, ans=0.125 2026-09-24 04:25:28,814 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=152286.66666666666, ans=0.2 2026-09-24 04:25:31,854 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=152320.0, ans=0.1 2026-09-24 04:25:35,919 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=152353.33333333334, ans=0.025 2026-09-24 04:25:42,730 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=152386.66666666666, ans=0.125 2026-09-24 04:25:46,278 INFO [train.py:1192] (0/2) Epoch 48, batch 700, loss[loss=0.2526, simple_loss=0.3637, pruned_loss=0.07075, over 24583.00 frames. ], tot_loss[loss=0.2677, simple_loss=0.3824, pruned_loss=0.07649, over 4682845.45 frames. ], batch size: 154, lr: 4.39e-03, grad_scale: 32.0 2026-09-24 04:25:50,009 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=152420.0, ans=0.125 2026-09-24 04:25:54,862 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.695e+02 3.461e+02 3.802e+02 4.262e+02 6.575e+02, threshold=7.604e+02, percent-clipped=0.0 2026-09-24 04:25:55,528 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=152453.33333333334, ans=0.125 2026-09-24 04:26:11,637 INFO [train.py:1192] (0/2) Epoch 48, batch 750, loss[loss=0.255, simple_loss=0.3753, pruned_loss=0.06732, over 24571.00 frames. ], tot_loss[loss=0.2669, simple_loss=0.3814, pruned_loss=0.07618, over 4714202.98 frames. ], batch size: 170, lr: 4.38e-03, grad_scale: 32.0 2026-09-24 04:26:18,546 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.23 vs. limit=6.0 2026-09-24 04:26:21,189 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.53 vs. limit=12.0 2026-09-24 04:26:24,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=152653.33333333334, ans=0.125 2026-09-24 04:26:37,351 INFO [train.py:1192] (0/2) Epoch 48, batch 800, loss[loss=0.2338, simple_loss=0.3425, pruned_loss=0.0625, over 24518.00 frames. ], tot_loss[loss=0.2674, simple_loss=0.3817, pruned_loss=0.07654, over 4738621.56 frames. ], batch size: 137, lr: 4.38e-03, grad_scale: 32.0 2026-09-24 04:26:45,995 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.614e+02 3.351e+02 3.680e+02 4.144e+02 5.789e+02, threshold=7.360e+02, percent-clipped=0.0 2026-09-24 04:26:47,594 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.30 vs. limit=15.0 2026-09-24 04:27:03,093 INFO [train.py:1192] (0/2) Epoch 48, batch 850, loss[loss=0.2737, simple_loss=0.3988, pruned_loss=0.07434, over 24552.00 frames. ], tot_loss[loss=0.2665, simple_loss=0.3812, pruned_loss=0.07597, over 4760520.95 frames. ], batch size: 204, lr: 4.38e-03, grad_scale: 32.0 2026-09-24 04:27:11,787 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.26 vs. limit=15.0 2026-09-24 04:27:16,570 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.24 vs. limit=15.0 2026-09-24 04:27:26,222 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=153053.33333333334, ans=0.125 2026-09-24 04:27:26,719 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=153053.33333333334, ans=0.125 2026-09-24 04:27:28,540 INFO [train.py:1192] (0/2) Epoch 48, batch 900, loss[loss=0.2265, simple_loss=0.3405, pruned_loss=0.05622, over 24554.00 frames. ], tot_loss[loss=0.2669, simple_loss=0.3814, pruned_loss=0.07619, over 4774038.52 frames. ], batch size: 137, lr: 4.38e-03, grad_scale: 32.0 2026-09-24 04:27:35,593 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=153120.0, ans=0.125 2026-09-24 04:27:37,269 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.574e+02 3.308e+02 3.681e+02 3.992e+02 6.338e+02, threshold=7.363e+02, percent-clipped=0.0 2026-09-24 04:27:40,318 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.36 vs. limit=15.0 2026-09-24 04:27:53,768 INFO [train.py:1192] (0/2) Epoch 48, batch 950, loss[loss=0.3821, simple_loss=0.4428, pruned_loss=0.1607, over 11505.00 frames. ], tot_loss[loss=0.268, simple_loss=0.3809, pruned_loss=0.0775, over 4715990.80 frames. ], batch size: 333, lr: 4.37e-03, grad_scale: 32.0 2026-09-24 04:27:55,356 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=153253.33333333334, ans=10.0 2026-09-24 04:27:58,005 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-48.pt 2026-09-24 04:28:05,083 INFO [train.py:1192] (0/2) Epoch 49, batch 0, loss[loss=0.2299, simple_loss=0.3467, pruned_loss=0.05656, over 24530.00 frames. ], tot_loss[loss=0.2299, simple_loss=0.3467, pruned_loss=0.05656, over 24530.00 frames. ], batch size: 137, lr: 4.33e-03, grad_scale: 32.0 2026-09-24 04:28:05,084 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 04:28:16,657 INFO [train.py:1224] (0/2) Epoch 49, validation: loss=0.1742, simple_loss=0.2928, pruned_loss=0.02777, over 2564189.00 frames. 2026-09-24 04:28:16,658 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 04:28:18,997 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=10.08 vs. limit=22.5 2026-09-24 04:28:26,706 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=153346.66666666666, ans=0.2 2026-09-24 04:28:31,903 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=153380.0, ans=0.125 2026-09-24 04:28:32,458 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=153380.0, ans=0.125 2026-09-24 04:28:33,042 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.64 vs. limit=15.0 2026-09-24 04:28:41,629 INFO [train.py:1192] (0/2) Epoch 49, batch 50, loss[loss=0.2378, simple_loss=0.3465, pruned_loss=0.06454, over 24326.00 frames. ], tot_loss[loss=0.2726, simple_loss=0.387, pruned_loss=0.07904, over 1075956.27 frames. ], batch size: 125, lr: 4.33e-03, grad_scale: 32.0 2026-09-24 04:28:45,909 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.772e+02 3.411e+02 3.790e+02 4.372e+02 7.575e+02, threshold=7.579e+02, percent-clipped=1.0 2026-09-24 04:28:48,587 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=153480.0, ans=0.0 2026-09-24 04:29:00,509 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=153546.66666666666, ans=0.05 2026-09-24 04:29:06,955 INFO [train.py:1192] (0/2) Epoch 49, batch 100, loss[loss=0.2628, simple_loss=0.3713, pruned_loss=0.07712, over 24628.00 frames. ], tot_loss[loss=0.2751, simple_loss=0.39, pruned_loss=0.08014, over 1902958.80 frames. ], batch size: 154, lr: 4.32e-03, grad_scale: 32.0 2026-09-24 04:29:25,373 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=153713.33333333334, ans=0.0 2026-09-24 04:29:25,610 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=6.05 vs. limit=15.0 2026-09-24 04:29:30,259 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=153746.66666666666, ans=0.125 2026-09-24 04:29:32,904 INFO [train.py:1192] (0/2) Epoch 49, batch 150, loss[loss=0.2314, simple_loss=0.334, pruned_loss=0.06444, over 24266.00 frames. ], tot_loss[loss=0.2719, simple_loss=0.3864, pruned_loss=0.07869, over 2557361.65 frames. ], batch size: 125, lr: 4.32e-03, grad_scale: 32.0 2026-09-24 04:29:34,829 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=153780.0, ans=0.125 2026-09-24 04:29:37,156 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.713e+02 3.405e+02 3.569e+02 4.203e+02 6.730e+02, threshold=7.137e+02, percent-clipped=0.0 2026-09-24 04:29:38,256 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=153813.33333333334, ans=0.125 2026-09-24 04:29:48,367 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=153880.0, ans=0.0 2026-09-24 04:29:58,648 INFO [train.py:1192] (0/2) Epoch 49, batch 200, loss[loss=0.3322, simple_loss=0.4256, pruned_loss=0.1194, over 21012.00 frames. ], tot_loss[loss=0.2699, simple_loss=0.3847, pruned_loss=0.07751, over 3053138.08 frames. ], batch size: 333, lr: 4.32e-03, grad_scale: 32.0 2026-09-24 04:30:06,005 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=153980.0, ans=0.07 2026-09-24 04:30:24,607 INFO [train.py:1192] (0/2) Epoch 49, batch 250, loss[loss=0.3248, simple_loss=0.4416, pruned_loss=0.104, over 24304.00 frames. ], tot_loss[loss=0.2695, simple_loss=0.3841, pruned_loss=0.07744, over 3442727.48 frames. ], batch size: 234, lr: 4.32e-03, grad_scale: 32.0 2026-09-24 04:30:24,704 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=154113.33333333334, ans=0.1 2026-09-24 04:30:28,990 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.668e+02 3.483e+02 3.921e+02 4.551e+02 6.304e+02, threshold=7.842e+02, percent-clipped=0.0 2026-09-24 04:30:30,074 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=154146.66666666666, ans=0.125 2026-09-24 04:30:39,003 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.26 vs. limit=12.0 2026-09-24 04:30:42,615 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=154213.33333333334, ans=0.0 2026-09-24 04:30:50,198 INFO [train.py:1192] (0/2) Epoch 49, batch 300, loss[loss=0.2752, simple_loss=0.3998, pruned_loss=0.07528, over 24531.00 frames. ], tot_loss[loss=0.268, simple_loss=0.3823, pruned_loss=0.0769, over 3748381.38 frames. ], batch size: 204, lr: 4.31e-03, grad_scale: 32.0 2026-09-24 04:31:10,059 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=13.57 vs. limit=15.0 2026-09-24 04:31:10,461 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=154413.33333333334, ans=0.09899494936611666 2026-09-24 04:31:15,199 INFO [train.py:1192] (0/2) Epoch 49, batch 350, loss[loss=0.2146, simple_loss=0.3272, pruned_loss=0.05102, over 24565.00 frames. ], tot_loss[loss=0.2682, simple_loss=0.3828, pruned_loss=0.07682, over 3991585.91 frames. ], batch size: 137, lr: 4.31e-03, grad_scale: 32.0 2026-09-24 04:31:19,892 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.470e+02 3.382e+02 3.658e+02 4.103e+02 5.631e+02, threshold=7.315e+02, percent-clipped=0.0 2026-09-24 04:31:24,682 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=154480.0, ans=0.125 2026-09-24 04:31:28,302 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=154513.33333333334, ans=0.125 2026-09-24 04:31:30,688 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.51 vs. limit=15.0 2026-09-24 04:31:31,204 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.00 vs. limit=22.5 2026-09-24 04:31:38,266 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.58 vs. limit=15.0 2026-09-24 04:31:39,515 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=154580.0, ans=0.0 2026-09-24 04:31:40,921 INFO [train.py:1192] (0/2) Epoch 49, batch 400, loss[loss=0.2894, simple_loss=0.3962, pruned_loss=0.09131, over 24584.00 frames. ], tot_loss[loss=0.2677, simple_loss=0.3824, pruned_loss=0.07652, over 4174622.52 frames. ], batch size: 170, lr: 4.31e-03, grad_scale: 32.0 2026-09-24 04:32:01,035 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=154746.66666666666, ans=0.125 2026-09-24 04:32:06,541 INFO [train.py:1192] (0/2) Epoch 49, batch 450, loss[loss=0.2753, simple_loss=0.3899, pruned_loss=0.08034, over 24633.00 frames. ], tot_loss[loss=0.2687, simple_loss=0.3832, pruned_loss=0.07715, over 4308888.36 frames. ], batch size: 175, lr: 4.31e-03, grad_scale: 32.0 2026-09-24 04:32:08,994 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=154780.0, ans=0.0 2026-09-24 04:32:10,594 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=154780.0, ans=0.1 2026-09-24 04:32:11,433 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.823e+02 3.406e+02 3.821e+02 4.417e+02 7.384e+02, threshold=7.641e+02, percent-clipped=1.0 2026-09-24 04:32:12,001 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=154813.33333333334, ans=0.125 2026-09-24 04:32:12,151 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.46 vs. limit=6.0 2026-09-24 04:32:13,386 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=154813.33333333334, ans=0.1 2026-09-24 04:32:20,556 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.44 vs. limit=22.5 2026-09-24 04:32:30,909 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=154913.33333333334, ans=0.1 2026-09-24 04:32:31,780 INFO [train.py:1192] (0/2) Epoch 49, batch 500, loss[loss=0.2897, simple_loss=0.4155, pruned_loss=0.08199, over 24489.00 frames. ], tot_loss[loss=0.2672, simple_loss=0.3815, pruned_loss=0.07647, over 4426258.04 frames. ], batch size: 218, lr: 4.31e-03, grad_scale: 32.0 2026-09-24 04:32:40,259 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=154980.0, ans=0.125 2026-09-24 04:32:43,448 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=155013.33333333334, ans=0.0 2026-09-24 04:32:47,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=155046.66666666666, ans=0.125 2026-09-24 04:32:47,755 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=155046.66666666666, ans=0.125 2026-09-24 04:32:52,426 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:32:53,400 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=155080.0, ans=0.05 2026-09-24 04:32:55,252 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=155080.0, ans=0.125 2026-09-24 04:32:57,669 INFO [train.py:1192] (0/2) Epoch 49, batch 550, loss[loss=0.296, simple_loss=0.4138, pruned_loss=0.08906, over 24248.00 frames. ], tot_loss[loss=0.2673, simple_loss=0.3818, pruned_loss=0.07638, over 4517344.35 frames. ], batch size: 257, lr: 4.30e-03, grad_scale: 32.0 2026-09-24 04:33:01,922 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.674e+02 3.320e+02 3.622e+02 4.066e+02 5.897e+02, threshold=7.244e+02, percent-clipped=0.0 2026-09-24 04:33:13,368 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=155213.33333333334, ans=0.5 2026-09-24 04:33:14,032 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=6.09 vs. limit=15.0 2026-09-24 04:33:23,652 INFO [train.py:1192] (0/2) Epoch 49, batch 600, loss[loss=0.2696, simple_loss=0.3912, pruned_loss=0.07401, over 24342.00 frames. ], tot_loss[loss=0.2678, simple_loss=0.3823, pruned_loss=0.07661, over 4583188.05 frames. ], batch size: 234, lr: 4.30e-03, grad_scale: 32.0 2026-09-24 04:33:24,237 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=155280.0, ans=0.09899494936611666 2026-09-24 04:33:48,806 INFO [train.py:1192] (0/2) Epoch 49, batch 650, loss[loss=0.2687, simple_loss=0.3828, pruned_loss=0.07728, over 24566.00 frames. ], tot_loss[loss=0.2669, simple_loss=0.3817, pruned_loss=0.07602, over 4648686.08 frames. ], batch size: 162, lr: 4.30e-03, grad_scale: 32.0 2026-09-24 04:33:53,183 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=155446.66666666666, ans=0.125 2026-09-24 04:33:53,474 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.674e+02 3.285e+02 3.739e+02 4.267e+02 7.018e+02, threshold=7.478e+02, percent-clipped=0.0 2026-09-24 04:33:53,587 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=155480.0, ans=0.125 2026-09-24 04:33:56,074 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=155480.0, ans=0.04949747468305833 2026-09-24 04:34:10,774 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=155580.0, ans=0.125 2026-09-24 04:34:14,580 INFO [train.py:1192] (0/2) Epoch 49, batch 700, loss[loss=0.2401, simple_loss=0.353, pruned_loss=0.06357, over 24581.00 frames. ], tot_loss[loss=0.2672, simple_loss=0.3823, pruned_loss=0.07606, over 4683083.08 frames. ], batch size: 154, lr: 4.30e-03, grad_scale: 32.0 2026-09-24 04:34:24,011 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=155680.0, ans=0.0 2026-09-24 04:34:28,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=155680.0, ans=0.1 2026-09-24 04:34:31,058 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.24 vs. limit=15.0 2026-09-24 04:34:37,836 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=155746.66666666666, ans=0.125 2026-09-24 04:34:39,547 INFO [train.py:1192] (0/2) Epoch 49, batch 750, loss[loss=0.2416, simple_loss=0.3671, pruned_loss=0.0581, over 24556.00 frames. ], tot_loss[loss=0.267, simple_loss=0.3817, pruned_loss=0.0761, over 4710896.49 frames. ], batch size: 170, lr: 4.29e-03, grad_scale: 32.0 2026-09-24 04:34:44,235 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.581e+02 3.365e+02 3.787e+02 4.317e+02 7.388e+02, threshold=7.575e+02, percent-clipped=0.0 2026-09-24 04:35:04,189 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.11 vs. limit=12.0 2026-09-24 04:35:04,932 INFO [train.py:1192] (0/2) Epoch 49, batch 800, loss[loss=0.243, simple_loss=0.3506, pruned_loss=0.06769, over 24533.00 frames. ], tot_loss[loss=0.2666, simple_loss=0.3812, pruned_loss=0.07599, over 4735958.27 frames. ], batch size: 137, lr: 4.29e-03, grad_scale: 32.0 2026-09-24 04:35:07,359 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:35:08,621 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=155946.66666666666, ans=0.125 2026-09-24 04:35:08,635 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=155946.66666666666, ans=0.2 2026-09-24 04:35:30,320 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=156113.33333333334, ans=0.0 2026-09-24 04:35:30,728 INFO [train.py:1192] (0/2) Epoch 49, batch 850, loss[loss=0.3206, simple_loss=0.4298, pruned_loss=0.1057, over 24590.00 frames. ], tot_loss[loss=0.2664, simple_loss=0.3809, pruned_loss=0.07594, over 4758961.42 frames. ], batch size: 198, lr: 4.29e-03, grad_scale: 32.0 2026-09-24 04:35:35,610 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.754e+02 3.290e+02 3.634e+02 4.203e+02 5.240e+02, threshold=7.268e+02, percent-clipped=0.0 2026-09-24 04:35:41,870 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=8.94 vs. limit=22.5 2026-09-24 04:35:42,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=156180.0, ans=0.0 2026-09-24 04:35:44,463 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=156180.0, ans=0.125 2026-09-24 04:35:44,490 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=156180.0, ans=0.125 2026-09-24 04:35:52,561 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=156246.66666666666, ans=0.07 2026-09-24 04:35:53,496 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=156246.66666666666, ans=0.0 2026-09-24 04:35:56,851 INFO [train.py:1192] (0/2) Epoch 49, batch 900, loss[loss=0.24, simple_loss=0.3484, pruned_loss=0.06582, over 24554.00 frames. ], tot_loss[loss=0.2674, simple_loss=0.3819, pruned_loss=0.0765, over 4772557.27 frames. ], batch size: 137, lr: 4.29e-03, grad_scale: 32.0 2026-09-24 04:36:13,824 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=156380.0, ans=0.125 2026-09-24 04:36:22,053 INFO [train.py:1192] (0/2) Epoch 49, batch 950, loss[loss=0.3623, simple_loss=0.4366, pruned_loss=0.144, over 11337.00 frames. ], tot_loss[loss=0.268, simple_loss=0.3809, pruned_loss=0.07756, over 4713140.46 frames. ], batch size: 333, lr: 4.28e-03, grad_scale: 32.0 2026-09-24 04:36:23,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=156446.66666666666, ans=0.125 2026-09-24 04:36:26,265 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-49.pt 2026-09-24 04:36:31,064 INFO [train.py:1192] (0/2) Epoch 50, batch 0, loss[loss=0.2462, simple_loss=0.3584, pruned_loss=0.06704, over 24550.00 frames. ], tot_loss[loss=0.2462, simple_loss=0.3584, pruned_loss=0.06704, over 24550.00 frames. ], batch size: 137, lr: 4.24e-03, grad_scale: 32.0 2026-09-24 04:36:31,064 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 04:36:42,796 INFO [train.py:1224] (0/2) Epoch 50, validation: loss=0.1711, simple_loss=0.2901, pruned_loss=0.02606, over 2564189.00 frames. 2026-09-24 04:36:42,796 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 04:36:43,679 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.433e+02 3.385e+02 3.879e+02 4.350e+02 6.538e+02, threshold=7.759e+02, percent-clipped=0.0 2026-09-24 04:36:45,911 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=156473.33333333334, ans=0.125 2026-09-24 04:36:47,430 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=156506.66666666666, ans=0.125 2026-09-24 04:37:03,130 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=156606.66666666666, ans=0.1 2026-09-24 04:37:03,920 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=156606.66666666666, ans=0.0 2026-09-24 04:37:08,434 INFO [train.py:1192] (0/2) Epoch 50, batch 50, loss[loss=0.236, simple_loss=0.3403, pruned_loss=0.06584, over 24243.00 frames. ], tot_loss[loss=0.2732, simple_loss=0.3864, pruned_loss=0.08003, over 1076459.35 frames. ], batch size: 125, lr: 4.24e-03, grad_scale: 32.0 2026-09-24 04:37:08,532 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=156640.0, ans=0.025 2026-09-24 04:37:08,537 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=156640.0, ans=0.0 2026-09-24 04:37:10,549 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=6.14 vs. limit=15.0 2026-09-24 04:37:12,591 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=8.86 vs. limit=22.5 2026-09-24 04:37:16,564 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.56 vs. limit=15.0 2026-09-24 04:37:25,506 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.46 vs. limit=15.0 2026-09-24 04:37:34,293 INFO [train.py:1192] (0/2) Epoch 50, batch 100, loss[loss=0.2627, simple_loss=0.3734, pruned_loss=0.07594, over 24603.00 frames. ], tot_loss[loss=0.2752, simple_loss=0.39, pruned_loss=0.08017, over 1903667.18 frames. ], batch size: 154, lr: 4.24e-03, grad_scale: 32.0 2026-09-24 04:37:35,234 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.636e+02 3.396e+02 3.722e+02 4.209e+02 6.213e+02, threshold=7.444e+02, percent-clipped=0.0 2026-09-24 04:37:46,956 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.38 vs. limit=15.0 2026-09-24 04:37:47,308 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=156873.33333333334, ans=0.0 2026-09-24 04:37:50,552 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=156906.66666666666, ans=0.0 2026-09-24 04:37:59,478 INFO [train.py:1192] (0/2) Epoch 50, batch 150, loss[loss=0.2406, simple_loss=0.3424, pruned_loss=0.06941, over 24245.00 frames. ], tot_loss[loss=0.2697, simple_loss=0.385, pruned_loss=0.07722, over 2557099.68 frames. ], batch size: 125, lr: 4.23e-03, grad_scale: 32.0 2026-09-24 04:38:02,751 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=156973.33333333334, ans=0.125 2026-09-24 04:38:02,763 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=156973.33333333334, ans=0.0 2026-09-24 04:38:13,335 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=157040.0, ans=0.1 2026-09-24 04:38:23,817 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=157106.66666666666, ans=0.2 2026-09-24 04:38:24,990 INFO [train.py:1192] (0/2) Epoch 50, batch 200, loss[loss=0.3067, simple_loss=0.4148, pruned_loss=0.09932, over 20996.00 frames. ], tot_loss[loss=0.2667, simple_loss=0.3826, pruned_loss=0.07546, over 3054076.45 frames. ], batch size: 333, lr: 4.23e-03, grad_scale: 32.0 2026-09-24 04:38:26,000 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.389e+02 3.343e+02 3.775e+02 4.499e+02 6.276e+02, threshold=7.550e+02, percent-clipped=0.0 2026-09-24 04:38:41,976 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=157240.0, ans=0.0 2026-09-24 04:38:42,593 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=157240.0, ans=0.0 2026-09-24 04:38:51,434 INFO [train.py:1192] (0/2) Epoch 50, batch 250, loss[loss=0.3082, simple_loss=0.4232, pruned_loss=0.09657, over 24308.00 frames. ], tot_loss[loss=0.2673, simple_loss=0.3825, pruned_loss=0.07603, over 3443353.74 frames. ], batch size: 234, lr: 4.23e-03, grad_scale: 32.0 2026-09-24 04:38:58,488 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=157340.0, ans=0.125 2026-09-24 04:39:00,029 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=157340.0, ans=0.125 2026-09-24 04:39:04,239 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=157373.33333333334, ans=0.2 2026-09-24 04:39:05,279 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=157373.33333333334, ans=0.125 2026-09-24 04:39:06,146 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=157406.66666666666, ans=0.025 2026-09-24 04:39:10,233 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.59 vs. limit=22.5 2026-09-24 04:39:11,809 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=157440.0, ans=0.1 2026-09-24 04:39:16,947 INFO [train.py:1192] (0/2) Epoch 50, batch 300, loss[loss=0.2756, simple_loss=0.4001, pruned_loss=0.07557, over 24565.00 frames. ], tot_loss[loss=0.2666, simple_loss=0.3813, pruned_loss=0.07593, over 3748283.51 frames. ], batch size: 204, lr: 4.23e-03, grad_scale: 32.0 2026-09-24 04:39:18,000 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.829e+02 3.385e+02 3.880e+02 4.348e+02 7.252e+02, threshold=7.759e+02, percent-clipped=0.0 2026-09-24 04:39:34,898 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=157573.33333333334, ans=0.2 2026-09-24 04:39:37,485 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.12 vs. limit=15.0 2026-09-24 04:39:42,199 INFO [train.py:1192] (0/2) Epoch 50, batch 350, loss[loss=0.2529, simple_loss=0.356, pruned_loss=0.07491, over 24563.00 frames. ], tot_loss[loss=0.2671, simple_loss=0.3823, pruned_loss=0.07595, over 3992023.83 frames. ], batch size: 137, lr: 4.23e-03, grad_scale: 32.0 2026-09-24 04:39:45,724 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=157640.0, ans=0.0 2026-09-24 04:39:57,801 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=157740.0, ans=0.1 2026-09-24 04:39:59,845 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.86 vs. limit=15.0 2026-09-24 04:40:05,243 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=157773.33333333334, ans=0.0 2026-09-24 04:40:08,040 INFO [train.py:1192] (0/2) Epoch 50, batch 400, loss[loss=0.2521, simple_loss=0.3656, pruned_loss=0.06925, over 24582.00 frames. ], tot_loss[loss=0.2674, simple_loss=0.3823, pruned_loss=0.07621, over 4177001.91 frames. ], batch size: 170, lr: 4.22e-03, grad_scale: 32.0 2026-09-24 04:40:08,595 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=157806.66666666666, ans=0.015 2026-09-24 04:40:09,092 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.477e+02 3.343e+02 3.833e+02 4.625e+02 8.353e+02, threshold=7.667e+02, percent-clipped=1.0 2026-09-24 04:40:12,073 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=157806.66666666666, ans=0.125 2026-09-24 04:40:12,946 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=157840.0, ans=0.125 2026-09-24 04:40:26,149 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.26 vs. limit=22.5 2026-09-24 04:40:33,871 INFO [train.py:1192] (0/2) Epoch 50, batch 450, loss[loss=0.2721, simple_loss=0.3902, pruned_loss=0.07701, over 24638.00 frames. ], tot_loss[loss=0.2684, simple_loss=0.383, pruned_loss=0.07687, over 4310514.85 frames. ], batch size: 175, lr: 4.22e-03, grad_scale: 32.0 2026-09-24 04:40:35,197 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.64 vs. limit=10.0 2026-09-24 04:40:39,801 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=158006.66666666666, ans=0.125 2026-09-24 04:40:58,850 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=158106.66666666666, ans=0.125 2026-09-24 04:40:59,742 INFO [train.py:1192] (0/2) Epoch 50, batch 500, loss[loss=0.3109, simple_loss=0.431, pruned_loss=0.09545, over 24486.00 frames. ], tot_loss[loss=0.2681, simple_loss=0.3823, pruned_loss=0.07697, over 4428049.06 frames. ], batch size: 218, lr: 4.22e-03, grad_scale: 32.0 2026-09-24 04:41:00,682 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.605e+02 3.299e+02 3.819e+02 4.283e+02 6.545e+02, threshold=7.637e+02, percent-clipped=0.0 2026-09-24 04:41:00,805 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=158140.0, ans=0.0 2026-09-24 04:41:08,600 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=158173.33333333334, ans=0.2 2026-09-24 04:41:17,785 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=158240.0, ans=0.015 2026-09-24 04:41:25,472 INFO [train.py:1192] (0/2) Epoch 50, batch 550, loss[loss=0.2797, simple_loss=0.4091, pruned_loss=0.07515, over 24228.00 frames. ], tot_loss[loss=0.2681, simple_loss=0.3825, pruned_loss=0.07688, over 4517506.86 frames. ], batch size: 257, lr: 4.22e-03, grad_scale: 32.0 2026-09-24 04:41:29,892 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=158306.66666666666, ans=0.0 2026-09-24 04:41:33,761 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=18.83 vs. limit=22.5 2026-09-24 04:41:42,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=158406.66666666666, ans=0.0 2026-09-24 04:41:45,956 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=158440.0, ans=0.0 2026-09-24 04:41:51,765 INFO [train.py:1192] (0/2) Epoch 50, batch 600, loss[loss=0.2728, simple_loss=0.4024, pruned_loss=0.07161, over 24324.00 frames. ], tot_loss[loss=0.2683, simple_loss=0.3831, pruned_loss=0.07679, over 4584824.27 frames. ], batch size: 234, lr: 4.21e-03, grad_scale: 32.0 2026-09-24 04:41:52,797 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.780e+02 3.282e+02 3.695e+02 4.091e+02 6.346e+02, threshold=7.389e+02, percent-clipped=0.0 2026-09-24 04:41:58,236 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=158506.66666666666, ans=0.2 2026-09-24 04:42:05,628 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=158540.0, ans=0.025 2026-09-24 04:42:15,874 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=158606.66666666666, ans=0.0 2026-09-24 04:42:16,987 INFO [train.py:1192] (0/2) Epoch 50, batch 650, loss[loss=0.2635, simple_loss=0.3784, pruned_loss=0.07432, over 24563.00 frames. ], tot_loss[loss=0.2661, simple_loss=0.3814, pruned_loss=0.07544, over 4650287.88 frames. ], batch size: 162, lr: 4.21e-03, grad_scale: 32.0 2026-09-24 04:42:20,083 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.14 vs. limit=15.0 2026-09-24 04:42:26,221 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:42:28,534 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=158706.66666666666, ans=0.0 2026-09-24 04:42:30,505 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=158706.66666666666, ans=10.0 2026-09-24 04:42:30,507 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=158706.66666666666, ans=0.025 2026-09-24 04:42:42,470 INFO [train.py:1192] (0/2) Epoch 50, batch 700, loss[loss=0.2601, simple_loss=0.3675, pruned_loss=0.07639, over 24573.00 frames. ], tot_loss[loss=0.2664, simple_loss=0.3818, pruned_loss=0.0755, over 4682125.42 frames. ], batch size: 154, lr: 4.21e-03, grad_scale: 32.0 2026-09-24 04:42:42,917 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=158806.66666666666, ans=0.0 2026-09-24 04:42:43,349 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=158806.66666666666, ans=0.035 2026-09-24 04:42:43,380 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=158806.66666666666, ans=0.07 2026-09-24 04:42:43,769 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.653e+02 3.394e+02 3.848e+02 4.389e+02 7.114e+02, threshold=7.696e+02, percent-clipped=0.0 2026-09-24 04:42:58,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=158906.66666666666, ans=0.0 2026-09-24 04:43:08,562 INFO [train.py:1192] (0/2) Epoch 50, batch 750, loss[loss=0.2551, simple_loss=0.3764, pruned_loss=0.06694, over 24556.00 frames. ], tot_loss[loss=0.2658, simple_loss=0.3808, pruned_loss=0.07538, over 4709349.47 frames. ], batch size: 170, lr: 4.21e-03, grad_scale: 32.0 2026-09-24 04:43:10,588 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.74 vs. limit=15.0 2026-09-24 04:43:10,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=158973.33333333334, ans=0.125 2026-09-24 04:43:11,418 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.13 vs. limit=15.0 2026-09-24 04:43:11,463 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten.whitening_limit, batch_count=158973.33333333334, ans=15.0 2026-09-24 04:43:13,463 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.06 vs. limit=6.0 2026-09-24 04:43:33,942 INFO [train.py:1192] (0/2) Epoch 50, batch 800, loss[loss=0.218, simple_loss=0.3307, pruned_loss=0.05269, over 24524.00 frames. ], tot_loss[loss=0.2653, simple_loss=0.3804, pruned_loss=0.07508, over 4735052.06 frames. ], batch size: 137, lr: 4.21e-03, grad_scale: 32.0 2026-09-24 04:43:34,776 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.450e+02 3.348e+02 3.704e+02 4.242e+02 6.225e+02, threshold=7.409e+02, percent-clipped=0.0 2026-09-24 04:43:41,702 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=159173.33333333334, ans=0.125 2026-09-24 04:43:42,106 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=159173.33333333334, ans=0.1 2026-09-24 04:43:51,548 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=159240.0, ans=0.2 2026-09-24 04:43:56,078 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=159273.33333333334, ans=0.0 2026-09-24 04:43:57,474 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=159273.33333333334, ans=0.025 2026-09-24 04:43:58,854 INFO [train.py:1192] (0/2) Epoch 50, batch 850, loss[loss=0.2815, simple_loss=0.4085, pruned_loss=0.07725, over 24520.00 frames. ], tot_loss[loss=0.2648, simple_loss=0.38, pruned_loss=0.07477, over 4757822.32 frames. ], batch size: 204, lr: 4.20e-03, grad_scale: 32.0 2026-09-24 04:44:11,301 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:44:23,820 INFO [train.py:1192] (0/2) Epoch 50, batch 900, loss[loss=0.2045, simple_loss=0.3228, pruned_loss=0.04305, over 24563.00 frames. ], tot_loss[loss=0.2651, simple_loss=0.3804, pruned_loss=0.07496, over 4773217.18 frames. ], batch size: 137, lr: 4.20e-03, grad_scale: 32.0 2026-09-24 04:44:25,033 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.433e+02 3.308e+02 3.810e+02 4.336e+02 6.433e+02, threshold=7.620e+02, percent-clipped=0.0 2026-09-24 04:44:43,348 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=159573.33333333334, ans=0.0 2026-09-24 04:44:45,089 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.28 vs. limit=22.5 2026-09-24 04:44:49,059 INFO [train.py:1192] (0/2) Epoch 50, batch 950, loss[loss=0.3943, simple_loss=0.4508, pruned_loss=0.1689, over 10693.00 frames. ], tot_loss[loss=0.2659, simple_loss=0.3797, pruned_loss=0.07603, over 4712062.80 frames. ], batch size: 334, lr: 4.20e-03, grad_scale: 32.0 2026-09-24 04:44:49,126 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:44:50,522 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=159640.0, ans=0.125 2026-09-24 04:44:50,927 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=159640.0, ans=0.1 2026-09-24 04:44:53,582 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-50.pt 2026-09-24 04:45:01,167 INFO [train.py:1192] (0/2) Epoch 51, batch 0, loss[loss=0.2315, simple_loss=0.3475, pruned_loss=0.05774, over 24563.00 frames. ], tot_loss[loss=0.2315, simple_loss=0.3475, pruned_loss=0.05774, over 24563.00 frames. ], batch size: 137, lr: 4.16e-03, grad_scale: 32.0 2026-09-24 04:45:01,168 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 04:45:12,233 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.3134, 3.3889, 3.2733, 3.1941], device='cuda:0') 2026-09-24 04:45:12,821 INFO [train.py:1224] (0/2) Epoch 51, validation: loss=0.1742, simple_loss=0.2926, pruned_loss=0.02794, over 2564189.00 frames. 2026-09-24 04:45:12,822 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 04:45:18,856 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=159700.0, ans=0.125 2026-09-24 04:45:28,815 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=159766.66666666666, ans=0.05 2026-09-24 04:45:33,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=159800.0, ans=0.025 2026-09-24 04:45:35,098 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.751e+02 3.480e+02 3.802e+02 4.431e+02 6.259e+02, threshold=7.604e+02, percent-clipped=0.0 2026-09-24 04:45:38,324 INFO [train.py:1192] (0/2) Epoch 51, batch 50, loss[loss=0.2261, simple_loss=0.331, pruned_loss=0.06065, over 24312.00 frames. ], tot_loss[loss=0.2766, simple_loss=0.3889, pruned_loss=0.08217, over 1076973.16 frames. ], batch size: 125, lr: 4.16e-03, grad_scale: 32.0 2026-09-24 04:45:46,408 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=159866.66666666666, ans=0.125 2026-09-24 04:45:50,381 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.91 vs. limit=15.0 2026-09-24 04:46:03,019 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-48000.pt 2026-09-24 04:46:03,740 INFO [train.py:1192] (0/2) Epoch 51, batch 100, loss[loss=0.2772, simple_loss=0.3774, pruned_loss=0.08851, over 24606.00 frames. ], tot_loss[loss=0.2753, simple_loss=0.3896, pruned_loss=0.08049, over 1904736.32 frames. ], batch size: 154, lr: 4.15e-03, grad_scale: 32.0 2026-09-24 04:46:12,423 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=160033.33333333334, ans=0.1 2026-09-24 04:46:22,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=160100.0, ans=0.2 2026-09-24 04:46:26,901 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.485e+02 3.390e+02 3.893e+02 4.368e+02 6.652e+02, threshold=7.786e+02, percent-clipped=0.0 2026-09-24 04:46:30,049 INFO [train.py:1192] (0/2) Epoch 51, batch 150, loss[loss=0.2525, simple_loss=0.3487, pruned_loss=0.07821, over 24236.00 frames. ], tot_loss[loss=0.2713, simple_loss=0.3856, pruned_loss=0.07846, over 2558132.44 frames. ], batch size: 125, lr: 4.15e-03, grad_scale: 32.0 2026-09-24 04:46:34,026 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=15.0 2026-09-24 04:46:45,105 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=160266.66666666666, ans=0.1 2026-09-24 04:46:52,808 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=160300.0, ans=0.0 2026-09-24 04:46:55,492 INFO [train.py:1192] (0/2) Epoch 51, batch 200, loss[loss=0.2795, simple_loss=0.392, pruned_loss=0.08348, over 21213.00 frames. ], tot_loss[loss=0.2682, simple_loss=0.3832, pruned_loss=0.07663, over 3056621.47 frames. ], batch size: 333, lr: 4.15e-03, grad_scale: 32.0 2026-09-24 04:46:55,788 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.81 vs. limit=15.0 2026-09-24 04:46:56,552 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=160333.33333333334, ans=0.2 2026-09-24 04:47:01,818 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=160366.66666666666, ans=0.125 2026-09-24 04:47:04,368 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=160366.66666666666, ans=0.09899494936611666 2026-09-24 04:47:09,227 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=160400.0, ans=0.0 2026-09-24 04:47:17,477 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.506e+02 3.435e+02 3.856e+02 4.490e+02 6.082e+02, threshold=7.712e+02, percent-clipped=0.0 2026-09-24 04:47:20,317 INFO [train.py:1192] (0/2) Epoch 51, batch 250, loss[loss=0.3113, simple_loss=0.4326, pruned_loss=0.09494, over 24312.00 frames. ], tot_loss[loss=0.2671, simple_loss=0.3823, pruned_loss=0.07601, over 3446232.49 frames. ], batch size: 234, lr: 4.15e-03, grad_scale: 32.0 2026-09-24 04:47:24,582 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=160500.0, ans=0.0 2026-09-24 04:47:27,923 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=160533.33333333334, ans=0.125 2026-09-24 04:47:30,255 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=160566.66666666666, ans=0.125 2026-09-24 04:47:33,908 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=160566.66666666666, ans=0.125 2026-09-24 04:47:37,767 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=160600.0, ans=0.2 2026-09-24 04:47:44,576 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=160633.33333333334, ans=0.125 2026-09-24 04:47:46,481 INFO [train.py:1192] (0/2) Epoch 51, batch 300, loss[loss=0.2813, simple_loss=0.3984, pruned_loss=0.08208, over 24552.00 frames. ], tot_loss[loss=0.266, simple_loss=0.3808, pruned_loss=0.07557, over 3751503.86 frames. ], batch size: 204, lr: 4.14e-03, grad_scale: 32.0 2026-09-24 04:47:48,352 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.70 vs. limit=15.0 2026-09-24 04:47:50,493 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.64 vs. limit=10.0 2026-09-24 04:47:52,197 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=160700.0, ans=0.0 2026-09-24 04:47:57,456 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.08 vs. limit=22.5 2026-09-24 04:48:02,355 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=160766.66666666666, ans=10.0 2026-09-24 04:48:08,514 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=160800.0, ans=0.2 2026-09-24 04:48:08,887 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.699e+02 3.332e+02 3.581e+02 4.177e+02 6.227e+02, threshold=7.162e+02, percent-clipped=0.0 2026-09-24 04:48:10,139 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.62 vs. limit=15.0 2026-09-24 04:48:12,028 INFO [train.py:1192] (0/2) Epoch 51, batch 350, loss[loss=0.2274, simple_loss=0.339, pruned_loss=0.05783, over 24586.00 frames. ], tot_loss[loss=0.2664, simple_loss=0.3816, pruned_loss=0.07562, over 3990741.24 frames. ], batch size: 137, lr: 4.14e-03, grad_scale: 32.0 2026-09-24 04:48:16,771 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=160866.66666666666, ans=0.1 2026-09-24 04:48:18,233 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=160866.66666666666, ans=0.0 2026-09-24 04:48:18,278 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:48:23,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=160900.0, ans=0.125 2026-09-24 04:48:29,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=160933.33333333334, ans=0.0 2026-09-24 04:48:33,720 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.16 vs. limit=15.0 2026-09-24 04:48:36,801 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=160966.66666666666, ans=0.125 2026-09-24 04:48:37,663 INFO [train.py:1192] (0/2) Epoch 51, batch 400, loss[loss=0.2673, simple_loss=0.3871, pruned_loss=0.07377, over 24547.00 frames. ], tot_loss[loss=0.2659, simple_loss=0.3811, pruned_loss=0.07531, over 4178210.65 frames. ], batch size: 170, lr: 4.14e-03, grad_scale: 32.0 2026-09-24 04:48:42,672 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=161033.33333333334, ans=0.025 2026-09-24 04:48:43,205 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=161033.33333333334, ans=0.125 2026-09-24 04:48:54,258 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer_ff2.min_abs, batch_count=161100.0, ans=0.1 2026-09-24 04:48:55,758 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.11 vs. limit=15.0 2026-09-24 04:48:58,242 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=161133.33333333334, ans=0.1 2026-09-24 04:48:59,947 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.681e+02 3.395e+02 3.852e+02 4.592e+02 6.632e+02, threshold=7.704e+02, percent-clipped=0.0 2026-09-24 04:49:02,798 INFO [train.py:1192] (0/2) Epoch 51, batch 450, loss[loss=0.2735, simple_loss=0.3898, pruned_loss=0.07856, over 24626.00 frames. ], tot_loss[loss=0.2664, simple_loss=0.3817, pruned_loss=0.07555, over 4309352.04 frames. ], batch size: 175, lr: 4.14e-03, grad_scale: 32.0 2026-09-24 04:49:03,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=161166.66666666666, ans=0.0 2026-09-24 04:49:11,014 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.34 vs. limit=15.0 2026-09-24 04:49:11,929 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=161200.0, ans=0.125 2026-09-24 04:49:19,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=161266.66666666666, ans=0.2 2026-09-24 04:49:20,981 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=161266.66666666666, ans=0.125 2026-09-24 04:49:24,949 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=161300.0, ans=0.125 2026-09-24 04:49:28,285 INFO [train.py:1192] (0/2) Epoch 51, batch 500, loss[loss=0.3023, simple_loss=0.4226, pruned_loss=0.09099, over 24486.00 frames. ], tot_loss[loss=0.2652, simple_loss=0.3801, pruned_loss=0.07514, over 4427003.80 frames. ], batch size: 218, lr: 4.14e-03, grad_scale: 32.0 2026-09-24 04:49:30,634 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=161333.33333333334, ans=0.125 2026-09-24 04:49:35,370 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=161366.66666666666, ans=0.2 2026-09-24 04:49:41,027 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=161400.0, ans=10.0 2026-09-24 04:49:50,390 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.493e+02 3.341e+02 3.648e+02 4.164e+02 5.336e+02, threshold=7.296e+02, percent-clipped=0.0 2026-09-24 04:49:53,806 INFO [train.py:1192] (0/2) Epoch 51, batch 550, loss[loss=0.2625, simple_loss=0.3901, pruned_loss=0.06751, over 24265.00 frames. ], tot_loss[loss=0.2656, simple_loss=0.3806, pruned_loss=0.0753, over 4517291.45 frames. ], batch size: 257, lr: 4.13e-03, grad_scale: 32.0 2026-09-24 04:50:01,456 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=161533.33333333334, ans=0.035 2026-09-24 04:50:16,897 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=161633.33333333334, ans=0.125 2026-09-24 04:50:19,491 INFO [train.py:1192] (0/2) Epoch 51, batch 600, loss[loss=0.2913, simple_loss=0.4162, pruned_loss=0.08314, over 24418.00 frames. ], tot_loss[loss=0.2656, simple_loss=0.381, pruned_loss=0.07515, over 4583656.36 frames. ], batch size: 235, lr: 4.13e-03, grad_scale: 32.0 2026-09-24 04:50:21,905 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=161666.66666666666, ans=0.025 2026-09-24 04:50:27,402 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.57 vs. limit=22.5 2026-09-24 04:50:31,459 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=161733.33333333334, ans=0.025 2026-09-24 04:50:34,044 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:50:41,557 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.597e+02 3.326e+02 3.679e+02 4.212e+02 6.429e+02, threshold=7.358e+02, percent-clipped=0.0 2026-09-24 04:50:44,524 INFO [train.py:1192] (0/2) Epoch 51, batch 650, loss[loss=0.2744, simple_loss=0.3822, pruned_loss=0.08331, over 24556.00 frames. ], tot_loss[loss=0.2647, simple_loss=0.3802, pruned_loss=0.07456, over 4649300.36 frames. ], batch size: 162, lr: 4.13e-03, grad_scale: 32.0 2026-09-24 04:50:49,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=161866.66666666666, ans=0.1 2026-09-24 04:50:55,530 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.41 vs. limit=15.0 2026-09-24 04:51:03,685 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=161933.33333333334, ans=0.125 2026-09-24 04:51:04,620 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=161966.66666666666, ans=0.5 2026-09-24 04:51:07,472 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=161966.66666666666, ans=0.125 2026-09-24 04:51:09,866 INFO [train.py:1192] (0/2) Epoch 51, batch 700, loss[loss=0.2614, simple_loss=0.3706, pruned_loss=0.07604, over 24573.00 frames. ], tot_loss[loss=0.265, simple_loss=0.3808, pruned_loss=0.07455, over 4682008.27 frames. ], batch size: 154, lr: 4.13e-03, grad_scale: 32.0 2026-09-24 04:51:10,512 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=162000.0, ans=0.125 2026-09-24 04:51:14,848 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=162033.33333333334, ans=0.2 2026-09-24 04:51:20,636 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=162066.66666666666, ans=0.125 2026-09-24 04:51:25,174 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=162100.0, ans=0.0 2026-09-24 04:51:29,756 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.75 vs. limit=15.0 2026-09-24 04:51:32,180 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.650e+02 3.501e+02 3.853e+02 4.434e+02 7.375e+02, threshold=7.706e+02, percent-clipped=1.0 2026-09-24 04:51:35,132 INFO [train.py:1192] (0/2) Epoch 51, batch 750, loss[loss=0.2626, simple_loss=0.3814, pruned_loss=0.07186, over 24565.00 frames. ], tot_loss[loss=0.2644, simple_loss=0.3799, pruned_loss=0.07443, over 4708960.40 frames. ], batch size: 170, lr: 4.13e-03, grad_scale: 32.0 2026-09-24 04:51:38,289 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=162166.66666666666, ans=0.1 2026-09-24 04:51:38,792 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=162166.66666666666, ans=0.025 2026-09-24 04:51:43,680 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.72 vs. limit=8.0 2026-09-24 04:51:44,517 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=162200.0, ans=0.125 2026-09-24 04:51:47,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=162233.33333333334, ans=0.125 2026-09-24 04:51:52,716 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=162266.66666666666, ans=0.2 2026-09-24 04:51:57,519 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=162300.0, ans=0.125 2026-09-24 04:52:00,970 INFO [train.py:1192] (0/2) Epoch 51, batch 800, loss[loss=0.2046, simple_loss=0.3208, pruned_loss=0.04422, over 24549.00 frames. ], tot_loss[loss=0.2647, simple_loss=0.38, pruned_loss=0.07472, over 4734358.87 frames. ], batch size: 137, lr: 4.12e-03, grad_scale: 32.0 2026-09-24 04:52:04,452 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=162333.33333333334, ans=0.125 2026-09-24 04:52:05,515 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=162333.33333333334, ans=0.0 2026-09-24 04:52:23,769 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.585e+02 3.311e+02 3.770e+02 4.405e+02 6.210e+02, threshold=7.541e+02, percent-clipped=0.0 2026-09-24 04:52:26,556 INFO [train.py:1192] (0/2) Epoch 51, batch 850, loss[loss=0.2752, simple_loss=0.397, pruned_loss=0.07676, over 24592.00 frames. ], tot_loss[loss=0.2637, simple_loss=0.3792, pruned_loss=0.07415, over 4757558.46 frames. ], batch size: 198, lr: 4.12e-03, grad_scale: 32.0 2026-09-24 04:52:31,500 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=162533.33333333334, ans=0.2 2026-09-24 04:52:34,766 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=162533.33333333334, ans=0.125 2026-09-24 04:52:35,536 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.21 vs. limit=22.5 2026-09-24 04:52:52,884 INFO [train.py:1192] (0/2) Epoch 51, batch 900, loss[loss=0.2279, simple_loss=0.3443, pruned_loss=0.0558, over 24577.00 frames. ], tot_loss[loss=0.2649, simple_loss=0.3801, pruned_loss=0.07484, over 4771840.66 frames. ], batch size: 137, lr: 4.12e-03, grad_scale: 32.0 2026-09-24 04:52:55,804 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.40 vs. limit=10.0 2026-09-24 04:52:56,789 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=162666.66666666666, ans=0.125 2026-09-24 04:52:59,704 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=162700.0, ans=0.1 2026-09-24 04:53:13,180 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=162800.0, ans=0.1 2026-09-24 04:53:15,092 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.707e+02 3.515e+02 3.906e+02 4.366e+02 6.602e+02, threshold=7.811e+02, percent-clipped=0.0 2026-09-24 04:53:16,568 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=162800.0, ans=0.05 2026-09-24 04:53:17,799 INFO [train.py:1192] (0/2) Epoch 51, batch 950, loss[loss=0.3542, simple_loss=0.4228, pruned_loss=0.1428, over 11369.00 frames. ], tot_loss[loss=0.2656, simple_loss=0.3791, pruned_loss=0.07606, over 4713546.61 frames. ], batch size: 333, lr: 4.12e-03, grad_scale: 32.0 2026-09-24 04:53:21,946 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-51.pt 2026-09-24 04:53:43,616 INFO [train.py:1192] (0/2) Epoch 52, batch 0, loss[loss=0.2347, simple_loss=0.3497, pruned_loss=0.05981, over 24536.00 frames. ], tot_loss[loss=0.2347, simple_loss=0.3497, pruned_loss=0.05981, over 24536.00 frames. ], batch size: 137, lr: 4.08e-03, grad_scale: 32.0 2026-09-24 04:53:43,616 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 04:53:51,873 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.0227, 2.0533, 2.4045, 2.0944, 1.7855, 2.3360, 1.2806, 1.9694], device='cuda:0') 2026-09-24 04:53:53,176 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.6905, 2.2576, 3.8227, 2.3348], device='cuda:0') 2026-09-24 04:53:54,767 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.1.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.2589, 3.0649, 2.8532, 2.1859], device='cuda:0') 2026-09-24 04:53:55,115 INFO [train.py:1224] (0/2) Epoch 52, validation: loss=0.1726, simple_loss=0.2914, pruned_loss=0.02685, over 2564189.00 frames. 2026-09-24 04:53:55,115 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 04:54:07,679 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=162926.66666666666, ans=0.0 2026-09-24 04:54:10,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=162960.0, ans=0.0 2026-09-24 04:54:17,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=162993.33333333334, ans=0.125 2026-09-24 04:54:20,528 INFO [train.py:1192] (0/2) Epoch 52, batch 50, loss[loss=0.2341, simple_loss=0.3423, pruned_loss=0.06298, over 24237.00 frames. ], tot_loss[loss=0.2724, simple_loss=0.3872, pruned_loss=0.07886, over 1074776.55 frames. ], batch size: 125, lr: 4.07e-03, grad_scale: 32.0 2026-09-24 04:54:24,633 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=163026.66666666666, ans=0.0 2026-09-24 04:54:35,653 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=163126.66666666666, ans=0.125 2026-09-24 04:54:37,796 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.42 vs. limit=15.0 2026-09-24 04:54:38,535 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.799e+02 3.476e+02 3.824e+02 4.146e+02 5.729e+02, threshold=7.649e+02, percent-clipped=0.0 2026-09-24 04:54:43,928 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=163160.0, ans=0.1 2026-09-24 04:54:45,738 INFO [train.py:1192] (0/2) Epoch 52, batch 100, loss[loss=0.2624, simple_loss=0.3727, pruned_loss=0.07609, over 24610.00 frames. ], tot_loss[loss=0.2737, simple_loss=0.389, pruned_loss=0.07919, over 1903660.56 frames. ], batch size: 154, lr: 4.07e-03, grad_scale: 64.0 2026-09-24 04:54:59,648 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=163260.0, ans=0.2 2026-09-24 04:55:01,525 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=163293.33333333334, ans=0.125 2026-09-24 04:55:11,078 INFO [train.py:1192] (0/2) Epoch 52, batch 150, loss[loss=0.2226, simple_loss=0.3353, pruned_loss=0.05497, over 24312.00 frames. ], tot_loss[loss=0.2687, simple_loss=0.3843, pruned_loss=0.0765, over 2557572.95 frames. ], batch size: 125, lr: 4.07e-03, grad_scale: 64.0 2026-09-24 04:55:12,846 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=163360.0, ans=0.125 2026-09-24 04:55:15,793 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.23 vs. limit=6.0 2026-09-24 04:55:16,004 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=163393.33333333334, ans=0.1 2026-09-24 04:55:20,340 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=163426.66666666666, ans=0.125 2026-09-24 04:55:23,362 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.71 vs. limit=22.5 2026-09-24 04:55:27,324 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=163460.0, ans=0.1 2026-09-24 04:55:28,705 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.605e+02 3.284e+02 3.723e+02 4.398e+02 6.163e+02, threshold=7.447e+02, percent-clipped=0.0 2026-09-24 04:55:35,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=163493.33333333334, ans=0.07 2026-09-24 04:55:35,787 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=163526.66666666666, ans=0.125 2026-09-24 04:55:36,225 INFO [train.py:1192] (0/2) Epoch 52, batch 200, loss[loss=0.3015, simple_loss=0.4056, pruned_loss=0.09864, over 21049.00 frames. ], tot_loss[loss=0.2658, simple_loss=0.3816, pruned_loss=0.07502, over 3054841.00 frames. ], batch size: 333, lr: 4.07e-03, grad_scale: 64.0 2026-09-24 04:55:45,715 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=163560.0, ans=0.125 2026-09-24 04:56:01,902 INFO [train.py:1192] (0/2) Epoch 52, batch 250, loss[loss=0.3117, simple_loss=0.4303, pruned_loss=0.09654, over 24289.00 frames. ], tot_loss[loss=0.2648, simple_loss=0.3807, pruned_loss=0.07451, over 3443506.86 frames. ], batch size: 234, lr: 4.07e-03, grad_scale: 64.0 2026-09-24 04:56:07,473 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=163726.66666666666, ans=0.125 2026-09-24 04:56:12,967 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=163760.0, ans=0.125 2026-09-24 04:56:20,171 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.910e+02 3.359e+02 3.886e+02 4.376e+02 6.164e+02, threshold=7.772e+02, percent-clipped=0.0 2026-09-24 04:56:26,126 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.26 vs. limit=15.0 2026-09-24 04:56:27,292 INFO [train.py:1192] (0/2) Epoch 52, batch 300, loss[loss=0.2646, simple_loss=0.392, pruned_loss=0.06859, over 24582.00 frames. ], tot_loss[loss=0.2643, simple_loss=0.3794, pruned_loss=0.07457, over 3749483.70 frames. ], batch size: 204, lr: 4.06e-03, grad_scale: 64.0 2026-09-24 04:56:27,926 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=163860.0, ans=0.125 2026-09-24 04:56:39,290 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=163926.66666666666, ans=0.0 2026-09-24 04:56:39,792 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=163926.66666666666, ans=0.0 2026-09-24 04:56:39,799 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=163926.66666666666, ans=0.0 2026-09-24 04:56:46,958 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=163960.0, ans=0.1 2026-09-24 04:56:51,008 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=163993.33333333334, ans=0.125 2026-09-24 04:56:53,216 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=11.07 vs. limit=22.5 2026-09-24 04:56:53,605 INFO [train.py:1192] (0/2) Epoch 52, batch 350, loss[loss=0.2344, simple_loss=0.3428, pruned_loss=0.06302, over 24579.00 frames. ], tot_loss[loss=0.266, simple_loss=0.3813, pruned_loss=0.07539, over 3988873.10 frames. ], batch size: 137, lr: 4.06e-03, grad_scale: 64.0 2026-09-24 04:56:55,699 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=164026.66666666666, ans=0.0 2026-09-24 04:56:58,502 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=164060.0, ans=0.125 2026-09-24 04:57:11,792 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.781e+02 3.431e+02 3.812e+02 4.631e+02 6.402e+02, threshold=7.623e+02, percent-clipped=0.0 2026-09-24 04:57:12,397 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=164126.66666666666, ans=0.125 2026-09-24 04:57:17,225 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=164160.0, ans=0.2 2026-09-24 04:57:18,477 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=164160.0, ans=0.0 2026-09-24 04:57:19,352 INFO [train.py:1192] (0/2) Epoch 52, batch 400, loss[loss=0.258, simple_loss=0.3745, pruned_loss=0.07077, over 24563.00 frames. ], tot_loss[loss=0.2648, simple_loss=0.3802, pruned_loss=0.0747, over 4176771.89 frames. ], batch size: 170, lr: 4.06e-03, grad_scale: 64.0 2026-09-24 04:57:23,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=164193.33333333334, ans=0.025 2026-09-24 04:57:34,010 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.74 vs. limit=10.0 2026-09-24 04:57:35,871 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:57:38,782 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=164326.66666666666, ans=0.0 2026-09-24 04:57:44,888 INFO [train.py:1192] (0/2) Epoch 52, batch 450, loss[loss=0.2708, simple_loss=0.3927, pruned_loss=0.07439, over 24622.00 frames. ], tot_loss[loss=0.2657, simple_loss=0.381, pruned_loss=0.07515, over 4310970.93 frames. ], batch size: 175, lr: 4.06e-03, grad_scale: 64.0 2026-09-24 04:57:48,285 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 04:57:54,747 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=8.97 vs. limit=22.5 2026-09-24 04:57:56,470 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=164426.66666666666, ans=0.125 2026-09-24 04:58:02,627 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.537e+02 3.288e+02 3.722e+02 4.227e+02 6.045e+02, threshold=7.443e+02, percent-clipped=0.0 2026-09-24 04:58:09,871 INFO [train.py:1192] (0/2) Epoch 52, batch 500, loss[loss=0.3007, simple_loss=0.4231, pruned_loss=0.08917, over 24511.00 frames. ], tot_loss[loss=0.2639, simple_loss=0.3792, pruned_loss=0.0743, over 4428696.37 frames. ], batch size: 218, lr: 4.06e-03, grad_scale: 64.0 2026-09-24 04:58:11,963 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.30 vs. limit=15.0 2026-09-24 04:58:15,314 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=164560.0, ans=0.125 2026-09-24 04:58:26,045 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=164626.66666666666, ans=0.0 2026-09-24 04:58:27,050 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=164626.66666666666, ans=0.125 2026-09-24 04:58:35,343 INFO [train.py:1192] (0/2) Epoch 52, batch 550, loss[loss=0.2835, simple_loss=0.4099, pruned_loss=0.07855, over 24269.00 frames. ], tot_loss[loss=0.2647, simple_loss=0.3803, pruned_loss=0.07452, over 4517710.54 frames. ], batch size: 257, lr: 4.05e-03, grad_scale: 64.0 2026-09-24 04:58:43,026 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=164726.66666666666, ans=0.0 2026-09-24 04:58:53,757 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.511e+02 3.288e+02 3.632e+02 3.999e+02 5.951e+02, threshold=7.265e+02, percent-clipped=0.0 2026-09-24 04:58:54,915 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=164793.33333333334, ans=0.2 2026-09-24 04:59:00,947 INFO [train.py:1192] (0/2) Epoch 52, batch 600, loss[loss=0.2927, simple_loss=0.4104, pruned_loss=0.08753, over 24331.00 frames. ], tot_loss[loss=0.2664, simple_loss=0.3816, pruned_loss=0.07555, over 4584303.09 frames. ], batch size: 234, lr: 4.05e-03, grad_scale: 64.0 2026-09-24 04:59:04,601 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=164860.0, ans=0.125 2026-09-24 04:59:14,323 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=164926.66666666666, ans=0.5 2026-09-24 04:59:19,801 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=164960.0, ans=0.1 2026-09-24 04:59:23,382 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=164993.33333333334, ans=0.125 2026-09-24 04:59:25,247 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=164993.33333333334, ans=0.0 2026-09-24 04:59:26,564 INFO [train.py:1192] (0/2) Epoch 52, batch 650, loss[loss=0.2615, simple_loss=0.3765, pruned_loss=0.07325, over 24565.00 frames. ], tot_loss[loss=0.2651, simple_loss=0.3806, pruned_loss=0.07476, over 4649919.84 frames. ], batch size: 162, lr: 4.05e-03, grad_scale: 64.0 2026-09-24 04:59:29,995 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.69 vs. limit=15.0 2026-09-24 04:59:45,092 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.682e+02 3.336e+02 3.710e+02 4.177e+02 6.779e+02, threshold=7.421e+02, percent-clipped=0.0 2026-09-24 04:59:51,911 INFO [train.py:1192] (0/2) Epoch 52, batch 700, loss[loss=0.2355, simple_loss=0.3536, pruned_loss=0.05865, over 24576.00 frames. ], tot_loss[loss=0.2654, simple_loss=0.3811, pruned_loss=0.07482, over 4682717.91 frames. ], batch size: 154, lr: 4.05e-03, grad_scale: 64.0 2026-09-24 05:00:14,795 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=165326.66666666666, ans=10.0 2026-09-24 05:00:17,521 INFO [train.py:1192] (0/2) Epoch 52, batch 750, loss[loss=0.2835, simple_loss=0.397, pruned_loss=0.08496, over 24562.00 frames. ], tot_loss[loss=0.2654, simple_loss=0.3804, pruned_loss=0.07517, over 4710393.53 frames. ], batch size: 170, lr: 4.05e-03, grad_scale: 64.0 2026-09-24 05:00:18,962 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=165360.0, ans=0.2 2026-09-24 05:00:36,100 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.774e+02 3.253e+02 3.701e+02 4.355e+02 6.161e+02, threshold=7.402e+02, percent-clipped=0.0 2026-09-24 05:00:36,222 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=165460.0, ans=0.125 2026-09-24 05:00:42,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=165493.33333333334, ans=0.2 2026-09-24 05:00:43,104 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=165526.66666666666, ans=0.125 2026-09-24 05:00:43,551 INFO [train.py:1192] (0/2) Epoch 52, batch 800, loss[loss=0.2416, simple_loss=0.3542, pruned_loss=0.06448, over 24553.00 frames. ], tot_loss[loss=0.2649, simple_loss=0.3799, pruned_loss=0.07497, over 4735742.96 frames. ], batch size: 137, lr: 4.04e-03, grad_scale: 64.0 2026-09-24 05:00:56,843 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=165593.33333333334, ans=0.125 2026-09-24 05:01:01,586 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=165626.66666666666, ans=0.025 2026-09-24 05:01:08,876 INFO [train.py:1192] (0/2) Epoch 52, batch 850, loss[loss=0.274, simple_loss=0.3954, pruned_loss=0.07629, over 24602.00 frames. ], tot_loss[loss=0.2639, simple_loss=0.3791, pruned_loss=0.07432, over 4758829.25 frames. ], batch size: 198, lr: 4.04e-03, grad_scale: 64.0 2026-09-24 05:01:09,491 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=165693.33333333334, ans=0.2 2026-09-24 05:01:25,820 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=165793.33333333334, ans=0.0 2026-09-24 05:01:27,625 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:01:27,954 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.431e+02 3.277e+02 3.764e+02 4.341e+02 6.225e+02, threshold=7.527e+02, percent-clipped=0.0 2026-09-24 05:01:32,966 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=165826.66666666666, ans=0.0 2026-09-24 05:01:34,859 INFO [train.py:1192] (0/2) Epoch 52, batch 900, loss[loss=0.2275, simple_loss=0.3492, pruned_loss=0.05286, over 24549.00 frames. ], tot_loss[loss=0.2653, simple_loss=0.3802, pruned_loss=0.07518, over 4772854.12 frames. ], batch size: 137, lr: 4.04e-03, grad_scale: 32.0 2026-09-24 05:01:35,420 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=165860.0, ans=0.95 2026-09-24 05:01:37,346 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=165860.0, ans=0.0 2026-09-24 05:01:51,583 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=165960.0, ans=0.025 2026-09-24 05:01:52,057 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=165960.0, ans=0.125 2026-09-24 05:01:57,041 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=165993.33333333334, ans=0.0 2026-09-24 05:02:00,373 INFO [train.py:1192] (0/2) Epoch 52, batch 950, loss[loss=0.3507, simple_loss=0.4197, pruned_loss=0.1408, over 11368.00 frames. ], tot_loss[loss=0.2656, simple_loss=0.3789, pruned_loss=0.07612, over 4715511.16 frames. ], batch size: 333, lr: 4.04e-03, grad_scale: 16.0 2026-09-24 05:02:04,542 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-52.pt 2026-09-24 05:02:10,259 INFO [train.py:1192] (0/2) Epoch 53, batch 0, loss[loss=0.213, simple_loss=0.3365, pruned_loss=0.04472, over 24553.00 frames. ], tot_loss[loss=0.213, simple_loss=0.3365, pruned_loss=0.04472, over 24553.00 frames. ], batch size: 137, lr: 4.00e-03, grad_scale: 32.0 2026-09-24 05:02:10,259 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 05:02:12,589 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.6334, 2.7940, 2.6008, 1.7028], device='cuda:0') 2026-09-24 05:02:21,902 INFO [train.py:1224] (0/2) Epoch 53, validation: loss=0.171, simple_loss=0.2896, pruned_loss=0.0262, over 2564189.00 frames. 2026-09-24 05:02:21,907 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 05:02:23,591 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.90 vs. limit=22.5 2026-09-24 05:02:31,875 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=166120.0, ans=0.125 2026-09-24 05:02:37,248 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.655e+02 3.515e+02 4.091e+02 4.690e+02 6.599e+02, threshold=8.183e+02, percent-clipped=0.0 2026-09-24 05:02:43,692 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=166186.66666666666, ans=0.125 2026-09-24 05:02:44,673 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=166186.66666666666, ans=0.05 2026-09-24 05:02:47,641 INFO [train.py:1192] (0/2) Epoch 53, batch 50, loss[loss=0.2251, simple_loss=0.3302, pruned_loss=0.05999, over 24254.00 frames. ], tot_loss[loss=0.2702, simple_loss=0.3848, pruned_loss=0.07779, over 1075340.90 frames. ], batch size: 125, lr: 4.00e-03, grad_scale: 32.0 2026-09-24 05:02:50,221 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=166220.0, ans=0.0 2026-09-24 05:02:50,253 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:03:09,487 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=8.16 vs. limit=15.0 2026-09-24 05:03:13,183 INFO [train.py:1192] (0/2) Epoch 53, batch 100, loss[loss=0.2638, simple_loss=0.3751, pruned_loss=0.07621, over 24609.00 frames. ], tot_loss[loss=0.2731, simple_loss=0.3888, pruned_loss=0.07866, over 1904477.86 frames. ], batch size: 154, lr: 4.00e-03, grad_scale: 32.0 2026-09-24 05:03:13,811 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=166386.66666666666, ans=0.125 2026-09-24 05:03:17,320 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=166386.66666666666, ans=0.0 2026-09-24 05:03:19,326 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=12.0 2026-09-24 05:03:28,099 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.355e+02 3.302e+02 3.763e+02 4.279e+02 8.783e+02, threshold=7.526e+02, percent-clipped=1.0 2026-09-24 05:03:38,533 INFO [train.py:1192] (0/2) Epoch 53, batch 150, loss[loss=0.2271, simple_loss=0.3353, pruned_loss=0.05945, over 24240.00 frames. ], tot_loss[loss=0.2681, simple_loss=0.384, pruned_loss=0.07607, over 2558425.16 frames. ], batch size: 125, lr: 3.99e-03, grad_scale: 32.0 2026-09-24 05:03:49,033 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=166620.0, ans=0.025 2026-09-24 05:03:49,035 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=166620.0, ans=0.2 2026-09-24 05:03:54,265 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=166653.33333333334, ans=0.0 2026-09-24 05:03:57,658 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=166653.33333333334, ans=0.125 2026-09-24 05:04:03,897 INFO [train.py:1192] (0/2) Epoch 53, batch 200, loss[loss=0.3136, simple_loss=0.4099, pruned_loss=0.1087, over 21017.00 frames. ], tot_loss[loss=0.2671, simple_loss=0.3829, pruned_loss=0.0756, over 3054651.88 frames. ], batch size: 333, lr: 3.99e-03, grad_scale: 32.0 2026-09-24 05:04:17,604 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=166786.66666666666, ans=0.125 2026-09-24 05:04:18,896 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.437e+02 3.418e+02 3.818e+02 4.297e+02 6.430e+02, threshold=7.637e+02, percent-clipped=0.0 2026-09-24 05:04:21,637 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.26 vs. limit=10.0 2026-09-24 05:04:22,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=166820.0, ans=0.125 2026-09-24 05:04:29,153 INFO [train.py:1192] (0/2) Epoch 53, batch 250, loss[loss=0.2705, simple_loss=0.3983, pruned_loss=0.07133, over 24301.00 frames. ], tot_loss[loss=0.265, simple_loss=0.3811, pruned_loss=0.07448, over 3443270.68 frames. ], batch size: 234, lr: 3.99e-03, grad_scale: 32.0 2026-09-24 05:04:43,496 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.31 vs. limit=15.0 2026-09-24 05:04:45,767 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=166986.66666666666, ans=0.125 2026-09-24 05:04:47,413 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.96 vs. limit=6.0 2026-09-24 05:04:52,079 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=167020.0, ans=0.2 2026-09-24 05:04:53,430 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=167020.0, ans=0.125 2026-09-24 05:04:54,228 INFO [train.py:1192] (0/2) Epoch 53, batch 300, loss[loss=0.2891, simple_loss=0.411, pruned_loss=0.08357, over 24549.00 frames. ], tot_loss[loss=0.265, simple_loss=0.3805, pruned_loss=0.07477, over 3747993.83 frames. ], batch size: 204, lr: 3.99e-03, grad_scale: 32.0 2026-09-24 05:04:59,242 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=167086.66666666666, ans=0.125 2026-09-24 05:05:03,567 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=167086.66666666666, ans=10.0 2026-09-24 05:05:09,121 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.681e+02 3.368e+02 3.702e+02 4.143e+02 5.571e+02, threshold=7.403e+02, percent-clipped=0.0 2026-09-24 05:05:12,907 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=167153.33333333334, ans=0.125 2026-09-24 05:05:19,229 INFO [train.py:1192] (0/2) Epoch 53, batch 350, loss[loss=0.2123, simple_loss=0.3246, pruned_loss=0.04997, over 24576.00 frames. ], tot_loss[loss=0.265, simple_loss=0.381, pruned_loss=0.07454, over 3993013.20 frames. ], batch size: 137, lr: 3.99e-03, grad_scale: 32.0 2026-09-24 05:05:21,599 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.84 vs. limit=15.0 2026-09-24 05:05:27,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=167253.33333333334, ans=0.0 2026-09-24 05:05:38,391 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=167320.0, ans=0.125 2026-09-24 05:05:45,351 INFO [train.py:1192] (0/2) Epoch 53, batch 400, loss[loss=0.2834, simple_loss=0.3987, pruned_loss=0.08404, over 24590.00 frames. ], tot_loss[loss=0.2642, simple_loss=0.38, pruned_loss=0.07418, over 4177379.07 frames. ], batch size: 170, lr: 3.98e-03, grad_scale: 32.0 2026-09-24 05:05:55,821 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=167453.33333333334, ans=0.125 2026-09-24 05:06:00,577 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:06:00,880 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.584e+02 3.416e+02 3.853e+02 4.638e+02 5.855e+02, threshold=7.707e+02, percent-clipped=0.0 2026-09-24 05:06:01,707 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=11.67 vs. limit=15.0 2026-09-24 05:06:02,110 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.25 vs. limit=15.0 2026-09-24 05:06:11,079 INFO [train.py:1192] (0/2) Epoch 53, batch 450, loss[loss=0.2621, simple_loss=0.3839, pruned_loss=0.0701, over 24625.00 frames. ], tot_loss[loss=0.2652, simple_loss=0.3808, pruned_loss=0.07481, over 4311300.24 frames. ], batch size: 175, lr: 3.98e-03, grad_scale: 32.0 2026-09-24 05:06:13,480 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=13.32 vs. limit=22.5 2026-09-24 05:06:34,025 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=167686.66666666666, ans=0.125 2026-09-24 05:06:34,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=167686.66666666666, ans=0.0 2026-09-24 05:06:36,748 INFO [train.py:1192] (0/2) Epoch 53, batch 500, loss[loss=0.3106, simple_loss=0.4307, pruned_loss=0.09526, over 24480.00 frames. ], tot_loss[loss=0.2643, simple_loss=0.3797, pruned_loss=0.07443, over 4428379.26 frames. ], batch size: 218, lr: 3.98e-03, grad_scale: 32.0 2026-09-24 05:06:37,413 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=167720.0, ans=0.125 2026-09-24 05:06:52,084 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.590e+02 3.274e+02 3.651e+02 4.065e+02 5.653e+02, threshold=7.301e+02, percent-clipped=0.0 2026-09-24 05:06:55,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer_ff2.min_abs, batch_count=167820.0, ans=0.1 2026-09-24 05:06:58,038 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=167853.33333333334, ans=0.125 2026-09-24 05:07:00,847 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=167853.33333333334, ans=0.0 2026-09-24 05:07:02,155 INFO [train.py:1192] (0/2) Epoch 53, batch 550, loss[loss=0.2878, simple_loss=0.4157, pruned_loss=0.0799, over 24283.00 frames. ], tot_loss[loss=0.264, simple_loss=0.3796, pruned_loss=0.07419, over 4517548.80 frames. ], batch size: 257, lr: 3.98e-03, grad_scale: 32.0 2026-09-24 05:07:08,115 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=167920.0, ans=0.2 2026-09-24 05:07:17,298 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=167986.66666666666, ans=0.025 2026-09-24 05:07:27,640 INFO [train.py:1192] (0/2) Epoch 53, batch 600, loss[loss=0.2731, simple_loss=0.4013, pruned_loss=0.07251, over 24331.00 frames. ], tot_loss[loss=0.264, simple_loss=0.3799, pruned_loss=0.07407, over 4584410.21 frames. ], batch size: 234, lr: 3.98e-03, grad_scale: 32.0 2026-09-24 05:07:27,710 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=168053.33333333334, ans=0.0 2026-09-24 05:07:33,192 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=168086.66666666666, ans=0.125 2026-09-24 05:07:33,691 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=168086.66666666666, ans=0.1 2026-09-24 05:07:37,918 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=168120.0, ans=0.0 2026-09-24 05:07:42,731 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.643e+02 3.309e+02 3.591e+02 3.976e+02 6.557e+02, threshold=7.181e+02, percent-clipped=0.0 2026-09-24 05:07:42,963 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.72 vs. limit=15.0 2026-09-24 05:07:48,272 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=168186.66666666666, ans=0.0 2026-09-24 05:07:51,292 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=168186.66666666666, ans=0.1 2026-09-24 05:07:52,652 INFO [train.py:1192] (0/2) Epoch 53, batch 650, loss[loss=0.2783, simple_loss=0.3888, pruned_loss=0.08391, over 24552.00 frames. ], tot_loss[loss=0.2626, simple_loss=0.3786, pruned_loss=0.07331, over 4649885.80 frames. ], batch size: 162, lr: 3.97e-03, grad_scale: 32.0 2026-09-24 05:07:55,080 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=168220.0, ans=0.0 2026-09-24 05:08:05,944 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=168286.66666666666, ans=0.125 2026-09-24 05:08:06,512 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=168286.66666666666, ans=0.1 2026-09-24 05:08:08,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=168320.0, ans=0.07 2026-09-24 05:08:14,838 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=168353.33333333334, ans=0.0 2026-09-24 05:08:18,293 INFO [train.py:1192] (0/2) Epoch 53, batch 700, loss[loss=0.2687, simple_loss=0.3744, pruned_loss=0.08153, over 24569.00 frames. ], tot_loss[loss=0.2638, simple_loss=0.3797, pruned_loss=0.07393, over 4682161.94 frames. ], batch size: 154, lr: 3.97e-03, grad_scale: 32.0 2026-09-24 05:08:20,176 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=168386.66666666666, ans=0.1 2026-09-24 05:08:27,544 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=168420.0, ans=0.0 2026-09-24 05:08:31,928 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=168453.33333333334, ans=0.0 2026-09-24 05:08:32,754 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=168453.33333333334, ans=0.2 2026-09-24 05:08:33,536 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.683e+02 3.418e+02 3.806e+02 4.319e+02 6.633e+02, threshold=7.612e+02, percent-clipped=0.0 2026-09-24 05:08:33,846 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.25 vs. limit=22.5 2026-09-24 05:08:43,354 INFO [train.py:1192] (0/2) Epoch 53, batch 750, loss[loss=0.268, simple_loss=0.3824, pruned_loss=0.07678, over 24565.00 frames. ], tot_loss[loss=0.2625, simple_loss=0.3781, pruned_loss=0.07342, over 4709221.42 frames. ], batch size: 170, lr: 3.97e-03, grad_scale: 32.0 2026-09-24 05:08:47,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=168553.33333333334, ans=0.125 2026-09-24 05:08:47,528 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=168553.33333333334, ans=0.0 2026-09-24 05:08:49,052 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=168586.66666666666, ans=0.1 2026-09-24 05:08:51,826 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=168586.66666666666, ans=0.0 2026-09-24 05:09:05,622 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=168686.66666666666, ans=0.025 2026-09-24 05:09:06,562 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=168686.66666666666, ans=0.0 2026-09-24 05:09:08,889 INFO [train.py:1192] (0/2) Epoch 53, batch 800, loss[loss=0.2077, simple_loss=0.3242, pruned_loss=0.04557, over 24549.00 frames. ], tot_loss[loss=0.2627, simple_loss=0.3782, pruned_loss=0.07359, over 4739023.46 frames. ], batch size: 137, lr: 3.97e-03, grad_scale: 32.0 2026-09-24 05:09:09,382 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=168720.0, ans=0.1 2026-09-24 05:09:22,216 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:09:22,987 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:09:24,341 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.559e+02 3.349e+02 3.707e+02 4.167e+02 6.733e+02, threshold=7.413e+02, percent-clipped=0.0 2026-09-24 05:09:28,362 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.64 vs. limit=15.0 2026-09-24 05:09:28,712 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=168820.0, ans=0.5 2026-09-24 05:09:34,556 INFO [train.py:1192] (0/2) Epoch 53, batch 850, loss[loss=0.2933, simple_loss=0.4109, pruned_loss=0.08784, over 24526.00 frames. ], tot_loss[loss=0.2633, simple_loss=0.3787, pruned_loss=0.07395, over 4760584.56 frames. ], batch size: 204, lr: 3.97e-03, grad_scale: 32.0 2026-09-24 05:09:35,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=168886.66666666666, ans=0.125 2026-09-24 05:10:00,564 INFO [train.py:1192] (0/2) Epoch 53, batch 900, loss[loss=0.2317, simple_loss=0.3514, pruned_loss=0.05605, over 24533.00 frames. ], tot_loss[loss=0.264, simple_loss=0.3796, pruned_loss=0.07424, over 4774278.82 frames. ], batch size: 137, lr: 3.96e-03, grad_scale: 32.0 2026-09-24 05:10:08,963 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=169086.66666666666, ans=0.125 2026-09-24 05:10:15,702 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.552e+02 3.417e+02 3.958e+02 4.535e+02 7.546e+02, threshold=7.916e+02, percent-clipped=1.0 2026-09-24 05:10:24,411 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=169186.66666666666, ans=0.2 2026-09-24 05:10:25,368 INFO [train.py:1192] (0/2) Epoch 53, batch 950, loss[loss=0.3419, simple_loss=0.4212, pruned_loss=0.1313, over 11035.00 frames. ], tot_loss[loss=0.264, simple_loss=0.3781, pruned_loss=0.07495, over 4709567.13 frames. ], batch size: 333, lr: 3.96e-03, grad_scale: 32.0 2026-09-24 05:10:27,851 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=20.99 vs. limit=22.5 2026-09-24 05:10:29,666 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-53.pt 2026-09-24 05:10:37,219 INFO [train.py:1192] (0/2) Epoch 54, batch 0, loss[loss=0.2212, simple_loss=0.3415, pruned_loss=0.05046, over 24578.00 frames. ], tot_loss[loss=0.2212, simple_loss=0.3415, pruned_loss=0.05046, over 24578.00 frames. ], batch size: 137, lr: 3.93e-03, grad_scale: 32.0 2026-09-24 05:10:37,220 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 05:10:40,391 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.1218, 1.7502, 3.0397, 2.0128], device='cuda:0') 2026-09-24 05:10:48,902 INFO [train.py:1224] (0/2) Epoch 54, validation: loss=0.1725, simple_loss=0.2912, pruned_loss=0.02692, over 2564189.00 frames. 2026-09-24 05:10:48,903 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 05:10:54,335 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=169280.0, ans=0.025 2026-09-24 05:10:57,812 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=169280.0, ans=0.125 2026-09-24 05:11:04,992 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.21 vs. limit=15.0 2026-09-24 05:11:09,986 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=169380.0, ans=0.0 2026-09-24 05:11:13,890 INFO [train.py:1192] (0/2) Epoch 54, batch 50, loss[loss=0.2087, simple_loss=0.3219, pruned_loss=0.04776, over 24264.00 frames. ], tot_loss[loss=0.2726, simple_loss=0.3868, pruned_loss=0.0792, over 1075720.56 frames. ], batch size: 125, lr: 3.92e-03, grad_scale: 32.0 2026-09-24 05:11:14,522 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=169413.33333333334, ans=0.0 2026-09-24 05:11:18,903 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=169446.66666666666, ans=0.125 2026-09-24 05:11:23,511 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=169446.66666666666, ans=0.125 2026-09-24 05:11:25,120 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.865e+02 3.454e+02 4.001e+02 4.532e+02 7.399e+02, threshold=8.002e+02, percent-clipped=0.0 2026-09-24 05:11:26,647 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=169480.0, ans=0.125 2026-09-24 05:11:30,111 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=169513.33333333334, ans=0.0 2026-09-24 05:11:38,305 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.14 vs. limit=10.0 2026-09-24 05:11:39,437 INFO [train.py:1192] (0/2) Epoch 54, batch 100, loss[loss=0.2674, simple_loss=0.3808, pruned_loss=0.07702, over 24616.00 frames. ], tot_loss[loss=0.2749, simple_loss=0.3902, pruned_loss=0.07974, over 1904140.63 frames. ], batch size: 154, lr: 3.92e-03, grad_scale: 32.0 2026-09-24 05:11:40,991 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.70 vs. limit=22.5 2026-09-24 05:11:45,002 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=169613.33333333334, ans=0.125 2026-09-24 05:11:48,289 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=169613.33333333334, ans=0.1 2026-09-24 05:11:48,762 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=169613.33333333334, ans=0.1 2026-09-24 05:11:52,617 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=169646.66666666666, ans=0.2 2026-09-24 05:12:04,595 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=169746.66666666666, ans=0.1 2026-09-24 05:12:05,050 INFO [train.py:1192] (0/2) Epoch 54, batch 150, loss[loss=0.244, simple_loss=0.347, pruned_loss=0.07049, over 24251.00 frames. ], tot_loss[loss=0.2692, simple_loss=0.3849, pruned_loss=0.07677, over 2557666.70 frames. ], batch size: 125, lr: 3.92e-03, grad_scale: 32.0 2026-09-24 05:12:09,393 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=169746.66666666666, ans=0.125 2026-09-24 05:12:15,794 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=169813.33333333334, ans=0.0 2026-09-24 05:12:16,182 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.710e+02 3.321e+02 3.761e+02 4.264e+02 6.680e+02, threshold=7.522e+02, percent-clipped=0.0 2026-09-24 05:12:27,170 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=169880.0, ans=0.5 2026-09-24 05:12:28,425 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.90 vs. limit=6.0 2026-09-24 05:12:30,902 INFO [train.py:1192] (0/2) Epoch 54, batch 200, loss[loss=0.2946, simple_loss=0.4024, pruned_loss=0.0934, over 21156.00 frames. ], tot_loss[loss=0.2676, simple_loss=0.3832, pruned_loss=0.07596, over 3055233.02 frames. ], batch size: 333, lr: 3.92e-03, grad_scale: 32.0 2026-09-24 05:12:34,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=169913.33333333334, ans=0.125 2026-09-24 05:12:37,540 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=169946.66666666666, ans=0.0 2026-09-24 05:12:40,244 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=169946.66666666666, ans=0.125 2026-09-24 05:12:53,490 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=170046.66666666666, ans=0.0 2026-09-24 05:12:56,743 INFO [train.py:1192] (0/2) Epoch 54, batch 250, loss[loss=0.2969, simple_loss=0.4219, pruned_loss=0.08593, over 24323.00 frames. ], tot_loss[loss=0.2669, simple_loss=0.3825, pruned_loss=0.07567, over 3442977.49 frames. ], batch size: 234, lr: 3.92e-03, grad_scale: 32.0 2026-09-24 05:12:59,326 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=170080.0, ans=0.1 2026-09-24 05:13:02,844 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=170113.33333333334, ans=0.125 2026-09-24 05:13:08,134 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.630e+02 3.409e+02 3.936e+02 4.619e+02 6.254e+02, threshold=7.872e+02, percent-clipped=0.0 2026-09-24 05:13:14,664 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=170180.0, ans=0.125 2026-09-24 05:13:15,154 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=170180.0, ans=0.125 2026-09-24 05:13:22,278 INFO [train.py:1192] (0/2) Epoch 54, batch 300, loss[loss=0.2935, simple_loss=0.4128, pruned_loss=0.08711, over 24539.00 frames. ], tot_loss[loss=0.2659, simple_loss=0.3811, pruned_loss=0.07532, over 3749561.72 frames. ], batch size: 204, lr: 3.91e-03, grad_scale: 32.0 2026-09-24 05:13:24,852 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=170246.66666666666, ans=0.1 2026-09-24 05:13:47,882 INFO [train.py:1192] (0/2) Epoch 54, batch 350, loss[loss=0.2155, simple_loss=0.3241, pruned_loss=0.05343, over 24579.00 frames. ], tot_loss[loss=0.266, simple_loss=0.3816, pruned_loss=0.07526, over 3993040.46 frames. ], batch size: 137, lr: 3.91e-03, grad_scale: 32.0 2026-09-24 05:13:48,007 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=170413.33333333334, ans=0.125 2026-09-24 05:13:59,591 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.512e+02 3.353e+02 3.675e+02 4.247e+02 6.490e+02, threshold=7.351e+02, percent-clipped=0.0 2026-09-24 05:14:01,713 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=170480.0, ans=0.2 2026-09-24 05:14:05,736 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2.whitening_limit, batch_count=170513.33333333334, ans=15.0 2026-09-24 05:14:09,419 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=170546.66666666666, ans=0.2 2026-09-24 05:14:13,364 INFO [train.py:1192] (0/2) Epoch 54, batch 400, loss[loss=0.2774, simple_loss=0.394, pruned_loss=0.08041, over 24561.00 frames. ], tot_loss[loss=0.2646, simple_loss=0.3803, pruned_loss=0.0744, over 4177084.27 frames. ], batch size: 170, lr: 3.91e-03, grad_scale: 32.0 2026-09-24 05:14:17,411 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=170580.0, ans=0.1 2026-09-24 05:14:38,545 INFO [train.py:1192] (0/2) Epoch 54, batch 450, loss[loss=0.276, simple_loss=0.3993, pruned_loss=0.07633, over 24619.00 frames. ], tot_loss[loss=0.2645, simple_loss=0.3805, pruned_loss=0.0743, over 4314097.53 frames. ], batch size: 175, lr: 3.91e-03, grad_scale: 32.0 2026-09-24 05:14:40,680 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=170746.66666666666, ans=0.0 2026-09-24 05:14:41,254 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=170746.66666666666, ans=0.2 2026-09-24 05:14:45,684 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.27 vs. limit=15.0 2026-09-24 05:14:50,272 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.675e+02 3.328e+02 3.837e+02 4.286e+02 5.852e+02, threshold=7.675e+02, percent-clipped=0.0 2026-09-24 05:14:55,497 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=170846.66666666666, ans=0.125 2026-09-24 05:15:00,510 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=170880.0, ans=0.125 2026-09-24 05:15:04,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=170913.33333333334, ans=0.95 2026-09-24 05:15:04,410 INFO [train.py:1192] (0/2) Epoch 54, batch 500, loss[loss=0.2721, simple_loss=0.3959, pruned_loss=0.07412, over 24508.00 frames. ], tot_loss[loss=0.2625, simple_loss=0.3782, pruned_loss=0.07334, over 4430340.64 frames. ], batch size: 218, lr: 3.91e-03, grad_scale: 32.0 2026-09-24 05:15:06,500 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=170913.33333333334, ans=0.125 2026-09-24 05:15:23,926 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=171013.33333333334, ans=0.0 2026-09-24 05:15:25,399 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=171046.66666666666, ans=0.07 2026-09-24 05:15:30,148 INFO [train.py:1192] (0/2) Epoch 54, batch 550, loss[loss=0.2963, simple_loss=0.4164, pruned_loss=0.08814, over 24280.00 frames. ], tot_loss[loss=0.2629, simple_loss=0.379, pruned_loss=0.07344, over 4519678.16 frames. ], batch size: 257, lr: 3.90e-03, grad_scale: 32.0 2026-09-24 05:15:33,841 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=171080.0, ans=0.1 2026-09-24 05:15:41,308 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.598e+02 3.316e+02 3.672e+02 4.016e+02 8.596e+02, threshold=7.345e+02, percent-clipped=1.0 2026-09-24 05:15:44,638 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=171146.66666666666, ans=0.125 2026-09-24 05:15:50,094 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=171213.33333333334, ans=0.0 2026-09-24 05:15:50,584 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=171213.33333333334, ans=0.0 2026-09-24 05:15:56,053 INFO [train.py:1192] (0/2) Epoch 54, batch 600, loss[loss=0.2567, simple_loss=0.3908, pruned_loss=0.06129, over 24344.00 frames. ], tot_loss[loss=0.2636, simple_loss=0.3796, pruned_loss=0.07377, over 4586021.16 frames. ], batch size: 234, lr: 3.90e-03, grad_scale: 32.0 2026-09-24 05:16:03,303 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=171280.0, ans=0.0 2026-09-24 05:16:04,333 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.50 vs. limit=15.0 2026-09-24 05:16:08,807 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=171313.33333333334, ans=0.0 2026-09-24 05:16:11,642 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=171346.66666666666, ans=0.0 2026-09-24 05:16:12,241 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=7.31 vs. limit=12.0 2026-09-24 05:16:18,168 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=4.80 vs. limit=15.0 2026-09-24 05:16:20,177 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=171380.0, ans=0.0 2026-09-24 05:16:21,027 INFO [train.py:1192] (0/2) Epoch 54, batch 650, loss[loss=0.2968, simple_loss=0.4004, pruned_loss=0.09661, over 24573.00 frames. ], tot_loss[loss=0.2626, simple_loss=0.3786, pruned_loss=0.07327, over 4650986.58 frames. ], batch size: 162, lr: 3.90e-03, grad_scale: 32.0 2026-09-24 05:16:26,806 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=21.70 vs. limit=22.5 2026-09-24 05:16:32,625 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.684e+02 3.310e+02 3.698e+02 4.173e+02 6.242e+02, threshold=7.396e+02, percent-clipped=0.0 2026-09-24 05:16:40,072 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=171513.33333333334, ans=0.125 2026-09-24 05:16:40,616 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=171513.33333333334, ans=0.1 2026-09-24 05:16:45,740 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=171546.66666666666, ans=0.95 2026-09-24 05:16:47,141 INFO [train.py:1192] (0/2) Epoch 54, batch 700, loss[loss=0.234, simple_loss=0.3503, pruned_loss=0.05881, over 24566.00 frames. ], tot_loss[loss=0.2633, simple_loss=0.3796, pruned_loss=0.07353, over 4683972.82 frames. ], batch size: 154, lr: 3.90e-03, grad_scale: 32.0 2026-09-24 05:16:51,319 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=171580.0, ans=0.125 2026-09-24 05:16:51,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=171580.0, ans=0.1 2026-09-24 05:16:56,506 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=171613.33333333334, ans=0.025 2026-09-24 05:16:57,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=171646.66666666666, ans=0.125 2026-09-24 05:16:59,797 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.49 vs. limit=6.0 2026-09-24 05:17:03,679 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:17:12,647 INFO [train.py:1192] (0/2) Epoch 54, batch 750, loss[loss=0.275, simple_loss=0.3914, pruned_loss=0.07933, over 24572.00 frames. ], tot_loss[loss=0.2626, simple_loss=0.3786, pruned_loss=0.07334, over 4710706.63 frames. ], batch size: 170, lr: 3.90e-03, grad_scale: 32.0 2026-09-24 05:17:23,450 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=171813.33333333334, ans=0.125 2026-09-24 05:17:23,791 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.549e+02 3.380e+02 3.817e+02 4.305e+02 6.996e+02, threshold=7.634e+02, percent-clipped=0.0 2026-09-24 05:17:24,044 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten.whitening_limit, batch_count=171813.33333333334, ans=22.5 2026-09-24 05:17:25,959 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=171813.33333333334, ans=0.0 2026-09-24 05:17:29,719 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.39 vs. limit=15.0 2026-09-24 05:17:32,603 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.95 vs. limit=15.0 2026-09-24 05:17:38,400 INFO [train.py:1192] (0/2) Epoch 54, batch 800, loss[loss=0.2323, simple_loss=0.3475, pruned_loss=0.05856, over 24536.00 frames. ], tot_loss[loss=0.2621, simple_loss=0.378, pruned_loss=0.07304, over 4736597.44 frames. ], batch size: 137, lr: 3.90e-03, grad_scale: 32.0 2026-09-24 05:17:41,709 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=171913.33333333334, ans=0.125 2026-09-24 05:17:46,950 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.38 vs. limit=15.0 2026-09-24 05:17:52,531 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.37 vs. limit=15.0 2026-09-24 05:18:02,826 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=172046.66666666666, ans=0.125 2026-09-24 05:18:04,224 INFO [train.py:1192] (0/2) Epoch 54, batch 850, loss[loss=0.2559, simple_loss=0.3836, pruned_loss=0.06414, over 24600.00 frames. ], tot_loss[loss=0.2618, simple_loss=0.378, pruned_loss=0.07279, over 4760207.75 frames. ], batch size: 198, lr: 3.89e-03, grad_scale: 32.0 2026-09-24 05:18:08,367 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=172080.0, ans=0.125 2026-09-24 05:18:15,652 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.670e+02 3.373e+02 3.656e+02 4.269e+02 5.742e+02, threshold=7.313e+02, percent-clipped=0.0 2026-09-24 05:18:21,133 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=172180.0, ans=0.0 2026-09-24 05:18:30,019 INFO [train.py:1192] (0/2) Epoch 54, batch 900, loss[loss=0.2316, simple_loss=0.3497, pruned_loss=0.05676, over 24536.00 frames. ], tot_loss[loss=0.2629, simple_loss=0.3788, pruned_loss=0.07348, over 4773163.88 frames. ], batch size: 137, lr: 3.89e-03, grad_scale: 32.0 2026-09-24 05:18:37,052 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=172280.0, ans=0.2 2026-09-24 05:18:44,030 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=172313.33333333334, ans=0.1 2026-09-24 05:18:52,772 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:18:55,144 INFO [train.py:1192] (0/2) Epoch 54, batch 950, loss[loss=0.3694, simple_loss=0.4313, pruned_loss=0.1537, over 12014.00 frames. ], tot_loss[loss=0.2642, simple_loss=0.3784, pruned_loss=0.075, over 4714197.63 frames. ], batch size: 334, lr: 3.89e-03, grad_scale: 32.0 2026-09-24 05:18:59,790 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-54.pt 2026-09-24 05:19:05,385 INFO [train.py:1192] (0/2) Epoch 55, batch 0, loss[loss=0.209, simple_loss=0.3291, pruned_loss=0.04448, over 24541.00 frames. ], tot_loss[loss=0.209, simple_loss=0.3291, pruned_loss=0.04448, over 24541.00 frames. ], batch size: 137, lr: 3.85e-03, grad_scale: 32.0 2026-09-24 05:19:05,385 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 05:19:12,491 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.7069, 3.2712, 3.6453, 3.2668], device='cuda:0') 2026-09-24 05:19:17,035 INFO [train.py:1224] (0/2) Epoch 55, validation: loss=0.1719, simple_loss=0.2905, pruned_loss=0.02667, over 2564189.00 frames. 2026-09-24 05:19:17,035 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 05:19:22,392 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=172473.33333333334, ans=0.125 2026-09-24 05:19:24,274 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.693e+02 3.542e+02 4.212e+02 4.755e+02 6.619e+02, threshold=8.424e+02, percent-clipped=0.0 2026-09-24 05:19:26,045 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=172473.33333333334, ans=0.0 2026-09-24 05:19:28,356 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=172506.66666666666, ans=0.2 2026-09-24 05:19:42,850 INFO [train.py:1192] (0/2) Epoch 55, batch 50, loss[loss=0.2163, simple_loss=0.3257, pruned_loss=0.05345, over 24300.00 frames. ], tot_loss[loss=0.2689, simple_loss=0.3836, pruned_loss=0.07712, over 1075424.58 frames. ], batch size: 125, lr: 3.85e-03, grad_scale: 32.0 2026-09-24 05:19:53,298 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=172673.33333333334, ans=0.125 2026-09-24 05:19:59,331 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=172706.66666666666, ans=0.0 2026-09-24 05:20:08,618 INFO [train.py:1192] (0/2) Epoch 55, batch 100, loss[loss=0.2787, simple_loss=0.3894, pruned_loss=0.08395, over 24589.00 frames. ], tot_loss[loss=0.2734, simple_loss=0.3889, pruned_loss=0.07894, over 1904281.91 frames. ], batch size: 154, lr: 3.85e-03, grad_scale: 64.0 2026-09-24 05:20:14,357 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=172806.66666666666, ans=0.5 2026-09-24 05:20:15,919 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.943e+02 3.406e+02 3.885e+02 4.242e+02 5.650e+02, threshold=7.770e+02, percent-clipped=0.0 2026-09-24 05:20:15,998 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=172806.66666666666, ans=0.025 2026-09-24 05:20:16,546 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=172806.66666666666, ans=0.0 2026-09-24 05:20:20,791 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=172840.0, ans=0.0 2026-09-24 05:20:25,674 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=172873.33333333334, ans=0.0 2026-09-24 05:20:28,689 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=172906.66666666666, ans=0.1 2026-09-24 05:20:30,725 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=172906.66666666666, ans=0.0 2026-09-24 05:20:30,734 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=172906.66666666666, ans=0.125 2026-09-24 05:20:33,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=172906.66666666666, ans=0.0 2026-09-24 05:20:33,186 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=172906.66666666666, ans=0.07 2026-09-24 05:20:34,109 INFO [train.py:1192] (0/2) Epoch 55, batch 150, loss[loss=0.2149, simple_loss=0.3223, pruned_loss=0.05374, over 24242.00 frames. ], tot_loss[loss=0.2673, simple_loss=0.3833, pruned_loss=0.07564, over 2557853.43 frames. ], batch size: 125, lr: 3.85e-03, grad_scale: 64.0 2026-09-24 05:20:38,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=172973.33333333334, ans=0.07 2026-09-24 05:20:49,407 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.25 vs. limit=22.5 2026-09-24 05:20:59,611 INFO [train.py:1192] (0/2) Epoch 55, batch 200, loss[loss=0.3146, simple_loss=0.4162, pruned_loss=0.1066, over 21103.00 frames. ], tot_loss[loss=0.2665, simple_loss=0.3823, pruned_loss=0.07536, over 3056366.66 frames. ], batch size: 333, lr: 3.85e-03, grad_scale: 64.0 2026-09-24 05:21:07,127 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.318e+02 3.421e+02 3.801e+02 4.436e+02 7.383e+02, threshold=7.602e+02, percent-clipped=0.0 2026-09-24 05:21:20,077 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=173240.0, ans=0.125 2026-09-24 05:21:20,594 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=173240.0, ans=0.1 2026-09-24 05:21:21,770 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.02 vs. limit=22.5 2026-09-24 05:21:24,966 INFO [train.py:1192] (0/2) Epoch 55, batch 250, loss[loss=0.2777, simple_loss=0.4031, pruned_loss=0.07613, over 24308.00 frames. ], tot_loss[loss=0.2651, simple_loss=0.3809, pruned_loss=0.07471, over 3446157.00 frames. ], batch size: 234, lr: 3.84e-03, grad_scale: 64.0 2026-09-24 05:21:25,076 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=173273.33333333334, ans=0.0 2026-09-24 05:21:30,257 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=173306.66666666666, ans=0.125 2026-09-24 05:21:33,085 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=173306.66666666666, ans=0.125 2026-09-24 05:21:34,059 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-52000.pt 2026-09-24 05:21:39,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=173340.0, ans=0.125 2026-09-24 05:21:51,004 INFO [train.py:1192] (0/2) Epoch 55, batch 300, loss[loss=0.2801, simple_loss=0.3978, pruned_loss=0.08121, over 24531.00 frames. ], tot_loss[loss=0.2647, simple_loss=0.3799, pruned_loss=0.07473, over 3750586.30 frames. ], batch size: 204, lr: 3.84e-03, grad_scale: 64.0 2026-09-24 05:21:58,132 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.734e+02 3.486e+02 3.863e+02 4.417e+02 5.821e+02, threshold=7.726e+02, percent-clipped=0.0 2026-09-24 05:22:12,106 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=173573.33333333334, ans=0.125 2026-09-24 05:22:16,130 INFO [train.py:1192] (0/2) Epoch 55, batch 350, loss[loss=0.2346, simple_loss=0.3407, pruned_loss=0.06428, over 24560.00 frames. ], tot_loss[loss=0.2643, simple_loss=0.3803, pruned_loss=0.07421, over 3990078.30 frames. ], batch size: 137, lr: 3.84e-03, grad_scale: 64.0 2026-09-24 05:22:25,848 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=173673.33333333334, ans=0.0 2026-09-24 05:22:32,648 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass_mid.scale_min, batch_count=173706.66666666666, ans=0.2 2026-09-24 05:22:36,936 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.64 vs. limit=8.0 2026-09-24 05:22:37,297 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.36 vs. limit=15.0 2026-09-24 05:22:37,633 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=173740.0, ans=0.125 2026-09-24 05:22:38,117 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=173740.0, ans=0.0 2026-09-24 05:22:40,400 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=173740.0, ans=0.0 2026-09-24 05:22:41,734 INFO [train.py:1192] (0/2) Epoch 55, batch 400, loss[loss=0.2765, simple_loss=0.3898, pruned_loss=0.08166, over 24578.00 frames. ], tot_loss[loss=0.2636, simple_loss=0.3796, pruned_loss=0.07381, over 4177531.48 frames. ], batch size: 170, lr: 3.84e-03, grad_scale: 64.0 2026-09-24 05:22:48,944 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.557e+02 3.486e+02 3.848e+02 4.662e+02 6.190e+02, threshold=7.697e+02, percent-clipped=0.0 2026-09-24 05:22:57,510 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=173873.33333333334, ans=0.0 2026-09-24 05:23:00,211 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=173873.33333333334, ans=0.2 2026-09-24 05:23:06,460 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=173906.66666666666, ans=0.0 2026-09-24 05:23:07,716 INFO [train.py:1192] (0/2) Epoch 55, batch 450, loss[loss=0.2953, simple_loss=0.4083, pruned_loss=0.09119, over 24611.00 frames. ], tot_loss[loss=0.2648, simple_loss=0.3805, pruned_loss=0.07455, over 4309501.89 frames. ], batch size: 175, lr: 3.84e-03, grad_scale: 64.0 2026-09-24 05:23:14,722 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=173973.33333333334, ans=0.04949747468305833 2026-09-24 05:23:22,226 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.89 vs. limit=15.0 2026-09-24 05:23:32,424 INFO [train.py:1192] (0/2) Epoch 55, batch 500, loss[loss=0.2674, simple_loss=0.3944, pruned_loss=0.07018, over 24478.00 frames. ], tot_loss[loss=0.2626, simple_loss=0.3784, pruned_loss=0.07339, over 4426730.77 frames. ], batch size: 218, lr: 3.84e-03, grad_scale: 64.0 2026-09-24 05:23:34,478 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=174106.66666666666, ans=0.0 2026-09-24 05:23:35,813 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=4.75 vs. limit=15.0 2026-09-24 05:23:39,755 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.453e+02 3.364e+02 3.704e+02 4.452e+02 5.986e+02, threshold=7.409e+02, percent-clipped=0.0 2026-09-24 05:23:40,505 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=16.14 vs. limit=22.5 2026-09-24 05:23:52,936 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=174240.0, ans=0.125 2026-09-24 05:23:58,230 INFO [train.py:1192] (0/2) Epoch 55, batch 550, loss[loss=0.257, simple_loss=0.3872, pruned_loss=0.06343, over 24301.00 frames. ], tot_loss[loss=0.2628, simple_loss=0.3788, pruned_loss=0.0734, over 4516379.78 frames. ], batch size: 257, lr: 3.83e-03, grad_scale: 64.0 2026-09-24 05:24:09,027 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.26 vs. limit=15.0 2026-09-24 05:24:22,426 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=174406.66666666666, ans=0.0 2026-09-24 05:24:22,940 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=174406.66666666666, ans=0.125 2026-09-24 05:24:24,183 INFO [train.py:1192] (0/2) Epoch 55, batch 600, loss[loss=0.2865, simple_loss=0.4125, pruned_loss=0.08028, over 24324.00 frames. ], tot_loss[loss=0.264, simple_loss=0.3801, pruned_loss=0.07394, over 4583587.77 frames. ], batch size: 234, lr: 3.83e-03, grad_scale: 32.0 2026-09-24 05:24:28,367 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=174440.0, ans=0.125 2026-09-24 05:24:31,742 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.760e+02 3.340e+02 3.684e+02 4.234e+02 5.839e+02, threshold=7.369e+02, percent-clipped=0.0 2026-09-24 05:24:32,387 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=174473.33333333334, ans=0.125 2026-09-24 05:24:32,412 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:24:37,257 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=174506.66666666666, ans=0.0 2026-09-24 05:24:38,543 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=174506.66666666666, ans=0.0 2026-09-24 05:24:41,547 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=174540.0, ans=0.125 2026-09-24 05:24:46,974 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=174573.33333333334, ans=0.125 2026-09-24 05:24:49,157 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=174573.33333333334, ans=0.125 2026-09-24 05:24:50,237 INFO [train.py:1192] (0/2) Epoch 55, batch 650, loss[loss=0.2856, simple_loss=0.3969, pruned_loss=0.08719, over 24560.00 frames. ], tot_loss[loss=0.2627, simple_loss=0.3788, pruned_loss=0.07332, over 4649263.21 frames. ], batch size: 162, lr: 3.83e-03, grad_scale: 32.0 2026-09-24 05:25:03,456 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=174673.33333333334, ans=0.07 2026-09-24 05:25:03,871 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=174673.33333333334, ans=0.2 2026-09-24 05:25:05,969 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.15 vs. limit=15.0 2026-09-24 05:25:15,225 INFO [train.py:1192] (0/2) Epoch 55, batch 700, loss[loss=0.249, simple_loss=0.3636, pruned_loss=0.0672, over 24603.00 frames. ], tot_loss[loss=0.2628, simple_loss=0.3793, pruned_loss=0.07313, over 4682289.79 frames. ], batch size: 154, lr: 3.83e-03, grad_scale: 32.0 2026-09-24 05:25:18,507 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=174773.33333333334, ans=0.125 2026-09-24 05:25:22,403 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.835e+02 3.386e+02 3.666e+02 3.991e+02 5.815e+02, threshold=7.332e+02, percent-clipped=0.0 2026-09-24 05:25:28,833 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=174840.0, ans=0.1 2026-09-24 05:25:33,155 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=174873.33333333334, ans=0.95 2026-09-24 05:25:34,559 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=174906.66666666666, ans=0.0 2026-09-24 05:25:40,795 INFO [train.py:1192] (0/2) Epoch 55, batch 750, loss[loss=0.2571, simple_loss=0.3793, pruned_loss=0.06746, over 24547.00 frames. ], tot_loss[loss=0.2621, simple_loss=0.3783, pruned_loss=0.07291, over 4709740.32 frames. ], batch size: 170, lr: 3.83e-03, grad_scale: 32.0 2026-09-24 05:25:44,366 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=174940.0, ans=0.2 2026-09-24 05:26:06,493 INFO [train.py:1192] (0/2) Epoch 55, batch 800, loss[loss=0.228, simple_loss=0.3397, pruned_loss=0.05822, over 24581.00 frames. ], tot_loss[loss=0.2614, simple_loss=0.3778, pruned_loss=0.07253, over 4735272.53 frames. ], batch size: 137, lr: 3.82e-03, grad_scale: 32.0 2026-09-24 05:26:10,931 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=175106.66666666666, ans=0.0 2026-09-24 05:26:14,557 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.433e+02 3.429e+02 3.889e+02 4.225e+02 6.137e+02, threshold=7.777e+02, percent-clipped=0.0 2026-09-24 05:26:17,089 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=175173.33333333334, ans=0.125 2026-09-24 05:26:18,485 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.37 vs. limit=6.0 2026-09-24 05:26:18,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=175173.33333333334, ans=0.125 2026-09-24 05:26:20,665 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.86 vs. limit=15.0 2026-09-24 05:26:22,397 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=175206.66666666666, ans=0.125 2026-09-24 05:26:24,820 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=175206.66666666666, ans=0.0 2026-09-24 05:26:29,442 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=175240.0, ans=0.125 2026-09-24 05:26:29,961 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.62 vs. limit=12.0 2026-09-24 05:26:32,234 INFO [train.py:1192] (0/2) Epoch 55, batch 850, loss[loss=0.2647, simple_loss=0.3897, pruned_loss=0.0699, over 24594.00 frames. ], tot_loss[loss=0.2623, simple_loss=0.3783, pruned_loss=0.07317, over 4757944.12 frames. ], batch size: 198, lr: 3.82e-03, grad_scale: 32.0 2026-09-24 05:26:35,444 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=175273.33333333334, ans=0.0 2026-09-24 05:26:36,977 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:26:45,788 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=175340.0, ans=0.025 2026-09-24 05:26:47,447 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=175373.33333333334, ans=0.125 2026-09-24 05:26:48,754 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=175373.33333333334, ans=0.2 2026-09-24 05:26:52,532 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.41 vs. limit=15.0 2026-09-24 05:26:57,632 INFO [train.py:1192] (0/2) Epoch 55, batch 900, loss[loss=0.2485, simple_loss=0.3604, pruned_loss=0.06828, over 24539.00 frames. ], tot_loss[loss=0.2621, simple_loss=0.3785, pruned_loss=0.07287, over 4771806.53 frames. ], batch size: 137, lr: 3.82e-03, grad_scale: 32.0 2026-09-24 05:27:00,595 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=175440.0, ans=0.2 2026-09-24 05:27:05,926 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.592e+02 3.331e+02 3.802e+02 4.333e+02 6.243e+02, threshold=7.604e+02, percent-clipped=0.0 2026-09-24 05:27:15,138 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=175540.0, ans=0.0 2026-09-24 05:27:15,563 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=175540.0, ans=0.04949747468305833 2026-09-24 05:27:19,680 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.max_positive, batch_count=175573.33333333334, ans=0.95 2026-09-24 05:27:23,441 INFO [train.py:1192] (0/2) Epoch 55, batch 950, loss[loss=0.3963, simple_loss=0.4536, pruned_loss=0.1695, over 11704.00 frames. ], tot_loss[loss=0.2627, simple_loss=0.3774, pruned_loss=0.074, over 4714006.17 frames. ], batch size: 333, lr: 3.82e-03, grad_scale: 32.0 2026-09-24 05:27:27,934 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-55.pt 2026-09-24 05:27:33,773 INFO [train.py:1192] (0/2) Epoch 56, batch 0, loss[loss=0.2177, simple_loss=0.3363, pruned_loss=0.0495, over 24536.00 frames. ], tot_loss[loss=0.2177, simple_loss=0.3363, pruned_loss=0.0495, over 24536.00 frames. ], batch size: 137, lr: 3.78e-03, grad_scale: 32.0 2026-09-24 05:27:33,773 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 05:27:42,989 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.9158, 2.2804, 2.4581, 2.0740, 2.4528, 2.4901, 2.5115, 2.1894], device='cuda:0') 2026-09-24 05:27:45,130 INFO [train.py:1224] (0/2) Epoch 56, validation: loss=0.1714, simple_loss=0.2901, pruned_loss=0.02629, over 2564189.00 frames. 2026-09-24 05:27:45,130 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 05:27:48,001 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=175633.33333333334, ans=0.2 2026-09-24 05:27:58,222 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=175700.0, ans=0.0 2026-09-24 05:28:04,741 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=175733.33333333334, ans=0.09899494936611666 2026-09-24 05:28:07,613 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=175766.66666666666, ans=0.125 2026-09-24 05:28:10,397 INFO [train.py:1192] (0/2) Epoch 56, batch 50, loss[loss=0.2279, simple_loss=0.3342, pruned_loss=0.06078, over 24259.00 frames. ], tot_loss[loss=0.27, simple_loss=0.3844, pruned_loss=0.0778, over 1076473.93 frames. ], batch size: 125, lr: 3.78e-03, grad_scale: 32.0 2026-09-24 05:28:13,967 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.642e+02 3.572e+02 4.045e+02 4.731e+02 9.270e+02, threshold=8.089e+02, percent-clipped=2.0 2026-09-24 05:28:16,340 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=175833.33333333334, ans=0.125 2026-09-24 05:28:23,341 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=175866.66666666666, ans=0.125 2026-09-24 05:28:35,526 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=175933.33333333334, ans=0.1 2026-09-24 05:28:36,933 INFO [train.py:1192] (0/2) Epoch 56, batch 100, loss[loss=0.2506, simple_loss=0.3666, pruned_loss=0.06736, over 24595.00 frames. ], tot_loss[loss=0.2727, simple_loss=0.3886, pruned_loss=0.07846, over 1905775.98 frames. ], batch size: 154, lr: 3.78e-03, grad_scale: 32.0 2026-09-24 05:28:38,118 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:28:51,341 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=176033.33333333334, ans=0.1 2026-09-24 05:28:57,168 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=176100.0, ans=0.125 2026-09-24 05:28:58,080 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=176100.0, ans=0.2 2026-09-24 05:29:02,509 INFO [train.py:1192] (0/2) Epoch 56, batch 150, loss[loss=0.2176, simple_loss=0.3243, pruned_loss=0.05543, over 24270.00 frames. ], tot_loss[loss=0.2679, simple_loss=0.3836, pruned_loss=0.0761, over 2559247.51 frames. ], batch size: 125, lr: 3.78e-03, grad_scale: 32.0 2026-09-24 05:29:05,893 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.706e+02 3.375e+02 3.744e+02 4.234e+02 6.625e+02, threshold=7.487e+02, percent-clipped=0.0 2026-09-24 05:29:08,216 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=176166.66666666666, ans=0.125 2026-09-24 05:29:08,696 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=176166.66666666666, ans=0.125 2026-09-24 05:29:27,742 INFO [train.py:1192] (0/2) Epoch 56, batch 200, loss[loss=0.3245, simple_loss=0.4235, pruned_loss=0.1127, over 21081.00 frames. ], tot_loss[loss=0.2659, simple_loss=0.3818, pruned_loss=0.07503, over 3055300.74 frames. ], batch size: 333, lr: 3.78e-03, grad_scale: 32.0 2026-09-24 05:29:27,835 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=176300.0, ans=0.125 2026-09-24 05:29:31,740 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.44 vs. limit=15.0 2026-09-24 05:29:39,481 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=176366.66666666666, ans=0.125 2026-09-24 05:29:40,844 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=176366.66666666666, ans=0.0 2026-09-24 05:29:43,764 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=176400.0, ans=0.04949747468305833 2026-09-24 05:29:49,421 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=176433.33333333334, ans=0.125 2026-09-24 05:29:53,237 INFO [train.py:1192] (0/2) Epoch 56, batch 250, loss[loss=0.2999, simple_loss=0.4173, pruned_loss=0.09127, over 24313.00 frames. ], tot_loss[loss=0.2644, simple_loss=0.3807, pruned_loss=0.07403, over 3445755.81 frames. ], batch size: 234, lr: 3.78e-03, grad_scale: 32.0 2026-09-24 05:29:56,600 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.70 vs. limit=6.0 2026-09-24 05:29:57,044 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.699e+02 3.326e+02 3.793e+02 4.270e+02 6.257e+02, threshold=7.587e+02, percent-clipped=0.0 2026-09-24 05:30:00,946 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=176500.0, ans=0.2 2026-09-24 05:30:01,783 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=176500.0, ans=0.125 2026-09-24 05:30:04,951 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.14 vs. limit=15.0 2026-09-24 05:30:16,834 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=4.68 vs. limit=15.0 2026-09-24 05:30:18,516 INFO [train.py:1192] (0/2) Epoch 56, batch 300, loss[loss=0.2967, simple_loss=0.4075, pruned_loss=0.09291, over 24526.00 frames. ], tot_loss[loss=0.2635, simple_loss=0.3795, pruned_loss=0.07378, over 3750782.86 frames. ], batch size: 204, lr: 3.77e-03, grad_scale: 32.0 2026-09-24 05:30:29,815 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=176700.0, ans=0.1 2026-09-24 05:30:30,856 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=176700.0, ans=0.125 2026-09-24 05:30:34,226 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=176733.33333333334, ans=0.07 2026-09-24 05:30:43,697 INFO [train.py:1192] (0/2) Epoch 56, batch 350, loss[loss=0.2142, simple_loss=0.3272, pruned_loss=0.05062, over 24559.00 frames. ], tot_loss[loss=0.264, simple_loss=0.38, pruned_loss=0.07397, over 3991627.59 frames. ], batch size: 137, lr: 3.77e-03, grad_scale: 32.0 2026-09-24 05:30:46,782 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=176800.0, ans=0.04949747468305833 2026-09-24 05:30:47,106 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.508e+02 3.265e+02 3.602e+02 4.252e+02 5.165e+02, threshold=7.204e+02, percent-clipped=0.0 2026-09-24 05:30:47,413 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.20 vs. limit=15.0 2026-09-24 05:30:55,793 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.00 vs. limit=6.0 2026-09-24 05:30:56,271 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=176866.66666666666, ans=0.2 2026-09-24 05:30:56,789 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=176866.66666666666, ans=0.025 2026-09-24 05:31:06,690 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=176933.33333333334, ans=0.0 2026-09-24 05:31:09,370 INFO [train.py:1192] (0/2) Epoch 56, batch 400, loss[loss=0.2911, simple_loss=0.3963, pruned_loss=0.09302, over 24585.00 frames. ], tot_loss[loss=0.2628, simple_loss=0.3789, pruned_loss=0.07333, over 4175139.76 frames. ], batch size: 170, lr: 3.77e-03, grad_scale: 32.0 2026-09-24 05:31:19,049 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.51 vs. limit=22.5 2026-09-24 05:31:22,652 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=177033.33333333334, ans=0.125 2026-09-24 05:31:27,107 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:31:34,647 INFO [train.py:1192] (0/2) Epoch 56, batch 450, loss[loss=0.2807, simple_loss=0.4045, pruned_loss=0.07852, over 24613.00 frames. ], tot_loss[loss=0.2632, simple_loss=0.3793, pruned_loss=0.0736, over 4309055.04 frames. ], batch size: 175, lr: 3.77e-03, grad_scale: 32.0 2026-09-24 05:31:37,866 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.807e+02 3.424e+02 3.853e+02 4.456e+02 6.389e+02, threshold=7.705e+02, percent-clipped=0.0 2026-09-24 05:31:42,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=177166.66666666666, ans=0.125 2026-09-24 05:31:48,064 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=177200.0, ans=0.1 2026-09-24 05:31:48,257 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.97 vs. limit=22.5 2026-09-24 05:31:57,497 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=177266.66666666666, ans=0.2 2026-09-24 05:31:59,294 INFO [train.py:1192] (0/2) Epoch 56, batch 500, loss[loss=0.2757, simple_loss=0.3942, pruned_loss=0.07854, over 24517.00 frames. ], tot_loss[loss=0.2615, simple_loss=0.3775, pruned_loss=0.07277, over 4426540.43 frames. ], batch size: 218, lr: 3.77e-03, grad_scale: 32.0 2026-09-24 05:32:05,037 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=177333.33333333334, ans=0.0 2026-09-24 05:32:15,270 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.54 vs. limit=15.0 2026-09-24 05:32:24,903 INFO [train.py:1192] (0/2) Epoch 56, batch 550, loss[loss=0.2826, simple_loss=0.4089, pruned_loss=0.07818, over 24270.00 frames. ], tot_loss[loss=0.2622, simple_loss=0.3782, pruned_loss=0.07313, over 4516513.43 frames. ], batch size: 257, lr: 3.76e-03, grad_scale: 32.0 2026-09-24 05:32:28,151 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.644e+02 3.338e+02 3.718e+02 4.019e+02 5.234e+02, threshold=7.435e+02, percent-clipped=0.0 2026-09-24 05:32:31,710 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.60 vs. limit=6.0 2026-09-24 05:32:32,542 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=177500.0, ans=0.1 2026-09-24 05:32:44,580 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=177566.66666666666, ans=0.125 2026-09-24 05:32:47,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=177600.0, ans=0.125 2026-09-24 05:32:48,253 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=177600.0, ans=0.1 2026-09-24 05:32:50,685 INFO [train.py:1192] (0/2) Epoch 56, batch 600, loss[loss=0.2949, simple_loss=0.4158, pruned_loss=0.08704, over 24421.00 frames. ], tot_loss[loss=0.2629, simple_loss=0.379, pruned_loss=0.07338, over 4582599.00 frames. ], batch size: 235, lr: 3.76e-03, grad_scale: 32.0 2026-09-24 05:32:53,804 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.48 vs. limit=15.0 2026-09-24 05:33:00,339 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=177666.66666666666, ans=0.125 2026-09-24 05:33:08,831 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=177733.33333333334, ans=0.125 2026-09-24 05:33:12,250 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=177766.66666666666, ans=0.125 2026-09-24 05:33:16,211 INFO [train.py:1192] (0/2) Epoch 56, batch 650, loss[loss=0.2568, simple_loss=0.3752, pruned_loss=0.06925, over 24556.00 frames. ], tot_loss[loss=0.2617, simple_loss=0.3782, pruned_loss=0.0726, over 4648355.81 frames. ], batch size: 162, lr: 3.76e-03, grad_scale: 32.0 2026-09-24 05:33:16,313 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=177800.0, ans=0.0 2026-09-24 05:33:17,186 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=177800.0, ans=0.125 2026-09-24 05:33:17,249 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=177800.0, ans=0.09899494936611666 2026-09-24 05:33:19,819 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.562e+02 3.403e+02 3.720e+02 4.184e+02 6.335e+02, threshold=7.440e+02, percent-clipped=0.0 2026-09-24 05:33:22,140 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=177833.33333333334, ans=0.2 2026-09-24 05:33:27,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=177866.66666666666, ans=0.0 2026-09-24 05:33:28,860 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=177866.66666666666, ans=0.0 2026-09-24 05:33:32,222 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=177900.0, ans=0.0 2026-09-24 05:33:32,663 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=177900.0, ans=0.0 2026-09-24 05:33:38,426 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.17 vs. limit=15.0 2026-09-24 05:33:42,131 INFO [train.py:1192] (0/2) Epoch 56, batch 700, loss[loss=0.2657, simple_loss=0.3715, pruned_loss=0.07991, over 24554.00 frames. ], tot_loss[loss=0.2625, simple_loss=0.3791, pruned_loss=0.073, over 4680726.11 frames. ], batch size: 154, lr: 3.76e-03, grad_scale: 32.0 2026-09-24 05:33:56,552 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:34:03,656 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.16 vs. limit=15.0 2026-09-24 05:34:07,517 INFO [train.py:1192] (0/2) Epoch 56, batch 750, loss[loss=0.2652, simple_loss=0.3843, pruned_loss=0.073, over 24555.00 frames. ], tot_loss[loss=0.2617, simple_loss=0.378, pruned_loss=0.07269, over 4712060.37 frames. ], batch size: 170, lr: 3.76e-03, grad_scale: 32.0 2026-09-24 05:34:11,078 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.813e+02 3.441e+02 3.869e+02 4.291e+02 6.518e+02, threshold=7.738e+02, percent-clipped=0.0 2026-09-24 05:34:11,586 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=178133.33333333334, ans=0.125 2026-09-24 05:34:16,984 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.99 vs. limit=6.0 2026-09-24 05:34:31,005 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=178266.66666666666, ans=0.025 2026-09-24 05:34:32,697 INFO [train.py:1192] (0/2) Epoch 56, batch 800, loss[loss=0.2297, simple_loss=0.3443, pruned_loss=0.05758, over 24552.00 frames. ], tot_loss[loss=0.2606, simple_loss=0.3773, pruned_loss=0.07197, over 4736120.20 frames. ], batch size: 137, lr: 3.76e-03, grad_scale: 32.0 2026-09-24 05:34:48,044 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:34:58,004 INFO [train.py:1192] (0/2) Epoch 56, batch 850, loss[loss=0.3053, simple_loss=0.4206, pruned_loss=0.09505, over 24572.00 frames. ], tot_loss[loss=0.2602, simple_loss=0.377, pruned_loss=0.07168, over 4758704.63 frames. ], batch size: 198, lr: 3.75e-03, grad_scale: 32.0 2026-09-24 05:34:59,834 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:35:01,691 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.451e+02 3.249e+02 3.700e+02 4.107e+02 5.695e+02, threshold=7.400e+02, percent-clipped=0.0 2026-09-24 05:35:18,607 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.10 vs. limit=10.0 2026-09-24 05:35:19,440 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=178600.0, ans=0.2 2026-09-24 05:35:23,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=178633.33333333334, ans=0.125 2026-09-24 05:35:23,077 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=178633.33333333334, ans=0.125 2026-09-24 05:35:23,536 INFO [train.py:1192] (0/2) Epoch 56, batch 900, loss[loss=0.2078, simple_loss=0.3319, pruned_loss=0.04185, over 24597.00 frames. ], tot_loss[loss=0.2603, simple_loss=0.3771, pruned_loss=0.07173, over 4772683.48 frames. ], batch size: 137, lr: 3.75e-03, grad_scale: 32.0 2026-09-24 05:35:29,907 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.48 vs. limit=15.0 2026-09-24 05:35:31,762 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=178666.66666666666, ans=0.125 2026-09-24 05:35:47,376 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=178766.66666666666, ans=0.2 2026-09-24 05:35:49,015 INFO [train.py:1192] (0/2) Epoch 56, batch 950, loss[loss=0.3782, simple_loss=0.4387, pruned_loss=0.1588, over 11124.00 frames. ], tot_loss[loss=0.2622, simple_loss=0.3772, pruned_loss=0.07363, over 4713444.12 frames. ], batch size: 333, lr: 3.75e-03, grad_scale: 32.0 2026-09-24 05:35:52,352 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.987e+02 3.455e+02 4.002e+02 4.893e+02 8.245e+02, threshold=8.003e+02, percent-clipped=1.0 2026-09-24 05:35:53,388 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-56.pt 2026-09-24 05:35:59,334 INFO [train.py:1192] (0/2) Epoch 57, batch 0, loss[loss=0.2267, simple_loss=0.3491, pruned_loss=0.05219, over 24558.00 frames. ], tot_loss[loss=0.2267, simple_loss=0.3491, pruned_loss=0.05219, over 24558.00 frames. ], batch size: 137, lr: 3.72e-03, grad_scale: 32.0 2026-09-24 05:35:59,334 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 05:36:10,803 INFO [train.py:1224] (0/2) Epoch 57, validation: loss=0.1707, simple_loss=0.2894, pruned_loss=0.02596, over 2564189.00 frames. 2026-09-24 05:36:10,804 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 05:36:24,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=178893.33333333334, ans=0.1 2026-09-24 05:36:32,153 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=178960.0, ans=0.125 2026-09-24 05:36:36,822 INFO [train.py:1192] (0/2) Epoch 57, batch 50, loss[loss=0.2233, simple_loss=0.3324, pruned_loss=0.05709, over 24231.00 frames. ], tot_loss[loss=0.2664, simple_loss=0.3819, pruned_loss=0.07541, over 1075106.53 frames. ], batch size: 125, lr: 3.72e-03, grad_scale: 32.0 2026-09-24 05:36:41,032 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=178993.33333333334, ans=0.125 2026-09-24 05:36:41,554 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=179026.66666666666, ans=0.0 2026-09-24 05:36:52,831 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=179093.33333333334, ans=0.0 2026-09-24 05:36:54,514 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=179093.33333333334, ans=0.0 2026-09-24 05:36:56,065 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=179093.33333333334, ans=0.125 2026-09-24 05:37:01,832 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.678e+02 3.460e+02 3.786e+02 4.432e+02 6.420e+02, threshold=7.573e+02, percent-clipped=0.0 2026-09-24 05:37:02,588 INFO [train.py:1192] (0/2) Epoch 57, batch 100, loss[loss=0.2802, simple_loss=0.3852, pruned_loss=0.08763, over 24620.00 frames. ], tot_loss[loss=0.2715, simple_loss=0.3873, pruned_loss=0.07781, over 1903183.57 frames. ], batch size: 154, lr: 3.71e-03, grad_scale: 32.0 2026-09-24 05:37:18,956 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.08 vs. limit=6.0 2026-09-24 05:37:21,433 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=179260.0, ans=0.125 2026-09-24 05:37:22,941 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=179293.33333333334, ans=0.2 2026-09-24 05:37:27,577 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=179326.66666666666, ans=0.125 2026-09-24 05:37:27,900 INFO [train.py:1192] (0/2) Epoch 57, batch 150, loss[loss=0.2075, simple_loss=0.3291, pruned_loss=0.04291, over 24243.00 frames. ], tot_loss[loss=0.2653, simple_loss=0.382, pruned_loss=0.07432, over 2557123.85 frames. ], batch size: 125, lr: 3.71e-03, grad_scale: 32.0 2026-09-24 05:37:35,560 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=179360.0, ans=0.025 2026-09-24 05:37:41,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=179393.33333333334, ans=0.125 2026-09-24 05:37:53,207 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.516e+02 3.407e+02 3.900e+02 4.333e+02 6.747e+02, threshold=7.800e+02, percent-clipped=0.0 2026-09-24 05:37:53,700 INFO [train.py:1192] (0/2) Epoch 57, batch 200, loss[loss=0.3047, simple_loss=0.4099, pruned_loss=0.09976, over 21063.00 frames. ], tot_loss[loss=0.2639, simple_loss=0.3804, pruned_loss=0.07372, over 3054397.30 frames. ], batch size: 333, lr: 3.71e-03, grad_scale: 32.0 2026-09-24 05:37:56,775 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=179493.33333333334, ans=0.125 2026-09-24 05:37:58,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=179526.66666666666, ans=0.04949747468305833 2026-09-24 05:37:59,826 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=179526.66666666666, ans=0.035 2026-09-24 05:38:02,213 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.46 vs. limit=15.0 2026-09-24 05:38:11,220 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=179593.33333333334, ans=0.2 2026-09-24 05:38:18,998 INFO [train.py:1192] (0/2) Epoch 57, batch 250, loss[loss=0.2733, simple_loss=0.4028, pruned_loss=0.07192, over 24313.00 frames. ], tot_loss[loss=0.263, simple_loss=0.3795, pruned_loss=0.07328, over 3445630.47 frames. ], batch size: 234, lr: 3.71e-03, grad_scale: 32.0 2026-09-24 05:38:19,112 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=179660.0, ans=0.2 2026-09-24 05:38:25,988 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_ff2.min_abs, batch_count=179693.33333333334, ans=0.1 2026-09-24 05:38:43,576 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.470e+02 3.240e+02 3.731e+02 4.326e+02 5.664e+02, threshold=7.462e+02, percent-clipped=0.0 2026-09-24 05:38:44,049 INFO [train.py:1192] (0/2) Epoch 57, batch 300, loss[loss=0.2603, simple_loss=0.3858, pruned_loss=0.06741, over 24565.00 frames. ], tot_loss[loss=0.2618, simple_loss=0.3781, pruned_loss=0.07277, over 3749769.94 frames. ], batch size: 204, lr: 3.71e-03, grad_scale: 32.0 2026-09-24 05:38:48,958 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=179860.0, ans=0.125 2026-09-24 05:39:02,645 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=179926.66666666666, ans=0.125 2026-09-24 05:39:10,028 INFO [train.py:1192] (0/2) Epoch 57, batch 350, loss[loss=0.2284, simple_loss=0.3409, pruned_loss=0.05801, over 24567.00 frames. ], tot_loss[loss=0.2632, simple_loss=0.3795, pruned_loss=0.07349, over 3992705.18 frames. ], batch size: 137, lr: 3.71e-03, grad_scale: 32.0 2026-09-24 05:39:12,592 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=179993.33333333334, ans=0.125 2026-09-24 05:39:28,897 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=180093.33333333334, ans=0.125 2026-09-24 05:39:31,699 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 05:39:35,104 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.864e+02 3.559e+02 4.088e+02 4.633e+02 6.508e+02, threshold=8.176e+02, percent-clipped=0.0 2026-09-24 05:39:35,581 INFO [train.py:1192] (0/2) Epoch 57, batch 400, loss[loss=0.2745, simple_loss=0.3868, pruned_loss=0.08115, over 24549.00 frames. ], tot_loss[loss=0.2627, simple_loss=0.3789, pruned_loss=0.07326, over 4176051.91 frames. ], batch size: 170, lr: 3.70e-03, grad_scale: 32.0 2026-09-24 05:39:41,099 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=180193.33333333334, ans=0.1 2026-09-24 05:39:41,167 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.prob, batch_count=180193.33333333334, ans=0.125 2026-09-24 05:40:00,533 INFO [train.py:1192] (0/2) Epoch 57, batch 450, loss[loss=0.2715, simple_loss=0.3891, pruned_loss=0.07692, over 24642.00 frames. ], tot_loss[loss=0.2625, simple_loss=0.3787, pruned_loss=0.07311, over 4310628.01 frames. ], batch size: 175, lr: 3.70e-03, grad_scale: 32.0 2026-09-24 05:40:02,181 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=180326.66666666666, ans=0.0 2026-09-24 05:40:12,552 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.59 vs. limit=6.0 2026-09-24 05:40:18,377 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=180426.66666666666, ans=0.2 2026-09-24 05:40:24,244 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=180460.0, ans=0.1 2026-09-24 05:40:25,548 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.658e+02 3.346e+02 3.702e+02 4.138e+02 6.456e+02, threshold=7.404e+02, percent-clipped=0.0 2026-09-24 05:40:25,737 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.24 vs. limit=15.0 2026-09-24 05:40:26,096 INFO [train.py:1192] (0/2) Epoch 57, batch 500, loss[loss=0.3035, simple_loss=0.4237, pruned_loss=0.09167, over 24483.00 frames. ], tot_loss[loss=0.2618, simple_loss=0.3778, pruned_loss=0.07294, over 4427935.39 frames. ], batch size: 218, lr: 3.70e-03, grad_scale: 32.0 2026-09-24 05:40:30,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=180526.66666666666, ans=0.125 2026-09-24 05:40:34,133 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=180526.66666666666, ans=0.125 2026-09-24 05:40:38,984 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=180560.0, ans=0.1 2026-09-24 05:40:40,488 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.32 vs. limit=6.0 2026-09-24 05:40:40,829 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=180593.33333333334, ans=0.125 2026-09-24 05:40:50,857 INFO [train.py:1192] (0/2) Epoch 57, batch 550, loss[loss=0.291, simple_loss=0.4194, pruned_loss=0.08129, over 24257.00 frames. ], tot_loss[loss=0.261, simple_loss=0.3775, pruned_loss=0.07222, over 4517755.78 frames. ], batch size: 257, lr: 3.70e-03, grad_scale: 32.0 2026-09-24 05:41:03,605 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=180726.66666666666, ans=0.0 2026-09-24 05:41:04,295 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=7.94 vs. limit=15.0 2026-09-24 05:41:05,054 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=180726.66666666666, ans=0.0 2026-09-24 05:41:08,371 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=180760.0, ans=0.2 2026-09-24 05:41:08,375 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=180760.0, ans=0.125 2026-09-24 05:41:16,158 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.91 vs. limit=8.0 2026-09-24 05:41:16,221 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.732e+02 3.299e+02 3.672e+02 4.156e+02 5.943e+02, threshold=7.343e+02, percent-clipped=0.0 2026-09-24 05:41:16,799 INFO [train.py:1192] (0/2) Epoch 57, batch 600, loss[loss=0.3024, simple_loss=0.4237, pruned_loss=0.09057, over 24324.00 frames. ], tot_loss[loss=0.262, simple_loss=0.3786, pruned_loss=0.07273, over 4584433.24 frames. ], batch size: 234, lr: 3.70e-03, grad_scale: 32.0 2026-09-24 05:41:42,015 INFO [train.py:1192] (0/2) Epoch 57, batch 650, loss[loss=0.2664, simple_loss=0.3818, pruned_loss=0.07547, over 24570.00 frames. ], tot_loss[loss=0.2614, simple_loss=0.3779, pruned_loss=0.07246, over 4649755.25 frames. ], batch size: 162, lr: 3.70e-03, grad_scale: 32.0 2026-09-24 05:41:50,322 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=181026.66666666666, ans=0.025 2026-09-24 05:42:07,960 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.698e+02 3.523e+02 3.824e+02 4.374e+02 7.238e+02, threshold=7.648e+02, percent-clipped=0.0 2026-09-24 05:42:07,966 INFO [train.py:1192] (0/2) Epoch 57, batch 700, loss[loss=0.2477, simple_loss=0.3587, pruned_loss=0.06838, over 24557.00 frames. ], tot_loss[loss=0.2621, simple_loss=0.3787, pruned_loss=0.07273, over 4684863.33 frames. ], batch size: 154, lr: 3.69e-03, grad_scale: 32.0 2026-09-24 05:42:12,118 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=181160.0, ans=0.2 2026-09-24 05:42:21,584 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=181226.66666666666, ans=0.125 2026-09-24 05:42:33,037 INFO [train.py:1192] (0/2) Epoch 57, batch 750, loss[loss=0.2638, simple_loss=0.3854, pruned_loss=0.07112, over 24582.00 frames. ], tot_loss[loss=0.261, simple_loss=0.3775, pruned_loss=0.07227, over 4712200.09 frames. ], batch size: 170, lr: 3.69e-03, grad_scale: 32.0 2026-09-24 05:42:34,803 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=181326.66666666666, ans=0.2 2026-09-24 05:42:36,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=181326.66666666666, ans=0.125 2026-09-24 05:42:38,525 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.51 vs. limit=15.0 2026-09-24 05:42:43,887 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=181393.33333333334, ans=0.0 2026-09-24 05:42:44,866 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=181393.33333333334, ans=0.1 2026-09-24 05:42:50,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=181426.66666666666, ans=0.04949747468305833 2026-09-24 05:42:56,460 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=181460.0, ans=0.0 2026-09-24 05:42:58,300 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.714e+02 3.444e+02 3.874e+02 4.390e+02 6.289e+02, threshold=7.748e+02, percent-clipped=0.0 2026-09-24 05:42:58,306 INFO [train.py:1192] (0/2) Epoch 57, batch 800, loss[loss=0.2189, simple_loss=0.3352, pruned_loss=0.05131, over 24579.00 frames. ], tot_loss[loss=0.261, simple_loss=0.3774, pruned_loss=0.07231, over 4741225.17 frames. ], batch size: 137, lr: 3.69e-03, grad_scale: 32.0 2026-09-24 05:43:05,385 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.77 vs. limit=15.0 2026-09-24 05:43:06,540 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=181526.66666666666, ans=10.0 2026-09-24 05:43:11,854 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=181560.0, ans=0.0 2026-09-24 05:43:23,938 INFO [train.py:1192] (0/2) Epoch 57, batch 850, loss[loss=0.2993, simple_loss=0.4203, pruned_loss=0.08914, over 24536.00 frames. ], tot_loss[loss=0.2606, simple_loss=0.3772, pruned_loss=0.07201, over 4761942.51 frames. ], batch size: 204, lr: 3.69e-03, grad_scale: 32.0 2026-09-24 05:43:24,029 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=181660.0, ans=0.125 2026-09-24 05:43:29,664 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=181693.33333333334, ans=0.09899494936611666 2026-09-24 05:43:38,856 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=181726.66666666666, ans=0.025 2026-09-24 05:43:43,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=181760.0, ans=0.0 2026-09-24 05:43:49,506 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.660e+02 3.443e+02 3.791e+02 4.284e+02 6.953e+02, threshold=7.582e+02, percent-clipped=0.0 2026-09-24 05:43:49,512 INFO [train.py:1192] (0/2) Epoch 57, batch 900, loss[loss=0.2086, simple_loss=0.3299, pruned_loss=0.04358, over 24545.00 frames. ], tot_loss[loss=0.2602, simple_loss=0.3769, pruned_loss=0.07177, over 4775056.56 frames. ], batch size: 137, lr: 3.69e-03, grad_scale: 32.0 2026-09-24 05:43:54,297 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=181860.0, ans=0.0 2026-09-24 05:43:58,407 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.19 vs. limit=10.0 2026-09-24 05:44:08,288 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=181926.66666666666, ans=0.125 2026-09-24 05:44:09,642 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=181960.0, ans=0.2 2026-09-24 05:44:10,060 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=181960.0, ans=0.0 2026-09-24 05:44:11,092 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.min_abs, batch_count=181960.0, ans=0.5 2026-09-24 05:44:11,609 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=181960.0, ans=0.07 2026-09-24 05:44:14,494 INFO [train.py:1192] (0/2) Epoch 57, batch 950, loss[loss=0.3647, simple_loss=0.4264, pruned_loss=0.1515, over 11814.00 frames. ], tot_loss[loss=0.2604, simple_loss=0.3757, pruned_loss=0.07251, over 4707936.27 frames. ], batch size: 334, lr: 3.69e-03, grad_scale: 32.0 2026-09-24 05:44:18,577 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-57.pt 2026-09-24 05:44:25,449 INFO [train.py:1192] (0/2) Epoch 58, batch 0, loss[loss=0.2167, simple_loss=0.3415, pruned_loss=0.04595, over 24562.00 frames. ], tot_loss[loss=0.2167, simple_loss=0.3415, pruned_loss=0.04595, over 24562.00 frames. ], batch size: 137, lr: 3.65e-03, grad_scale: 32.0 2026-09-24 05:44:25,449 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 05:44:33,207 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.0694, 2.9097, 2.4675, 2.1675], device='cuda:0') 2026-09-24 05:44:36,020 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.0.layers.0.self_attn_weights, attn_weights_entropy = tensor([4.4371, 4.4985, 4.2238, 4.0838], device='cuda:0') 2026-09-24 05:44:36,213 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([4.1796, 3.7440, 3.6334, 4.0008], device='cuda:0') 2026-09-24 05:44:37,022 INFO [train.py:1224] (0/2) Epoch 58, validation: loss=0.1721, simple_loss=0.2908, pruned_loss=0.02669, over 2564189.00 frames. 2026-09-24 05:44:37,023 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 05:44:59,008 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.814e+02 3.593e+02 4.032e+02 4.676e+02 7.320e+02, threshold=8.064e+02, percent-clipped=0.0 2026-09-24 05:44:59,699 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.27 vs. limit=12.0 2026-09-24 05:45:02,914 INFO [train.py:1192] (0/2) Epoch 58, batch 50, loss[loss=0.2231, simple_loss=0.3335, pruned_loss=0.05633, over 24245.00 frames. ], tot_loss[loss=0.2651, simple_loss=0.3816, pruned_loss=0.07429, over 1075448.86 frames. ], batch size: 125, lr: 3.65e-03, grad_scale: 32.0 2026-09-24 05:45:03,996 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=182186.66666666666, ans=0.125 2026-09-24 05:45:07,198 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=182220.0, ans=0.125 2026-09-24 05:45:13,642 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.18 vs. limit=15.0 2026-09-24 05:45:13,912 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=182253.33333333334, ans=0.125 2026-09-24 05:45:21,795 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=182286.66666666666, ans=0.0 2026-09-24 05:45:28,135 INFO [train.py:1192] (0/2) Epoch 58, batch 100, loss[loss=0.2629, simple_loss=0.373, pruned_loss=0.07643, over 24596.00 frames. ], tot_loss[loss=0.2685, simple_loss=0.3856, pruned_loss=0.07574, over 1903077.99 frames. ], batch size: 154, lr: 3.65e-03, grad_scale: 32.0 2026-09-24 05:45:34,163 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=182386.66666666666, ans=0.0 2026-09-24 05:45:34,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=182386.66666666666, ans=0.125 2026-09-24 05:45:34,784 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.whiten, num_groups=1, num_channels=192, metric=4.40 vs. limit=12.0 2026-09-24 05:45:37,835 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=182420.0, ans=0.125 2026-09-24 05:45:42,206 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=182420.0, ans=0.1 2026-09-24 05:45:46,096 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=182453.33333333334, ans=0.0 2026-09-24 05:45:49,735 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.618e+02 3.344e+02 3.795e+02 4.291e+02 6.356e+02, threshold=7.589e+02, percent-clipped=0.0 2026-09-24 05:45:53,690 INFO [train.py:1192] (0/2) Epoch 58, batch 150, loss[loss=0.2443, simple_loss=0.3402, pruned_loss=0.07418, over 24259.00 frames. ], tot_loss[loss=0.2633, simple_loss=0.3805, pruned_loss=0.07308, over 2557803.90 frames. ], batch size: 125, lr: 3.65e-03, grad_scale: 32.0 2026-09-24 05:45:56,557 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=182520.0, ans=0.125 2026-09-24 05:45:58,721 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.77 vs. limit=15.0 2026-09-24 05:46:02,301 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=182553.33333333334, ans=0.0 2026-09-24 05:46:03,616 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=182586.66666666666, ans=0.1 2026-09-24 05:46:03,743 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.97 vs. limit=15.0 2026-09-24 05:46:12,423 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=182620.0, ans=0.125 2026-09-24 05:46:13,882 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=182620.0, ans=0.125 2026-09-24 05:46:19,925 INFO [train.py:1192] (0/2) Epoch 58, batch 200, loss[loss=0.3112, simple_loss=0.4123, pruned_loss=0.105, over 21153.00 frames. ], tot_loss[loss=0.2623, simple_loss=0.3792, pruned_loss=0.07271, over 3054715.78 frames. ], batch size: 333, lr: 3.65e-03, grad_scale: 32.0 2026-09-24 05:46:26,499 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.31 vs. limit=12.0 2026-09-24 05:46:34,105 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=182753.33333333334, ans=0.025 2026-09-24 05:46:41,401 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.696e+02 3.457e+02 3.741e+02 4.445e+02 6.120e+02, threshold=7.483e+02, percent-clipped=0.0 2026-09-24 05:46:45,728 INFO [train.py:1192] (0/2) Epoch 58, batch 250, loss[loss=0.2819, simple_loss=0.411, pruned_loss=0.07637, over 24295.00 frames. ], tot_loss[loss=0.2616, simple_loss=0.3785, pruned_loss=0.07238, over 3444327.22 frames. ], batch size: 234, lr: 3.64e-03, grad_scale: 32.0 2026-09-24 05:46:55,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=182920.0, ans=0.125 2026-09-24 05:47:09,703 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.65 vs. limit=15.0 2026-09-24 05:47:11,806 INFO [train.py:1192] (0/2) Epoch 58, batch 300, loss[loss=0.298, simple_loss=0.4193, pruned_loss=0.08835, over 24536.00 frames. ], tot_loss[loss=0.2617, simple_loss=0.378, pruned_loss=0.07273, over 3750722.88 frames. ], batch size: 204, lr: 3.64e-03, grad_scale: 32.0 2026-09-24 05:47:11,889 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=183020.0, ans=0.04949747468305833 2026-09-24 05:47:15,118 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=183020.0, ans=0.2 2026-09-24 05:47:16,010 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=183053.33333333334, ans=0.125 2026-09-24 05:47:33,099 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.515e+02 3.314e+02 3.805e+02 4.215e+02 6.606e+02, threshold=7.609e+02, percent-clipped=0.0 2026-09-24 05:47:37,117 INFO [train.py:1192] (0/2) Epoch 58, batch 350, loss[loss=0.2385, simple_loss=0.3498, pruned_loss=0.06364, over 24582.00 frames. ], tot_loss[loss=0.263, simple_loss=0.3794, pruned_loss=0.07331, over 3993453.47 frames. ], batch size: 137, lr: 3.64e-03, grad_scale: 32.0 2026-09-24 05:47:56,658 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=183320.0, ans=0.0 2026-09-24 05:48:02,034 INFO [train.py:1192] (0/2) Epoch 58, batch 400, loss[loss=0.2707, simple_loss=0.3876, pruned_loss=0.07691, over 24584.00 frames. ], tot_loss[loss=0.2622, simple_loss=0.3788, pruned_loss=0.07283, over 4179642.50 frames. ], batch size: 170, lr: 3.64e-03, grad_scale: 32.0 2026-09-24 05:48:02,335 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=13.04 vs. limit=22.5 2026-09-24 05:48:09,599 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=183386.66666666666, ans=0.0 2026-09-24 05:48:16,900 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=5.60 vs. limit=15.0 2026-09-24 05:48:23,412 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.871e+02 3.378e+02 3.805e+02 4.366e+02 6.990e+02, threshold=7.609e+02, percent-clipped=0.0 2026-09-24 05:48:27,157 INFO [train.py:1192] (0/2) Epoch 58, batch 450, loss[loss=0.2443, simple_loss=0.3759, pruned_loss=0.05641, over 24621.00 frames. ], tot_loss[loss=0.2624, simple_loss=0.3788, pruned_loss=0.073, over 4313788.51 frames. ], batch size: 175, lr: 3.64e-03, grad_scale: 32.0 2026-09-24 05:48:40,095 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=183586.66666666666, ans=0.1 2026-09-24 05:48:40,134 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.31 vs. limit=15.0 2026-09-24 05:48:52,868 INFO [train.py:1192] (0/2) Epoch 58, batch 500, loss[loss=0.3045, simple_loss=0.4168, pruned_loss=0.09606, over 24495.00 frames. ], tot_loss[loss=0.2613, simple_loss=0.3773, pruned_loss=0.07261, over 4430547.89 frames. ], batch size: 218, lr: 3.64e-03, grad_scale: 32.0 2026-09-24 05:49:06,602 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.39 vs. limit=8.0 2026-09-24 05:49:09,276 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=183786.66666666666, ans=0.125 2026-09-24 05:49:13,307 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=183820.0, ans=0.5 2026-09-24 05:49:14,455 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.501e+02 3.193e+02 3.750e+02 4.124e+02 5.753e+02, threshold=7.500e+02, percent-clipped=0.0 2026-09-24 05:49:14,552 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=183820.0, ans=0.0 2026-09-24 05:49:17,168 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=183820.0, ans=0.0 2026-09-24 05:49:18,789 INFO [train.py:1192] (0/2) Epoch 58, batch 550, loss[loss=0.2708, simple_loss=0.3966, pruned_loss=0.0725, over 24304.00 frames. ], tot_loss[loss=0.2615, simple_loss=0.3776, pruned_loss=0.07269, over 4520011.48 frames. ], batch size: 257, lr: 3.63e-03, grad_scale: 32.0 2026-09-24 05:49:19,952 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=183853.33333333334, ans=0.125 2026-09-24 05:49:23,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=183886.66666666666, ans=0.1 2026-09-24 05:49:25,194 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=183886.66666666666, ans=0.1 2026-09-24 05:49:29,114 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.17 vs. limit=6.0 2026-09-24 05:49:36,715 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.63 vs. limit=6.0 2026-09-24 05:49:37,132 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=183953.33333333334, ans=0.0 2026-09-24 05:49:40,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=183986.66666666666, ans=0.125 2026-09-24 05:49:44,582 INFO [train.py:1192] (0/2) Epoch 58, batch 600, loss[loss=0.2659, simple_loss=0.3921, pruned_loss=0.06982, over 24342.00 frames. ], tot_loss[loss=0.2619, simple_loss=0.3782, pruned_loss=0.0728, over 4585672.11 frames. ], batch size: 234, lr: 3.63e-03, grad_scale: 32.0 2026-09-24 05:49:56,854 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=184086.66666666666, ans=0.125 2026-09-24 05:50:05,768 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.673e+02 3.291e+02 3.709e+02 4.176e+02 6.941e+02, threshold=7.419e+02, percent-clipped=0.0 2026-09-24 05:50:10,282 INFO [train.py:1192] (0/2) Epoch 58, batch 650, loss[loss=0.2733, simple_loss=0.387, pruned_loss=0.07979, over 24549.00 frames. ], tot_loss[loss=0.2608, simple_loss=0.3774, pruned_loss=0.07209, over 4650784.32 frames. ], batch size: 162, lr: 3.63e-03, grad_scale: 32.0 2026-09-24 05:50:12,702 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.23 vs. limit=22.5 2026-09-24 05:50:16,105 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=184220.0, ans=0.125 2026-09-24 05:50:18,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=184220.0, ans=0.125 2026-09-24 05:50:21,800 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=184253.33333333334, ans=0.125 2026-09-24 05:50:28,081 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=184286.66666666666, ans=0.125 2026-09-24 05:50:33,009 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.99 vs. limit=22.5 2026-09-24 05:50:36,105 INFO [train.py:1192] (0/2) Epoch 58, batch 700, loss[loss=0.2483, simple_loss=0.3641, pruned_loss=0.06623, over 24576.00 frames. ], tot_loss[loss=0.2606, simple_loss=0.3776, pruned_loss=0.07177, over 4682605.30 frames. ], batch size: 154, lr: 3.63e-03, grad_scale: 32.0 2026-09-24 05:50:39,678 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=184353.33333333334, ans=0.1 2026-09-24 05:50:44,232 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=6.99 vs. limit=15.0 2026-09-24 05:50:48,511 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=184420.0, ans=0.0 2026-09-24 05:50:57,734 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.707e+02 3.433e+02 3.912e+02 4.428e+02 7.211e+02, threshold=7.824e+02, percent-clipped=0.0 2026-09-24 05:50:58,414 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.16 vs. limit=15.0 2026-09-24 05:51:01,757 INFO [train.py:1192] (0/2) Epoch 58, batch 750, loss[loss=0.2694, simple_loss=0.3871, pruned_loss=0.0758, over 24556.00 frames. ], tot_loss[loss=0.2596, simple_loss=0.3765, pruned_loss=0.07134, over 4710379.17 frames. ], batch size: 170, lr: 3.63e-03, grad_scale: 32.0 2026-09-24 05:51:03,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=184520.0, ans=0.2 2026-09-24 05:51:11,666 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=184586.66666666666, ans=0.1 2026-09-24 05:51:21,855 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=184653.33333333334, ans=0.0 2026-09-24 05:51:27,059 INFO [train.py:1192] (0/2) Epoch 58, batch 800, loss[loss=0.2254, simple_loss=0.343, pruned_loss=0.05385, over 24544.00 frames. ], tot_loss[loss=0.2594, simple_loss=0.3761, pruned_loss=0.0713, over 4734953.98 frames. ], batch size: 137, lr: 3.63e-03, grad_scale: 32.0 2026-09-24 05:51:45,979 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=184786.66666666666, ans=0.0 2026-09-24 05:51:48,213 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.583e+02 3.476e+02 3.817e+02 4.450e+02 5.594e+02, threshold=7.634e+02, percent-clipped=0.0 2026-09-24 05:51:52,209 INFO [train.py:1192] (0/2) Epoch 58, batch 850, loss[loss=0.2802, simple_loss=0.3982, pruned_loss=0.08108, over 24582.00 frames. ], tot_loss[loss=0.2586, simple_loss=0.3755, pruned_loss=0.07079, over 4757565.02 frames. ], batch size: 198, lr: 3.63e-03, grad_scale: 32.0 2026-09-24 05:51:58,792 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=184886.66666666666, ans=0.125 2026-09-24 05:52:05,351 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=184920.0, ans=0.2 2026-09-24 05:52:17,221 INFO [train.py:1192] (0/2) Epoch 58, batch 900, loss[loss=0.222, simple_loss=0.3416, pruned_loss=0.05123, over 24545.00 frames. ], tot_loss[loss=0.2588, simple_loss=0.3758, pruned_loss=0.07089, over 4771866.23 frames. ], batch size: 137, lr: 3.62e-03, grad_scale: 32.0 2026-09-24 05:52:19,237 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=185020.0, ans=0.125 2026-09-24 05:52:23,267 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=185053.33333333334, ans=0.025 2026-09-24 05:52:25,180 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=185053.33333333334, ans=0.125 2026-09-24 05:52:25,669 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=185053.33333333334, ans=0.2 2026-09-24 05:52:38,728 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.652e+02 3.344e+02 3.958e+02 4.577e+02 6.010e+02, threshold=7.916e+02, percent-clipped=0.0 2026-09-24 05:52:42,580 INFO [train.py:1192] (0/2) Epoch 58, batch 950, loss[loss=0.3485, simple_loss=0.4073, pruned_loss=0.1449, over 11384.00 frames. ], tot_loss[loss=0.2593, simple_loss=0.3749, pruned_loss=0.07181, over 4709935.04 frames. ], batch size: 333, lr: 3.62e-03, grad_scale: 32.0 2026-09-24 05:52:46,778 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-58.pt 2026-09-24 05:52:52,719 INFO [train.py:1192] (0/2) Epoch 59, batch 0, loss[loss=0.2148, simple_loss=0.3391, pruned_loss=0.04521, over 24557.00 frames. ], tot_loss[loss=0.2148, simple_loss=0.3391, pruned_loss=0.04521, over 24557.00 frames. ], batch size: 137, lr: 3.59e-03, grad_scale: 32.0 2026-09-24 05:52:52,720 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 05:52:54,283 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.1334, 1.7729, 3.1645, 1.9351], device='cuda:0') 2026-09-24 05:53:04,541 INFO [train.py:1224] (0/2) Epoch 59, validation: loss=0.171, simple_loss=0.2895, pruned_loss=0.02627, over 2564189.00 frames. 2026-09-24 05:53:04,542 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 05:53:04,726 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.28 vs. limit=10.0 2026-09-24 05:53:10,470 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=185246.66666666666, ans=0.125 2026-09-24 05:53:12,432 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=185246.66666666666, ans=0.0 2026-09-24 05:53:27,029 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=185346.66666666666, ans=0.125 2026-09-24 05:53:30,308 INFO [train.py:1192] (0/2) Epoch 59, batch 50, loss[loss=0.2388, simple_loss=0.3419, pruned_loss=0.06788, over 24256.00 frames. ], tot_loss[loss=0.2687, simple_loss=0.3835, pruned_loss=0.07695, over 1075131.43 frames. ], batch size: 125, lr: 3.59e-03, grad_scale: 32.0 2026-09-24 05:53:39,979 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.37 vs. limit=15.0 2026-09-24 05:53:47,502 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.691e+02 3.534e+02 3.979e+02 4.502e+02 6.245e+02, threshold=7.957e+02, percent-clipped=0.0 2026-09-24 05:53:48,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=185480.0, ans=0.125 2026-09-24 05:53:53,506 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=185513.33333333334, ans=0.0 2026-09-24 05:53:55,373 INFO [train.py:1192] (0/2) Epoch 59, batch 100, loss[loss=0.2503, simple_loss=0.3629, pruned_loss=0.06883, over 24607.00 frames. ], tot_loss[loss=0.271, simple_loss=0.3874, pruned_loss=0.07726, over 1904671.37 frames. ], batch size: 154, lr: 3.59e-03, grad_scale: 32.0 2026-09-24 05:54:06,539 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.72 vs. limit=15.0 2026-09-24 05:54:21,465 INFO [train.py:1192] (0/2) Epoch 59, batch 150, loss[loss=0.1955, simple_loss=0.3102, pruned_loss=0.04044, over 24266.00 frames. ], tot_loss[loss=0.265, simple_loss=0.3816, pruned_loss=0.07425, over 2558946.03 frames. ], batch size: 125, lr: 3.59e-03, grad_scale: 32.0 2026-09-24 05:54:27,160 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=185746.66666666666, ans=0.025 2026-09-24 05:54:35,822 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=185780.0, ans=0.125 2026-09-24 05:54:36,276 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_positive, batch_count=185813.33333333334, ans=0.05 2026-09-24 05:54:38,134 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.96 vs. limit=15.0 2026-09-24 05:54:38,851 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.662e+02 3.284e+02 3.697e+02 4.244e+02 6.453e+02, threshold=7.394e+02, percent-clipped=0.0 2026-09-24 05:54:44,355 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=185846.66666666666, ans=0.125 2026-09-24 05:54:47,286 INFO [train.py:1192] (0/2) Epoch 59, batch 200, loss[loss=0.2933, simple_loss=0.4027, pruned_loss=0.092, over 21165.00 frames. ], tot_loss[loss=0.2629, simple_loss=0.3798, pruned_loss=0.07303, over 3055520.16 frames. ], batch size: 333, lr: 3.58e-03, grad_scale: 32.0 2026-09-24 05:54:53,463 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=185913.33333333334, ans=0.0 2026-09-24 05:55:00,740 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=185946.66666666666, ans=0.2 2026-09-24 05:55:04,883 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=185980.0, ans=0.025 2026-09-24 05:55:06,882 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=185980.0, ans=0.0 2026-09-24 05:55:12,730 INFO [train.py:1192] (0/2) Epoch 59, batch 250, loss[loss=0.287, simple_loss=0.4119, pruned_loss=0.08103, over 24321.00 frames. ], tot_loss[loss=0.2609, simple_loss=0.3782, pruned_loss=0.07182, over 3443889.17 frames. ], batch size: 234, lr: 3.58e-03, grad_scale: 32.0 2026-09-24 05:55:16,213 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.01 vs. limit=10.0 2026-09-24 05:55:21,207 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=186080.0, ans=0.125 2026-09-24 05:55:27,759 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=186146.66666666666, ans=0.025 2026-09-24 05:55:30,137 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.774e+02 3.471e+02 3.847e+02 4.480e+02 8.835e+02, threshold=7.694e+02, percent-clipped=1.0 2026-09-24 05:55:38,443 INFO [train.py:1192] (0/2) Epoch 59, batch 300, loss[loss=0.2855, simple_loss=0.4063, pruned_loss=0.08234, over 24548.00 frames. ], tot_loss[loss=0.2615, simple_loss=0.3778, pruned_loss=0.07263, over 3752096.71 frames. ], batch size: 204, lr: 3.58e-03, grad_scale: 32.0 2026-09-24 05:55:47,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=186280.0, ans=0.0 2026-09-24 05:55:51,195 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=6.54 vs. limit=15.0 2026-09-24 05:55:52,108 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=186280.0, ans=0.125 2026-09-24 05:55:53,083 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=186313.33333333334, ans=0.125 2026-09-24 05:56:00,341 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.96 vs. limit=15.0 2026-09-24 05:56:03,679 INFO [train.py:1192] (0/2) Epoch 59, batch 350, loss[loss=0.2342, simple_loss=0.3453, pruned_loss=0.06154, over 24568.00 frames. ], tot_loss[loss=0.2611, simple_loss=0.378, pruned_loss=0.07208, over 3991098.48 frames. ], batch size: 137, lr: 3.58e-03, grad_scale: 32.0 2026-09-24 05:56:06,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=186380.0, ans=0.05 2026-09-24 05:56:08,213 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=186413.33333333334, ans=0.125 2026-09-24 05:56:13,725 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=186446.66666666666, ans=0.2 2026-09-24 05:56:20,967 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.677e+02 3.293e+02 3.742e+02 4.328e+02 7.534e+02, threshold=7.484e+02, percent-clipped=0.0 2026-09-24 05:56:24,846 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=186513.33333333334, ans=0.125 2026-09-24 05:56:28,954 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.41 vs. limit=15.0 2026-09-24 05:56:29,151 INFO [train.py:1192] (0/2) Epoch 59, batch 400, loss[loss=0.2568, simple_loss=0.3736, pruned_loss=0.06997, over 24556.00 frames. ], tot_loss[loss=0.2603, simple_loss=0.3772, pruned_loss=0.07172, over 4175675.08 frames. ], batch size: 170, lr: 3.58e-03, grad_scale: 32.0 2026-09-24 05:56:46,602 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-56000.pt 2026-09-24 05:56:54,934 INFO [train.py:1192] (0/2) Epoch 59, batch 450, loss[loss=0.2785, simple_loss=0.405, pruned_loss=0.076, over 24633.00 frames. ], tot_loss[loss=0.2613, simple_loss=0.3782, pruned_loss=0.07217, over 4310338.62 frames. ], batch size: 175, lr: 3.58e-03, grad_scale: 32.0 2026-09-24 05:56:59,280 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=186746.66666666666, ans=0.1 2026-09-24 05:57:03,525 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=186746.66666666666, ans=0.1 2026-09-24 05:57:08,176 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=186780.0, ans=0.95 2026-09-24 05:57:11,571 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=186813.33333333334, ans=0.1 2026-09-24 05:57:11,988 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.672e+02 3.301e+02 3.744e+02 4.193e+02 6.017e+02, threshold=7.488e+02, percent-clipped=0.0 2026-09-24 05:57:19,055 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=186846.66666666666, ans=0.025 2026-09-24 05:57:20,274 INFO [train.py:1192] (0/2) Epoch 59, batch 500, loss[loss=0.2864, simple_loss=0.4097, pruned_loss=0.08157, over 24466.00 frames. ], tot_loss[loss=0.2603, simple_loss=0.3769, pruned_loss=0.07189, over 4428206.09 frames. ], batch size: 218, lr: 3.57e-03, grad_scale: 32.0 2026-09-24 05:57:20,474 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.21 vs. limit=15.0 2026-09-24 05:57:23,900 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=186880.0, ans=0.125 2026-09-24 05:57:32,652 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=186946.66666666666, ans=0.1 2026-09-24 05:57:43,095 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.21 vs. limit=15.0 2026-09-24 05:57:45,403 INFO [train.py:1192] (0/2) Epoch 59, batch 550, loss[loss=0.2872, simple_loss=0.4126, pruned_loss=0.0809, over 24246.00 frames. ], tot_loss[loss=0.2606, simple_loss=0.3773, pruned_loss=0.07195, over 4517600.10 frames. ], batch size: 257, lr: 3.57e-03, grad_scale: 32.0 2026-09-24 05:57:50,291 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=187080.0, ans=0.1 2026-09-24 05:58:02,550 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.630e+02 3.301e+02 3.638e+02 4.319e+02 6.236e+02, threshold=7.276e+02, percent-clipped=0.0 2026-09-24 05:58:03,138 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=187146.66666666666, ans=0.1 2026-09-24 05:58:10,787 INFO [train.py:1192] (0/2) Epoch 59, batch 600, loss[loss=0.2748, simple_loss=0.4043, pruned_loss=0.07259, over 24323.00 frames. ], tot_loss[loss=0.2608, simple_loss=0.3778, pruned_loss=0.07188, over 4584596.55 frames. ], batch size: 234, lr: 3.57e-03, grad_scale: 32.0 2026-09-24 05:58:21,478 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.99 vs. limit=10.0 2026-09-24 05:58:30,352 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=187313.33333333334, ans=0.125 2026-09-24 05:58:32,718 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=187346.66666666666, ans=0.1 2026-09-24 05:58:36,319 INFO [train.py:1192] (0/2) Epoch 59, batch 650, loss[loss=0.2652, simple_loss=0.3779, pruned_loss=0.07622, over 24559.00 frames. ], tot_loss[loss=0.2596, simple_loss=0.3767, pruned_loss=0.07127, over 4649867.05 frames. ], batch size: 162, lr: 3.57e-03, grad_scale: 32.0 2026-09-24 05:58:46,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=187446.66666666666, ans=0.125 2026-09-24 05:58:50,810 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=187446.66666666666, ans=0.0 2026-09-24 05:58:52,169 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=187480.0, ans=0.125 2026-09-24 05:58:53,470 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.591e+02 3.411e+02 3.788e+02 4.386e+02 6.509e+02, threshold=7.576e+02, percent-clipped=0.0 2026-09-24 05:58:54,076 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=187480.0, ans=0.125 2026-09-24 05:58:54,110 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=187480.0, ans=0.125 2026-09-24 05:59:01,734 INFO [train.py:1192] (0/2) Epoch 59, batch 700, loss[loss=0.2511, simple_loss=0.3647, pruned_loss=0.06871, over 24565.00 frames. ], tot_loss[loss=0.2601, simple_loss=0.3775, pruned_loss=0.07136, over 4682733.04 frames. ], batch size: 154, lr: 3.57e-03, grad_scale: 32.0 2026-09-24 05:59:04,073 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=187546.66666666666, ans=0.125 2026-09-24 05:59:15,202 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=187613.33333333334, ans=0.0 2026-09-24 05:59:27,692 INFO [train.py:1192] (0/2) Epoch 59, batch 750, loss[loss=0.2608, simple_loss=0.3817, pruned_loss=0.06993, over 24547.00 frames. ], tot_loss[loss=0.2599, simple_loss=0.3767, pruned_loss=0.07152, over 4711246.28 frames. ], batch size: 170, lr: 3.57e-03, grad_scale: 32.0 2026-09-24 05:59:34,975 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=187746.66666666666, ans=0.2 2026-09-24 05:59:45,330 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.834e+02 3.595e+02 4.005e+02 4.409e+02 6.130e+02, threshold=8.011e+02, percent-clipped=0.0 2026-09-24 05:59:45,987 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.14 vs. limit=15.0 2026-09-24 05:59:51,383 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=187846.66666666666, ans=0.1 2026-09-24 05:59:53,263 INFO [train.py:1192] (0/2) Epoch 59, batch 800, loss[loss=0.2152, simple_loss=0.3311, pruned_loss=0.0496, over 24558.00 frames. ], tot_loss[loss=0.2591, simple_loss=0.3761, pruned_loss=0.07105, over 4736884.06 frames. ], batch size: 137, lr: 3.57e-03, grad_scale: 64.0 2026-09-24 05:59:58,561 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=187913.33333333334, ans=0.035 2026-09-24 05:59:59,212 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=6.20 vs. limit=15.0 2026-09-24 06:00:00,070 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=187913.33333333334, ans=0.125 2026-09-24 06:00:07,717 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=187946.66666666666, ans=0.1 2026-09-24 06:00:12,186 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=187980.0, ans=0.125 2026-09-24 06:00:18,896 INFO [train.py:1192] (0/2) Epoch 59, batch 850, loss[loss=0.2758, simple_loss=0.3976, pruned_loss=0.07698, over 24580.00 frames. ], tot_loss[loss=0.259, simple_loss=0.3761, pruned_loss=0.07098, over 4759606.44 frames. ], batch size: 198, lr: 3.56e-03, grad_scale: 64.0 2026-09-24 06:00:26,795 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=188080.0, ans=0.125 2026-09-24 06:00:36,162 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.731e+02 3.395e+02 3.830e+02 4.384e+02 6.066e+02, threshold=7.661e+02, percent-clipped=0.0 2026-09-24 06:00:44,043 INFO [train.py:1192] (0/2) Epoch 59, batch 900, loss[loss=0.2124, simple_loss=0.3354, pruned_loss=0.04473, over 24551.00 frames. ], tot_loss[loss=0.2594, simple_loss=0.3764, pruned_loss=0.07116, over 4773615.04 frames. ], batch size: 137, lr: 3.56e-03, grad_scale: 64.0 2026-09-24 06:00:53,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.const_attention_rate, batch_count=188280.0, ans=0.025 2026-09-24 06:00:54,153 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=188280.0, ans=0.125 2026-09-24 06:01:03,430 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=188346.66666666666, ans=0.0 2026-09-24 06:01:05,817 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=188346.66666666666, ans=0.0 2026-09-24 06:01:06,821 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:01:08,622 INFO [train.py:1192] (0/2) Epoch 59, batch 950, loss[loss=0.3904, simple_loss=0.4493, pruned_loss=0.1658, over 11089.00 frames. ], tot_loss[loss=0.2594, simple_loss=0.3752, pruned_loss=0.07187, over 4713601.27 frames. ], batch size: 333, lr: 3.56e-03, grad_scale: 32.0 2026-09-24 06:01:13,045 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-59.pt 2026-09-24 06:01:17,701 INFO [train.py:1192] (0/2) Epoch 60, batch 0, loss[loss=0.2164, simple_loss=0.3322, pruned_loss=0.05032, over 24583.00 frames. ], tot_loss[loss=0.2164, simple_loss=0.3322, pruned_loss=0.05032, over 24583.00 frames. ], batch size: 137, lr: 3.53e-03, grad_scale: 32.0 2026-09-24 06:01:17,702 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 06:01:28,823 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.1.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.7486, 2.4881, 2.6298, 2.5376], device='cuda:0') 2026-09-24 06:01:29,342 INFO [train.py:1224] (0/2) Epoch 60, validation: loss=0.1697, simple_loss=0.2883, pruned_loss=0.0255, over 2564189.00 frames. 2026-09-24 06:01:29,392 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 06:01:37,295 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=188440.0, ans=0.125 2026-09-24 06:01:37,318 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=188440.0, ans=0.1 2026-09-24 06:01:42,911 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.788e+02 3.464e+02 3.986e+02 4.513e+02 6.611e+02, threshold=7.972e+02, percent-clipped=0.0 2026-09-24 06:01:45,167 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=188506.66666666666, ans=0.0 2026-09-24 06:01:51,861 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.29 vs. limit=15.0 2026-09-24 06:01:53,139 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:01:54,595 INFO [train.py:1192] (0/2) Epoch 60, batch 50, loss[loss=0.2021, simple_loss=0.3173, pruned_loss=0.04346, over 24265.00 frames. ], tot_loss[loss=0.2654, simple_loss=0.3813, pruned_loss=0.07475, over 1077213.06 frames. ], batch size: 125, lr: 3.53e-03, grad_scale: 32.0 2026-09-24 06:02:03,603 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=188606.66666666666, ans=0.125 2026-09-24 06:02:11,751 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=188673.33333333334, ans=0.125 2026-09-24 06:02:12,271 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=188673.33333333334, ans=0.07 2026-09-24 06:02:12,794 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=188673.33333333334, ans=0.125 2026-09-24 06:02:19,596 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.87 vs. limit=6.0 2026-09-24 06:02:19,925 INFO [train.py:1192] (0/2) Epoch 60, batch 100, loss[loss=0.2672, simple_loss=0.3781, pruned_loss=0.07813, over 24604.00 frames. ], tot_loss[loss=0.269, simple_loss=0.3863, pruned_loss=0.07583, over 1904827.75 frames. ], batch size: 154, lr: 3.53e-03, grad_scale: 32.0 2026-09-24 06:02:24,474 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=188773.33333333334, ans=0.125 2026-09-24 06:02:33,607 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.858e+02 3.594e+02 3.944e+02 4.374e+02 5.505e+02, threshold=7.888e+02, percent-clipped=0.0 2026-09-24 06:02:36,008 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=188840.0, ans=0.0 2026-09-24 06:02:36,987 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=188840.0, ans=0.0 2026-09-24 06:02:42,787 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=188873.33333333334, ans=0.0 2026-09-24 06:02:44,870 INFO [train.py:1192] (0/2) Epoch 60, batch 150, loss[loss=0.2242, simple_loss=0.3322, pruned_loss=0.05806, over 24293.00 frames. ], tot_loss[loss=0.2637, simple_loss=0.3811, pruned_loss=0.07319, over 2557210.12 frames. ], batch size: 125, lr: 3.53e-03, grad_scale: 32.0 2026-09-24 06:03:03,213 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=189006.66666666666, ans=0.125 2026-09-24 06:03:07,360 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer_ff2.min_abs, batch_count=189040.0, ans=0.1 2026-09-24 06:03:07,376 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=189040.0, ans=0.125 2026-09-24 06:03:10,719 INFO [train.py:1192] (0/2) Epoch 60, batch 200, loss[loss=0.3035, simple_loss=0.4084, pruned_loss=0.09928, over 21061.00 frames. ], tot_loss[loss=0.2621, simple_loss=0.3795, pruned_loss=0.07238, over 3054552.36 frames. ], batch size: 333, lr: 3.52e-03, grad_scale: 32.0 2026-09-24 06:03:19,693 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=189106.66666666666, ans=0.025 2026-09-24 06:03:24,458 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.616e+02 3.276e+02 3.806e+02 4.339e+02 6.683e+02, threshold=7.612e+02, percent-clipped=0.0 2026-09-24 06:03:34,163 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.39 vs. limit=12.0 2026-09-24 06:03:36,158 INFO [train.py:1192] (0/2) Epoch 60, batch 250, loss[loss=0.2783, simple_loss=0.4027, pruned_loss=0.07695, over 24304.00 frames. ], tot_loss[loss=0.2609, simple_loss=0.3781, pruned_loss=0.0718, over 3443061.61 frames. ], batch size: 234, lr: 3.52e-03, grad_scale: 32.0 2026-09-24 06:03:45,916 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:03:46,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=189306.66666666666, ans=0.0 2026-09-24 06:04:01,877 INFO [train.py:1192] (0/2) Epoch 60, batch 300, loss[loss=0.2836, simple_loss=0.406, pruned_loss=0.08061, over 24571.00 frames. ], tot_loss[loss=0.26, simple_loss=0.377, pruned_loss=0.07152, over 3749264.57 frames. ], batch size: 204, lr: 3.52e-03, grad_scale: 32.0 2026-09-24 06:04:15,632 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.845e+02 3.363e+02 3.702e+02 4.210e+02 5.489e+02, threshold=7.404e+02, percent-clipped=0.0 2026-09-24 06:04:27,313 INFO [train.py:1192] (0/2) Epoch 60, batch 350, loss[loss=0.2129, simple_loss=0.3278, pruned_loss=0.04896, over 24572.00 frames. ], tot_loss[loss=0.2606, simple_loss=0.378, pruned_loss=0.07163, over 3992770.44 frames. ], batch size: 137, lr: 3.52e-03, grad_scale: 32.0 2026-09-24 06:04:52,395 INFO [train.py:1192] (0/2) Epoch 60, batch 400, loss[loss=0.2685, simple_loss=0.383, pruned_loss=0.07705, over 24557.00 frames. ], tot_loss[loss=0.2598, simple_loss=0.3771, pruned_loss=0.07121, over 4178968.49 frames. ], batch size: 170, lr: 3.52e-03, grad_scale: 32.0 2026-09-24 06:04:53,187 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.77 vs. limit=15.0 2026-09-24 06:05:06,102 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.726e+02 3.383e+02 3.799e+02 4.527e+02 6.363e+02, threshold=7.598e+02, percent-clipped=0.0 2026-09-24 06:05:10,735 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=189840.0, ans=0.2 2026-09-24 06:05:13,308 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=189873.33333333334, ans=0.0 2026-09-24 06:05:18,083 INFO [train.py:1192] (0/2) Epoch 60, batch 450, loss[loss=0.2736, simple_loss=0.3862, pruned_loss=0.08048, over 24611.00 frames. ], tot_loss[loss=0.2606, simple_loss=0.3777, pruned_loss=0.07176, over 4311322.95 frames. ], batch size: 175, lr: 3.52e-03, grad_scale: 32.0 2026-09-24 06:05:34,713 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=190006.66666666666, ans=0.1 2026-09-24 06:05:43,568 INFO [train.py:1192] (0/2) Epoch 60, batch 500, loss[loss=0.2783, simple_loss=0.4014, pruned_loss=0.0776, over 24505.00 frames. ], tot_loss[loss=0.259, simple_loss=0.3759, pruned_loss=0.07103, over 4428877.87 frames. ], batch size: 218, lr: 3.52e-03, grad_scale: 32.0 2026-09-24 06:05:43,728 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys.whitening_limit, batch_count=190073.33333333334, ans=6.0 2026-09-24 06:05:49,484 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=190106.66666666666, ans=0.1 2026-09-24 06:05:57,318 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.393e+02 3.216e+02 3.484e+02 4.055e+02 5.792e+02, threshold=6.968e+02, percent-clipped=0.0 2026-09-24 06:06:00,369 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.min_positive, batch_count=190173.33333333334, ans=0.05 2026-09-24 06:06:09,332 INFO [train.py:1192] (0/2) Epoch 60, batch 550, loss[loss=0.2938, simple_loss=0.4146, pruned_loss=0.08655, over 24254.00 frames. ], tot_loss[loss=0.2605, simple_loss=0.3774, pruned_loss=0.07179, over 4518435.95 frames. ], batch size: 257, lr: 3.51e-03, grad_scale: 32.0 2026-09-24 06:06:13,491 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=190240.0, ans=0.125 2026-09-24 06:06:29,255 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.47 vs. limit=15.0 2026-09-24 06:06:31,993 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=190373.33333333334, ans=0.125 2026-09-24 06:06:34,617 INFO [train.py:1192] (0/2) Epoch 60, batch 600, loss[loss=0.2936, simple_loss=0.4117, pruned_loss=0.0878, over 24320.00 frames. ], tot_loss[loss=0.2613, simple_loss=0.3781, pruned_loss=0.07229, over 4585107.55 frames. ], batch size: 234, lr: 3.51e-03, grad_scale: 32.0 2026-09-24 06:06:35,195 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=190406.66666666666, ans=0.04949747468305833 2026-09-24 06:06:35,209 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=190406.66666666666, ans=0.0 2026-09-24 06:06:40,558 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=190440.0, ans=0.125 2026-09-24 06:06:43,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=190440.0, ans=0.0 2026-09-24 06:06:48,113 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=190473.33333333334, ans=0.125 2026-09-24 06:06:48,511 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.621e+02 3.279e+02 3.656e+02 4.277e+02 6.065e+02, threshold=7.312e+02, percent-clipped=0.0 2026-09-24 06:06:53,560 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.47 vs. limit=15.0 2026-09-24 06:06:58,904 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=190540.0, ans=0.1 2026-09-24 06:07:00,623 INFO [train.py:1192] (0/2) Epoch 60, batch 650, loss[loss=0.2651, simple_loss=0.3848, pruned_loss=0.07267, over 24559.00 frames. ], tot_loss[loss=0.2611, simple_loss=0.3779, pruned_loss=0.07218, over 4650420.49 frames. ], batch size: 162, lr: 3.51e-03, grad_scale: 32.0 2026-09-24 06:07:10,859 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=190640.0, ans=0.125 2026-09-24 06:07:11,859 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=190640.0, ans=0.1 2026-09-24 06:07:18,304 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=190673.33333333334, ans=0.125 2026-09-24 06:07:20,081 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=190706.66666666666, ans=0.1 2026-09-24 06:07:25,600 INFO [train.py:1192] (0/2) Epoch 60, batch 700, loss[loss=0.2552, simple_loss=0.3682, pruned_loss=0.07112, over 24593.00 frames. ], tot_loss[loss=0.2614, simple_loss=0.3787, pruned_loss=0.07202, over 4682217.96 frames. ], batch size: 154, lr: 3.51e-03, grad_scale: 32.0 2026-09-24 06:07:29,133 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.26 vs. limit=15.0 2026-09-24 06:07:30,045 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=190740.0, ans=0.125 2026-09-24 06:07:33,903 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=190773.33333333334, ans=0.0 2026-09-24 06:07:36,743 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer_na.min_abs, batch_count=190806.66666666666, ans=0.02 2026-09-24 06:07:39,147 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=190806.66666666666, ans=0.1 2026-09-24 06:07:39,500 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.842e+02 3.414e+02 3.718e+02 4.175e+02 5.828e+02, threshold=7.437e+02, percent-clipped=0.0 2026-09-24 06:07:41,545 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=190840.0, ans=0.1 2026-09-24 06:07:50,671 INFO [train.py:1192] (0/2) Epoch 60, batch 750, loss[loss=0.2661, simple_loss=0.3829, pruned_loss=0.07471, over 24577.00 frames. ], tot_loss[loss=0.2602, simple_loss=0.3772, pruned_loss=0.07165, over 4709869.06 frames. ], batch size: 170, lr: 3.51e-03, grad_scale: 32.0 2026-09-24 06:07:53,926 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=190906.66666666666, ans=0.125 2026-09-24 06:08:04,255 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=190973.33333333334, ans=0.1 2026-09-24 06:08:05,598 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=191006.66666666666, ans=10.0 2026-09-24 06:08:13,960 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=191040.0, ans=0.0 2026-09-24 06:08:15,835 INFO [train.py:1192] (0/2) Epoch 60, batch 800, loss[loss=0.1945, simple_loss=0.3152, pruned_loss=0.03689, over 24553.00 frames. ], tot_loss[loss=0.259, simple_loss=0.3761, pruned_loss=0.07099, over 4734836.96 frames. ], batch size: 137, lr: 3.51e-03, grad_scale: 32.0 2026-09-24 06:08:15,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=191073.33333333334, ans=0.125 2026-09-24 06:08:28,239 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=191140.0, ans=0.125 2026-09-24 06:08:29,585 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.454e+02 3.407e+02 3.822e+02 4.313e+02 6.694e+02, threshold=7.644e+02, percent-clipped=0.0 2026-09-24 06:08:33,615 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.97 vs. limit=12.0 2026-09-24 06:08:41,208 INFO [train.py:1192] (0/2) Epoch 60, batch 850, loss[loss=0.2694, simple_loss=0.398, pruned_loss=0.07035, over 24592.00 frames. ], tot_loss[loss=0.2588, simple_loss=0.376, pruned_loss=0.07086, over 4757962.62 frames. ], batch size: 198, lr: 3.50e-03, grad_scale: 32.0 2026-09-24 06:08:47,224 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=8.69 vs. limit=22.5 2026-09-24 06:08:58,550 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=191340.0, ans=0.125 2026-09-24 06:09:06,696 INFO [train.py:1192] (0/2) Epoch 60, batch 900, loss[loss=0.221, simple_loss=0.335, pruned_loss=0.05347, over 24550.00 frames. ], tot_loss[loss=0.2601, simple_loss=0.3769, pruned_loss=0.07164, over 4771958.64 frames. ], batch size: 137, lr: 3.50e-03, grad_scale: 32.0 2026-09-24 06:09:11,419 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=191440.0, ans=0.0 2026-09-24 06:09:13,979 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:09:19,106 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.75 vs. limit=15.0 2026-09-24 06:09:19,813 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.616e+02 3.471e+02 3.853e+02 4.480e+02 6.109e+02, threshold=7.707e+02, percent-clipped=0.0 2026-09-24 06:09:22,286 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=191506.66666666666, ans=0.125 2026-09-24 06:09:31,555 INFO [train.py:1192] (0/2) Epoch 60, batch 950, loss[loss=0.3569, simple_loss=0.421, pruned_loss=0.1465, over 10813.00 frames. ], tot_loss[loss=0.2599, simple_loss=0.3754, pruned_loss=0.07225, over 4712512.64 frames. ], batch size: 333, lr: 3.50e-03, grad_scale: 32.0 2026-09-24 06:09:35,752 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-60.pt 2026-09-24 06:09:41,446 INFO [train.py:1192] (0/2) Epoch 61, batch 0, loss[loss=0.2132, simple_loss=0.3352, pruned_loss=0.04557, over 24564.00 frames. ], tot_loss[loss=0.2132, simple_loss=0.3352, pruned_loss=0.04557, over 24564.00 frames. ], batch size: 137, lr: 3.47e-03, grad_scale: 32.0 2026-09-24 06:09:41,446 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 06:09:51,653 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([3.3405, 2.4181, 3.8342, 1.5303], device='cuda:0') 2026-09-24 06:09:53,106 INFO [train.py:1224] (0/2) Epoch 61, validation: loss=0.17, simple_loss=0.2887, pruned_loss=0.02562, over 2564189.00 frames. 2026-09-24 06:09:53,106 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 06:10:00,225 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=191633.33333333334, ans=0.0 2026-09-24 06:10:18,463 INFO [train.py:1192] (0/2) Epoch 61, batch 50, loss[loss=0.2465, simple_loss=0.3519, pruned_loss=0.07057, over 24257.00 frames. ], tot_loss[loss=0.267, simple_loss=0.3825, pruned_loss=0.07578, over 1078153.07 frames. ], batch size: 125, lr: 3.47e-03, grad_scale: 32.0 2026-09-24 06:10:18,545 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=191766.66666666666, ans=0.1 2026-09-24 06:10:20,500 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=191766.66666666666, ans=0.0 2026-09-24 06:10:28,374 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.687e+02 3.624e+02 4.017e+02 4.823e+02 1.136e+03, threshold=8.035e+02, percent-clipped=1.0 2026-09-24 06:10:31,833 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=191833.33333333334, ans=0.07 2026-09-24 06:10:37,380 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=191866.66666666666, ans=0.1 2026-09-24 06:10:44,283 INFO [train.py:1192] (0/2) Epoch 61, batch 100, loss[loss=0.2449, simple_loss=0.3629, pruned_loss=0.06343, over 24621.00 frames. ], tot_loss[loss=0.2661, simple_loss=0.3837, pruned_loss=0.07428, over 1905590.56 frames. ], batch size: 154, lr: 3.47e-03, grad_scale: 32.0 2026-09-24 06:10:45,034 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.54 vs. limit=10.0 2026-09-24 06:10:47,487 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=191933.33333333334, ans=0.125 2026-09-24 06:10:47,619 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=15.0 2026-09-24 06:10:52,554 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.15 vs. limit=22.5 2026-09-24 06:10:53,443 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=191966.66666666666, ans=0.125 2026-09-24 06:10:54,614 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.60 vs. limit=22.5 2026-09-24 06:10:55,028 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=192000.0, ans=0.2 2026-09-24 06:11:01,445 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.86 vs. limit=8.0 2026-09-24 06:11:03,762 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=192033.33333333334, ans=0.0 2026-09-24 06:11:10,555 INFO [train.py:1192] (0/2) Epoch 61, batch 150, loss[loss=0.2219, simple_loss=0.3327, pruned_loss=0.05558, over 24266.00 frames. ], tot_loss[loss=0.2628, simple_loss=0.38, pruned_loss=0.07276, over 2559712.84 frames. ], batch size: 125, lr: 3.47e-03, grad_scale: 32.0 2026-09-24 06:11:12,172 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=192100.0, ans=0.125 2026-09-24 06:11:13,810 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.86 vs. limit=15.0 2026-09-24 06:11:19,765 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.514e+02 3.411e+02 3.849e+02 4.309e+02 6.280e+02, threshold=7.698e+02, percent-clipped=0.0 2026-09-24 06:11:22,010 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=192166.66666666666, ans=0.1 2026-09-24 06:11:22,621 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.34 vs. limit=15.0 2026-09-24 06:11:33,470 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=192233.33333333334, ans=0.2 2026-09-24 06:11:35,626 INFO [train.py:1192] (0/2) Epoch 61, batch 200, loss[loss=0.321, simple_loss=0.4247, pruned_loss=0.1087, over 21073.00 frames. ], tot_loss[loss=0.2605, simple_loss=0.3782, pruned_loss=0.07141, over 3056257.69 frames. ], batch size: 333, lr: 3.47e-03, grad_scale: 32.0 2026-09-24 06:11:45,782 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten.whitening_limit, batch_count=192333.33333333334, ans=22.5 2026-09-24 06:11:50,043 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=192333.33333333334, ans=0.1 2026-09-24 06:11:54,235 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=192366.66666666666, ans=0.125 2026-09-24 06:11:55,455 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.78 vs. limit=15.0 2026-09-24 06:12:01,365 INFO [train.py:1192] (0/2) Epoch 61, batch 250, loss[loss=0.2869, simple_loss=0.4139, pruned_loss=0.07995, over 24311.00 frames. ], tot_loss[loss=0.2606, simple_loss=0.3779, pruned_loss=0.0717, over 3443881.92 frames. ], batch size: 234, lr: 3.46e-03, grad_scale: 32.0 2026-09-24 06:12:08,821 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=192466.66666666666, ans=0.0 2026-09-24 06:12:11,212 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.616e+02 3.434e+02 3.850e+02 4.566e+02 6.399e+02, threshold=7.699e+02, percent-clipped=0.0 2026-09-24 06:12:19,147 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.64 vs. limit=22.5 2026-09-24 06:12:21,304 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.51 vs. limit=15.0 2026-09-24 06:12:26,500 INFO [train.py:1192] (0/2) Epoch 61, batch 300, loss[loss=0.2649, simple_loss=0.391, pruned_loss=0.06938, over 24538.00 frames. ], tot_loss[loss=0.2587, simple_loss=0.3759, pruned_loss=0.07078, over 3749694.23 frames. ], batch size: 204, lr: 3.46e-03, grad_scale: 32.0 2026-09-24 06:12:28,117 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=192600.0, ans=0.0 2026-09-24 06:12:37,895 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=192666.66666666666, ans=0.0 2026-09-24 06:12:43,538 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=192700.0, ans=0.1 2026-09-24 06:12:52,166 INFO [train.py:1192] (0/2) Epoch 61, batch 350, loss[loss=0.2136, simple_loss=0.3272, pruned_loss=0.04996, over 24583.00 frames. ], tot_loss[loss=0.2598, simple_loss=0.3772, pruned_loss=0.07117, over 3994221.08 frames. ], batch size: 137, lr: 3.46e-03, grad_scale: 32.0 2026-09-24 06:12:56,530 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=192766.66666666666, ans=0.0 2026-09-24 06:12:57,000 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=192800.0, ans=0.125 2026-09-24 06:13:01,848 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.749e+02 3.413e+02 3.727e+02 4.235e+02 7.272e+02, threshold=7.455e+02, percent-clipped=0.0 2026-09-24 06:13:03,894 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.86 vs. limit=6.0 2026-09-24 06:13:04,260 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.scale_min, batch_count=192833.33333333334, ans=0.2 2026-09-24 06:13:04,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=192833.33333333334, ans=0.2 2026-09-24 06:13:06,133 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff2_skip_rate, batch_count=192833.33333333334, ans=0.0 2026-09-24 06:13:17,078 INFO [train.py:1192] (0/2) Epoch 61, batch 400, loss[loss=0.2533, simple_loss=0.374, pruned_loss=0.06631, over 24555.00 frames. ], tot_loss[loss=0.259, simple_loss=0.3763, pruned_loss=0.07081, over 4180567.05 frames. ], batch size: 170, lr: 3.46e-03, grad_scale: 32.0 2026-09-24 06:13:22,630 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.78 vs. limit=15.0 2026-09-24 06:13:24,943 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=192966.66666666666, ans=0.015 2026-09-24 06:13:33,380 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=193033.33333333334, ans=0.0 2026-09-24 06:13:43,515 INFO [train.py:1192] (0/2) Epoch 61, batch 450, loss[loss=0.2672, simple_loss=0.3833, pruned_loss=0.07556, over 24643.00 frames. ], tot_loss[loss=0.2599, simple_loss=0.3771, pruned_loss=0.07131, over 4313555.76 frames. ], batch size: 175, lr: 3.46e-03, grad_scale: 32.0 2026-09-24 06:13:45,709 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=193100.0, ans=0.1 2026-09-24 06:13:53,054 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.637e+02 3.431e+02 3.745e+02 4.317e+02 6.858e+02, threshold=7.490e+02, percent-clipped=0.0 2026-09-24 06:13:57,337 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=193166.66666666666, ans=0.0 2026-09-24 06:14:03,118 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=193200.0, ans=0.0 2026-09-24 06:14:09,331 INFO [train.py:1192] (0/2) Epoch 61, batch 500, loss[loss=0.2756, simple_loss=0.3986, pruned_loss=0.07627, over 24527.00 frames. ], tot_loss[loss=0.259, simple_loss=0.3757, pruned_loss=0.07116, over 4430639.13 frames. ], batch size: 218, lr: 3.46e-03, grad_scale: 32.0 2026-09-24 06:14:09,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=193266.66666666666, ans=0.125 2026-09-24 06:14:26,963 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.61 vs. limit=6.0 2026-09-24 06:14:33,068 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.97 vs. limit=15.0 2026-09-24 06:14:34,749 INFO [train.py:1192] (0/2) Epoch 61, batch 550, loss[loss=0.2574, simple_loss=0.39, pruned_loss=0.06235, over 24307.00 frames. ], tot_loss[loss=0.2587, simple_loss=0.3757, pruned_loss=0.07084, over 4519835.60 frames. ], batch size: 257, lr: 3.46e-03, grad_scale: 32.0 2026-09-24 06:14:39,960 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.50 vs. limit=15.0 2026-09-24 06:14:44,570 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.586e+02 3.162e+02 3.590e+02 3.957e+02 6.076e+02, threshold=7.179e+02, percent-clipped=0.0 2026-09-24 06:15:00,776 INFO [train.py:1192] (0/2) Epoch 61, batch 600, loss[loss=0.2648, simple_loss=0.3965, pruned_loss=0.06653, over 24304.00 frames. ], tot_loss[loss=0.2598, simple_loss=0.3769, pruned_loss=0.07139, over 4586318.64 frames. ], batch size: 234, lr: 3.45e-03, grad_scale: 32.0 2026-09-24 06:15:07,864 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=193633.33333333334, ans=0.125 2026-09-24 06:15:23,840 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=193733.33333333334, ans=0.0 2026-09-24 06:15:26,175 INFO [train.py:1192] (0/2) Epoch 61, batch 650, loss[loss=0.2636, simple_loss=0.3794, pruned_loss=0.07389, over 24566.00 frames. ], tot_loss[loss=0.2589, simple_loss=0.3761, pruned_loss=0.07082, over 4651494.48 frames. ], batch size: 162, lr: 3.45e-03, grad_scale: 16.0 2026-09-24 06:15:28,611 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=193766.66666666666, ans=0.95 2026-09-24 06:15:30,036 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=193766.66666666666, ans=0.1 2026-09-24 06:15:35,680 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=193800.0, ans=0.125 2026-09-24 06:15:36,529 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.685e+02 3.370e+02 3.618e+02 4.011e+02 6.223e+02, threshold=7.236e+02, percent-clipped=0.0 2026-09-24 06:15:41,114 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=193833.33333333334, ans=0.125 2026-09-24 06:15:46,303 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=193866.66666666666, ans=0.125 2026-09-24 06:15:48,518 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=11.00 vs. limit=15.0 2026-09-24 06:15:51,736 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=193933.33333333334, ans=0.125 2026-09-24 06:15:52,128 INFO [train.py:1192] (0/2) Epoch 61, batch 700, loss[loss=0.252, simple_loss=0.3656, pruned_loss=0.06924, over 24551.00 frames. ], tot_loss[loss=0.2602, simple_loss=0.3774, pruned_loss=0.07153, over 4683943.94 frames. ], batch size: 154, lr: 3.45e-03, grad_scale: 16.0 2026-09-24 06:16:00,998 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=193966.66666666666, ans=0.09899494936611666 2026-09-24 06:16:03,657 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=194000.0, ans=0.125 2026-09-24 06:16:15,540 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=194066.66666666666, ans=0.0 2026-09-24 06:16:17,965 INFO [train.py:1192] (0/2) Epoch 61, batch 750, loss[loss=0.2661, simple_loss=0.3873, pruned_loss=0.07243, over 24562.00 frames. ], tot_loss[loss=0.259, simple_loss=0.376, pruned_loss=0.07097, over 4710844.63 frames. ], batch size: 170, lr: 3.45e-03, grad_scale: 16.0 2026-09-24 06:16:19,608 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=194100.0, ans=0.125 2026-09-24 06:16:25,170 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=194133.33333333334, ans=0.125 2026-09-24 06:16:25,339 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.40 vs. limit=10.0 2026-09-24 06:16:28,057 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.911e+02 3.503e+02 4.008e+02 4.518e+02 6.214e+02, threshold=8.016e+02, percent-clipped=0.0 2026-09-24 06:16:32,686 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.46 vs. limit=6.0 2026-09-24 06:16:37,653 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.18 vs. limit=15.0 2026-09-24 06:16:39,686 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.26 vs. limit=15.0 2026-09-24 06:16:40,046 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=194233.33333333334, ans=0.025 2026-09-24 06:16:43,080 INFO [train.py:1192] (0/2) Epoch 61, batch 800, loss[loss=0.2312, simple_loss=0.3415, pruned_loss=0.06048, over 24533.00 frames. ], tot_loss[loss=0.2592, simple_loss=0.3762, pruned_loss=0.07115, over 4739764.02 frames. ], batch size: 137, lr: 3.45e-03, grad_scale: 32.0 2026-09-24 06:16:44,172 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=194266.66666666666, ans=0.0 2026-09-24 06:16:45,722 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=194266.66666666666, ans=0.015 2026-09-24 06:16:47,457 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=194266.66666666666, ans=0.125 2026-09-24 06:16:53,253 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=194333.33333333334, ans=0.125 2026-09-24 06:17:00,713 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=194366.66666666666, ans=0.125 2026-09-24 06:17:08,615 INFO [train.py:1192] (0/2) Epoch 61, batch 850, loss[loss=0.255, simple_loss=0.3792, pruned_loss=0.0654, over 24603.00 frames. ], tot_loss[loss=0.2593, simple_loss=0.3761, pruned_loss=0.07126, over 4761923.23 frames. ], batch size: 198, lr: 3.45e-03, grad_scale: 32.0 2026-09-24 06:17:19,023 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.850e+02 3.451e+02 3.900e+02 4.481e+02 6.204e+02, threshold=7.800e+02, percent-clipped=0.0 2026-09-24 06:17:22,656 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:17:25,412 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=194533.33333333334, ans=0.125 2026-09-24 06:17:34,281 INFO [train.py:1192] (0/2) Epoch 61, batch 900, loss[loss=0.2065, simple_loss=0.3261, pruned_loss=0.04342, over 24549.00 frames. ], tot_loss[loss=0.2592, simple_loss=0.3761, pruned_loss=0.07117, over 4774908.16 frames. ], batch size: 137, lr: 3.45e-03, grad_scale: 32.0 2026-09-24 06:17:38,738 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=194600.0, ans=0.1 2026-09-24 06:17:50,311 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.94 vs. limit=22.5 2026-09-24 06:17:59,267 INFO [train.py:1192] (0/2) Epoch 61, batch 950, loss[loss=0.3588, simple_loss=0.4313, pruned_loss=0.1432, over 11522.00 frames. ], tot_loss[loss=0.2597, simple_loss=0.3753, pruned_loss=0.0721, over 4716190.71 frames. ], batch size: 333, lr: 3.44e-03, grad_scale: 32.0 2026-09-24 06:18:03,584 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-61.pt 2026-09-24 06:18:07,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=194793.33333333334, ans=0.125 2026-09-24 06:18:08,384 INFO [train.py:1192] (0/2) Epoch 62, batch 0, loss[loss=0.2194, simple_loss=0.3365, pruned_loss=0.05112, over 24538.00 frames. ], tot_loss[loss=0.2194, simple_loss=0.3365, pruned_loss=0.05112, over 24538.00 frames. ], batch size: 137, lr: 3.42e-03, grad_scale: 32.0 2026-09-24 06:18:08,385 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 06:18:20,016 INFO [train.py:1224] (0/2) Epoch 62, validation: loss=0.1718, simple_loss=0.29, pruned_loss=0.02679, over 2564189.00 frames. 2026-09-24 06:18:20,017 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 06:18:25,909 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.504e+02 3.508e+02 3.896e+02 4.877e+02 6.832e+02, threshold=7.792e+02, percent-clipped=0.0 2026-09-24 06:18:27,510 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=194826.66666666666, ans=0.125 2026-09-24 06:18:39,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=194926.66666666666, ans=0.2 2026-09-24 06:18:45,135 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=194926.66666666666, ans=0.025 2026-09-24 06:18:46,003 INFO [train.py:1192] (0/2) Epoch 62, batch 50, loss[loss=0.2379, simple_loss=0.348, pruned_loss=0.0639, over 24221.00 frames. ], tot_loss[loss=0.2656, simple_loss=0.3815, pruned_loss=0.07487, over 1074349.80 frames. ], batch size: 125, lr: 3.41e-03, grad_scale: 32.0 2026-09-24 06:18:51,910 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=194993.33333333334, ans=0.0 2026-09-24 06:18:56,194 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=195026.66666666666, ans=0.1 2026-09-24 06:19:06,588 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=195093.33333333334, ans=0.125 2026-09-24 06:19:09,863 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=195093.33333333334, ans=0.015 2026-09-24 06:19:11,752 INFO [train.py:1192] (0/2) Epoch 62, batch 100, loss[loss=0.2622, simple_loss=0.3729, pruned_loss=0.07573, over 24598.00 frames. ], tot_loss[loss=0.2673, simple_loss=0.3846, pruned_loss=0.07496, over 1903910.62 frames. ], batch size: 154, lr: 3.41e-03, grad_scale: 32.0 2026-09-24 06:19:13,454 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.59 vs. limit=6.0 2026-09-24 06:19:17,833 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.792e+02 3.477e+02 3.857e+02 4.199e+02 8.261e+02, threshold=7.714e+02, percent-clipped=1.0 2026-09-24 06:19:24,394 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=195193.33333333334, ans=0.1 2026-09-24 06:19:27,129 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=195226.66666666666, ans=0.125 2026-09-24 06:19:30,332 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=195226.66666666666, ans=0.09899494936611666 2026-09-24 06:19:36,620 INFO [train.py:1192] (0/2) Epoch 62, batch 150, loss[loss=0.2269, simple_loss=0.3313, pruned_loss=0.06129, over 24219.00 frames. ], tot_loss[loss=0.2616, simple_loss=0.3794, pruned_loss=0.07192, over 2557604.09 frames. ], batch size: 125, lr: 3.41e-03, grad_scale: 32.0 2026-09-24 06:19:40,498 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.98 vs. limit=6.0 2026-09-24 06:19:42,132 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=195326.66666666666, ans=0.125 2026-09-24 06:19:49,706 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.98 vs. limit=10.0 2026-09-24 06:19:54,482 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=195393.33333333334, ans=0.1 2026-09-24 06:20:01,477 INFO [train.py:1192] (0/2) Epoch 62, batch 200, loss[loss=0.3056, simple_loss=0.4074, pruned_loss=0.1019, over 20949.00 frames. ], tot_loss[loss=0.2604, simple_loss=0.3779, pruned_loss=0.07138, over 3053528.43 frames. ], batch size: 333, lr: 3.41e-03, grad_scale: 32.0 2026-09-24 06:20:07,471 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.623e+02 3.457e+02 3.962e+02 4.499e+02 6.381e+02, threshold=7.924e+02, percent-clipped=0.0 2026-09-24 06:20:11,355 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=195526.66666666666, ans=0.1 2026-09-24 06:20:11,985 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=195526.66666666666, ans=15.0 2026-09-24 06:20:12,809 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=195526.66666666666, ans=0.125 2026-09-24 06:20:20,669 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer2.prob, batch_count=195560.0, ans=0.125 2026-09-24 06:20:23,019 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=195593.33333333334, ans=0.07 2026-09-24 06:20:27,547 INFO [train.py:1192] (0/2) Epoch 62, batch 250, loss[loss=0.2921, simple_loss=0.4142, pruned_loss=0.08497, over 24305.00 frames. ], tot_loss[loss=0.2598, simple_loss=0.3773, pruned_loss=0.07121, over 3442634.40 frames. ], batch size: 234, lr: 3.41e-03, grad_scale: 32.0 2026-09-24 06:20:38,895 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=11.15 vs. limit=15.0 2026-09-24 06:20:52,590 INFO [train.py:1192] (0/2) Epoch 62, batch 300, loss[loss=0.261, simple_loss=0.3925, pruned_loss=0.06475, over 24516.00 frames. ], tot_loss[loss=0.259, simple_loss=0.376, pruned_loss=0.07098, over 3748165.97 frames. ], batch size: 204, lr: 3.41e-03, grad_scale: 32.0 2026-09-24 06:20:58,304 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.643e+02 3.333e+02 3.738e+02 4.237e+02 6.549e+02, threshold=7.477e+02, percent-clipped=0.0 2026-09-24 06:21:02,361 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=195860.0, ans=0.125 2026-09-24 06:21:07,570 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=195893.33333333334, ans=0.125 2026-09-24 06:21:10,465 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=195893.33333333334, ans=0.0 2026-09-24 06:21:10,481 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:21:17,885 INFO [train.py:1192] (0/2) Epoch 62, batch 350, loss[loss=0.2134, simple_loss=0.323, pruned_loss=0.05189, over 24582.00 frames. ], tot_loss[loss=0.2598, simple_loss=0.3771, pruned_loss=0.07124, over 3988217.17 frames. ], batch size: 137, lr: 3.41e-03, grad_scale: 32.0 2026-09-24 06:21:26,359 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.40 vs. limit=6.0 2026-09-24 06:21:43,346 INFO [train.py:1192] (0/2) Epoch 62, batch 400, loss[loss=0.2828, simple_loss=0.4016, pruned_loss=0.08199, over 24576.00 frames. ], tot_loss[loss=0.2592, simple_loss=0.3765, pruned_loss=0.07092, over 4175905.14 frames. ], batch size: 170, lr: 3.40e-03, grad_scale: 32.0 2026-09-24 06:21:45,666 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=196126.66666666666, ans=0.125 2026-09-24 06:21:49,067 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.838e+02 3.389e+02 3.914e+02 4.523e+02 6.865e+02, threshold=7.829e+02, percent-clipped=0.0 2026-09-24 06:21:53,304 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=196193.33333333334, ans=0.2 2026-09-24 06:22:03,422 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=196260.0, ans=0.2 2026-09-24 06:22:08,539 INFO [train.py:1192] (0/2) Epoch 62, batch 450, loss[loss=0.2758, simple_loss=0.3946, pruned_loss=0.07848, over 24628.00 frames. ], tot_loss[loss=0.2596, simple_loss=0.3769, pruned_loss=0.07114, over 4310800.20 frames. ], batch size: 175, lr: 3.40e-03, grad_scale: 32.0 2026-09-24 06:22:09,157 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=196293.33333333334, ans=0.125 2026-09-24 06:22:11,149 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=196293.33333333334, ans=0.125 2026-09-24 06:22:14,826 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=196326.66666666666, ans=0.125 2026-09-24 06:22:24,767 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=196393.33333333334, ans=0.04949747468305833 2026-09-24 06:22:25,571 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.57 vs. limit=12.0 2026-09-24 06:22:33,603 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.95 vs. limit=22.5 2026-09-24 06:22:33,889 INFO [train.py:1192] (0/2) Epoch 62, batch 500, loss[loss=0.2835, simple_loss=0.4122, pruned_loss=0.07736, over 24531.00 frames. ], tot_loss[loss=0.2589, simple_loss=0.3757, pruned_loss=0.07105, over 4428180.91 frames. ], batch size: 218, lr: 3.40e-03, grad_scale: 32.0 2026-09-24 06:22:40,093 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.696e+02 3.424e+02 3.778e+02 4.258e+02 6.180e+02, threshold=7.556e+02, percent-clipped=0.0 2026-09-24 06:22:44,007 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=196526.66666666666, ans=0.2 2026-09-24 06:22:45,871 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:22:59,923 INFO [train.py:1192] (0/2) Epoch 62, batch 550, loss[loss=0.2841, simple_loss=0.4174, pruned_loss=0.07543, over 24275.00 frames. ], tot_loss[loss=0.2591, simple_loss=0.3763, pruned_loss=0.07098, over 4518530.94 frames. ], batch size: 257, lr: 3.40e-03, grad_scale: 32.0 2026-09-24 06:23:20,216 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass_mid.scale_min, batch_count=196760.0, ans=0.2 2026-09-24 06:23:22,987 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=196760.0, ans=0.125 2026-09-24 06:23:25,013 INFO [train.py:1192] (0/2) Epoch 62, batch 600, loss[loss=0.2748, simple_loss=0.403, pruned_loss=0.07329, over 24318.00 frames. ], tot_loss[loss=0.2594, simple_loss=0.3768, pruned_loss=0.07102, over 4586111.80 frames. ], batch size: 234, lr: 3.40e-03, grad_scale: 32.0 2026-09-24 06:23:27,051 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=196793.33333333334, ans=0.125 2026-09-24 06:23:28,051 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=196793.33333333334, ans=0.1 2026-09-24 06:23:29,463 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=196826.66666666666, ans=0.125 2026-09-24 06:23:30,925 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.705e+02 3.299e+02 3.732e+02 4.086e+02 5.579e+02, threshold=7.464e+02, percent-clipped=0.0 2026-09-24 06:23:50,018 INFO [train.py:1192] (0/2) Epoch 62, batch 650, loss[loss=0.2515, simple_loss=0.3731, pruned_loss=0.06499, over 24561.00 frames. ], tot_loss[loss=0.2578, simple_loss=0.3754, pruned_loss=0.07009, over 4651503.73 frames. ], batch size: 162, lr: 3.40e-03, grad_scale: 32.0 2026-09-24 06:23:54,840 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=196993.33333333334, ans=0.125 2026-09-24 06:24:02,955 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=197026.66666666666, ans=0.2 2026-09-24 06:24:15,511 INFO [train.py:1192] (0/2) Epoch 62, batch 700, loss[loss=0.2418, simple_loss=0.3557, pruned_loss=0.06402, over 24562.00 frames. ], tot_loss[loss=0.2577, simple_loss=0.3757, pruned_loss=0.06985, over 4683595.77 frames. ], batch size: 154, lr: 3.40e-03, grad_scale: 32.0 2026-09-24 06:24:17,709 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=197126.66666666666, ans=0.0 2026-09-24 06:24:20,726 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=197160.0, ans=0.125 2026-09-24 06:24:21,540 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.887e+02 3.370e+02 3.867e+02 4.370e+02 6.944e+02, threshold=7.734e+02, percent-clipped=0.0 2026-09-24 06:24:21,645 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=197160.0, ans=0.0 2026-09-24 06:24:23,826 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=197160.0, ans=0.125 2026-09-24 06:24:23,837 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=197160.0, ans=0.1 2026-09-24 06:24:25,240 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=197193.33333333334, ans=0.0 2026-09-24 06:24:26,096 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=197193.33333333334, ans=0.05 2026-09-24 06:24:26,558 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=10.83 vs. limit=12.0 2026-09-24 06:24:40,635 INFO [train.py:1192] (0/2) Epoch 62, batch 750, loss[loss=0.2365, simple_loss=0.3643, pruned_loss=0.05441, over 24550.00 frames. ], tot_loss[loss=0.2569, simple_loss=0.3746, pruned_loss=0.06955, over 4711439.03 frames. ], batch size: 170, lr: 3.39e-03, grad_scale: 32.0 2026-09-24 06:24:51,756 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=197360.0, ans=0.125 2026-09-24 06:24:52,973 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.61 vs. limit=8.0 2026-09-24 06:25:05,419 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=197426.66666666666, ans=0.125 2026-09-24 06:25:06,419 INFO [train.py:1192] (0/2) Epoch 62, batch 800, loss[loss=0.2223, simple_loss=0.3367, pruned_loss=0.05401, over 24549.00 frames. ], tot_loss[loss=0.2568, simple_loss=0.3746, pruned_loss=0.06955, over 4736634.38 frames. ], batch size: 137, lr: 3.39e-03, grad_scale: 32.0 2026-09-24 06:25:09,063 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=197460.0, ans=0.025 2026-09-24 06:25:12,833 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.689e+02 3.485e+02 3.958e+02 4.384e+02 6.565e+02, threshold=7.916e+02, percent-clipped=0.0 2026-09-24 06:25:27,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=197593.33333333334, ans=0.07 2026-09-24 06:25:27,076 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=197593.33333333334, ans=0.125 2026-09-24 06:25:31,745 INFO [train.py:1192] (0/2) Epoch 62, batch 850, loss[loss=0.2766, simple_loss=0.4022, pruned_loss=0.07548, over 24572.00 frames. ], tot_loss[loss=0.2569, simple_loss=0.3746, pruned_loss=0.06955, over 4759165.13 frames. ], batch size: 198, lr: 3.39e-03, grad_scale: 32.0 2026-09-24 06:25:42,405 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.35 vs. limit=15.0 2026-09-24 06:25:57,804 INFO [train.py:1192] (0/2) Epoch 62, batch 900, loss[loss=0.2326, simple_loss=0.3437, pruned_loss=0.06082, over 24575.00 frames. ], tot_loss[loss=0.2575, simple_loss=0.3752, pruned_loss=0.06993, over 4772813.62 frames. ], batch size: 137, lr: 3.39e-03, grad_scale: 32.0 2026-09-24 06:26:04,314 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.853e+02 3.460e+02 4.122e+02 4.592e+02 8.345e+02, threshold=8.244e+02, percent-clipped=1.0 2026-09-24 06:26:04,833 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=197826.66666666666, ans=0.1 2026-09-24 06:26:13,409 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=197893.33333333334, ans=0.2 2026-09-24 06:26:21,239 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=197926.66666666666, ans=0.0 2026-09-24 06:26:23,048 INFO [train.py:1192] (0/2) Epoch 62, batch 950, loss[loss=0.3414, simple_loss=0.4187, pruned_loss=0.1321, over 11875.00 frames. ], tot_loss[loss=0.2575, simple_loss=0.374, pruned_loss=0.07053, over 4712830.43 frames. ], batch size: 333, lr: 3.39e-03, grad_scale: 32.0 2026-09-24 06:26:27,380 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-62.pt 2026-09-24 06:26:33,329 INFO [train.py:1192] (0/2) Epoch 63, batch 0, loss[loss=0.2063, simple_loss=0.3256, pruned_loss=0.04356, over 24584.00 frames. ], tot_loss[loss=0.2063, simple_loss=0.3256, pruned_loss=0.04356, over 24584.00 frames. ], batch size: 137, lr: 3.36e-03, grad_scale: 32.0 2026-09-24 06:26:33,329 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 06:26:44,986 INFO [train.py:1224] (0/2) Epoch 63, validation: loss=0.1706, simple_loss=0.2889, pruned_loss=0.02612, over 2564189.00 frames. 2026-09-24 06:26:44,986 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 06:26:49,923 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.43 vs. limit=15.0 2026-09-24 06:26:50,322 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=198020.0, ans=0.125 2026-09-24 06:26:54,392 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=198020.0, ans=0.125 2026-09-24 06:26:54,908 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=198053.33333333334, ans=0.125 2026-09-24 06:27:05,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=198120.0, ans=0.0 2026-09-24 06:27:08,025 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=198120.0, ans=0.125 2026-09-24 06:27:10,463 INFO [train.py:1192] (0/2) Epoch 63, batch 50, loss[loss=0.2328, simple_loss=0.339, pruned_loss=0.06326, over 24249.00 frames. ], tot_loss[loss=0.2655, simple_loss=0.3822, pruned_loss=0.07437, over 1076222.19 frames. ], batch size: 125, lr: 3.36e-03, grad_scale: 32.0 2026-09-24 06:27:12,233 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.726e+02 3.514e+02 4.081e+02 4.504e+02 6.153e+02, threshold=8.162e+02, percent-clipped=0.0 2026-09-24 06:27:12,796 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=198153.33333333334, ans=0.1 2026-09-24 06:27:13,746 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=198153.33333333334, ans=0.0 2026-09-24 06:27:14,252 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=198153.33333333334, ans=0.125 2026-09-24 06:27:23,516 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=198220.0, ans=0.0 2026-09-24 06:27:25,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=198253.33333333334, ans=0.1 2026-09-24 06:27:35,847 INFO [train.py:1192] (0/2) Epoch 63, batch 100, loss[loss=0.2467, simple_loss=0.3598, pruned_loss=0.06684, over 24617.00 frames. ], tot_loss[loss=0.2665, simple_loss=0.3843, pruned_loss=0.07428, over 1905678.54 frames. ], batch size: 154, lr: 3.36e-03, grad_scale: 32.0 2026-09-24 06:27:45,408 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=198353.33333333334, ans=0.2 2026-09-24 06:27:45,832 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=198386.66666666666, ans=0.125 2026-09-24 06:27:49,334 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.37 vs. limit=15.0 2026-09-24 06:28:00,787 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=198453.33333333334, ans=0.0 2026-09-24 06:28:01,702 INFO [train.py:1192] (0/2) Epoch 63, batch 150, loss[loss=0.2007, simple_loss=0.3131, pruned_loss=0.04419, over 24229.00 frames. ], tot_loss[loss=0.2618, simple_loss=0.38, pruned_loss=0.07183, over 2558841.88 frames. ], batch size: 125, lr: 3.36e-03, grad_scale: 32.0 2026-09-24 06:28:01,779 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=198486.66666666666, ans=0.1 2026-09-24 06:28:03,634 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.814e+02 3.458e+02 3.872e+02 4.270e+02 5.583e+02, threshold=7.744e+02, percent-clipped=0.0 2026-09-24 06:28:07,513 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=198520.0, ans=0.1 2026-09-24 06:28:17,211 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=198586.66666666666, ans=0.125 2026-09-24 06:28:27,414 INFO [train.py:1192] (0/2) Epoch 63, batch 200, loss[loss=0.3178, simple_loss=0.415, pruned_loss=0.1103, over 21009.00 frames. ], tot_loss[loss=0.26, simple_loss=0.3781, pruned_loss=0.07093, over 3055306.79 frames. ], batch size: 333, lr: 3.36e-03, grad_scale: 32.0 2026-09-24 06:28:41,564 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.42 vs. limit=15.0 2026-09-24 06:28:45,163 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=198753.33333333334, ans=0.0 2026-09-24 06:28:50,832 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=198786.66666666666, ans=0.0 2026-09-24 06:28:52,838 INFO [train.py:1192] (0/2) Epoch 63, batch 250, loss[loss=0.2805, simple_loss=0.41, pruned_loss=0.07547, over 24295.00 frames. ], tot_loss[loss=0.2604, simple_loss=0.3781, pruned_loss=0.07136, over 3444643.98 frames. ], batch size: 234, lr: 3.35e-03, grad_scale: 16.0 2026-09-24 06:28:55,580 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.737e+02 3.490e+02 3.926e+02 4.577e+02 6.260e+02, threshold=7.853e+02, percent-clipped=0.0 2026-09-24 06:29:01,997 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.71 vs. limit=15.0 2026-09-24 06:29:05,039 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=198886.66666666666, ans=0.0 2026-09-24 06:29:14,144 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=198953.33333333334, ans=0.125 2026-09-24 06:29:17,924 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.56 vs. limit=15.0 2026-09-24 06:29:18,058 INFO [train.py:1192] (0/2) Epoch 63, batch 300, loss[loss=0.2587, simple_loss=0.3863, pruned_loss=0.06552, over 24556.00 frames. ], tot_loss[loss=0.2587, simple_loss=0.3762, pruned_loss=0.07065, over 3750255.17 frames. ], batch size: 204, lr: 3.35e-03, grad_scale: 16.0 2026-09-24 06:29:18,603 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=198986.66666666666, ans=0.125 2026-09-24 06:29:20,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=198986.66666666666, ans=0.2 2026-09-24 06:29:24,024 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.17 vs. limit=22.5 2026-09-24 06:29:29,106 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=199053.33333333334, ans=0.0 2026-09-24 06:29:29,108 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=199053.33333333334, ans=0.2 2026-09-24 06:29:33,938 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=199086.66666666666, ans=0.5 2026-09-24 06:29:34,018 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.73 vs. limit=15.0 2026-09-24 06:29:34,755 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=199086.66666666666, ans=0.1 2026-09-24 06:29:43,383 INFO [train.py:1192] (0/2) Epoch 63, batch 350, loss[loss=0.2163, simple_loss=0.3307, pruned_loss=0.05096, over 24571.00 frames. ], tot_loss[loss=0.259, simple_loss=0.3767, pruned_loss=0.07068, over 3993553.06 frames. ], batch size: 137, lr: 3.35e-03, grad_scale: 16.0 2026-09-24 06:29:44,478 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.10 vs. limit=15.0 2026-09-24 06:29:45,498 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.09 vs. limit=22.5 2026-09-24 06:29:45,836 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.672e+02 3.255e+02 3.680e+02 4.171e+02 6.940e+02, threshold=7.361e+02, percent-clipped=0.0 2026-09-24 06:29:49,871 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=199186.66666666666, ans=0.125 2026-09-24 06:30:05,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=199286.66666666666, ans=0.125 2026-09-24 06:30:08,945 INFO [train.py:1192] (0/2) Epoch 63, batch 400, loss[loss=0.2893, simple_loss=0.3973, pruned_loss=0.09063, over 24568.00 frames. ], tot_loss[loss=0.2591, simple_loss=0.3766, pruned_loss=0.07078, over 4180432.66 frames. ], batch size: 170, lr: 3.35e-03, grad_scale: 32.0 2026-09-24 06:30:11,343 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=199320.0, ans=0.125 2026-09-24 06:30:20,733 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.08 vs. limit=10.0 2026-09-24 06:30:22,071 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=199386.66666666666, ans=0.1 2026-09-24 06:30:23,448 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=199420.0, ans=0.125 2026-09-24 06:30:34,533 INFO [train.py:1192] (0/2) Epoch 63, batch 450, loss[loss=0.2771, simple_loss=0.3975, pruned_loss=0.07836, over 24609.00 frames. ], tot_loss[loss=0.2593, simple_loss=0.3768, pruned_loss=0.07089, over 4313581.92 frames. ], batch size: 175, lr: 3.35e-03, grad_scale: 32.0 2026-09-24 06:30:37,032 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.697e+02 3.269e+02 3.844e+02 4.496e+02 6.455e+02, threshold=7.689e+02, percent-clipped=0.0 2026-09-24 06:30:39,926 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=199520.0, ans=0.1 2026-09-24 06:30:46,124 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=199553.33333333334, ans=0.125 2026-09-24 06:30:48,985 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=199553.33333333334, ans=0.125 2026-09-24 06:30:50,473 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=199586.66666666666, ans=0.0 2026-09-24 06:30:55,875 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=199620.0, ans=0.125 2026-09-24 06:31:00,202 INFO [train.py:1192] (0/2) Epoch 63, batch 500, loss[loss=0.3011, simple_loss=0.4166, pruned_loss=0.0928, over 24460.00 frames. ], tot_loss[loss=0.2579, simple_loss=0.3752, pruned_loss=0.07032, over 4431051.68 frames. ], batch size: 218, lr: 3.35e-03, grad_scale: 32.0 2026-09-24 06:31:04,096 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=199653.33333333334, ans=0.125 2026-09-24 06:31:07,904 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=199686.66666666666, ans=0.0 2026-09-24 06:31:08,433 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=199686.66666666666, ans=0.1 2026-09-24 06:31:10,644 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=199720.0, ans=0.125 2026-09-24 06:31:11,562 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=199720.0, ans=0.0 2026-09-24 06:31:14,486 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=199720.0, ans=0.0 2026-09-24 06:31:18,262 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=199753.33333333334, ans=0.0 2026-09-24 06:31:21,316 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=199786.66666666666, ans=0.0 2026-09-24 06:31:25,651 INFO [train.py:1192] (0/2) Epoch 63, batch 550, loss[loss=0.2687, simple_loss=0.3979, pruned_loss=0.06971, over 24283.00 frames. ], tot_loss[loss=0.2589, simple_loss=0.3761, pruned_loss=0.07085, over 4519838.74 frames. ], batch size: 257, lr: 3.35e-03, grad_scale: 32.0 2026-09-24 06:31:26,315 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=7.57 vs. limit=12.0 2026-09-24 06:31:27,959 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.555e+02 3.292e+02 3.652e+02 4.147e+02 6.169e+02, threshold=7.304e+02, percent-clipped=0.0 2026-09-24 06:31:31,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=199853.33333333334, ans=0.125 2026-09-24 06:31:33,551 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=199853.33333333334, ans=0.0 2026-09-24 06:31:36,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=199886.66666666666, ans=0.125 2026-09-24 06:31:45,128 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.29 vs. limit=15.0 2026-09-24 06:31:46,120 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=5.39 vs. limit=6.0 2026-09-24 06:31:50,749 INFO [train.py:1192] (0/2) Epoch 63, batch 600, loss[loss=0.261, simple_loss=0.394, pruned_loss=0.06397, over 24408.00 frames. ], tot_loss[loss=0.2593, simple_loss=0.3767, pruned_loss=0.071, over 4586130.40 frames. ], batch size: 235, lr: 3.34e-03, grad_scale: 32.0 2026-09-24 06:31:52,661 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-60000.pt 2026-09-24 06:32:02,549 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.80 vs. limit=22.5 2026-09-24 06:32:05,012 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=200053.33333333334, ans=0.125 2026-09-24 06:32:08,687 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=200086.66666666666, ans=0.0 2026-09-24 06:32:09,243 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=200086.66666666666, ans=0.0 2026-09-24 06:32:12,575 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=200120.0, ans=0.2 2026-09-24 06:32:15,438 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=200120.0, ans=0.0 2026-09-24 06:32:16,628 INFO [train.py:1192] (0/2) Epoch 63, batch 650, loss[loss=0.2632, simple_loss=0.3806, pruned_loss=0.07295, over 24554.00 frames. ], tot_loss[loss=0.2577, simple_loss=0.3755, pruned_loss=0.06996, over 4651266.20 frames. ], batch size: 162, lr: 3.34e-03, grad_scale: 32.0 2026-09-24 06:32:19,332 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.689e+02 3.268e+02 3.715e+02 4.211e+02 6.195e+02, threshold=7.429e+02, percent-clipped=0.0 2026-09-24 06:32:21,400 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=200186.66666666666, ans=0.025 2026-09-24 06:32:23,158 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=200186.66666666666, ans=0.0 2026-09-24 06:32:25,918 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=200220.0, ans=0.125 2026-09-24 06:32:27,937 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=200220.0, ans=0.2 2026-09-24 06:32:41,838 INFO [train.py:1192] (0/2) Epoch 63, batch 700, loss[loss=0.2411, simple_loss=0.356, pruned_loss=0.06305, over 24561.00 frames. ], tot_loss[loss=0.2582, simple_loss=0.3761, pruned_loss=0.07012, over 4686423.55 frames. ], batch size: 154, lr: 3.34e-03, grad_scale: 32.0 2026-09-24 06:32:48,972 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=200353.33333333334, ans=0.2 2026-09-24 06:32:55,713 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=200386.66666666666, ans=0.125 2026-09-24 06:32:59,832 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=200420.0, ans=0.1 2026-09-24 06:33:05,218 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.36 vs. limit=15.0 2026-09-24 06:33:06,114 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=200453.33333333334, ans=0.2 2026-09-24 06:33:07,093 INFO [train.py:1192] (0/2) Epoch 63, batch 750, loss[loss=0.2496, simple_loss=0.3768, pruned_loss=0.06118, over 24557.00 frames. ], tot_loss[loss=0.2573, simple_loss=0.375, pruned_loss=0.0698, over 4712428.62 frames. ], batch size: 170, lr: 3.34e-03, grad_scale: 32.0 2026-09-24 06:33:07,777 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.64 vs. limit=15.0 2026-09-24 06:33:09,912 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.690e+02 3.492e+02 3.957e+02 4.555e+02 6.218e+02, threshold=7.914e+02, percent-clipped=0.0 2026-09-24 06:33:14,867 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=200520.0, ans=0.1 2026-09-24 06:33:18,001 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=8.30 vs. limit=15.0 2026-09-24 06:33:18,766 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=200553.33333333334, ans=0.2 2026-09-24 06:33:23,149 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=200586.66666666666, ans=0.0 2026-09-24 06:33:25,574 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=200586.66666666666, ans=0.2 2026-09-24 06:33:32,821 INFO [train.py:1192] (0/2) Epoch 63, batch 800, loss[loss=0.2174, simple_loss=0.3353, pruned_loss=0.04971, over 24559.00 frames. ], tot_loss[loss=0.2575, simple_loss=0.3751, pruned_loss=0.06994, over 4741800.94 frames. ], batch size: 137, lr: 3.34e-03, grad_scale: 32.0 2026-09-24 06:33:35,741 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=200653.33333333334, ans=0.0 2026-09-24 06:33:36,276 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=9.11 vs. limit=12.0 2026-09-24 06:33:38,704 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=200686.66666666666, ans=0.125 2026-09-24 06:33:48,029 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.20 vs. limit=22.5 2026-09-24 06:33:57,514 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=200786.66666666666, ans=0.125 2026-09-24 06:33:58,365 INFO [train.py:1192] (0/2) Epoch 63, batch 850, loss[loss=0.303, simple_loss=0.4192, pruned_loss=0.09335, over 24545.00 frames. ], tot_loss[loss=0.257, simple_loss=0.3747, pruned_loss=0.06964, over 4762230.66 frames. ], batch size: 204, lr: 3.34e-03, grad_scale: 32.0 2026-09-24 06:33:59,989 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=200820.0, ans=0.125 2026-09-24 06:34:00,914 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.543e+02 3.497e+02 3.836e+02 4.377e+02 6.099e+02, threshold=7.673e+02, percent-clipped=0.0 2026-09-24 06:34:12,827 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=200886.66666666666, ans=0.1 2026-09-24 06:34:12,840 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=200886.66666666666, ans=0.0 2026-09-24 06:34:13,364 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=200920.0, ans=0.1 2026-09-24 06:34:14,586 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=200920.0, ans=0.09899494936611666 2026-09-24 06:34:19,839 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=200953.33333333334, ans=0.125 2026-09-24 06:34:22,265 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.36 vs. limit=6.0 2026-09-24 06:34:24,351 INFO [train.py:1192] (0/2) Epoch 63, batch 900, loss[loss=0.214, simple_loss=0.3343, pruned_loss=0.04683, over 24568.00 frames. ], tot_loss[loss=0.2574, simple_loss=0.3751, pruned_loss=0.06988, over 4775236.39 frames. ], batch size: 137, lr: 3.34e-03, grad_scale: 32.0 2026-09-24 06:34:25,823 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=200986.66666666666, ans=0.125 2026-09-24 06:34:27,778 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.96 vs. limit=15.0 2026-09-24 06:34:30,795 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=201020.0, ans=0.125 2026-09-24 06:34:43,651 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=201086.66666666666, ans=0.1 2026-09-24 06:34:49,489 INFO [train.py:1192] (0/2) Epoch 63, batch 950, loss[loss=0.3209, simple_loss=0.4019, pruned_loss=0.1199, over 11702.00 frames. ], tot_loss[loss=0.2581, simple_loss=0.3742, pruned_loss=0.07095, over 4714185.87 frames. ], batch size: 333, lr: 3.33e-03, grad_scale: 32.0 2026-09-24 06:34:51,756 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.689e+02 3.427e+02 4.031e+02 4.632e+02 6.244e+02, threshold=8.063e+02, percent-clipped=0.0 2026-09-24 06:34:53,578 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-63.pt 2026-09-24 06:34:58,196 INFO [train.py:1192] (0/2) Epoch 64, batch 0, loss[loss=0.2043, simple_loss=0.3285, pruned_loss=0.04003, over 24580.00 frames. ], tot_loss[loss=0.2043, simple_loss=0.3285, pruned_loss=0.04003, over 24580.00 frames. ], batch size: 137, lr: 3.31e-03, grad_scale: 32.0 2026-09-24 06:34:58,196 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 06:35:09,802 INFO [train.py:1224] (0/2) Epoch 64, validation: loss=0.1703, simple_loss=0.2888, pruned_loss=0.02589, over 2564189.00 frames. 2026-09-24 06:35:09,802 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 06:35:12,094 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=201180.0, ans=0.0 2026-09-24 06:35:12,601 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=201180.0, ans=0.0 2026-09-24 06:35:20,205 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=201246.66666666666, ans=0.5 2026-09-24 06:35:21,321 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=201246.66666666666, ans=0.125 2026-09-24 06:35:22,412 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=7.156e-02 2026-09-24 06:35:32,290 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=201313.33333333334, ans=0.0 2026-09-24 06:35:34,209 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=201313.33333333334, ans=0.125 2026-09-24 06:35:35,566 INFO [train.py:1192] (0/2) Epoch 64, batch 50, loss[loss=0.2228, simple_loss=0.3336, pruned_loss=0.05603, over 24242.00 frames. ], tot_loss[loss=0.2632, simple_loss=0.3803, pruned_loss=0.07302, over 1076703.47 frames. ], batch size: 125, lr: 3.31e-03, grad_scale: 32.0 2026-09-24 06:35:38,922 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=201346.66666666666, ans=0.2 2026-09-24 06:35:56,414 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=201480.0, ans=0.0 2026-09-24 06:35:59,807 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.810e+02 3.494e+02 3.954e+02 4.353e+02 6.086e+02, threshold=7.909e+02, percent-clipped=0.0 2026-09-24 06:36:01,129 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=201513.33333333334, ans=0.125 2026-09-24 06:36:01,518 INFO [train.py:1192] (0/2) Epoch 64, batch 100, loss[loss=0.269, simple_loss=0.375, pruned_loss=0.08149, over 24619.00 frames. ], tot_loss[loss=0.2673, simple_loss=0.385, pruned_loss=0.07476, over 1904687.35 frames. ], batch size: 154, lr: 3.31e-03, grad_scale: 32.0 2026-09-24 06:36:08,553 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=10.55 vs. limit=22.5 2026-09-24 06:36:21,161 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.10 vs. limit=12.0 2026-09-24 06:36:26,888 INFO [train.py:1192] (0/2) Epoch 64, batch 150, loss[loss=0.218, simple_loss=0.3288, pruned_loss=0.05358, over 24302.00 frames. ], tot_loss[loss=0.2617, simple_loss=0.3797, pruned_loss=0.07184, over 2558537.46 frames. ], batch size: 125, lr: 3.30e-03, grad_scale: 32.0 2026-09-24 06:36:29,753 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=201680.0, ans=0.1 2026-09-24 06:36:50,961 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.955e+02 3.309e+02 3.704e+02 4.199e+02 6.164e+02, threshold=7.408e+02, percent-clipped=0.0 2026-09-24 06:36:52,519 INFO [train.py:1192] (0/2) Epoch 64, batch 200, loss[loss=0.2994, simple_loss=0.4053, pruned_loss=0.09681, over 21046.00 frames. ], tot_loss[loss=0.2605, simple_loss=0.3785, pruned_loss=0.07123, over 3055144.83 frames. ], batch size: 333, lr: 3.30e-03, grad_scale: 32.0 2026-09-24 06:36:57,059 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=201880.0, ans=0.125 2026-09-24 06:37:17,626 INFO [train.py:1192] (0/2) Epoch 64, batch 250, loss[loss=0.2868, simple_loss=0.4139, pruned_loss=0.07988, over 24303.00 frames. ], tot_loss[loss=0.2594, simple_loss=0.3773, pruned_loss=0.07076, over 3445298.42 frames. ], batch size: 234, lr: 3.30e-03, grad_scale: 32.0 2026-09-24 06:37:17,910 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.18 vs. limit=10.0 2026-09-24 06:37:20,163 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=202013.33333333334, ans=0.1 2026-09-24 06:37:41,608 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.474e+02 3.346e+02 3.765e+02 4.247e+02 6.018e+02, threshold=7.530e+02, percent-clipped=0.0 2026-09-24 06:37:43,183 INFO [train.py:1192] (0/2) Epoch 64, batch 300, loss[loss=0.27, simple_loss=0.4, pruned_loss=0.06995, over 24526.00 frames. ], tot_loss[loss=0.2581, simple_loss=0.376, pruned_loss=0.07014, over 3750161.35 frames. ], batch size: 204, lr: 3.30e-03, grad_scale: 32.0 2026-09-24 06:37:58,705 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.58 vs. limit=22.5 2026-09-24 06:37:58,968 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=202280.0, ans=0.025 2026-09-24 06:38:08,215 INFO [train.py:1192] (0/2) Epoch 64, batch 350, loss[loss=0.217, simple_loss=0.3277, pruned_loss=0.05318, over 24585.00 frames. ], tot_loss[loss=0.2579, simple_loss=0.3763, pruned_loss=0.0698, over 3993333.76 frames. ], batch size: 137, lr: 3.30e-03, grad_scale: 32.0 2026-09-24 06:38:17,640 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=202380.0, ans=0.2 2026-09-24 06:38:21,248 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.16 vs. limit=22.5 2026-09-24 06:38:24,091 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=202446.66666666666, ans=0.0 2026-09-24 06:38:31,788 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.529e+02 3.427e+02 3.829e+02 4.368e+02 6.561e+02, threshold=7.657e+02, percent-clipped=0.0 2026-09-24 06:38:32,447 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=202480.0, ans=0.025 2026-09-24 06:38:33,357 INFO [train.py:1192] (0/2) Epoch 64, batch 400, loss[loss=0.2495, simple_loss=0.3752, pruned_loss=0.06191, over 24562.00 frames. ], tot_loss[loss=0.257, simple_loss=0.3752, pruned_loss=0.06937, over 4178595.93 frames. ], batch size: 170, lr: 3.30e-03, grad_scale: 32.0 2026-09-24 06:38:33,440 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=202513.33333333334, ans=0.025 2026-09-24 06:38:36,390 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.11 vs. limit=10.0 2026-09-24 06:38:46,122 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=7.13 vs. limit=15.0 2026-09-24 06:38:47,876 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=202580.0, ans=0.125 2026-09-24 06:38:55,579 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=202646.66666666666, ans=0.2 2026-09-24 06:38:57,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=202646.66666666666, ans=0.025 2026-09-24 06:38:57,440 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=202646.66666666666, ans=0.125 2026-09-24 06:38:58,346 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=202646.66666666666, ans=0.125 2026-09-24 06:38:59,184 INFO [train.py:1192] (0/2) Epoch 64, batch 450, loss[loss=0.2669, simple_loss=0.3875, pruned_loss=0.07319, over 24618.00 frames. ], tot_loss[loss=0.258, simple_loss=0.3758, pruned_loss=0.07011, over 4314123.87 frames. ], batch size: 175, lr: 3.30e-03, grad_scale: 32.0 2026-09-24 06:39:07,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=202713.33333333334, ans=0.125 2026-09-24 06:39:07,586 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=202713.33333333334, ans=0.2 2026-09-24 06:39:09,441 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=202746.66666666666, ans=0.125 2026-09-24 06:39:15,945 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.44 vs. limit=15.0 2026-09-24 06:39:23,158 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.645e+02 3.315e+02 3.849e+02 4.427e+02 7.168e+02, threshold=7.697e+02, percent-clipped=0.0 2026-09-24 06:39:24,690 INFO [train.py:1192] (0/2) Epoch 64, batch 500, loss[loss=0.2682, simple_loss=0.3979, pruned_loss=0.06924, over 24507.00 frames. ], tot_loss[loss=0.2571, simple_loss=0.3745, pruned_loss=0.06989, over 4430275.07 frames. ], batch size: 218, lr: 3.29e-03, grad_scale: 32.0 2026-09-24 06:39:44,067 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=202946.66666666666, ans=0.95 2026-09-24 06:39:45,809 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=202980.0, ans=0.025 2026-09-24 06:39:47,393 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=202980.0, ans=0.1 2026-09-24 06:39:50,387 INFO [train.py:1192] (0/2) Epoch 64, batch 550, loss[loss=0.2647, simple_loss=0.3937, pruned_loss=0.06783, over 24282.00 frames. ], tot_loss[loss=0.2577, simple_loss=0.3753, pruned_loss=0.07, over 4519238.95 frames. ], batch size: 257, lr: 3.29e-03, grad_scale: 32.0 2026-09-24 06:39:52,596 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=203013.33333333334, ans=0.0 2026-09-24 06:39:59,673 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=203046.66666666666, ans=0.125 2026-09-24 06:40:07,204 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=203113.33333333334, ans=0.125 2026-09-24 06:40:12,359 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=203146.66666666666, ans=0.1 2026-09-24 06:40:12,362 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=203146.66666666666, ans=0.0 2026-09-24 06:40:14,232 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.612e+02 3.344e+02 3.554e+02 4.056e+02 6.649e+02, threshold=7.108e+02, percent-clipped=0.0 2026-09-24 06:40:16,065 INFO [train.py:1192] (0/2) Epoch 64, batch 600, loss[loss=0.2723, simple_loss=0.394, pruned_loss=0.07536, over 24369.00 frames. ], tot_loss[loss=0.2584, simple_loss=0.3761, pruned_loss=0.0703, over 4586819.40 frames. ], batch size: 234, lr: 3.29e-03, grad_scale: 32.0 2026-09-24 06:40:27,613 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.43 vs. limit=8.0 2026-09-24 06:40:33,305 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=203280.0, ans=0.125 2026-09-24 06:40:41,346 INFO [train.py:1192] (0/2) Epoch 64, batch 650, loss[loss=0.2514, simple_loss=0.3698, pruned_loss=0.06648, over 24560.00 frames. ], tot_loss[loss=0.2579, simple_loss=0.3755, pruned_loss=0.07014, over 4651660.29 frames. ], batch size: 162, lr: 3.29e-03, grad_scale: 32.0 2026-09-24 06:40:48,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=203380.0, ans=0.125 2026-09-24 06:40:49,560 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=203380.0, ans=0.125 2026-09-24 06:40:49,678 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.48 vs. limit=22.5 2026-09-24 06:40:51,652 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=203413.33333333334, ans=0.2 2026-09-24 06:41:05,045 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.666e+02 3.248e+02 3.709e+02 4.337e+02 6.887e+02, threshold=7.418e+02, percent-clipped=0.0 2026-09-24 06:41:06,436 INFO [train.py:1192] (0/2) Epoch 64, batch 700, loss[loss=0.2413, simple_loss=0.3563, pruned_loss=0.06314, over 24562.00 frames. ], tot_loss[loss=0.2577, simple_loss=0.3758, pruned_loss=0.0698, over 4684726.96 frames. ], batch size: 154, lr: 3.29e-03, grad_scale: 32.0 2026-09-24 06:41:29,530 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=203646.66666666666, ans=0.125 2026-09-24 06:41:30,062 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=9.94 vs. limit=22.5 2026-09-24 06:41:31,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=203646.66666666666, ans=0.125 2026-09-24 06:41:32,390 INFO [train.py:1192] (0/2) Epoch 64, batch 750, loss[loss=0.2747, simple_loss=0.3926, pruned_loss=0.07842, over 24550.00 frames. ], tot_loss[loss=0.257, simple_loss=0.3745, pruned_loss=0.06975, over 4712245.62 frames. ], batch size: 170, lr: 3.29e-03, grad_scale: 32.0 2026-09-24 06:41:39,338 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=203713.33333333334, ans=0.125 2026-09-24 06:41:46,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=203746.66666666666, ans=0.025 2026-09-24 06:41:51,167 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.27 vs. limit=15.0 2026-09-24 06:41:54,008 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=203813.33333333334, ans=0.025 2026-09-24 06:41:54,447 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=203813.33333333334, ans=0.125 2026-09-24 06:41:56,202 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.674e+02 3.434e+02 4.021e+02 4.647e+02 7.395e+02, threshold=8.043e+02, percent-clipped=0.0 2026-09-24 06:41:58,035 INFO [train.py:1192] (0/2) Epoch 64, batch 800, loss[loss=0.2285, simple_loss=0.3471, pruned_loss=0.05496, over 24546.00 frames. ], tot_loss[loss=0.2567, simple_loss=0.3743, pruned_loss=0.06957, over 4741348.82 frames. ], batch size: 137, lr: 3.29e-03, grad_scale: 32.0 2026-09-24 06:42:02,705 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=3.91 vs. limit=5.0 2026-09-24 06:42:13,726 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=203946.66666666666, ans=0.125 2026-09-24 06:42:23,706 INFO [train.py:1192] (0/2) Epoch 64, batch 850, loss[loss=0.2855, simple_loss=0.4117, pruned_loss=0.07966, over 24551.00 frames. ], tot_loss[loss=0.2564, simple_loss=0.3741, pruned_loss=0.06942, over 4764293.88 frames. ], batch size: 204, lr: 3.29e-03, grad_scale: 32.0 2026-09-24 06:42:42,156 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=204113.33333333334, ans=0.1 2026-09-24 06:42:47,169 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.581e+02 3.618e+02 4.019e+02 4.678e+02 6.392e+02, threshold=8.038e+02, percent-clipped=0.0 2026-09-24 06:42:48,276 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.42 vs. limit=15.0 2026-09-24 06:42:48,523 INFO [train.py:1192] (0/2) Epoch 64, batch 900, loss[loss=0.2213, simple_loss=0.3348, pruned_loss=0.05387, over 24560.00 frames. ], tot_loss[loss=0.2567, simple_loss=0.3744, pruned_loss=0.06952, over 4776977.90 frames. ], batch size: 137, lr: 3.28e-03, grad_scale: 32.0 2026-09-24 06:42:49,062 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=204180.0, ans=0.1 2026-09-24 06:42:51,466 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=204180.0, ans=0.125 2026-09-24 06:42:53,283 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=204213.33333333334, ans=0.0 2026-09-24 06:42:53,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=204213.33333333334, ans=0.125 2026-09-24 06:42:53,933 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.38 vs. limit=22.5 2026-09-24 06:42:56,987 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.49 vs. limit=15.0 2026-09-24 06:43:00,460 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=18.24 vs. limit=22.5 2026-09-24 06:43:00,728 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=204246.66666666666, ans=0.0 2026-09-24 06:43:02,196 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=204246.66666666666, ans=0.125 2026-09-24 06:43:02,882 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.97 vs. limit=6.0 2026-09-24 06:43:10,351 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=204313.33333333334, ans=0.2 2026-09-24 06:43:13,902 INFO [train.py:1192] (0/2) Epoch 64, batch 950, loss[loss=0.3691, simple_loss=0.4295, pruned_loss=0.1544, over 11065.00 frames. ], tot_loss[loss=0.2576, simple_loss=0.3736, pruned_loss=0.07083, over 4713650.71 frames. ], batch size: 334, lr: 3.28e-03, grad_scale: 32.0 2026-09-24 06:43:18,261 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-64.pt 2026-09-24 06:43:23,835 INFO [train.py:1192] (0/2) Epoch 65, batch 0, loss[loss=0.2027, simple_loss=0.3254, pruned_loss=0.03999, over 24569.00 frames. ], tot_loss[loss=0.2027, simple_loss=0.3254, pruned_loss=0.03999, over 24569.00 frames. ], batch size: 137, lr: 3.26e-03, grad_scale: 32.0 2026-09-24 06:43:23,835 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 06:43:35,372 INFO [train.py:1224] (0/2) Epoch 65, validation: loss=0.1694, simple_loss=0.2879, pruned_loss=0.02545, over 2564189.00 frames. 2026-09-24 06:43:35,372 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 06:43:35,464 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=204373.33333333334, ans=0.125 2026-09-24 06:43:36,776 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=204373.33333333334, ans=0.125 2026-09-24 06:43:52,993 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=204473.33333333334, ans=0.2 2026-09-24 06:43:55,233 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.690e+02 3.480e+02 4.057e+02 4.389e+02 1.010e+03, threshold=8.114e+02, percent-clipped=1.0 2026-09-24 06:43:55,769 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=204506.66666666666, ans=0.1 2026-09-24 06:43:57,009 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:44:00,886 INFO [train.py:1192] (0/2) Epoch 65, batch 50, loss[loss=0.2224, simple_loss=0.3273, pruned_loss=0.05869, over 24247.00 frames. ], tot_loss[loss=0.2629, simple_loss=0.3792, pruned_loss=0.07336, over 1076308.80 frames. ], batch size: 125, lr: 3.26e-03, grad_scale: 32.0 2026-09-24 06:44:01,001 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass_mid.scale_min, batch_count=204540.0, ans=0.2 2026-09-24 06:44:02,960 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.29 vs. limit=15.0 2026-09-24 06:44:12,248 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.34 vs. limit=15.0 2026-09-24 06:44:13,098 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=204606.66666666666, ans=0.0 2026-09-24 06:44:24,540 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=204673.33333333334, ans=0.125 2026-09-24 06:44:26,486 INFO [train.py:1192] (0/2) Epoch 65, batch 100, loss[loss=0.2515, simple_loss=0.3641, pruned_loss=0.06946, over 24626.00 frames. ], tot_loss[loss=0.2669, simple_loss=0.384, pruned_loss=0.0749, over 1905157.65 frames. ], batch size: 154, lr: 3.25e-03, grad_scale: 32.0 2026-09-24 06:44:32,283 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=4.47 vs. limit=15.0 2026-09-24 06:44:35,111 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=204740.0, ans=0.2 2026-09-24 06:44:44,753 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=7.47 vs. limit=15.0 2026-09-24 06:44:45,954 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.589e+02 3.249e+02 3.693e+02 4.114e+02 5.857e+02, threshold=7.386e+02, percent-clipped=0.0 2026-09-24 06:44:52,172 INFO [train.py:1192] (0/2) Epoch 65, batch 150, loss[loss=0.2063, simple_loss=0.3178, pruned_loss=0.04742, over 24220.00 frames. ], tot_loss[loss=0.2614, simple_loss=0.379, pruned_loss=0.07191, over 2558759.59 frames. ], batch size: 125, lr: 3.25e-03, grad_scale: 32.0 2026-09-24 06:45:05,693 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=6.18 vs. limit=10.0 2026-09-24 06:45:10,432 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.25 vs. limit=22.5 2026-09-24 06:45:17,146 INFO [train.py:1192] (0/2) Epoch 65, batch 200, loss[loss=0.329, simple_loss=0.4255, pruned_loss=0.1162, over 21188.00 frames. ], tot_loss[loss=0.2594, simple_loss=0.3773, pruned_loss=0.07077, over 3055601.36 frames. ], batch size: 333, lr: 3.25e-03, grad_scale: 32.0 2026-09-24 06:45:17,703 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=205040.0, ans=0.0 2026-09-24 06:45:17,705 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=205040.0, ans=0.125 2026-09-24 06:45:18,248 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.69 vs. limit=10.0 2026-09-24 06:45:18,697 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=205040.0, ans=0.125 2026-09-24 06:45:19,634 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=205040.0, ans=0.1 2026-09-24 06:45:20,199 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=205040.0, ans=0.125 2026-09-24 06:45:23,309 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.13 vs. limit=10.0 2026-09-24 06:45:24,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=205073.33333333334, ans=0.125 2026-09-24 06:45:34,942 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=205140.0, ans=0.1 2026-09-24 06:45:36,742 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.542e+02 3.387e+02 3.702e+02 4.365e+02 5.971e+02, threshold=7.404e+02, percent-clipped=0.0 2026-09-24 06:45:37,761 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=205173.33333333334, ans=0.2 2026-09-24 06:45:42,603 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.92 vs. limit=15.0 2026-09-24 06:45:42,846 INFO [train.py:1192] (0/2) Epoch 65, batch 250, loss[loss=0.2853, simple_loss=0.4113, pruned_loss=0.07964, over 24291.00 frames. ], tot_loss[loss=0.2581, simple_loss=0.3763, pruned_loss=0.06998, over 3445596.11 frames. ], batch size: 234, lr: 3.25e-03, grad_scale: 32.0 2026-09-24 06:46:00,535 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=205306.66666666666, ans=0.1 2026-09-24 06:46:07,907 INFO [train.py:1192] (0/2) Epoch 65, batch 300, loss[loss=0.2657, simple_loss=0.3958, pruned_loss=0.06782, over 24545.00 frames. ], tot_loss[loss=0.2576, simple_loss=0.3753, pruned_loss=0.06991, over 3751721.11 frames. ], batch size: 204, lr: 3.25e-03, grad_scale: 32.0 2026-09-24 06:46:16,677 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=205406.66666666666, ans=0.0 2026-09-24 06:46:18,934 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=205440.0, ans=0.0 2026-09-24 06:46:20,302 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=205440.0, ans=0.125 2026-09-24 06:46:21,701 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=205440.0, ans=0.1 2026-09-24 06:46:23,669 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.44 vs. limit=15.0 2026-09-24 06:46:27,621 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.897e+02 3.396e+02 3.848e+02 4.457e+02 6.052e+02, threshold=7.696e+02, percent-clipped=0.0 2026-09-24 06:46:33,595 INFO [train.py:1192] (0/2) Epoch 65, batch 350, loss[loss=0.2219, simple_loss=0.335, pruned_loss=0.05442, over 24569.00 frames. ], tot_loss[loss=0.2587, simple_loss=0.3766, pruned_loss=0.07039, over 3994572.35 frames. ], batch size: 137, lr: 3.25e-03, grad_scale: 64.0 2026-09-24 06:46:45,738 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=205606.66666666666, ans=0.2 2026-09-24 06:46:52,303 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=205640.0, ans=0.09899494936611666 2026-09-24 06:46:52,826 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.35 vs. limit=15.0 2026-09-24 06:46:56,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=205673.33333333334, ans=0.125 2026-09-24 06:46:58,630 INFO [train.py:1192] (0/2) Epoch 65, batch 400, loss[loss=0.2764, simple_loss=0.3886, pruned_loss=0.08215, over 24558.00 frames. ], tot_loss[loss=0.258, simple_loss=0.3758, pruned_loss=0.07009, over 4179148.14 frames. ], batch size: 170, lr: 3.25e-03, grad_scale: 64.0 2026-09-24 06:47:18,815 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.618e+02 3.420e+02 3.842e+02 4.354e+02 6.399e+02, threshold=7.683e+02, percent-clipped=0.0 2026-09-24 06:47:19,845 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.skip_rate, batch_count=205840.0, ans=0.035 2026-09-24 06:47:24,758 INFO [train.py:1192] (0/2) Epoch 65, batch 450, loss[loss=0.3071, simple_loss=0.4161, pruned_loss=0.09901, over 24639.00 frames. ], tot_loss[loss=0.2586, simple_loss=0.3764, pruned_loss=0.0704, over 4311275.20 frames. ], batch size: 175, lr: 3.25e-03, grad_scale: 64.0 2026-09-24 06:47:36,444 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=205940.0, ans=0.125 2026-09-24 06:47:38,232 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.68 vs. limit=10.0 2026-09-24 06:47:39,615 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.88 vs. limit=12.0 2026-09-24 06:47:50,111 INFO [train.py:1192] (0/2) Epoch 65, batch 500, loss[loss=0.3042, simple_loss=0.4248, pruned_loss=0.09174, over 24519.00 frames. ], tot_loss[loss=0.2575, simple_loss=0.3749, pruned_loss=0.07003, over 4429204.69 frames. ], batch size: 218, lr: 3.24e-03, grad_scale: 64.0 2026-09-24 06:47:57,513 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=206073.33333333334, ans=0.1 2026-09-24 06:48:05,883 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.31 vs. limit=10.0 2026-09-24 06:48:09,453 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.664e+02 3.408e+02 3.757e+02 4.234e+02 7.032e+02, threshold=7.514e+02, percent-clipped=0.0 2026-09-24 06:48:15,447 INFO [train.py:1192] (0/2) Epoch 65, batch 550, loss[loss=0.2706, simple_loss=0.3987, pruned_loss=0.07126, over 24199.00 frames. ], tot_loss[loss=0.2577, simple_loss=0.3754, pruned_loss=0.07004, over 4518613.77 frames. ], batch size: 257, lr: 3.24e-03, grad_scale: 64.0 2026-09-24 06:48:41,086 INFO [train.py:1192] (0/2) Epoch 65, batch 600, loss[loss=0.2844, simple_loss=0.4107, pruned_loss=0.07907, over 24323.00 frames. ], tot_loss[loss=0.2574, simple_loss=0.3753, pruned_loss=0.06973, over 4586201.96 frames. ], batch size: 234, lr: 3.24e-03, grad_scale: 32.0 2026-09-24 06:48:44,100 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=206373.33333333334, ans=0.2 2026-09-24 06:48:47,109 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=206406.66666666666, ans=0.125 2026-09-24 06:48:52,824 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=206440.0, ans=0.125 2026-09-24 06:48:53,308 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=206440.0, ans=0.0 2026-09-24 06:48:54,099 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.48 vs. limit=22.5 2026-09-24 06:48:56,261 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=206473.33333333334, ans=0.125 2026-09-24 06:49:01,405 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.814e+02 3.398e+02 3.684e+02 4.322e+02 6.606e+02, threshold=7.368e+02, percent-clipped=0.0 2026-09-24 06:49:05,479 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.91 vs. limit=15.0 2026-09-24 06:49:06,101 INFO [train.py:1192] (0/2) Epoch 65, batch 650, loss[loss=0.2638, simple_loss=0.3795, pruned_loss=0.0741, over 24558.00 frames. ], tot_loss[loss=0.2563, simple_loss=0.3744, pruned_loss=0.06908, over 4651665.09 frames. ], batch size: 162, lr: 3.24e-03, grad_scale: 32.0 2026-09-24 06:49:11,441 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.54 vs. limit=15.0 2026-09-24 06:49:31,601 INFO [train.py:1192] (0/2) Epoch 65, batch 700, loss[loss=0.2331, simple_loss=0.3511, pruned_loss=0.05755, over 24564.00 frames. ], tot_loss[loss=0.2569, simple_loss=0.3752, pruned_loss=0.06932, over 4685422.55 frames. ], batch size: 154, lr: 3.24e-03, grad_scale: 32.0 2026-09-24 06:49:32,654 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=206706.66666666666, ans=0.1 2026-09-24 06:49:34,682 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.30 vs. limit=15.0 2026-09-24 06:49:38,828 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=206740.0, ans=0.125 2026-09-24 06:49:39,597 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=12.56 vs. limit=22.5 2026-09-24 06:49:44,201 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:49:46,031 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=206773.33333333334, ans=0.0 2026-09-24 06:49:47,956 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=206806.66666666666, ans=0.1 2026-09-24 06:49:52,782 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.587e+02 3.473e+02 3.863e+02 4.424e+02 6.086e+02, threshold=7.726e+02, percent-clipped=0.0 2026-09-24 06:49:57,494 INFO [train.py:1192] (0/2) Epoch 65, batch 750, loss[loss=0.2431, simple_loss=0.3703, pruned_loss=0.05799, over 24568.00 frames. ], tot_loss[loss=0.2561, simple_loss=0.3741, pruned_loss=0.06899, over 4712112.35 frames. ], batch size: 170, lr: 3.24e-03, grad_scale: 32.0 2026-09-24 06:49:58,490 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=206873.33333333334, ans=0.125 2026-09-24 06:50:22,735 INFO [train.py:1192] (0/2) Epoch 65, batch 800, loss[loss=0.2179, simple_loss=0.3335, pruned_loss=0.05114, over 24549.00 frames. ], tot_loss[loss=0.2557, simple_loss=0.3738, pruned_loss=0.06878, over 4737149.71 frames. ], batch size: 137, lr: 3.24e-03, grad_scale: 32.0 2026-09-24 06:50:24,172 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=207040.0, ans=0.125 2026-09-24 06:50:42,768 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.42 vs. limit=6.0 2026-09-24 06:50:43,051 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.588e+02 3.517e+02 3.795e+02 4.231e+02 7.044e+02, threshold=7.591e+02, percent-clipped=0.0 2026-09-24 06:50:46,109 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=8.86 vs. limit=22.5 2026-09-24 06:50:47,911 INFO [train.py:1192] (0/2) Epoch 65, batch 850, loss[loss=0.2793, simple_loss=0.4065, pruned_loss=0.07603, over 24599.00 frames. ], tot_loss[loss=0.255, simple_loss=0.3732, pruned_loss=0.06839, over 4760942.51 frames. ], batch size: 198, lr: 3.24e-03, grad_scale: 32.0 2026-09-24 06:51:04,531 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=207306.66666666666, ans=0.0 2026-09-24 06:51:05,668 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=207306.66666666666, ans=0.0 2026-09-24 06:51:07,965 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=207340.0, ans=0.0 2026-09-24 06:51:13,800 INFO [train.py:1192] (0/2) Epoch 65, batch 900, loss[loss=0.2272, simple_loss=0.3427, pruned_loss=0.05588, over 24574.00 frames. ], tot_loss[loss=0.2557, simple_loss=0.374, pruned_loss=0.06873, over 4773967.49 frames. ], batch size: 137, lr: 3.23e-03, grad_scale: 32.0 2026-09-24 06:51:14,503 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.71 vs. limit=15.0 2026-09-24 06:51:16,170 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=207373.33333333334, ans=0.1 2026-09-24 06:51:22,982 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=207440.0, ans=0.1 2026-09-24 06:51:24,957 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.46 vs. limit=6.0 2026-09-24 06:51:32,468 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=207473.33333333334, ans=0.125 2026-09-24 06:51:33,391 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.511e+02 3.421e+02 4.215e+02 4.846e+02 8.093e+02, threshold=8.430e+02, percent-clipped=1.0 2026-09-24 06:51:38,709 INFO [train.py:1192] (0/2) Epoch 65, batch 950, loss[loss=0.3287, simple_loss=0.4059, pruned_loss=0.1257, over 11712.00 frames. ], tot_loss[loss=0.2561, simple_loss=0.3729, pruned_loss=0.06965, over 4712141.72 frames. ], batch size: 334, lr: 3.23e-03, grad_scale: 32.0 2026-09-24 06:51:41,245 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=207540.0, ans=0.1 2026-09-24 06:51:41,759 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=9.65 vs. limit=12.0 2026-09-24 06:51:42,920 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-65.pt 2026-09-24 06:51:50,075 INFO [train.py:1192] (0/2) Epoch 66, batch 0, loss[loss=0.2115, simple_loss=0.3313, pruned_loss=0.04584, over 24551.00 frames. ], tot_loss[loss=0.2115, simple_loss=0.3313, pruned_loss=0.04584, over 24551.00 frames. ], batch size: 137, lr: 3.21e-03, grad_scale: 32.0 2026-09-24 06:51:50,075 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 06:51:55,969 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.6283, 2.4956, 2.6265, 2.1878], device='cuda:0') 2026-09-24 06:52:01,616 INFO [train.py:1224] (0/2) Epoch 66, validation: loss=0.1703, simple_loss=0.2885, pruned_loss=0.02602, over 2564189.00 frames. 2026-09-24 06:52:01,621 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 06:52:16,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=207633.33333333334, ans=0.0 2026-09-24 06:52:17,209 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.80 vs. limit=12.0 2026-09-24 06:52:23,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=207700.0, ans=0.1 2026-09-24 06:52:27,748 INFO [train.py:1192] (0/2) Epoch 66, batch 50, loss[loss=0.2661, simple_loss=0.3603, pruned_loss=0.08591, over 24278.00 frames. ], tot_loss[loss=0.2658, simple_loss=0.3822, pruned_loss=0.07467, over 1076807.86 frames. ], batch size: 125, lr: 3.21e-03, grad_scale: 32.0 2026-09-24 06:52:41,877 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=207800.0, ans=0.0 2026-09-24 06:52:43,634 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.716e+02 3.550e+02 3.980e+02 4.670e+02 6.089e+02, threshold=7.959e+02, percent-clipped=0.0 2026-09-24 06:52:51,059 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=207866.66666666666, ans=0.0 2026-09-24 06:52:53,540 INFO [train.py:1192] (0/2) Epoch 66, batch 100, loss[loss=0.2489, simple_loss=0.3653, pruned_loss=0.06626, over 24615.00 frames. ], tot_loss[loss=0.2669, simple_loss=0.3844, pruned_loss=0.07464, over 1905049.34 frames. ], batch size: 154, lr: 3.20e-03, grad_scale: 32.0 2026-09-24 06:53:11,200 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=208000.0, ans=0.1 2026-09-24 06:53:14,119 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=208033.33333333334, ans=0.125 2026-09-24 06:53:17,320 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=208033.33333333334, ans=0.125 2026-09-24 06:53:19,162 INFO [train.py:1192] (0/2) Epoch 66, batch 150, loss[loss=0.1935, simple_loss=0.3086, pruned_loss=0.03916, over 24282.00 frames. ], tot_loss[loss=0.2603, simple_loss=0.3785, pruned_loss=0.07107, over 2557006.35 frames. ], batch size: 125, lr: 3.20e-03, grad_scale: 32.0 2026-09-24 06:53:20,721 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:53:24,207 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.90 vs. limit=22.5 2026-09-24 06:53:28,855 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.min_abs, batch_count=208133.33333333334, ans=0.5 2026-09-24 06:53:35,705 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.697e+02 3.405e+02 3.768e+02 4.274e+02 6.558e+02, threshold=7.536e+02, percent-clipped=0.0 2026-09-24 06:53:37,622 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=208166.66666666666, ans=0.125 2026-09-24 06:53:44,824 INFO [train.py:1192] (0/2) Epoch 66, batch 200, loss[loss=0.2993, simple_loss=0.4043, pruned_loss=0.09713, over 21045.00 frames. ], tot_loss[loss=0.2585, simple_loss=0.3767, pruned_loss=0.07015, over 3055799.16 frames. ], batch size: 333, lr: 3.20e-03, grad_scale: 32.0 2026-09-24 06:53:49,833 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.prob, batch_count=208266.66666666666, ans=0.125 2026-09-24 06:53:52,356 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.34 vs. limit=6.0 2026-09-24 06:53:54,903 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=208300.0, ans=0.125 2026-09-24 06:53:57,896 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=208300.0, ans=0.1 2026-09-24 06:54:04,207 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.49 vs. limit=15.0 2026-09-24 06:54:05,519 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=208366.66666666666, ans=0.125 2026-09-24 06:54:10,415 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=208400.0, ans=0.125 2026-09-24 06:54:10,843 INFO [train.py:1192] (0/2) Epoch 66, batch 250, loss[loss=0.2723, simple_loss=0.4019, pruned_loss=0.07137, over 24313.00 frames. ], tot_loss[loss=0.2573, simple_loss=0.3755, pruned_loss=0.06953, over 3446137.23 frames. ], batch size: 234, lr: 3.20e-03, grad_scale: 32.0 2026-09-24 06:54:17,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=208433.33333333334, ans=0.125 2026-09-24 06:54:22,979 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=208466.66666666666, ans=0.125 2026-09-24 06:54:26,502 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.729e+02 3.440e+02 3.866e+02 4.640e+02 6.209e+02, threshold=7.733e+02, percent-clipped=0.0 2026-09-24 06:54:35,695 INFO [train.py:1192] (0/2) Epoch 66, batch 300, loss[loss=0.2532, simple_loss=0.3865, pruned_loss=0.05991, over 24548.00 frames. ], tot_loss[loss=0.2557, simple_loss=0.3739, pruned_loss=0.06871, over 3751040.49 frames. ], batch size: 204, lr: 3.20e-03, grad_scale: 32.0 2026-09-24 06:54:54,664 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.93 vs. limit=6.0 2026-09-24 06:54:57,713 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.25 vs. limit=15.0 2026-09-24 06:54:59,198 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.40 vs. limit=15.0 2026-09-24 06:55:01,036 INFO [train.py:1192] (0/2) Epoch 66, batch 350, loss[loss=0.2299, simple_loss=0.3434, pruned_loss=0.05824, over 24564.00 frames. ], tot_loss[loss=0.2571, simple_loss=0.3753, pruned_loss=0.06938, over 3993752.08 frames. ], batch size: 137, lr: 3.20e-03, grad_scale: 32.0 2026-09-24 06:55:03,907 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=208733.33333333334, ans=0.0 2026-09-24 06:55:09,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=208766.66666666666, ans=0.125 2026-09-24 06:55:12,619 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=208800.0, ans=0.07 2026-09-24 06:55:16,964 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.757e+02 3.335e+02 3.673e+02 4.164e+02 6.012e+02, threshold=7.346e+02, percent-clipped=0.0 2026-09-24 06:55:19,433 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=208833.33333333334, ans=0.2 2026-09-24 06:55:21,195 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=208866.66666666666, ans=0.025 2026-09-24 06:55:22,455 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.18 vs. limit=15.0 2026-09-24 06:55:26,228 INFO [train.py:1192] (0/2) Epoch 66, batch 400, loss[loss=0.2423, simple_loss=0.3706, pruned_loss=0.05697, over 24576.00 frames. ], tot_loss[loss=0.2556, simple_loss=0.3743, pruned_loss=0.06846, over 4179541.08 frames. ], batch size: 170, lr: 3.20e-03, grad_scale: 32.0 2026-09-24 06:55:38,015 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.05 vs. limit=12.0 2026-09-24 06:55:46,031 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.60 vs. limit=22.5 2026-09-24 06:55:46,728 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=209033.33333333334, ans=0.1 2026-09-24 06:55:51,368 INFO [train.py:1192] (0/2) Epoch 66, batch 450, loss[loss=0.25, simple_loss=0.3749, pruned_loss=0.06254, over 24636.00 frames. ], tot_loss[loss=0.2565, simple_loss=0.3748, pruned_loss=0.06908, over 4315199.45 frames. ], batch size: 175, lr: 3.20e-03, grad_scale: 32.0 2026-09-24 06:55:51,499 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:56:03,785 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.11 vs. limit=15.0 2026-09-24 06:56:04,659 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.32 vs. limit=15.0 2026-09-24 06:56:07,235 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.474e+02 3.290e+02 3.686e+02 4.229e+02 6.187e+02, threshold=7.373e+02, percent-clipped=0.0 2026-09-24 06:56:10,121 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=12.04 vs. limit=22.5 2026-09-24 06:56:16,537 INFO [train.py:1192] (0/2) Epoch 66, batch 500, loss[loss=0.285, simple_loss=0.4038, pruned_loss=0.08307, over 24510.00 frames. ], tot_loss[loss=0.2551, simple_loss=0.3734, pruned_loss=0.06842, over 4432602.27 frames. ], batch size: 218, lr: 3.19e-03, grad_scale: 32.0 2026-09-24 06:56:22,708 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=209266.66666666666, ans=0.0 2026-09-24 06:56:30,468 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=209300.0, ans=0.125 2026-09-24 06:56:33,136 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=209333.33333333334, ans=0.0 2026-09-24 06:56:34,644 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=209333.33333333334, ans=0.125 2026-09-24 06:56:35,677 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=209333.33333333334, ans=0.0 2026-09-24 06:56:39,998 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=209366.66666666666, ans=0.0 2026-09-24 06:56:41,123 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=209366.66666666666, ans=0.125 2026-09-24 06:56:41,924 INFO [train.py:1192] (0/2) Epoch 66, batch 550, loss[loss=0.2537, simple_loss=0.3825, pruned_loss=0.06246, over 24240.00 frames. ], tot_loss[loss=0.2566, simple_loss=0.3746, pruned_loss=0.06927, over 4521030.98 frames. ], batch size: 257, lr: 3.19e-03, grad_scale: 32.0 2026-09-24 06:56:43,794 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.42 vs. limit=6.0 2026-09-24 06:56:48,600 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=209433.33333333334, ans=0.1 2026-09-24 06:56:49,555 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=209433.33333333334, ans=0.125 2026-09-24 06:56:57,893 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.736e+02 3.320e+02 3.729e+02 3.919e+02 5.850e+02, threshold=7.458e+02, percent-clipped=0.0 2026-09-24 06:56:57,985 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=209500.0, ans=0.125 2026-09-24 06:56:58,506 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=209500.0, ans=0.125 2026-09-24 06:57:04,250 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.min_positive, batch_count=209533.33333333334, ans=0.025 2026-09-24 06:57:06,924 INFO [train.py:1192] (0/2) Epoch 66, batch 600, loss[loss=0.2633, simple_loss=0.395, pruned_loss=0.06582, over 24353.00 frames. ], tot_loss[loss=0.2566, simple_loss=0.3749, pruned_loss=0.06916, over 4587273.33 frames. ], batch size: 234, lr: 3.19e-03, grad_scale: 32.0 2026-09-24 06:57:07,523 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=209566.66666666666, ans=0.07 2026-09-24 06:57:09,433 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=209566.66666666666, ans=0.125 2026-09-24 06:57:09,915 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=209566.66666666666, ans=0.0 2026-09-24 06:57:15,465 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.34 vs. limit=15.0 2026-09-24 06:57:15,899 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=209600.0, ans=0.0 2026-09-24 06:57:16,783 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=209633.33333333334, ans=0.1 2026-09-24 06:57:23,110 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff2_skip_rate, batch_count=209666.66666666666, ans=0.0 2026-09-24 06:57:25,908 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer2.prob, batch_count=209666.66666666666, ans=0.125 2026-09-24 06:57:31,048 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=209700.0, ans=0.125 2026-09-24 06:57:31,970 INFO [train.py:1192] (0/2) Epoch 66, batch 650, loss[loss=0.2735, simple_loss=0.3827, pruned_loss=0.0822, over 24569.00 frames. ], tot_loss[loss=0.2552, simple_loss=0.3738, pruned_loss=0.06832, over 4652474.18 frames. ], batch size: 162, lr: 3.19e-03, grad_scale: 32.0 2026-09-24 06:57:32,056 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 06:57:45,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=209800.0, ans=0.0 2026-09-24 06:57:46,545 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=209800.0, ans=0.1 2026-09-24 06:57:46,559 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=209800.0, ans=0.1 2026-09-24 06:57:48,344 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.742e+02 3.431e+02 3.744e+02 4.216e+02 6.535e+02, threshold=7.489e+02, percent-clipped=0.0 2026-09-24 06:57:57,160 INFO [train.py:1192] (0/2) Epoch 66, batch 700, loss[loss=0.2384, simple_loss=0.3531, pruned_loss=0.06184, over 24558.00 frames. ], tot_loss[loss=0.2561, simple_loss=0.3746, pruned_loss=0.06879, over 4684076.52 frames. ], batch size: 154, lr: 3.19e-03, grad_scale: 32.0 2026-09-24 06:58:00,899 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.77 vs. limit=15.0 2026-09-24 06:58:07,755 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.37 vs. limit=15.0 2026-09-24 06:58:12,048 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.78 vs. limit=6.0 2026-09-24 06:58:17,260 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=9.24 vs. limit=15.0 2026-09-24 06:58:23,017 INFO [train.py:1192] (0/2) Epoch 66, batch 750, loss[loss=0.2513, simple_loss=0.3743, pruned_loss=0.06415, over 24576.00 frames. ], tot_loss[loss=0.2547, simple_loss=0.3731, pruned_loss=0.0682, over 4710864.92 frames. ], batch size: 170, lr: 3.19e-03, grad_scale: 32.0 2026-09-24 06:58:25,565 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=210066.66666666666, ans=0.125 2026-09-24 06:58:36,557 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.14 vs. limit=15.0 2026-09-24 06:58:38,895 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=210166.66666666666, ans=0.0 2026-09-24 06:58:39,816 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 3.007e+02 3.496e+02 4.142e+02 4.613e+02 7.327e+02, threshold=8.284e+02, percent-clipped=0.0 2026-09-24 06:58:40,953 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=210166.66666666666, ans=0.1 2026-09-24 06:58:46,359 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.07 vs. limit=12.0 2026-09-24 06:58:47,531 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=210200.0, ans=0.07 2026-09-24 06:58:48,439 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=210233.33333333334, ans=0.125 2026-09-24 06:58:48,822 INFO [train.py:1192] (0/2) Epoch 66, batch 800, loss[loss=0.2136, simple_loss=0.337, pruned_loss=0.04513, over 24548.00 frames. ], tot_loss[loss=0.2552, simple_loss=0.3735, pruned_loss=0.06851, over 4739879.36 frames. ], batch size: 137, lr: 3.19e-03, grad_scale: 32.0 2026-09-24 06:58:49,421 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=210233.33333333334, ans=0.1 2026-09-24 06:59:00,852 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.82 vs. limit=6.0 2026-09-24 06:59:08,636 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.67 vs. limit=15.0 2026-09-24 06:59:13,921 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=210400.0, ans=0.125 2026-09-24 06:59:14,283 INFO [train.py:1192] (0/2) Epoch 66, batch 850, loss[loss=0.2545, simple_loss=0.3857, pruned_loss=0.06167, over 24547.00 frames. ], tot_loss[loss=0.2551, simple_loss=0.3733, pruned_loss=0.06843, over 4761531.53 frames. ], batch size: 204, lr: 3.19e-03, grad_scale: 32.0 2026-09-24 06:59:14,380 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=210400.0, ans=0.125 2026-09-24 06:59:21,783 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=210433.33333333334, ans=0.125 2026-09-24 06:59:23,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=210433.33333333334, ans=0.125 2026-09-24 06:59:24,695 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=210466.66666666666, ans=0.0 2026-09-24 06:59:26,845 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=12.35 vs. limit=15.0 2026-09-24 06:59:30,796 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.620e+02 3.507e+02 3.896e+02 4.569e+02 6.621e+02, threshold=7.791e+02, percent-clipped=0.0 2026-09-24 06:59:33,324 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=6.98 vs. limit=15.0 2026-09-24 06:59:38,417 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=210533.33333333334, ans=0.125 2026-09-24 06:59:40,242 INFO [train.py:1192] (0/2) Epoch 66, batch 900, loss[loss=0.2118, simple_loss=0.3333, pruned_loss=0.04519, over 24595.00 frames. ], tot_loss[loss=0.2554, simple_loss=0.3736, pruned_loss=0.06855, over 4774981.77 frames. ], batch size: 137, lr: 3.18e-03, grad_scale: 32.0 2026-09-24 06:59:43,623 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=210566.66666666666, ans=0.0 2026-09-24 06:59:51,909 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=14.22 vs. limit=15.0 2026-09-24 06:59:52,113 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=210633.33333333334, ans=0.1 2026-09-24 07:00:05,314 INFO [train.py:1192] (0/2) Epoch 66, batch 950, loss[loss=0.3481, simple_loss=0.4262, pruned_loss=0.135, over 11720.00 frames. ], tot_loss[loss=0.256, simple_loss=0.3727, pruned_loss=0.06961, over 4708156.51 frames. ], batch size: 334, lr: 3.18e-03, grad_scale: 16.0 2026-09-24 07:00:09,683 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-66.pt 2026-09-24 07:00:15,381 INFO [train.py:1192] (0/2) Epoch 67, batch 0, loss[loss=0.2138, simple_loss=0.337, pruned_loss=0.04532, over 24603.00 frames. ], tot_loss[loss=0.2138, simple_loss=0.337, pruned_loss=0.04532, over 24603.00 frames. ], batch size: 137, lr: 3.16e-03, grad_scale: 32.0 2026-09-24 07:00:15,381 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 07:00:26,987 INFO [train.py:1224] (0/2) Epoch 67, validation: loss=0.1709, simple_loss=0.2889, pruned_loss=0.02645, over 2564189.00 frames. 2026-09-24 07:00:26,987 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 07:00:27,100 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:00:32,787 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=210793.33333333334, ans=0.125 2026-09-24 07:00:39,935 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.612e+02 3.476e+02 4.064e+02 4.784e+02 8.984e+02, threshold=8.128e+02, percent-clipped=1.0 2026-09-24 07:00:40,042 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=210826.66666666666, ans=0.2 2026-09-24 07:00:52,611 INFO [train.py:1192] (0/2) Epoch 67, batch 50, loss[loss=0.2054, simple_loss=0.3241, pruned_loss=0.04338, over 24276.00 frames. ], tot_loss[loss=0.2616, simple_loss=0.3792, pruned_loss=0.07204, over 1075859.66 frames. ], batch size: 125, lr: 3.16e-03, grad_scale: 32.0 2026-09-24 07:00:53,169 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:01:09,281 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=211026.66666666666, ans=10.0 2026-09-24 07:01:12,393 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=211060.0, ans=0.125 2026-09-24 07:01:17,681 INFO [train.py:1192] (0/2) Epoch 67, batch 100, loss[loss=0.2575, simple_loss=0.374, pruned_loss=0.07052, over 24639.00 frames. ], tot_loss[loss=0.2654, simple_loss=0.3837, pruned_loss=0.07352, over 1903967.51 frames. ], batch size: 154, lr: 3.16e-03, grad_scale: 32.0 2026-09-24 07:01:21,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=211093.33333333334, ans=0.0 2026-09-24 07:01:23,363 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.07 vs. limit=10.0 2026-09-24 07:01:26,451 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=211126.66666666666, ans=0.125 2026-09-24 07:01:30,025 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.574e+02 3.541e+02 3.935e+02 4.334e+02 7.130e+02, threshold=7.871e+02, percent-clipped=0.0 2026-09-24 07:01:33,187 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=211193.33333333334, ans=0.125 2026-09-24 07:01:34,468 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.36 vs. limit=15.0 2026-09-24 07:01:38,389 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=211226.66666666666, ans=0.125 2026-09-24 07:01:41,909 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=211226.66666666666, ans=0.025 2026-09-24 07:01:43,177 INFO [train.py:1192] (0/2) Epoch 67, batch 150, loss[loss=0.1922, simple_loss=0.31, pruned_loss=0.03717, over 24266.00 frames. ], tot_loss[loss=0.2604, simple_loss=0.3789, pruned_loss=0.07101, over 2557698.19 frames. ], batch size: 125, lr: 3.16e-03, grad_scale: 32.0 2026-09-24 07:01:53,492 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=211326.66666666666, ans=0.125 2026-09-24 07:01:55,613 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.67 vs. limit=22.5 2026-09-24 07:02:00,145 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=211360.0, ans=0.0 2026-09-24 07:02:02,806 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.58 vs. limit=15.0 2026-09-24 07:02:09,270 INFO [train.py:1192] (0/2) Epoch 67, batch 200, loss[loss=0.3262, simple_loss=0.4223, pruned_loss=0.115, over 21065.00 frames. ], tot_loss[loss=0.2587, simple_loss=0.377, pruned_loss=0.07024, over 3055351.68 frames. ], batch size: 333, lr: 3.15e-03, grad_scale: 32.0 2026-09-24 07:02:16,412 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:02:22,142 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.513e+02 3.399e+02 3.723e+02 4.270e+02 5.747e+02, threshold=7.446e+02, percent-clipped=0.0 2026-09-24 07:02:24,587 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=211526.66666666666, ans=0.125 2026-09-24 07:02:35,315 INFO [train.py:1192] (0/2) Epoch 67, batch 250, loss[loss=0.296, simple_loss=0.4199, pruned_loss=0.08601, over 24306.00 frames. ], tot_loss[loss=0.2579, simple_loss=0.376, pruned_loss=0.06991, over 3443556.15 frames. ], batch size: 234, lr: 3.15e-03, grad_scale: 32.0 2026-09-24 07:02:36,975 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=211593.33333333334, ans=0.125 2026-09-24 07:02:44,071 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.50 vs. limit=6.0 2026-09-24 07:02:46,374 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=211660.0, ans=0.2 2026-09-24 07:02:54,561 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=211693.33333333334, ans=0.125 2026-09-24 07:02:58,236 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=211726.66666666666, ans=0.1 2026-09-24 07:03:01,166 INFO [train.py:1192] (0/2) Epoch 67, batch 300, loss[loss=0.2729, simple_loss=0.3909, pruned_loss=0.07743, over 24547.00 frames. ], tot_loss[loss=0.2583, simple_loss=0.3757, pruned_loss=0.0704, over 3752665.44 frames. ], batch size: 204, lr: 3.15e-03, grad_scale: 32.0 2026-09-24 07:03:04,168 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.skip_rate, batch_count=211760.0, ans=0.04949747468305833 2026-09-24 07:03:07,577 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.attention_skip_rate, batch_count=211793.33333333334, ans=0.0 2026-09-24 07:03:13,841 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.648e+02 3.406e+02 3.799e+02 4.131e+02 6.192e+02, threshold=7.597e+02, percent-clipped=0.0 2026-09-24 07:03:21,441 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=211893.33333333334, ans=0.125 2026-09-24 07:03:25,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=211893.33333333334, ans=0.1 2026-09-24 07:03:26,873 INFO [train.py:1192] (0/2) Epoch 67, batch 350, loss[loss=0.2019, simple_loss=0.3204, pruned_loss=0.04172, over 24571.00 frames. ], tot_loss[loss=0.2584, simple_loss=0.3765, pruned_loss=0.07018, over 3994609.51 frames. ], batch size: 137, lr: 3.15e-03, grad_scale: 32.0 2026-09-24 07:03:43,744 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=212026.66666666666, ans=0.09899494936611666 2026-09-24 07:03:47,498 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.96 vs. limit=22.5 2026-09-24 07:03:48,460 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=212060.0, ans=0.125 2026-09-24 07:03:52,995 INFO [train.py:1192] (0/2) Epoch 67, batch 400, loss[loss=0.2549, simple_loss=0.3682, pruned_loss=0.07077, over 24558.00 frames. ], tot_loss[loss=0.2583, simple_loss=0.3761, pruned_loss=0.0703, over 4181191.47 frames. ], batch size: 170, lr: 3.15e-03, grad_scale: 32.0 2026-09-24 07:03:57,418 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=9.76 vs. limit=15.0 2026-09-24 07:04:06,105 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.632e+02 3.467e+02 3.832e+02 4.396e+02 7.099e+02, threshold=7.664e+02, percent-clipped=0.0 2026-09-24 07:04:09,357 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=212193.33333333334, ans=0.125 2026-09-24 07:04:13,861 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=212226.66666666666, ans=0.0 2026-09-24 07:04:18,260 INFO [train.py:1192] (0/2) Epoch 67, batch 450, loss[loss=0.2749, simple_loss=0.4015, pruned_loss=0.07414, over 24632.00 frames. ], tot_loss[loss=0.2575, simple_loss=0.3757, pruned_loss=0.06971, over 4313300.09 frames. ], batch size: 175, lr: 3.15e-03, grad_scale: 32.0 2026-09-24 07:04:24,183 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=212293.33333333334, ans=0.125 2026-09-24 07:04:32,807 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=212326.66666666666, ans=0.0 2026-09-24 07:04:40,287 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=212393.33333333334, ans=0.125 2026-09-24 07:04:42,284 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=212393.33333333334, ans=0.2 2026-09-24 07:04:44,113 INFO [train.py:1192] (0/2) Epoch 67, batch 500, loss[loss=0.2946, simple_loss=0.4167, pruned_loss=0.08626, over 24484.00 frames. ], tot_loss[loss=0.2573, simple_loss=0.3748, pruned_loss=0.06993, over 4430228.04 frames. ], batch size: 218, lr: 3.15e-03, grad_scale: 32.0 2026-09-24 07:04:50,963 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=212460.0, ans=0.1 2026-09-24 07:04:51,896 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=212460.0, ans=0.0 2026-09-24 07:04:54,285 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=212493.33333333334, ans=0.07 2026-09-24 07:04:56,829 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.703e+02 3.364e+02 3.669e+02 4.236e+02 6.673e+02, threshold=7.338e+02, percent-clipped=0.0 2026-09-24 07:05:02,014 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=212526.66666666666, ans=0.125 2026-09-24 07:05:02,022 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=212526.66666666666, ans=0.125 2026-09-24 07:05:04,109 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer1.prob, batch_count=212560.0, ans=0.125 2026-09-24 07:05:05,497 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=212560.0, ans=0.125 2026-09-24 07:05:09,564 INFO [train.py:1192] (0/2) Epoch 67, batch 550, loss[loss=0.2784, simple_loss=0.4109, pruned_loss=0.07293, over 24215.00 frames. ], tot_loss[loss=0.2572, simple_loss=0.3749, pruned_loss=0.06972, over 4519304.23 frames. ], batch size: 257, lr: 3.15e-03, grad_scale: 32.0 2026-09-24 07:05:28,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=212693.33333333334, ans=0.125 2026-09-24 07:05:35,537 INFO [train.py:1192] (0/2) Epoch 67, batch 600, loss[loss=0.284, simple_loss=0.4158, pruned_loss=0.07616, over 24446.00 frames. ], tot_loss[loss=0.2573, simple_loss=0.3754, pruned_loss=0.06959, over 4585477.55 frames. ], batch size: 235, lr: 3.14e-03, grad_scale: 32.0 2026-09-24 07:05:36,428 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=212760.0, ans=0.1 2026-09-24 07:05:44,965 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.min_positive, batch_count=212793.33333333334, ans=0.05 2026-09-24 07:05:48,924 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.724e+02 3.420e+02 3.759e+02 4.319e+02 7.115e+02, threshold=7.517e+02, percent-clipped=0.0 2026-09-24 07:05:54,290 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=212860.0, ans=0.125 2026-09-24 07:05:58,297 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=212893.33333333334, ans=0.1 2026-09-24 07:06:01,453 INFO [train.py:1192] (0/2) Epoch 67, batch 650, loss[loss=0.2502, simple_loss=0.3702, pruned_loss=0.0651, over 24554.00 frames. ], tot_loss[loss=0.2566, simple_loss=0.3747, pruned_loss=0.0692, over 4650568.51 frames. ], batch size: 162, lr: 3.14e-03, grad_scale: 32.0 2026-09-24 07:06:06,484 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.skip_rate, batch_count=212960.0, ans=0.07 2026-09-24 07:06:16,059 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=213026.66666666666, ans=0.2 2026-09-24 07:06:26,440 INFO [train.py:1192] (0/2) Epoch 67, batch 700, loss[loss=0.248, simple_loss=0.3659, pruned_loss=0.06503, over 24561.00 frames. ], tot_loss[loss=0.2568, simple_loss=0.3752, pruned_loss=0.06918, over 4682044.41 frames. ], batch size: 154, lr: 3.14e-03, grad_scale: 32.0 2026-09-24 07:06:39,314 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.576e+02 3.459e+02 3.714e+02 4.345e+02 6.399e+02, threshold=7.428e+02, percent-clipped=0.0 2026-09-24 07:06:45,111 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=213193.33333333334, ans=0.2 2026-09-24 07:06:47,566 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=213226.66666666666, ans=0.0 2026-09-24 07:06:48,146 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=8.70 vs. limit=12.0 2026-09-24 07:06:52,194 INFO [train.py:1192] (0/2) Epoch 67, batch 750, loss[loss=0.2513, simple_loss=0.3752, pruned_loss=0.06371, over 24550.00 frames. ], tot_loss[loss=0.2563, simple_loss=0.3742, pruned_loss=0.06921, over 4713166.70 frames. ], batch size: 170, lr: 3.14e-03, grad_scale: 32.0 2026-09-24 07:06:55,759 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=213260.0, ans=0.125 2026-09-24 07:07:00,239 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=213293.33333333334, ans=0.0 2026-09-24 07:07:03,538 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-64000.pt 2026-09-24 07:07:04,407 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.30 vs. limit=15.0 2026-09-24 07:07:18,308 INFO [train.py:1192] (0/2) Epoch 67, batch 800, loss[loss=0.2152, simple_loss=0.3344, pruned_loss=0.048, over 24544.00 frames. ], tot_loss[loss=0.2564, simple_loss=0.3741, pruned_loss=0.0693, over 4737153.70 frames. ], batch size: 137, lr: 3.14e-03, grad_scale: 32.0 2026-09-24 07:07:31,425 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.871e+02 3.614e+02 4.001e+02 4.536e+02 6.366e+02, threshold=8.002e+02, percent-clipped=0.0 2026-09-24 07:07:32,931 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=213493.33333333334, ans=0.1 2026-09-24 07:07:35,723 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=213526.66666666666, ans=0.125 2026-09-24 07:07:36,309 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.58 vs. limit=22.5 2026-09-24 07:07:39,252 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=213560.0, ans=0.125 2026-09-24 07:07:43,985 INFO [train.py:1192] (0/2) Epoch 67, batch 850, loss[loss=0.2841, simple_loss=0.4001, pruned_loss=0.08404, over 24562.00 frames. ], tot_loss[loss=0.2554, simple_loss=0.3735, pruned_loss=0.06866, over 4761307.48 frames. ], batch size: 204, lr: 3.14e-03, grad_scale: 32.0 2026-09-24 07:07:44,627 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.const_attention_rate, batch_count=213593.33333333334, ans=0.025 2026-09-24 07:07:46,610 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=213593.33333333334, ans=0.125 2026-09-24 07:07:51,172 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=213626.66666666666, ans=0.0 2026-09-24 07:08:02,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=213693.33333333334, ans=0.1 2026-09-24 07:08:09,081 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.prob, batch_count=213760.0, ans=0.125 2026-09-24 07:08:09,559 INFO [train.py:1192] (0/2) Epoch 67, batch 900, loss[loss=0.2401, simple_loss=0.3509, pruned_loss=0.06465, over 24550.00 frames. ], tot_loss[loss=0.256, simple_loss=0.3742, pruned_loss=0.06891, over 4775049.57 frames. ], batch size: 137, lr: 3.14e-03, grad_scale: 32.0 2026-09-24 07:08:22,420 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.594e+02 3.509e+02 4.010e+02 4.495e+02 6.260e+02, threshold=8.019e+02, percent-clipped=0.0 2026-09-24 07:08:35,373 INFO [train.py:1192] (0/2) Epoch 67, batch 950, loss[loss=0.3527, simple_loss=0.4183, pruned_loss=0.1436, over 11152.00 frames. ], tot_loss[loss=0.256, simple_loss=0.3729, pruned_loss=0.06954, over 4714719.91 frames. ], batch size: 333, lr: 3.14e-03, grad_scale: 32.0 2026-09-24 07:08:39,593 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-67.pt 2026-09-24 07:08:44,514 INFO [train.py:1192] (0/2) Epoch 68, batch 0, loss[loss=0.208, simple_loss=0.3308, pruned_loss=0.04263, over 24571.00 frames. ], tot_loss[loss=0.208, simple_loss=0.3308, pruned_loss=0.04263, over 24571.00 frames. ], batch size: 137, lr: 3.11e-03, grad_scale: 32.0 2026-09-24 07:08:44,514 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 07:08:47,520 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.8983, 2.1958, 3.2489, 1.4368], device='cuda:0') 2026-09-24 07:08:47,837 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.7766, 2.6770, 2.1587, 3.1945], device='cuda:0') 2026-09-24 07:08:56,321 INFO [train.py:1224] (0/2) Epoch 68, validation: loss=0.17, simple_loss=0.2884, pruned_loss=0.0258, over 2564189.00 frames. 2026-09-24 07:08:56,322 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 07:08:56,427 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:09:00,891 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=213953.33333333334, ans=0.125 2026-09-24 07:09:01,871 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=213986.66666666666, ans=0.125 2026-09-24 07:09:03,393 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.43 vs. limit=15.0 2026-09-24 07:09:12,439 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=214053.33333333334, ans=0.125 2026-09-24 07:09:19,017 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=214086.66666666666, ans=0.0 2026-09-24 07:09:20,056 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.min_positive, batch_count=214086.66666666666, ans=0.05 2026-09-24 07:09:22,136 INFO [train.py:1192] (0/2) Epoch 68, batch 50, loss[loss=0.2133, simple_loss=0.3211, pruned_loss=0.0528, over 24275.00 frames. ], tot_loss[loss=0.2599, simple_loss=0.3774, pruned_loss=0.07117, over 1075514.83 frames. ], batch size: 125, lr: 3.11e-03, grad_scale: 32.0 2026-09-24 07:09:29,610 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=214153.33333333334, ans=0.5 2026-09-24 07:09:30,324 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.745e+02 3.640e+02 4.106e+02 4.924e+02 7.084e+02, threshold=8.211e+02, percent-clipped=0.0 2026-09-24 07:09:35,735 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer2.prob, batch_count=214186.66666666666, ans=0.125 2026-09-24 07:09:46,862 INFO [train.py:1192] (0/2) Epoch 68, batch 100, loss[loss=0.2594, simple_loss=0.3705, pruned_loss=0.07413, over 24608.00 frames. ], tot_loss[loss=0.2629, simple_loss=0.3812, pruned_loss=0.07229, over 1904108.05 frames. ], batch size: 154, lr: 3.11e-03, grad_scale: 32.0 2026-09-24 07:10:12,471 INFO [train.py:1192] (0/2) Epoch 68, batch 150, loss[loss=0.2138, simple_loss=0.3228, pruned_loss=0.05244, over 24272.00 frames. ], tot_loss[loss=0.2594, simple_loss=0.3776, pruned_loss=0.0706, over 2557598.19 frames. ], batch size: 125, lr: 3.11e-03, grad_scale: 32.0 2026-09-24 07:10:13,583 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=214453.33333333334, ans=0.125 2026-09-24 07:10:16,079 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.85 vs. limit=12.0 2026-09-24 07:10:21,413 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.370e+02 3.475e+02 3.740e+02 4.253e+02 5.776e+02, threshold=7.480e+02, percent-clipped=0.0 2026-09-24 07:10:29,194 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=9.42 vs. limit=15.0 2026-09-24 07:10:31,249 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=214553.33333333334, ans=0.0 2026-09-24 07:10:35,105 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.85 vs. limit=12.0 2026-09-24 07:10:38,400 INFO [train.py:1192] (0/2) Epoch 68, batch 200, loss[loss=0.3179, simple_loss=0.422, pruned_loss=0.1069, over 21136.00 frames. ], tot_loss[loss=0.2575, simple_loss=0.3761, pruned_loss=0.0694, over 3054733.54 frames. ], batch size: 333, lr: 3.11e-03, grad_scale: 32.0 2026-09-24 07:10:39,273 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn1.whiten, num_groups=1, num_channels=192, metric=13.00 vs. limit=22.5 2026-09-24 07:10:44,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=214653.33333333334, ans=0.1 2026-09-24 07:10:55,382 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.48 vs. limit=15.0 2026-09-24 07:10:56,848 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=214720.0, ans=0.95 2026-09-24 07:10:59,717 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=214753.33333333334, ans=0.025 2026-09-24 07:11:03,806 INFO [train.py:1192] (0/2) Epoch 68, batch 250, loss[loss=0.277, simple_loss=0.4065, pruned_loss=0.07373, over 24300.00 frames. ], tot_loss[loss=0.2569, simple_loss=0.3757, pruned_loss=0.06903, over 3443243.02 frames. ], batch size: 234, lr: 3.11e-03, grad_scale: 32.0 2026-09-24 07:11:04,512 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=214786.66666666666, ans=0.0 2026-09-24 07:11:10,367 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=8.72 vs. limit=22.5 2026-09-24 07:11:12,904 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.745e+02 3.398e+02 3.956e+02 4.810e+02 6.309e+02, threshold=7.912e+02, percent-clipped=0.0 2026-09-24 07:11:21,970 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=214886.66666666666, ans=0.0 2026-09-24 07:11:27,018 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.31 vs. limit=12.0 2026-09-24 07:11:29,748 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=214953.33333333334, ans=10.0 2026-09-24 07:11:30,097 INFO [train.py:1192] (0/2) Epoch 68, batch 300, loss[loss=0.28, simple_loss=0.3983, pruned_loss=0.08086, over 24564.00 frames. ], tot_loss[loss=0.256, simple_loss=0.3745, pruned_loss=0.06877, over 3752258.19 frames. ], batch size: 204, lr: 3.11e-03, grad_scale: 32.0 2026-09-24 07:11:32,599 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten.whitening_limit, batch_count=214953.33333333334, ans=15.0 2026-09-24 07:11:39,243 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=214986.66666666666, ans=0.1 2026-09-24 07:11:41,645 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=215020.0, ans=0.125 2026-09-24 07:11:45,531 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=215053.33333333334, ans=0.125 2026-09-24 07:11:46,073 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.63 vs. limit=15.0 2026-09-24 07:11:47,332 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=215053.33333333334, ans=0.125 2026-09-24 07:11:53,490 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=3.90 vs. limit=12.0 2026-09-24 07:11:55,921 INFO [train.py:1192] (0/2) Epoch 68, batch 350, loss[loss=0.2284, simple_loss=0.3376, pruned_loss=0.05963, over 24563.00 frames. ], tot_loss[loss=0.2562, simple_loss=0.375, pruned_loss=0.06871, over 3994698.16 frames. ], batch size: 137, lr: 3.10e-03, grad_scale: 32.0 2026-09-24 07:11:58,616 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=215120.0, ans=0.125 2026-09-24 07:12:04,078 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.822e+02 3.399e+02 3.724e+02 4.282e+02 5.845e+02, threshold=7.447e+02, percent-clipped=0.0 2026-09-24 07:12:20,082 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=215253.33333333334, ans=0.0 2026-09-24 07:12:21,559 INFO [train.py:1192] (0/2) Epoch 68, batch 400, loss[loss=0.2642, simple_loss=0.3817, pruned_loss=0.07342, over 24570.00 frames. ], tot_loss[loss=0.2561, simple_loss=0.3747, pruned_loss=0.06873, over 4178637.29 frames. ], batch size: 170, lr: 3.10e-03, grad_scale: 32.0 2026-09-24 07:12:38,048 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=215386.66666666666, ans=0.125 2026-09-24 07:12:47,197 INFO [train.py:1192] (0/2) Epoch 68, batch 450, loss[loss=0.2764, simple_loss=0.3965, pruned_loss=0.0782, over 24634.00 frames. ], tot_loss[loss=0.2575, simple_loss=0.3757, pruned_loss=0.0697, over 4312046.76 frames. ], batch size: 175, lr: 3.10e-03, grad_scale: 32.0 2026-09-24 07:12:53,689 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=215486.66666666666, ans=0.0 2026-09-24 07:12:55,952 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.825e+02 3.489e+02 3.928e+02 4.300e+02 6.364e+02, threshold=7.857e+02, percent-clipped=0.0 2026-09-24 07:12:58,335 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=215520.0, ans=0.1 2026-09-24 07:12:59,849 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=215520.0, ans=10.0 2026-09-24 07:13:06,997 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer2.prob, batch_count=215586.66666666666, ans=0.125 2026-09-24 07:13:08,662 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=215586.66666666666, ans=0.1 2026-09-24 07:13:12,698 INFO [train.py:1192] (0/2) Epoch 68, batch 500, loss[loss=0.2696, simple_loss=0.3961, pruned_loss=0.07151, over 24513.00 frames. ], tot_loss[loss=0.256, simple_loss=0.374, pruned_loss=0.06906, over 4429556.62 frames. ], batch size: 218, lr: 3.10e-03, grad_scale: 32.0 2026-09-24 07:13:15,007 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.21 vs. limit=15.0 2026-09-24 07:13:15,835 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=215620.0, ans=0.2 2026-09-24 07:13:17,362 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=215653.33333333334, ans=0.0 2026-09-24 07:13:19,109 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.09 vs. limit=15.0 2026-09-24 07:13:21,404 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=215653.33333333334, ans=0.125 2026-09-24 07:13:28,839 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.38 vs. limit=15.0 2026-09-24 07:13:38,729 INFO [train.py:1192] (0/2) Epoch 68, batch 550, loss[loss=0.2674, simple_loss=0.3983, pruned_loss=0.06831, over 24225.00 frames. ], tot_loss[loss=0.2565, simple_loss=0.3746, pruned_loss=0.06916, over 4518791.78 frames. ], batch size: 257, lr: 3.10e-03, grad_scale: 32.0 2026-09-24 07:13:44,212 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=5.59 vs. limit=15.0 2026-09-24 07:13:46,479 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=215820.0, ans=0.0 2026-09-24 07:13:47,376 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.517e+02 3.330e+02 3.662e+02 4.043e+02 5.524e+02, threshold=7.324e+02, percent-clipped=0.0 2026-09-24 07:13:50,220 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=215853.33333333334, ans=0.025 2026-09-24 07:14:02,038 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.02 vs. limit=22.5 2026-09-24 07:14:04,509 INFO [train.py:1192] (0/2) Epoch 68, batch 600, loss[loss=0.2922, simple_loss=0.4179, pruned_loss=0.08323, over 24349.00 frames. ], tot_loss[loss=0.2565, simple_loss=0.375, pruned_loss=0.06895, over 4584924.75 frames. ], batch size: 234, lr: 3.10e-03, grad_scale: 32.0 2026-09-24 07:14:29,990 INFO [train.py:1192] (0/2) Epoch 68, batch 650, loss[loss=0.275, simple_loss=0.3871, pruned_loss=0.08142, over 24574.00 frames. ], tot_loss[loss=0.2562, simple_loss=0.3747, pruned_loss=0.06882, over 4650346.18 frames. ], batch size: 162, lr: 3.10e-03, grad_scale: 32.0 2026-09-24 07:14:30,068 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=216120.0, ans=0.1 2026-09-24 07:14:38,831 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.864e+02 3.384e+02 3.669e+02 4.044e+02 6.181e+02, threshold=7.338e+02, percent-clipped=0.0 2026-09-24 07:14:40,335 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=216186.66666666666, ans=0.0 2026-09-24 07:14:51,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=216253.33333333334, ans=0.125 2026-09-24 07:14:51,685 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.33 vs. limit=15.0 2026-09-24 07:14:55,706 INFO [train.py:1192] (0/2) Epoch 68, batch 700, loss[loss=0.2527, simple_loss=0.362, pruned_loss=0.07174, over 24565.00 frames. ], tot_loss[loss=0.2562, simple_loss=0.375, pruned_loss=0.06873, over 4682252.90 frames. ], batch size: 154, lr: 3.10e-03, grad_scale: 32.0 2026-09-24 07:14:58,190 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=216286.66666666666, ans=0.5 2026-09-24 07:15:01,955 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=216320.0, ans=0.125 2026-09-24 07:15:10,167 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=216353.33333333334, ans=0.0 2026-09-24 07:15:10,815 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.63 vs. limit=12.0 2026-09-24 07:15:15,218 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=216386.66666666666, ans=0.125 2026-09-24 07:15:17,693 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=216420.0, ans=0.09899494936611666 2026-09-24 07:15:21,786 INFO [train.py:1192] (0/2) Epoch 68, batch 750, loss[loss=0.274, simple_loss=0.3854, pruned_loss=0.08134, over 24555.00 frames. ], tot_loss[loss=0.2556, simple_loss=0.3739, pruned_loss=0.06862, over 4708935.73 frames. ], batch size: 170, lr: 3.09e-03, grad_scale: 32.0 2026-09-24 07:15:24,046 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=216453.33333333334, ans=0.125 2026-09-24 07:15:30,336 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.632e+02 3.551e+02 4.134e+02 4.479e+02 7.371e+02, threshold=8.267e+02, percent-clipped=1.0 2026-09-24 07:15:42,959 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=216586.66666666666, ans=0.1 2026-09-24 07:15:45,973 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:15:46,846 INFO [train.py:1192] (0/2) Epoch 68, batch 800, loss[loss=0.2162, simple_loss=0.3322, pruned_loss=0.05007, over 24529.00 frames. ], tot_loss[loss=0.2549, simple_loss=0.3734, pruned_loss=0.06825, over 4734100.62 frames. ], batch size: 137, lr: 3.09e-03, grad_scale: 32.0 2026-09-24 07:15:52,255 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.27 vs. limit=22.5 2026-09-24 07:15:59,076 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=216686.66666666666, ans=0.125 2026-09-24 07:16:01,988 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=216720.0, ans=0.07 2026-09-24 07:16:04,131 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=216720.0, ans=0.125 2026-09-24 07:16:08,078 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=11.27 vs. limit=15.0 2026-09-24 07:16:11,718 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=6.67 vs. limit=15.0 2026-09-24 07:16:12,510 INFO [train.py:1192] (0/2) Epoch 68, batch 850, loss[loss=0.2857, simple_loss=0.4023, pruned_loss=0.08456, over 24604.00 frames. ], tot_loss[loss=0.2545, simple_loss=0.3729, pruned_loss=0.06802, over 4757513.29 frames. ], batch size: 198, lr: 3.09e-03, grad_scale: 32.0 2026-09-24 07:16:21,233 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.733e+02 3.458e+02 3.987e+02 4.457e+02 5.664e+02, threshold=7.974e+02, percent-clipped=0.0 2026-09-24 07:16:22,825 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=216853.33333333334, ans=0.1 2026-09-24 07:16:31,153 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.42 vs. limit=15.0 2026-09-24 07:16:37,528 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=10.51 vs. limit=15.0 2026-09-24 07:16:38,118 INFO [train.py:1192] (0/2) Epoch 68, batch 900, loss[loss=0.2218, simple_loss=0.3419, pruned_loss=0.05084, over 24532.00 frames. ], tot_loss[loss=0.2542, simple_loss=0.3727, pruned_loss=0.06779, over 4771112.76 frames. ], batch size: 137, lr: 3.09e-03, grad_scale: 32.0 2026-09-24 07:16:44,056 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=216986.66666666666, ans=0.2 2026-09-24 07:16:47,880 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=217020.0, ans=0.1 2026-09-24 07:16:56,301 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=217053.33333333334, ans=0.2 2026-09-24 07:16:58,134 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=217086.66666666666, ans=0.1 2026-09-24 07:17:00,376 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.attention_skip_rate, batch_count=217086.66666666666, ans=0.0 2026-09-24 07:17:02,315 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=217120.0, ans=0.1 2026-09-24 07:17:02,733 INFO [train.py:1192] (0/2) Epoch 68, batch 950, loss[loss=0.3515, simple_loss=0.429, pruned_loss=0.137, over 11022.00 frames. ], tot_loss[loss=0.2542, simple_loss=0.3715, pruned_loss=0.06849, over 4713749.95 frames. ], batch size: 334, lr: 3.09e-03, grad_scale: 32.0 2026-09-24 07:17:07,067 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-68.pt 2026-09-24 07:17:40,884 INFO [train.py:1192] (0/2) Epoch 69, batch 0, loss[loss=0.2054, simple_loss=0.3306, pruned_loss=0.04015, over 24571.00 frames. ], tot_loss[loss=0.2054, simple_loss=0.3306, pruned_loss=0.04015, over 24571.00 frames. ], batch size: 137, lr: 3.07e-03, grad_scale: 32.0 2026-09-24 07:17:40,884 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 07:17:52,462 INFO [train.py:1224] (0/2) Epoch 69, validation: loss=0.1691, simple_loss=0.2872, pruned_loss=0.02547, over 2564189.00 frames. 2026-09-24 07:17:52,462 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 07:17:54,026 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=217146.66666666666, ans=0.1 2026-09-24 07:17:56,833 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.733e+02 3.531e+02 4.038e+02 4.647e+02 6.528e+02, threshold=8.076e+02, percent-clipped=0.0 2026-09-24 07:17:59,736 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.28 vs. limit=22.5 2026-09-24 07:18:02,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=217213.33333333334, ans=0.125 2026-09-24 07:18:05,464 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=217213.33333333334, ans=0.0 2026-09-24 07:18:09,342 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=217246.66666666666, ans=0.0 2026-09-24 07:18:15,754 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=217280.0, ans=0.125 2026-09-24 07:18:17,522 INFO [train.py:1192] (0/2) Epoch 69, batch 50, loss[loss=0.2339, simple_loss=0.3431, pruned_loss=0.06238, over 24236.00 frames. ], tot_loss[loss=0.261, simple_loss=0.3784, pruned_loss=0.07183, over 1075259.70 frames. ], batch size: 125, lr: 3.07e-03, grad_scale: 32.0 2026-09-24 07:18:19,634 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=217313.33333333334, ans=0.125 2026-09-24 07:18:20,123 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=217313.33333333334, ans=0.125 2026-09-24 07:18:23,728 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=217346.66666666666, ans=0.125 2026-09-24 07:18:27,581 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.13 vs. limit=12.0 2026-09-24 07:18:31,278 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.73 vs. limit=15.0 2026-09-24 07:18:38,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=217446.66666666666, ans=0.125 2026-09-24 07:18:42,242 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=217446.66666666666, ans=0.0 2026-09-24 07:18:43,404 INFO [train.py:1192] (0/2) Epoch 69, batch 100, loss[loss=0.2411, simple_loss=0.365, pruned_loss=0.0586, over 24628.00 frames. ], tot_loss[loss=0.2634, simple_loss=0.3822, pruned_loss=0.07227, over 1903678.37 frames. ], batch size: 154, lr: 3.06e-03, grad_scale: 64.0 2026-09-24 07:18:47,700 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.741e+02 3.588e+02 3.935e+02 4.341e+02 5.984e+02, threshold=7.870e+02, percent-clipped=0.0 2026-09-24 07:19:01,795 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:19:08,937 INFO [train.py:1192] (0/2) Epoch 69, batch 150, loss[loss=0.2167, simple_loss=0.3236, pruned_loss=0.0549, over 24274.00 frames. ], tot_loss[loss=0.2581, simple_loss=0.3771, pruned_loss=0.06952, over 2557660.89 frames. ], batch size: 125, lr: 3.06e-03, grad_scale: 64.0 2026-09-24 07:19:34,964 INFO [train.py:1192] (0/2) Epoch 69, batch 200, loss[loss=0.2887, simple_loss=0.3957, pruned_loss=0.09084, over 20898.00 frames. ], tot_loss[loss=0.256, simple_loss=0.3751, pruned_loss=0.06846, over 3054546.27 frames. ], batch size: 333, lr: 3.06e-03, grad_scale: 64.0 2026-09-24 07:19:39,199 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.799e+02 3.352e+02 3.841e+02 4.410e+02 6.405e+02, threshold=7.682e+02, percent-clipped=0.0 2026-09-24 07:19:44,987 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=217880.0, ans=0.0 2026-09-24 07:19:53,339 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=217913.33333333334, ans=0.125 2026-09-24 07:19:58,100 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.34 vs. limit=10.0 2026-09-24 07:20:00,536 INFO [train.py:1192] (0/2) Epoch 69, batch 250, loss[loss=0.2969, simple_loss=0.417, pruned_loss=0.08836, over 24335.00 frames. ], tot_loss[loss=0.256, simple_loss=0.3747, pruned_loss=0.06863, over 3442823.16 frames. ], batch size: 234, lr: 3.06e-03, grad_scale: 32.0 2026-09-24 07:20:00,638 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=217980.0, ans=0.2 2026-09-24 07:20:07,580 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=218013.33333333334, ans=0.125 2026-09-24 07:20:09,417 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=218013.33333333334, ans=0.125 2026-09-24 07:20:12,674 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=218046.66666666666, ans=0.1 2026-09-24 07:20:14,054 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=218046.66666666666, ans=0.0 2026-09-24 07:20:24,891 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=218113.33333333334, ans=0.125 2026-09-24 07:20:24,910 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=218113.33333333334, ans=0.95 2026-09-24 07:20:26,401 INFO [train.py:1192] (0/2) Epoch 69, batch 300, loss[loss=0.276, simple_loss=0.4033, pruned_loss=0.0743, over 24538.00 frames. ], tot_loss[loss=0.2568, simple_loss=0.3748, pruned_loss=0.06935, over 3747693.99 frames. ], batch size: 204, lr: 3.06e-03, grad_scale: 32.0 2026-09-24 07:20:31,668 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.606e+02 3.510e+02 4.031e+02 4.663e+02 6.511e+02, threshold=8.061e+02, percent-clipped=0.0 2026-09-24 07:20:44,466 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=218246.66666666666, ans=0.2 2026-09-24 07:20:44,904 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=218246.66666666666, ans=0.125 2026-09-24 07:20:52,928 INFO [train.py:1192] (0/2) Epoch 69, batch 350, loss[loss=0.2213, simple_loss=0.336, pruned_loss=0.05329, over 24563.00 frames. ], tot_loss[loss=0.2578, simple_loss=0.3761, pruned_loss=0.06976, over 3991042.39 frames. ], batch size: 137, lr: 3.06e-03, grad_scale: 32.0 2026-09-24 07:21:01,248 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=218346.66666666666, ans=0.1 2026-09-24 07:21:02,042 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=218346.66666666666, ans=0.0 2026-09-24 07:21:02,533 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=218346.66666666666, ans=0.125 2026-09-24 07:21:09,467 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer_ff3.min_abs, batch_count=218413.33333333334, ans=0.2 2026-09-24 07:21:09,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=218413.33333333334, ans=0.125 2026-09-24 07:21:11,807 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.1.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:21:18,286 INFO [train.py:1192] (0/2) Epoch 69, batch 400, loss[loss=0.277, simple_loss=0.3935, pruned_loss=0.08029, over 24569.00 frames. ], tot_loss[loss=0.2565, simple_loss=0.3749, pruned_loss=0.06904, over 4177491.35 frames. ], batch size: 170, lr: 3.06e-03, grad_scale: 32.0 2026-09-24 07:21:21,431 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.90 vs. limit=6.0 2026-09-24 07:21:23,427 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.919e+02 3.473e+02 3.848e+02 4.402e+02 6.832e+02, threshold=7.696e+02, percent-clipped=0.0 2026-09-24 07:21:24,589 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=218513.33333333334, ans=0.125 2026-09-24 07:21:33,771 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=218580.0, ans=0.025 2026-09-24 07:21:37,913 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=218580.0, ans=0.2 2026-09-24 07:21:42,585 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=218613.33333333334, ans=0.125 2026-09-24 07:21:44,375 INFO [train.py:1192] (0/2) Epoch 69, batch 450, loss[loss=0.2493, simple_loss=0.3722, pruned_loss=0.06319, over 24626.00 frames. ], tot_loss[loss=0.2573, simple_loss=0.3754, pruned_loss=0.06958, over 4308405.24 frames. ], batch size: 175, lr: 3.06e-03, grad_scale: 16.0 2026-09-24 07:21:49,001 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=218646.66666666666, ans=0.1 2026-09-24 07:21:54,461 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=218713.33333333334, ans=0.1 2026-09-24 07:21:56,354 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=5.48 vs. limit=15.0 2026-09-24 07:22:02,587 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=218746.66666666666, ans=0.125 2026-09-24 07:22:09,415 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=218780.0, ans=0.125 2026-09-24 07:22:10,551 INFO [train.py:1192] (0/2) Epoch 69, batch 500, loss[loss=0.2717, simple_loss=0.4006, pruned_loss=0.07142, over 24515.00 frames. ], tot_loss[loss=0.2555, simple_loss=0.3737, pruned_loss=0.06871, over 4427188.98 frames. ], batch size: 218, lr: 3.06e-03, grad_scale: 16.0 2026-09-24 07:22:12,679 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=218813.33333333334, ans=0.0 2026-09-24 07:22:12,684 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=218813.33333333334, ans=0.125 2026-09-24 07:22:16,002 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.776e+02 3.243e+02 3.613e+02 4.164e+02 1.083e+03, threshold=7.226e+02, percent-clipped=1.0 2026-09-24 07:22:17,695 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.max_abs, batch_count=218846.66666666666, ans=10.0 2026-09-24 07:22:20,164 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:22:21,824 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.prob, batch_count=218880.0, ans=0.125 2026-09-24 07:22:21,852 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=218880.0, ans=0.1 2026-09-24 07:22:25,891 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=218913.33333333334, ans=0.125 2026-09-24 07:22:34,051 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=218946.66666666666, ans=0.0 2026-09-24 07:22:36,592 INFO [train.py:1192] (0/2) Epoch 69, batch 550, loss[loss=0.3137, simple_loss=0.4372, pruned_loss=0.09507, over 24280.00 frames. ], tot_loss[loss=0.2558, simple_loss=0.3742, pruned_loss=0.06876, over 4517232.57 frames. ], batch size: 257, lr: 3.05e-03, grad_scale: 16.0 2026-09-24 07:22:39,789 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.37 vs. limit=15.0 2026-09-24 07:22:42,282 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=4.40 vs. limit=15.0 2026-09-24 07:22:53,347 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=219080.0, ans=0.125 2026-09-24 07:22:59,677 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.convnext.out_whiten, num_groups=1, num_channels=128, metric=3.97 vs. limit=5.0 2026-09-24 07:23:01,291 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=219113.33333333334, ans=0.2 2026-09-24 07:23:02,642 INFO [train.py:1192] (0/2) Epoch 69, batch 600, loss[loss=0.2875, simple_loss=0.4103, pruned_loss=0.08233, over 24440.00 frames. ], tot_loss[loss=0.2559, simple_loss=0.3745, pruned_loss=0.06861, over 4585090.01 frames. ], batch size: 235, lr: 3.05e-03, grad_scale: 16.0 2026-09-24 07:23:06,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=219146.66666666666, ans=0.0 2026-09-24 07:23:08,221 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.677e+02 3.268e+02 3.901e+02 4.607e+02 6.489e+02, threshold=7.802e+02, percent-clipped=0.0 2026-09-24 07:23:22,070 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=219246.66666666666, ans=0.125 2026-09-24 07:23:24,083 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer1.prob, batch_count=219280.0, ans=0.125 2026-09-24 07:23:24,534 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=219280.0, ans=0.0 2026-09-24 07:23:27,001 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.75 vs. limit=10.0 2026-09-24 07:23:27,819 INFO [train.py:1192] (0/2) Epoch 69, batch 650, loss[loss=0.2711, simple_loss=0.3824, pruned_loss=0.07996, over 24564.00 frames. ], tot_loss[loss=0.2545, simple_loss=0.3735, pruned_loss=0.06776, over 4650348.10 frames. ], batch size: 162, lr: 3.05e-03, grad_scale: 16.0 2026-09-24 07:23:29,233 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:23:52,570 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.99 vs. limit=12.0 2026-09-24 07:23:53,974 INFO [train.py:1192] (0/2) Epoch 69, batch 700, loss[loss=0.2504, simple_loss=0.3635, pruned_loss=0.06864, over 24569.00 frames. ], tot_loss[loss=0.2548, simple_loss=0.374, pruned_loss=0.06781, over 4683326.88 frames. ], batch size: 154, lr: 3.05e-03, grad_scale: 16.0 2026-09-24 07:23:59,796 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.624e+02 3.327e+02 3.649e+02 4.152e+02 7.303e+02, threshold=7.297e+02, percent-clipped=0.0 2026-09-24 07:24:16,990 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:24:18,866 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=219613.33333333334, ans=0.09899494936611666 2026-09-24 07:24:19,851 INFO [train.py:1192] (0/2) Epoch 69, batch 750, loss[loss=0.2548, simple_loss=0.3752, pruned_loss=0.06721, over 24560.00 frames. ], tot_loss[loss=0.2537, simple_loss=0.3725, pruned_loss=0.06746, over 4710234.61 frames. ], batch size: 170, lr: 3.05e-03, grad_scale: 16.0 2026-09-24 07:24:40,051 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.03 vs. limit=15.0 2026-09-24 07:24:40,524 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.78 vs. limit=10.0 2026-09-24 07:24:40,793 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.scale_min, batch_count=219780.0, ans=0.2 2026-09-24 07:24:42,596 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.self_attn2.whiten, num_groups=1, num_channels=192, metric=13.23 vs. limit=22.5 2026-09-24 07:24:45,599 INFO [train.py:1192] (0/2) Epoch 69, batch 800, loss[loss=0.2058, simple_loss=0.3241, pruned_loss=0.0438, over 24552.00 frames. ], tot_loss[loss=0.2539, simple_loss=0.3727, pruned_loss=0.06758, over 4734737.04 frames. ], batch size: 137, lr: 3.05e-03, grad_scale: 32.0 2026-09-24 07:24:46,972 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=219813.33333333334, ans=0.1 2026-09-24 07:24:47,485 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=219813.33333333334, ans=0.2 2026-09-24 07:24:49,575 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=219813.33333333334, ans=0.0 2026-09-24 07:24:51,633 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.655e+02 3.384e+02 3.857e+02 4.281e+02 7.154e+02, threshold=7.715e+02, percent-clipped=0.0 2026-09-24 07:24:54,840 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.52 vs. limit=15.0 2026-09-24 07:24:55,735 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.04 vs. limit=15.0 2026-09-24 07:24:58,160 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=219880.0, ans=0.2 2026-09-24 07:25:01,475 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.attention_skip_rate, batch_count=219913.33333333334, ans=0.0 2026-09-24 07:25:06,030 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=219946.66666666666, ans=0.125 2026-09-24 07:25:11,908 INFO [train.py:1192] (0/2) Epoch 69, batch 850, loss[loss=0.2717, simple_loss=0.3986, pruned_loss=0.07236, over 24583.00 frames. ], tot_loss[loss=0.2544, simple_loss=0.373, pruned_loss=0.06787, over 4757766.07 frames. ], batch size: 198, lr: 3.05e-03, grad_scale: 32.0 2026-09-24 07:25:17,865 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.07 vs. limit=22.5 2026-09-24 07:25:18,246 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=220013.33333333334, ans=0.125 2026-09-24 07:25:18,779 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=220013.33333333334, ans=0.0 2026-09-24 07:25:24,309 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.69 vs. limit=12.0 2026-09-24 07:25:28,184 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.24 vs. limit=6.0 2026-09-24 07:25:38,144 INFO [train.py:1192] (0/2) Epoch 69, batch 900, loss[loss=0.208, simple_loss=0.3319, pruned_loss=0.0421, over 24544.00 frames. ], tot_loss[loss=0.2547, simple_loss=0.3734, pruned_loss=0.06804, over 4772279.46 frames. ], batch size: 137, lr: 3.05e-03, grad_scale: 32.0 2026-09-24 07:25:43,774 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.692e+02 3.583e+02 4.252e+02 4.779e+02 6.888e+02, threshold=8.504e+02, percent-clipped=0.0 2026-09-24 07:25:53,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=220246.66666666666, ans=0.125 2026-09-24 07:26:03,477 INFO [train.py:1192] (0/2) Epoch 69, batch 950, loss[loss=0.3633, simple_loss=0.432, pruned_loss=0.1474, over 11293.00 frames. ], tot_loss[loss=0.2553, simple_loss=0.3722, pruned_loss=0.06914, over 4714775.89 frames. ], batch size: 334, lr: 3.05e-03, grad_scale: 32.0 2026-09-24 07:26:05,501 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer2.prob, batch_count=220313.33333333334, ans=0.125 2026-09-24 07:26:07,840 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-69.pt 2026-09-24 07:26:13,677 INFO [train.py:1192] (0/2) Epoch 70, batch 0, loss[loss=0.1996, simple_loss=0.3245, pruned_loss=0.03729, over 24545.00 frames. ], tot_loss[loss=0.1996, simple_loss=0.3245, pruned_loss=0.03729, over 24545.00 frames. ], batch size: 137, lr: 3.02e-03, grad_scale: 32.0 2026-09-24 07:26:13,677 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 07:26:20,361 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.9419, 2.8097, 2.3691, 3.3969], device='cuda:0') 2026-09-24 07:26:25,448 INFO [train.py:1224] (0/2) Epoch 70, validation: loss=0.1687, simple_loss=0.2872, pruned_loss=0.02509, over 2564189.00 frames. 2026-09-24 07:26:25,449 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 07:26:30,184 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=220373.33333333334, ans=0.125 2026-09-24 07:26:31,124 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=220373.33333333334, ans=0.1 2026-09-24 07:26:39,502 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=220406.66666666666, ans=0.0 2026-09-24 07:26:45,473 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=220473.33333333334, ans=0.025 2026-09-24 07:26:51,172 INFO [train.py:1192] (0/2) Epoch 70, batch 50, loss[loss=0.233, simple_loss=0.339, pruned_loss=0.0635, over 24238.00 frames. ], tot_loss[loss=0.2586, simple_loss=0.3765, pruned_loss=0.07035, over 1077347.73 frames. ], batch size: 125, lr: 3.02e-03, grad_scale: 32.0 2026-09-24 07:26:52,421 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.735e+02 3.450e+02 4.028e+02 5.068e+02 6.444e+02, threshold=8.056e+02, percent-clipped=0.0 2026-09-24 07:26:53,618 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.12 vs. limit=15.0 2026-09-24 07:26:55,264 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.93 vs. limit=15.0 2026-09-24 07:27:03,832 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=220573.33333333334, ans=0.0 2026-09-24 07:27:03,986 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.09 vs. limit=10.0 2026-09-24 07:27:16,697 INFO [train.py:1192] (0/2) Epoch 70, batch 100, loss[loss=0.2253, simple_loss=0.3453, pruned_loss=0.0527, over 24618.00 frames. ], tot_loss[loss=0.2621, simple_loss=0.3808, pruned_loss=0.0717, over 1905074.29 frames. ], batch size: 154, lr: 3.02e-03, grad_scale: 32.0 2026-09-24 07:27:20,272 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.89 vs. limit=22.5 2026-09-24 07:27:32,712 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=220773.33333333334, ans=0.05 2026-09-24 07:27:41,336 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=220806.66666666666, ans=0.1 2026-09-24 07:27:42,707 INFO [train.py:1192] (0/2) Epoch 70, batch 150, loss[loss=0.2069, simple_loss=0.3198, pruned_loss=0.04702, over 24249.00 frames. ], tot_loss[loss=0.2572, simple_loss=0.3762, pruned_loss=0.06904, over 2557675.21 frames. ], batch size: 125, lr: 3.02e-03, grad_scale: 32.0 2026-09-24 07:27:44,616 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.678e+02 3.486e+02 3.795e+02 4.140e+02 5.567e+02, threshold=7.590e+02, percent-clipped=0.0 2026-09-24 07:28:02,776 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=220973.33333333334, ans=0.125 2026-09-24 07:28:08,732 INFO [train.py:1192] (0/2) Epoch 70, batch 200, loss[loss=0.2829, simple_loss=0.3922, pruned_loss=0.08679, over 21002.00 frames. ], tot_loss[loss=0.255, simple_loss=0.3745, pruned_loss=0.06773, over 3053596.18 frames. ], batch size: 333, lr: 3.02e-03, grad_scale: 32.0 2026-09-24 07:28:21,449 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=221073.33333333334, ans=0.125 2026-09-24 07:28:27,129 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=221106.66666666666, ans=0.125 2026-09-24 07:28:34,549 INFO [train.py:1192] (0/2) Epoch 70, batch 250, loss[loss=0.2744, simple_loss=0.403, pruned_loss=0.07291, over 24312.00 frames. ], tot_loss[loss=0.2536, simple_loss=0.3732, pruned_loss=0.06704, over 3443040.16 frames. ], batch size: 234, lr: 3.02e-03, grad_scale: 32.0 2026-09-24 07:28:36,014 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.660e+02 3.395e+02 3.896e+02 4.437e+02 8.047e+02, threshold=7.793e+02, percent-clipped=1.0 2026-09-24 07:28:40,592 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=221206.66666666666, ans=0.0 2026-09-24 07:28:59,605 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=221340.0, ans=0.0 2026-09-24 07:29:00,012 INFO [train.py:1192] (0/2) Epoch 70, batch 300, loss[loss=0.25, simple_loss=0.3772, pruned_loss=0.06139, over 24508.00 frames. ], tot_loss[loss=0.2533, simple_loss=0.3721, pruned_loss=0.06721, over 3748010.37 frames. ], batch size: 204, lr: 3.02e-03, grad_scale: 32.0 2026-09-24 07:29:05,666 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=221373.33333333334, ans=0.125 2026-09-24 07:29:06,143 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:29:24,530 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten.whitening_limit, batch_count=221473.33333333334, ans=15.0 2026-09-24 07:29:26,352 INFO [train.py:1192] (0/2) Epoch 70, batch 350, loss[loss=0.2005, simple_loss=0.3176, pruned_loss=0.04174, over 24556.00 frames. ], tot_loss[loss=0.2543, simple_loss=0.3736, pruned_loss=0.06749, over 3991478.13 frames. ], batch size: 137, lr: 3.02e-03, grad_scale: 32.0 2026-09-24 07:29:27,079 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.13 vs. limit=15.0 2026-09-24 07:29:27,842 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.642e+02 3.393e+02 3.774e+02 4.172e+02 6.242e+02, threshold=7.549e+02, percent-clipped=0.0 2026-09-24 07:29:39,068 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=221573.33333333334, ans=0.125 2026-09-24 07:29:51,881 INFO [train.py:1192] (0/2) Epoch 70, batch 400, loss[loss=0.2432, simple_loss=0.3629, pruned_loss=0.06178, over 24564.00 frames. ], tot_loss[loss=0.2544, simple_loss=0.3732, pruned_loss=0.0678, over 4178732.53 frames. ], batch size: 170, lr: 3.01e-03, grad_scale: 32.0 2026-09-24 07:29:56,464 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.17 vs. limit=10.0 2026-09-24 07:30:03,743 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=221740.0, ans=0.1 2026-09-24 07:30:09,475 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=221773.33333333334, ans=0.125 2026-09-24 07:30:16,521 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=221806.66666666666, ans=0.0 2026-09-24 07:30:18,062 INFO [train.py:1192] (0/2) Epoch 70, batch 450, loss[loss=0.2736, simple_loss=0.3953, pruned_loss=0.07592, over 24619.00 frames. ], tot_loss[loss=0.2548, simple_loss=0.3737, pruned_loss=0.06798, over 4311230.97 frames. ], batch size: 175, lr: 3.01e-03, grad_scale: 32.0 2026-09-24 07:30:19,415 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.680e+02 3.391e+02 3.831e+02 4.554e+02 6.756e+02, threshold=7.662e+02, percent-clipped=0.0 2026-09-24 07:30:21,777 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=221840.0, ans=0.0 2026-09-24 07:30:34,177 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=221940.0, ans=0.0 2026-09-24 07:30:39,978 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=221973.33333333334, ans=0.0 2026-09-24 07:30:43,624 INFO [train.py:1192] (0/2) Epoch 70, batch 500, loss[loss=0.2637, simple_loss=0.3932, pruned_loss=0.06711, over 24525.00 frames. ], tot_loss[loss=0.2538, simple_loss=0.3724, pruned_loss=0.06754, over 4429087.45 frames. ], batch size: 218, lr: 3.01e-03, grad_scale: 32.0 2026-09-24 07:30:46,347 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=5.62 vs. limit=15.0 2026-09-24 07:30:48,733 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=222040.0, ans=0.025 2026-09-24 07:30:51,675 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=222040.0, ans=0.2 2026-09-24 07:30:53,143 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=222040.0, ans=0.125 2026-09-24 07:31:03,406 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=222106.66666666666, ans=0.5 2026-09-24 07:31:09,730 INFO [train.py:1192] (0/2) Epoch 70, batch 550, loss[loss=0.2954, simple_loss=0.4213, pruned_loss=0.08479, over 24229.00 frames. ], tot_loss[loss=0.2545, simple_loss=0.3731, pruned_loss=0.06791, over 4518143.48 frames. ], batch size: 257, lr: 3.01e-03, grad_scale: 32.0 2026-09-24 07:31:11,276 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.566e+02 3.420e+02 3.675e+02 4.165e+02 5.906e+02, threshold=7.350e+02, percent-clipped=0.0 2026-09-24 07:31:12,079 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.30 vs. limit=22.5 2026-09-24 07:31:24,217 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=192, metric=4.24 vs. limit=15.0 2026-09-24 07:31:33,316 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=222306.66666666666, ans=0.125 2026-09-24 07:31:35,476 INFO [train.py:1192] (0/2) Epoch 70, batch 600, loss[loss=0.2647, simple_loss=0.3938, pruned_loss=0.06778, over 24371.00 frames. ], tot_loss[loss=0.2548, simple_loss=0.3737, pruned_loss=0.06796, over 4584380.33 frames. ], batch size: 234, lr: 3.01e-03, grad_scale: 32.0 2026-09-24 07:31:35,570 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.max_positive, batch_count=222340.0, ans=0.95 2026-09-24 07:31:49,416 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=222406.66666666666, ans=0.2 2026-09-24 07:31:50,761 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=222440.0, ans=0.125 2026-09-24 07:31:56,660 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=222473.33333333334, ans=0.04949747468305833 2026-09-24 07:31:58,673 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.65 vs. limit=15.0 2026-09-24 07:32:00,848 INFO [train.py:1192] (0/2) Epoch 70, batch 650, loss[loss=0.2679, simple_loss=0.3829, pruned_loss=0.07646, over 24576.00 frames. ], tot_loss[loss=0.2541, simple_loss=0.3731, pruned_loss=0.06755, over 4649863.34 frames. ], batch size: 162, lr: 3.01e-03, grad_scale: 32.0 2026-09-24 07:32:02,769 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.728e+02 3.370e+02 3.663e+02 4.136e+02 7.034e+02, threshold=7.327e+02, percent-clipped=0.0 2026-09-24 07:32:11,297 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=222573.33333333334, ans=0.0 2026-09-24 07:32:24,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=222640.0, ans=0.125 2026-09-24 07:32:26,816 INFO [train.py:1192] (0/2) Epoch 70, batch 700, loss[loss=0.2374, simple_loss=0.3535, pruned_loss=0.06069, over 24585.00 frames. ], tot_loss[loss=0.2546, simple_loss=0.3738, pruned_loss=0.06771, over 4685093.58 frames. ], batch size: 154, lr: 3.01e-03, grad_scale: 32.0 2026-09-24 07:32:38,466 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=222740.0, ans=0.125 2026-09-24 07:32:53,108 INFO [train.py:1192] (0/2) Epoch 70, batch 750, loss[loss=0.2652, simple_loss=0.3869, pruned_loss=0.07179, over 24560.00 frames. ], tot_loss[loss=0.2541, simple_loss=0.3728, pruned_loss=0.06772, over 4712260.98 frames. ], batch size: 170, lr: 3.01e-03, grad_scale: 32.0 2026-09-24 07:32:54,669 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.865e+02 3.519e+02 3.933e+02 4.610e+02 7.367e+02, threshold=7.866e+02, percent-clipped=1.0 2026-09-24 07:33:10,320 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=222940.0, ans=0.125 2026-09-24 07:33:15,278 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=222973.33333333334, ans=0.0 2026-09-24 07:33:19,270 INFO [train.py:1192] (0/2) Epoch 70, batch 800, loss[loss=0.2166, simple_loss=0.3329, pruned_loss=0.05009, over 24531.00 frames. ], tot_loss[loss=0.2543, simple_loss=0.3727, pruned_loss=0.0679, over 4740841.73 frames. ], batch size: 137, lr: 3.00e-03, grad_scale: 32.0 2026-09-24 07:33:26,746 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=223040.0, ans=0.0 2026-09-24 07:33:44,865 INFO [train.py:1192] (0/2) Epoch 70, batch 850, loss[loss=0.275, simple_loss=0.3981, pruned_loss=0.07592, over 24543.00 frames. ], tot_loss[loss=0.2537, simple_loss=0.3724, pruned_loss=0.06746, over 4764558.63 frames. ], batch size: 204, lr: 3.00e-03, grad_scale: 32.0 2026-09-24 07:33:44,969 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=223173.33333333334, ans=0.125 2026-09-24 07:33:46,671 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.632e+02 3.514e+02 3.965e+02 4.665e+02 6.328e+02, threshold=7.930e+02, percent-clipped=0.0 2026-09-24 07:33:53,736 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=223206.66666666666, ans=15.0 2026-09-24 07:33:55,485 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=223240.0, ans=0.0 2026-09-24 07:33:57,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=223240.0, ans=0.09899494936611666 2026-09-24 07:33:59,203 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.48 vs. limit=12.0 2026-09-24 07:34:01,169 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:34:04,612 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=223273.33333333334, ans=0.0 2026-09-24 07:34:05,060 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=223273.33333333334, ans=0.125 2026-09-24 07:34:07,079 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.prob, batch_count=223306.66666666666, ans=0.125 2026-09-24 07:34:09,585 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=223306.66666666666, ans=0.025 2026-09-24 07:34:11,226 INFO [train.py:1192] (0/2) Epoch 70, batch 900, loss[loss=0.2187, simple_loss=0.3358, pruned_loss=0.05078, over 24535.00 frames. ], tot_loss[loss=0.2542, simple_loss=0.373, pruned_loss=0.0677, over 4776823.47 frames. ], batch size: 137, lr: 3.00e-03, grad_scale: 32.0 2026-09-24 07:34:17,052 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.36 vs. limit=15.0 2026-09-24 07:34:24,261 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=223406.66666666666, ans=0.125 2026-09-24 07:34:29,537 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=223440.0, ans=0.125 2026-09-24 07:34:30,499 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=223440.0, ans=0.125 2026-09-24 07:34:36,433 INFO [train.py:1192] (0/2) Epoch 70, batch 950, loss[loss=0.3367, simple_loss=0.4179, pruned_loss=0.1277, over 11250.00 frames. ], tot_loss[loss=0.254, simple_loss=0.3715, pruned_loss=0.06825, over 4707624.06 frames. ], batch size: 333, lr: 3.00e-03, grad_scale: 32.0 2026-09-24 07:34:37,455 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer_ff2.min_abs, batch_count=223506.66666666666, ans=0.1 2026-09-24 07:34:37,871 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.739e+02 3.481e+02 3.831e+02 4.616e+02 8.227e+02, threshold=7.662e+02, percent-clipped=1.0 2026-09-24 07:34:40,654 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-70.pt 2026-09-24 07:34:47,726 INFO [train.py:1192] (0/2) Epoch 71, batch 0, loss[loss=0.2194, simple_loss=0.3416, pruned_loss=0.04861, over 24561.00 frames. ], tot_loss[loss=0.2194, simple_loss=0.3416, pruned_loss=0.04861, over 24561.00 frames. ], batch size: 137, lr: 2.98e-03, grad_scale: 32.0 2026-09-24 07:34:47,727 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 07:34:52,107 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.9467, 2.3078, 3.1143, 1.6781], device='cuda:0') 2026-09-24 07:34:57,177 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([1.9137, 3.1022, 2.9690, 2.6781], device='cuda:0') 2026-09-24 07:34:59,540 INFO [train.py:1224] (0/2) Epoch 71, validation: loss=0.1691, simple_loss=0.2875, pruned_loss=0.02536, over 2564189.00 frames. 2026-09-24 07:34:59,540 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 07:35:01,503 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=223533.33333333334, ans=0.025 2026-09-24 07:35:08,680 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.97 vs. limit=15.0 2026-09-24 07:35:09,299 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.63 vs. limit=8.0 2026-09-24 07:35:14,545 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=223633.33333333334, ans=0.125 2026-09-24 07:35:16,801 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.99 vs. limit=15.0 2026-09-24 07:35:24,955 INFO [train.py:1192] (0/2) Epoch 71, batch 50, loss[loss=0.2312, simple_loss=0.3363, pruned_loss=0.063, over 24275.00 frames. ], tot_loss[loss=0.2584, simple_loss=0.377, pruned_loss=0.06995, over 1075683.17 frames. ], batch size: 125, lr: 2.98e-03, grad_scale: 32.0 2026-09-24 07:35:25,504 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=223700.0, ans=0.125 2026-09-24 07:35:31,927 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=223733.33333333334, ans=0.0 2026-09-24 07:35:43,935 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.attention_skip_rate, batch_count=223800.0, ans=0.0 2026-09-24 07:35:48,598 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.629e+02 3.501e+02 3.879e+02 4.453e+02 6.125e+02, threshold=7.759e+02, percent-clipped=0.0 2026-09-24 07:35:51,056 INFO [train.py:1192] (0/2) Epoch 71, batch 100, loss[loss=0.2596, simple_loss=0.3718, pruned_loss=0.07374, over 24596.00 frames. ], tot_loss[loss=0.2608, simple_loss=0.3804, pruned_loss=0.07059, over 1903997.65 frames. ], batch size: 154, lr: 2.98e-03, grad_scale: 32.0 2026-09-24 07:35:54,697 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.41 vs. limit=15.0 2026-09-24 07:35:56,005 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:35:56,665 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.30 vs. limit=10.0 2026-09-24 07:36:06,972 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:36:09,612 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=223966.66666666666, ans=0.125 2026-09-24 07:36:16,649 INFO [train.py:1192] (0/2) Epoch 71, batch 150, loss[loss=0.2383, simple_loss=0.3449, pruned_loss=0.06583, over 24298.00 frames. ], tot_loss[loss=0.2564, simple_loss=0.3759, pruned_loss=0.06841, over 2556349.00 frames. ], batch size: 125, lr: 2.98e-03, grad_scale: 32.0 2026-09-24 07:36:22,472 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.prob, batch_count=224066.66666666666, ans=0.125 2026-09-24 07:36:30,605 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=224100.0, ans=0.125 2026-09-24 07:36:34,296 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=224133.33333333334, ans=0.04949747468305833 2026-09-24 07:36:36,483 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=13.52 vs. limit=22.5 2026-09-24 07:36:37,656 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.max_abs, batch_count=224166.66666666666, ans=10.0 2026-09-24 07:36:40,097 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.787e+02 3.394e+02 3.776e+02 4.264e+02 6.298e+02, threshold=7.553e+02, percent-clipped=0.0 2026-09-24 07:36:41,099 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=224166.66666666666, ans=0.125 2026-09-24 07:36:42,471 INFO [train.py:1192] (0/2) Epoch 71, batch 200, loss[loss=0.2909, simple_loss=0.3988, pruned_loss=0.09154, over 21038.00 frames. ], tot_loss[loss=0.2543, simple_loss=0.3743, pruned_loss=0.06715, over 3054013.31 frames. ], batch size: 333, lr: 2.98e-03, grad_scale: 32.0 2026-09-24 07:36:47,977 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=224233.33333333334, ans=0.125 2026-09-24 07:37:03,713 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=224333.33333333334, ans=0.0 2026-09-24 07:37:05,605 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=4.41 vs. limit=15.0 2026-09-24 07:37:09,169 INFO [train.py:1192] (0/2) Epoch 71, batch 250, loss[loss=0.2812, simple_loss=0.4091, pruned_loss=0.07663, over 24412.00 frames. ], tot_loss[loss=0.2539, simple_loss=0.3735, pruned_loss=0.06713, over 3442870.43 frames. ], batch size: 235, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:37:15,374 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=224400.0, ans=0.0 2026-09-24 07:37:19,606 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=224433.33333333334, ans=0.125 2026-09-24 07:37:22,177 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.09 vs. limit=22.5 2026-09-24 07:37:32,153 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=224500.0, ans=0.125 2026-09-24 07:37:32,567 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.491e+02 3.620e+02 4.048e+02 4.670e+02 6.190e+02, threshold=8.096e+02, percent-clipped=0.0 2026-09-24 07:37:35,403 INFO [train.py:1192] (0/2) Epoch 71, batch 300, loss[loss=0.2489, simple_loss=0.3768, pruned_loss=0.0605, over 24537.00 frames. ], tot_loss[loss=0.2531, simple_loss=0.3724, pruned_loss=0.06692, over 3747524.59 frames. ], batch size: 204, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:37:36,886 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=224533.33333333334, ans=0.1 2026-09-24 07:37:46,856 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=224600.0, ans=0.125 2026-09-24 07:38:01,653 INFO [train.py:1192] (0/2) Epoch 71, batch 350, loss[loss=0.2166, simple_loss=0.3296, pruned_loss=0.05179, over 24569.00 frames. ], tot_loss[loss=0.2549, simple_loss=0.3741, pruned_loss=0.06783, over 3992181.55 frames. ], batch size: 137, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:38:12,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=224766.66666666666, ans=0.1 2026-09-24 07:38:15,878 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=224766.66666666666, ans=0.0 2026-09-24 07:38:16,399 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.57 vs. limit=15.0 2026-09-24 07:38:22,485 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=224833.33333333334, ans=0.125 2026-09-24 07:38:25,052 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.748e+02 3.403e+02 3.758e+02 4.281e+02 6.376e+02, threshold=7.516e+02, percent-clipped=0.0 2026-09-24 07:38:27,490 INFO [train.py:1192] (0/2) Epoch 71, batch 400, loss[loss=0.2506, simple_loss=0.3666, pruned_loss=0.06728, over 24577.00 frames. ], tot_loss[loss=0.254, simple_loss=0.3731, pruned_loss=0.0674, over 4175680.19 frames. ], batch size: 170, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:38:28,901 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=224866.66666666666, ans=0.125 2026-09-24 07:38:35,131 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=224900.0, ans=0.1 2026-09-24 07:38:38,017 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.88 vs. limit=10.0 2026-09-24 07:38:38,712 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=224933.33333333334, ans=0.0 2026-09-24 07:38:38,724 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=224933.33333333334, ans=0.05 2026-09-24 07:38:47,623 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=225000.0, ans=0.1 2026-09-24 07:38:52,918 INFO [train.py:1192] (0/2) Epoch 71, batch 450, loss[loss=0.2919, simple_loss=0.4053, pruned_loss=0.08928, over 24640.00 frames. ], tot_loss[loss=0.255, simple_loss=0.3738, pruned_loss=0.06811, over 4310474.59 frames. ], batch size: 175, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:38:58,630 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff2_skip_rate, batch_count=225066.66666666666, ans=0.0 2026-09-24 07:39:02,456 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn1.whiten.whitening_limit, batch_count=225066.66666666666, ans=22.5 2026-09-24 07:39:09,249 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=225133.33333333334, ans=0.0 2026-09-24 07:39:13,386 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.98 vs. limit=10.0 2026-09-24 07:39:14,246 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=225166.66666666666, ans=0.125 2026-09-24 07:39:15,146 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=225166.66666666666, ans=0.0 2026-09-24 07:39:15,518 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.538e+02 3.379e+02 3.840e+02 4.380e+02 6.758e+02, threshold=7.679e+02, percent-clipped=0.0 2026-09-24 07:39:18,491 INFO [train.py:1192] (0/2) Epoch 71, batch 500, loss[loss=0.2697, simple_loss=0.4005, pruned_loss=0.06943, over 24514.00 frames. ], tot_loss[loss=0.2532, simple_loss=0.372, pruned_loss=0.06722, over 4428645.32 frames. ], batch size: 218, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:39:45,110 INFO [train.py:1192] (0/2) Epoch 71, batch 550, loss[loss=0.2535, simple_loss=0.3866, pruned_loss=0.06022, over 24214.00 frames. ], tot_loss[loss=0.2539, simple_loss=0.3729, pruned_loss=0.06747, over 4518395.76 frames. ], batch size: 257, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:39:46,870 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.92 vs. limit=12.0 2026-09-24 07:39:50,899 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.35 vs. limit=15.0 2026-09-24 07:40:04,221 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer2.prob, batch_count=225466.66666666666, ans=0.125 2026-09-24 07:40:08,166 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.593e+02 3.323e+02 3.699e+02 4.232e+02 6.947e+02, threshold=7.397e+02, percent-clipped=0.0 2026-09-24 07:40:10,453 INFO [train.py:1192] (0/2) Epoch 71, batch 600, loss[loss=0.2872, simple_loss=0.4147, pruned_loss=0.07981, over 24324.00 frames. ], tot_loss[loss=0.2543, simple_loss=0.3732, pruned_loss=0.0677, over 4584556.90 frames. ], batch size: 234, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:40:20,025 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=225600.0, ans=0.2 2026-09-24 07:40:23,170 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=225600.0, ans=0.025 2026-09-24 07:40:35,626 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=225700.0, ans=0.1 2026-09-24 07:40:35,657 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=10.92 vs. limit=12.0 2026-09-24 07:40:36,019 INFO [train.py:1192] (0/2) Epoch 71, batch 650, loss[loss=0.2693, simple_loss=0.3848, pruned_loss=0.07692, over 24558.00 frames. ], tot_loss[loss=0.2539, simple_loss=0.3727, pruned_loss=0.06753, over 4650042.51 frames. ], batch size: 162, lr: 2.97e-03, grad_scale: 32.0 2026-09-24 07:40:38,032 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=12.52 vs. limit=22.5 2026-09-24 07:40:39,323 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.skip_rate, batch_count=225700.0, ans=0.04949747468305833 2026-09-24 07:40:40,268 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer1.prob, batch_count=225700.0, ans=0.125 2026-09-24 07:40:43,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=225733.33333333334, ans=0.09899494936611666 2026-09-24 07:40:47,923 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=225766.66666666666, ans=0.1 2026-09-24 07:40:47,942 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=225766.66666666666, ans=0.125 2026-09-24 07:40:49,431 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=225766.66666666666, ans=0.125 2026-09-24 07:40:51,259 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=225800.0, ans=0.0 2026-09-24 07:40:53,265 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=225800.0, ans=0.07 2026-09-24 07:40:56,391 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.max_abs, batch_count=225833.33333333334, ans=10.0 2026-09-24 07:40:59,064 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=225833.33333333334, ans=0.1 2026-09-24 07:40:59,967 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.716e+02 3.479e+02 3.828e+02 4.312e+02 7.119e+02, threshold=7.656e+02, percent-clipped=0.0 2026-09-24 07:41:01,784 INFO [train.py:1192] (0/2) Epoch 71, batch 700, loss[loss=0.2361, simple_loss=0.3555, pruned_loss=0.05834, over 24554.00 frames. ], tot_loss[loss=0.2548, simple_loss=0.3737, pruned_loss=0.06793, over 4682279.92 frames. ], batch size: 154, lr: 2.96e-03, grad_scale: 32.0 2026-09-24 07:41:17,737 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.max_abs, batch_count=225966.66666666666, ans=10.0 2026-09-24 07:41:20,797 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=9.49 vs. limit=15.0 2026-09-24 07:41:21,525 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=225966.66666666666, ans=0.125 2026-09-24 07:41:26,821 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.10 vs. limit=15.0 2026-09-24 07:41:27,539 INFO [train.py:1192] (0/2) Epoch 71, batch 750, loss[loss=0.2472, simple_loss=0.3733, pruned_loss=0.06057, over 24571.00 frames. ], tot_loss[loss=0.2539, simple_loss=0.3726, pruned_loss=0.06765, over 4713209.97 frames. ], batch size: 170, lr: 2.96e-03, grad_scale: 32.0 2026-09-24 07:41:40,114 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=226100.0, ans=0.0 2026-09-24 07:41:48,191 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.47 vs. limit=15.0 2026-09-24 07:41:49,010 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=226166.66666666666, ans=0.1 2026-09-24 07:41:51,394 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.692e+02 3.469e+02 3.978e+02 4.612e+02 7.251e+02, threshold=7.957e+02, percent-clipped=0.0 2026-09-24 07:41:53,392 INFO [train.py:1192] (0/2) Epoch 71, batch 800, loss[loss=0.2246, simple_loss=0.3375, pruned_loss=0.0558, over 24560.00 frames. ], tot_loss[loss=0.2536, simple_loss=0.3723, pruned_loss=0.0675, over 4738277.14 frames. ], batch size: 137, lr: 2.96e-03, grad_scale: 32.0 2026-09-24 07:41:53,931 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=226200.0, ans=0.05 2026-09-24 07:42:00,053 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer2.prob, batch_count=226233.33333333334, ans=0.125 2026-09-24 07:42:04,695 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.37 vs. limit=15.0 2026-09-24 07:42:05,147 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=226266.66666666666, ans=0.125 2026-09-24 07:42:16,486 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:42:19,353 INFO [train.py:1192] (0/2) Epoch 71, batch 850, loss[loss=0.2671, simple_loss=0.3931, pruned_loss=0.07053, over 24612.00 frames. ], tot_loss[loss=0.2536, simple_loss=0.3724, pruned_loss=0.0674, over 4760332.84 frames. ], batch size: 198, lr: 2.96e-03, grad_scale: 32.0 2026-09-24 07:42:21,346 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=226366.66666666666, ans=0.0 2026-09-24 07:42:34,600 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=1.91 vs. limit=6.0 2026-09-24 07:42:42,964 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.466e+02 3.586e+02 3.961e+02 4.460e+02 5.915e+02, threshold=7.923e+02, percent-clipped=0.0 2026-09-24 07:42:44,813 INFO [train.py:1192] (0/2) Epoch 71, batch 900, loss[loss=0.2019, simple_loss=0.3237, pruned_loss=0.04003, over 24553.00 frames. ], tot_loss[loss=0.254, simple_loss=0.3727, pruned_loss=0.06764, over 4773530.85 frames. ], batch size: 137, lr: 2.96e-03, grad_scale: 32.0 2026-09-24 07:42:48,625 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff2_skip_rate, batch_count=226533.33333333334, ans=0.0 2026-09-24 07:43:00,978 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=9.21 vs. limit=15.0 2026-09-24 07:43:03,871 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=226633.33333333334, ans=0.1 2026-09-24 07:43:04,255 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-68000.pt 2026-09-24 07:43:10,065 INFO [train.py:1192] (0/2) Epoch 71, batch 950, loss[loss=0.3456, simple_loss=0.4178, pruned_loss=0.1367, over 11719.00 frames. ], tot_loss[loss=0.2538, simple_loss=0.3712, pruned_loss=0.06816, over 4711331.25 frames. ], batch size: 333, lr: 2.96e-03, grad_scale: 32.0 2026-09-24 07:43:14,489 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-71.pt 2026-09-24 07:43:19,962 INFO [train.py:1192] (0/2) Epoch 72, batch 0, loss[loss=0.2107, simple_loss=0.3285, pruned_loss=0.04644, over 24570.00 frames. ], tot_loss[loss=0.2107, simple_loss=0.3285, pruned_loss=0.04644, over 24570.00 frames. ], batch size: 137, lr: 2.94e-03, grad_scale: 32.0 2026-09-24 07:43:19,962 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 07:43:31,728 INFO [train.py:1224] (0/2) Epoch 72, validation: loss=0.169, simple_loss=0.2872, pruned_loss=0.02538, over 2564189.00 frames. 2026-09-24 07:43:31,729 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 07:43:44,462 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=226793.33333333334, ans=0.125 2026-09-24 07:43:51,425 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.743e+02 3.499e+02 3.913e+02 4.779e+02 6.698e+02, threshold=7.827e+02, percent-clipped=0.0 2026-09-24 07:43:52,992 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=226860.0, ans=0.125 2026-09-24 07:43:57,810 INFO [train.py:1192] (0/2) Epoch 72, batch 50, loss[loss=0.2052, simple_loss=0.3188, pruned_loss=0.04578, over 24342.00 frames. ], tot_loss[loss=0.2608, simple_loss=0.3787, pruned_loss=0.07146, over 1075572.50 frames. ], batch size: 125, lr: 2.94e-03, grad_scale: 32.0 2026-09-24 07:44:02,617 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.75 vs. limit=22.5 2026-09-24 07:44:09,101 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=226960.0, ans=10.0 2026-09-24 07:44:23,436 INFO [train.py:1192] (0/2) Epoch 72, batch 100, loss[loss=0.2599, simple_loss=0.3709, pruned_loss=0.07447, over 24622.00 frames. ], tot_loss[loss=0.2639, simple_loss=0.3822, pruned_loss=0.07282, over 1904091.76 frames. ], batch size: 154, lr: 2.94e-03, grad_scale: 32.0 2026-09-24 07:44:24,098 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=8.88 vs. limit=22.5 2026-09-24 07:44:24,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=227060.0, ans=0.0 2026-09-24 07:44:27,920 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=227093.33333333334, ans=0.025 2026-09-24 07:44:43,112 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.867e+02 3.517e+02 3.844e+02 4.529e+02 6.217e+02, threshold=7.689e+02, percent-clipped=0.0 2026-09-24 07:44:43,206 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.prob, batch_count=227160.0, ans=0.125 2026-09-24 07:44:44,485 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=227193.33333333334, ans=0.125 2026-09-24 07:44:49,366 INFO [train.py:1192] (0/2) Epoch 72, batch 150, loss[loss=0.1972, simple_loss=0.3161, pruned_loss=0.0391, over 24276.00 frames. ], tot_loss[loss=0.2584, simple_loss=0.3768, pruned_loss=0.06998, over 2557673.33 frames. ], batch size: 125, lr: 2.94e-03, grad_scale: 32.0 2026-09-24 07:44:57,336 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=227260.0, ans=0.0 2026-09-24 07:44:57,350 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=227260.0, ans=0.2 2026-09-24 07:45:00,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=227293.33333333334, ans=0.025 2026-09-24 07:45:15,647 INFO [train.py:1192] (0/2) Epoch 72, batch 200, loss[loss=0.2825, simple_loss=0.3944, pruned_loss=0.08529, over 21017.00 frames. ], tot_loss[loss=0.2571, simple_loss=0.3759, pruned_loss=0.06914, over 3053655.65 frames. ], batch size: 333, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:45:23,772 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=227426.66666666666, ans=0.125 2026-09-24 07:45:29,064 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.const_attention_rate, batch_count=227460.0, ans=0.025 2026-09-24 07:45:30,969 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=227493.33333333334, ans=0.125 2026-09-24 07:45:35,043 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.856e+02 3.499e+02 3.903e+02 4.417e+02 8.811e+02, threshold=7.805e+02, percent-clipped=1.0 2026-09-24 07:45:41,419 INFO [train.py:1192] (0/2) Epoch 72, batch 250, loss[loss=0.2532, simple_loss=0.386, pruned_loss=0.06022, over 24315.00 frames. ], tot_loss[loss=0.2551, simple_loss=0.3742, pruned_loss=0.06805, over 3442684.97 frames. ], batch size: 234, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:45:48,524 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=227593.33333333334, ans=0.0 2026-09-24 07:45:53,239 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer1.prob, batch_count=227626.66666666666, ans=0.125 2026-09-24 07:45:58,783 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.out_combiner.scale_min, batch_count=227660.0, ans=0.2 2026-09-24 07:46:00,573 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.96 vs. limit=6.0 2026-09-24 07:46:03,869 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=227693.33333333334, ans=0.0 2026-09-24 07:46:06,545 INFO [train.py:1192] (0/2) Epoch 72, batch 300, loss[loss=0.2799, simple_loss=0.4103, pruned_loss=0.07479, over 24548.00 frames. ], tot_loss[loss=0.254, simple_loss=0.3731, pruned_loss=0.06748, over 3747225.71 frames. ], batch size: 204, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:46:07,969 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=227726.66666666666, ans=0.0 2026-09-24 07:46:26,608 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.613e+02 3.439e+02 3.792e+02 4.297e+02 6.948e+02, threshold=7.583e+02, percent-clipped=0.0 2026-09-24 07:46:26,855 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.37 vs. limit=15.0 2026-09-24 07:46:33,069 INFO [train.py:1192] (0/2) Epoch 72, batch 350, loss[loss=0.2203, simple_loss=0.3345, pruned_loss=0.05304, over 24593.00 frames. ], tot_loss[loss=0.255, simple_loss=0.3739, pruned_loss=0.068, over 3990677.39 frames. ], batch size: 137, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:46:35,173 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=227893.33333333334, ans=0.07 2026-09-24 07:46:41,217 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff3_skip_rate, batch_count=227926.66666666666, ans=0.0 2026-09-24 07:46:49,137 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.97 vs. limit=8.0 2026-09-24 07:46:59,148 INFO [train.py:1192] (0/2) Epoch 72, batch 400, loss[loss=0.247, simple_loss=0.3687, pruned_loss=0.06271, over 24551.00 frames. ], tot_loss[loss=0.2541, simple_loss=0.3729, pruned_loss=0.06769, over 4173865.68 frames. ], batch size: 170, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:47:08,219 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=228093.33333333334, ans=0.0 2026-09-24 07:47:18,230 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.698e+02 3.303e+02 3.749e+02 4.406e+02 6.679e+02, threshold=7.497e+02, percent-clipped=0.0 2026-09-24 07:47:20,068 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=228193.33333333334, ans=0.125 2026-09-24 07:47:24,747 INFO [train.py:1192] (0/2) Epoch 72, batch 450, loss[loss=0.2582, simple_loss=0.382, pruned_loss=0.06718, over 24618.00 frames. ], tot_loss[loss=0.2548, simple_loss=0.3736, pruned_loss=0.06802, over 4306334.86 frames. ], batch size: 175, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:47:25,476 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=10.24 vs. limit=12.0 2026-09-24 07:47:27,324 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=228226.66666666666, ans=0.0 2026-09-24 07:47:40,647 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=9.31 vs. limit=15.0 2026-09-24 07:47:50,212 INFO [train.py:1192] (0/2) Epoch 72, batch 500, loss[loss=0.2716, simple_loss=0.401, pruned_loss=0.07113, over 24504.00 frames. ], tot_loss[loss=0.2538, simple_loss=0.3721, pruned_loss=0.06768, over 4425006.26 frames. ], batch size: 218, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:47:54,931 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=228426.66666666666, ans=0.125 2026-09-24 07:47:57,276 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=228426.66666666666, ans=0.2 2026-09-24 07:48:10,152 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.588e+02 3.380e+02 3.944e+02 4.318e+02 5.448e+02, threshold=7.889e+02, percent-clipped=0.0 2026-09-24 07:48:10,748 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=228526.66666666666, ans=0.2 2026-09-24 07:48:16,316 INFO [train.py:1192] (0/2) Epoch 72, batch 550, loss[loss=0.2696, simple_loss=0.3978, pruned_loss=0.07073, over 24256.00 frames. ], tot_loss[loss=0.2544, simple_loss=0.3729, pruned_loss=0.06794, over 4515813.75 frames. ], batch size: 257, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:48:16,422 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=228560.0, ans=0.125 2026-09-24 07:48:16,943 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=228560.0, ans=0.125 2026-09-24 07:48:20,743 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=228560.0, ans=0.125 2026-09-24 07:48:25,646 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:48:35,256 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=228660.0, ans=0.025 2026-09-24 07:48:42,328 INFO [train.py:1192] (0/2) Epoch 72, batch 600, loss[loss=0.2681, simple_loss=0.4005, pruned_loss=0.06791, over 24423.00 frames. ], tot_loss[loss=0.2551, simple_loss=0.3737, pruned_loss=0.06824, over 4582978.30 frames. ], batch size: 235, lr: 2.93e-03, grad_scale: 32.0 2026-09-24 07:49:01,645 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.785e+02 3.329e+02 3.709e+02 4.336e+02 7.314e+02, threshold=7.418e+02, percent-clipped=0.0 2026-09-24 07:49:07,598 INFO [train.py:1192] (0/2) Epoch 72, batch 650, loss[loss=0.2412, simple_loss=0.361, pruned_loss=0.06076, over 24569.00 frames. ], tot_loss[loss=0.253, simple_loss=0.3721, pruned_loss=0.06692, over 4648930.05 frames. ], batch size: 162, lr: 2.92e-03, grad_scale: 32.0 2026-09-24 07:49:10,205 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=228893.33333333334, ans=0.0 2026-09-24 07:49:15,796 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=228926.66666666666, ans=0.125 2026-09-24 07:49:18,414 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.84 vs. limit=15.0 2026-09-24 07:49:22,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=228960.0, ans=0.0 2026-09-24 07:49:26,448 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=228993.33333333334, ans=0.07 2026-09-24 07:49:31,789 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=229026.66666666666, ans=0.0 2026-09-24 07:49:33,606 INFO [train.py:1192] (0/2) Epoch 72, batch 700, loss[loss=0.2618, simple_loss=0.3738, pruned_loss=0.07491, over 24566.00 frames. ], tot_loss[loss=0.2532, simple_loss=0.3728, pruned_loss=0.06679, over 4682318.01 frames. ], batch size: 154, lr: 2.92e-03, grad_scale: 32.0 2026-09-24 07:49:33,807 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=4.56 vs. limit=12.0 2026-09-24 07:49:50,637 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=229160.0, ans=0.125 2026-09-24 07:49:53,195 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.792e+02 3.430e+02 3.933e+02 4.419e+02 6.993e+02, threshold=7.865e+02, percent-clipped=0.0 2026-09-24 07:49:56,028 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=229193.33333333334, ans=0.125 2026-09-24 07:49:59,601 INFO [train.py:1192] (0/2) Epoch 72, batch 750, loss[loss=0.2739, simple_loss=0.3887, pruned_loss=0.07952, over 24571.00 frames. ], tot_loss[loss=0.2525, simple_loss=0.3719, pruned_loss=0.06659, over 4709365.45 frames. ], batch size: 170, lr: 2.92e-03, grad_scale: 32.0 2026-09-24 07:50:12,759 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=229293.33333333334, ans=0.125 2026-09-24 07:50:23,121 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=4.56 vs. limit=15.0 2026-09-24 07:50:25,266 INFO [train.py:1192] (0/2) Epoch 72, batch 800, loss[loss=0.205, simple_loss=0.3272, pruned_loss=0.04142, over 24545.00 frames. ], tot_loss[loss=0.252, simple_loss=0.3714, pruned_loss=0.06631, over 4735557.00 frames. ], batch size: 137, lr: 2.92e-03, grad_scale: 32.0 2026-09-24 07:50:39,629 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=229460.0, ans=0.125 2026-09-24 07:50:42,419 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=229493.33333333334, ans=0.025 2026-09-24 07:50:44,704 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.710e+02 3.454e+02 3.890e+02 4.486e+02 6.628e+02, threshold=7.780e+02, percent-clipped=0.0 2026-09-24 07:50:44,812 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.scale_min, batch_count=229493.33333333334, ans=0.2 2026-09-24 07:50:49,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=229526.66666666666, ans=0.125 2026-09-24 07:50:51,052 INFO [train.py:1192] (0/2) Epoch 72, batch 850, loss[loss=0.2657, simple_loss=0.3945, pruned_loss=0.06844, over 24608.00 frames. ], tot_loss[loss=0.2514, simple_loss=0.371, pruned_loss=0.06591, over 4758620.69 frames. ], batch size: 198, lr: 2.92e-03, grad_scale: 32.0 2026-09-24 07:50:55,971 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=229593.33333333334, ans=0.1 2026-09-24 07:51:03,721 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=11.85 vs. limit=12.0 2026-09-24 07:51:14,961 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=229693.33333333334, ans=0.0 2026-09-24 07:51:15,413 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer2.min_positive, batch_count=229693.33333333334, ans=0.05 2026-09-24 07:51:16,282 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=229726.66666666666, ans=0.125 2026-09-24 07:51:16,641 INFO [train.py:1192] (0/2) Epoch 72, batch 900, loss[loss=0.2135, simple_loss=0.334, pruned_loss=0.04651, over 24557.00 frames. ], tot_loss[loss=0.252, simple_loss=0.3714, pruned_loss=0.06627, over 4772324.61 frames. ], batch size: 137, lr: 2.92e-03, grad_scale: 32.0 2026-09-24 07:51:27,386 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=229793.33333333334, ans=0.0 2026-09-24 07:51:28,755 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.2.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 07:51:36,062 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.684e+02 3.390e+02 3.920e+02 4.670e+02 6.022e+02, threshold=7.840e+02, percent-clipped=0.0 2026-09-24 07:51:40,631 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=229860.0, ans=0.0 2026-09-24 07:51:42,075 INFO [train.py:1192] (0/2) Epoch 72, batch 950, loss[loss=0.322, simple_loss=0.4052, pruned_loss=0.1194, over 11475.00 frames. ], tot_loss[loss=0.2534, simple_loss=0.3712, pruned_loss=0.06782, over 4713614.31 frames. ], batch size: 333, lr: 2.92e-03, grad_scale: 16.0 2026-09-24 07:51:42,163 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=229893.33333333334, ans=0.1 2026-09-24 07:51:46,451 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-72.pt 2026-09-24 07:51:52,270 INFO [train.py:1192] (0/2) Epoch 73, batch 0, loss[loss=0.1986, simple_loss=0.3207, pruned_loss=0.03826, over 24559.00 frames. ], tot_loss[loss=0.1986, simple_loss=0.3207, pruned_loss=0.03826, over 24559.00 frames. ], batch size: 137, lr: 2.90e-03, grad_scale: 32.0 2026-09-24 07:51:52,270 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 07:52:04,060 INFO [train.py:1224] (0/2) Epoch 73, validation: loss=0.1694, simple_loss=0.2874, pruned_loss=0.0257, over 2564189.00 frames. 2026-09-24 07:52:04,060 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 07:52:09,313 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=229953.33333333334, ans=0.125 2026-09-24 07:52:11,617 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=229953.33333333334, ans=0.125 2026-09-24 07:52:15,652 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=229986.66666666666, ans=0.0 2026-09-24 07:52:20,762 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.40 vs. limit=15.0 2026-09-24 07:52:22,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=230020.0, ans=10.0 2026-09-24 07:52:29,779 INFO [train.py:1192] (0/2) Epoch 73, batch 50, loss[loss=0.2306, simple_loss=0.339, pruned_loss=0.06112, over 24243.00 frames. ], tot_loss[loss=0.259, simple_loss=0.3774, pruned_loss=0.07028, over 1076510.97 frames. ], batch size: 125, lr: 2.90e-03, grad_scale: 32.0 2026-09-24 07:52:34,947 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=230120.0, ans=0.125 2026-09-24 07:52:45,190 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.833e+02 3.477e+02 3.944e+02 4.589e+02 6.689e+02, threshold=7.889e+02, percent-clipped=0.0 2026-09-24 07:52:47,788 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass_mid.scale_min, batch_count=230186.66666666666, ans=0.2 2026-09-24 07:52:54,225 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=230220.0, ans=0.0 2026-09-24 07:52:55,009 INFO [train.py:1192] (0/2) Epoch 73, batch 100, loss[loss=0.232, simple_loss=0.352, pruned_loss=0.05599, over 24612.00 frames. ], tot_loss[loss=0.2593, simple_loss=0.3795, pruned_loss=0.0696, over 1904630.50 frames. ], batch size: 154, lr: 2.90e-03, grad_scale: 32.0 2026-09-24 07:53:08,760 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=230320.0, ans=0.125 2026-09-24 07:53:21,073 INFO [train.py:1192] (0/2) Epoch 73, batch 150, loss[loss=0.2097, simple_loss=0.3238, pruned_loss=0.04783, over 24268.00 frames. ], tot_loss[loss=0.2553, simple_loss=0.3753, pruned_loss=0.06769, over 2557812.44 frames. ], batch size: 125, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:53:36,846 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.546e+02 3.340e+02 3.719e+02 4.243e+02 6.082e+02, threshold=7.438e+02, percent-clipped=0.0 2026-09-24 07:53:37,459 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=230520.0, ans=0.125 2026-09-24 07:53:44,220 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=230553.33333333334, ans=0.1 2026-09-24 07:53:45,895 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=230553.33333333334, ans=0.125 2026-09-24 07:53:46,857 INFO [train.py:1192] (0/2) Epoch 73, batch 200, loss[loss=0.2984, simple_loss=0.4056, pruned_loss=0.09559, over 21170.00 frames. ], tot_loss[loss=0.2527, simple_loss=0.3728, pruned_loss=0.06631, over 3055059.54 frames. ], batch size: 333, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:54:05,466 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=230686.66666666666, ans=0.0 2026-09-24 07:54:12,661 INFO [train.py:1192] (0/2) Epoch 73, batch 250, loss[loss=0.3036, simple_loss=0.4241, pruned_loss=0.09155, over 24321.00 frames. ], tot_loss[loss=0.2526, simple_loss=0.3725, pruned_loss=0.06632, over 3444108.22 frames. ], batch size: 234, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:54:14,668 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=230753.33333333334, ans=0.125 2026-09-24 07:54:19,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=230786.66666666666, ans=0.125 2026-09-24 07:54:23,698 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=230820.0, ans=0.1 2026-09-24 07:54:26,362 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.36 vs. limit=22.5 2026-09-24 07:54:28,512 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.785e+02 3.488e+02 4.066e+02 4.516e+02 6.561e+02, threshold=8.133e+02, percent-clipped=0.0 2026-09-24 07:54:28,738 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.05 vs. limit=10.0 2026-09-24 07:54:29,833 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.21 vs. limit=10.0 2026-09-24 07:54:35,575 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module2.whiten, num_groups=1, num_channels=192, metric=10.89 vs. limit=15.0 2026-09-24 07:54:38,696 INFO [train.py:1192] (0/2) Epoch 73, batch 300, loss[loss=0.2514, simple_loss=0.3786, pruned_loss=0.06213, over 24534.00 frames. ], tot_loss[loss=0.2534, simple_loss=0.3723, pruned_loss=0.06723, over 3748875.90 frames. ], batch size: 204, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:54:43,490 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.34 vs. limit=15.0 2026-09-24 07:54:47,176 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=230953.33333333334, ans=0.125 2026-09-24 07:54:57,017 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=231020.0, ans=0.2 2026-09-24 07:54:58,963 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=231053.33333333334, ans=0.125 2026-09-24 07:55:04,726 INFO [train.py:1192] (0/2) Epoch 73, batch 350, loss[loss=0.203, simple_loss=0.3206, pruned_loss=0.04268, over 24599.00 frames. ], tot_loss[loss=0.2547, simple_loss=0.3736, pruned_loss=0.06792, over 3992855.89 frames. ], batch size: 137, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:55:04,851 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=231086.66666666666, ans=0.0 2026-09-24 07:55:20,672 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=9.94 vs. limit=12.0 2026-09-24 07:55:20,945 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.837e+02 3.420e+02 3.800e+02 4.422e+02 6.792e+02, threshold=7.601e+02, percent-clipped=0.0 2026-09-24 07:55:24,040 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=231186.66666666666, ans=0.125 2026-09-24 07:55:30,544 INFO [train.py:1192] (0/2) Epoch 73, batch 400, loss[loss=0.2769, simple_loss=0.3926, pruned_loss=0.08054, over 24553.00 frames. ], tot_loss[loss=0.2545, simple_loss=0.3733, pruned_loss=0.06785, over 4180655.57 frames. ], batch size: 170, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:55:37,950 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=12.88 vs. limit=15.0 2026-09-24 07:55:38,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=231286.66666666666, ans=0.2 2026-09-24 07:55:44,334 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=231320.0, ans=0.95 2026-09-24 07:55:56,249 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=231420.0, ans=0.125 2026-09-24 07:55:56,676 INFO [train.py:1192] (0/2) Epoch 73, batch 450, loss[loss=0.2867, simple_loss=0.3955, pruned_loss=0.08901, over 24633.00 frames. ], tot_loss[loss=0.2559, simple_loss=0.3744, pruned_loss=0.06875, over 4314191.02 frames. ], batch size: 175, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:56:01,695 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.36 vs. limit=15.0 2026-09-24 07:56:10,194 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=231486.66666666666, ans=0.125 2026-09-24 07:56:12,492 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.564e+02 3.410e+02 3.808e+02 4.266e+02 5.624e+02, threshold=7.616e+02, percent-clipped=0.0 2026-09-24 07:56:22,350 INFO [train.py:1192] (0/2) Epoch 73, batch 500, loss[loss=0.2988, simple_loss=0.4131, pruned_loss=0.09223, over 24524.00 frames. ], tot_loss[loss=0.2549, simple_loss=0.3731, pruned_loss=0.06838, over 4430619.47 frames. ], batch size: 218, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:56:27,956 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn1.whiten, num_groups=1, num_channels=256, metric=11.83 vs. limit=22.5 2026-09-24 07:56:46,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=231720.0, ans=0.2 2026-09-24 07:56:48,158 INFO [train.py:1192] (0/2) Epoch 73, batch 550, loss[loss=0.246, simple_loss=0.3775, pruned_loss=0.05729, over 24343.00 frames. ], tot_loss[loss=0.2551, simple_loss=0.3735, pruned_loss=0.06832, over 4519589.85 frames. ], batch size: 257, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:56:59,381 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module1.balancer1.min_positive, batch_count=231820.0, ans=0.025 2026-09-24 07:57:02,999 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=231853.33333333334, ans=0.0 2026-09-24 07:57:03,946 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.688e+02 3.309e+02 3.713e+02 4.161e+02 6.571e+02, threshold=7.426e+02, percent-clipped=0.0 2026-09-24 07:57:13,999 INFO [train.py:1192] (0/2) Epoch 73, batch 600, loss[loss=0.2495, simple_loss=0.3801, pruned_loss=0.05944, over 24345.00 frames. ], tot_loss[loss=0.2551, simple_loss=0.3738, pruned_loss=0.06814, over 4585735.53 frames. ], batch size: 234, lr: 2.89e-03, grad_scale: 32.0 2026-09-24 07:57:16,571 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.33 vs. limit=15.0 2026-09-24 07:57:18,863 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.hidden_balancer.prob, batch_count=231953.33333333334, ans=0.125 2026-09-24 07:57:20,956 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=231953.33333333334, ans=0.1 2026-09-24 07:57:33,109 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.58 vs. limit=15.0 2026-09-24 07:57:34,489 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=232020.0, ans=0.0 2026-09-24 07:57:40,364 INFO [train.py:1192] (0/2) Epoch 73, batch 650, loss[loss=0.2581, simple_loss=0.3726, pruned_loss=0.07175, over 24560.00 frames. ], tot_loss[loss=0.2543, simple_loss=0.3731, pruned_loss=0.06778, over 4651247.02 frames. ], batch size: 162, lr: 2.88e-03, grad_scale: 32.0 2026-09-24 07:57:55,934 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module1.balancer1.prob, batch_count=232186.66666666666, ans=0.125 2026-09-24 07:57:55,975 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=232186.66666666666, ans=0.5 2026-09-24 07:57:56,635 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.960e+02 3.399e+02 3.702e+02 4.237e+02 6.090e+02, threshold=7.404e+02, percent-clipped=0.0 2026-09-24 07:58:05,961 INFO [train.py:1192] (0/2) Epoch 73, batch 700, loss[loss=0.2569, simple_loss=0.3692, pruned_loss=0.07225, over 24566.00 frames. ], tot_loss[loss=0.2546, simple_loss=0.3737, pruned_loss=0.06779, over 4683836.68 frames. ], batch size: 154, lr: 2.88e-03, grad_scale: 32.0 2026-09-24 07:58:15,139 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.nonlin_attention.balancer.prob, batch_count=232286.66666666666, ans=0.125 2026-09-24 07:58:19,971 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=8.93 vs. limit=15.0 2026-09-24 07:58:22,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=232353.33333333334, ans=0.125 2026-09-24 07:58:27,642 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=232386.66666666666, ans=0.0 2026-09-24 07:58:32,090 INFO [train.py:1192] (0/2) Epoch 73, batch 750, loss[loss=0.2613, simple_loss=0.3843, pruned_loss=0.06914, over 24559.00 frames. ], tot_loss[loss=0.2537, simple_loss=0.3726, pruned_loss=0.06745, over 4711468.40 frames. ], batch size: 170, lr: 2.88e-03, grad_scale: 32.0 2026-09-24 07:58:35,036 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=5.59 vs. limit=10.0 2026-09-24 07:58:38,510 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=232453.33333333334, ans=0.125 2026-09-24 07:58:42,888 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=232486.66666666666, ans=0.0 2026-09-24 07:58:48,339 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.899e+02 3.525e+02 4.022e+02 4.642e+02 6.227e+02, threshold=8.044e+02, percent-clipped=0.0 2026-09-24 07:58:48,563 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.41 vs. limit=6.0 2026-09-24 07:58:49,950 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=232520.0, ans=0.5 2026-09-24 07:58:50,859 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=232520.0, ans=0.1 2026-09-24 07:58:54,301 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=232553.33333333334, ans=0.125 2026-09-24 07:58:57,950 INFO [train.py:1192] (0/2) Epoch 73, batch 800, loss[loss=0.2333, simple_loss=0.342, pruned_loss=0.06227, over 24566.00 frames. ], tot_loss[loss=0.2531, simple_loss=0.3721, pruned_loss=0.0671, over 4740862.63 frames. ], batch size: 137, lr: 2.88e-03, grad_scale: 32.0 2026-09-24 07:59:01,792 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=232586.66666666666, ans=0.0 2026-09-24 07:59:17,704 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=8.38 vs. limit=15.0 2026-09-24 07:59:23,489 INFO [train.py:1192] (0/2) Epoch 73, batch 850, loss[loss=0.2688, simple_loss=0.3907, pruned_loss=0.07346, over 24566.00 frames. ], tot_loss[loss=0.2526, simple_loss=0.3717, pruned_loss=0.0667, over 4763245.50 frames. ], batch size: 204, lr: 2.88e-03, grad_scale: 32.0 2026-09-24 07:59:27,293 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=10.29 vs. limit=15.0 2026-09-24 07:59:39,302 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 3.019e+02 3.518e+02 3.918e+02 4.685e+02 6.562e+02, threshold=7.837e+02, percent-clipped=0.0 2026-09-24 07:59:48,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=232886.66666666666, ans=0.1 2026-09-24 07:59:49,227 INFO [train.py:1192] (0/2) Epoch 73, batch 900, loss[loss=0.2115, simple_loss=0.3285, pruned_loss=0.04726, over 24560.00 frames. ], tot_loss[loss=0.2526, simple_loss=0.3719, pruned_loss=0.06665, over 4775652.99 frames. ], batch size: 137, lr: 2.88e-03, grad_scale: 32.0 2026-09-24 07:59:50,785 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.ff2_skip_rate, batch_count=232920.0, ans=0.0 2026-09-24 08:00:02,410 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward2.hidden_balancer.prob, batch_count=232986.66666666666, ans=0.125 2026-09-24 08:00:04,842 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=9.559e-02 2026-09-24 08:00:06,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=233020.0, ans=10.0 2026-09-24 08:00:14,509 INFO [train.py:1192] (0/2) Epoch 73, batch 950, loss[loss=0.3435, simple_loss=0.4239, pruned_loss=0.1315, over 11373.00 frames. ], tot_loss[loss=0.2531, simple_loss=0.3709, pruned_loss=0.06763, over 4709013.06 frames. ], batch size: 333, lr: 2.88e-03, grad_scale: 16.0 2026-09-24 08:00:18,876 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-73.pt 2026-09-24 08:00:24,720 INFO [train.py:1192] (0/2) Epoch 74, batch 0, loss[loss=0.2133, simple_loss=0.3326, pruned_loss=0.04706, over 24583.00 frames. ], tot_loss[loss=0.2133, simple_loss=0.3326, pruned_loss=0.04706, over 24583.00 frames. ], batch size: 137, lr: 2.86e-03, grad_scale: 32.0 2026-09-24 08:00:24,721 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 08:00:30,434 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.3.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.8816, 2.8920, 3.3199, 2.7180, 2.5087, 3.2530, 1.9688, 2.6247], device='cuda:0') 2026-09-24 08:00:36,539 INFO [train.py:1224] (0/2) Epoch 74, validation: loss=0.1684, simple_loss=0.2869, pruned_loss=0.02499, over 2564189.00 frames. 2026-09-24 08:00:36,539 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 08:00:42,563 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=233146.66666666666, ans=0.2 2026-09-24 08:00:45,168 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=233146.66666666666, ans=0.2 2026-09-24 08:00:47,441 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.29 vs. limit=12.0 2026-09-24 08:00:48,696 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.549e+02 3.454e+02 3.895e+02 4.574e+02 7.934e+02, threshold=7.790e+02, percent-clipped=1.0 2026-09-24 08:01:01,803 INFO [train.py:1192] (0/2) Epoch 74, batch 50, loss[loss=0.2178, simple_loss=0.331, pruned_loss=0.0523, over 24261.00 frames. ], tot_loss[loss=0.2601, simple_loss=0.3781, pruned_loss=0.0711, over 1075466.87 frames. ], batch size: 125, lr: 2.86e-03, grad_scale: 32.0 2026-09-24 08:01:11,957 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer1.min_positive, batch_count=233346.66666666666, ans=0.025 2026-09-24 08:01:16,508 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=233346.66666666666, ans=0.04949747468305833 2026-09-24 08:01:23,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=233413.33333333334, ans=0.125 2026-09-24 08:01:25,927 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=3.81 vs. limit=10.0 2026-09-24 08:01:27,548 INFO [train.py:1192] (0/2) Epoch 74, batch 100, loss[loss=0.2378, simple_loss=0.3547, pruned_loss=0.06042, over 24608.00 frames. ], tot_loss[loss=0.2614, simple_loss=0.3804, pruned_loss=0.07114, over 1905076.17 frames. ], batch size: 154, lr: 2.86e-03, grad_scale: 32.0 2026-09-24 08:01:39,488 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=233513.33333333334, ans=0.2 2026-09-24 08:01:39,792 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.764e+02 3.427e+02 3.827e+02 4.293e+02 5.928e+02, threshold=7.654e+02, percent-clipped=0.0 2026-09-24 08:01:44,629 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=233546.66666666666, ans=0.1 2026-09-24 08:01:52,370 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.08 vs. limit=15.0 2026-09-24 08:01:53,104 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=233613.33333333334, ans=0.0 2026-09-24 08:01:53,443 INFO [train.py:1192] (0/2) Epoch 74, batch 150, loss[loss=0.2046, simple_loss=0.3227, pruned_loss=0.04329, over 24248.00 frames. ], tot_loss[loss=0.2566, simple_loss=0.3758, pruned_loss=0.06867, over 2558728.55 frames. ], batch size: 125, lr: 2.86e-03, grad_scale: 32.0 2026-09-24 08:01:54,477 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=233613.33333333334, ans=0.125 2026-09-24 08:01:56,503 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.29 vs. limit=6.0 2026-09-24 08:02:02,115 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=233646.66666666666, ans=0.125 2026-09-24 08:02:06,247 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=233680.0, ans=0.125 2026-09-24 08:02:18,790 INFO [train.py:1192] (0/2) Epoch 74, batch 200, loss[loss=0.2817, simple_loss=0.3894, pruned_loss=0.08701, over 20991.00 frames. ], tot_loss[loss=0.2555, simple_loss=0.3746, pruned_loss=0.06821, over 3055028.27 frames. ], batch size: 333, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:02:21,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.min_positive, batch_count=233780.0, ans=0.025 2026-09-24 08:02:23,231 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=233813.33333333334, ans=0.1 2026-09-24 08:02:30,707 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.787e+02 3.340e+02 3.805e+02 4.289e+02 6.745e+02, threshold=7.609e+02, percent-clipped=0.0 2026-09-24 08:02:31,813 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.const_attention_rate, batch_count=233846.66666666666, ans=0.025 2026-09-24 08:02:36,245 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer1.prob, batch_count=233880.0, ans=0.125 2026-09-24 08:02:41,675 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=233913.33333333334, ans=0.015 2026-09-24 08:02:43,994 INFO [train.py:1192] (0/2) Epoch 74, batch 250, loss[loss=0.266, simple_loss=0.4002, pruned_loss=0.06595, over 24303.00 frames. ], tot_loss[loss=0.2541, simple_loss=0.3732, pruned_loss=0.06752, over 3444079.93 frames. ], batch size: 234, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:02:54,052 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=234013.33333333334, ans=0.1 2026-09-24 08:02:58,725 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=234013.33333333334, ans=0.0 2026-09-24 08:03:03,078 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=234046.66666666666, ans=0.015 2026-09-24 08:03:06,078 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=234080.0, ans=0.125 2026-09-24 08:03:10,273 INFO [train.py:1192] (0/2) Epoch 74, batch 300, loss[loss=0.3035, simple_loss=0.4205, pruned_loss=0.09325, over 24564.00 frames. ], tot_loss[loss=0.253, simple_loss=0.372, pruned_loss=0.06696, over 3748736.33 frames. ], batch size: 204, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:03:17,310 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=10.33 vs. limit=15.0 2026-09-24 08:03:22,607 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.787e+02 3.564e+02 3.883e+02 4.568e+02 6.180e+02, threshold=7.765e+02, percent-clipped=0.0 2026-09-24 08:03:25,772 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=7.89 vs. limit=15.0 2026-09-24 08:03:33,970 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.min_positive, batch_count=234246.66666666666, ans=0.025 2026-09-24 08:03:35,316 INFO [train.py:1192] (0/2) Epoch 74, batch 350, loss[loss=0.2087, simple_loss=0.3226, pruned_loss=0.04737, over 24557.00 frames. ], tot_loss[loss=0.2531, simple_loss=0.3727, pruned_loss=0.06676, over 3988722.59 frames. ], batch size: 137, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:03:41,213 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.const_attention_rate, batch_count=234313.33333333334, ans=0.025 2026-09-24 08:03:42,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=234313.33333333334, ans=0.125 2026-09-24 08:03:44,814 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass_mid.scale_min, batch_count=234313.33333333334, ans=0.2 2026-09-24 08:04:01,396 INFO [train.py:1192] (0/2) Epoch 74, batch 400, loss[loss=0.2695, simple_loss=0.389, pruned_loss=0.07498, over 24565.00 frames. ], tot_loss[loss=0.2531, simple_loss=0.3725, pruned_loss=0.06685, over 4173224.93 frames. ], batch size: 170, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:04:09,378 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_skip_rate, batch_count=234480.0, ans=0.0 2026-09-24 08:04:09,954 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.whiten.whitening_limit, batch_count=234480.0, ans=12.0 2026-09-24 08:04:13,288 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.675e+02 3.316e+02 3.775e+02 4.302e+02 5.883e+02, threshold=7.551e+02, percent-clipped=0.0 2026-09-24 08:04:26,714 INFO [train.py:1192] (0/2) Epoch 74, batch 450, loss[loss=0.273, simple_loss=0.3862, pruned_loss=0.07991, over 24612.00 frames. ], tot_loss[loss=0.2532, simple_loss=0.3725, pruned_loss=0.06701, over 4307284.29 frames. ], batch size: 175, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:04:31,333 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.03 vs. limit=15.0 2026-09-24 08:04:44,634 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=234713.33333333334, ans=0.125 2026-09-24 08:04:45,641 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=234713.33333333334, ans=0.125 2026-09-24 08:04:50,353 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 08:04:50,882 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=234746.66666666666, ans=0.1 2026-09-24 08:04:53,058 INFO [train.py:1192] (0/2) Epoch 74, batch 500, loss[loss=0.2601, simple_loss=0.3926, pruned_loss=0.06379, over 24525.00 frames. ], tot_loss[loss=0.2525, simple_loss=0.3714, pruned_loss=0.06676, over 4425920.68 frames. ], batch size: 218, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:05:05,585 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.703e+02 3.488e+02 3.882e+02 4.333e+02 6.074e+02, threshold=7.764e+02, percent-clipped=0.0 2026-09-24 08:05:13,192 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer1.prob, batch_count=234880.0, ans=0.125 2026-09-24 08:05:19,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=234946.66666666666, ans=0.125 2026-09-24 08:05:19,764 INFO [train.py:1192] (0/2) Epoch 74, batch 550, loss[loss=0.2911, simple_loss=0.4159, pruned_loss=0.08313, over 24260.00 frames. ], tot_loss[loss=0.2532, simple_loss=0.3723, pruned_loss=0.06704, over 4516107.36 frames. ], batch size: 257, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:05:45,467 INFO [train.py:1192] (0/2) Epoch 74, batch 600, loss[loss=0.268, simple_loss=0.4013, pruned_loss=0.06735, over 24301.00 frames. ], tot_loss[loss=0.2537, simple_loss=0.3728, pruned_loss=0.06725, over 4582822.53 frames. ], batch size: 234, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:05:57,771 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.632e+02 3.324e+02 3.672e+02 4.258e+02 7.730e+02, threshold=7.345e+02, percent-clipped=0.0 2026-09-24 08:06:00,650 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=235213.33333333334, ans=0.1 2026-09-24 08:06:04,796 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=235213.33333333334, ans=0.0 2026-09-24 08:06:06,539 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.31 vs. limit=22.5 2026-09-24 08:06:11,285 INFO [train.py:1192] (0/2) Epoch 74, batch 650, loss[loss=0.2641, simple_loss=0.385, pruned_loss=0.07161, over 24577.00 frames. ], tot_loss[loss=0.2524, simple_loss=0.3718, pruned_loss=0.06644, over 4648764.18 frames. ], batch size: 162, lr: 2.85e-03, grad_scale: 32.0 2026-09-24 08:06:11,874 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=235280.0, ans=0.04949747468305833 2026-09-24 08:06:13,667 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=235280.0, ans=0.1 2026-09-24 08:06:21,926 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=235346.66666666666, ans=0.0 2026-09-24 08:06:25,488 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=235346.66666666666, ans=0.125 2026-09-24 08:06:27,666 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=235380.0, ans=0.1 2026-09-24 08:06:30,972 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=235380.0, ans=0.125 2026-09-24 08:06:34,750 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=12.41 vs. limit=15.0 2026-09-24 08:06:37,370 INFO [train.py:1192] (0/2) Epoch 74, batch 700, loss[loss=0.2622, simple_loss=0.3745, pruned_loss=0.07495, over 24583.00 frames. ], tot_loss[loss=0.2531, simple_loss=0.3727, pruned_loss=0.06676, over 4682312.77 frames. ], batch size: 154, lr: 2.84e-03, grad_scale: 32.0 2026-09-24 08:06:48,250 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=235513.33333333334, ans=0.0 2026-09-24 08:06:50,061 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.897e+02 3.449e+02 3.815e+02 4.155e+02 6.261e+02, threshold=7.629e+02, percent-clipped=0.0 2026-09-24 08:06:53,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass_mid.scale_min, batch_count=235546.66666666666, ans=0.2 2026-09-24 08:06:59,125 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=235580.0, ans=0.125 2026-09-24 08:07:02,036 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=235580.0, ans=0.1 2026-09-24 08:07:03,434 INFO [train.py:1192] (0/2) Epoch 74, batch 750, loss[loss=0.2401, simple_loss=0.3671, pruned_loss=0.0566, over 24555.00 frames. ], tot_loss[loss=0.252, simple_loss=0.3715, pruned_loss=0.06629, over 4710507.89 frames. ], batch size: 170, lr: 2.84e-03, grad_scale: 32.0 2026-09-24 08:07:05,926 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=235613.33333333334, ans=0.125 2026-09-24 08:07:16,383 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=235680.0, ans=0.07 2026-09-24 08:07:19,220 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=235713.33333333334, ans=0.1 2026-09-24 08:07:25,436 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=235746.66666666666, ans=0.125 2026-09-24 08:07:26,402 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer1.prob, batch_count=235746.66666666666, ans=0.125 2026-09-24 08:07:27,897 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer2.prob, batch_count=235746.66666666666, ans=0.125 2026-09-24 08:07:28,763 INFO [train.py:1192] (0/2) Epoch 74, batch 800, loss[loss=0.2129, simple_loss=0.33, pruned_loss=0.0479, over 24534.00 frames. ], tot_loss[loss=0.2513, simple_loss=0.3707, pruned_loss=0.06593, over 4735075.89 frames. ], batch size: 137, lr: 2.84e-03, grad_scale: 32.0 2026-09-24 08:07:41,909 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.661e+02 3.511e+02 3.915e+02 4.395e+02 7.015e+02, threshold=7.829e+02, percent-clipped=0.0 2026-09-24 08:07:45,814 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=235880.0, ans=0.0 2026-09-24 08:07:54,993 INFO [train.py:1192] (0/2) Epoch 74, batch 850, loss[loss=0.2482, simple_loss=0.3776, pruned_loss=0.0594, over 24585.00 frames. ], tot_loss[loss=0.2513, simple_loss=0.3708, pruned_loss=0.06588, over 4757968.07 frames. ], batch size: 198, lr: 2.84e-03, grad_scale: 32.0 2026-09-24 08:07:57,084 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=235946.66666666666, ans=0.125 2026-09-24 08:08:03,063 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=235980.0, ans=0.125 2026-09-24 08:08:04,640 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=235980.0, ans=0.0 2026-09-24 08:08:07,786 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=236013.33333333334, ans=0.1 2026-09-24 08:08:13,780 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.min_positive, batch_count=236046.66666666666, ans=0.05 2026-09-24 08:08:17,663 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=236080.0, ans=0.125 2026-09-24 08:08:21,586 INFO [train.py:1192] (0/2) Epoch 74, batch 900, loss[loss=0.2176, simple_loss=0.34, pruned_loss=0.04759, over 24565.00 frames. ], tot_loss[loss=0.2524, simple_loss=0.3719, pruned_loss=0.06646, over 4771737.28 frames. ], batch size: 137, lr: 2.84e-03, grad_scale: 32.0 2026-09-24 08:08:34,074 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=236180.0, ans=0.125 2026-09-24 08:08:34,520 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.503e+02 3.414e+02 3.841e+02 4.540e+02 6.153e+02, threshold=7.681e+02, percent-clipped=0.0 2026-09-24 08:08:36,494 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 08:08:36,539 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn1.whiten.whitening_limit, batch_count=236213.33333333334, ans=22.5 2026-09-24 08:08:41,712 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=1.400e-01 2026-09-24 08:08:46,498 INFO [train.py:1192] (0/2) Epoch 74, batch 950, loss[loss=0.3033, simple_loss=0.3812, pruned_loss=0.1127, over 11469.00 frames. ], tot_loss[loss=0.2526, simple_loss=0.3707, pruned_loss=0.06725, over 4715954.39 frames. ], batch size: 334, lr: 2.84e-03, grad_scale: 32.0 2026-09-24 08:08:47,883 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=9.13 vs. limit=15.0 2026-09-24 08:08:50,955 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-74.pt 2026-09-24 08:08:58,646 INFO [train.py:1192] (0/2) Epoch 75, batch 0, loss[loss=0.1882, simple_loss=0.3197, pruned_loss=0.02833, over 24561.00 frames. ], tot_loss[loss=0.1882, simple_loss=0.3197, pruned_loss=0.02833, over 24561.00 frames. ], batch size: 137, lr: 2.82e-03, grad_scale: 32.0 2026-09-24 08:08:58,646 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 08:09:06,910 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.0.layers.1.self_attn_weights, attn_weights_entropy = tensor([5.2770, 4.7185, 4.7192, 5.1590], device='cuda:0') 2026-09-24 08:09:10,525 INFO [train.py:1224] (0/2) Epoch 75, validation: loss=0.1675, simple_loss=0.2858, pruned_loss=0.02463, over 2564189.00 frames. 2026-09-24 08:09:10,525 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 08:09:22,372 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=236373.33333333334, ans=0.025 2026-09-24 08:09:24,295 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=236373.33333333334, ans=0.125 2026-09-24 08:09:25,620 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=236406.66666666666, ans=0.0 2026-09-24 08:09:33,488 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_abs, batch_count=236440.0, ans=0.5 2026-09-24 08:09:35,801 INFO [train.py:1192] (0/2) Epoch 75, batch 50, loss[loss=0.2092, simple_loss=0.3203, pruned_loss=0.04902, over 24334.00 frames. ], tot_loss[loss=0.258, simple_loss=0.3767, pruned_loss=0.0697, over 1075703.54 frames. ], batch size: 125, lr: 2.82e-03, grad_scale: 32.0 2026-09-24 08:09:37,225 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward3.hidden_balancer.prob, batch_count=236473.33333333334, ans=0.125 2026-09-24 08:09:42,714 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=236506.66666666666, ans=0.1 2026-09-24 08:09:42,722 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=236506.66666666666, ans=0.2 2026-09-24 08:09:42,891 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn2.whiten, num_groups=1, num_channels=192, metric=11.36 vs. limit=22.5 2026-09-24 08:09:44,422 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.860e+02 3.663e+02 4.192e+02 4.797e+02 6.594e+02, threshold=8.383e+02, percent-clipped=0.0 2026-09-24 08:10:00,838 INFO [train.py:1192] (0/2) Epoch 75, batch 100, loss[loss=0.2361, simple_loss=0.3496, pruned_loss=0.06134, over 24625.00 frames. ], tot_loss[loss=0.2606, simple_loss=0.3804, pruned_loss=0.07043, over 1904080.85 frames. ], batch size: 154, lr: 2.82e-03, grad_scale: 32.0 2026-09-24 08:10:04,639 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=236640.0, ans=0.2 2026-09-24 08:10:09,013 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn1.whiten, num_groups=1, num_channels=192, metric=10.84 vs. limit=22.5 2026-09-24 08:10:15,225 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.const_attention_rate, batch_count=236706.66666666666, ans=0.025 2026-09-24 08:10:20,831 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=236740.0, ans=0.0 2026-09-24 08:10:23,435 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=236773.33333333334, ans=0.0 2026-09-24 08:10:26,892 INFO [train.py:1192] (0/2) Epoch 75, batch 150, loss[loss=0.1927, simple_loss=0.3099, pruned_loss=0.0377, over 24259.00 frames. ], tot_loss[loss=0.2568, simple_loss=0.3763, pruned_loss=0.06869, over 2557359.60 frames. ], batch size: 125, lr: 2.82e-03, grad_scale: 32.0 2026-09-24 08:10:28,081 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=236806.66666666666, ans=0.0 2026-09-24 08:10:32,398 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten.whitening_limit, batch_count=236840.0, ans=15.0 2026-09-24 08:10:35,889 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.735e+02 3.496e+02 3.861e+02 4.306e+02 6.041e+02, threshold=7.723e+02, percent-clipped=0.0 2026-09-24 08:10:52,126 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.89 vs. limit=22.5 2026-09-24 08:10:52,130 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.whiten.whitening_limit, batch_count=236940.0, ans=12.0 2026-09-24 08:10:53,125 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.21 vs. limit=15.0 2026-09-24 08:10:53,327 INFO [train.py:1192] (0/2) Epoch 75, batch 200, loss[loss=0.2838, simple_loss=0.3929, pruned_loss=0.08738, over 21054.00 frames. ], tot_loss[loss=0.2548, simple_loss=0.3746, pruned_loss=0.06755, over 3054842.90 frames. ], batch size: 333, lr: 2.82e-03, grad_scale: 32.0 2026-09-24 08:10:55,669 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=236973.33333333334, ans=0.95 2026-09-24 08:10:56,122 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=236973.33333333334, ans=0.04949747468305833 2026-09-24 08:11:01,265 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=237006.66666666666, ans=0.0 2026-09-24 08:11:05,244 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.93 vs. limit=15.0 2026-09-24 08:11:18,734 INFO [train.py:1192] (0/2) Epoch 75, batch 250, loss[loss=0.2764, simple_loss=0.4092, pruned_loss=0.07182, over 24294.00 frames. ], tot_loss[loss=0.2519, simple_loss=0.3722, pruned_loss=0.06582, over 3444878.32 frames. ], batch size: 234, lr: 2.82e-03, grad_scale: 32.0 2026-09-24 08:11:27,134 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.670e+02 3.484e+02 3.917e+02 4.560e+02 6.379e+02, threshold=7.834e+02, percent-clipped=0.0 2026-09-24 08:11:38,954 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=237273.33333333334, ans=0.0 2026-09-24 08:11:43,649 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=237273.33333333334, ans=0.125 2026-09-24 08:11:45,042 INFO [train.py:1192] (0/2) Epoch 75, batch 300, loss[loss=0.2609, simple_loss=0.3936, pruned_loss=0.06415, over 24541.00 frames. ], tot_loss[loss=0.252, simple_loss=0.3718, pruned_loss=0.06613, over 3749170.57 frames. ], batch size: 204, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:11:51,477 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=237340.0, ans=0.1 2026-09-24 08:12:10,519 INFO [train.py:1192] (0/2) Epoch 75, batch 350, loss[loss=0.213, simple_loss=0.3268, pruned_loss=0.04954, over 24589.00 frames. ], tot_loss[loss=0.2531, simple_loss=0.3729, pruned_loss=0.06665, over 3992269.04 frames. ], batch size: 137, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:12:11,115 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=237473.33333333334, ans=0.125 2026-09-24 08:12:19,183 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.729e+02 3.346e+02 3.794e+02 4.475e+02 7.372e+02, threshold=7.587e+02, percent-clipped=0.0 2026-09-24 08:12:21,633 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module1.whiten, num_groups=1, num_channels=192, metric=7.41 vs. limit=15.0 2026-09-24 08:12:31,164 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer1.max_abs, batch_count=237606.66666666666, ans=10.0 2026-09-24 08:12:35,723 INFO [train.py:1192] (0/2) Epoch 75, batch 400, loss[loss=0.2498, simple_loss=0.3727, pruned_loss=0.06345, over 24596.00 frames. ], tot_loss[loss=0.2518, simple_loss=0.3717, pruned_loss=0.06595, over 4177970.71 frames. ], batch size: 170, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:12:41,827 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=237673.33333333334, ans=0.035 2026-09-24 08:12:45,355 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.3.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 08:12:51,361 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=237740.0, ans=0.1 2026-09-24 08:12:58,412 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=237773.33333333334, ans=0.125 2026-09-24 08:13:01,227 INFO [train.py:1192] (0/2) Epoch 75, batch 450, loss[loss=0.2816, simple_loss=0.4035, pruned_loss=0.07983, over 24649.00 frames. ], tot_loss[loss=0.2528, simple_loss=0.3724, pruned_loss=0.06663, over 4312259.48 frames. ], batch size: 175, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:13:05,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=237806.66666666666, ans=0.125 2026-09-24 08:13:10,049 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.880e+02 3.483e+02 3.872e+02 4.424e+02 7.917e+02, threshold=7.745e+02, percent-clipped=1.0 2026-09-24 08:13:18,114 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=237906.66666666666, ans=0.125 2026-09-24 08:13:22,451 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=5.60 vs. limit=12.0 2026-09-24 08:13:22,689 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=237940.0, ans=0.0 2026-09-24 08:13:27,557 INFO [train.py:1192] (0/2) Epoch 75, batch 500, loss[loss=0.2893, simple_loss=0.414, pruned_loss=0.08229, over 24503.00 frames. ], tot_loss[loss=0.2522, simple_loss=0.3714, pruned_loss=0.06645, over 4428872.07 frames. ], batch size: 218, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:13:46,049 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.73 vs. limit=8.0 2026-09-24 08:13:53,904 INFO [train.py:1192] (0/2) Epoch 75, batch 550, loss[loss=0.2868, simple_loss=0.4094, pruned_loss=0.08211, over 24257.00 frames. ], tot_loss[loss=0.2532, simple_loss=0.3723, pruned_loss=0.06706, over 4518116.58 frames. ], batch size: 257, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:13:57,311 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=238140.0, ans=0.5 2026-09-24 08:14:01,682 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=238173.33333333334, ans=0.025 2026-09-24 08:14:02,108 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.737e+02 3.424e+02 3.830e+02 4.349e+02 6.866e+02, threshold=7.661e+02, percent-clipped=0.0 2026-09-24 08:14:08,631 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.04 vs. limit=22.5 2026-09-24 08:14:12,701 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=238240.0, ans=0.0 2026-09-24 08:14:14,635 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=238273.33333333334, ans=0.0 2026-09-24 08:14:19,010 INFO [train.py:1192] (0/2) Epoch 75, batch 600, loss[loss=0.2814, simple_loss=0.4097, pruned_loss=0.07657, over 24289.00 frames. ], tot_loss[loss=0.253, simple_loss=0.3725, pruned_loss=0.0668, over 4586685.28 frames. ], batch size: 234, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:14:33,041 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=238373.33333333334, ans=0.125 2026-09-24 08:14:39,797 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=238440.0, ans=0.125 2026-09-24 08:14:43,902 INFO [train.py:1192] (0/2) Epoch 75, batch 650, loss[loss=0.2381, simple_loss=0.3632, pruned_loss=0.05652, over 24563.00 frames. ], tot_loss[loss=0.251, simple_loss=0.3709, pruned_loss=0.06555, over 4651897.68 frames. ], batch size: 162, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:14:45,785 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=238473.33333333334, ans=0.0 2026-09-24 08:14:52,140 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.765e+02 3.291e+02 3.723e+02 4.199e+02 5.918e+02, threshold=7.446e+02, percent-clipped=0.0 2026-09-24 08:14:53,385 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=238540.0, ans=0.0 2026-09-24 08:15:00,979 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.nonlin_attention.balancer.prob, batch_count=238573.33333333334, ans=0.125 2026-09-24 08:15:03,532 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=6.50 vs. limit=15.0 2026-09-24 08:15:07,098 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=238606.66666666666, ans=0.125 2026-09-24 08:15:08,966 INFO [train.py:1192] (0/2) Epoch 75, batch 700, loss[loss=0.2464, simple_loss=0.3594, pruned_loss=0.06668, over 24595.00 frames. ], tot_loss[loss=0.2516, simple_loss=0.3716, pruned_loss=0.06581, over 4684128.85 frames. ], batch size: 154, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:15:19,936 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=238706.66666666666, ans=0.2 2026-09-24 08:15:25,058 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=7.86 vs. limit=15.0 2026-09-24 08:15:25,831 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=238740.0, ans=0.2 2026-09-24 08:15:27,493 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.92 vs. limit=6.0 2026-09-24 08:15:34,278 INFO [train.py:1192] (0/2) Epoch 75, batch 750, loss[loss=0.2521, simple_loss=0.3789, pruned_loss=0.06264, over 24560.00 frames. ], tot_loss[loss=0.2505, simple_loss=0.3702, pruned_loss=0.0654, over 4710777.04 frames. ], batch size: 170, lr: 2.81e-03, grad_scale: 32.0 2026-09-24 08:15:35,881 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.min_positive, batch_count=238806.66666666666, ans=0.05 2026-09-24 08:15:35,882 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=238806.66666666666, ans=0.0 2026-09-24 08:15:42,469 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=238840.0, ans=0.0 2026-09-24 08:15:43,332 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.908e+02 3.604e+02 4.002e+02 4.730e+02 6.100e+02, threshold=8.003e+02, percent-clipped=0.0 2026-09-24 08:15:44,401 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer2.prob, batch_count=238873.33333333334, ans=0.125 2026-09-24 08:15:56,666 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=238940.0, ans=0.125 2026-09-24 08:15:57,129 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=238940.0, ans=0.1 2026-09-24 08:15:58,019 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=238940.0, ans=0.2 2026-09-24 08:15:59,903 INFO [train.py:1192] (0/2) Epoch 75, batch 800, loss[loss=0.2244, simple_loss=0.3357, pruned_loss=0.05658, over 24554.00 frames. ], tot_loss[loss=0.2506, simple_loss=0.3702, pruned_loss=0.06549, over 4735592.73 frames. ], batch size: 137, lr: 2.80e-03, grad_scale: 32.0 2026-09-24 08:16:19,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=239106.66666666666, ans=0.0 2026-09-24 08:16:21,332 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.out_whiten.whitening_limit, batch_count=239106.66666666666, ans=8.0 2026-09-24 08:16:24,966 INFO [train.py:1192] (0/2) Epoch 75, batch 850, loss[loss=0.2844, simple_loss=0.3979, pruned_loss=0.08539, over 24581.00 frames. ], tot_loss[loss=0.2504, simple_loss=0.37, pruned_loss=0.06543, over 4758635.56 frames. ], batch size: 198, lr: 2.80e-03, grad_scale: 32.0 2026-09-24 08:16:25,090 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 08:16:27,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=239140.0, ans=0.125 2026-09-24 08:16:27,482 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=239140.0, ans=0.1 2026-09-24 08:16:33,718 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.830e+02 3.516e+02 3.911e+02 4.413e+02 6.230e+02, threshold=7.822e+02, percent-clipped=0.0 2026-09-24 08:16:36,183 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=239206.66666666666, ans=0.0 2026-09-24 08:16:36,737 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=9.08 vs. limit=15.0 2026-09-24 08:16:38,137 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.25 vs. limit=22.5 2026-09-24 08:16:46,358 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=239273.33333333334, ans=0.0 2026-09-24 08:16:47,248 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=239273.33333333334, ans=0.1 2026-09-24 08:16:49,647 INFO [train.py:1192] (0/2) Epoch 75, batch 900, loss[loss=0.2365, simple_loss=0.352, pruned_loss=0.06047, over 24554.00 frames. ], tot_loss[loss=0.2506, simple_loss=0.3701, pruned_loss=0.06551, over 4772367.33 frames. ], batch size: 137, lr: 2.80e-03, grad_scale: 32.0 2026-09-24 08:16:59,924 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=239373.33333333334, ans=0.1 2026-09-24 08:17:01,382 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=239373.33333333334, ans=0.2 2026-09-24 08:17:14,525 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=239473.33333333334, ans=0.125 2026-09-24 08:17:14,642 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=17.74 vs. limit=15.0 2026-09-24 08:17:14,895 INFO [train.py:1192] (0/2) Epoch 75, batch 950, loss[loss=0.3194, simple_loss=0.4026, pruned_loss=0.1181, over 11135.00 frames. ], tot_loss[loss=0.2508, simple_loss=0.3691, pruned_loss=0.06627, over 4713206.90 frames. ], batch size: 333, lr: 2.80e-03, grad_scale: 32.0 2026-09-24 08:17:17,490 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=239473.33333333334, ans=0.125 2026-09-24 08:17:19,172 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-75.pt 2026-09-24 08:17:26,061 INFO [train.py:1192] (0/2) Epoch 76, batch 0, loss[loss=0.2141, simple_loss=0.3363, pruned_loss=0.04595, over 24557.00 frames. ], tot_loss[loss=0.2141, simple_loss=0.3363, pruned_loss=0.04595, over 24557.00 frames. ], batch size: 137, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:17:26,061 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 08:17:37,694 INFO [train.py:1224] (0/2) Epoch 76, validation: loss=0.1691, simple_loss=0.2872, pruned_loss=0.0255, over 2564189.00 frames. 2026-09-24 08:17:37,695 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 08:17:37,786 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=239500.0, ans=0.1 2026-09-24 08:17:42,601 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.621e+02 3.523e+02 3.917e+02 4.585e+02 8.203e+02, threshold=7.834e+02, percent-clipped=2.0 2026-09-24 08:17:42,721 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=239533.33333333334, ans=0.1 2026-09-24 08:17:48,638 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass_mid.scale_min, batch_count=239566.66666666666, ans=0.2 2026-09-24 08:18:03,691 INFO [train.py:1192] (0/2) Epoch 76, batch 50, loss[loss=0.2087, simple_loss=0.3207, pruned_loss=0.04836, over 24326.00 frames. ], tot_loss[loss=0.2565, simple_loss=0.3752, pruned_loss=0.06894, over 1075211.03 frames. ], batch size: 125, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:18:06,303 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=239666.66666666666, ans=0.0 2026-09-24 08:18:09,862 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=239700.0, ans=0.125 2026-09-24 08:18:29,758 INFO [train.py:1192] (0/2) Epoch 76, batch 100, loss[loss=0.2296, simple_loss=0.346, pruned_loss=0.05655, over 24615.00 frames. ], tot_loss[loss=0.26, simple_loss=0.3794, pruned_loss=0.07029, over 1903574.95 frames. ], batch size: 154, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:18:34,250 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.830e+02 3.495e+02 3.949e+02 4.420e+02 6.131e+02, threshold=7.897e+02, percent-clipped=0.0 2026-09-24 08:18:35,222 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=239866.66666666666, ans=0.125 2026-09-24 08:18:41,836 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.prob, batch_count=239900.0, ans=0.125 2026-09-24 08:18:51,169 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.attention_skip_rate, batch_count=239966.66666666666, ans=0.0 2026-09-24 08:18:53,622 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=239966.66666666666, ans=0.125 2026-09-24 08:18:54,580 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-72000.pt 2026-09-24 08:18:55,256 INFO [train.py:1192] (0/2) Epoch 76, batch 150, loss[loss=0.2249, simple_loss=0.331, pruned_loss=0.0594, over 24246.00 frames. ], tot_loss[loss=0.2546, simple_loss=0.3744, pruned_loss=0.0674, over 2557637.34 frames. ], batch size: 125, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:18:55,920 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=240000.0, ans=0.2 2026-09-24 08:18:59,826 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass_mid.scale_min, batch_count=240033.33333333334, ans=0.2 2026-09-24 08:19:08,783 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=240066.66666666666, ans=0.025 2026-09-24 08:19:15,717 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=7.11 vs. limit=15.0 2026-09-24 08:19:16,556 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=240133.33333333334, ans=0.0 2026-09-24 08:19:21,032 INFO [train.py:1192] (0/2) Epoch 76, batch 200, loss[loss=0.2561, simple_loss=0.3738, pruned_loss=0.06915, over 21019.00 frames. ], tot_loss[loss=0.2524, simple_loss=0.3724, pruned_loss=0.06616, over 3054236.52 frames. ], batch size: 333, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:19:25,646 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.638e+02 3.457e+02 3.825e+02 4.494e+02 1.295e+03, threshold=7.650e+02, percent-clipped=1.0 2026-09-24 08:19:27,614 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=240200.0, ans=0.0 2026-09-24 08:19:30,389 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=240200.0, ans=0.125 2026-09-24 08:19:31,781 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=240233.33333333334, ans=0.1 2026-09-24 08:19:38,888 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=240266.66666666666, ans=0.1 2026-09-24 08:19:40,559 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=240266.66666666666, ans=0.2 2026-09-24 08:19:46,106 INFO [train.py:1192] (0/2) Epoch 76, batch 250, loss[loss=0.2791, simple_loss=0.4079, pruned_loss=0.07513, over 24310.00 frames. ], tot_loss[loss=0.2525, simple_loss=0.3724, pruned_loss=0.06628, over 3442878.34 frames. ], batch size: 234, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:20:00,158 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=240400.0, ans=0.125 2026-09-24 08:20:02,721 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.attention_skip_rate, batch_count=240433.33333333334, ans=0.0 2026-09-24 08:20:12,605 INFO [train.py:1192] (0/2) Epoch 76, batch 300, loss[loss=0.2434, simple_loss=0.3774, pruned_loss=0.05475, over 24552.00 frames. ], tot_loss[loss=0.2519, simple_loss=0.3715, pruned_loss=0.06617, over 3749740.14 frames. ], batch size: 204, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:20:14,271 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.69 vs. limit=15.0 2026-09-24 08:20:16,401 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=240500.0, ans=0.125 2026-09-24 08:20:16,837 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.720e+02 3.464e+02 3.928e+02 4.450e+02 6.560e+02, threshold=7.856e+02, percent-clipped=0.0 2026-09-24 08:20:18,646 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=240533.33333333334, ans=0.125 2026-09-24 08:20:21,646 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=240533.33333333334, ans=0.1 2026-09-24 08:20:22,130 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=240566.66666666666, ans=0.1 2026-09-24 08:20:30,877 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=240600.0, ans=0.0 2026-09-24 08:20:32,251 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=11.22 vs. limit=22.5 2026-09-24 08:20:32,653 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=240633.33333333334, ans=0.0 2026-09-24 08:20:35,058 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer2.prob, batch_count=240633.33333333334, ans=0.125 2026-09-24 08:20:37,710 INFO [train.py:1192] (0/2) Epoch 76, batch 350, loss[loss=0.2201, simple_loss=0.3344, pruned_loss=0.05289, over 24575.00 frames. ], tot_loss[loss=0.2522, simple_loss=0.3721, pruned_loss=0.06616, over 3995038.40 frames. ], batch size: 137, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:20:37,975 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.44 vs. limit=15.0 2026-09-24 08:20:41,536 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer2.prob, batch_count=240666.66666666666, ans=0.125 2026-09-24 08:20:51,646 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=240733.33333333334, ans=0.0 2026-09-24 08:20:56,492 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=240766.66666666666, ans=0.1 2026-09-24 08:21:02,984 INFO [train.py:1192] (0/2) Epoch 76, batch 400, loss[loss=0.2541, simple_loss=0.3737, pruned_loss=0.06723, over 24574.00 frames. ], tot_loss[loss=0.2517, simple_loss=0.3713, pruned_loss=0.06601, over 4178549.59 frames. ], batch size: 170, lr: 2.78e-03, grad_scale: 32.0 2026-09-24 08:21:07,148 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.763e+02 3.331e+02 3.627e+02 4.142e+02 8.048e+02, threshold=7.254e+02, percent-clipped=1.0 2026-09-24 08:21:14,410 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.22 vs. limit=15.0 2026-09-24 08:21:25,968 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.prob, batch_count=240966.66666666666, ans=0.125 2026-09-24 08:21:27,871 INFO [train.py:1192] (0/2) Epoch 76, batch 450, loss[loss=0.2536, simple_loss=0.3823, pruned_loss=0.06247, over 24639.00 frames. ], tot_loss[loss=0.252, simple_loss=0.3717, pruned_loss=0.06615, over 4310801.77 frames. ], batch size: 175, lr: 2.77e-03, grad_scale: 32.0 2026-09-24 08:21:29,798 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=5.91 vs. limit=15.0 2026-09-24 08:21:37,926 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.balancer1.prob, batch_count=241066.66666666666, ans=0.125 2026-09-24 08:21:48,317 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=241133.33333333334, ans=0.07 2026-09-24 08:21:49,400 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=9.83 vs. limit=15.0 2026-09-24 08:21:52,774 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=241166.66666666666, ans=0.125 2026-09-24 08:21:53,138 INFO [train.py:1192] (0/2) Epoch 76, batch 500, loss[loss=0.2754, simple_loss=0.3982, pruned_loss=0.07633, over 24472.00 frames. ], tot_loss[loss=0.2516, simple_loss=0.3708, pruned_loss=0.06617, over 4428157.92 frames. ], batch size: 218, lr: 2.77e-03, grad_scale: 16.0 2026-09-24 08:21:53,365 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=7.13 vs. limit=15.0 2026-09-24 08:21:53,684 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer2.prob, batch_count=241166.66666666666, ans=0.125 2026-09-24 08:21:58,110 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.491e+02 3.246e+02 3.684e+02 4.227e+02 7.383e+02, threshold=7.368e+02, percent-clipped=1.0 2026-09-24 08:21:58,229 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=241200.0, ans=0.0 2026-09-24 08:22:01,706 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer1.prob, batch_count=241200.0, ans=0.125 2026-09-24 08:22:01,745 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=241200.0, ans=0.125 2026-09-24 08:22:06,238 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module1.balancer1.prob, batch_count=241233.33333333334, ans=0.125 2026-09-24 08:22:08,803 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=3.57 vs. limit=15.0 2026-09-24 08:22:09,195 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=241266.66666666666, ans=0.1 2026-09-24 08:22:18,732 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=241333.33333333334, ans=0.125 2026-09-24 08:22:19,051 INFO [train.py:1192] (0/2) Epoch 76, batch 550, loss[loss=0.2832, simple_loss=0.4065, pruned_loss=0.07993, over 24315.00 frames. ], tot_loss[loss=0.2516, simple_loss=0.3711, pruned_loss=0.06608, over 4517555.06 frames. ], batch size: 257, lr: 2.77e-03, grad_scale: 16.0 2026-09-24 08:22:41,044 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.19 vs. limit=15.0 2026-09-24 08:22:45,483 INFO [train.py:1192] (0/2) Epoch 76, batch 600, loss[loss=0.2824, simple_loss=0.4121, pruned_loss=0.0764, over 24302.00 frames. ], tot_loss[loss=0.2523, simple_loss=0.3719, pruned_loss=0.06636, over 4583133.84 frames. ], batch size: 234, lr: 2.77e-03, grad_scale: 16.0 2026-09-24 08:22:46,584 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.bypass.scale_min, batch_count=241500.0, ans=0.2 2026-09-24 08:22:50,183 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=241533.33333333334, ans=0.025 2026-09-24 08:22:50,574 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.795e+02 3.387e+02 3.819e+02 4.436e+02 6.290e+02, threshold=7.639e+02, percent-clipped=0.0 2026-09-24 08:22:57,637 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.79 vs. limit=15.0 2026-09-24 08:23:04,344 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.72 vs. limit=15.0 2026-09-24 08:23:10,495 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.ff3_skip_rate, batch_count=241633.33333333334, ans=0.0 2026-09-24 08:23:11,331 INFO [train.py:1192] (0/2) Epoch 76, batch 650, loss[loss=0.2482, simple_loss=0.3676, pruned_loss=0.06437, over 24566.00 frames. ], tot_loss[loss=0.252, simple_loss=0.3715, pruned_loss=0.06626, over 4649077.47 frames. ], batch size: 162, lr: 2.77e-03, grad_scale: 16.0 2026-09-24 08:23:26,603 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=241766.66666666666, ans=0.0 2026-09-24 08:23:37,768 INFO [train.py:1192] (0/2) Epoch 76, batch 700, loss[loss=0.2557, simple_loss=0.3613, pruned_loss=0.07507, over 24571.00 frames. ], tot_loss[loss=0.2525, simple_loss=0.3722, pruned_loss=0.06636, over 4681389.15 frames. ], batch size: 154, lr: 2.77e-03, grad_scale: 16.0 2026-09-24 08:23:42,683 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.586e+02 3.452e+02 3.738e+02 4.196e+02 6.416e+02, threshold=7.476e+02, percent-clipped=0.0 2026-09-24 08:23:51,503 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=241900.0, ans=0.125 2026-09-24 08:24:03,537 INFO [train.py:1192] (0/2) Epoch 76, batch 750, loss[loss=0.2641, simple_loss=0.3861, pruned_loss=0.07108, over 24583.00 frames. ], tot_loss[loss=0.2523, simple_loss=0.3717, pruned_loss=0.06651, over 4712501.58 frames. ], batch size: 170, lr: 2.77e-03, grad_scale: 16.0 2026-09-24 08:24:05,812 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.out_balancer.prob, batch_count=242000.0, ans=0.125 2026-09-24 08:24:12,742 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.ff3_skip_rate, batch_count=242066.66666666666, ans=0.0 2026-09-24 08:24:21,366 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=242100.0, ans=0.125 2026-09-24 08:24:23,240 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=8.68 vs. limit=15.0 2026-09-24 08:24:24,666 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module2.balancer1.prob, batch_count=242133.33333333334, ans=0.125 2026-09-24 08:24:26,499 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.58 vs. limit=15.0 2026-09-24 08:24:28,133 INFO [train.py:1192] (0/2) Epoch 76, batch 800, loss[loss=0.2222, simple_loss=0.3376, pruned_loss=0.0534, over 24527.00 frames. ], tot_loss[loss=0.2518, simple_loss=0.3712, pruned_loss=0.06625, over 4737166.82 frames. ], batch size: 137, lr: 2.77e-03, grad_scale: 32.0 2026-09-24 08:24:32,938 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.807e+02 3.450e+02 3.951e+02 4.539e+02 6.485e+02, threshold=7.903e+02, percent-clipped=0.0 2026-09-24 08:24:46,004 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=242266.66666666666, ans=0.125 2026-09-24 08:24:48,747 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=242300.0, ans=0.125 2026-09-24 08:24:53,154 INFO [train.py:1192] (0/2) Epoch 76, batch 850, loss[loss=0.2401, simple_loss=0.3718, pruned_loss=0.0542, over 24518.00 frames. ], tot_loss[loss=0.2511, simple_loss=0.3705, pruned_loss=0.06583, over 4760079.55 frames. ], batch size: 204, lr: 2.77e-03, grad_scale: 32.0 2026-09-24 08:25:00,739 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=242366.66666666666, ans=0.125 2026-09-24 08:25:00,761 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=242366.66666666666, ans=0.125 2026-09-24 08:25:04,842 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=242400.0, ans=0.0 2026-09-24 08:25:13,704 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.attention_skip_rate, batch_count=242466.66666666666, ans=0.0 2026-09-24 08:25:16,747 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=9.79 vs. limit=12.0 2026-09-24 08:25:18,491 INFO [train.py:1192] (0/2) Epoch 76, batch 900, loss[loss=0.1974, simple_loss=0.321, pruned_loss=0.03689, over 24583.00 frames. ], tot_loss[loss=0.2511, simple_loss=0.3706, pruned_loss=0.06582, over 4774168.75 frames. ], batch size: 137, lr: 2.77e-03, grad_scale: 32.0 2026-09-24 08:25:23,806 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.726e+02 3.589e+02 4.035e+02 4.771e+02 7.613e+02, threshold=8.071e+02, percent-clipped=0.0 2026-09-24 08:25:25,434 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=242533.33333333334, ans=0.125 2026-09-24 08:25:34,432 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=242600.0, ans=0.125 2026-09-24 08:25:35,377 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.scale_min, batch_count=242600.0, ans=0.2 2026-09-24 08:25:43,465 INFO [train.py:1192] (0/2) Epoch 76, batch 950, loss[loss=0.3614, simple_loss=0.4331, pruned_loss=0.1448, over 11788.00 frames. ], tot_loss[loss=0.2514, simple_loss=0.3693, pruned_loss=0.0668, over 4713913.17 frames. ], batch size: 333, lr: 2.76e-03, grad_scale: 16.0 2026-09-24 08:25:47,675 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-76.pt 2026-09-24 08:25:52,694 INFO [train.py:1192] (0/2) Epoch 77, batch 0, loss[loss=0.2235, simple_loss=0.3379, pruned_loss=0.05452, over 24555.00 frames. ], tot_loss[loss=0.2235, simple_loss=0.3379, pruned_loss=0.05452, over 24555.00 frames. ], batch size: 137, lr: 2.75e-03, grad_scale: 32.0 2026-09-24 08:25:52,702 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 08:26:02,645 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.6563, 2.5340, 2.1676, 3.0659], device='cuda:0') 2026-09-24 08:26:04,146 INFO [train.py:1224] (0/2) Epoch 77, validation: loss=0.1678, simple_loss=0.286, pruned_loss=0.0248, over 2564189.00 frames. 2026-09-24 08:26:04,146 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 08:26:21,851 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.33 vs. limit=15.0 2026-09-24 08:26:25,181 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=242826.66666666666, ans=0.0 2026-09-24 08:26:28,230 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff3_skip_rate, batch_count=242826.66666666666, ans=0.0 2026-09-24 08:26:29,269 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=242860.0, ans=0.1 2026-09-24 08:26:29,667 INFO [train.py:1192] (0/2) Epoch 77, batch 50, loss[loss=0.225, simple_loss=0.334, pruned_loss=0.05802, over 24233.00 frames. ], tot_loss[loss=0.2581, simple_loss=0.3763, pruned_loss=0.06996, over 1075957.56 frames. ], batch size: 125, lr: 2.75e-03, grad_scale: 32.0 2026-09-24 08:26:31,055 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.642e+02 3.542e+02 4.135e+02 4.724e+02 7.671e+02, threshold=8.269e+02, percent-clipped=0.0 2026-09-24 08:26:35,166 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=242893.33333333334, ans=0.125 2026-09-24 08:26:40,419 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=242926.66666666666, ans=0.125 2026-09-24 08:26:40,423 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_module2.balancer2.min_positive, batch_count=242926.66666666666, ans=0.05 2026-09-24 08:26:48,073 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=242960.0, ans=0.125 2026-09-24 08:26:51,720 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=15.98 vs. limit=22.5 2026-09-24 08:26:54,951 INFO [train.py:1192] (0/2) Epoch 77, batch 100, loss[loss=0.2488, simple_loss=0.3647, pruned_loss=0.06643, over 24618.00 frames. ], tot_loss[loss=0.2615, simple_loss=0.3804, pruned_loss=0.07132, over 1904292.46 frames. ], batch size: 154, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:26:57,376 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=243026.66666666666, ans=0.125 2026-09-24 08:27:00,093 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=243060.0, ans=0.125 2026-09-24 08:27:14,654 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=243160.0, ans=0.0 2026-09-24 08:27:20,270 INFO [train.py:1192] (0/2) Epoch 77, batch 150, loss[loss=0.2154, simple_loss=0.3261, pruned_loss=0.05232, over 24249.00 frames. ], tot_loss[loss=0.2559, simple_loss=0.3755, pruned_loss=0.06818, over 2558001.56 frames. ], batch size: 125, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:27:21,585 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.750e+02 3.580e+02 3.897e+02 4.233e+02 6.143e+02, threshold=7.794e+02, percent-clipped=0.0 2026-09-24 08:27:23,046 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=243193.33333333334, ans=0.125 2026-09-24 08:27:24,023 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=243193.33333333334, ans=0.2 2026-09-24 08:27:24,994 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer1.prob, batch_count=243226.66666666666, ans=0.125 2026-09-24 08:27:37,151 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=243293.33333333334, ans=0.125 2026-09-24 08:27:45,599 INFO [train.py:1192] (0/2) Epoch 77, batch 200, loss[loss=0.3123, simple_loss=0.4118, pruned_loss=0.1064, over 20918.00 frames. ], tot_loss[loss=0.2528, simple_loss=0.373, pruned_loss=0.06632, over 3054284.06 frames. ], batch size: 333, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:27:48,489 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=14.09 vs. limit=15.0 2026-09-24 08:27:50,529 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=243393.33333333334, ans=0.125 2026-09-24 08:27:59,220 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=243426.66666666666, ans=0.125 2026-09-24 08:28:01,341 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.03 vs. limit=15.0 2026-09-24 08:28:06,454 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=243493.33333333334, ans=0.0 2026-09-24 08:28:06,559 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=6.36 vs. limit=15.0 2026-09-24 08:28:08,937 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.ff2_skip_rate, batch_count=243493.33333333334, ans=0.0 2026-09-24 08:28:11,392 INFO [train.py:1192] (0/2) Epoch 77, batch 250, loss[loss=0.2945, simple_loss=0.4189, pruned_loss=0.08505, over 24322.00 frames. ], tot_loss[loss=0.2526, simple_loss=0.3726, pruned_loss=0.0663, over 3443107.53 frames. ], batch size: 234, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:28:12,831 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.673e+02 3.585e+02 3.996e+02 4.594e+02 6.658e+02, threshold=7.992e+02, percent-clipped=0.0 2026-09-24 08:28:31,789 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=243660.0, ans=0.125 2026-09-24 08:28:37,016 INFO [train.py:1192] (0/2) Epoch 77, batch 300, loss[loss=0.2648, simple_loss=0.3895, pruned_loss=0.07003, over 24549.00 frames. ], tot_loss[loss=0.2518, simple_loss=0.3716, pruned_loss=0.06604, over 3748699.58 frames. ], batch size: 204, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:28:48,961 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass.scale_min, batch_count=243760.0, ans=0.2 2026-09-24 08:28:50,520 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=243760.0, ans=0.0 2026-09-24 08:29:01,015 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=243826.66666666666, ans=0.0 2026-09-24 08:29:02,461 INFO [train.py:1192] (0/2) Epoch 77, batch 350, loss[loss=0.2175, simple_loss=0.3315, pruned_loss=0.05171, over 24568.00 frames. ], tot_loss[loss=0.2517, simple_loss=0.372, pruned_loss=0.06568, over 3992105.13 frames. ], batch size: 137, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:29:03,127 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=243860.0, ans=0.0 2026-09-24 08:29:03,146 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer2.prob, batch_count=243860.0, ans=0.125 2026-09-24 08:29:03,943 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.724e+02 3.449e+02 3.803e+02 4.343e+02 6.609e+02, threshold=7.606e+02, percent-clipped=0.0 2026-09-24 08:29:06,484 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=8.98 vs. limit=15.0 2026-09-24 08:29:22,252 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff3_skip_rate, batch_count=243993.33333333334, ans=0.0 2026-09-24 08:29:26,105 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.77 vs. limit=6.0 2026-09-24 08:29:27,882 INFO [train.py:1192] (0/2) Epoch 77, batch 400, loss[loss=0.2527, simple_loss=0.3712, pruned_loss=0.06706, over 24538.00 frames. ], tot_loss[loss=0.2513, simple_loss=0.3714, pruned_loss=0.06561, over 4175658.91 frames. ], batch size: 170, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:29:31,846 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.conv_module2.whiten, num_groups=1, num_channels=192, metric=6.88 vs. limit=15.0 2026-09-24 08:29:40,608 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=244093.33333333334, ans=0.1 2026-09-24 08:29:43,095 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=244126.66666666666, ans=0.125 2026-09-24 08:29:44,098 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=10.14 vs. limit=22.5 2026-09-24 08:29:48,424 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.out_combiner.scale_min, batch_count=244160.0, ans=0.2 2026-09-24 08:29:53,397 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=244193.33333333334, ans=0.0 2026-09-24 08:29:53,822 INFO [train.py:1192] (0/2) Epoch 77, batch 450, loss[loss=0.2598, simple_loss=0.386, pruned_loss=0.06675, over 24615.00 frames. ], tot_loss[loss=0.2526, simple_loss=0.3723, pruned_loss=0.06649, over 4313622.62 frames. ], batch size: 175, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:29:54,095 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=3.37 vs. limit=12.0 2026-09-24 08:29:54,370 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=244193.33333333334, ans=0.125 2026-09-24 08:29:55,136 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.791e+02 3.381e+02 3.816e+02 4.471e+02 6.865e+02, threshold=7.633e+02, percent-clipped=0.0 2026-09-24 08:30:11,419 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=244293.33333333334, ans=0.125 2026-09-24 08:30:18,923 INFO [train.py:1192] (0/2) Epoch 77, batch 500, loss[loss=0.2847, simple_loss=0.4114, pruned_loss=0.07902, over 24495.00 frames. ], tot_loss[loss=0.252, simple_loss=0.3712, pruned_loss=0.06635, over 4430843.08 frames. ], batch size: 218, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:30:24,614 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=244393.33333333334, ans=0.125 2026-09-24 08:30:27,950 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward2.hidden_balancer.prob, batch_count=244393.33333333334, ans=0.125 2026-09-24 08:30:29,784 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module2.balancer2.prob, batch_count=244426.66666666666, ans=0.125 2026-09-24 08:30:44,991 INFO [train.py:1192] (0/2) Epoch 77, batch 550, loss[loss=0.2666, simple_loss=0.3929, pruned_loss=0.07011, over 24251.00 frames. ], tot_loss[loss=0.2516, simple_loss=0.3711, pruned_loss=0.06603, over 4519452.56 frames. ], batch size: 257, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:30:46,547 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.668e+02 3.350e+02 3.888e+02 4.304e+02 6.561e+02, threshold=7.776e+02, percent-clipped=0.0 2026-09-24 08:30:50,024 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.97 vs. limit=15.0 2026-09-24 08:30:55,360 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=244593.33333333334, ans=0.09899494936611666 2026-09-24 08:30:56,300 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=244593.33333333334, ans=0.0 2026-09-24 08:30:59,850 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=244626.66666666666, ans=0.125 2026-09-24 08:31:03,070 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=244626.66666666666, ans=0.2 2026-09-24 08:31:05,397 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=18.01 vs. limit=22.5 2026-09-24 08:31:10,906 INFO [train.py:1192] (0/2) Epoch 77, batch 600, loss[loss=0.2963, simple_loss=0.4195, pruned_loss=0.08661, over 24342.00 frames. ], tot_loss[loss=0.2519, simple_loss=0.3717, pruned_loss=0.06608, over 4583902.60 frames. ], batch size: 234, lr: 2.74e-03, grad_scale: 32.0 2026-09-24 08:31:37,035 INFO [train.py:1192] (0/2) Epoch 77, batch 650, loss[loss=0.2595, simple_loss=0.3722, pruned_loss=0.07337, over 24568.00 frames. ], tot_loss[loss=0.2516, simple_loss=0.3712, pruned_loss=0.06597, over 4649768.73 frames. ], batch size: 162, lr: 2.73e-03, grad_scale: 32.0 2026-09-24 08:31:38,307 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.482e+02 3.401e+02 3.705e+02 4.317e+02 6.289e+02, threshold=7.410e+02, percent-clipped=0.0 2026-09-24 08:31:42,083 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=244893.33333333334, ans=0.2 2026-09-24 08:32:02,497 INFO [train.py:1192] (0/2) Epoch 77, batch 700, loss[loss=0.2577, simple_loss=0.3659, pruned_loss=0.07472, over 24574.00 frames. ], tot_loss[loss=0.2518, simple_loss=0.3716, pruned_loss=0.06599, over 4683266.38 frames. ], batch size: 154, lr: 2.73e-03, grad_scale: 32.0 2026-09-24 08:32:07,808 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=245060.0, ans=0.0 2026-09-24 08:32:09,795 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_skip_rate, batch_count=245060.0, ans=0.0 2026-09-24 08:32:15,112 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=11.98 vs. limit=15.0 2026-09-24 08:32:15,780 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=7.43 vs. limit=15.0 2026-09-24 08:32:18,696 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.2.prob, batch_count=245126.66666666666, ans=0.125 2026-09-24 08:32:25,261 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.prob, batch_count=245160.0, ans=0.125 2026-09-24 08:32:25,262 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=245160.0, ans=0.035 2026-09-24 08:32:27,515 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=245193.33333333334, ans=0.1 2026-09-24 08:32:27,970 INFO [train.py:1192] (0/2) Epoch 77, batch 750, loss[loss=0.2569, simple_loss=0.3793, pruned_loss=0.06731, over 24556.00 frames. ], tot_loss[loss=0.2516, simple_loss=0.371, pruned_loss=0.06612, over 4711061.59 frames. ], batch size: 170, lr: 2.73e-03, grad_scale: 32.0 2026-09-24 08:32:29,437 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_skip_rate, batch_count=245193.33333333334, ans=0.0 2026-09-24 08:32:29,821 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 3.022e+02 3.446e+02 3.846e+02 4.397e+02 7.250e+02, threshold=7.693e+02, percent-clipped=0.0 2026-09-24 08:32:34,182 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.const_attention_rate, batch_count=245226.66666666666, ans=0.025 2026-09-24 08:32:36,104 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=245226.66666666666, ans=0.0 2026-09-24 08:32:36,132 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=245226.66666666666, ans=0.0 2026-09-24 08:32:40,371 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=245260.0, ans=0.125 2026-09-24 08:32:43,633 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=192, metric=8.01 vs. limit=15.0 2026-09-24 08:32:51,556 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.const_attention_rate, batch_count=245326.66666666666, ans=0.025 2026-09-24 08:32:53,146 INFO [train.py:1192] (0/2) Epoch 77, batch 800, loss[loss=0.2254, simple_loss=0.3378, pruned_loss=0.05652, over 24566.00 frames. ], tot_loss[loss=0.251, simple_loss=0.3706, pruned_loss=0.06575, over 4735943.56 frames. ], batch size: 137, lr: 2.73e-03, grad_scale: 32.0 2026-09-24 08:32:55,089 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=245360.0, ans=0.1 2026-09-24 08:32:57,719 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass_mid.scale_min, batch_count=245393.33333333334, ans=0.2 2026-09-24 08:32:59,286 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=245393.33333333334, ans=0.125 2026-09-24 08:33:00,394 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=8.55 vs. limit=22.5 2026-09-24 08:33:00,777 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=245393.33333333334, ans=0.0 2026-09-24 08:33:05,711 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.98 vs. limit=15.0 2026-09-24 08:33:18,010 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=245526.66666666666, ans=0.1 2026-09-24 08:33:18,387 INFO [train.py:1192] (0/2) Epoch 77, batch 850, loss[loss=0.259, simple_loss=0.3867, pruned_loss=0.06562, over 24628.00 frames. ], tot_loss[loss=0.2503, simple_loss=0.3701, pruned_loss=0.06531, over 4759107.79 frames. ], batch size: 198, lr: 2.73e-03, grad_scale: 32.0 2026-09-24 08:33:18,953 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=245526.66666666666, ans=0.0 2026-09-24 08:33:19,866 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.524e+02 3.493e+02 4.068e+02 4.856e+02 6.731e+02, threshold=8.137e+02, percent-clipped=0.0 2026-09-24 08:33:21,733 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=245526.66666666666, ans=0.1 2026-09-24 08:33:36,051 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=245626.66666666666, ans=0.2 2026-09-24 08:33:43,723 INFO [train.py:1192] (0/2) Epoch 77, batch 900, loss[loss=0.2157, simple_loss=0.3378, pruned_loss=0.04681, over 24555.00 frames. ], tot_loss[loss=0.2509, simple_loss=0.3707, pruned_loss=0.0656, over 4772716.73 frames. ], batch size: 137, lr: 2.73e-03, grad_scale: 32.0 2026-09-24 08:33:43,832 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=245693.33333333334, ans=0.125 2026-09-24 08:33:48,579 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_skip_rate, batch_count=245726.66666666666, ans=0.0 2026-09-24 08:33:58,221 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.skip_rate, batch_count=245760.0, ans=0.07 2026-09-24 08:34:00,989 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=245793.33333333334, ans=0.125 2026-09-24 08:34:07,608 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=245826.66666666666, ans=0.125 2026-09-24 08:34:09,055 INFO [train.py:1192] (0/2) Epoch 77, batch 950, loss[loss=0.2923, simple_loss=0.3726, pruned_loss=0.106, over 11329.00 frames. ], tot_loss[loss=0.2507, simple_loss=0.369, pruned_loss=0.06621, over 4717578.21 frames. ], batch size: 333, lr: 2.73e-03, grad_scale: 32.0 2026-09-24 08:34:10,525 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.720e+02 3.408e+02 3.913e+02 4.584e+02 6.615e+02, threshold=7.826e+02, percent-clipped=0.0 2026-09-24 08:34:13,460 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-77.pt 2026-09-24 08:34:20,058 INFO [train.py:1192] (0/2) Epoch 78, batch 0, loss[loss=0.1946, simple_loss=0.3204, pruned_loss=0.03434, over 24567.00 frames. ], tot_loss[loss=0.1946, simple_loss=0.3204, pruned_loss=0.03434, over 24567.00 frames. ], batch size: 137, lr: 2.71e-03, grad_scale: 32.0 2026-09-24 08:34:20,058 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 08:34:21,929 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([1.7296, 1.3890, 2.4066, 1.5326], device='cuda:0') 2026-09-24 08:34:24,386 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.2.encoder.layers.1.self_attn_weights, attn_weights_entropy = tensor([2.6277, 4.2114, 4.0646, 3.4986], device='cuda:0') 2026-09-24 08:34:26,400 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.8101, 2.1579, 2.9847, 1.4651], device='cuda:0') 2026-09-24 08:34:31,826 INFO [train.py:1224] (0/2) Epoch 78, validation: loss=0.169, simple_loss=0.2871, pruned_loss=0.02545, over 2564189.00 frames. 2026-09-24 08:34:31,892 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 08:34:56,846 INFO [train.py:1192] (0/2) Epoch 78, batch 50, loss[loss=0.2006, simple_loss=0.317, pruned_loss=0.04217, over 24250.00 frames. ], tot_loss[loss=0.2556, simple_loss=0.3747, pruned_loss=0.06822, over 1076795.74 frames. ], batch size: 125, lr: 2.71e-03, grad_scale: 32.0 2026-09-24 08:35:20,367 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.706e+02 3.564e+02 3.879e+02 4.906e+02 6.303e+02, threshold=7.759e+02, percent-clipped=0.0 2026-09-24 08:35:22,860 INFO [train.py:1192] (0/2) Epoch 78, batch 100, loss[loss=0.2558, simple_loss=0.3658, pruned_loss=0.07289, over 24604.00 frames. ], tot_loss[loss=0.2601, simple_loss=0.3798, pruned_loss=0.07022, over 1904920.00 frames. ], batch size: 154, lr: 2.71e-03, grad_scale: 32.0 2026-09-24 08:35:24,674 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer2.prob, batch_count=246220.0, ans=0.125 2026-09-24 08:35:25,725 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=246220.0, ans=0.125 2026-09-24 08:35:26,201 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_skip_rate, batch_count=246220.0, ans=0.0 2026-09-24 08:35:26,635 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=246220.0, ans=0.0 2026-09-24 08:35:31,065 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.42 vs. limit=15.0 2026-09-24 08:35:32,593 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=4.07 vs. limit=15.0 2026-09-24 08:35:33,828 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=246286.66666666666, ans=0.125 2026-09-24 08:35:35,011 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=2.92 vs. limit=6.0 2026-09-24 08:35:39,094 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=246320.0, ans=0.09899494936611666 2026-09-24 08:35:40,951 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer1.min_positive, batch_count=246320.0, ans=0.025 2026-09-24 08:35:44,811 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.13 vs. limit=15.0 2026-09-24 08:35:48,804 INFO [train.py:1192] (0/2) Epoch 78, batch 150, loss[loss=0.2061, simple_loss=0.3185, pruned_loss=0.04681, over 24247.00 frames. ], tot_loss[loss=0.2545, simple_loss=0.3742, pruned_loss=0.06739, over 2558218.39 frames. ], batch size: 125, lr: 2.71e-03, grad_scale: 16.0 2026-09-24 08:35:52,060 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.ff2_skip_rate, batch_count=246386.66666666666, ans=0.0 2026-09-24 08:35:57,770 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.conv_module2.balancer2.prob, batch_count=246420.0, ans=0.125 2026-09-24 08:36:04,327 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=246486.66666666666, ans=0.125 2026-09-24 08:36:13,174 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.731e+02 3.462e+02 3.795e+02 4.434e+02 6.400e+02, threshold=7.589e+02, percent-clipped=0.0 2026-09-24 08:36:15,516 INFO [train.py:1192] (0/2) Epoch 78, batch 200, loss[loss=0.2831, simple_loss=0.3964, pruned_loss=0.08491, over 21184.00 frames. ], tot_loss[loss=0.2538, simple_loss=0.3734, pruned_loss=0.06706, over 3055889.77 frames. ], batch size: 333, lr: 2.71e-03, grad_scale: 16.0 2026-09-24 08:36:21,773 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=246586.66666666666, ans=0.125 2026-09-24 08:36:26,973 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.out_proj.dropout_p, batch_count=246620.0, ans=0.1 2026-09-24 08:36:30,366 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=246653.33333333334, ans=0.2 2026-09-24 08:36:32,998 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=5.33 vs. limit=15.0 2026-09-24 08:36:41,013 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=246720.0, ans=0.125 2026-09-24 08:36:41,359 INFO [train.py:1192] (0/2) Epoch 78, batch 250, loss[loss=0.2738, simple_loss=0.4027, pruned_loss=0.07248, over 24288.00 frames. ], tot_loss[loss=0.2535, simple_loss=0.3731, pruned_loss=0.06696, over 3444228.96 frames. ], batch size: 234, lr: 2.71e-03, grad_scale: 16.0 2026-09-24 08:36:44,551 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=246720.0, ans=0.125 2026-09-24 08:36:51,414 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=246786.66666666666, ans=0.2 2026-09-24 08:37:04,963 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.840e+02 3.577e+02 4.129e+02 4.659e+02 7.262e+02, threshold=8.259e+02, percent-clipped=0.0 2026-09-24 08:37:06,940 INFO [train.py:1192] (0/2) Epoch 78, batch 300, loss[loss=0.256, simple_loss=0.3861, pruned_loss=0.06294, over 24575.00 frames. ], tot_loss[loss=0.2517, simple_loss=0.3714, pruned_loss=0.06601, over 3750407.11 frames. ], batch size: 204, lr: 2.71e-03, grad_scale: 16.0 2026-09-24 08:37:08,887 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.4.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 08:37:12,517 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=192, metric=6.02 vs. limit=15.0 2026-09-24 08:37:15,873 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=246920.0, ans=0.0 2026-09-24 08:37:18,216 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module2.balancer2.prob, batch_count=246953.33333333334, ans=0.125 2026-09-24 08:37:29,288 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.skip_rate, batch_count=247020.0, ans=0.07 2026-09-24 08:37:32,107 INFO [train.py:1192] (0/2) Epoch 78, batch 350, loss[loss=0.1942, simple_loss=0.3135, pruned_loss=0.0375, over 24565.00 frames. ], tot_loss[loss=0.2515, simple_loss=0.3717, pruned_loss=0.06564, over 3993307.34 frames. ], batch size: 137, lr: 2.70e-03, grad_scale: 16.0 2026-09-24 08:37:47,125 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=247153.33333333334, ans=0.125 2026-09-24 08:37:48,175 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=247153.33333333334, ans=0.0 2026-09-24 08:37:56,125 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.713e+02 3.444e+02 3.822e+02 4.330e+02 1.621e+03, threshold=7.644e+02, percent-clipped=1.0 2026-09-24 08:37:58,173 INFO [train.py:1192] (0/2) Epoch 78, batch 400, loss[loss=0.2521, simple_loss=0.3708, pruned_loss=0.06669, over 24547.00 frames. ], tot_loss[loss=0.2506, simple_loss=0.3707, pruned_loss=0.06527, over 4181478.29 frames. ], batch size: 170, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:38:14,293 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_skip_rate, batch_count=247320.0, ans=0.0 2026-09-24 08:38:18,945 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=247353.33333333334, ans=0.125 2026-09-24 08:38:22,956 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=247353.33333333334, ans=0.0 2026-09-24 08:38:24,361 INFO [train.py:1192] (0/2) Epoch 78, batch 450, loss[loss=0.2437, simple_loss=0.3697, pruned_loss=0.05885, over 24613.00 frames. ], tot_loss[loss=0.2511, simple_loss=0.3711, pruned_loss=0.06555, over 4313926.90 frames. ], batch size: 175, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:38:29,484 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=247420.0, ans=0.0 2026-09-24 08:38:29,995 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer1.prob, batch_count=247420.0, ans=0.125 2026-09-24 08:38:35,322 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.64 vs. limit=6.0 2026-09-24 08:38:41,057 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.skip_rate, batch_count=247486.66666666666, ans=0.04949747468305833 2026-09-24 08:38:48,573 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.706e+02 3.246e+02 3.816e+02 4.281e+02 5.996e+02, threshold=7.631e+02, percent-clipped=0.0 2026-09-24 08:38:49,355 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=5.00 vs. limit=6.0 2026-09-24 08:38:50,692 INFO [train.py:1192] (0/2) Epoch 78, batch 500, loss[loss=0.293, simple_loss=0.417, pruned_loss=0.08449, over 24497.00 frames. ], tot_loss[loss=0.251, simple_loss=0.3704, pruned_loss=0.06584, over 4430582.00 frames. ], batch size: 218, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:38:50,810 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer1.max_abs, batch_count=247553.33333333334, ans=10.0 2026-09-24 08:38:56,061 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=247586.66666666666, ans=0.125 2026-09-24 08:39:02,441 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.balancer2.prob, batch_count=247620.0, ans=0.125 2026-09-24 08:39:04,618 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer1.prob, batch_count=247620.0, ans=0.125 2026-09-24 08:39:06,587 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.88 vs. limit=22.5 2026-09-24 08:39:10,305 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer1.prob, batch_count=247653.33333333334, ans=0.125 2026-09-24 08:39:16,618 INFO [train.py:1192] (0/2) Epoch 78, batch 550, loss[loss=0.2653, simple_loss=0.3941, pruned_loss=0.06826, over 24290.00 frames. ], tot_loss[loss=0.2507, simple_loss=0.3704, pruned_loss=0.06546, over 4519437.61 frames. ], batch size: 257, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:39:21,614 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=247753.33333333334, ans=0.0 2026-09-24 08:39:35,572 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=247820.0, ans=0.125 2026-09-24 08:39:36,473 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=247853.33333333334, ans=0.125 2026-09-24 08:39:39,234 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=8.98 vs. limit=10.0 2026-09-24 08:39:40,765 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.871e+02 3.377e+02 3.784e+02 4.275e+02 5.619e+02, threshold=7.568e+02, percent-clipped=0.0 2026-09-24 08:39:42,789 INFO [train.py:1192] (0/2) Epoch 78, batch 600, loss[loss=0.2816, simple_loss=0.4046, pruned_loss=0.07937, over 24336.00 frames. ], tot_loss[loss=0.2517, simple_loss=0.3714, pruned_loss=0.06599, over 4585435.82 frames. ], batch size: 234, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:39:42,865 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=247886.66666666666, ans=0.2 2026-09-24 08:39:49,186 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.max_positive, batch_count=247920.0, ans=0.95 2026-09-24 08:39:49,217 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff2_skip_rate, batch_count=247920.0, ans=0.0 2026-09-24 08:39:49,623 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=247920.0, ans=0.125 2026-09-24 08:39:54,331 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_skip_rate, batch_count=247953.33333333334, ans=0.0 2026-09-24 08:40:04,644 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=248020.0, ans=0.125 2026-09-24 08:40:08,342 INFO [train.py:1192] (0/2) Epoch 78, batch 650, loss[loss=0.266, simple_loss=0.3795, pruned_loss=0.07626, over 24558.00 frames. ], tot_loss[loss=0.2506, simple_loss=0.3704, pruned_loss=0.06539, over 4650754.83 frames. ], batch size: 162, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:40:13,255 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer2.prob, batch_count=248086.66666666666, ans=0.125 2026-09-24 08:40:21,782 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer2.prob, batch_count=248120.0, ans=0.125 2026-09-24 08:40:23,957 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.31 vs. limit=6.0 2026-09-24 08:40:28,208 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.28 vs. limit=15.0 2026-09-24 08:40:31,926 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.810e+02 3.472e+02 3.856e+02 4.289e+02 7.171e+02, threshold=7.712e+02, percent-clipped=0.0 2026-09-24 08:40:34,067 INFO [train.py:1192] (0/2) Epoch 78, batch 700, loss[loss=0.2221, simple_loss=0.3398, pruned_loss=0.05222, over 24591.00 frames. ], tot_loss[loss=0.2506, simple_loss=0.3707, pruned_loss=0.06531, over 4684260.50 frames. ], batch size: 154, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:40:51,336 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.balancer1.prob, batch_count=248320.0, ans=0.125 2026-09-24 08:40:59,690 INFO [train.py:1192] (0/2) Epoch 78, batch 750, loss[loss=0.2521, simple_loss=0.377, pruned_loss=0.06367, over 24570.00 frames. ], tot_loss[loss=0.2501, simple_loss=0.3699, pruned_loss=0.06517, over 4711406.98 frames. ], batch size: 170, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:41:04,671 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.17 vs. limit=12.0 2026-09-24 08:41:13,862 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=248453.33333333334, ans=0.125 2026-09-24 08:41:16,983 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.0.layers.1.self_attn_weights, loss-sum=0.000e+00 2026-09-24 08:41:23,175 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.910e+02 3.459e+02 3.880e+02 4.813e+02 7.336e+02, threshold=7.760e+02, percent-clipped=0.0 2026-09-24 08:41:23,390 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=6.82 vs. limit=12.0 2026-09-24 08:41:25,113 INFO [train.py:1192] (0/2) Epoch 78, batch 800, loss[loss=0.2211, simple_loss=0.3374, pruned_loss=0.05241, over 24564.00 frames. ], tot_loss[loss=0.2501, simple_loss=0.3699, pruned_loss=0.06519, over 4740814.06 frames. ], batch size: 137, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:41:25,201 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.balancer2.prob, batch_count=248553.33333333334, ans=0.125 2026-09-24 08:41:31,153 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.conv_module2.balancer1.prob, batch_count=248586.66666666666, ans=0.125 2026-09-24 08:41:34,505 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer2.prob, batch_count=248586.66666666666, ans=0.125 2026-09-24 08:41:48,804 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=192, metric=4.70 vs. limit=10.0 2026-09-24 08:41:50,416 INFO [train.py:1192] (0/2) Epoch 78, batch 850, loss[loss=0.2822, simple_loss=0.4083, pruned_loss=0.07803, over 24551.00 frames. ], tot_loss[loss=0.2501, simple_loss=0.3699, pruned_loss=0.06513, over 4761868.75 frames. ], batch size: 204, lr: 2.70e-03, grad_scale: 32.0 2026-09-24 08:41:53,450 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=5.68 vs. limit=15.0 2026-09-24 08:41:53,931 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.bypass.skip_rate, batch_count=248720.0, ans=0.07 2026-09-24 08:42:06,057 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=248820.0, ans=0.125 2026-09-24 08:42:06,098 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.out_combiner.scale_min, batch_count=248820.0, ans=0.2 2026-09-24 08:42:13,894 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.596e+02 3.659e+02 4.152e+02 4.826e+02 6.895e+02, threshold=8.305e+02, percent-clipped=0.0 2026-09-24 08:42:16,183 INFO [train.py:1192] (0/2) Epoch 78, batch 900, loss[loss=0.2102, simple_loss=0.3279, pruned_loss=0.04628, over 24574.00 frames. ], tot_loss[loss=0.2507, simple_loss=0.3705, pruned_loss=0.06541, over 4774751.55 frames. ], batch size: 137, lr: 2.69e-03, grad_scale: 32.0 2026-09-24 08:42:29,793 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.const_attention_rate, batch_count=248953.33333333334, ans=0.025 2026-09-24 08:42:32,937 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.33 vs. limit=22.5 2026-09-24 08:42:36,430 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.bypass_mid.scale_min, batch_count=249020.0, ans=0.2 2026-09-24 08:42:36,936 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_module1.balancer1.prob, batch_count=249020.0, ans=0.125 2026-09-24 08:42:41,710 INFO [train.py:1192] (0/2) Epoch 78, batch 950, loss[loss=0.3139, simple_loss=0.3928, pruned_loss=0.1175, over 10914.00 frames. ], tot_loss[loss=0.2508, simple_loss=0.3691, pruned_loss=0.06632, over 4705902.74 frames. ], batch size: 333, lr: 2.69e-03, grad_scale: 32.0 2026-09-24 08:42:42,414 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff2_skip_rate, batch_count=249053.33333333334, ans=0.0 2026-09-24 08:42:46,053 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-78.pt 2026-09-24 08:42:51,047 INFO [train.py:1192] (0/2) Epoch 79, batch 0, loss[loss=0.2111, simple_loss=0.3384, pruned_loss=0.04187, over 24574.00 frames. ], tot_loss[loss=0.2111, simple_loss=0.3384, pruned_loss=0.04187, over 24574.00 frames. ], batch size: 137, lr: 2.68e-03, grad_scale: 32.0 2026-09-24 08:42:51,047 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 08:43:02,755 INFO [train.py:1224] (0/2) Epoch 79, validation: loss=0.1687, simple_loss=0.2865, pruned_loss=0.02543, over 2564189.00 frames. 2026-09-24 08:43:02,755 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 08:43:12,528 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff3_skip_rate, batch_count=249146.66666666666, ans=0.0 2026-09-24 08:43:15,494 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward1.out_whiten, num_groups=1, num_channels=256, metric=7.87 vs. limit=15.0 2026-09-24 08:43:20,615 INFO [scaling.py:1024] (0/2) Whitening: name=encoder_embed.out_whiten, num_groups=1, num_channels=192, metric=5.31 vs. limit=8.0 2026-09-24 08:43:21,940 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.542e+02 3.541e+02 3.974e+02 4.654e+02 7.476e+02, threshold=7.947e+02, percent-clipped=0.0 2026-09-24 08:43:28,496 INFO [train.py:1192] (0/2) Epoch 79, batch 50, loss[loss=0.1976, simple_loss=0.3188, pruned_loss=0.03816, over 24266.00 frames. ], tot_loss[loss=0.2577, simple_loss=0.3769, pruned_loss=0.06927, over 1075590.19 frames. ], batch size: 125, lr: 2.68e-03, grad_scale: 32.0 2026-09-24 08:43:38,582 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.attention_skip_rate, batch_count=249313.33333333334, ans=0.0 2026-09-24 08:43:52,150 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=249380.0, ans=0.125 2026-09-24 08:43:53,999 INFO [train.py:1192] (0/2) Epoch 79, batch 100, loss[loss=0.2485, simple_loss=0.3668, pruned_loss=0.06508, over 24641.00 frames. ], tot_loss[loss=0.2602, simple_loss=0.38, pruned_loss=0.07016, over 1904199.25 frames. ], batch size: 154, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:43:55,914 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=1.294e-02 2026-09-24 08:43:56,753 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=249413.33333333334, ans=0.0 2026-09-24 08:44:03,959 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.49 vs. limit=15.0 2026-09-24 08:44:05,612 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=249480.0, ans=0.0 2026-09-24 08:44:07,386 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=249480.0, ans=0.1 2026-09-24 08:44:11,059 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=249513.33333333334, ans=0.1 2026-09-24 08:44:12,888 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.912e+02 3.453e+02 3.822e+02 4.249e+02 5.886e+02, threshold=7.645e+02, percent-clipped=0.0 2026-09-24 08:44:19,340 INFO [train.py:1192] (0/2) Epoch 79, batch 150, loss[loss=0.2393, simple_loss=0.3451, pruned_loss=0.06672, over 24319.00 frames. ], tot_loss[loss=0.2557, simple_loss=0.3754, pruned_loss=0.06793, over 2557235.33 frames. ], batch size: 125, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:44:25,534 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.balancer2.prob, batch_count=249613.33333333334, ans=0.125 2026-09-24 08:44:31,185 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer1.prob, batch_count=249646.66666666666, ans=0.125 2026-09-24 08:44:44,965 INFO [train.py:1192] (0/2) Epoch 79, batch 200, loss[loss=0.2788, simple_loss=0.3903, pruned_loss=0.0836, over 21101.00 frames. ], tot_loss[loss=0.2521, simple_loss=0.3723, pruned_loss=0.06599, over 3053868.58 frames. ], batch size: 334, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:44:45,208 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.33 vs. limit=15.0 2026-09-24 08:44:54,309 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.balancer_ff3.min_abs, batch_count=249780.0, ans=0.2 2026-09-24 08:44:58,604 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=249813.33333333334, ans=0.0 2026-09-24 08:45:03,023 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer1.prob, batch_count=249846.66666666666, ans=0.125 2026-09-24 08:45:04,799 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.847e+02 3.520e+02 4.008e+02 4.520e+02 7.241e+02, threshold=8.015e+02, percent-clipped=0.0 2026-09-24 08:45:06,564 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=249880.0, ans=0.125 2026-09-24 08:45:08,028 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer1.prob, batch_count=249880.0, ans=0.125 2026-09-24 08:45:10,619 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.feed_forward1.out_whiten, num_groups=1, num_channels=192, metric=8.04 vs. limit=15.0 2026-09-24 08:45:11,303 INFO [train.py:1192] (0/2) Epoch 79, batch 250, loss[loss=0.2665, simple_loss=0.4029, pruned_loss=0.06507, over 24301.00 frames. ], tot_loss[loss=0.2523, simple_loss=0.3722, pruned_loss=0.0662, over 3443266.65 frames. ], batch size: 234, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:45:23,500 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module2.balancer2.prob, batch_count=249980.0, ans=0.125 2026-09-24 08:45:30,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass.scale_min, batch_count=250013.33333333334, ans=0.2 2026-09-24 08:45:36,641 INFO [train.py:1192] (0/2) Epoch 79, batch 300, loss[loss=0.244, simple_loss=0.3793, pruned_loss=0.05437, over 24517.00 frames. ], tot_loss[loss=0.2505, simple_loss=0.3703, pruned_loss=0.06536, over 3747741.54 frames. ], batch size: 204, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:45:41,980 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.53 vs. limit=22.5 2026-09-24 08:45:45,340 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.prob, batch_count=250113.33333333334, ans=0.125 2026-09-24 08:45:52,303 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff2_skip_rate, batch_count=250180.0, ans=0.0 2026-09-24 08:45:55,603 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.824e+02 3.495e+02 3.883e+02 4.427e+02 6.886e+02, threshold=7.766e+02, percent-clipped=0.0 2026-09-24 08:45:58,251 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.ff3_skip_rate, batch_count=250213.33333333334, ans=0.0 2026-09-24 08:46:02,106 INFO [train.py:1192] (0/2) Epoch 79, batch 350, loss[loss=0.2116, simple_loss=0.3261, pruned_loss=0.04853, over 24604.00 frames. ], tot_loss[loss=0.2514, simple_loss=0.3712, pruned_loss=0.0658, over 3992278.35 frames. ], batch size: 137, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:46:08,333 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=250280.0, ans=0.015 2026-09-24 08:46:13,128 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.attention_skip_rate, batch_count=250313.33333333334, ans=0.0 2026-09-24 08:46:24,671 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module2.balancer1.prob, batch_count=250380.0, ans=0.125 2026-09-24 08:46:27,840 INFO [train.py:1192] (0/2) Epoch 79, batch 400, loss[loss=0.2766, simple_loss=0.3918, pruned_loss=0.08067, over 24568.00 frames. ], tot_loss[loss=0.251, simple_loss=0.3709, pruned_loss=0.06553, over 4180118.61 frames. ], batch size: 170, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:46:28,368 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.balancer_ff3.min_abs, batch_count=250413.33333333334, ans=0.2 2026-09-24 08:46:36,107 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.ff3_skip_rate, batch_count=250446.66666666666, ans=0.0 2026-09-24 08:46:42,856 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=250513.33333333334, ans=0.0 2026-09-24 08:46:46,704 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.633e+02 3.490e+02 3.885e+02 4.618e+02 6.233e+02, threshold=7.769e+02, percent-clipped=0.0 2026-09-24 08:46:52,715 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.bypass.skip_rate, batch_count=250580.0, ans=0.04949747468305833 2026-09-24 08:46:53,076 INFO [train.py:1192] (0/2) Epoch 79, batch 450, loss[loss=0.2431, simple_loss=0.3692, pruned_loss=0.05846, over 24620.00 frames. ], tot_loss[loss=0.2522, simple_loss=0.3718, pruned_loss=0.06633, over 4313148.96 frames. ], batch size: 175, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:46:57,153 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=17.61 vs. limit=22.5 2026-09-24 08:46:57,448 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=250580.0, ans=0.1 2026-09-24 08:47:11,831 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=250680.0, ans=0.125 2026-09-24 08:47:18,493 INFO [train.py:1192] (0/2) Epoch 79, batch 500, loss[loss=0.2554, simple_loss=0.3859, pruned_loss=0.06243, over 24491.00 frames. ], tot_loss[loss=0.2511, simple_loss=0.3704, pruned_loss=0.06587, over 4431024.05 frames. ], batch size: 218, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:47:18,810 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=3.01 vs. limit=6.0 2026-09-24 08:47:22,028 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.out_combiner.scale_min, batch_count=250746.66666666666, ans=0.2 2026-09-24 08:47:28,455 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.50 vs. limit=12.0 2026-09-24 08:47:36,421 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=4, num_channels=128, metric=4.35 vs. limit=6.0 2026-09-24 08:47:37,680 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.52 vs. limit=15.0 2026-09-24 08:47:37,919 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.693e+02 3.381e+02 3.663e+02 4.213e+02 5.999e+02, threshold=7.326e+02, percent-clipped=0.0 2026-09-24 08:47:42,835 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_skip_rate, batch_count=250880.0, ans=0.0 2026-09-24 08:47:44,126 INFO [train.py:1192] (0/2) Epoch 79, batch 550, loss[loss=0.2817, simple_loss=0.4089, pruned_loss=0.07722, over 24289.00 frames. ], tot_loss[loss=0.2507, simple_loss=0.3704, pruned_loss=0.06551, over 4520111.10 frames. ], batch size: 257, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:47:48,947 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.skip_rate, batch_count=250946.66666666666, ans=0.035 2026-09-24 08:47:49,541 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.65 vs. limit=6.0 2026-09-24 08:47:50,039 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.whiten, num_groups=1, num_channels=256, metric=5.48 vs. limit=12.0 2026-09-24 08:47:53,844 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=250980.0, ans=0.125 2026-09-24 08:47:58,380 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.bypass_mid.scale_min, batch_count=250980.0, ans=0.2 2026-09-24 08:48:01,163 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_module1.balancer1.prob, batch_count=251013.33333333334, ans=0.125 2026-09-24 08:48:03,971 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.const_attention_rate, batch_count=251046.66666666666, ans=0.025 2026-09-24 08:48:09,490 INFO [train.py:1192] (0/2) Epoch 79, batch 600, loss[loss=0.279, simple_loss=0.4038, pruned_loss=0.07706, over 24286.00 frames. ], tot_loss[loss=0.2515, simple_loss=0.3714, pruned_loss=0.06578, over 4584527.64 frames. ], batch size: 234, lr: 2.67e-03, grad_scale: 32.0 2026-09-24 08:48:16,380 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.conv_skip_rate, batch_count=251113.33333333334, ans=0.0 2026-09-24 08:48:21,606 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.ff3_skip_rate, batch_count=251146.66666666666, ans=0.0 2026-09-24 08:48:28,256 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.682e+02 3.335e+02 3.736e+02 4.552e+02 5.747e+02, threshold=7.472e+02, percent-clipped=0.0 2026-09-24 08:48:34,679 INFO [train.py:1192] (0/2) Epoch 79, batch 650, loss[loss=0.254, simple_loss=0.3746, pruned_loss=0.06665, over 24580.00 frames. ], tot_loss[loss=0.2507, simple_loss=0.3706, pruned_loss=0.06538, over 4650133.59 frames. ], batch size: 162, lr: 2.66e-03, grad_scale: 32.0 2026-09-24 08:48:36,709 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=9.22 vs. limit=15.0 2026-09-24 08:48:47,121 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=251313.33333333334, ans=0.125 2026-09-24 08:48:48,347 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.const_attention_rate, batch_count=251313.33333333334, ans=0.025 2026-09-24 08:48:51,348 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=251346.66666666666, ans=0.1 2026-09-24 08:48:53,186 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.conv_skip_rate, batch_count=251346.66666666666, ans=0.0 2026-09-24 08:48:57,933 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer1.prob, batch_count=251380.0, ans=0.125 2026-09-24 08:49:00,176 INFO [train.py:1192] (0/2) Epoch 79, batch 700, loss[loss=0.2234, simple_loss=0.3425, pruned_loss=0.0522, over 24558.00 frames. ], tot_loss[loss=0.2509, simple_loss=0.3712, pruned_loss=0.06532, over 4683229.95 frames. ], batch size: 154, lr: 2.66e-03, grad_scale: 32.0 2026-09-24 08:49:00,416 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=8.13 vs. limit=15.0 2026-09-24 08:49:08,100 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=251446.66666666666, ans=0.125 2026-09-24 08:49:08,159 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.conv_module1.balancer2.prob, batch_count=251446.66666666666, ans=0.125 2026-09-24 08:49:15,253 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer2.prob, batch_count=251513.33333333334, ans=0.125 2026-09-24 08:49:18,762 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.ff2_skip_rate, batch_count=251513.33333333334, ans=0.0 2026-09-24 08:49:19,128 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.914e+02 3.511e+02 3.902e+02 4.459e+02 6.947e+02, threshold=7.805e+02, percent-clipped=0.0 2026-09-24 08:49:19,939 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.8.prob, batch_count=251546.66666666666, ans=0.125 2026-09-24 08:49:23,785 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=251546.66666666666, ans=0.125 2026-09-24 08:49:25,065 INFO [train.py:1192] (0/2) Epoch 79, batch 750, loss[loss=0.2561, simple_loss=0.3815, pruned_loss=0.0653, over 24571.00 frames. ], tot_loss[loss=0.2503, simple_loss=0.3702, pruned_loss=0.06519, over 4710965.85 frames. ], batch size: 170, lr: 2.66e-03, grad_scale: 32.0 2026-09-24 08:49:28,588 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.bypass.scale_min, batch_count=251580.0, ans=0.2 2026-09-24 08:49:30,339 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.conv_module1.balancer2.prob, batch_count=251613.33333333334, ans=0.125 2026-09-24 08:49:48,642 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward1.hidden_balancer.prob, batch_count=251713.33333333334, ans=0.125 2026-09-24 08:49:50,418 INFO [train.py:1192] (0/2) Epoch 79, batch 800, loss[loss=0.1938, simple_loss=0.3147, pruned_loss=0.03647, over 24557.00 frames. ], tot_loss[loss=0.2497, simple_loss=0.3697, pruned_loss=0.06482, over 4737691.14 frames. ], batch size: 137, lr: 2.66e-03, grad_scale: 32.0 2026-09-24 08:49:53,584 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.0.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.84 vs. limit=22.5 2026-09-24 08:49:58,071 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.conv_module2.balancer2.min_abs, batch_count=251780.0, ans=0.5 2026-09-24 08:50:05,883 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=251846.66666666666, ans=0.125 2026-09-24 08:50:10,035 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.832e+02 3.485e+02 3.865e+02 4.570e+02 6.004e+02, threshold=7.731e+02, percent-clipped=0.0 2026-09-24 08:50:15,841 INFO [train.py:1192] (0/2) Epoch 79, batch 850, loss[loss=0.2629, simple_loss=0.3922, pruned_loss=0.06686, over 24619.00 frames. ], tot_loss[loss=0.2489, simple_loss=0.369, pruned_loss=0.0644, over 4761150.78 frames. ], batch size: 198, lr: 2.66e-03, grad_scale: 32.0 2026-09-24 08:50:22,634 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.1.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=10.13 vs. limit=15.0 2026-09-24 08:50:34,375 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.bypass.scale_min, batch_count=252013.33333333334, ans=0.2 2026-09-24 08:50:34,379 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff3_skip_rate, batch_count=252013.33333333334, ans=0.0 2026-09-24 08:50:41,386 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.52 vs. limit=15.0 2026-09-24 08:50:41,599 INFO [train.py:1192] (0/2) Epoch 79, batch 900, loss[loss=0.19, simple_loss=0.3147, pruned_loss=0.03262, over 24533.00 frames. ], tot_loss[loss=0.2505, simple_loss=0.3704, pruned_loss=0.06529, over 4774268.30 frames. ], batch size: 137, lr: 2.66e-03, grad_scale: 32.0 2026-09-24 08:50:46,904 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.1.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=3.56 vs. limit=15.0 2026-09-24 08:50:47,225 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=1.641e-01 2026-09-24 08:50:48,177 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.ff3_skip_rate, batch_count=252113.33333333334, ans=0.0 2026-09-24 08:50:50,503 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module2.balancer1.prob, batch_count=252113.33333333334, ans=0.125 2026-09-24 08:50:56,831 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=252180.0, ans=0.1 2026-09-24 08:50:57,884 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=252180.0, ans=0.125 2026-09-24 08:51:01,709 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.431e+02 3.649e+02 3.988e+02 4.637e+02 8.380e+02, threshold=7.977e+02, percent-clipped=1.0 2026-09-24 08:51:06,979 INFO [train.py:1192] (0/2) Epoch 79, batch 950, loss[loss=0.3584, simple_loss=0.4295, pruned_loss=0.1436, over 11569.00 frames. ], tot_loss[loss=0.2507, simple_loss=0.3693, pruned_loss=0.06602, over 4713147.22 frames. ], batch size: 334, lr: 2.66e-03, grad_scale: 16.0 2026-09-24 08:51:11,534 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-79.pt 2026-09-24 08:51:16,296 INFO [train.py:1192] (0/2) Epoch 80, batch 0, loss[loss=0.2128, simple_loss=0.335, pruned_loss=0.04526, over 24552.00 frames. ], tot_loss[loss=0.2128, simple_loss=0.335, pruned_loss=0.04526, over 24552.00 frames. ], batch size: 137, lr: 2.64e-03, grad_scale: 32.0 2026-09-24 08:51:16,296 INFO [train.py:1215] (0/2) Computing validation loss 2026-09-24 08:51:18,077 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.5.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.6832, 2.1812, 2.8765, 1.5531], device='cuda:0') 2026-09-24 08:51:24,785 INFO [zipformer.py:1764] (0/2) name=encoder.encoders.4.encoder.layers.0.self_attn_weights, attn_weights_entropy = tensor([2.0433, 2.9893, 2.5800, 2.2545], device='cuda:0') 2026-09-24 08:51:27,874 INFO [train.py:1224] (0/2) Epoch 80, validation: loss=0.1689, simple_loss=0.2867, pruned_loss=0.02552, over 2564189.00 frames. 2026-09-24 08:51:27,874 INFO [train.py:1225] (0/2) Maximum memory allocated so far is 14447MB 2026-09-24 08:51:34,250 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.feed_forward3.out_whiten, num_groups=1, num_channels=256, metric=10.07 vs. limit=15.0 2026-09-24 08:51:40,632 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=252340.0, ans=0.1 2026-09-24 08:51:41,316 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.conv_module2.whiten, num_groups=1, num_channels=256, metric=4.44 vs. limit=15.0 2026-09-24 08:51:54,048 INFO [train.py:1192] (0/2) Epoch 80, batch 50, loss[loss=0.2364, simple_loss=0.3424, pruned_loss=0.06514, over 24269.00 frames. ], tot_loss[loss=0.2554, simple_loss=0.3749, pruned_loss=0.06793, over 1076970.45 frames. ], batch size: 125, lr: 2.64e-03, grad_scale: 32.0 2026-09-24 08:51:57,332 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=252440.0, ans=0.1 2026-09-24 08:51:58,034 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=9.82 vs. limit=10.0 2026-09-24 08:51:59,141 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_skip_rate, batch_count=252473.33333333334, ans=0.0 2026-09-24 08:52:09,731 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.794e+02 3.586e+02 3.951e+02 4.531e+02 1.002e+03, threshold=7.902e+02, percent-clipped=1.0 2026-09-24 08:52:12,218 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.balancer1.prob, batch_count=252540.0, ans=0.125 2026-09-24 08:52:13,076 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=252540.0, ans=0.0 2026-09-24 08:52:19,588 INFO [train.py:1192] (0/2) Epoch 80, batch 100, loss[loss=0.2651, simple_loss=0.3804, pruned_loss=0.07492, over 24575.00 frames. ], tot_loss[loss=0.2591, simple_loss=0.3794, pruned_loss=0.06941, over 1905217.49 frames. ], batch size: 154, lr: 2.64e-03, grad_scale: 32.0 2026-09-24 08:52:19,705 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.balancer2.prob, batch_count=252606.66666666666, ans=0.125 2026-09-24 08:52:22,684 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=4.18 vs. limit=15.0 2026-09-24 08:52:25,481 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.attention_skip_rate, batch_count=252640.0, ans=0.0 2026-09-24 08:52:25,872 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=252640.0, ans=0.125 2026-09-24 08:52:40,932 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.0.bypass.skip_rate, batch_count=252740.0, ans=0.09899494936611666 2026-09-24 08:52:44,898 INFO [train.py:1192] (0/2) Epoch 80, batch 150, loss[loss=0.2205, simple_loss=0.33, pruned_loss=0.05551, over 24257.00 frames. ], tot_loss[loss=0.2543, simple_loss=0.3748, pruned_loss=0.06692, over 2558733.26 frames. ], batch size: 125, lr: 2.64e-03, grad_scale: 32.0 2026-09-24 08:52:45,995 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=252773.33333333334, ans=0.1 2026-09-24 08:52:55,686 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.ff2_skip_rate, batch_count=252840.0, ans=0.0 2026-09-24 08:53:00,766 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.639e+02 3.480e+02 3.903e+02 4.526e+02 1.036e+03, threshold=7.805e+02, percent-clipped=1.0 2026-09-24 08:53:07,706 INFO [scaling.py:1120] (0/2) WithLoss: name=encoder.encoders.5.encoder.layers.0.self_attn_weights, loss-sum=0.000e+00 2026-09-24 08:53:10,450 INFO [train.py:1192] (0/2) Epoch 80, batch 200, loss[loss=0.2914, simple_loss=0.3962, pruned_loss=0.09329, over 20981.00 frames. ], tot_loss[loss=0.2527, simple_loss=0.3729, pruned_loss=0.06619, over 3055321.82 frames. ], batch size: 333, lr: 2.64e-03, grad_scale: 32.0 2026-09-24 08:53:18,580 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module2.balancer1.prob, batch_count=252973.33333333334, ans=0.125 2026-09-24 08:53:20,019 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=252973.33333333334, ans=0.0 2026-09-24 08:53:35,457 INFO [train.py:1192] (0/2) Epoch 80, batch 250, loss[loss=0.2704, simple_loss=0.3994, pruned_loss=0.07066, over 24307.00 frames. ], tot_loss[loss=0.2517, simple_loss=0.372, pruned_loss=0.06572, over 3444701.93 frames. ], batch size: 234, lr: 2.64e-03, grad_scale: 16.0 2026-09-24 08:53:37,257 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.dropout.p, batch_count=253106.66666666666, ans=0.1 2026-09-24 08:53:38,365 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.out_combiner.scale_min, batch_count=253106.66666666666, ans=0.2 2026-09-24 08:53:42,368 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.hidden_balancer.prob, batch_count=253140.0, ans=0.125 2026-09-24 08:53:47,715 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.bypass.scale_min, batch_count=253173.33333333334, ans=0.2 2026-09-24 08:53:51,625 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.914e+02 3.683e+02 4.085e+02 4.754e+02 7.887e+02, threshold=8.170e+02, percent-clipped=1.0 2026-09-24 08:53:51,849 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.4.encoder.layers.0.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.95 vs. limit=15.0 2026-09-24 08:53:55,797 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.attention_skip_rate, batch_count=253240.0, ans=0.0 2026-09-24 08:53:58,900 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.balancer_na.min_abs, batch_count=253240.0, ans=0.02 2026-09-24 08:54:00,955 INFO [train.py:1192] (0/2) Epoch 80, batch 300, loss[loss=0.253, simple_loss=0.3827, pruned_loss=0.06163, over 24535.00 frames. ], tot_loss[loss=0.2513, simple_loss=0.3712, pruned_loss=0.06572, over 3751247.35 frames. ], batch size: 204, lr: 2.64e-03, grad_scale: 16.0 2026-09-24 08:54:01,223 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.nonlin_attention.whiten1, num_groups=1, num_channels=144, metric=6.68 vs. limit=10.0 2026-09-24 08:54:09,148 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module2.balancer2.min_abs, batch_count=253306.66666666666, ans=0.5 2026-09-24 08:54:09,176 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=253306.66666666666, ans=0.0 2026-09-24 08:54:09,536 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/checkpoint-76000.pt 2026-09-24 08:54:14,050 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.conv_module1.balancer1.prob, batch_count=253340.0, ans=0.125 2026-09-24 08:54:15,071 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass.skip_rate, batch_count=253340.0, ans=0.09899494936611666 2026-09-24 08:54:16,864 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.conv_module1.balancer2.prob, batch_count=253373.33333333334, ans=0.125 2026-09-24 08:54:26,228 INFO [train.py:1192] (0/2) Epoch 80, batch 350, loss[loss=0.1974, simple_loss=0.3186, pruned_loss=0.03814, over 24571.00 frames. ], tot_loss[loss=0.2514, simple_loss=0.3717, pruned_loss=0.06553, over 3995487.87 frames. ], batch size: 137, lr: 2.64e-03, grad_scale: 16.0 2026-09-24 08:54:42,597 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.556e+02 3.480e+02 3.784e+02 4.358e+02 6.303e+02, threshold=7.569e+02, percent-clipped=0.0 2026-09-24 08:54:47,717 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.5.encoder.layers.1.conv_module1.whiten, num_groups=1, num_channels=256, metric=3.46 vs. limit=15.0 2026-09-24 08:54:50,884 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.conv_skip_rate, batch_count=253573.33333333334, ans=0.0 2026-09-24 08:54:51,739 INFO [train.py:1192] (0/2) Epoch 80, batch 400, loss[loss=0.2295, simple_loss=0.3581, pruned_loss=0.0505, over 24558.00 frames. ], tot_loss[loss=0.2509, simple_loss=0.371, pruned_loss=0.0654, over 4182590.03 frames. ], batch size: 170, lr: 2.64e-03, grad_scale: 32.0 2026-09-24 08:54:52,347 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.attention_skip_rate, batch_count=253606.66666666666, ans=0.0 2026-09-24 08:54:56,349 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=9.38 vs. limit=22.5 2026-09-24 08:54:59,206 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=253640.0, ans=0.0 2026-09-24 08:55:00,331 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.bypass_mid.scale_min, batch_count=253640.0, ans=0.2 2026-09-24 08:55:17,534 INFO [train.py:1192] (0/2) Epoch 80, batch 450, loss[loss=0.2661, simple_loss=0.39, pruned_loss=0.07106, over 24649.00 frames. ], tot_loss[loss=0.2519, simple_loss=0.3717, pruned_loss=0.066, over 4315001.45 frames. ], batch size: 175, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:55:29,035 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=253840.0, ans=0.1 2026-09-24 08:55:33,644 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.615e+02 3.296e+02 3.721e+02 4.183e+02 6.082e+02, threshold=7.443e+02, percent-clipped=0.0 2026-09-24 08:55:40,835 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward2.hidden_balancer.prob, batch_count=253906.66666666666, ans=0.125 2026-09-24 08:55:42,350 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.0.nonlin_attention.balancer.prob, batch_count=253906.66666666666, ans=0.125 2026-09-24 08:55:43,183 INFO [train.py:1192] (0/2) Epoch 80, batch 500, loss[loss=0.2841, simple_loss=0.4083, pruned_loss=0.08001, over 24496.00 frames. ], tot_loss[loss=0.2504, simple_loss=0.3699, pruned_loss=0.06543, over 4431688.35 frames. ], batch size: 218, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:55:44,759 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.1.self_attn2.whiten, num_groups=1, num_channels=256, metric=8.01 vs. limit=22.5 2026-09-24 08:55:45,147 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.2.encoder.layers.1.self_attn_weights.pos_emb_skip_rate, batch_count=253940.0, ans=0.0 2026-09-24 08:55:45,671 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.feed_forward1.out_proj.dropout_p, batch_count=253940.0, ans=0.1 2026-09-24 08:55:48,961 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=253973.33333333334, ans=0.125 2026-09-24 08:55:58,811 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.3.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=254040.0, ans=0.125 2026-09-24 08:55:59,308 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.feed_forward1.out_proj.dropout_p, batch_count=254040.0, ans=0.1 2026-09-24 08:56:03,077 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.feed_forward3.hidden_balancer.prob, batch_count=254040.0, ans=0.125 2026-09-24 08:56:06,314 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.convnext.layerdrop_rate, batch_count=254073.33333333334, ans=0.015 2026-09-24 08:56:09,264 INFO [train.py:1192] (0/2) Epoch 80, batch 550, loss[loss=0.2795, simple_loss=0.4091, pruned_loss=0.07495, over 24235.00 frames. ], tot_loss[loss=0.2512, simple_loss=0.3709, pruned_loss=0.06573, over 4520055.70 frames. ], batch size: 257, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:56:14,075 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward1.hidden_balancer.prob, batch_count=254140.0, ans=0.125 2026-09-24 08:56:25,532 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.819e+02 3.448e+02 3.764e+02 4.262e+02 8.012e+02, threshold=7.528e+02, percent-clipped=1.0 2026-09-24 08:56:34,696 INFO [train.py:1192] (0/2) Epoch 80, batch 600, loss[loss=0.2729, simple_loss=0.4008, pruned_loss=0.0725, over 24335.00 frames. ], tot_loss[loss=0.2516, simple_loss=0.3713, pruned_loss=0.06592, over 4585520.37 frames. ], batch size: 234, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:56:38,170 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=254273.33333333334, ans=0.125 2026-09-24 08:56:55,661 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.const_attention_rate, batch_count=254406.66666666666, ans=0.025 2026-09-24 08:56:57,440 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=254406.66666666666, ans=0.125 2026-09-24 08:56:59,897 INFO [train.py:1192] (0/2) Epoch 80, batch 650, loss[loss=0.2488, simple_loss=0.3657, pruned_loss=0.06599, over 24556.00 frames. ], tot_loss[loss=0.2504, simple_loss=0.3704, pruned_loss=0.06522, over 4650597.57 frames. ], batch size: 162, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:57:01,419 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.ff3_skip_rate, batch_count=254440.0, ans=0.0 2026-09-24 08:57:10,948 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.feed_forward2.out_whiten, num_groups=1, num_channels=256, metric=6.78 vs. limit=15.0 2026-09-24 08:57:13,434 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder_embed.conv.5.prob, batch_count=254506.66666666666, ans=0.125 2026-09-24 08:57:15,960 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.878e+02 3.361e+02 3.729e+02 4.333e+02 5.678e+02, threshold=7.458e+02, percent-clipped=0.0 2026-09-24 08:57:25,199 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.self_attn_weights.whiten_keys, num_groups=8, num_channels=256, metric=4.78 vs. limit=6.0 2026-09-24 08:57:25,384 INFO [train.py:1192] (0/2) Epoch 80, batch 700, loss[loss=0.2316, simple_loss=0.3509, pruned_loss=0.05617, over 24573.00 frames. ], tot_loss[loss=0.251, simple_loss=0.3712, pruned_loss=0.06538, over 4683946.28 frames. ], batch size: 154, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:57:32,165 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.balancer1.prob, batch_count=254640.0, ans=0.125 2026-09-24 08:57:35,780 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.conv_module1.balancer2.prob, batch_count=254673.33333333334, ans=0.125 2026-09-24 08:57:38,902 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.bypass.scale_min, batch_count=254673.33333333334, ans=0.2 2026-09-24 08:57:44,974 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward1.hidden_balancer.prob, batch_count=254740.0, ans=0.125 2026-09-24 08:57:45,563 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.2.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.56 vs. limit=12.0 2026-09-24 08:57:50,349 INFO [train.py:1192] (0/2) Epoch 80, batch 750, loss[loss=0.2543, simple_loss=0.3778, pruned_loss=0.06545, over 24560.00 frames. ], tot_loss[loss=0.2502, simple_loss=0.3701, pruned_loss=0.06512, over 4710637.71 frames. ], batch size: 170, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:57:52,331 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.whiten, num_groups=1, num_channels=256, metric=4.91 vs. limit=12.0 2026-09-24 08:57:59,976 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.feed_forward3.hidden_balancer.prob, batch_count=254840.0, ans=0.125 2026-09-24 08:58:00,476 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.0.balancer2.prob, batch_count=254840.0, ans=0.125 2026-09-24 08:58:03,624 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.1.whiten, num_groups=1, num_channels=192, metric=3.39 vs. limit=12.0 2026-09-24 08:58:06,371 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.846e+02 3.517e+02 4.029e+02 4.805e+02 7.034e+02, threshold=8.058e+02, percent-clipped=0.0 2026-09-24 08:58:15,523 INFO [train.py:1192] (0/2) Epoch 80, batch 800, loss[loss=0.2172, simple_loss=0.334, pruned_loss=0.05018, over 24554.00 frames. ], tot_loss[loss=0.2498, simple_loss=0.3699, pruned_loss=0.06489, over 4739893.94 frames. ], batch size: 137, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:58:17,752 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.1.feed_forward3.hidden_balancer.prob, batch_count=254940.0, ans=0.125 2026-09-24 08:58:18,580 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.0.conv_module1.balancer2.min_abs, batch_count=254940.0, ans=0.5 2026-09-24 08:58:19,165 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.3.encoder.layers.0.nonlin_attention.whiten2, num_groups=1, num_channels=256, metric=4.89 vs. limit=15.0 2026-09-24 08:58:40,279 INFO [train.py:1192] (0/2) Epoch 80, batch 850, loss[loss=0.2772, simple_loss=0.4029, pruned_loss=0.07568, over 24574.00 frames. ], tot_loss[loss=0.2492, simple_loss=0.3695, pruned_loss=0.06448, over 4761016.22 frames. ], batch size: 204, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:58:42,565 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.0.layers.0.conv_module1.whiten, num_groups=1, num_channels=192, metric=11.60 vs. limit=15.0 2026-09-24 08:58:50,334 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.nonlin_attention.balancer.prob, batch_count=255173.33333333334, ans=0.125 2026-09-24 08:58:56,878 WARNING [optim.py:487] (0/2) Clipping_scale=2.0, grad-norm quartiles 2.839e+02 3.514e+02 3.983e+02 4.549e+02 6.342e+02, threshold=7.965e+02, percent-clipped=0.0 2026-09-24 08:58:56,968 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.0.layers.1.bypass.scale_min, batch_count=255206.66666666666, ans=0.2 2026-09-24 08:59:01,877 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.5.encoder.layers.0.self_attn_weights.pos_emb_skip_rate, batch_count=255240.0, ans=0.0 2026-09-24 08:59:06,377 INFO [train.py:1192] (0/2) Epoch 80, batch 900, loss[loss=0.225, simple_loss=0.3387, pruned_loss=0.05568, over 24565.00 frames. ], tot_loss[loss=0.2505, simple_loss=0.3704, pruned_loss=0.06525, over 4774537.71 frames. ], batch size: 137, lr: 2.63e-03, grad_scale: 32.0 2026-09-24 08:59:19,210 INFO [scaling.py:1024] (0/2) Whitening: name=encoder.encoders.1.encoder.layers.0.self_attn1.whiten, num_groups=1, num_channels=256, metric=17.16 vs. limit=22.5 2026-09-24 08:59:24,377 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.4.encoder.layers.1.feed_forward1.out_proj.dropout_p, batch_count=255373.33333333334, ans=0.1 2026-09-24 08:59:27,948 INFO [scaling.py:214] (0/2) ScheduledFloat: name=encoder.encoders.1.encoder.layers.1.feed_forward2.hidden_balancer.prob, batch_count=255406.66666666666, ans=0.125 2026-09-24 08:59:31,832 INFO [train.py:1192] (0/2) Epoch 80, batch 950, loss[loss=0.3001, simple_loss=0.3968, pruned_loss=0.1017, over 12183.00 frames. ], tot_loss[loss=0.2511, simple_loss=0.3696, pruned_loss=0.06632, over 4714057.29 frames. ], batch size: 333, lr: 2.63e-03, grad_scale: 16.0 2026-09-24 08:59:36,214 INFO [checkpoint.py:75] (0/2) Saving checkpoint to zipformer/exp-spanish-atomic-phones/epoch-80.pt 2026-09-24 08:59:36,437 INFO [train.py:1469] (0/2) Done!