2026-04-15 02:32:17,999 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://0b739bc5175046dfb1.gradio.live full_url=https://0b739bc5175046dfb1.gradio.live?project=w2v-bert-test2-full&write_token=gp2ky6KSVE7qcW9V_t_2zzkj0S4t9UrlpTWiuoKNA5g
2026-04-15 02:32:18,206 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 02:33:12,668 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 02:33:17,050 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 02:33:17,052 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 02:33:17,053 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 02:33:17,053 | INFO | rank=0 | resolving dataset sources
2026-04-15 02:33:17,055 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 02:33:17,055 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 02:33:20,680 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=3.6s
2026-04-15 02:33:20,682 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 02:33:22,464 | INFO | rank=0 | dataset audio decode support ready elapsed=5.4s
2026-04-15 02:33:22,464 | INFO | rank=0 | building dataset split audit
2026-04-15 02:33:22,465 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 02:33:23,113 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=49972.3/s elapsed=0.6s
2026-04-15 02:33:23,773 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=49529.2/s elapsed=1.3s
2026-04-15 02:33:24,449 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=48996.7/s elapsed=2.0s
2026-04-15 02:33:25,124 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=48757.6/s elapsed=2.7s
2026-04-15 02:33:25,799 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=48604.9/s elapsed=3.3s
2026-04-15 02:33:26,462 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=48645.8/s elapsed=4.0s
2026-04-15 02:33:27,121 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=48725.7/s elapsed=4.7s
2026-04-15 02:33:27,780 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=48779.0/s elapsed=5.3s
2026-04-15 02:33:28,441 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=48809.3/s elapsed=6.0s
2026-04-15 02:33:29,101 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=48839.0/s elapsed=6.6s
2026-04-15 02:33:29,761 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=48862.9/s elapsed=7.3s
2026-04-15 02:33:30,421 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=48883.1/s elapsed=8.0s
2026-04-15 02:33:31,111 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=48730.7/s elapsed=8.6s
2026-04-15 02:33:31,825 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=48479.4/s elapsed=9.4s
2026-04-15 02:33:32,529 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=48308.2/s elapsed=10.1s
2026-04-15 02:33:33,251 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=48079.0/s elapsed=10.8s
2026-04-15 02:33:34,037 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=47615.2/s elapsed=11.6s
2026-04-15 02:33:34,728 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=47574.2/s elapsed=12.3s
2026-04-15 02:33:35,426 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=47512.9/s elapsed=13.0s
2026-04-15 02:33:36,236 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=47071.2/s elapsed=13.8s
2026-04-15 02:33:36,237 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=13.8s
2026-04-15 02:33:36,237 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 02:33:36,259 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=46018.0/s elapsed=0.0s
2026-04-15 02:33:36,280 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=47382.9/s elapsed=0.0s
2026-04-15 02:33:36,300 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=47658.2/s elapsed=0.1s
2026-04-15 02:33:36,305 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=47664.2/s elapsed=0.1s
2026-04-15 02:33:36,305 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.1s
2026-04-15 02:33:40,428 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=23.4s
2026-04-15 02:33:40,628 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 02:34:32,340 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=51.7s
2026-04-15 02:34:32,341 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 02:34:32,372 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 02:34:35,755 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 dataloader_backend=yomikomi num_workers=12 dataloader_worker_threads=1 metadata_workers=8 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 02:34:35,755 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=12 worker_threads=1 backend=yomikomi
2026-04-15 02:34:35,755 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 02:34:36,148 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=82540.2/s elapsed=0.4s
2026-04-15 02:34:36,541 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=82513.9/s elapsed=0.8s
2026-04-15 02:34:36,953 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=81178.6/s elapsed=1.2s
2026-04-15 02:34:37,357 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=80940.5/s elapsed=1.6s
2026-04-15 02:34:37,753 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=81104.2/s elapsed=2.0s
2026-04-15 02:34:38,139 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=81579.1/s elapsed=2.4s
2026-04-15 02:34:38,524 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=81927.1/s elapsed=2.8s
2026-04-15 02:34:38,979 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=80433.1/s elapsed=3.2s
2026-04-15 02:34:39,379 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=80493.8/s elapsed=3.6s
2026-04-15 02:34:39,764 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=80852.3/s elapsed=4.0s
2026-04-15 02:34:40,149 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=81139.2/s elapsed=4.4s
2026-04-15 02:34:40,535 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=81364.1/s elapsed=4.8s
2026-04-15 02:34:40,925 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=81500.8/s elapsed=5.2s
2026-04-15 02:34:41,311 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=81669.3/s elapsed=5.6s
2026-04-15 02:34:41,697 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=81820.3/s elapsed=5.9s
2026-04-15 02:34:42,085 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=81928.7/s elapsed=6.3s
2026-04-15 02:34:42,481 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=81924.1/s elapsed=6.7s
2026-04-15 02:34:42,868 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=82025.7/s elapsed=7.1s
2026-04-15 02:34:43,254 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=82125.0/s elapsed=7.5s
2026-04-15 02:34:43,640 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=82210.9/s elapsed=7.9s
2026-04-15 02:34:51,436 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=15.7s
2026-04-15 02:34:51,437 | INFO | rank=0 | train dataloader metadata ready elapsed=15.7s
2026-04-15 02:34:51,437 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 02:34:51,961 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=61959.8/s elapsed=0.5s
2026-04-15 02:34:52,478 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=62284.2/s elapsed=1.0s
2026-04-15 02:34:52,998 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=62309.4/s elapsed=1.6s
2026-04-15 02:34:53,519 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=62272.6/s elapsed=2.1s
2026-04-15 02:34:54,038 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=62322.4/s elapsed=2.6s
2026-04-15 02:34:54,552 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=62440.1/s elapsed=3.1s
2026-04-15 02:34:55,069 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=62474.3/s elapsed=3.6s
2026-04-15 02:34:55,588 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=62478.9/s elapsed=4.2s
2026-04-15 02:34:56,110 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=62436.0/s elapsed=4.7s
2026-04-15 02:34:56,630 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=62422.5/s elapsed=5.2s
2026-04-15 02:34:57,150 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=62415.5/s elapsed=5.7s
2026-04-15 02:34:57,670 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=62402.8/s elapsed=6.2s
2026-04-15 02:34:58,190 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=62399.1/s elapsed=6.8s
2026-04-15 02:34:58,705 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=62436.1/s elapsed=7.3s
2026-04-15 02:34:59,225 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=62430.5/s elapsed=7.8s
2026-04-15 02:34:59,743 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=62439.1/s elapsed=8.3s
2026-04-15 02:35:00,259 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=62462.5/s elapsed=8.8s
2026-04-15 02:35:00,775 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=62480.7/s elapsed=9.3s
2026-04-15 02:35:01,298 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=62455.5/s elapsed=9.9s
2026-04-15 02:35:01,821 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=62428.3/s elapsed=10.4s
2026-04-15 02:35:01,822 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 02:35:03,258 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=10.4s sort_elapsed=1.4s
2026-04-15 02:35:03,874 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 02:35:07,320 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=75255.9/s elapsed=3.4s
2026-04-15 02:35:07,587 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=78563.5/s elapsed=3.7s
2026-04-15 02:35:07,861 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=81298.9/s elapsed=4.0s
2026-04-15 02:35:09,945 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=58731.1/s elapsed=6.1s
2026-04-15 02:35:10,236 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=61139.2/s elapsed=6.4s
2026-04-15 02:35:10,520 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=63399.6/s elapsed=6.6s
2026-04-15 02:35:10,810 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=65425.0/s elapsed=6.9s
2026-04-15 02:35:11,151 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=66817.8/s elapsed=7.3s
2026-04-15 02:35:13,919 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=51629.5/s elapsed=10.0s
2026-04-15 02:35:14,234 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=53188.6/s elapsed=10.4s
2026-04-15 02:35:14,522 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=54791.6/s elapsed=10.6s
2026-04-15 02:35:14,799 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=56371.5/s elapsed=10.9s
2026-04-15 02:35:15,069 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=57906.7/s elapsed=11.2s
2026-04-15 02:35:15,082 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=12 worker_threads=1 backend=yomikomi
2026-04-15 02:35:15,082 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 02:35:15,097 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=65833.3/s elapsed=0.0s
2026-04-15 02:35:15,112 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=66391.3/s elapsed=0.0s
2026-04-15 02:35:15,127 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=66682.8/s elapsed=0.0s
2026-04-15 02:35:15,130 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=66477.6/s elapsed=0.0s
2026-04-15 02:35:15,169 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 02:35:15,169 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 02:35:15,169 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 02:35:15,186 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=59498.7/s elapsed=0.0s
2026-04-15 02:35:15,202 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=60803.4/s elapsed=0.0s
2026-04-15 02:35:15,219 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=60938.5/s elapsed=0.0s
2026-04-15 02:35:15,222 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=60923.4/s elapsed=0.1s
2026-04-15 02:35:15,222 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 02:35:15,224 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.1s sort_elapsed=0.0s
2026-04-15 02:35:15,225 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 02:35:15,231 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=152480.4/s elapsed=0.0s
2026-04-15 02:35:15,238 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=144940.0/s elapsed=0.0s
2026-04-15 02:35:15,246 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=141801.7/s elapsed=0.0s
2026-04-15 02:35:15,247 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=140757.5/s elapsed=0.0s
2026-04-15 02:35:15,248 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=42.9s
2026-04-15 02:35:15,248 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 02:36:33,321 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 02:36:33,478 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=1m 18.2s
2026-04-15 02:37:34,433 | INFO | rank=0 | resumed from artifacts/w2v-bert-test2-full/checkpoint_step_last.pt starting_epoch=2 global_step=12000 best_val_wer=0.4044
2026-04-15 02:37:34,707 | INFO | rank=0 | trackio initialized project=w2v-bert-test2-full run_name=w2v-bert-test2-full_20260414_233214_resume-e0002-s00012000 trackio_dir=artifacts/w2v-bert-test2-full/trackio elapsed_since_start=5m 19.7s
2026-04-15 02:37:34,707 | INFO | rank=0 | epoch 2/100 started train_batches=249868 val_batches=1355 grad_accumulation=4
2026-04-15 02:37:47,325 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 backend=yomikomi num_workers=12 prefetch_buffer_size=32 still waiting elapsed=12.6s
2026-04-15 02:38:05,369 | INFO | rank=0 | epoch=2 first_train_batch_ready elapsed=30.6s data_wait=30.5s batch_size=2 max_feature_frames=191 target_tokens=65
2026-04-15 02:38:12,603 | INFO | rank=0 | epoch=2 waiting for training batch 2/249868 backend=yomikomi num_workers=12 prefetch_buffer_size=32 still waiting elapsed=5.0s
2026-04-15 02:38:21,289 | INFO | rank=0 | epoch=2 waiting for training batch 3/249868 backend=yomikomi num_workers=12 prefetch_buffer_size=32 still waiting elapsed=5.0s
2026-04-15 02:38:35,105 | INFO | rank=0 | epoch=2 waiting for training batch 5/249868 backend=yomikomi num_workers=12 prefetch_buffer_size=32 still waiting elapsed=5.0s
2026-04-15 02:38:48,381 | INFO | rank=0 | epoch=2 waiting for training batch 6/249868 backend=yomikomi num_workers=12 prefetch_buffer_size=32 still waiting elapsed=5.0s
2026-04-15 02:38:58,405 | INFO | rank=0 | epoch=2 waiting for training batch 8/249868 backend=yomikomi num_workers=12 prefetch_buffer_size=32 still waiting elapsed=5.0s
2026-04-15 02:39:04,646 | INFO | rank=0 | batch_timing epoch=2 batch=8/249868 global_step=12002 data_wait=10.2s step_compute=1.0s batch_size=1 effective_frames=815 max_feature_frames=815 target_tokens=121
2026-04-15 02:39:09,648 | INFO | rank=0 | epoch=2 waiting for training batch 9/249868 backend=yomikomi num_workers=12 prefetch_buffer_size=32 still waiting elapsed=5.0s