2026-04-15 02:20:47,031 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://7b3e695e7caa6ea3d6.gradio.live full_url=https://7b3e695e7caa6ea3d6.gradio.live?project=w2v-bert-test2-full&write_token=A_RxMAjVoJzU60_FguTOCyHo9QePxhMMRYA6WS5RpM8
2026-04-15 02:20:47,234 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 02:21:41,580 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 02:21:46,046 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 02:21:46,049 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 02:21:46,050 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 02:21:46,050 | INFO | rank=0 | resolving dataset sources
2026-04-15 02:21:46,052 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 02:21:46,053 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 02:21:49,357 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=3.3s
2026-04-15 02:21:49,358 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 02:21:51,163 | INFO | rank=0 | dataset audio decode support ready elapsed=5.1s
2026-04-15 02:21:51,164 | INFO | rank=0 | building dataset split audit
2026-04-15 02:21:51,164 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 02:21:51,813 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=49907.4/s elapsed=0.6s
2026-04-15 02:21:52,461 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=49965.0/s elapsed=1.3s
2026-04-15 02:21:53,130 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=49462.3/s elapsed=2.0s
2026-04-15 02:21:53,796 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=49248.4/s elapsed=2.6s
2026-04-15 02:21:54,460 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=49163.9/s elapsed=3.3s
2026-04-15 02:21:55,114 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=49230.1/s elapsed=4.0s
2026-04-15 02:21:55,772 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=49241.1/s elapsed=4.6s
2026-04-15 02:21:56,446 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=49093.8/s elapsed=5.3s
2026-04-15 02:21:57,101 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=49135.6/s elapsed=5.9s
2026-04-15 02:21:57,755 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=49177.9/s elapsed=6.6s
2026-04-15 02:21:58,405 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=49237.5/s elapsed=7.2s
2026-04-15 02:21:59,057 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=49279.5/s elapsed=7.9s
2026-04-15 02:21:59,736 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=49156.0/s elapsed=8.6s
2026-04-15 02:22:00,438 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=48928.5/s elapsed=9.3s
2026-04-15 02:22:01,135 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=48760.8/s elapsed=10.0s
2026-04-15 02:22:01,848 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=48537.3/s elapsed=10.7s
2026-04-15 02:22:02,626 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=48072.5/s elapsed=11.5s
2026-04-15 02:22:03,307 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=48044.6/s elapsed=12.1s
2026-04-15 02:22:03,996 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=47989.4/s elapsed=12.8s
2026-04-15 02:22:04,810 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=47505.7/s elapsed=13.6s
2026-04-15 02:22:04,810 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=13.6s
2026-04-15 02:22:04,811 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 02:22:04,833 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=45157.3/s elapsed=0.0s
2026-04-15 02:22:04,853 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=47172.1/s elapsed=0.0s
2026-04-15 02:22:04,873 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=47712.5/s elapsed=0.1s
2026-04-15 02:22:04,878 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=47725.8/s elapsed=0.1s
2026-04-15 02:22:04,878 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.1s
2026-04-15 02:22:09,061 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=23.0s
2026-04-15 02:22:09,259 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 02:23:01,281 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=52.0s
2026-04-15 02:23:01,283 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 02:23:05,001 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=3.7s
2026-04-15 02:23:09,174 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 dataloader_backend=yomikomi num_workers=12 dataloader_worker_threads=1 metadata_workers=8 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 02:23:09,174 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=12 worker_threads=1 backend=yomikomi
2026-04-15 02:23:09,174 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 02:23:09,671 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=65201.9/s elapsed=0.5s
2026-04-15 02:23:10,165 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=65412.4/s elapsed=1.0s
2026-04-15 02:23:10,659 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=65508.6/s elapsed=1.5s
2026-04-15 02:23:11,176 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=64778.2/s elapsed=2.0s
2026-04-15 02:23:11,670 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=64933.9/s elapsed=2.5s
2026-04-15 02:23:12,171 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=64890.0/s elapsed=3.0s
2026-04-15 02:23:12,671 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=64880.5/s elapsed=3.5s
2026-04-15 02:23:13,167 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=64936.0/s elapsed=4.0s
2026-04-15 02:23:13,666 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=64948.5/s elapsed=4.5s
2026-04-15 02:23:14,165 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=64939.0/s elapsed=5.0s
2026-04-15 02:23:14,659 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=65000.0/s elapsed=5.5s
2026-04-15 02:23:15,152 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=65060.7/s elapsed=6.0s
2026-04-15 02:23:15,647 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=65100.1/s elapsed=6.5s
2026-04-15 02:23:16,142 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=65123.6/s elapsed=7.0s
2026-04-15 02:23:16,637 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=65149.5/s elapsed=7.5s
2026-04-15 02:23:17,132 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=65163.8/s elapsed=8.0s
2026-04-15 02:23:17,630 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=65165.8/s elapsed=8.5s
2026-04-15 02:23:18,126 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=65169.4/s elapsed=9.0s
2026-04-15 02:23:18,623 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=65171.8/s elapsed=9.4s
2026-04-15 02:23:19,118 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=65187.0/s elapsed=9.9s
2026-04-15 02:23:27,071 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=17.9s
2026-04-15 02:23:27,072 | INFO | rank=0 | train dataloader metadata ready elapsed=17.9s
2026-04-15 02:23:27,073 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 02:23:27,598 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=61678.9/s elapsed=0.5s
2026-04-15 02:23:28,124 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=61690.8/s elapsed=1.1s
2026-04-15 02:23:28,656 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=61405.3/s elapsed=1.6s
2026-04-15 02:23:29,186 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=61353.5/s elapsed=2.1s
2026-04-15 02:23:29,716 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=61310.6/s elapsed=2.6s
2026-04-15 02:23:30,241 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=61373.5/s elapsed=3.2s
2026-04-15 02:23:30,765 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=61446.9/s elapsed=3.7s
2026-04-15 02:23:31,293 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=61438.9/s elapsed=4.2s
2026-04-15 02:23:31,822 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=61425.0/s elapsed=4.7s
2026-04-15 02:23:32,351 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=61408.5/s elapsed=5.3s
2026-04-15 02:23:32,876 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=61441.6/s elapsed=5.8s
2026-04-15 02:23:33,398 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=61491.7/s elapsed=6.3s
2026-04-15 02:23:33,929 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=61453.2/s elapsed=6.9s
2026-04-15 02:23:34,461 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=61413.5/s elapsed=7.4s
2026-04-15 02:23:34,988 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=61421.7/s elapsed=7.9s
2026-04-15 02:23:35,512 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=61446.5/s elapsed=8.4s
2026-04-15 02:23:36,049 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=61384.3/s elapsed=9.0s
2026-04-15 02:23:36,579 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=61370.0/s elapsed=9.5s
2026-04-15 02:23:37,113 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=61338.9/s elapsed=10.0s
2026-04-15 02:23:37,643 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=61327.2/s elapsed=10.6s
2026-04-15 02:23:37,643 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 02:23:38,865 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=10.6s sort_elapsed=1.2s
2026-04-15 02:23:39,482 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 02:23:42,843 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=77149.0/s elapsed=3.4s
2026-04-15 02:23:43,104 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=80540.9/s elapsed=3.6s
2026-04-15 02:23:43,364 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=83490.5/s elapsed=3.9s
2026-04-15 02:23:45,329 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=60981.3/s elapsed=5.8s
2026-04-15 02:23:45,631 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=63258.6/s elapsed=6.1s
2026-04-15 02:23:45,922 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=65434.4/s elapsed=6.4s
2026-04-15 02:23:46,214 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=67404.8/s elapsed=6.7s
2026-04-15 02:23:46,507 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=69213.1/s elapsed=7.0s
2026-04-15 02:23:49,534 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=51591.9/s elapsed=10.1s
2026-04-15 02:23:49,849 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=53150.2/s elapsed=10.4s
2026-04-15 02:23:50,140 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=54738.9/s elapsed=10.7s
2026-04-15 02:23:50,422 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=56294.1/s elapsed=10.9s
2026-04-15 02:23:50,697 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=57802.4/s elapsed=11.2s
2026-04-15 02:24:00,383 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=12 worker_threads=1 backend=yomikomi
2026-04-15 02:24:00,384 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 02:24:00,400 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=62202.4/s elapsed=0.0s
2026-04-15 02:24:00,416 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=63555.4/s elapsed=0.0s
2026-04-15 02:24:00,431 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=63896.7/s elapsed=0.0s
2026-04-15 02:24:00,435 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=63642.9/s elapsed=0.1s
2026-04-15 02:24:00,473 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 02:24:00,474 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 02:24:00,474 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 02:24:00,490 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=61772.8/s elapsed=0.0s
2026-04-15 02:24:00,506 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=61999.9/s elapsed=0.0s
2026-04-15 02:24:00,523 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=61245.9/s elapsed=0.0s
2026-04-15 02:24:00,527 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=61045.6/s elapsed=0.1s
2026-04-15 02:24:00,527 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 02:24:00,529 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.1s sort_elapsed=0.0s
2026-04-15 02:24:00,529 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 02:24:00,536 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=149589.2/s elapsed=0.0s
2026-04-15 02:24:00,544 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=140589.0/s elapsed=0.0s
2026-04-15 02:24:00,552 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=135353.4/s elapsed=0.0s
2026-04-15 02:24:00,553 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=134757.6/s elapsed=0.0s
2026-04-15 02:24:00,554 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=55.6s
2026-04-15 02:24:00,554 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 02:25:19,825 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 02:25:19,990 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=1m 19.4s
2026-04-15 02:26:23,520 | INFO | rank=0 | resumed from artifacts/w2v-bert-test2-full/checkpoint_step_last.pt starting_epoch=2 global_step=12000 best_val_wer=0.4044
2026-04-15 02:26:23,801 | INFO | rank=0 | trackio initialized project=w2v-bert-test2-full run_name=w2v-bert-test2-full_20260414_232044_resume-e0002-s00012000 trackio_dir=artifacts/w2v-bert-test2-full/trackio elapsed_since_start=5m 39.6s
2026-04-15 02:26:23,802 | INFO | rank=0 | epoch 2/100 started train_batches=249868 val_batches=1355 grad_accumulation=4
2026-04-15 02:26:36,238 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=12 prefetch_factor=2 still waiting elapsed=12.4s
2026-04-15 02:27:06,243 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=12 prefetch_factor=2 still waiting elapsed=42.4s
2026-04-15 02:27:36,258 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=12 prefetch_factor=2 still waiting elapsed=1m 12.4s
2026-04-15 02:28:06,265 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=12 prefetch_factor=2 still waiting elapsed=1m 42.4s
2026-04-15 02:28:36,274 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=12 prefetch_factor=2 still waiting elapsed=2m 12.4s
2026-04-15 02:29:06,281 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=12 prefetch_factor=2 still waiting elapsed=2m 42.4s
2026-04-15 02:29:36,287 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=12 prefetch_factor=2 still waiting elapsed=3m 12.5s
2026-04-15 02:30:06,294 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=12 prefetch_factor=2 still waiting elapsed=3m 42.5s