2026-04-15 03:33:39,359 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://472122af0a4fe5759a.gradio.live full_url=https://472122af0a4fe5759a.gradio.live?project=w2v-bert-test2-full&write_token=wdSDgVbCIElKFDvO-Xzr24Re3nVgihVE2RicO1MJrtk
2026-04-15 03:33:39,539 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 03:33:46,676 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 03:33:47,313 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 03:33:47,313 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 03:33:47,314 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 03:33:47,314 | INFO | rank=0 | resolving dataset sources
2026-04-15 03:33:47,315 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 03:33:47,315 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 03:33:47,690 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=0.4s
2026-04-15 03:33:47,691 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 03:33:48,537 | INFO | rank=0 | dataset audio decode support ready elapsed=1.2s
2026-04-15 03:33:48,537 | INFO | rank=0 | building dataset split audit
2026-04-15 03:33:48,537 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 03:33:49,021 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=66996.2/s elapsed=0.5s
2026-04-15 03:33:49,503 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=67121.4/s elapsed=1.0s
2026-04-15 03:33:49,998 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=66554.0/s elapsed=1.5s
2026-04-15 03:33:50,494 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=66243.4/s elapsed=2.0s
2026-04-15 03:33:50,986 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=66178.0/s elapsed=2.4s
2026-04-15 03:33:51,470 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=66293.9/s elapsed=2.9s
2026-04-15 03:33:51,957 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=66336.8/s elapsed=3.4s
2026-04-15 03:33:52,444 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=66357.5/s elapsed=3.9s
2026-04-15 03:33:52,940 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=66246.4/s elapsed=4.4s
2026-04-15 03:33:53,424 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=66325.9/s elapsed=4.9s
2026-04-15 03:33:53,910 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=66356.5/s elapsed=5.4s
2026-04-15 03:33:54,396 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=66381.7/s elapsed=5.9s
2026-04-15 03:33:54,897 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=66247.4/s elapsed=6.4s
2026-04-15 03:33:55,410 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=66025.3/s elapsed=6.9s
2026-04-15 03:33:55,918 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=65866.3/s elapsed=7.4s
2026-04-15 03:33:56,446 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=65569.5/s elapsed=7.9s
2026-04-15 03:33:57,015 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=64988.4/s elapsed=8.5s
2026-04-15 03:33:57,530 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=64876.0/s elapsed=9.0s
2026-04-15 03:33:58,048 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=64748.8/s elapsed=9.5s
2026-04-15 03:33:58,605 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=64385.6/s elapsed=10.1s
2026-04-15 03:33:58,605 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=10.1s
2026-04-15 03:33:58,605 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 03:33:58,620 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=67076.8/s elapsed=0.0s
2026-04-15 03:33:58,635 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=66714.8/s elapsed=0.0s
2026-04-15 03:33:58,650 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=66841.4/s elapsed=0.0s
2026-04-15 03:33:58,653 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=66594.3/s elapsed=0.0s
2026-04-15 03:33:58,654 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.0s
2026-04-15 03:34:01,958 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=14.6s
2026-04-15 03:34:02,088 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 03:34:07,660 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=5.6s
2026-04-15 03:34:07,660 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 03:34:07,686 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 03:34:10,936 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 dataloader_backend=rust-parquet num_workers=16 dataloader_worker_threads=1 metadata_workers=8 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 03:34:10,936 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=16 worker_threads=1 backend=rust-parquet
2026-04-15 03:34:10,936 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 03:34:11,315 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=85661.6/s elapsed=0.4s
2026-04-15 03:34:11,694 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=85619.0/s elapsed=0.8s
2026-04-15 03:34:12,072 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=85626.6/s elapsed=1.1s
2026-04-15 03:34:12,450 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=85646.1/s elapsed=1.5s
2026-04-15 03:34:12,827 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=85701.8/s elapsed=1.9s
2026-04-15 03:34:13,206 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=85693.0/s elapsed=2.3s
2026-04-15 03:34:13,584 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=85699.0/s elapsed=2.6s
2026-04-15 03:34:13,963 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=85668.2/s elapsed=3.0s
2026-04-15 03:34:14,344 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=85599.4/s elapsed=3.4s
2026-04-15 03:34:14,748 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=85032.1/s elapsed=3.8s
2026-04-15 03:34:15,211 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=83402.1/s elapsed=4.3s
2026-04-15 03:34:15,691 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=81811.6/s elapsed=4.8s
2026-04-15 03:34:16,169 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=80532.0/s elapsed=5.2s
2026-04-15 03:34:16,645 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=79495.5/s elapsed=5.7s
2026-04-15 03:34:17,121 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=78606.6/s elapsed=6.2s
2026-04-15 03:34:17,600 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=77825.7/s elapsed=6.7s
2026-04-15 03:34:18,078 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=77157.2/s elapsed=7.1s
2026-04-15 03:34:18,557 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=76562.3/s elapsed=7.6s
2026-04-15 03:34:19,035 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=76038.2/s elapsed=8.1s
2026-04-15 03:34:19,514 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=75576.7/s elapsed=8.6s
2026-04-15 03:34:27,609 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=16.7s
2026-04-15 03:34:27,609 | INFO | rank=0 | train dataloader metadata ready elapsed=16.7s
2026-04-15 03:34:27,609 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 03:34:28,130 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=62218.5/s elapsed=0.5s
2026-04-15 03:34:28,649 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=62332.5/s elapsed=1.0s
2026-04-15 03:34:29,168 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=62389.6/s elapsed=1.6s
2026-04-15 03:34:29,686 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=62425.9/s elapsed=2.1s
2026-04-15 03:34:30,205 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=62430.0/s elapsed=2.6s
2026-04-15 03:34:30,719 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=62528.1/s elapsed=3.1s
2026-04-15 03:34:31,240 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=62493.9/s elapsed=3.6s
2026-04-15 03:34:31,759 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=62482.3/s elapsed=4.1s
2026-04-15 03:34:32,278 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=62478.0/s elapsed=4.7s
2026-04-15 03:34:32,795 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=62500.6/s elapsed=5.2s
2026-04-15 03:34:33,309 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=62557.0/s elapsed=5.7s
2026-04-15 03:34:33,822 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=62604.7/s elapsed=6.2s
2026-04-15 03:34:34,335 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=62650.7/s elapsed=6.7s
2026-04-15 03:34:34,853 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=62647.1/s elapsed=7.2s
2026-04-15 03:34:35,630 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=60619.7/s elapsed=8.0s
2026-04-15 03:34:36,504 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=58304.0/s elapsed=8.9s
2026-04-15 03:34:37,178 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=57584.1/s elapsed=9.6s
2026-04-15 03:34:37,775 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=57392.2/s elapsed=10.2s
2026-04-15 03:34:38,397 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=57087.9/s elapsed=10.8s
2026-04-15 03:34:39,065 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=56589.4/s elapsed=11.5s
2026-04-15 03:34:39,065 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 03:34:40,364 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=11.5s sort_elapsed=1.3s
2026-04-15 03:34:41,132 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 03:34:43,989 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=90778.1/s elapsed=2.9s
2026-04-15 03:34:44,362 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=90324.2/s elapsed=3.2s
2026-04-15 03:34:44,580 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=94006.7/s elapsed=3.4s
2026-04-15 03:34:45,552 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=80661.2/s elapsed=4.4s
2026-04-15 03:34:45,813 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=83095.7/s elapsed=4.7s
2026-04-15 03:34:46,151 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=83948.1/s elapsed=5.0s
2026-04-15 03:34:46,552 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=83732.5/s elapsed=5.4s
2026-04-15 03:34:46,894 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=84380.2/s elapsed=5.8s
2026-04-15 03:34:48,047 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=74998.9/s elapsed=6.9s
2026-04-15 03:34:48,369 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=76139.9/s elapsed=7.2s
2026-04-15 03:34:48,645 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=77651.1/s elapsed=7.5s
2026-04-15 03:34:48,943 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=78847.0/s elapsed=7.8s
2026-04-15 03:34:49,303 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=79336.1/s elapsed=8.2s
2026-04-15 03:34:49,305 | INFO | rank=0 | train dataloader rust-parquet feature cache index build started cache_dir=/home/yehor/squeezeformer/data/test-warm/feature-cache-w2v-bert/train batches=249868 num_workers=16 prefetch_batches=32
2026-04-15 03:36:41,453 | INFO | rank=0 | train dataloader rust-parquet feature cache index ready entries=650171 elapsed=112.1s
2026-04-15 03:36:41,454 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=16 worker_threads=1 backend=rust-parquet
2026-04-15 03:36:41,454 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 03:36:41,466 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=82671.2/s elapsed=0.0s
2026-04-15 03:36:41,478 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=83320.6/s elapsed=0.0s
2026-04-15 03:36:41,490 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=83630.9/s elapsed=0.0s
2026-04-15 03:36:41,493 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=83352.2/s elapsed=0.0s
2026-04-15 03:36:41,524 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 03:36:41,524 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 03:36:41,524 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 03:36:41,538 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=74184.4/s elapsed=0.0s
2026-04-15 03:36:41,550 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=76866.7/s elapsed=0.0s
2026-04-15 03:36:41,563 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=77533.3/s elapsed=0.0s
2026-04-15 03:36:41,566 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=77521.1/s elapsed=0.0s
2026-04-15 03:36:41,566 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 03:36:41,567 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.0s sort_elapsed=0.0s
2026-04-15 03:36:41,568 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 03:36:41,573 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=195376.6/s elapsed=0.0s
2026-04-15 03:36:41,579 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=183522.4/s elapsed=0.0s
2026-04-15 03:36:41,584 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=177974.9/s elapsed=0.0s
2026-04-15 03:36:41,586 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=176113.9/s elapsed=0.0s
2026-04-15 03:36:41,586 | INFO | rank=0 | validation dataloader rust-parquet feature cache index build started cache_dir=/home/yehor/squeezeformer/data/test-warm/feature-cache-w2v-bert/validation batches=1355 num_workers=16 prefetch_batches=32
2026-04-15 03:36:41,598 | INFO | rank=0 | validation dataloader rust-parquet feature cache index ready entries=3204 elapsed=0.0s
2026-04-15 03:36:41,598 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=2m 33.9s
2026-04-15 03:36:41,598 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 03:36:59,274 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 03:36:59,347 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=17.7s
2026-04-15 03:37:01,754 | INFO | rank=0 | resumed from artifacts/w2v-bert-test2-full/checkpoint_step_last.pt starting_epoch=2 global_step=12000 best_val_wer=0.4044
2026-04-15 03:37:02,016 | INFO | rank=0 | trackio initialized project=w2v-bert-test2-full run_name=w2v-bert-test2-full_20260415_003336_resume-e0002-s00012000 trackio_dir=artifacts/w2v-bert-test2-full/trackio elapsed_since_start=3m 25.1s
2026-04-15 03:37:02,016 | INFO | rank=0 | epoch 2/100 started train_batches=249868 val_batches=1355 grad_accumulation=4
2026-04-15 03:37:12,023 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 backend=rust-parquet num_workers=16 prefetch_batches=32 still waiting elapsed=10.0s
2026-04-15 03:37:42,023 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 backend=rust-parquet num_workers=16 prefetch_batches=32 still waiting elapsed=40.0s
2026-04-15 03:38:07,842 | INFO | rank=0 | epoch=2 first_train_batch_ready elapsed=1m 5.8s data_wait=1m 5.8s batch_size=2 max_feature_frames=191 target_tokens=65
2026-04-15 03:38:33,095 | INFO | rank=0 | epoch=2 waiting for training batch 20/249868 backend=rust-parquet num_workers=16 prefetch_batches=32 still waiting elapsed=5.0s