2026-04-15 02:13:54,407 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://c48f55dad8998ec769.gradio.live full_url=https://c48f55dad8998ec769.gradio.live?project=w2v-bert-test2-full&write_token=NyTE-83co1qKKPXgKoQxNsGcIXQn0ZtwIFmkewV2dLo
2026-04-15 02:13:54,598 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 02:14:50,327 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 02:14:52,292 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 02:14:52,294 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 02:14:52,294 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 02:14:52,295 | INFO | rank=0 | resolving dataset sources
2026-04-15 02:14:52,297 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 02:14:52,297 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 02:14:53,903 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=1.6s
2026-04-15 02:14:53,905 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 02:14:55,674 | INFO | rank=0 | dataset audio decode support ready elapsed=3.4s
2026-04-15 02:14:55,675 | INFO | rank=0 | building dataset split audit
2026-04-15 02:14:55,675 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 02:14:56,306 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=51318.1/s elapsed=0.6s
2026-04-15 02:14:56,935 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=51424.0/s elapsed=1.3s
2026-04-15 02:14:57,591 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=50741.8/s elapsed=1.9s
2026-04-15 02:14:58,248 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=50392.4/s elapsed=2.6s
2026-04-15 02:14:58,901 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=50239.1/s elapsed=3.2s
2026-04-15 02:14:59,537 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=50351.6/s elapsed=3.9s
2026-04-15 02:15:00,175 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=50416.6/s elapsed=4.5s
2026-04-15 02:15:00,817 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=50427.6/s elapsed=5.1s
2026-04-15 02:15:01,459 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=50428.2/s elapsed=5.8s
2026-04-15 02:15:02,102 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=50430.5/s elapsed=6.4s
2026-04-15 02:15:02,740 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=50463.8/s elapsed=7.1s
2026-04-15 02:15:03,390 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=50409.9/s elapsed=7.7s
2026-04-15 02:15:04,046 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=50331.4/s elapsed=8.4s
2026-04-15 02:15:04,725 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=50138.0/s elapsed=9.1s
2026-04-15 02:15:05,403 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=49975.0/s elapsed=9.7s
2026-04-15 02:15:06,095 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=49768.9/s elapsed=10.4s
2026-04-15 02:15:06,851 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=49300.7/s elapsed=11.2s
2026-04-15 02:15:07,513 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=49282.2/s elapsed=11.8s
2026-04-15 02:15:08,179 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=49250.2/s elapsed=12.5s
2026-04-15 02:15:08,926 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=48917.7/s elapsed=13.3s
2026-04-15 02:15:08,927 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=13.3s
2026-04-15 02:15:08,927 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 02:15:08,948 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=47373.1/s elapsed=0.0s
2026-04-15 02:15:08,968 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=48770.2/s elapsed=0.0s
2026-04-15 02:15:08,988 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=49147.9/s elapsed=0.1s
2026-04-15 02:15:08,993 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=48993.5/s elapsed=0.1s
2026-04-15 02:15:08,993 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.1s
2026-04-15 02:15:13,148 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=20.9s
2026-04-15 02:15:13,357 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 02:16:10,252 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=56.9s
2026-04-15 02:16:10,253 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 02:16:10,292 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 02:16:13,514 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 dataloader_backend=yomikomi num_workers=12 dataloader_worker_threads=1 metadata_workers=8 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 02:16:13,515 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=12 worker_threads=1 backend=yomikomi
2026-04-15 02:16:13,515 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 02:16:13,902 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=83891.3/s elapsed=0.4s
2026-04-15 02:16:14,290 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=83678.7/s elapsed=0.8s
2026-04-15 02:16:14,677 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=83685.4/s elapsed=1.2s
2026-04-15 02:16:15,064 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=83731.4/s elapsed=1.5s
2026-04-15 02:16:15,450 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=83767.7/s elapsed=1.9s
2026-04-15 02:16:15,836 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=83780.1/s elapsed=2.3s
2026-04-15 02:16:16,226 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=83687.9/s elapsed=2.7s
2026-04-15 02:16:16,614 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=83680.0/s elapsed=3.1s
2026-04-15 02:16:17,000 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=83714.7/s elapsed=3.5s
2026-04-15 02:16:17,387 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=83709.8/s elapsed=3.9s
2026-04-15 02:16:17,777 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=83657.6/s elapsed=4.3s
2026-04-15 02:16:18,164 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=83667.1/s elapsed=4.6s
2026-04-15 02:16:18,551 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=83678.4/s elapsed=5.0s
2026-04-15 02:16:18,939 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=83668.9/s elapsed=5.4s
2026-04-15 02:16:19,328 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=83639.5/s elapsed=5.8s
2026-04-15 02:16:19,716 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=83639.2/s elapsed=6.2s
2026-04-15 02:16:20,104 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=83625.8/s elapsed=6.6s
2026-04-15 02:16:20,498 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=83550.8/s elapsed=7.0s
2026-04-15 02:16:20,887 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=83533.7/s elapsed=7.4s
2026-04-15 02:16:21,274 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=83544.5/s elapsed=7.8s
2026-04-15 02:16:27,396 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=13.9s
2026-04-15 02:16:27,396 | INFO | rank=0 | train dataloader metadata ready elapsed=13.9s
2026-04-15 02:16:27,397 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 02:16:27,827 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=75308.6/s elapsed=0.4s
2026-04-15 02:16:28,258 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=75280.7/s elapsed=0.9s
2026-04-15 02:16:28,690 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=75185.8/s elapsed=1.3s
2026-04-15 02:16:29,118 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=75310.9/s elapsed=1.7s
2026-04-15 02:16:29,544 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=75463.4/s elapsed=2.1s
2026-04-15 02:16:29,969 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=75622.6/s elapsed=2.6s
2026-04-15 02:16:30,392 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=75754.6/s elapsed=3.0s
2026-04-15 02:16:30,818 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=75801.0/s elapsed=3.4s
2026-04-15 02:16:31,267 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=75380.4/s elapsed=3.9s
2026-04-15 02:16:31,697 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=75374.6/s elapsed=4.3s
2026-04-15 02:16:32,122 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=75459.6/s elapsed=4.7s
2026-04-15 02:16:32,549 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=75492.5/s elapsed=5.2s
2026-04-15 02:16:32,980 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=75475.0/s elapsed=5.6s
2026-04-15 02:16:33,408 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=75487.9/s elapsed=6.0s
2026-04-15 02:16:33,838 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=75478.6/s elapsed=6.4s
2026-04-15 02:16:34,288 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=75255.5/s elapsed=6.9s
2026-04-15 02:16:34,728 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=75163.0/s elapsed=7.3s
2026-04-15 02:16:35,168 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=75077.7/s elapsed=7.8s
2026-04-15 02:16:35,608 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=74999.8/s elapsed=8.2s
2026-04-15 02:16:36,050 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=74911.8/s elapsed=8.7s
2026-04-15 02:16:36,050 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 02:16:37,605 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=8.7s sort_elapsed=1.6s
2026-04-15 02:16:38,367 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 02:16:42,031 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=70762.8/s elapsed=3.7s
2026-04-15 02:16:42,453 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=71390.5/s elapsed=4.1s
2026-04-15 02:16:44,241 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=55172.9/s elapsed=5.9s
2026-04-15 02:16:44,502 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=58106.3/s elapsed=6.1s
2026-04-15 02:16:44,759 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=60844.9/s elapsed=6.4s
2026-04-15 02:16:45,015 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=63377.8/s elapsed=6.6s
2026-04-15 02:16:45,272 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=65707.7/s elapsed=6.9s
2026-04-15 02:16:45,526 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=67905.2/s elapsed=7.2s
2026-04-15 02:16:47,269 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=58254.2/s elapsed=8.9s
2026-04-15 02:16:47,476 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=60485.2/s elapsed=9.1s
2026-04-15 02:16:47,678 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=62655.9/s elapsed=9.3s
2026-04-15 02:16:47,878 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=64747.8/s elapsed=9.5s
2026-04-15 02:16:48,075 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=66769.5/s elapsed=9.7s
2026-04-15 02:16:55,323 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=12 worker_threads=1 backend=yomikomi
2026-04-15 02:16:55,323 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 02:16:55,336 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=78553.1/s elapsed=0.0s
2026-04-15 02:16:55,348 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=80531.7/s elapsed=0.0s
2026-04-15 02:16:55,360 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=81672.2/s elapsed=0.0s
2026-04-15 02:16:55,363 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=81441.5/s elapsed=0.0s
2026-04-15 02:16:55,394 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 02:16:55,394 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 02:16:55,394 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 02:16:55,407 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=75447.6/s elapsed=0.0s
2026-04-15 02:16:55,420 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=76435.2/s elapsed=0.0s
2026-04-15 02:16:55,433 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=76997.8/s elapsed=0.0s
2026-04-15 02:16:55,436 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=76964.0/s elapsed=0.0s
2026-04-15 02:16:55,436 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 02:16:55,437 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.0s sort_elapsed=0.0s
2026-04-15 02:16:55,438 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 02:16:55,443 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=186182.8/s elapsed=0.0s
2026-04-15 02:16:55,449 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=177555.4/s elapsed=0.0s
2026-04-15 02:16:55,455 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=175041.9/s elapsed=0.0s
2026-04-15 02:16:55,456 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=173819.7/s elapsed=0.0s
2026-04-15 02:16:55,457 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=45.2s
2026-04-15 02:16:55,457 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 02:18:09,861 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 02:18:10,018 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=1m 14.6s
2026-04-15 02:19:04,980 | INFO | rank=0 | resumed from artifacts/w2v-bert-test2-full/checkpoint_step_last.pt starting_epoch=2 global_step=12000 best_val_wer=0.4044
2026-04-15 02:19:05,263 | INFO | rank=0 | trackio initialized project=w2v-bert-test2-full run_name=w2v-bert-test2-full_20260414_231349_resume-e0002-s00012000 trackio_dir=artifacts/w2v-bert-test2-full/trackio elapsed_since_start=5m 15.7s
2026-04-15 02:19:05,264 | INFO | rank=0 | epoch 2/100 started train_batches=249868 val_batches=1355 grad_accumulation=4