2026-04-15 02:05:57,028 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://c48baa28476b9b2314.gradio.live full_url=https://c48baa28476b9b2314.gradio.live?project=w2v-bert-test2-full&write_token=kDXmuF5_5lV1pudArQ9Yb20xICSNNOJlGrBYj5noZaA
2026-04-15 02:05:57,226 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 02:06:58,397 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 02:07:00,184 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 02:07:00,185 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 02:07:00,186 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 02:07:00,186 | INFO | rank=0 | resolving dataset sources
2026-04-15 02:07:00,188 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 02:07:00,188 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 02:07:01,751 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=1.6s
2026-04-15 02:07:01,752 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 02:07:03,596 | INFO | rank=0 | dataset audio decode support ready elapsed=3.4s
2026-04-15 02:07:03,597 | INFO | rank=0 | building dataset split audit
2026-04-15 02:07:03,597 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 02:07:04,125 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=61403.6/s elapsed=0.5s
2026-04-15 02:07:04,653 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=61366.4/s elapsed=1.1s
2026-04-15 02:07:05,201 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=60603.8/s elapsed=1.6s
2026-04-15 02:07:05,749 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=60249.3/s elapsed=2.2s
2026-04-15 02:07:06,293 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=60106.2/s elapsed=2.7s
2026-04-15 02:07:06,835 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=60063.4/s elapsed=3.2s
2026-04-15 02:07:07,373 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=60080.0/s elapsed=3.8s
2026-04-15 02:07:07,914 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=60057.3/s elapsed=4.3s
2026-04-15 02:07:08,456 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=60031.7/s elapsed=4.9s
2026-04-15 02:07:08,999 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=60001.0/s elapsed=5.4s
2026-04-15 02:07:09,521 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=60179.3/s elapsed=5.9s
2026-04-15 02:07:10,033 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=60432.2/s elapsed=6.4s
2026-04-15 02:07:10,568 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=60446.2/s elapsed=7.0s
2026-04-15 02:07:11,123 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=60295.2/s elapsed=7.5s
2026-04-15 02:07:11,671 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=60213.0/s elapsed=8.1s
2026-04-15 02:07:12,250 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=59931.8/s elapsed=8.7s
2026-04-15 02:07:12,872 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=59408.0/s elapsed=9.3s
2026-04-15 02:07:13,420 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=59389.7/s elapsed=9.8s
2026-04-15 02:07:13,972 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=59357.3/s elapsed=10.4s
2026-04-15 02:07:14,579 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=59028.5/s elapsed=11.0s
2026-04-15 02:07:14,579 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=11.0s
2026-04-15 02:07:14,579 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 02:07:14,595 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=61793.1/s elapsed=0.0s
2026-04-15 02:07:14,611 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=62773.3/s elapsed=0.0s
2026-04-15 02:07:14,627 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=63023.1/s elapsed=0.0s
2026-04-15 02:07:14,630 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=62924.6/s elapsed=0.1s
2026-04-15 02:07:14,630 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.1s
2026-04-15 02:07:17,865 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=17.7s
2026-04-15 02:07:18,036 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 02:08:17,685 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=59.6s
2026-04-15 02:08:17,686 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 02:08:17,717 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 02:08:21,021 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 dataloader_backend=yomikomi num_workers=12 dataloader_worker_threads=1 metadata_workers=8 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 02:08:21,021 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=12 worker_threads=1 backend=yomikomi
2026-04-15 02:08:21,021 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 02:08:21,405 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=84405.6/s elapsed=0.4s
2026-04-15 02:08:21,790 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=84342.5/s elapsed=0.8s
2026-04-15 02:08:22,174 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=84343.1/s elapsed=1.2s
2026-04-15 02:08:22,558 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=84336.2/s elapsed=1.5s
2026-04-15 02:08:22,942 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=84363.3/s elapsed=1.9s
2026-04-15 02:08:23,327 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=84327.9/s elapsed=2.3s
2026-04-15 02:08:23,718 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=84126.9/s elapsed=2.7s
2026-04-15 02:08:24,103 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=84131.6/s elapsed=3.1s
2026-04-15 02:08:24,482 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=84288.0/s elapsed=3.5s
2026-04-15 02:08:24,859 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=84442.4/s elapsed=3.8s
2026-04-15 02:08:25,236 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=84587.6/s elapsed=4.2s
2026-04-15 02:08:25,612 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=84713.0/s elapsed=4.6s
2026-04-15 02:08:25,996 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=84695.5/s elapsed=5.0s
2026-04-15 02:08:26,375 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=84750.3/s elapsed=5.4s
2026-04-15 02:08:26,754 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=84806.0/s elapsed=5.7s
2026-04-15 02:08:27,131 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=84874.7/s elapsed=6.1s
2026-04-15 02:08:27,509 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=84928.1/s elapsed=6.5s
2026-04-15 02:08:27,888 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=84957.4/s elapsed=6.9s
2026-04-15 02:08:28,270 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=84958.6/s elapsed=7.2s
2026-04-15 02:08:28,650 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=84974.5/s elapsed=7.6s
2026-04-15 02:08:34,752 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=13.7s
2026-04-15 02:08:34,752 | INFO | rank=0 | train dataloader metadata ready elapsed=13.7s
2026-04-15 02:08:34,753 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 02:08:35,176 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=76553.5/s elapsed=0.4s
2026-04-15 02:08:35,613 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=75361.6/s elapsed=0.9s
2026-04-15 02:08:36,039 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=75590.5/s elapsed=1.3s
2026-04-15 02:08:36,456 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=76106.3/s elapsed=1.7s
2026-04-15 02:08:36,872 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=76481.0/s elapsed=2.1s
2026-04-15 02:08:37,288 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=76708.4/s elapsed=2.5s
2026-04-15 02:08:37,704 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=76864.6/s elapsed=3.0s
2026-04-15 02:08:38,123 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=76933.0/s elapsed=3.4s
2026-04-15 02:08:38,541 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=77012.4/s elapsed=3.8s
2026-04-15 02:08:38,957 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=77095.4/s elapsed=4.2s
2026-04-15 02:08:39,374 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=77148.9/s elapsed=4.6s
2026-04-15 02:08:39,789 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=77226.7/s elapsed=5.0s
2026-04-15 02:08:40,206 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=77260.8/s elapsed=5.5s
2026-04-15 02:08:40,623 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=77298.5/s elapsed=5.9s
2026-04-15 02:08:41,063 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=77042.5/s elapsed=6.3s
2026-04-15 02:08:41,482 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=77063.1/s elapsed=6.7s
2026-04-15 02:08:41,901 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=77080.1/s elapsed=7.1s
2026-04-15 02:08:42,324 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=77060.2/s elapsed=7.6s
2026-04-15 02:08:42,752 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=76988.3/s elapsed=8.0s
2026-04-15 02:08:43,188 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=76853.0/s elapsed=8.4s
2026-04-15 02:08:43,188 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 02:08:44,668 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=8.4s sort_elapsed=1.5s
2026-04-15 02:08:45,435 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 02:08:48,542 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=83479.2/s elapsed=3.1s
2026-04-15 02:08:48,778 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=87273.0/s elapsed=3.3s
2026-04-15 02:08:50,522 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=63712.8/s elapsed=5.1s
2026-04-15 02:08:50,735 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=67271.8/s elapsed=5.3s
2026-04-15 02:08:50,979 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=70157.3/s elapsed=5.5s
2026-04-15 02:08:51,252 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=72435.3/s elapsed=5.8s
2026-04-15 02:08:51,521 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=74566.4/s elapsed=6.1s
2026-04-15 02:08:51,770 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=76752.2/s elapsed=6.3s
2026-04-15 02:08:53,884 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=61382.5/s elapsed=8.4s
2026-04-15 02:08:54,094 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=63635.7/s elapsed=8.7s
2026-04-15 02:08:54,300 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=65811.3/s elapsed=8.9s
2026-04-15 02:08:54,500 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=67932.0/s elapsed=9.1s
2026-04-15 02:08:54,700 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=69963.7/s elapsed=9.3s
2026-04-15 02:09:01,768 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=12 worker_threads=1 backend=yomikomi
2026-04-15 02:09:01,769 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 02:09:01,781 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=80224.1/s elapsed=0.0s
2026-04-15 02:09:01,793 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=81560.2/s elapsed=0.0s
2026-04-15 02:09:01,805 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=82085.3/s elapsed=0.0s
2026-04-15 02:09:01,808 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=81750.5/s elapsed=0.0s
2026-04-15 02:09:01,838 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 02:09:01,838 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 02:09:01,839 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 02:09:01,851 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=80860.2/s elapsed=0.0s
2026-04-15 02:09:01,864 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=79143.7/s elapsed=0.0s
2026-04-15 02:09:01,876 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=79375.2/s elapsed=0.0s
2026-04-15 02:09:01,879 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=79074.4/s elapsed=0.0s
2026-04-15 02:09:01,879 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 02:09:01,881 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.0s sort_elapsed=0.0s
2026-04-15 02:09:01,881 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 02:09:01,887 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=193196.2/s elapsed=0.0s
2026-04-15 02:09:01,893 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=179451.1/s elapsed=0.0s
2026-04-15 02:09:01,898 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=176409.8/s elapsed=0.0s
2026-04-15 02:09:01,900 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=174458.8/s elapsed=0.0s
2026-04-15 02:09:01,900 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=44.2s
2026-04-15 02:09:01,900 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 02:10:23,345 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 02:10:23,505 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=1m 21.6s