2026-04-15 01:32:23,439 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://c3053a45e400b8b898.gradio.live full_url=https://c3053a45e400b8b898.gradio.live?project=w2v-bert-test2-full&write_token=dYg0BduJTWOxbtRd6XVcWq-2TSaYo_JPdE6msjGjSkA
2026-04-15 01:32:23,625 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 01:32:23,626 | INFO | rank=0 | auto-resume requested but no checkpoint found at artifacts/w2v-bert-test2-full/checkpoint_last.pt; starting a fresh run
2026-04-15 01:32:23,627 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 01:32:23,627 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 01:32:23,627 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 01:32:23,628 | INFO | rank=0 | resolving dataset sources
2026-04-15 01:32:23,630 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 01:32:23,630 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 01:32:24,447 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=0.8s
2026-04-15 01:32:24,448 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 01:32:26,959 | INFO | rank=0 | dataset audio decode support ready elapsed=3.3s
2026-04-15 01:32:26,959 | INFO | rank=0 | building dataset split audit
2026-04-15 01:32:26,959 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 01:32:27,459 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=64773.9/s elapsed=0.5s
2026-04-15 01:32:27,952 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=65290.0/s elapsed=1.0s
2026-04-15 01:32:28,461 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=64723.1/s elapsed=1.5s
2026-04-15 01:32:28,971 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=64449.5/s elapsed=2.0s
2026-04-15 01:32:29,476 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=64397.8/s elapsed=2.5s
2026-04-15 01:32:29,974 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=64495.5/s elapsed=3.0s
2026-04-15 01:32:30,472 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=64583.6/s elapsed=3.5s
2026-04-15 01:32:30,970 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=64652.6/s elapsed=4.0s
2026-04-15 01:32:31,466 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=64720.0/s elapsed=4.5s
2026-04-15 01:32:31,965 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=64743.5/s elapsed=5.0s
2026-04-15 01:32:32,464 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=64760.0/s elapsed=5.5s
2026-04-15 01:32:32,962 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=64789.8/s elapsed=6.0s
2026-04-15 01:32:33,477 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=64644.0/s elapsed=6.5s
2026-04-15 01:32:34,003 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=64422.9/s elapsed=7.0s
2026-04-15 01:32:34,521 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=64291.0/s elapsed=7.6s
2026-04-15 01:32:35,048 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=64110.5/s elapsed=8.1s
2026-04-15 01:32:35,612 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=63674.5/s elapsed=8.7s
2026-04-15 01:32:36,123 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=63661.8/s elapsed=9.2s
2026-04-15 01:32:36,636 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=63635.7/s elapsed=9.7s
2026-04-15 01:32:37,206 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=63259.0/s elapsed=10.2s
2026-04-15 01:32:37,207 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=10.2s
2026-04-15 01:32:37,207 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 01:32:37,221 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=68371.5/s elapsed=0.0s
2026-04-15 01:32:37,236 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=68169.4/s elapsed=0.0s
2026-04-15 01:32:37,251 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=68064.7/s elapsed=0.0s
2026-04-15 01:32:37,254 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=67788.9/s elapsed=0.0s
2026-04-15 01:32:37,254 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.0s
2026-04-15 01:32:40,479 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=16.8s
2026-04-15 01:32:40,639 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 01:32:40,640 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=False tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 01:32:40,647 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 01:32:43,859 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 num_workers=16 dataloader_worker_threads=1 metadata_workers=16 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 01:32:43,860 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=16 worker_threads=1
2026-04-15 01:32:43,860 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 01:32:44,243 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=84565.0/s elapsed=0.4s
2026-04-15 01:32:44,626 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=84605.6/s elapsed=0.8s
2026-04-15 01:32:45,011 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=84498.2/s elapsed=1.2s
2026-04-15 01:32:45,394 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=84488.9/s elapsed=1.5s
2026-04-15 01:32:45,778 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=84494.2/s elapsed=1.9s
2026-04-15 01:32:46,160 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=84544.4/s elapsed=2.3s
2026-04-15 01:32:46,543 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=84563.4/s elapsed=2.7s
2026-04-15 01:32:46,926 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=84574.2/s elapsed=3.1s
2026-04-15 01:32:47,307 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=84618.5/s elapsed=3.4s
2026-04-15 01:32:47,689 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=84638.8/s elapsed=3.8s
2026-04-15 01:32:48,071 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=84660.1/s elapsed=4.2s
2026-04-15 01:32:48,455 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=84647.6/s elapsed=4.6s
2026-04-15 01:32:48,838 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=84633.8/s elapsed=5.0s
2026-04-15 01:32:49,222 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=84624.5/s elapsed=5.4s
2026-04-15 01:32:49,604 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=84643.4/s elapsed=5.7s
2026-04-15 01:32:49,986 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=84652.7/s elapsed=6.1s
2026-04-15 01:32:50,369 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=84656.0/s elapsed=6.5s
2026-04-15 01:32:50,751 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=84665.6/s elapsed=6.9s
2026-04-15 01:32:51,133 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=84666.9/s elapsed=7.3s
2026-04-15 01:32:51,517 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=84658.3/s elapsed=7.7s
2026-04-15 01:32:57,589 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=13.7s
2026-04-15 01:32:57,589 | INFO | rank=0 | train dataloader metadata ready elapsed=13.7s
2026-04-15 01:32:57,589 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 01:32:58,006 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=77657.4/s elapsed=0.4s
2026-04-15 01:32:58,422 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=77819.6/s elapsed=0.8s
2026-04-15 01:32:58,839 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=77806.7/s elapsed=1.2s
2026-04-15 01:32:59,255 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=77821.9/s elapsed=1.7s
2026-04-15 01:32:59,673 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=77779.6/s elapsed=2.1s
2026-04-15 01:33:00,088 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=77808.5/s elapsed=2.5s
2026-04-15 01:33:00,503 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=77855.4/s elapsed=2.9s
2026-04-15 01:33:00,919 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=77868.5/s elapsed=3.3s
2026-04-15 01:33:01,337 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=77838.0/s elapsed=3.7s
2026-04-15 01:33:01,754 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=77825.3/s elapsed=4.2s
2026-04-15 01:33:02,173 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=77782.6/s elapsed=4.6s
2026-04-15 01:33:02,589 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=77792.2/s elapsed=5.0s
2026-04-15 01:33:03,004 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=77812.5/s elapsed=5.4s
2026-04-15 01:33:03,423 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=77780.5/s elapsed=5.8s
2026-04-15 01:33:03,844 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=77727.5/s elapsed=6.3s
2026-04-15 01:33:04,265 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=77677.8/s elapsed=6.7s
2026-04-15 01:33:04,686 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=77641.9/s elapsed=7.1s
2026-04-15 01:33:05,103 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=77647.9/s elapsed=7.5s
2026-04-15 01:33:05,521 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=77642.0/s elapsed=7.9s
2026-04-15 01:33:05,943 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=77595.3/s elapsed=8.4s
2026-04-15 01:33:05,952 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 01:33:06,614 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=8.4s sort_elapsed=0.7s
2026-04-15 01:33:06,943 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 01:33:08,653 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=151653.3/s elapsed=1.7s
2026-04-15 01:33:08,854 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=152679.5/s elapsed=1.9s
2026-04-15 01:33:09,056 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=153424.9/s elapsed=2.1s
2026-04-15 01:33:09,606 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=133884.9/s elapsed=2.7s
2026-04-15 01:33:09,812 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=135561.2/s elapsed=2.9s
2026-04-15 01:33:10,017 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=137080.4/s elapsed=3.1s
2026-04-15 01:33:10,221 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=138446.6/s elapsed=3.3s
2026-04-15 01:33:10,425 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=139650.7/s elapsed=3.5s
2026-04-15 01:33:10,631 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=140627.1/s elapsed=3.7s
2026-04-15 01:33:11,248 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=128001.3/s elapsed=4.3s
2026-04-15 01:33:11,462 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=129103.3/s elapsed=4.5s
2026-04-15 01:33:11,665 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=130413.7/s elapsed=4.7s
2026-04-15 01:33:11,867 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=131651.2/s elapsed=4.9s
2026-04-15 01:33:18,806 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=16 worker_threads=1
2026-04-15 01:33:18,806 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 01:33:18,818 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=82245.9/s elapsed=0.0s
2026-04-15 01:33:18,831 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=82318.3/s elapsed=0.0s
2026-04-15 01:33:18,842 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=82941.1/s elapsed=0.0s
2026-04-15 01:33:18,845 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=82523.6/s elapsed=0.0s
2026-04-15 01:33:18,875 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 01:33:18,876 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 01:33:18,876 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 01:33:18,888 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=80054.4/s elapsed=0.0s
2026-04-15 01:33:18,901 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=80010.8/s elapsed=0.0s
2026-04-15 01:33:18,913 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=79847.0/s elapsed=0.0s
2026-04-15 01:33:18,916 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=79399.1/s elapsed=0.0s
2026-04-15 01:33:18,916 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 01:33:18,918 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.0s sort_elapsed=0.0s
2026-04-15 01:33:18,918 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 01:33:18,923 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=189519.9/s elapsed=0.0s
2026-04-15 01:33:18,930 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=176308.7/s elapsed=0.0s
2026-04-15 01:33:18,936 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=172788.1/s elapsed=0.0s
2026-04-15 01:33:18,937 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=171438.5/s elapsed=0.0s
2026-04-15 01:33:18,937 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=38.3s
2026-04-15 01:33:18,941 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 01:33:21,493 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 01:33:21,578 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=2.6s
2026-04-15 01:33:21,858 | INFO | rank=0 | trackio initialized project=w2v-bert-test2-full run_name=w2v-bert-test2-full_20260414_223220 trackio_dir=artifacts/w2v-bert-test2-full/trackio elapsed_since_start=1m 1.5s
2026-04-15 01:33:21,859 | INFO | rank=0 | epoch 1/100 started train_batches=249868 val_batches=1355 grad_accumulation=4