2026-04-15 03:13:33,425 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://76c6876b3bb7f376b4.gradio.live full_url=https://76c6876b3bb7f376b4.gradio.live?project=w2v-bert-test2-full&write_token=mGhTSffvPnAMhFbTVo6k2DRZS5Y5fKM7WWHwkgeiDrk
2026-04-15 03:13:33,612 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 03:13:52,690 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 03:13:53,485 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 03:13:53,485 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 03:13:53,485 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 03:13:53,485 | INFO | rank=0 | resolving dataset sources
2026-04-15 03:13:53,486 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 03:13:53,486 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 03:13:53,866 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=0.4s
2026-04-15 03:13:53,866 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 03:13:54,935 | INFO | rank=0 | dataset audio decode support ready elapsed=1.4s
2026-04-15 03:13:54,936 | INFO | rank=0 | building dataset split audit
2026-04-15 03:13:54,936 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 03:13:55,419 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=67033.5/s elapsed=0.5s
2026-04-15 03:13:55,906 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=66782.8/s elapsed=1.0s
2026-04-15 03:13:56,399 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=66458.7/s elapsed=1.5s
2026-04-15 03:13:56,901 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=65953.0/s elapsed=2.0s
2026-04-15 03:13:57,394 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=65914.4/s elapsed=2.5s
2026-04-15 03:13:57,883 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=65988.9/s elapsed=2.9s
2026-04-15 03:13:58,369 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=66076.8/s elapsed=3.4s
2026-04-15 03:13:58,860 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=66081.7/s elapsed=3.9s
2026-04-15 03:13:59,351 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=66070.0/s elapsed=4.4s
2026-04-15 03:13:59,840 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=66082.3/s elapsed=4.9s
2026-04-15 03:14:00,330 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=66089.0/s elapsed=5.4s
2026-04-15 03:14:00,821 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=66084.1/s elapsed=5.9s
2026-04-15 03:14:01,326 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=65933.2/s elapsed=6.4s
2026-04-15 03:14:01,842 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=65707.0/s elapsed=6.9s
2026-04-15 03:14:02,352 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=65553.7/s elapsed=7.4s
2026-04-15 03:14:02,879 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=65287.1/s elapsed=7.9s
2026-04-15 03:14:03,445 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=64755.1/s elapsed=8.5s
2026-04-15 03:14:03,954 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=64689.4/s elapsed=9.0s
2026-04-15 03:14:04,468 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=64606.2/s elapsed=9.5s
2026-04-15 03:14:05,023 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=64265.1/s elapsed=10.1s
2026-04-15 03:14:05,023 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=10.1s
2026-04-15 03:14:05,023 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 03:14:05,038 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=67704.4/s elapsed=0.0s
2026-04-15 03:14:05,053 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=67310.5/s elapsed=0.0s
2026-04-15 03:14:05,068 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=67191.0/s elapsed=0.0s
2026-04-15 03:14:05,071 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=67066.7/s elapsed=0.0s
2026-04-15 03:14:05,071 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.0s
2026-04-15 03:14:08,350 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=14.9s
2026-04-15 03:14:08,480 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 03:14:14,345 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=5.9s
2026-04-15 03:14:14,346 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 03:14:14,371 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 03:14:17,638 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 dataloader_backend=rust-parquet num_workers=16 dataloader_worker_threads=1 metadata_workers=8 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 03:14:17,638 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=16 worker_threads=1 backend=rust-parquet
2026-04-15 03:14:17,638 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 03:14:18,026 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=83487.9/s elapsed=0.4s
2026-04-15 03:14:18,414 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=83510.3/s elapsed=0.8s
2026-04-15 03:14:18,803 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=83457.5/s elapsed=1.2s
2026-04-15 03:14:19,197 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=83167.9/s elapsed=1.6s
2026-04-15 03:14:19,584 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=83258.8/s elapsed=1.9s
2026-04-15 03:14:19,972 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=83328.9/s elapsed=2.3s
2026-04-15 03:14:20,358 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=83405.8/s elapsed=2.7s
2026-04-15 03:14:20,745 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=83466.1/s elapsed=3.1s
2026-04-15 03:14:21,133 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=83474.2/s elapsed=3.5s
2026-04-15 03:14:21,520 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=83490.8/s elapsed=3.9s
2026-04-15 03:14:21,908 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=83500.0/s elapsed=4.3s
2026-04-15 03:14:22,295 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=83526.5/s elapsed=4.7s
2026-04-15 03:14:22,683 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=83524.7/s elapsed=5.0s
2026-04-15 03:14:23,072 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=83498.4/s elapsed=5.4s
2026-04-15 03:14:23,462 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=83473.4/s elapsed=5.8s
2026-04-15 03:14:23,851 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=83466.6/s elapsed=6.2s
2026-04-15 03:14:24,240 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=83465.9/s elapsed=6.6s
2026-04-15 03:14:24,628 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=83469.5/s elapsed=7.0s
2026-04-15 03:14:25,016 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=83463.6/s elapsed=7.4s
2026-04-15 03:14:25,404 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=83467.3/s elapsed=7.8s
2026-04-15 03:14:31,647 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=14.0s
2026-04-15 03:14:31,647 | INFO | rank=0 | train dataloader metadata ready elapsed=14.0s
2026-04-15 03:14:31,648 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 03:14:32,061 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=78438.1/s elapsed=0.4s
2026-04-15 03:14:32,478 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=78077.5/s elapsed=0.8s
2026-04-15 03:14:32,896 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=77881.9/s elapsed=1.2s
2026-04-15 03:14:33,314 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=77782.6/s elapsed=1.7s
2026-04-15 03:14:33,732 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=77761.6/s elapsed=2.1s
2026-04-15 03:14:34,147 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=77804.4/s elapsed=2.5s
2026-04-15 03:14:34,563 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=77830.7/s elapsed=2.9s
2026-04-15 03:14:34,978 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=77847.8/s elapsed=3.3s
2026-04-15 03:14:35,394 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=77862.4/s elapsed=3.7s
2026-04-15 03:14:35,810 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=77876.8/s elapsed=4.2s
2026-04-15 03:14:36,225 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=77884.6/s elapsed=4.6s
2026-04-15 03:14:36,640 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=77904.8/s elapsed=5.0s
2026-04-15 03:14:37,057 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=77897.9/s elapsed=5.4s
2026-04-15 03:14:37,474 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=77881.1/s elapsed=5.8s
2026-04-15 03:14:37,889 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=77896.5/s elapsed=6.2s
2026-04-15 03:14:38,303 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=77916.6/s elapsed=6.7s
2026-04-15 03:14:38,719 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=77915.5/s elapsed=7.1s
2026-04-15 03:14:39,133 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=77938.7/s elapsed=7.5s
2026-04-15 03:14:39,546 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=77964.2/s elapsed=7.9s
2026-04-15 03:14:39,961 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=77970.7/s elapsed=8.3s
2026-04-15 03:14:39,962 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 03:14:40,600 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=8.3s sort_elapsed=0.6s
2026-04-15 03:14:40,923 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 03:14:42,661 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=149204.0/s elapsed=1.7s
2026-04-15 03:14:42,863 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=150383.7/s elapsed=1.9s
2026-04-15 03:14:43,404 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=130620.3/s elapsed=2.5s
2026-04-15 03:14:43,603 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=133007.3/s elapsed=2.7s
2026-04-15 03:14:43,802 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=135095.7/s elapsed=2.9s
2026-04-15 03:14:44,000 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=136914.7/s elapsed=3.1s
2026-04-15 03:14:44,200 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=138481.5/s elapsed=3.3s
2026-04-15 03:14:44,801 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=125363.1/s elapsed=3.9s
2026-04-15 03:14:45,003 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=127096.1/s elapsed=4.1s
2026-04-15 03:14:45,203 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=128746.0/s elapsed=4.3s
2026-04-15 03:14:45,402 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=130263.2/s elapsed=4.5s
2026-04-15 03:14:45,599 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=131698.3/s elapsed=4.7s
2026-04-15 03:14:45,796 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=133017.8/s elapsed=4.9s