2026-04-15 03:21:20,951 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://e3a165a06be1c820bc.gradio.live full_url=https://e3a165a06be1c820bc.gradio.live?project=w2v-bert-test2-full&write_token=DNvvx94gVOUgppujompeVRh-0-byboGAnxJVwF_TXXw
2026-04-15 03:21:21,133 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 03:21:28,168 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 03:21:28,978 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 03:21:28,979 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 03:21:28,979 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 03:21:28,979 | INFO | rank=0 | resolving dataset sources
2026-04-15 03:21:28,980 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 03:21:28,980 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 03:21:29,379 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=0.4s
2026-04-15 03:21:29,380 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 03:21:30,234 | INFO | rank=0 | dataset audio decode support ready elapsed=1.3s
2026-04-15 03:21:30,235 | INFO | rank=0 | building dataset split audit
2026-04-15 03:21:30,235 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 03:21:30,734 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=64922.5/s elapsed=0.5s
2026-04-15 03:21:31,229 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=65205.5/s elapsed=1.0s
2026-04-15 03:21:31,734 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=64884.2/s elapsed=1.5s
2026-04-15 03:21:32,233 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=64871.5/s elapsed=2.0s
2026-04-15 03:21:32,731 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=64925.5/s elapsed=2.5s
2026-04-15 03:21:33,225 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=65033.3/s elapsed=3.0s
2026-04-15 03:21:33,724 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=65029.6/s elapsed=3.5s
2026-04-15 03:21:34,223 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=65025.8/s elapsed=4.0s
2026-04-15 03:21:34,744 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=64687.3/s elapsed=4.5s
2026-04-15 03:21:35,351 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=63351.4/s elapsed=5.1s
2026-04-15 03:21:35,958 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=62300.8/s elapsed=5.7s
2026-04-15 03:21:36,563 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=61461.0/s elapsed=6.3s
2026-04-15 03:21:37,191 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=60576.5/s elapsed=7.0s
2026-04-15 03:21:37,831 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=59737.7/s elapsed=7.6s
2026-04-15 03:21:38,466 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=59069.8/s elapsed=8.2s
2026-04-15 03:21:39,117 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=58384.5/s elapsed=8.9s
2026-04-15 03:21:39,817 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=57501.1/s elapsed=9.6s
2026-04-15 03:21:40,444 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=57149.2/s elapsed=10.2s
2026-04-15 03:21:41,076 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=56803.7/s elapsed=10.8s
2026-04-15 03:21:41,762 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=56236.8/s elapsed=11.5s
2026-04-15 03:21:41,762 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=11.5s
2026-04-15 03:21:41,762 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 03:21:41,780 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=55000.8/s elapsed=0.0s
2026-04-15 03:21:41,799 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=55043.5/s elapsed=0.0s
2026-04-15 03:21:41,817 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=55111.5/s elapsed=0.1s
2026-04-15 03:21:41,821 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=54890.5/s elapsed=0.1s
2026-04-15 03:21:41,821 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.1s
2026-04-15 03:21:45,861 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=16.9s
2026-04-15 03:21:46,018 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 03:21:51,986 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=6.0s
2026-04-15 03:21:51,986 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 03:21:52,012 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 03:21:55,231 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 dataloader_backend=rust-parquet num_workers=16 dataloader_worker_threads=1 metadata_workers=8 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 03:21:55,231 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=16 worker_threads=1 backend=rust-parquet
2026-04-15 03:21:55,231 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 03:21:55,615 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=84306.2/s elapsed=0.4s
2026-04-15 03:21:56,000 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=84309.9/s elapsed=0.8s
2026-04-15 03:21:56,386 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=84184.4/s elapsed=1.2s
2026-04-15 03:21:56,772 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=84142.5/s elapsed=1.5s
2026-04-15 03:21:57,157 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=84129.0/s elapsed=1.9s
2026-04-15 03:21:57,541 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=84182.7/s elapsed=2.3s
2026-04-15 03:21:57,925 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=84227.8/s elapsed=2.7s
2026-04-15 03:21:58,310 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=84225.1/s elapsed=3.1s
2026-04-15 03:21:58,695 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=84222.4/s elapsed=3.5s
2026-04-15 03:21:59,079 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=84220.6/s elapsed=3.8s
2026-04-15 03:21:59,464 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=84228.3/s elapsed=4.2s
2026-04-15 03:21:59,848 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=84232.8/s elapsed=4.6s
2026-04-15 03:22:00,233 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=84231.2/s elapsed=5.0s
2026-04-15 03:22:00,619 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=84219.0/s elapsed=5.4s
2026-04-15 03:22:01,003 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=84230.1/s elapsed=5.8s
2026-04-15 03:22:01,388 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=84233.7/s elapsed=6.2s
2026-04-15 03:22:01,775 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=84204.0/s elapsed=6.5s
2026-04-15 03:22:02,160 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=84196.0/s elapsed=6.9s
2026-04-15 03:22:02,554 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=84095.0/s elapsed=7.3s
2026-04-15 03:22:02,939 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=84099.5/s elapsed=7.7s
2026-04-15 03:22:09,136 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=13.9s
2026-04-15 03:22:09,136 | INFO | rank=0 | train dataloader metadata ready elapsed=13.9s
2026-04-15 03:22:09,136 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 03:22:09,545 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=79365.4/s elapsed=0.4s
2026-04-15 03:22:09,952 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=79506.3/s elapsed=0.8s
2026-04-15 03:22:10,359 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=79524.3/s elapsed=1.2s
2026-04-15 03:22:10,768 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=79454.8/s elapsed=1.6s
2026-04-15 03:22:11,182 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=79229.1/s elapsed=2.0s
2026-04-15 03:22:11,590 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=79249.9/s elapsed=2.5s
2026-04-15 03:22:11,998 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=79285.3/s elapsed=2.9s
2026-04-15 03:22:12,409 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=79244.2/s elapsed=3.3s
2026-04-15 03:22:12,817 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=79259.0/s elapsed=3.7s
2026-04-15 03:22:13,232 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=79134.2/s elapsed=4.1s
2026-04-15 03:22:13,639 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=79189.0/s elapsed=4.5s
2026-04-15 03:22:14,045 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=79231.8/s elapsed=4.9s
2026-04-15 03:22:14,452 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=79261.0/s elapsed=5.3s
2026-04-15 03:22:14,863 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=79244.6/s elapsed=5.7s
2026-04-15 03:22:15,271 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=79254.0/s elapsed=6.1s
2026-04-15 03:22:15,680 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=79252.4/s elapsed=6.5s
2026-04-15 03:22:16,090 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=79237.9/s elapsed=7.0s
2026-04-15 03:22:16,498 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=79249.6/s elapsed=7.4s
2026-04-15 03:22:16,906 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=79261.0/s elapsed=7.8s
2026-04-15 03:22:17,315 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=79256.1/s elapsed=8.2s
2026-04-15 03:22:17,316 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 03:22:17,950 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=8.2s sort_elapsed=0.6s
2026-04-15 03:22:18,272 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 03:22:19,969 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=152732.5/s elapsed=1.7s
2026-04-15 03:22:20,167 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=153912.5/s elapsed=1.9s
2026-04-15 03:22:20,366 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=154725.3/s elapsed=2.1s
2026-04-15 03:22:20,912 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=135023.4/s elapsed=2.6s
2026-04-15 03:22:21,115 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=136784.5/s elapsed=2.8s
2026-04-15 03:22:21,315 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=138460.8/s elapsed=3.0s
2026-04-15 03:22:21,514 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=139944.1/s elapsed=3.2s
2026-04-15 03:22:21,715 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=141200.6/s elapsed=3.4s
2026-04-15 03:22:22,321 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=128062.1/s elapsed=4.0s
2026-04-15 03:22:22,524 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=129574.9/s elapsed=4.3s
2026-04-15 03:22:22,723 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=131055.8/s elapsed=4.5s
2026-04-15 03:22:22,922 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=132440.1/s elapsed=4.6s
2026-04-15 03:22:23,119 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=133744.0/s elapsed=4.8s