2026-04-15 01:38:11,822 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://e6c604419c13db812b.gradio.live full_url=https://e6c604419c13db812b.gradio.live?project=w2v-bert-test2-full&write_token=H77M57qdinqcG6YPpm1CteKQF_DD6-z66zA3N0zmldA
2026-04-15 01:38:12,005 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 01:38:26,300 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 01:38:27,175 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 01:38:27,176 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 01:38:27,176 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 01:38:27,176 | INFO | rank=0 | resolving dataset sources
2026-04-15 01:38:27,177 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 01:38:27,177 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 01:38:27,535 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=0.4s
2026-04-15 01:38:27,535 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 01:38:28,444 | INFO | rank=0 | dataset audio decode support ready elapsed=1.3s
2026-04-15 01:38:28,444 | INFO | rank=0 | building dataset split audit
2026-04-15 01:38:28,444 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 01:38:28,923 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=67664.1/s elapsed=0.5s
2026-04-15 01:38:29,403 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=67590.1/s elapsed=1.0s
2026-04-15 01:38:29,898 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=66854.2/s elapsed=1.5s
2026-04-15 01:38:30,390 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=66631.4/s elapsed=1.9s
2026-04-15 01:38:30,879 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=66553.8/s elapsed=2.4s
2026-04-15 01:38:31,358 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=66737.7/s elapsed=2.9s
2026-04-15 01:38:31,838 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=66851.2/s elapsed=3.4s
2026-04-15 01:38:32,323 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=66849.2/s elapsed=3.9s
2026-04-15 01:38:32,807 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=66860.2/s elapsed=4.4s
2026-04-15 01:38:33,285 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=66953.0/s elapsed=4.8s
2026-04-15 01:38:33,763 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=67027.0/s elapsed=5.3s
2026-04-15 01:38:34,245 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=67042.8/s elapsed=5.8s
2026-04-15 01:38:34,748 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=66843.2/s elapsed=6.3s
2026-04-15 01:38:35,254 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=66627.2/s elapsed=6.8s
2026-04-15 01:38:35,757 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=66482.0/s elapsed=7.3s
2026-04-15 01:38:36,279 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=66191.3/s elapsed=7.8s
2026-04-15 01:38:36,846 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=65577.4/s elapsed=8.4s
2026-04-15 01:38:37,360 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=65432.8/s elapsed=8.9s
2026-04-15 01:38:37,878 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=65280.0/s elapsed=9.4s
2026-04-15 01:38:38,435 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=64880.6/s elapsed=10.0s
2026-04-15 01:38:38,435 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=10.0s
2026-04-15 01:38:38,435 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 01:38:38,450 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=68662.0/s elapsed=0.0s
2026-04-15 01:38:38,465 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=68446.5/s elapsed=0.0s
2026-04-15 01:38:38,479 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=68310.8/s elapsed=0.0s
2026-04-15 01:38:38,483 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=68165.9/s elapsed=0.0s
2026-04-15 01:38:38,483 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.0s
2026-04-15 01:38:41,800 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=14.6s
2026-04-15 01:38:41,931 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 01:39:04,707 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=22.8s
2026-04-15 01:39:04,708 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 01:39:04,737 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 01:39:08,008 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 num_workers=16 dataloader_worker_threads=1 metadata_workers=16 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 01:39:08,008 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=16 worker_threads=1
2026-04-15 01:39:08,008 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 01:39:08,390 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=84794.6/s elapsed=0.4s
2026-04-15 01:39:08,773 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=84772.6/s elapsed=0.8s
2026-04-15 01:39:09,151 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=85114.7/s elapsed=1.1s
2026-04-15 01:39:09,528 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=85281.7/s elapsed=1.5s
2026-04-15 01:39:09,906 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=85405.7/s elapsed=1.9s
2026-04-15 01:39:10,287 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=85343.5/s elapsed=2.3s
2026-04-15 01:39:10,670 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=85241.2/s elapsed=2.7s
2026-04-15 01:39:11,052 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=85201.6/s elapsed=3.0s
2026-04-15 01:39:11,436 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=85097.6/s elapsed=3.4s
2026-04-15 01:39:11,819 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=85062.5/s elapsed=3.8s
2026-04-15 01:39:12,206 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=84936.0/s elapsed=4.2s
2026-04-15 01:39:12,589 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=84916.9/s elapsed=4.6s
2026-04-15 01:39:12,973 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=84862.2/s elapsed=5.0s
2026-04-15 01:39:13,355 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=84875.9/s elapsed=5.3s
2026-04-15 01:39:13,734 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=84914.0/s elapsed=5.7s
2026-04-15 01:39:14,115 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=84927.2/s elapsed=6.1s
2026-04-15 01:39:14,492 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=84987.1/s elapsed=6.5s
2026-04-15 01:39:14,870 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=85025.7/s elapsed=6.9s
2026-04-15 01:39:15,248 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=85062.7/s elapsed=7.2s
2026-04-15 01:39:15,625 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=85111.1/s elapsed=7.6s
2026-04-15 01:39:22,017 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=14.0s
2026-04-15 01:39:22,017 | INFO | rank=0 | train dataloader metadata ready elapsed=14.0s
2026-04-15 01:39:22,018 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 01:39:22,440 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=76829.4/s elapsed=0.4s
2026-04-15 01:39:22,879 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=75300.0/s elapsed=0.9s
2026-04-15 01:39:23,308 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=75341.3/s elapsed=1.3s
2026-04-15 01:39:23,729 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=75772.4/s elapsed=1.7s
2026-04-15 01:39:24,151 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=75963.8/s elapsed=2.1s
2026-04-15 01:39:24,571 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=76170.7/s elapsed=2.6s
2026-04-15 01:39:24,992 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=76280.7/s elapsed=3.0s
2026-04-15 01:39:25,412 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=76398.1/s elapsed=3.4s
2026-04-15 01:39:25,837 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=76390.9/s elapsed=3.8s
2026-04-15 01:39:26,257 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=76464.3/s elapsed=4.2s
2026-04-15 01:39:26,679 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=76497.1/s elapsed=4.7s
2026-04-15 01:39:27,100 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=76526.3/s elapsed=5.1s
2026-04-15 01:39:27,521 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=76564.1/s elapsed=5.5s
2026-04-15 01:39:27,943 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=76586.2/s elapsed=5.9s
2026-04-15 01:39:28,385 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=76363.0/s elapsed=6.4s
2026-04-15 01:39:28,814 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=76302.6/s elapsed=6.8s
2026-04-15 01:39:29,247 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=76222.3/s elapsed=7.2s
2026-04-15 01:39:29,676 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=76182.7/s elapsed=7.7s
2026-04-15 01:39:30,104 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=76155.0/s elapsed=8.1s
2026-04-15 01:39:30,533 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=76129.9/s elapsed=8.5s
2026-04-15 01:39:30,533 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 01:39:32,298 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=8.5s sort_elapsed=1.8s
2026-04-15 01:39:32,970 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 01:39:35,995 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=85740.1/s elapsed=3.0s
2026-04-15 01:39:36,249 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=88979.8/s elapsed=3.3s
2026-04-15 01:39:36,484 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=92249.1/s elapsed=3.5s
2026-04-15 01:39:38,489 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=64607.9/s elapsed=5.5s
2026-04-15 01:39:38,699 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=67897.1/s elapsed=5.7s
2026-04-15 01:39:38,905 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=71003.9/s elapsed=5.9s
2026-04-15 01:39:39,111 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=73892.7/s elapsed=6.1s
2026-04-15 01:39:39,317 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=76600.0/s elapsed=6.3s
2026-04-15 01:39:41,238 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=62723.0/s elapsed=8.3s
2026-04-15 01:39:41,451 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=64970.9/s elapsed=8.5s
2026-04-15 01:39:41,678 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=66998.1/s elapsed=8.7s
2026-04-15 01:39:41,907 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=68910.3/s elapsed=8.9s
2026-04-15 01:39:42,140 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=70692.1/s elapsed=9.2s
2026-04-15 01:39:49,556 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=16 worker_threads=1
2026-04-15 01:39:49,557 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 01:39:49,570 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=77527.9/s elapsed=0.0s
2026-04-15 01:39:49,582 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=80086.7/s elapsed=0.0s
2026-04-15 01:39:49,594 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=81322.0/s elapsed=0.0s
2026-04-15 01:39:49,596 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=81010.9/s elapsed=0.0s
2026-04-15 01:39:49,627 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 01:39:49,627 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 01:39:49,627 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 01:39:49,640 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=79029.4/s elapsed=0.0s
2026-04-15 01:39:49,652 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=78976.9/s elapsed=0.0s
2026-04-15 01:39:49,665 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=78799.9/s elapsed=0.0s
2026-04-15 01:39:49,668 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=78620.0/s elapsed=0.0s
2026-04-15 01:39:49,668 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 01:39:49,670 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.0s sort_elapsed=0.0s
2026-04-15 01:39:49,670 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 01:39:49,675 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=189625.3/s elapsed=0.0s
2026-04-15 01:39:49,681 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=180782.8/s elapsed=0.0s
2026-04-15 01:39:49,687 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=176445.2/s elapsed=0.0s
2026-04-15 01:39:49,688 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=175295.3/s elapsed=0.0s
2026-04-15 01:39:49,689 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=45.0s
2026-04-15 01:39:49,689 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 01:41:03,979 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 01:41:04,242 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=1m 14.6s
2026-04-15 01:41:53,004 | INFO | rank=0 | resumed from artifacts/w2v-bert-test2-full/checkpoint_step_last.pt starting_epoch=2 global_step=12000 best_val_wer=0.4044
2026-04-15 01:41:53,398 | INFO | rank=0 | trackio initialized project=w2v-bert-test2-full run_name=w2v-bert-test2-full_20260414_223809_resume-e0002-s00012000 trackio_dir=artifacts/w2v-bert-test2-full/trackio elapsed_since_start=3m 44.2s
2026-04-15 01:41:53,400 | INFO | rank=0 | epoch 2/100 started train_batches=249868 val_batches=1355 grad_accumulation=4