2026-04-15 01:59:14,664 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://494cae8c31ad73fee6.gradio.live full_url=https://494cae8c31ad73fee6.gradio.live?project=w2v-bert-test2-full&write_token=y-NhTbJQrRuJYhju4XLZMyoYap4G7cPnkI0JkdrheG4
2026-04-15 01:59:14,867 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 02:00:08,449 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 02:00:12,856 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 02:00:12,858 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 02:00:12,860 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 02:00:12,860 | INFO | rank=0 | resolving dataset sources
2026-04-15 02:00:12,862 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 02:00:12,862 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 02:00:16,540 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=3.7s
2026-04-15 02:00:16,542 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 02:00:18,279 | INFO | rank=0 | dataset audio decode support ready elapsed=5.4s
2026-04-15 02:00:18,279 | INFO | rank=0 | building dataset split audit
2026-04-15 02:00:18,280 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 02:00:18,913 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=51196.0/s elapsed=0.6s
2026-04-15 02:00:19,547 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=51133.2/s elapsed=1.3s
2026-04-15 02:00:20,202 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=50590.7/s elapsed=1.9s
2026-04-15 02:00:20,857 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=50308.6/s elapsed=2.6s
2026-04-15 02:00:21,504 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=50257.0/s elapsed=3.2s
2026-04-15 02:00:22,143 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=50334.7/s elapsed=3.9s
2026-04-15 02:00:22,783 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=50385.5/s elapsed=4.5s
2026-04-15 02:00:23,421 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=50429.7/s elapsed=5.1s
2026-04-15 02:00:24,063 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=50437.3/s elapsed=5.8s
2026-04-15 02:00:24,701 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=50474.4/s elapsed=6.4s
2026-04-15 02:00:25,336 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=50523.7/s elapsed=7.1s
2026-04-15 02:00:25,978 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=50526.2/s elapsed=7.7s
2026-04-15 02:00:26,639 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=50405.4/s elapsed=8.4s
2026-04-15 02:00:27,317 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=50209.6/s elapsed=9.0s
2026-04-15 02:00:27,996 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=50036.3/s elapsed=9.7s
2026-04-15 02:00:28,691 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=49808.3/s elapsed=10.4s
2026-04-15 02:00:29,447 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=49338.7/s elapsed=11.2s
2026-04-15 02:00:30,115 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=49293.2/s elapsed=11.8s
2026-04-15 02:00:30,787 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=49237.0/s elapsed=12.5s
2026-04-15 02:00:31,566 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=48790.5/s elapsed=13.3s
2026-04-15 02:00:31,567 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=13.3s
2026-04-15 02:00:31,568 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 02:00:31,589 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=46632.3/s elapsed=0.0s
2026-04-15 02:00:31,610 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=47893.7/s elapsed=0.0s
2026-04-15 02:00:31,630 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=48353.2/s elapsed=0.1s
2026-04-15 02:00:31,634 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=48157.9/s elapsed=0.1s
2026-04-15 02:00:31,635 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.1s
2026-04-15 02:00:35,754 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=22.9s
2026-04-15 02:00:35,952 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 02:01:27,563 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=51.6s
2026-04-15 02:01:27,565 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 02:01:30,838 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=3.3s
2026-04-15 02:01:34,863 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 num_workers=4 dataloader_worker_threads=1 metadata_workers=4 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 02:01:34,864 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=4 worker_threads=1
2026-04-15 02:01:34,864 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 02:01:35,347 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=67031.0/s elapsed=0.5s
2026-04-15 02:01:35,830 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=67117.0/s elapsed=1.0s
2026-04-15 02:01:36,312 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=67133.2/s elapsed=1.4s
2026-04-15 02:01:36,797 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=67078.9/s elapsed=1.9s
2026-04-15 02:01:37,281 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=67040.3/s elapsed=2.4s
2026-04-15 02:01:37,765 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=67028.7/s elapsed=2.9s
2026-04-15 02:01:38,250 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=66998.8/s elapsed=3.4s
2026-04-15 02:01:38,731 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=67045.2/s elapsed=3.9s
2026-04-15 02:01:39,212 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=67082.8/s elapsed=4.3s
2026-04-15 02:01:39,690 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=67161.1/s elapsed=4.8s
2026-04-15 02:01:40,167 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=67232.6/s elapsed=5.3s
2026-04-15 02:01:40,646 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=67265.2/s elapsed=5.8s
2026-04-15 02:01:41,130 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=67239.8/s elapsed=6.3s
2026-04-15 02:01:41,613 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=67233.7/s elapsed=6.7s
2026-04-15 02:01:42,103 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=67164.2/s elapsed=7.2s
2026-04-15 02:01:42,585 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=67166.6/s elapsed=7.7s
2026-04-15 02:01:43,066 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=67178.9/s elapsed=8.2s
2026-04-15 02:01:43,545 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=67203.0/s elapsed=8.7s
2026-04-15 02:01:44,031 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=67180.6/s elapsed=9.2s
2026-04-15 02:01:44,520 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=67132.7/s elapsed=9.7s
2026-04-15 02:01:52,473 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=17.6s
2026-04-15 02:01:52,473 | INFO | rank=0 | train dataloader metadata ready elapsed=17.6s
2026-04-15 02:01:52,474 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 02:01:53,001 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=61476.9/s elapsed=0.5s
2026-04-15 02:01:53,521 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=61895.5/s elapsed=1.0s
2026-04-15 02:01:54,048 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=61789.5/s elapsed=1.6s
2026-04-15 02:01:54,573 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=61756.5/s elapsed=2.1s
2026-04-15 02:01:55,096 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=61810.0/s elapsed=2.6s
2026-04-15 02:01:55,625 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=61710.1/s elapsed=3.2s
2026-04-15 02:01:56,154 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=61648.2/s elapsed=3.7s
2026-04-15 02:01:56,679 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=61660.8/s elapsed=4.2s
2026-04-15 02:01:57,209 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=61610.4/s elapsed=4.7s
2026-04-15 02:01:57,727 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=61707.1/s elapsed=5.3s
2026-04-15 02:01:58,248 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=61745.2/s elapsed=5.8s
2026-04-15 02:01:58,788 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=61603.9/s elapsed=6.3s
2026-04-15 02:01:59,316 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=61580.1/s elapsed=6.8s
2026-04-15 02:01:59,834 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=61656.0/s elapsed=7.4s
2026-04-15 02:02:00,357 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=61675.4/s elapsed=7.9s
2026-04-15 02:02:00,895 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=61584.4/s elapsed=8.4s
2026-04-15 02:02:01,428 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=61534.4/s elapsed=9.0s
2026-04-15 02:02:01,951 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=61558.1/s elapsed=9.5s
2026-04-15 02:02:02,479 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=61549.1/s elapsed=10.0s
2026-04-15 02:02:03,014 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=61500.6/s elapsed=10.5s
2026-04-15 02:02:03,015 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 02:02:04,818 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=10.5s sort_elapsed=1.8s
2026-04-15 02:02:05,470 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 02:02:08,546 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=84304.9/s elapsed=3.1s
2026-04-15 02:02:08,794 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=87745.2/s elapsed=3.3s
2026-04-15 02:02:09,045 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=90649.7/s elapsed=3.6s
2026-04-15 02:02:10,658 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=68725.1/s elapsed=5.2s
2026-04-15 02:02:10,955 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=70906.7/s elapsed=5.5s
2026-04-15 02:02:11,223 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=73238.3/s elapsed=5.8s
2026-04-15 02:02:11,484 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=75455.5/s elapsed=6.0s
2026-04-15 02:02:11,742 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=77517.3/s elapsed=6.3s
2026-04-15 02:02:14,818 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=55477.6/s elapsed=9.3s
2026-04-15 02:02:15,123 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=57083.4/s elapsed=9.7s
2026-04-15 02:02:15,406 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=58715.2/s elapsed=9.9s
2026-04-15 02:02:15,682 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=60303.6/s elapsed=10.2s
2026-04-15 02:02:15,952 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=61845.8/s elapsed=10.5s
2026-04-15 02:02:26,035 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=4 worker_threads=1
2026-04-15 02:02:26,036 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 02:02:26,052 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=61924.5/s elapsed=0.0s
2026-04-15 02:02:26,068 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=63553.7/s elapsed=0.0s
2026-04-15 02:02:26,083 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=64177.6/s elapsed=0.0s
2026-04-15 02:02:26,086 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=64152.1/s elapsed=0.0s
2026-04-15 02:02:26,125 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 02:02:26,125 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 02:02:26,125 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 02:02:26,141 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=62853.0/s elapsed=0.0s
2026-04-15 02:02:26,157 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=62799.4/s elapsed=0.0s
2026-04-15 02:02:26,174 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=62377.4/s elapsed=0.0s
2026-04-15 02:02:26,177 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=62139.8/s elapsed=0.1s
2026-04-15 02:02:26,177 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 02:02:26,179 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.1s sort_elapsed=0.0s
2026-04-15 02:02:26,180 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 02:02:26,186 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=151712.7/s elapsed=0.0s
2026-04-15 02:02:26,194 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=142913.1/s elapsed=0.0s
2026-04-15 02:02:26,201 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=140484.4/s elapsed=0.0s
2026-04-15 02:02:26,203 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=139129.2/s elapsed=0.0s
2026-04-15 02:02:26,203 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=55.4s
2026-04-15 02:02:26,204 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False