2026-04-15 03:39:45,236 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://45b271e28086c7072c.gradio.live full_url=https://45b271e28086c7072c.gradio.live?project=w2v-bert-test2-full&write_token=ceg_UC-uW5bltVjmF9NOSNWUi5NTMs1ABfUqO93lAyA
2026-04-15 03:39:45,416 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 03:39:53,170 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 03:39:53,796 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 03:39:53,797 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 03:39:53,797 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 03:39:53,797 | INFO | rank=0 | resolving dataset sources
2026-04-15 03:39:53,798 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 03:39:53,798 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 03:39:54,141 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=0.3s
2026-04-15 03:39:54,142 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 03:39:55,799 | INFO | rank=0 | dataset audio decode support ready elapsed=2.0s
2026-04-15 03:39:55,799 | INFO | rank=0 | building dataset split audit
2026-04-15 03:39:55,799 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 03:39:56,286 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=66586.6/s elapsed=0.5s
2026-04-15 03:39:56,773 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=66571.9/s elapsed=1.0s
2026-04-15 03:39:57,273 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=65980.2/s elapsed=1.5s
2026-04-15 03:39:57,770 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=65786.1/s elapsed=2.0s
2026-04-15 03:39:58,266 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=65701.2/s elapsed=2.5s
2026-04-15 03:39:58,760 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=65698.5/s elapsed=3.0s
2026-04-15 03:39:59,252 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=65718.4/s elapsed=3.5s
2026-04-15 03:39:59,743 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=65751.2/s elapsed=3.9s
2026-04-15 03:40:00,235 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=65769.5/s elapsed=4.4s
2026-04-15 03:40:00,727 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=65772.6/s elapsed=4.9s
2026-04-15 03:40:01,220 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=65774.2/s elapsed=5.4s
2026-04-15 03:40:01,713 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=65774.3/s elapsed=5.9s
2026-04-15 03:40:02,221 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=65614.8/s elapsed=6.4s
2026-04-15 03:40:02,741 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=65367.7/s elapsed=6.9s
2026-04-15 03:40:03,258 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=65185.3/s elapsed=7.5s
2026-04-15 03:40:03,788 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=64914.3/s elapsed=8.0s
2026-04-15 03:40:04,357 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=64390.6/s elapsed=8.6s
2026-04-15 03:40:04,872 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=64308.7/s elapsed=9.1s
2026-04-15 03:40:05,389 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=64219.5/s elapsed=9.6s
2026-04-15 03:40:05,948 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=63878.3/s elapsed=10.1s
2026-04-15 03:40:05,948 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=10.1s
2026-04-15 03:40:05,948 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 03:40:05,963 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=66703.7/s elapsed=0.0s
2026-04-15 03:40:05,978 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=66681.6/s elapsed=0.0s
2026-04-15 03:40:05,993 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=66770.8/s elapsed=0.0s
2026-04-15 03:40:05,996 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=66521.0/s elapsed=0.0s
2026-04-15 03:40:05,996 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.0s
2026-04-15 03:40:09,205 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=15.4s
2026-04-15 03:40:09,345 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 03:40:14,832 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=5.5s
2026-04-15 03:40:14,832 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 03:40:14,858 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 03:40:18,084 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 dataloader_backend=rust-parquet num_workers=16 dataloader_worker_threads=1 metadata_workers=8 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 03:40:18,084 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=16 worker_threads=1 backend=rust-parquet
2026-04-15 03:40:18,084 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 03:40:18,470 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=84054.6/s elapsed=0.4s
2026-04-15 03:40:18,857 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=83881.7/s elapsed=0.8s
2026-04-15 03:40:19,246 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=83691.8/s elapsed=1.2s
2026-04-15 03:40:19,636 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=83559.1/s elapsed=1.6s
2026-04-15 03:40:20,025 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=83504.8/s elapsed=1.9s
2026-04-15 03:40:20,412 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=83535.4/s elapsed=2.3s
2026-04-15 03:40:20,798 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=83593.3/s elapsed=2.7s
2026-04-15 03:40:21,185 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=83627.9/s elapsed=3.1s
2026-04-15 03:40:21,572 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=83641.9/s elapsed=3.5s
2026-04-15 03:40:21,958 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=83662.5/s elapsed=3.9s
2026-04-15 03:40:22,347 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=83643.7/s elapsed=4.3s
2026-04-15 03:40:22,735 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=83624.4/s elapsed=4.7s
2026-04-15 03:40:23,124 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=83601.0/s elapsed=5.0s
2026-04-15 03:40:23,513 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=83586.9/s elapsed=5.4s
2026-04-15 03:40:23,901 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=83576.3/s elapsed=5.8s
2026-04-15 03:40:24,291 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=83546.8/s elapsed=6.2s
2026-04-15 03:40:24,681 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=83528.1/s elapsed=6.6s
2026-04-15 03:40:25,072 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=83486.8/s elapsed=7.0s
2026-04-15 03:40:25,462 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=83463.3/s elapsed=7.4s
2026-04-15 03:40:25,851 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=83462.7/s elapsed=7.8s
2026-04-15 03:40:32,080 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=14.0s
2026-04-15 03:40:32,081 | INFO | rank=0 | train dataloader metadata ready elapsed=14.0s
2026-04-15 03:40:32,081 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 03:40:32,485 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=80162.0/s elapsed=0.4s
2026-04-15 03:40:32,889 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=80247.9/s elapsed=0.8s
2026-04-15 03:40:33,294 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=80144.1/s elapsed=1.2s
2026-04-15 03:40:33,701 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=80015.4/s elapsed=1.6s
2026-04-15 03:40:34,106 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=80016.8/s elapsed=2.0s
2026-04-15 03:40:34,511 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=80035.3/s elapsed=2.4s
2026-04-15 03:40:34,915 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=80053.8/s elapsed=2.8s
2026-04-15 03:40:35,320 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=80048.8/s elapsed=3.2s
2026-04-15 03:40:35,724 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=80071.5/s elapsed=3.6s
2026-04-15 03:40:36,128 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=80077.2/s elapsed=4.0s
2026-04-15 03:40:36,533 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=80085.8/s elapsed=4.5s
2026-04-15 03:40:36,938 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=80078.0/s elapsed=4.9s
2026-04-15 03:40:37,344 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=80063.7/s elapsed=5.3s
2026-04-15 03:40:37,750 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=80046.4/s elapsed=5.7s
2026-04-15 03:40:38,155 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=80040.7/s elapsed=6.1s
2026-04-15 03:40:38,560 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=80034.3/s elapsed=6.5s
2026-04-15 03:40:38,967 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=80014.7/s elapsed=6.9s
2026-04-15 03:40:39,373 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=80011.2/s elapsed=7.3s
2026-04-15 03:40:39,778 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=80008.9/s elapsed=7.7s
2026-04-15 03:40:40,185 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=79988.9/s elapsed=8.1s
2026-04-15 03:40:40,185 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 03:40:40,811 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=8.1s sort_elapsed=0.6s
2026-04-15 03:40:41,127 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 03:40:42,819 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=153253.9/s elapsed=1.7s
2026-04-15 03:40:43,018 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=154303.3/s elapsed=1.9s
2026-04-15 03:40:43,549 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=133838.3/s elapsed=2.4s
2026-04-15 03:40:43,751 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=135878.2/s elapsed=2.6s
2026-04-15 03:40:43,952 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=137677.3/s elapsed=2.8s
2026-04-15 03:40:44,154 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=139224.4/s elapsed=3.0s
2026-04-15 03:40:44,356 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=140532.5/s elapsed=3.2s
2026-04-15 03:40:44,558 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=141710.3/s elapsed=3.4s
2026-04-15 03:40:45,149 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=128934.5/s elapsed=4.0s
2026-04-15 03:40:45,351 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=130461.7/s elapsed=4.2s
2026-04-15 03:40:45,550 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=131908.3/s elapsed=4.4s
2026-04-15 03:40:45,748 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=133262.0/s elapsed=4.6s
2026-04-15 03:40:45,946 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=134514.7/s elapsed=4.8s
2026-04-15 03:40:45,948 | INFO | rank=0 | train dataloader rust-parquet feature cache index build started cache_dir=/home/yehor/squeezeformer/data/test-warm/feature-cache-w2v-bert/train batches=249868 num_workers=16 prefetch_batches=32
2026-04-15 03:40:46,582 | INFO | rank=0 | train dataloader rust-parquet feature cache index ready entries=650171 elapsed=0.6s
2026-04-15 03:40:46,582 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=16 worker_threads=1 backend=rust-parquet
2026-04-15 03:40:46,583 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 03:40:46,595 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=81723.3/s elapsed=0.0s
2026-04-15 03:40:46,607 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=82291.3/s elapsed=0.0s
2026-04-15 03:40:46,619 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=82766.6/s elapsed=0.0s
2026-04-15 03:40:46,621 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=82510.2/s elapsed=0.0s
2026-04-15 03:40:46,652 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 03:40:46,652 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 03:40:46,652 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 03:40:46,664 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=80905.5/s elapsed=0.0s
2026-04-15 03:40:46,677 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=80955.4/s elapsed=0.0s
2026-04-15 03:40:46,689 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=80872.0/s elapsed=0.0s
2026-04-15 03:40:46,692 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=80736.0/s elapsed=0.0s
2026-04-15 03:40:46,692 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 03:40:46,693 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.0s sort_elapsed=0.0s
2026-04-15 03:40:46,694 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 03:40:46,699 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=196295.2/s elapsed=0.0s
2026-04-15 03:40:46,705 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=178405.1/s elapsed=0.0s
2026-04-15 03:40:46,711 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=175536.4/s elapsed=0.0s
2026-04-15 03:40:46,712 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=174465.7/s elapsed=0.0s
2026-04-15 03:40:46,712 | INFO | rank=0 | validation dataloader rust-parquet feature cache index build started cache_dir=/home/yehor/squeezeformer/data/test-warm/feature-cache-w2v-bert/validation batches=1355 num_workers=16 prefetch_batches=32
2026-04-15 03:40:46,715 | INFO | rank=0 | validation dataloader rust-parquet feature cache index ready entries=3204 elapsed=0.0s
2026-04-15 03:40:46,715 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=31.9s
2026-04-15 03:40:46,715 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 03:41:04,448 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 03:41:04,519 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=17.8s
2026-04-15 03:41:06,927 | INFO | rank=0 | resumed from artifacts/w2v-bert-test2-full/checkpoint_step_last.pt starting_epoch=2 global_step=12000 best_val_wer=0.4044
2026-04-15 03:41:07,211 | INFO | rank=0 | trackio initialized project=w2v-bert-test2-full run_name=w2v-bert-test2-full_20260415_003942_resume-e0002-s00012000 trackio_dir=artifacts/w2v-bert-test2-full/trackio elapsed_since_start=1m 24.9s
2026-04-15 03:41:07,212 | INFO | rank=0 | epoch 2/100 started train_batches=249868 val_batches=1355 grad_accumulation=4
2026-04-15 03:41:17,226 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 backend=rust-parquet num_workers=16 prefetch_batches=32 still waiting elapsed=10.0s
2026-04-15 03:41:41,653 | INFO | rank=0 | train dataloader rust-parquet first loaded batch size=1 rust_hits=0 fallbacks=1 elapsed=32.3s
2026-04-15 03:41:47,226 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 backend=rust-parquet num_workers=16 prefetch_batches=32 still waiting elapsed=40.0s
2026-04-15 03:42:17,227 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 backend=rust-parquet num_workers=16 prefetch_batches=32 still waiting elapsed=1m 10.0s