2026-04-15 01:47:00,897 | INFO | rank=0 | trackio ui launched trackio_dir=artifacts/w2v-bert-test2-full/trackio local_url=http://127.0.0.1:7860/ share_url=https://cd7561e1db912cbdca.gradio.live full_url=https://cd7561e1db912cbdca.gradio.live?project=w2v-bert-test2-full&write_token=IbDnOZvCBg5uxXh_bXbHrdhho2nkn2UzRHPRhSjEkuc
2026-04-15 01:47:01,092 | INFO | rank=0 | hugging face checkpoint repository initialized repo_id=speech-uk/w2v-bert-test2-full repo_type=model path_in_repo=. checkpoint_format=safetensors folder=artifacts/w2v-bert-test2-full
2026-04-15 01:47:55,858 | INFO | rank=0 | auto-resume validated checkpoint path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt epoch=1 global_step=12000
2026-04-15 01:47:57,419 | INFO | rank=0 | starting training variant=sm zipformer_requested=False w2v_bert_requested=True device=cuda distributed=True world_size=2 output_dir=artifacts/w2v-bert-test2-full
2026-04-15 01:47:57,421 | INFO | rank=0 | loading LM scorer source=disabled
2026-04-15 01:47:57,421 | INFO | rank=0 | LM scorer ready source=disabled elapsed=0.0s
2026-04-15 01:47:57,421 | INFO | rank=0 | resolving dataset sources
2026-04-15 01:47:57,422 | INFO | rank=0 | dataset sources resolved train_count=4 validation_count=1 elapsed=0.0s
2026-04-15 01:47:57,422 | INFO | rank=0 | loading dataset records train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] validation_mode=external record_cache=True prevalidate_audio=False
2026-04-15 01:47:58,940 | INFO | rank=0 | dataset records available train_samples=648240 val_samples=3204 elapsed=1.5s
2026-04-15 01:47:58,942 | INFO | rank=0 | checking dataset audio decode support
2026-04-15 01:48:00,482 | INFO | rank=0 | dataset audio decode support ready elapsed=3.1s
2026-04-15 01:48:00,483 | INFO | rank=0 | building dataset split audit
2026-04-15 01:48:00,483 | INFO | rank=0 | split audit started split=train records=648240
2026-04-15 01:48:00,988 | INFO | rank=0 | split audit scanned train records progress=32412/648240 5.0% rate=64132.6/s elapsed=0.5s
2026-04-15 01:48:01,491 | INFO | rank=0 | split audit scanned train records progress=64824/648240 10.0% rate=64301.6/s elapsed=1.0s
2026-04-15 01:48:02,011 | INFO | rank=0 | split audit scanned train records progress=97236/648240 15.0% rate=63638.0/s elapsed=1.5s
2026-04-15 01:48:02,530 | INFO | rank=0 | split audit scanned train records progress=129648/648240 20.0% rate=63326.8/s elapsed=2.0s
2026-04-15 01:48:03,047 | INFO | rank=0 | split audit scanned train records progress=162060/648240 25.0% rate=63199.7/s elapsed=2.6s
2026-04-15 01:48:03,557 | INFO | rank=0 | split audit scanned train records progress=194472/648240 30.0% rate=63262.8/s elapsed=3.1s
2026-04-15 01:48:04,068 | INFO | rank=0 | split audit scanned train records progress=226884/648240 35.0% rate=63290.1/s elapsed=3.6s
2026-04-15 01:48:04,577 | INFO | rank=0 | split audit scanned train records progress=259296/648240 40.0% rate=63338.5/s elapsed=4.1s
2026-04-15 01:48:05,086 | INFO | rank=0 | split audit scanned train records progress=291708/648240 45.0% rate=63376.2/s elapsed=4.6s
2026-04-15 01:48:05,594 | INFO | rank=0 | split audit scanned train records progress=324120/648240 50.0% rate=63414.2/s elapsed=5.1s
2026-04-15 01:48:06,104 | INFO | rank=0 | split audit scanned train records progress=356532/648240 55.0% rate=63428.7/s elapsed=5.6s
2026-04-15 01:48:06,615 | INFO | rank=0 | split audit scanned train records progress=388944/648240 60.0% rate=63434.3/s elapsed=6.1s
2026-04-15 01:48:07,145 | INFO | rank=0 | split audit scanned train records progress=421356/648240 65.0% rate=63251.7/s elapsed=6.7s
2026-04-15 01:48:07,691 | INFO | rank=0 | split audit scanned train records progress=453768/648240 70.0% rate=62955.3/s elapsed=7.2s
2026-04-15 01:48:08,231 | INFO | rank=0 | split audit scanned train records progress=486180/648240 75.0% rate=62748.0/s elapsed=7.7s
2026-04-15 01:48:08,795 | INFO | rank=0 | split audit scanned train records progress=518592/648240 80.0% rate=62391.7/s elapsed=8.3s
2026-04-15 01:48:09,406 | INFO | rank=0 | split audit scanned train records progress=551004/648240 85.0% rate=61749.1/s elapsed=8.9s
2026-04-15 01:48:09,945 | INFO | rank=0 | split audit scanned train records progress=583416/648240 90.0% rate=61660.6/s elapsed=9.5s
2026-04-15 01:48:10,487 | INFO | rank=0 | split audit scanned train records progress=615828/648240 95.0% rate=61560.7/s elapsed=10.0s
2026-04-15 01:48:11,087 | INFO | rank=0 | split audit scanned train records progress=648240/648240 100.0% rate=61132.8/s elapsed=10.6s
2026-04-15 01:48:11,088 | INFO | rank=0 | split audit completed split=train records=648240 speakers=0 records_with_speaker_id=0 elapsed=10.6s
2026-04-15 01:48:11,088 | INFO | rank=0 | split audit started split=validation records=3204
2026-04-15 01:48:11,105 | INFO | rank=0 | split audit scanned validation records progress=1000/3204 31.2% rate=57841.5/s elapsed=0.0s
2026-04-15 01:48:11,121 | INFO | rank=0 | split audit scanned validation records progress=2000/3204 62.4% rate=59916.6/s elapsed=0.0s
2026-04-15 01:48:11,137 | INFO | rank=0 | split audit scanned validation records progress=3000/3204 93.6% rate=61118.7/s elapsed=0.0s
2026-04-15 01:48:11,140 | INFO | rank=0 | split audit scanned validation records progress=3204/3204 100.0% rate=61065.9/s elapsed=0.1s
2026-04-15 01:48:11,140 | INFO | rank=0 | split audit completed split=validation records=3204 speakers=0 records_with_speaker_id=0 elapsed=0.1s
2026-04-15 01:48:14,443 | INFO | rank=0 | loaded datasets train_sources=['/home/yehor/squeezeformer/data/cv22-opus-speech-filtered', '/home/yehor/squeezeformer/data/broadcast-opus-speech-filtered', '/home/yehor/squeezeformer/data/voa-opus-speech-filtered', '/home/yehor/squeezeformer/data/yodas2-opus-speech-filtered'] validation_sources=['/home/yehor/squeezeformer/data/cv10-uk-testset-clean-punctuated'] train_samples=648240 val_samples=3204 speaker_balance_ratio=1.000 elapsed=17.0s
2026-04-15 01:48:14,621 | INFO | rank=0 | saved audio preview samples requested=10 saved=10 dir=artifacts/w2v-bert-test2-full/audio_previews manifest=artifacts/w2v-bert-test2-full/audio_previews/manifest.jsonl
2026-04-15 01:49:05,254 | INFO | rank=0 | resume checkpoint loaded path=artifacts/w2v-bert-test2-full/checkpoint_step_last.pt elapsed=50.6s
2026-04-15 01:49:05,255 | INFO | rank=0 | preparing tokenizer mode=sentencepiece resume=True tokenizer_path=/home/yehor/squeezeformer/code/artifacts/w2v-bert-test-full/tokenizer.model
2026-04-15 01:49:05,285 | INFO | rank=0 | tokenizer ready vocab_size=512 elapsed=0.0s
2026-04-15 01:49:08,505 | INFO | rank=0 | building dataloaders train_samples=648240 val_samples=3204 distributed=True world_size=2 train_hours=1264.35 val_hours=4.63 num_workers=16 dataloader_worker_threads=8 metadata_workers=16 force_audio_metadata_probe=False persistent_workers=True prefetch_factor=2 train_in_order=False feature_cache_format=parquet
2026-04-15 01:49:08,505 | INFO | rank=0 | train dataloader create_dataloader started samples=648240 batch_size=8 shuffle=True num_workers=16 worker_threads=8
2026-04-15 01:49:08,505 | INFO | rank=0 | train dataloader estimating frames from cached audio metadata records=648240
2026-04-15 01:49:08,882 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=32412/648240 5.0% rate=85986.4/s elapsed=0.4s
2026-04-15 01:49:09,260 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=64824/648240 10.0% rate=85936.5/s elapsed=0.8s
2026-04-15 01:49:09,636 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=97236/648240 15.0% rate=86023.7/s elapsed=1.1s
2026-04-15 01:49:10,012 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=129648/648240 20.0% rate=86057.6/s elapsed=1.5s
2026-04-15 01:49:10,388 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=162060/648240 25.0% rate=86067.0/s elapsed=1.9s
2026-04-15 01:49:10,780 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=194472/648240 30.0% rate=85486.3/s elapsed=2.3s
2026-04-15 01:49:11,156 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=226884/648240 35.0% rate=85587.0/s elapsed=2.7s
2026-04-15 01:49:11,532 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=259296/648240 40.0% rate=85681.1/s elapsed=3.0s
2026-04-15 01:49:11,905 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=291708/648240 45.0% rate=85796.6/s elapsed=3.4s
2026-04-15 01:49:12,285 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=324120/648240 50.0% rate=85753.9/s elapsed=3.8s
2026-04-15 01:49:12,662 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=356532/648240 55.0% rate=85768.6/s elapsed=4.2s
2026-04-15 01:49:13,039 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=388944/648240 60.0% rate=85783.9/s elapsed=4.5s
2026-04-15 01:49:13,417 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=421356/648240 65.0% rate=85790.6/s elapsed=4.9s
2026-04-15 01:49:13,792 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=453768/648240 70.0% rate=85825.6/s elapsed=5.3s
2026-04-15 01:49:14,170 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=486180/648240 75.0% rate=85824.2/s elapsed=5.7s
2026-04-15 01:49:14,547 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=518592/648240 80.0% rate=85840.3/s elapsed=6.0s
2026-04-15 01:49:14,924 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=551004/648240 85.0% rate=85842.5/s elapsed=6.4s
2026-04-15 01:49:15,302 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=583416/648240 90.0% rate=85836.0/s elapsed=6.8s
2026-04-15 01:49:15,679 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=615828/648240 95.0% rate=85842.7/s elapsed=7.2s
2026-04-15 01:49:16,055 | INFO | rank=0 | train dataloader estimated frames from cached metadata progress=648240/648240 100.0% rate=85856.9/s elapsed=7.6s
2026-04-15 01:49:22,373 | INFO | rank=0 | train dataloader audio metadata complete records=648240 probe_needed=0 elapsed=13.9s
2026-04-15 01:49:22,373 | INFO | rank=0 | train dataloader metadata ready elapsed=13.9s
2026-04-15 01:49:22,373 | INFO | rank=0 | train dataloader adaptive sampler computing sort keys records=648240
2026-04-15 01:49:22,799 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=32412/648240 5.0% rate=76126.0/s elapsed=0.4s
2026-04-15 01:49:23,241 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=64824/648240 10.0% rate=74706.6/s elapsed=0.9s
2026-04-15 01:49:23,672 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=97236/648240 15.0% rate=74892.6/s elapsed=1.3s
2026-04-15 01:49:24,090 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=129648/648240 20.0% rate=75496.5/s elapsed=1.7s
2026-04-15 01:49:24,514 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=162060/648240 25.0% rate=75700.3/s elapsed=2.1s
2026-04-15 01:49:24,939 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=194472/648240 30.0% rate=75787.9/s elapsed=2.6s
2026-04-15 01:49:25,365 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=226884/648240 35.0% rate=75846.9/s elapsed=3.0s
2026-04-15 01:49:25,790 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=259296/648240 40.0% rate=75882.5/s elapsed=3.4s
2026-04-15 01:49:26,213 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=291708/648240 45.0% rate=75972.1/s elapsed=3.8s
2026-04-15 01:49:26,639 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=324120/648240 50.0% rate=75985.9/s elapsed=4.3s
2026-04-15 01:49:27,064 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=356532/648240 55.0% rate=76005.6/s elapsed=4.7s
2026-04-15 01:49:27,490 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=388944/648240 60.0% rate=76010.6/s elapsed=5.1s
2026-04-15 01:49:27,913 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=421356/648240 65.0% rate=76063.1/s elapsed=5.5s
2026-04-15 01:49:28,336 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=453768/648240 70.0% rate=76101.5/s elapsed=6.0s
2026-04-15 01:49:28,778 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=486180/648240 75.0% rate=75905.5/s elapsed=6.4s
2026-04-15 01:49:29,211 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=518592/648240 80.0% rate=75838.1/s elapsed=6.8s
2026-04-15 01:49:29,649 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=551004/648240 85.0% rate=75731.5/s elapsed=7.3s
2026-04-15 01:49:30,084 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=583416/648240 90.0% rate=75665.3/s elapsed=7.7s
2026-04-15 01:49:30,515 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=615828/648240 95.0% rate=75641.6/s elapsed=8.1s
2026-04-15 01:49:30,943 | INFO | rank=0 | train dataloader adaptive sampler computed sort keys progress=648240/648240 100.0% rate=75645.3/s elapsed=8.6s
2026-04-15 01:49:30,943 | INFO | rank=0 | train dataloader adaptive sampler sorting keyed records total=648240
2026-04-15 01:49:32,624 | INFO | rank=0 | train dataloader adaptive sampler sorted records total=648240 key_elapsed=8.6s sort_elapsed=1.7s
2026-04-15 01:49:33,210 | INFO | rank=0 | train dataloader adaptive sampler building batches records=648240 unit=frames budget=400
2026-04-15 01:49:36,209 | INFO | rank=0 | train dataloader adaptive sampler built batches=110791 progress=259296/648240 40.0% rate=86471.4/s elapsed=3.0s
2026-04-15 01:49:36,434 | INFO | rank=0 | train dataloader adaptive sampler built batches=143203 progress=291708/648240 45.0% rate=90465.3/s elapsed=3.2s
2026-04-15 01:49:36,666 | INFO | rank=0 | train dataloader adaptive sampler built batches=175615 progress=324120/648240 50.0% rate=93783.2/s elapsed=3.5s
2026-04-15 01:49:38,463 | INFO | rank=0 | train dataloader adaptive sampler built batches=208027 progress=356532/648240 55.0% rate=67873.6/s elapsed=5.3s
2026-04-15 01:49:38,670 | INFO | rank=0 | train dataloader adaptive sampler built batches=240439 progress=388944/648240 60.0% rate=71230.1/s elapsed=5.5s
2026-04-15 01:49:38,873 | INFO | rank=0 | train dataloader adaptive sampler built batches=272851 progress=421356/648240 65.0% rate=74393.4/s elapsed=5.7s
2026-04-15 01:49:39,076 | INFO | rank=0 | train dataloader adaptive sampler built batches=305263 progress=453768/648240 70.0% rate=77345.6/s elapsed=5.9s
2026-04-15 01:49:39,281 | INFO | rank=0 | train dataloader adaptive sampler built batches=337675 progress=486180/648240 75.0% rate=80075.3/s elapsed=6.1s
2026-04-15 01:49:41,078 | INFO | rank=0 | train dataloader adaptive sampler built batches=370087 progress=518592/648240 80.0% rate=65909.2/s elapsed=7.9s
2026-04-15 01:49:41,289 | INFO | rank=0 | train dataloader adaptive sampler built batches=402499 progress=551004/648240 85.0% rate=68197.9/s elapsed=8.1s
2026-04-15 01:49:41,492 | INFO | rank=0 | train dataloader adaptive sampler built batches=434911 progress=583416/648240 90.0% rate=70440.2/s elapsed=8.3s
2026-04-15 01:49:41,697 | INFO | rank=0 | train dataloader adaptive sampler built batches=467323 progress=615828/648240 95.0% rate=72555.9/s elapsed=8.5s
2026-04-15 01:49:41,896 | INFO | rank=0 | train dataloader adaptive sampler built batches=499735 progress=648240/648240 100.0% rate=74626.6/s elapsed=8.7s
2026-04-15 01:49:49,310 | INFO | rank=0 | validation dataloader create_dataloader started samples=3204 batch_size=8 shuffle=False num_workers=16 worker_threads=8
2026-04-15 01:49:49,311 | INFO | rank=0 | validation dataloader estimating frames from cached audio metadata records=3204
2026-04-15 01:49:49,324 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=1000/3204 31.2% rate=76917.2/s elapsed=0.0s
2026-04-15 01:49:49,336 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=2000/3204 62.4% rate=79822.9/s elapsed=0.0s
2026-04-15 01:49:49,348 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3000/3204 93.6% rate=81078.6/s elapsed=0.0s
2026-04-15 01:49:49,351 | INFO | rank=0 | validation dataloader estimated frames from cached metadata progress=3204/3204 100.0% rate=80949.8/s elapsed=0.0s
2026-04-15 01:49:49,382 | INFO | rank=0 | validation dataloader audio metadata complete records=3204 probe_needed=0 elapsed=0.1s
2026-04-15 01:49:49,382 | INFO | rank=0 | validation dataloader metadata ready elapsed=0.1s
2026-04-15 01:49:49,382 | INFO | rank=0 | validation dataloader adaptive sampler computing sort keys records=3204
2026-04-15 01:49:49,396 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=1000/3204 31.2% rate=73838.5/s elapsed=0.0s
2026-04-15 01:49:49,408 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=2000/3204 62.4% rate=76074.4/s elapsed=0.0s
2026-04-15 01:49:49,421 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3000/3204 93.6% rate=76728.1/s elapsed=0.0s
2026-04-15 01:49:49,424 | INFO | rank=0 | validation dataloader adaptive sampler computed sort keys progress=3204/3204 100.0% rate=76552.1/s elapsed=0.0s
2026-04-15 01:49:49,424 | INFO | rank=0 | validation dataloader adaptive sampler sorting keyed records total=3204
2026-04-15 01:49:49,426 | INFO | rank=0 | validation dataloader adaptive sampler sorted records total=3204 key_elapsed=0.0s sort_elapsed=0.0s
2026-04-15 01:49:49,426 | INFO | rank=0 | validation dataloader adaptive sampler building batches records=3204 unit=frames budget=400
2026-04-15 01:49:49,431 | INFO | rank=0 | validation dataloader adaptive sampler built batches=505 progress=1000/3204 31.2% rate=192692.6/s elapsed=0.0s
2026-04-15 01:49:49,437 | INFO | rank=0 | validation dataloader adaptive sampler built batches=1505 progress=2000/3204 62.4% rate=182268.2/s elapsed=0.0s
2026-04-15 01:49:49,443 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2505 progress=3000/3204 93.6% rate=179076.2/s elapsed=0.0s
2026-04-15 01:49:49,444 | INFO | rank=0 | validation dataloader adaptive sampler built batches=2709 progress=3204/3204 100.0% rate=177678.6/s elapsed=0.0s
2026-04-15 01:49:49,444 | INFO | rank=0 | dataloaders ready train_batches=249868 val_batches=1355 elapsed=44.2s
2026-04-15 01:49:49,445 | INFO | rank=0 | building model architecture=w2v-bert variant=sm dtype=bfloat16 compile=False aed=False liberta=False audio_teacher=False
2026-04-15 01:51:08,186 | INFO | rank=0 | wrapping model with DistributedDataParallel find_unused_parameters=True
2026-04-15 01:51:08,369 | INFO | rank=0 | model and auxiliaries ready params=606203200 liberta=False audio_teacher=False elapsed=1m 18.9s
2026-04-15 01:51:58,275 | INFO | rank=0 | resumed from artifacts/w2v-bert-test2-full/checkpoint_step_last.pt starting_epoch=2 global_step=12000 best_val_wer=0.4044
2026-04-15 01:51:58,618 | INFO | rank=0 | trackio initialized project=w2v-bert-test2-full run_name=w2v-bert-test2-full_20260414_224658_resume-e0002-s00012000 trackio_dir=artifacts/w2v-bert-test2-full/trackio elapsed_since_start=5m 0.3s
2026-04-15 01:51:58,621 | INFO | rank=0 | epoch 2/100 started train_batches=249868 val_batches=1355 grad_accumulation=4
2026-04-15 01:56:04,746 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=16 prefetch_factor=2 still waiting elapsed=10.0s
2026-04-15 01:56:34,770 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=16 prefetch_factor=2 still waiting elapsed=40.0s
2026-04-15 01:57:04,790 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=16 prefetch_factor=2 still waiting elapsed=1m 10.1s
2026-04-15 01:57:34,806 | INFO | rank=0 | epoch=2 waiting for training batch 1/249868 num_workers=16 prefetch_factor=2 still waiting elapsed=1m 40.1s