File size: 3,907 Bytes
d23210e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | 2026-04-18 16:03:49.122 | INFO | robotmdar.dtype.logger:set:45 - Logging initialized for experiment: OfficialFullTrainabilityResume
2026-04-18 16:03:49.124 | INFO | robotmdar.dtype.logger:set:46 - Experiment directory: /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullTrainabilityResume/train-mvae-20260418_160348
2026-04-18 16:03:49.206 | INFO | robotmdar.dtype.seed:set:10 - Setting seed: 0
2026-04-18 16:03:49.272 | INFO | robotmdar.dataloader.data:_load_data:200 - Loading train data...
2026-04-18 16:03:49.632 | INFO | robotmdar.dataloader.data:_load_statistics:356 - statistics.yaml not found, inferred fps=50 from labels/npz dataset
2026-04-18 16:03:49.645 | INFO | robotmdar.dataloader.data:_load_raw_labels_npz_dataset:364 - Detected labels.json + npz dataset layout, using compatibility loader...
2026-04-18 16:03:49.960 | INFO | robotmdar.dataloader.data:_load_raw_labels_npz_dataset:376 - Loading 21837 samples from /data/haozhe/zzn/VAR_FM/ws/project/dataset/HumanML3D_babel_compat/train/labels.json...
2026-04-18 16:05:02.864 | INFO | robotmdar.dataloader.data:_load_data:228 - Found 21228 valid samples out of 21837
2026-04-18 16:05:02.864 | INFO | robotmdar.dataloader.data:_load_text_embeddings:415 - Loading cached text embeddings...
2026-04-18 16:05:05.611 | INFO | robotmdar.dataloader.data:_load_meanstd:457 - Loading cached mean/std from /data/haozhe/zzn/VAR_FM/ws/project/dataset/HumanML3D_babel_compat/meanstd.pkl...
2026-04-18 16:05:05.855 | INFO | robotmdar.dataloader.data:_load_data:200 - Loading val data...
2026-04-18 16:05:06.165 | INFO | robotmdar.dataloader.data:_load_statistics:356 - statistics.yaml not found, inferred fps=50 from labels/npz dataset
2026-04-18 16:05:06.179 | INFO | robotmdar.dataloader.data:_load_raw_labels_npz_dataset:364 - Detected labels.json + npz dataset layout, using compatibility loader...
2026-04-18 16:05:06.191 | INFO | robotmdar.dataloader.data:_load_raw_labels_npz_dataset:376 - Loading 2427 samples from /data/haozhe/zzn/VAR_FM/ws/project/dataset/HumanML3D_babel_compat/val/labels.json...
2026-04-18 16:05:13.530 | INFO | robotmdar.dataloader.data:_load_data:228 - Found 2368 valid samples out of 2427
2026-04-18 16:05:13.531 | INFO | robotmdar.dataloader.data:_load_text_embeddings:415 - Loading cached text embeddings...
2026-04-18 16:05:13.908 | INFO | robotmdar.dataloader.data:_load_meanstd:457 - Loading cached mean/std from /data/haozhe/zzn/VAR_FM/ws/project/dataset/HumanML3D_babel_compat/meanstd.pkl...
2026-04-18 16:05:14.899 | INFO | robotmdar.train.manager:hold_model:580 - MVAEManager: Holding VAE model and optimizer
2026-04-18 16:05:15.716 | INFO | robotmdar.train.manager:load_model:676 - Loaded CKPT model & optimizer from /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullTrainability/train-mvae-20260418_154541/ckpt_1000.pth
2026-04-18 16:05:15.717 | INFO | robotmdar.train.manager:load_model:677 - CKPT step: 1000
2026-04-18 16:27:51.371 | INFO | robotmdar.train.manager:save_model:659 - Saved model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullTrainabilityResume/train-mvae-20260418_160348/ckpt_5000.pth
2026-04-18 16:27:51.372 | INFO | robotmdar.train.manager:save_model:660 - Current step: 5000
2026-04-18 16:56:20.654 | INFO | robotmdar.train.manager:save_model:659 - Saved model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullTrainabilityResume/train-mvae-20260418_160348/ckpt_10000.pth
2026-04-18 16:56:20.654 | INFO | robotmdar.train.manager:save_model:660 - Current step: 10000
|