File size: 7,243 Bytes
d23210e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | 2026-04-18 18:13:59.485 | INFO | robotmdar.dtype.logger:set:45 - Logging initialized for experiment: OfficialFullDARFromMVAE10000
2026-04-18 18:13:59.488 | INFO | robotmdar.dtype.logger:set:46 - Experiment directory: /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357
2026-04-18 18:13:59.935 | INFO | robotmdar.dataloader.data:_load_data:226 - Loading train data...
2026-04-18 18:14:00.607 | INFO | robotmdar.dataloader.data:_load_statistics:382 - statistics.yaml not found, inferred fps=50 from labels/npz dataset
2026-04-18 18:14:00.624 | INFO | robotmdar.dataloader.data:_load_raw_labels_npz_dataset:390 - Detected labels.json + npz dataset layout, using compatibility loader...
2026-04-18 18:14:01.053 | INFO | robotmdar.dataloader.data:_load_raw_labels_npz_dataset:402 - Loading 21837 samples from /data/haozhe/zzn/VAR_FM/ws/project/dataset/HumanML3D_babel_compat/train/labels.json...
2026-04-18 18:22:12.507 | INFO | robotmdar.dataloader.data:_load_data:254 - Found 21228 valid samples out of 21837
2026-04-18 18:22:12.508 | INFO | robotmdar.dataloader.data:_load_text_embeddings:439 - Loading cached text embeddings...
2026-04-18 18:22:15.591 | INFO | robotmdar.dataloader.data:_load_meanstd:481 - Loading cached mean/std from /data/haozhe/zzn/VAR_FM/ws/project/dataset/HumanML3D_babel_compat/meanstd.pkl...
2026-04-18 18:22:15.717 | INFO | robotmdar.dataloader.data:_load_data:226 - Loading val data...
2026-04-18 18:22:16.478 | INFO | robotmdar.dataloader.data:_load_statistics:382 - statistics.yaml not found, inferred fps=50 from labels/npz dataset
2026-04-18 18:22:16.502 | INFO | robotmdar.dataloader.data:_load_raw_labels_npz_dataset:390 - Detected labels.json + npz dataset layout, using compatibility loader...
2026-04-18 18:22:16.519 | INFO | robotmdar.dataloader.data:_load_raw_labels_npz_dataset:402 - Loading 2427 samples from /data/haozhe/zzn/VAR_FM/ws/project/dataset/HumanML3D_babel_compat/val/labels.json...
2026-04-18 18:23:10.640 | INFO | robotmdar.dataloader.data:_load_data:254 - Found 2368 valid samples out of 2427
2026-04-18 18:23:10.641 | INFO | robotmdar.dataloader.data:_load_text_embeddings:439 - Loading cached text embeddings...
2026-04-18 18:23:11.431 | INFO | robotmdar.dataloader.data:_load_meanstd:481 - Loading cached mean/std from /data/haozhe/zzn/VAR_FM/ws/project/dataset/HumanML3D_babel_compat/meanstd.pkl...
2026-04-18 18:23:23.215 | INFO | robotmdar.train.manager:hold_model:703 - DARManager: Holding denoiser models and optimizer. VAE loaded from checkpoint.
2026-04-18 18:23:23.216 | INFO | robotmdar.train.manager:hold_model:739 - VAE cached to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/vae.pth
2026-04-18 18:23:24.361 | INFO | robotmdar.train.manager:_load_vae_from_checkpoint:783 - Loaded VAE from checkpoint: /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/vae.pth
2026-04-18 18:23:24.370 | INFO | robotmdar.train.manager:_load_vae_from_checkpoint:784 - (latent_mean, latent_std): 0, 1
2026-04-18 19:30:20.043 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_5000.pth
2026-04-18 19:30:20.044 | INFO | robotmdar.train.manager:save_model:878 - Current step: 5000
2026-04-18 19:42:17.783 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_10000.pth
2026-04-18 19:42:17.783 | INFO | robotmdar.train.manager:save_model:878 - Current step: 10000
2026-04-18 19:53:49.066 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_15000.pth
2026-04-18 19:53:49.067 | INFO | robotmdar.train.manager:save_model:878 - Current step: 15000
2026-04-18 20:05:32.026 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_20000.pth
2026-04-18 20:05:32.027 | INFO | robotmdar.train.manager:save_model:878 - Current step: 20000
2026-04-18 20:17:09.833 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_25000.pth
2026-04-18 20:17:09.834 | INFO | robotmdar.train.manager:save_model:878 - Current step: 25000
2026-04-18 20:28:39.797 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_30000.pth
2026-04-18 20:28:39.798 | INFO | robotmdar.train.manager:save_model:878 - Current step: 30000
2026-04-18 20:40:13.665 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_35000.pth
2026-04-18 20:40:13.666 | INFO | robotmdar.train.manager:save_model:878 - Current step: 35000
2026-04-18 20:51:59.270 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_40000.pth
2026-04-18 20:51:59.270 | INFO | robotmdar.train.manager:save_model:878 - Current step: 40000
2026-04-18 21:06:14.895 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_45000.pth
2026-04-18 21:06:14.895 | INFO | robotmdar.train.manager:save_model:878 - Current step: 45000
2026-04-18 21:17:58.694 | INFO | robotmdar.train.manager:save_model:877 - Saved DAR model & optimizer to /data/haozhe/zzn/VAR_FM/ws/project/P_1_Embodied-AI/_reference/TextOp_official/TextOpRobotMDAR/logs/RobotMDAR/OfficialFullDARFromMVAE10000/train-dar-20260418_181357/ckpt_50000.pth
2026-04-18 21:17:58.695 | INFO | robotmdar.train.manager:save_model:878 - Current step: 50000
|