ACT CVAE, cubes+cylinder position holdout, chunk50, 100k steps, eval_loss 0.2041

README.md

@@ -0,0 +1,108 @@
+---
+library_name: lerobot
+license: apache-2.0
+tags: [robotics, act, so101, imitation-learning, lerobot]
+datasets: [BrutalCaesar/phi_so101_cubes_cylinder_v1]
+pipeline_tag: robotics
+---
+
+# ACT (CVAE) - SO-101 - 3 objects, 2 containers - wrist+front - chunk 50
+
+Action Chunking Transformer, standard CVAE objective (`use_vae=true`, `kl_weight=10.0`), trained with
+[LeRobot](https://github.com/huggingface/lerobot) 0.6.0.
+
+Task: pick up one of three objects (red 25 mm cube, yellow cylinder, white 45 mm cube) and place it in
+whichever container is on the table (cardboard box or white 3D-printed bin).
+
+**Paired baseline**: [`act_so101_cubcyl_poshold_chunk50_bc`](https://huggingface.co/BrutalCaesar/act_so101_cubcyl_poshold_chunk50_bc)
+— identical in every respect except `use_vae=false`. Trained to answer whether the CVAE objective
+matters on this data.
+
+**Final eval_loss (held-out 30 episodes): 0.2041.** Note this includes the `kl_weight x KL` term and
+the BC model's does not, so the two numbers are not directly comparable.
+
+## The latent is collapsed, and lowering the penalty did not change that
+
+Measured across a β sweep at `kl_weight` = 10, 1 and 0.1: all three finished with train loss
+0.056-0.057, identical to the BC run's 0.059, which has no KL term at all. At β=0.1 a KL of 1 nat
+would add 0.1 to the loss; nothing of the sort appears, putting the KL below roughly 0.02 nats
+regardless of β.
+
+So the collapse is not caused by the penalty. The likely explanation is that `observation.state`
+already discloses which grasp approach is underway for most frames — the demonstrations are
+multimodal at the task level but close to deterministic per-observation, which is what the encoder
+actually conditions on.
+
+Practical consequence: **this model behaves close to a deterministic chunk regressor.** Do not expect
+behavioural diversity from the latent.
+
+## ✅ Camera keys are correct — wire by name
+
+| Physical camera | Observation key |
+|---|---|
+| wrist (gripper module) | `observation.images.wrist` |
+| front (desk-level Brio 101) | `observation.images.front` |
+
+Verified against pixels at episodes 8, 39 and 67 of the source dataset. **Do not** apply the
+transposed mapping from the `8bin` models — that swap belongs to a different dataset.
+
+The `top` camera was recorded but is **not** an input to this policy.
+
+## Held-out episodes — evaluate on these
+
+30 episodes were excluded from training: 5 from each of the 6 object x container blocks.
+
+```
+[0,1,2,3,4, 20,21,22,23,24, 45,46,47,48,49, 65,66,67,68,69, 90,91,92,93,94, 110,111,112,113,114]
+```
+
+Objects and containers, by `episode_index`: red cube 0-39, yellow cylinder 40-79, white cube 80-119;
+cardboard box in the first 20 of each block, white 3D-printed bin in the second 20. Only one
+container was on the table at a time, so the visible bin is always the target.
+
+## Inputs / outputs
+
+| | Shape |
+|---|---|
+| `observation.images.wrist` | `(3, 480, 640)` |
+| `observation.images.front` | `(3, 480, 640)` |
+| `observation.state` | `(6,)` |
+| **action** | `(50, 6)` — 50-step chunk, **1.67 s at 30 fps**, all executed |
+
+Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights. Load through
+`ACTPolicy.from_pretrained` or actions come out in the wrong units.
+
+## 🚨 Evaluation caveat that applies to both models
+
+These two policies are **indistinguishable on held-out loss**. Final `eval_loss`: 0.2041 (CVAE) vs
+0.1988 (BC). But a single model's eval bounces by ~0.008 between consecutive checkpoints, so the
+0.005 spread **between** models is smaller than the noise within either one.
+
+Worse, the metric is biased for this data. The demonstrations contain roughly three distinct grasp
+approaches per object, and **held-out L1 systematically rewards a policy that averages them** — a
+policy committing crisply to approach 2 is penalised on every held-out episode that used approach 1
+or 3. BC's marginally lower number is exactly what an averaging model would produce.
+
+**Only scored rollouts can rank these.** Do not conclude from the loss.
+
+## Training
+
+| | |
+|---|---|
+| Dataset | [`phi_so101_cubes_cylinder_v1`](https://huggingface.co/datasets/BrutalCaesar/phi_so101_cubes_cylinder_v1), 90 of 120 episodes, 49,969 frames |
+| Steps / batch / seed | 100,000 / 8 / 1000 (16.0 epochs) |
+| Optimizer | AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4 |
+| Backbone | ResNet-18, ImageNet-pretrained, not frozen |
+| `n_obs_steps` | 1 — single frame, no history |
+| Hardware | 1x H200, Northeastern Explorer |
+
+No overfitting was observed: `eval_loss` is flat from step 60,000 to 100,000, so any checkpoint in
+that range is equivalent. This is the 100,000-step checkpoint.
+
+## Known limitations
+
+- `n_obs_steps=1`, so no velocity information.
+- Scene-dependent: the cardboard box and white bin as used during recording. A different container is a domain shift.
+- One room, one lighting setup, one operator.
+- ⚠️ The operator observed a camera frame-rate warning on battery power during roughly the first 50 episodes of collection. Recorded as reported; a pixel-level check found 4-9% stalled frames uniformly across the dataset with no early-vs-late pattern, so this is **unconfirmed**.
+- ⚠️ The white cube is low contrast against the light table in the overhead view. Not an input here (top camera unused), but it is the object most likely to underperform.

config.json

@@ -0,0 +1,72 @@
+{
+    "type": "act",
+    "n_obs_steps": 1,
+    "input_features": {
+        "observation.state": {
+            "type": "STATE",
+            "shape": [
+                6
+            ]
+        },
+        "observation.images.wrist": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        },
+        "observation.images.front": {
+            "type": "VISUAL",
+            "shape": [
+                3,
+                480,
+                640
+            ]
+        }
+    },
+    "output_features": {
+        "action": {
+            "type": "ACTION",
+            "shape": [
+                6
+            ]
+        }
+    },
+    "device": "cuda",
+    "use_amp": false,
+    "use_peft": false,
+    "push_to_hub": false,
+    "repo_id": null,
+    "private": null,
+    "tags": null,
+    "license": null,
+    "pretrained_path": null,
+    "pretrained_revision": null,
+    "chunk_size": 50,
+    "n_action_steps": 50,
+    "normalization_mapping": {
+        "VISUAL": "MEAN_STD",
+        "STATE": "MEAN_STD",
+        "ACTION": "MEAN_STD"
+    },
+    "vision_backbone": "resnet18",
+    "pretrained_backbone_weights": "ResNet18_Weights.IMAGENET1K_V1",
+    "replace_final_stride_with_dilation": false,
+    "pre_norm": false,
+    "dim_model": 512,
+    "n_heads": 8,
+    "dim_feedforward": 3200,
+    "feedforward_activation": "relu",
+    "n_encoder_layers": 4,
+    "n_decoder_layers": 1,
+    "use_vae": true,
+    "latent_dim": 32,
+    "n_vae_encoder_layers": 4,
+    "temporal_ensemble_coeff": null,
+    "dropout": 0.1,
+    "kl_weight": 10.0,
+    "optimizer_lr": 1e-05,
+    "optimizer_weight_decay": 0.0001,
+    "optimizer_lr_backbone": 1e-05
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:743ea32f2abe16c7c769dd710104927fa94c6c3654f17ea3e6c72b7eaf981d6b
+size 206494928

policy_postprocessor.json

@@ -0,0 +1,32 @@
+{
+  "name": "policy_postprocessor",
+  "steps": [
+    {
+      "registry_name": "unnormalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MEAN_STD",
+          "ACTION": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_postprocessor_step_0_unnormalizer_processor.safetensors"
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cpu",
+        "float_dtype": null
+      }
+    }
+  ]
+}

policy_postprocessor_step_0_unnormalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:709c02731a00cdbc979e0288e10069e14204c6dc34ca9ac0995a16a2141ad3dc
+size 8528

policy_preprocessor.json

@@ -0,0 +1,64 @@
+{
+  "name": "policy_preprocessor",
+  "steps": [
+    {
+      "registry_name": "rename_observations_processor",
+      "config": {
+        "rename_map": {}
+      }
+    },
+    {
+      "registry_name": "to_batch_processor",
+      "config": {}
+    },
+    {
+      "registry_name": "device_processor",
+      "config": {
+        "device": "cuda",
+        "float_dtype": null
+      }
+    },
+    {
+      "registry_name": "normalizer_processor",
+      "config": {
+        "eps": 1e-08,
+        "features": {
+          "observation.state": {
+            "type": "STATE",
+            "shape": [
+              6
+            ]
+          },
+          "observation.images.wrist": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "observation.images.front": {
+            "type": "VISUAL",
+            "shape": [
+              3,
+              480,
+              640
+            ]
+          },
+          "action": {
+            "type": "ACTION",
+            "shape": [
+              6
+            ]
+          }
+        },
+        "norm_map": {
+          "VISUAL": "MEAN_STD",
+          "STATE": "MEAN_STD",
+          "ACTION": "MEAN_STD"
+        }
+      },
+      "state_file": "policy_preprocessor_step_3_normalizer_processor.safetensors"
+    }
+  ]
+}

policy_preprocessor_step_3_normalizer_processor.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b48043906358fa82dcd15ba6cfbfaaa9a8e10aa9f08fa4c0e6adaed14cc6cef2
+size 8536

train_config.json

@@ -0,0 +1,340 @@
+{
+    "dataset": {
+        "repo_id": "BrutalCaesar/phi_so101_cubes_cylinder_v1",
+        "root": null,
+        "episodes": [
+            5,
+            6,
+            7,
+            8,
+            9,
+            10,
+            11,
+            12,
+            13,
+            14,
+            15,
+            16,
+            17,
+            18,
+            19,
+            25,
+            26,
+            27,
+            28,
+            29,
+            30,
+            31,
+            32,
+            33,
+            34,
+            35,
+            36,
+            37,
+            38,
+            39,
+            40,
+            41,
+            42,
+            43,
+            44,
+            50,
+            51,
+            52,
+            53,
+            54,
+            55,
+            56,
+            57,
+            58,
+            59,
+            60,
+            61,
+            62,
+            63,
+            64,
+            70,
+            71,
+            72,
+            73,
+            74,
+            75,
+            76,
+            77,
+            78,
+            79,
+            80,
+            81,
+            82,
+            83,
+            84,
+            85,
+            86,
+            87,
+            88,
+            89,
+            95,
+            96,
+            97,
+            98,
+            99,
+            100,
+            101,
+            102,
+            103,
+            104,
+            105,
+            106,
+            107,
+            108,
+            109,
+            115,
+            116,
+            117,
+            118,
+            119,
+            0,
+            1,
+            2,
+            3,
+            4,
+            20,
+            21,
+            22,
+            23,
+            24,
+            45,
+            46,
+            47,
+            48,
+            49,
+            65,
+            66,
+            67,
+            68,
+            69,
+            90,
+            91,
+            92,
+            93,
+            94,
+            110,
+            111,
+            112,
+            113,
+            114
+        ],
+        "image_transforms": {
+            "enable": false,
+            "max_num_transforms": 3,
+            "random_order": false,
+            "tfs": {
+                "brightness": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "brightness": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "contrast": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "contrast": [
+                            0.8,
+                            1.2
+                        ]
+                    }
+                },
+                "saturation": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "saturation": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "hue": {
+                    "weight": 1.0,
+                    "type": "ColorJitter",
+                    "kwargs": {
+                        "hue": [
+                            -0.05,
+                            0.05
+                        ]
+                    }
+                },
+                "sharpness": {
+                    "weight": 1.0,
+                    "type": "SharpnessJitter",
+                    "kwargs": {
+                        "sharpness": [
+                            0.5,
+                            1.5
+                        ]
+                    }
+                },
+                "affine": {
+                    "weight": 1.0,
+                    "type": "RandomAffine",
+                    "kwargs": {
+                        "degrees": [
+                            -5.0,
+                            5.0
+                        ],
+                        "translate": [
+                            0.05,
+                            0.05
+                        ]
+                    }
+                }
+            }
+        },
+        "revision": null,
+        "use_imagenet_stats": true,
+        "video_backend": "pyav",
+        "return_uint8": false,
+        "depth_output_unit": "mm",
+        "streaming": false,
+        "eval_split": 0.25
+    },
+    "env": null,
+    "policy": {
+        "type": "act",
+        "n_obs_steps": 1,
+        "input_features": {
+            "observation.state": {
+                "type": "STATE",
+                "shape": [
+                    6
+                ]
+            },
+            "observation.images.wrist": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    480,
+                    640
+                ]
+            },
+            "observation.images.front": {
+                "type": "VISUAL",
+                "shape": [
+                    3,
+                    480,
+                    640
+                ]
+            }
+        },
+        "output_features": {
+            "action": {
+                "type": "ACTION",
+                "shape": [
+                    6
+                ]
+            }
+        },
+        "device": "cuda",
+        "use_amp": false,
+        "use_peft": false,
+        "push_to_hub": false,
+        "repo_id": null,
+        "private": null,
+        "tags": null,
+        "license": null,
+        "pretrained_path": null,
+        "pretrained_revision": null,
+        "chunk_size": 50,
+        "n_action_steps": 50,
+        "normalization_mapping": {
+            "VISUAL": "MEAN_STD",
+            "STATE": "MEAN_STD",
+            "ACTION": "MEAN_STD"
+        },
+        "vision_backbone": "resnet18",
+        "pretrained_backbone_weights": "ResNet18_Weights.IMAGENET1K_V1",
+        "replace_final_stride_with_dilation": false,
+        "pre_norm": false,
+        "dim_model": 512,
+        "n_heads": 8,
+        "dim_feedforward": 3200,
+        "feedforward_activation": "relu",
+        "n_encoder_layers": 4,
+        "n_decoder_layers": 1,
+        "use_vae": true,
+        "latent_dim": 32,
+        "n_vae_encoder_layers": 4,
+        "temporal_ensemble_coeff": null,
+        "dropout": 0.1,
+        "kl_weight": 10.0,
+        "optimizer_lr": 1e-05,
+        "optimizer_weight_decay": 0.0001,
+        "optimizer_lr_backbone": 1e-05
+    },
+    "reward_model": null,
+    "output_dir": "/scratch/gupta.yashv/phi/results/act_cubcyl_poshold_chunk50_cvae",
+    "job_name": "act_cubcyl_poshold_chunk50_cvae",
+    "resume": false,
+    "seed": 1000,
+    "cudnn_deterministic": false,
+    "num_workers": 16,
+    "batch_size": 8,
+    "prefetch_factor": 4,
+    "persistent_workers": true,
+    "steps": 100000,
+    "env_eval_freq": 20000,
+    "log_freq": 200,
+    "eval_steps": 10000,
+    "max_eval_samples": 0,
+    "tolerance_s": 0.0001,
+    "save_checkpoint": true,
+    "save_freq": 20000,
+    "use_policy_training_preset": true,
+    "optimizer": {
+        "type": "adamw",
+        "lr": 1e-05,
+        "weight_decay": 0.0001,
+        "grad_clip_norm": 10.0,
+        "betas": [
+            0.9,
+            0.999
+        ],
+        "eps": 1e-08
+    },
+    "scheduler": null,
+    "eval": {
+        "n_episodes": 50,
+        "batch_size": 50,
+        "use_async_envs": true,
+        "recording": false,
+        "recording_repo_id": null,
+        "recording_private": false
+    },
+    "wandb": {
+        "enable": true,
+        "disable_artifact": false,
+        "project": "phi-act",
+        "entity": null,
+        "notes": "cubes+cylinder, position holdout (5 eps from each of 6 object x bin blocks, 30 held out). use_vae=true. chunk=50. eval_loss on the held-out 30 every 10000 steps.",
+        "run_id": "hipg6v1q",
+        "mode": "online",
+        "add_tags": true
+    },
+    "peft": null,
+    "job": {
+        "target": null,
+        "image": "huggingface/lerobot-gpu:latest",
+        "timeout": "2d",
+        "detach": false,
+        "tags": []
+    },
+    "save_checkpoint_to_hub": false,
+    "sample_weighting": null,
+    "rename_map": {},
+    "checkpoint_path": null
+}