BrutalCaesar/act_so101_cubcyl_poshold_chunk50_cvae_3cam

ACT (CVAE) - SO-101 - 3 cameras - 3 objects, 2 containers - chunk 50

Action Chunking Transformer with the standard CVAE objective (use_vae=true, kl_weight=10.0), trained on all three camera feeds, with LeRobot 0.6.0. 51.57M parameters.

Task: pick up one of three objects (red 25 mm cube, yellow cylinder, white 45 mm cube) and place it in whichever container is on the table.

Final eval_loss on the held-out 30 episodes: 0.2006 — the second-lowest of six variants, and within noise of all of them.

Paired baseline: act_so101_cubcyl_poshold_chunk50_bc_3cam, identical except use_vae=false.

The latent is collapsed, and the penalty is not why

A β sweep at kl_weight = 10, 1 and 0.1 all finished with train loss 0.056-0.057 — identical to the BC run's 0.059, which has no KL term at all. At β=0.1 a KL of 1 nat would add 0.1 to the loss and nothing of the sort appears, putting the KL under roughly 0.02 nats regardless of β.

So collapse is not caused by the penalty. The likely reason is that observation.state already discloses which grasp approach is underway for most frames: the demonstrations are multimodal at the task level but close to deterministic per-observation, which is what the encoder conditions on.

Treat this as a near-deterministic chunk regressor. Do not expect behavioural diversity from the latent.

✅ Camera keys are correct — and all THREE are required

Physical camera	Observation key
wrist (gripper module)	observation.images.wrist
front (desk-level Brio 101)	observation.images.front
top (overhead, boom arm)	observation.images.top

This policy will not run without the overhead camera connected. Its two-camera sibling (..._chunk50_cvae) takes wrist+front only.

Keys verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the transposed mapping from the 8bin models.

Held-out episodes — evaluate on these

[0,1,2,3,4, 20,21,22,23,24, 45,46,47,48,49, 65,66,67,68,69, 90,91,92,93,94, 110,111,112,113,114]

5 episodes from each of the 6 object x container blocks. By episode_index: red cube (25 mm) 0-39, yellow cylinder 40-79, white cube (45 mm) 80-119; cardboard box in the first 20 of each block, white 3D-printed bin in the second 20. Only one container was on the table at a time, so the visible bin is always the target.

Inputs / outputs

	Shape
observation.images.wrist	(3, 480, 640)
observation.images.front	(3, 480, 640)
observation.images.top	(3, 480, 640)
observation.state	(6,)
action	(50, 6) — 50-step chunk, 1.67 s at 30 fps, all executed

903 encoder tokens (301 per camera at 640x480, plus latent and state), against 602 for the two-camera variants. Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights — load via ACTPolicy.from_pretrained.

🚨 All six variants are indistinguishable on held-out loss

Variant	cameras	latent	eval_loss @100k
cvae	2	CVAE β=10	0.2041
bc	2	none	0.1988
cvae_kl1	2	CVAE β=1	0.2014
cvae_kl0p1	2	CVAE β=0.1	0.2018
cvae_3cam	3	CVAE β=10	0.2006
bc_3cam	3	none	0.2067

The full spread is 0.0079. A single run's eval_loss bounces by ~0.008 between consecutive checkpoints. The between-model differences do not exceed the within-model noise.

And the metric is biased for this data: the demonstrations contain roughly three distinct grasp approaches per object, so held-out L1 rewards a policy that averages them over one that commits to a single valid approach. A worse number here may be a better policy on the robot.

Only scored rollouts can rank these.

Training

Dataset	phi_so101_cubes_cylinder_v1, 90 of 120 episodes, 49,969 frames
Steps / batch / seed	100,000 / 8 / 1000 (16.0 epochs)
Optimizer	AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone	ResNet-18 per camera, ImageNet-pretrained, not frozen
n_obs_steps	1 — single frame, no history
Hardware	1x H200, 2 h 00 m wall, 5.7 GB VRAM

eval_loss is flat from step 60,000 to 100,000, so no overfitting penalty; this is the 100,000-step checkpoint.

Known limitations

• n_obs_steps=1, so no velocity information.
• Scene-dependent: the cardboard box and white bin as used during recording, and the camera poses as mounted. A moved camera or a different container is a domain shift.
• One room, one lighting setup, one operator.
• ⚠️ A camera frame-rate warning was observed on battery power during roughly the first 50 episodes of collection. Reported but unconfirmed — a pixel-level check found 4-9% stalled frames uniformly with no early-vs-late pattern.
• ⚠️ The white cube is low contrast against the light table, most visibly in the overhead view — which this policy uses. Episodes 80-119 are the likeliest to underperform.