BrutalCaesar/act_so101_cubcyl_recovery_chunk50_aug_3cam

ACT · SO-101 · cubes + cylinder + recovery · 3 cameras · image augmentation ON

Trained by Φ (Physical Hardware Intelligence), the Northeastern Silicon Valley robotics SIG. Code: physical-hardware-intelligence/phi.

This is the augmented arm of an A/B. The control is act_so101_cubcyl_recovery_chunk50_noaug_3cam — identical in every respect except dataset.image_transforms.enable.

---

🚨 Two things will silently break this policy

Neither raises an error. Both look exactly like "the model isn't very good."

1. Calibration frame

A policy emits joint angles in the calibration frame of the machine that recorded its training data. This dataset was recorded on one specific laptop. Run this checkpoint against a different calibration and the arm moves smoothly to the wrong place.

We hit this for real on 2026-08-10: this dataset's policy mis-grasped on a second team member's machine, and copying the original calibration file across fixed it with no retraining.

cp configs/calibration/robots/so_follower/phi_follower.json \
   ~/.cache/huggingface/lerobot/calibration/robots/so_follower/
python -m phi.utils.compare_calibration phi_follower <whatever-you-were-using>

phi_follower.json is committed in the repo and is the frame every Φ dataset and checkpoint shares. The root cause is that LeRobot hardcodes wrist_roll as the full-turn motor (range 0–4095), so unlike every other joint its calibration pose never cancels out — two calibrations of the same physical arm differed on it by 60.4°.

2. Camera keys and their physical mapping

This policy expects exactly three image inputs:

observation.images.wrist
observation.images.front
observation.images.top

All three at 3 × 480 × 640. Miss one and it will not load; swap two and it loads fine and behaves badly.

⚠️ Do not trust a written-down camera index order, including one you read in our repo. macOS has no udev, so device indices reorder silently between sessions. Verify index → physical camera every single session before a rollout.

---

What it is

Policy	ACT with CVAE (use_vae=true, kl_weight=10.0)
Parameters	51,571,590 (+45,824 frozen buffers: BatchNorm stats + VAE pos-enc)
Backbone	ResNet18, ImageNet pretrained, BatchNorm frozen (LeRobot ACT default)
chunk_size / n_action_steps	50 / 50 (1.67 s open-loop at 30 fps)
Cameras	3 × 640×480
Image augmentation	ON — LeRobot stock stack, 3 of 6 sampled per frame
	brightness contrast saturation hue sharpness (photometric)
	affine — RandomAffine, degrees (-5, 5), translate 0.05 ⚠️ see below
Steps · batch · seed	100,000 · 8 · 1000
Wall time	2 h 54 m on one H200 (57% more than the control)

Training data

BrutalCaesar/phi_so101_cubes_cylinder_recovery_v1 — 143 episodes / 81,943 frames = 120 clean + 23 failure-then-recovery.

The 23 recovery episodes each contain a miss or a drop followed by a successful human correction. They are not negative examples: ACT's L1 loss always pulls the prediction toward the demonstrated action, so an episode that ended in failure would teach the failure. They work because each one recovers and completes the placement — positive supervision at off-distribution states.

Split. All 23 recovery episodes are in training. The 30-episode holdout is drawn from clean episodes only, 5 from each of the 6 object × container blocks:

holdout:  0-4 · 20-24 · 45-49 · 65-69 · 90-94 · 110-114        (eval_split = 0.209)
train  :  the remaining 113, including all of 120-142

Results

eval_loss (ACT's full L1 + KL objective) on the 30 clean held-out episodes, at each saved checkpoint:

step	20k	40k	60k	80k	100k
this run (aug on)	0.2149	0.2112	0.2056	0.2046	0.2088
control (aug off)	0.2222	0.2056	0.2058	0.2053	0.2052

The two runs are within ~2% of each other throughout. Single seed. Do not read a winner out of this.

⚠️ Note that 100k is this run's worst late checkpoint. It regressed from 0.2046 at 80k to 0.2088 at 100k. The 80k weights are published separately on the step-80000 branch of this repo. main is 100k so that the A/B against the control uses a fixed step rather than selection on a noisy metric.

A pre-registered caveat about affine

Recorded before either run started, so it cannot be rationalised after the fact. translate=0.05 is ±5% of the frame = ±32 px of 640 ≈ 2 cm at our workspace scale, about one cube width — and RandomAffine moves the image while the action label stays put. It therefore teaches "these joint targets are still correct even though the object appears a cube to the left," which is precisely the invariance a precision grasp must not have.

The held-out loss above does not resolve this either way, and as noted below it is close to blind to grasp precision. If this checkpoint underperforms the control on real rollouts, re-run with only affine zeroed before concluding augmentation as a category does not help. The phi repo has that ablation pre-wired (AFFINE_OFF=1).

🚨 What has NOT been measured

Not scored yet — no success rate on the real arm exists for this checkpoint.

The baseline it has to beat is measured. act_so101_cubcyl_poshold_chunk50_cvae_3cam (same 3 cameras, same holdout, no recovery data) was scored 2026-08-10 over 17 rollouts: 27% success / 0.418 mean progress on 11 held-out, vs 50% / 0.600 on 6 control episodes it trained on. Held-out by object: red cube 67%, yellow cylinder 25%, white cube (45 mm) 0% (0/4).

🔑 The recorded failure mode is the one the recovery episodes target. Operator note on held-out episode 20: if not grasped, the gripper just hovers above and closes and opens repetitively. So the pre-committed prediction for this checkpoint is that hover-and-cycle failures become second attempts.

Two specific things the loss above cannot tell you:

1. Whether recovery improved. The holdout is entirely clean and contains no induced failure, so nothing in it tests recovery. That needs a rollout where the object is deliberately displaced mid-episode to see whether the policy re-approaches.
2. Whether grasp precision changed. L1 over a 50-step chunk is dominated by the large transit motions, not by the millimetre-critical instant of closing the gripper.

Running it

lerobot-rollout \
  --strategy.type=base \
  --policy.path=BrutalCaesar/act_so101_cubcyl_recovery_chunk50_aug_3cam \
  --robot.type=so101_follower \
  --robot.port=<your-port> \
  --robot.id=phi_follower \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: <verify>, width: 640, height: 480, fps: 30},
                     front: {type: opencv, index_or_path: <verify>, width: 640, height: 480, fps: 30},
                     top:   {type: opencv, index_or_path: <verify>, width: 640, height: 480, fps: 30} }" \
  --task="pick up the cubes/cylinder and place it in the box" \
  --duration=60

Camera config must match what was used at recording time, and the calibration must be phi_follower.json. See the two warnings above.

Provenance

Training script configs/hpc/train_recovery.sbatch, write-up experiments/2026-08-10_act-recovery-augmentation.md, both in the phi repo. Full training config is in train_config.json in this repo, including the exact 143-entry episode order that produces the split above.