madokalif/xlerobot-pi05-bottle

XLeRobot pi0.5 - bottle pick-and-place (sim)

pi0.5 LoRA fine-tune of pi05_base for the XLeRobot's left arm picking a Life Extension Two-Per-Day bottle off a table and standing it on a randomized green pad, trained entirely in ManiSkill. Two stages: a fixed-scene run (step-*) and a domain-randomized fine-tune on top of it (dr2-step-*), which is the one to use.

• Configs: pi05_xlerobot_orange (stage 1) and pi05_xlerobot_orange_dr (stage 2), registered into openpi by experiments/vla_orange/openpi_xlerobot/install_openpi.py
• Model: Pi0Config(pi05=True), gemma_2b_lora + gemma_300m_lora, action_horizon 10, EMA off
• Training: batch 32, FSDP across 2x RTX 3090. Stage 1 stopped at 33k steps; stage 2 ran 12k steps from stage 1's weights (fresh optimizer state — the dataset and its norm stats differ), peak LR 5e-5 cosine to 5e-6.
• Data: scripted-IK demos collected in ManiSkill, head + wrist cameras at 224x224. Stage 1: 1024 episodes, fixed scene. Stage 2: 512 episodes with table height 0.760-0.860 m, per-env lighting, background walls/floor/clutter, camera pose+fov jitter, and a rest pose moved off the spawn region's sightline. Scripted success 88.9%.
• Prompt: put the bottle on the green pad

Results

eval_sim.py, 32 episodes each, closed-loop through the same websocket policy server used for deployment. The scripted policy that generated the data scores 88.9%.

checkpoint	scene	episode cap	success	grasp	lift
step-33000 (stage 1)	fixed	280	71.9%	-	-
step-33000 (stage 1)	domain-randomized	280	0%	-	-
dr2-step-7000	domain-randomized	280	34.4%	90.6%	71.9%
dr2-step-11999	domain-randomized	280	56.2%	84.4%	84.4%
dr2-step-11999	domain-randomized	400	71.9%	84.4%	78.1%
dr2-step-11999	DR, table pinned to 0.851 m	280	50.0%	90.6%	75.0%
dr2-step-11999	DR, table pinned to 0.851 m	400	68.8%	93.8%	75.0%

The episode cap matters more than it looks. The scripted demos average 254 steps and the env registers max_episode_steps=280, leaving 26 steps of slack; the learned policy is slower than the script, so raising the cap to 400 converts most "failures" into successes — the lifted-but-not-placed rate falls from 28% to 9%. Budget at least 400 steps at deployment (more on hardware, where deploy_client.py's per-step clamp slows the arm further).

Motion profile. The commanded per-step joint deltas are bursty rather than uniformly fast: median 0.010 rad but p99 0.25 rad, with 3–6% of steps exceeding deploy_client's 0.067 rad clamp. The bursts are not chunk-boundary artifacts — jump statistics are flat across the position within the action chunk. An RTC (real-time chunking, arXiv:2506.07339) guided sampler is available for these checkpoints (experiments/vla_orange/openpi_xlerobot/pi0_rtc.py, installed into openpi by install_openpi.py); serve as usual and send optional prev_actions / inference_delay / rtc_guidance_weight request keys to activate it. In zero-latency sim it neither helps nor hurts (the splice discontinuity it repairs only exists under real inference latency); it costs ~65% extra inference time (270 ms vs ~165 ms on an RTX 3090). Note the short action horizon (10 steps) bounds how much latency RTC can hide: at 30 Hz control the chunk spans 333 ms, so plan for a lower control rate or a longer retrained horizon if end-to-end latency approaches that.

Checkpoints

<name>/ is an orbax checkpoint directory exactly as openpi writes it:

• params/ - weights, enough to serve the policy
• train_state/ - optimizer state; needed to --resume training from this point

uv run scripts/serve_policy.py policy:checkpoint \
    --policy.config=pi05_xlerobot_orange_dr \
    --policy.dir=<path>/dr2-step-11999

dr2-step-11999 is the final stage-2 checkpoint: num_train_steps=12000 counts steps 0..11999, so 11999 is the last one written — there is no dr2-step-12000. Serve dr2-* under pi05_xlerobot_orange_dr and step-* under pi05_xlerobot_orange; the two have different norm stats and swapping them silently denormalizes actions with the wrong statistics.

Evaluating a DR checkpoint: do not use --no-dr

eval_sim.py --no-dr (and safety_probe.py, which hardcodes the same thing) turns off two things at once: the visual randomization AND the arm's start posture, because PickOrangeEnv._rest_for_height returns the legacy Q_REST verbatim when randomization is off. Every stage-2 episode starts from a different rest pose, so a dr2-* checkpoint measured that way is being started from a posture it has never seen — grasp rate collapses from 91% to 9% on that alone. The flag is correct for the stage-1 step-* checkpoints, which is what it was written for. To pin one scene parameter for a stage-2 checkpoint, keep domain_randomization=True and constrain that parameter instead (e.g. table_z_range=(0.851, 0.851)).

Observation and action spaces

key	shape	meaning
observation/image	224x224x3 uint8	head camera
observation/wrist_image	224x224x3 uint8	acting (left) arm's wrist camera
observation/state	6 float32	[Rotation, Pitch, Elbow, Wrist_Pitch, Wrist_Roll, Jaw], absolute radians
actions	10x6 float32	same 6-D vector, absolute radians, action horizon 10

The robot is an XLeRobot 0.4.0: two SO-101 arms on a trolley whose top is flush with the table at 0.775 m. Only the left arm acts; the right arm is frozen in frame, and the base and head are frozen.

Known limits

• Sim-only. No real robot data went into training, and no hardware run has completed the task. On the committed real captures the policy does respond to real frames (its action separates from the blank-image action by 0.28), but that response is about 26% of the steering range it shows on sim frames — real images are read, less sharply than sim ones.
• Commanded motion is faster than deployment allows. Per-step joint jumps reach 0.40 rad against deploy_client.py's ~0.067 rad clamp, so the clamp engages constantly and the arm lags the intended trajectory.
• A bare scene is out of distribution. The randomization covers the nominal value of every continuous parameter, but the structural elements are always on: back wall, floor overlay, exactly 4 clutter pieces, shadows. An empty fixed scene scores well below the randomized one.
• Height safety is clear: at the measured bench height (0.851 m) none of 1120 commanded actions put the jaw tip below the tabletop, so the floor guard never engages.

Licensing

These are LoRA fine-tunes of pi05_base, which is built on PaliGemma and a Gemma action expert, so the weights are governed by the Gemma Terms of Use (openpi ships LICENSE_GEMMA.txt alongside its Apache-2.0 code for exactly this reason). openpi itself is Apache-2.0.