arielchen07/pi05_qwen_behavior_15tasks_aug_with_ir_train_bs256_4gpu_23dim

pi0.5-Qwen — BEHAVIOR-1K 15-task, dot-IR conditioned

Checkpoint at step 100000 of the training config pi05_qwen_behavior_15tasks_aug_with_ir_train_bs256_4gpu_23dim.

A pi0.5 policy with a Qwen3.5-4B backbone, trained on 15 BEHAVIOR-1K manipulation task families (behavior_15tasks_aug), 23-dim action space, action horizon 32.

The 4th image ("dot IR")

Unlike a stock pi0.5 this model takes four camera images: head + two wrists, plus an intermediate-representation image that is the head RGB with a colored dot drawn at the target object's projected center. It is declared in the config as extra_image_keys=("object_center_0_rgb",) with max_token_len=320.

At eval time that image is rendered server-side from an object center the sim client sends over the wire — there is nothing to precompute. Enable with WITH_IR=1 and IR_STYLE=dot (the default). IR_STYLE=heatmap belongs to a different experiment and will silently produce garbage with this checkpoint.

Contents

100000/
  model.safetensors                                    # 9.2 GB, bf16
  metadata.pt
  assets/behavior_15tasks_aug/train/norm_stats.json    # REQUIRED

optimizer.pt has been removed — this is an inference-only release.

Keep the tree intact. norm_stats.json is what de-normalizes the predicted actions; load model.safetensors without it and the policy runs clean while emitting mis-scaled actions, with no error.

Usage

hf download arielchen07/pi05_qwen_behavior_15tasks_aug_with_ir_train_bs256_4gpu_23dim \
    --local-dir /your/path/15tasks_ir

Then, with https://github.com/arielchen07/openpi-spatialvla (branch eval-handoff-15tasks-ir):

CKPT_DIR=/your/path/15tasks_ir STEP=100000 \
CONFIG=pi05_qwen_behavior_15tasks_aug_with_ir_train_bs256_4gpu_23dim \
TASK=reach_dice SLICE=train_sanity WITH_IR=1 IR_STYLE=dot \
bash scripts/behavior_eval/_submit_pool_eval.sh

Running the sim eval additionally needs BEHAVIOR-1K + OmniGibson + Isaac Sim, and the eval task instances (BDDL/TRO). See HANDOFF.md in the eval bundle.

Training

batch size	256 (4 GPU × 64)
LR schedule	cosine, warmup 4k, peak 4e-4, decay_steps 1e6 (no anneal), floor 2e-5
optimizer	AdamW, grad clip 1.0
precision	bfloat16
backbone	frozen; prefix backbone compiled
norm stats	reused from the no-IR 15-task run (image-independent)

Caveats

Held-out performance is substantially below train-sanity. That gap is the research finding, not a broken setup: the policy exhibits a nearest-candidate grounding shortcut and largely fails to generalize to novel object layouts. Do not read low held-out numbers as a configuration error.