RoboPRO Evaluation Pipeline — Concise Recreate Guide
Purpose: give another team the minimum information needed to rebuild the RoboPRO evaluation pipeline, run the simulator, connect a model, and reproduce the reported SR/HSR results. Socket direction is only a launch detail; reproducibility mostly depends on simulator version, seed banks, task configs, model wrapper, eval knobs, and result aggregation.
1. Reproducibility contract
An eval cell is:
(scene, task, bench_config, checkpoint, seed_bank, num_seeds)
For each seed, RoboPRO runs one SAPIEN/RoboTwin episode, sends observations to the model, executes the returned action chunk, and writes metrics/videos.
To reproduce results, match:
- RoboPRO/customized_robotwin code and assets
- simulator version and render settings
- checkpoint, model code, and model wrapper
- seed bank, queue file, and
EVAL_TEST_NUM - camera selection, preprocessing, proprio mode, and action execution horizon
- same
organize_results.pyscoring path
SR = success rate. HSR = hard success rate: success with no collision.
2. Existing GB10 layout
On the GB10 cluster, the runnable installation is group-readable:
/shared_work/jack/eval_root
├── env.sh
├── code/RoboPRO/ # benchmark + customized_robotwin
│ └── customized_robotwin/script/ # eval_policy_client.py, policy_model_server.py
├── code/DA3-XVLA/ # model code
├── envs/{robotwin,xvla,robotwin_b1} # sim env, model env, SAPIEN b1 swap
├── checkpoints/, ablation_ckpts/ # model weights
├── slurm_eval_node.sh # per-node worker entrypoint
├── stage_k320_node.sh, stage_giants_node.sh
├── organize_results.py
├── orchestrate_*.sh
└── runs/<RUN>/ # queue, leases, logs, raw metrics
/shared_work/jack/robopro-eval-pipeline # $BUNDLE
├── harness/eval_worker_steal_dualgpu.sh
└── seeds/ # fixed seed banks + queue files
To rebuild elsewhere, preserve this structure or update paths consistently in env.sh, stagers, workers, and orchestrators.
3. Runtime architecture
The simulator and model are separate processes connected by one TCP socket. The simulator owns physics, task reset, success checks, videos, and metrics. The model owns checkpoint loading and inference. Observations/actions are sent as length-prefixed JSON-with-numpy frames.
Two launch directions are supported:
MODEL server + SIM client # default in current worker
SIM server + MODEL client # alternate direction
Treat this as plumbing. The reproducible eval pipeline is the same either way: sim sends observations, model returns action chunks, sim executes and scores.
4. Model wrapper contract
The eval is model-agnostic as long as the model is exposed through a policy wrapper.
Default path:
policy/dxvla/deploy_policy.py:get_model(usr_args)
policy/dxvla/dxvla_model.py:DXVLA wrapper
policy/dxvla/deploy_policy.yml
To swap in another model, either reuse the dxvla wrapper with a new CKPT_NAME / GIANT_CODE_DIR, or create policy/<name>/deploy_policy.py and pass --policy_name <name>.
The wrapper should provide either the high-level policy methods:
def reset_model(): ...
def set_language(instruction): ...
def update_observation_window(rgb_list, proprio, extrinsics, intrinsics): ...
def get_action(): ... # [num_actions, action_dim]
or, for the existing DXVLA wrapper style:
def generate_actions(input_ids, image_input, image_mask, domain_id, proprio, ...) -> [B, num_actions, action_dim]
Inputs are already prepared by the eval stack. Typical image_input is [B, V, 3, H, W], language is tokenized in input_ids, and proprio is provided according to the eval setting. Output is a chunk of end-effector actions, typically 30 steps, with 10-D per arm/action block: xyz + rot6d + gripper.
5. Exact eval knobs used for comparable results
Use these for the k320 clean reference configuration. Reported reference: 58% SR / 49% HSR.
DXVLA_SAPIEN_B1=1 # SAPIEN 3.0.0b1, not 3.0.3
WORKERS_PER_GPU=1 # do not raise on GB10; sharing corrupts sim
DXVLA_DENOISER=optix # OIDN is a silent no-op on GB10 Blackwell
DXVLA_RT_SPP=32 # ray-trace samples/pixel
XVLA_POSED_DA3=1 # pass camera extrinsics/intrinsics to DA3 path
XVLA_RGB_INPUT=1
XVLA_SKIP_GEOMETRY_REINIT=1 # keep trained geometry weights
DXVLA_FAKE_PROPRIO=1 # proprio = last commanded action
DXVLA_ACTION_SMOOTH=0.7 # cross-chunk EMA smoothing
DXVLA_SMOOTH_ROT=1 # smooth xyz + rot6d, not just xyz
DXVLA_EXEC_ACTIONS=0 # k320: execute full 30-action chunk
DXVLA_PRIMARY_CAM=countertop
DXVLA_VIDEO_CAM=countertop_camera
USE_EVAL_SEEDS=1
EVAL_RESUME_SEEDS=1
EVAL_TEST_NUM=<N>
Notes:
- k320 used
DXVLA_EXEC_ACTIONS=0. - Giant ablations used
DXVLA_EXEC_ACTIONS=20to generate 30 actions, execute 20, then re-query. - Keep
DXVLA_EXEC_ACTIONSconsistent within comparisons. - All final numbers used
DXVLA_ACTION_SMOOTH=0.7andDXVLA_SMOOTH_ROT=1.
6. Seed banks and queues
Use the same seeds to reproduce the same episodes. Seed banks live under:
$BUNDLE/seeds/
# where $BUNDLE=/shared_work/jack/robopro-eval-pipeline
The eval reads:
<SEEDS_ROOT>/<task>/<task_config>.txt
| eval | SEEDS_ROOT |
queue file | seeds/task |
|---|---|---|---|
| clean | k320_first10_seeds |
k320_first10_queue.txt |
10 seeds, 40000–40009 |
clutter d6–d15 |
clutter_seeds_hf |
clutter_queue_hf_d<N>.txt |
2 per task/level |
| clean fill-in seeds 5–9 | k320_seeds5to9 |
k320_first10_queue.txt |
5 seeds, 40005–40009 |
Example:
$BUNDLE/seeds/k320_first10_seeds/put_cup_in_box/bench_demo_study_clean.txt
# 40000 40001 ... 40009
For exact reproduction, set SEEDS_ROOT, USE_EVAL_SEEDS=1, and the same EVAL_TEST_NUM used in the reported run.
7. Worker, lease queue, and resume
A scalable run uses one shared run directory and many workers.
runs/<RUN>/queue.txt # one scene/task cell per line
runs/<RUN>/queue.txt.leases/ # atomic lock dirs for claimed cells
runs/<RUN>/eval_result/ # raw per-episode metrics
Worker loop:
self-stage code/env/checkpoint if needed
start persistent model process
claim queue cell by atomic mkdir lease
run missing seeds only
write metrics/videos
claim next cell
Resume behavior:
EVAL_RESUME_SEEDS=1counts existing_metrics.jsonlentries and runs only missing seeds.- Rerunning the same run directory skips completed work.
- To clear stale locks, use:
rm -rf runs/<RUN>/queue.txt.leases
Do not use rm -f; queue.txt.leases is a directory.
8. Minimal GB10 run recipe
Example: k320 clean, 5 seeds, node pool.
source /shared_work/jack/eval_root/env.sh
RUN=k320_clean
mkdir -p runs/$RUN/eval_result
cp $BUNDLE/seeds/k320_first10_queue.txt runs/$RUN/queue.txt
rm -rf runs/$RUN/queue.txt.leases
DXVLA_SAPIEN_B1=1 \
STUDY_RUN=$RUN \
CKPT_NAME=da3-xvla-k320-ckpt90k \
BENCH_CONFIG=clean \
SEEDS_ROOT=$BUNDLE/seeds/k320_first10_seeds \
EVAL_TEST_NUM=5 \
WORKERS_PER_GPU=1 \
srun --partition=gb10 \
--nodelist=trt-gb10-[3-15] \
--nodes=13 \
--ntasks-per-node=1 \
--gres=gpu:1 \
--cpus-per-task=18 \
-t 600 \
--kill-on-bad-exit=0 \
bash /shared_work/jack/eval_root/slurm_eval_node.sh
python organize_results.py $RUN k320
Resume by rerunning the same block. Increase EVAL_TEST_NUM to fill in more seeds; completed seeds are skipped.
For giant/geostack models, additionally set model-specific vars such as:
GIANT_CODE_DIR=<model_code_dir>
XVLA_DA3_NATIVE_INPUT=1 # K=160 giant variants
DXVLA_GEOSTACK_STEP=20000 # GeoStack variant, if applicable
Use each checkpoint's EVAL_INTERFACE.md when available.
9. Output and scoring
Raw metrics stream to:
runs/<RUN>/eval_result/<task>/dxvla/bench_demo_<scene>_<cfg>/.../_metrics.jsonl
Each line includes fields like:
{"task": "...", "seed": 40000, "success": true, "hard_success": true}
Aggregate with:
python organize_results.py <RUN> <model_name>
Organized outputs:
runs/<model>/<scene>/<task>/seed<N>.mp4
runs/<model>/<scene>/results.txt
runs/<model>/results.txt
organize_results.py deduplicates latest-per-seed before computing SR/HSR.