# RoboPRO Evaluation Pipeline — Concise Recreate Guide Purpose: give another team the minimum information needed to rebuild the RoboPRO evaluation pipeline, run the simulator, connect a model, and reproduce the reported SR/HSR results. Socket direction is only a launch detail; reproducibility mostly depends on simulator version, seed banks, task configs, model wrapper, eval knobs, and result aggregation. --- ## 1. Reproducibility contract An eval cell is: ```text (scene, task, bench_config, checkpoint, seed_bank, num_seeds) ``` For each seed, RoboPRO runs one SAPIEN/RoboTwin episode, sends observations to the model, executes the returned action chunk, and writes metrics/videos. To reproduce results, match: - RoboPRO/customized_robotwin code and assets - simulator version and render settings - checkpoint, model code, and model wrapper - seed bank, queue file, and `EVAL_TEST_NUM` - camera selection, preprocessing, proprio mode, and action execution horizon - same `organize_results.py` scoring path SR = success rate. HSR = hard success rate: success with no collision. --- ## 2. Existing GB10 layout On the GB10 cluster, the runnable installation is group-readable: ```text /shared_work/jack/eval_root ├── env.sh ├── code/RoboPRO/ # benchmark + customized_robotwin │ └── customized_robotwin/script/ # eval_policy_client.py, policy_model_server.py ├── code/DA3-XVLA/ # model code ├── envs/{robotwin,xvla,robotwin_b1} # sim env, model env, SAPIEN b1 swap ├── checkpoints/, ablation_ckpts/ # model weights ├── slurm_eval_node.sh # per-node worker entrypoint ├── stage_k320_node.sh, stage_giants_node.sh ├── organize_results.py ├── orchestrate_*.sh └── runs// # queue, leases, logs, raw metrics /shared_work/jack/robopro-eval-pipeline # $BUNDLE ├── harness/eval_worker_steal_dualgpu.sh └── seeds/ # fixed seed banks + queue files ``` To rebuild elsewhere, preserve this structure or update paths consistently in `env.sh`, stagers, workers, and orchestrators. --- ## 3. Runtime architecture The simulator and model are separate processes connected by one TCP socket. The simulator owns physics, task reset, success checks, videos, and metrics. The model owns checkpoint loading and inference. Observations/actions are sent as length-prefixed JSON-with-numpy frames. Two launch directions are supported: ```text MODEL server + SIM client # default in current worker SIM server + MODEL client # alternate direction ``` Treat this as plumbing. The reproducible eval pipeline is the same either way: sim sends observations, model returns action chunks, sim executes and scores. --- ## 4. Model wrapper contract The eval is model-agnostic as long as the model is exposed through a policy wrapper. Default path: ```text policy/dxvla/deploy_policy.py:get_model(usr_args) policy/dxvla/dxvla_model.py:DXVLA wrapper policy/dxvla/deploy_policy.yml ``` To swap in another model, either reuse the `dxvla` wrapper with a new `CKPT_NAME` / `GIANT_CODE_DIR`, or create `policy//deploy_policy.py` and pass `--policy_name `. The wrapper should provide either the high-level policy methods: ```python def reset_model(): ... def set_language(instruction): ... def update_observation_window(rgb_list, proprio, extrinsics, intrinsics): ... def get_action(): ... # [num_actions, action_dim] ``` or, for the existing DXVLA wrapper style: ```python def generate_actions(input_ids, image_input, image_mask, domain_id, proprio, ...) -> [B, num_actions, action_dim] ``` Inputs are already prepared by the eval stack. Typical `image_input` is `[B, V, 3, H, W]`, language is tokenized in `input_ids`, and proprio is provided according to the eval setting. Output is a chunk of end-effector actions, typically 30 steps, with 10-D per arm/action block: xyz + rot6d + gripper. --- ## 5. Exact eval knobs used for comparable results Use these for the k320 clean reference configuration. Reported reference: **58% SR / 49% HSR**. ```bash DXVLA_SAPIEN_B1=1 # SAPIEN 3.0.0b1, not 3.0.3 WORKERS_PER_GPU=1 # do not raise on GB10; sharing corrupts sim DXVLA_DENOISER=optix # OIDN is a silent no-op on GB10 Blackwell DXVLA_RT_SPP=32 # ray-trace samples/pixel XVLA_POSED_DA3=1 # pass camera extrinsics/intrinsics to DA3 path XVLA_RGB_INPUT=1 XVLA_SKIP_GEOMETRY_REINIT=1 # keep trained geometry weights DXVLA_FAKE_PROPRIO=1 # proprio = last commanded action DXVLA_ACTION_SMOOTH=0.7 # cross-chunk EMA smoothing DXVLA_SMOOTH_ROT=1 # smooth xyz + rot6d, not just xyz DXVLA_EXEC_ACTIONS=0 # k320: execute full 30-action chunk DXVLA_PRIMARY_CAM=countertop DXVLA_VIDEO_CAM=countertop_camera USE_EVAL_SEEDS=1 EVAL_RESUME_SEEDS=1 EVAL_TEST_NUM= ``` Notes: - k320 used `DXVLA_EXEC_ACTIONS=0`. - Giant ablations used `DXVLA_EXEC_ACTIONS=20` to generate 30 actions, execute 20, then re-query. - Keep `DXVLA_EXEC_ACTIONS` consistent within comparisons. - All final numbers used `DXVLA_ACTION_SMOOTH=0.7` and `DXVLA_SMOOTH_ROT=1`. --- ## 6. Seed banks and queues Use the same seeds to reproduce the same episodes. Seed banks live under: ```text $BUNDLE/seeds/ # where $BUNDLE=/shared_work/jack/robopro-eval-pipeline ``` The eval reads: ```text //.txt ``` | eval | `SEEDS_ROOT` | queue file | seeds/task | |---|---|---|---| | clean | `k320_first10_seeds` | `k320_first10_queue.txt` | 10 seeds, `40000`–`40009` | | clutter `d6`–`d15` | `clutter_seeds_hf` | `clutter_queue_hf_d.txt` | 2 per task/level | | clean fill-in seeds 5–9 | `k320_seeds5to9` | `k320_first10_queue.txt` | 5 seeds, `40005`–`40009` | Example: ```text $BUNDLE/seeds/k320_first10_seeds/put_cup_in_box/bench_demo_study_clean.txt # 40000 40001 ... 40009 ``` For exact reproduction, set `SEEDS_ROOT`, `USE_EVAL_SEEDS=1`, and the same `EVAL_TEST_NUM` used in the reported run. --- ## 7. Worker, lease queue, and resume A scalable run uses one shared run directory and many workers. ```text runs//queue.txt # one scene/task cell per line runs//queue.txt.leases/ # atomic lock dirs for claimed cells runs//eval_result/ # raw per-episode metrics ``` Worker loop: ```text self-stage code/env/checkpoint if needed start persistent model process claim queue cell by atomic mkdir lease run missing seeds only write metrics/videos claim next cell ``` Resume behavior: - `EVAL_RESUME_SEEDS=1` counts existing `_metrics.jsonl` entries and runs only missing seeds. - Rerunning the same run directory skips completed work. - To clear stale locks, use: ```bash rm -rf runs//queue.txt.leases ``` Do not use `rm -f`; `queue.txt.leases` is a directory. --- ## 8. Minimal GB10 run recipe Example: k320 clean, 5 seeds, node pool. ```bash source /shared_work/jack/eval_root/env.sh RUN=k320_clean mkdir -p runs/$RUN/eval_result cp $BUNDLE/seeds/k320_first10_queue.txt runs/$RUN/queue.txt rm -rf runs/$RUN/queue.txt.leases DXVLA_SAPIEN_B1=1 \ STUDY_RUN=$RUN \ CKPT_NAME=da3-xvla-k320-ckpt90k \ BENCH_CONFIG=clean \ SEEDS_ROOT=$BUNDLE/seeds/k320_first10_seeds \ EVAL_TEST_NUM=5 \ WORKERS_PER_GPU=1 \ srun --partition=gb10 \ --nodelist=trt-gb10-[3-15] \ --nodes=13 \ --ntasks-per-node=1 \ --gres=gpu:1 \ --cpus-per-task=18 \ -t 600 \ --kill-on-bad-exit=0 \ bash /shared_work/jack/eval_root/slurm_eval_node.sh python organize_results.py $RUN k320 ``` Resume by rerunning the same block. Increase `EVAL_TEST_NUM` to fill in more seeds; completed seeds are skipped. For giant/geostack models, additionally set model-specific vars such as: ```bash GIANT_CODE_DIR= XVLA_DA3_NATIVE_INPUT=1 # K=160 giant variants DXVLA_GEOSTACK_STEP=20000 # GeoStack variant, if applicable ``` Use each checkpoint's `EVAL_INTERFACE.md` when available. --- ## 9. Output and scoring Raw metrics stream to: ```text runs//eval_result//dxvla/bench_demo__/.../_metrics.jsonl ``` Each line includes fields like: ```json {"task": "...", "seed": 40000, "success": true, "hard_success": true} ``` Aggregate with: ```bash python organize_results.py ``` Organized outputs: ```text runs////seed.mp4 runs///results.txt runs//results.txt ``` `organize_results.py` deduplicates latest-per-seed before computing SR/HSR.