ROBOPRO_EVAL_RECREATE_RESULTS_CONCISE_MERGED.md

# RoboPRO Evaluation Pipeline — Concise Recreate Guide

Purpose: give another team the minimum information needed to rebuild the RoboPRO evaluation pipeline, run the simulator, connect a model, and reproduce the reported SR/HSR results. Socket direction is only a launch detail; reproducibility mostly depends on simulator version, seed banks, task configs, model wrapper, eval knobs, and result aggregation.

---

## 1. Reproducibility contract

An eval cell is:

```text
(scene, task, bench_config, checkpoint, seed_bank, num_seeds)
```

For each seed, RoboPRO runs one SAPIEN/RoboTwin episode, sends observations to the model, executes the returned action chunk, and writes metrics/videos.

To reproduce results, match:

- RoboPRO/customized_robotwin code and assets
- simulator version and render settings
- checkpoint, model code, and model wrapper
- seed bank, queue file, and `EVAL_TEST_NUM`
- camera selection, preprocessing, proprio mode, and action execution horizon
- same `organize_results.py` scoring path

SR = success rate. HSR = hard success rate: success with no collision.

---

## 2. Existing GB10 layout

On the GB10 cluster, the runnable installation is group-readable:

```text
/shared_work/jack/eval_root
├── env.sh
├── code/RoboPRO/                         # benchmark + customized_robotwin
│   └── customized_robotwin/script/        # eval_policy_client.py, policy_model_server.py
├── code/DA3-XVLA/                        # model code
├── envs/{robotwin,xvla,robotwin_b1}       # sim env, model env, SAPIEN b1 swap
├── checkpoints/, ablation_ckpts/          # model weights
├── slurm_eval_node.sh                     # per-node worker entrypoint
├── stage_k320_node.sh, stage_giants_node.sh
├── organize_results.py
├── orchestrate_*.sh
└── runs/<RUN>/                            # queue, leases, logs, raw metrics

/shared_work/jack/robopro-eval-pipeline    # $BUNDLE
├── harness/eval_worker_steal_dualgpu.sh
└── seeds/                                 # fixed seed banks + queue files
```

To rebuild elsewhere, preserve this structure or update paths consistently in `env.sh`, stagers, workers, and orchestrators.

---

## 3. Runtime architecture

The simulator and model are separate processes connected by one TCP socket. The simulator owns physics, task reset, success checks, videos, and metrics. The model owns checkpoint loading and inference. Observations/actions are sent as length-prefixed JSON-with-numpy frames.

Two launch directions are supported:

```text
MODEL server + SIM client   # default in current worker
SIM server + MODEL client   # alternate direction
```

Treat this as plumbing. The reproducible eval pipeline is the same either way: sim sends observations, model returns action chunks, sim executes and scores.

---

## 4. Model wrapper contract

The eval is model-agnostic as long as the model is exposed through a policy wrapper.

Default path:

```text
policy/dxvla/deploy_policy.py:get_model(usr_args)
policy/dxvla/dxvla_model.py:DXVLA wrapper
policy/dxvla/deploy_policy.yml
```

To swap in another model, either reuse the `dxvla` wrapper with a new `CKPT_NAME` / `GIANT_CODE_DIR`, or create `policy/<name>/deploy_policy.py` and pass `--policy_name <name>`.

The wrapper should provide either the high-level policy methods:

```python
def reset_model(): ...
def set_language(instruction): ...
def update_observation_window(rgb_list, proprio, extrinsics, intrinsics): ...
def get_action(): ...  # [num_actions, action_dim]
```

or, for the existing DXVLA wrapper style:

```python
def generate_actions(input_ids, image_input, image_mask, domain_id, proprio, ...) -> [B, num_actions, action_dim]
```

Inputs are already prepared by the eval stack. Typical `image_input` is `[B, V, 3, H, W]`, language is tokenized in `input_ids`, and proprio is provided according to the eval setting. Output is a chunk of end-effector actions, typically 30 steps, with 10-D per arm/action block: xyz + rot6d + gripper.

---

## 5. Exact eval knobs used for comparable results

Use these for the k320 clean reference configuration. Reported reference: **58% SR / 49% HSR**.

```bash
DXVLA_SAPIEN_B1=1             # SAPIEN 3.0.0b1, not 3.0.3
WORKERS_PER_GPU=1             # do not raise on GB10; sharing corrupts sim
DXVLA_DENOISER=optix          # OIDN is a silent no-op on GB10 Blackwell
DXVLA_RT_SPP=32               # ray-trace samples/pixel
XVLA_POSED_DA3=1              # pass camera extrinsics/intrinsics to DA3 path
XVLA_RGB_INPUT=1
XVLA_SKIP_GEOMETRY_REINIT=1   # keep trained geometry weights
DXVLA_FAKE_PROPRIO=1          # proprio = last commanded action
DXVLA_ACTION_SMOOTH=0.7       # cross-chunk EMA smoothing
DXVLA_SMOOTH_ROT=1            # smooth xyz + rot6d, not just xyz
DXVLA_EXEC_ACTIONS=0          # k320: execute full 30-action chunk
DXVLA_PRIMARY_CAM=countertop
DXVLA_VIDEO_CAM=countertop_camera
USE_EVAL_SEEDS=1
EVAL_RESUME_SEEDS=1
EVAL_TEST_NUM=<N>
```

Notes:

- k320 used `DXVLA_EXEC_ACTIONS=0`.
- Giant ablations used `DXVLA_EXEC_ACTIONS=20` to generate 30 actions, execute 20, then re-query.
- Keep `DXVLA_EXEC_ACTIONS` consistent within comparisons.
- All final numbers used `DXVLA_ACTION_SMOOTH=0.7` and `DXVLA_SMOOTH_ROT=1`.

---

## 6. Seed banks and queues

Use the same seeds to reproduce the same episodes. Seed banks live under:

```text
$BUNDLE/seeds/
# where $BUNDLE=/shared_work/jack/robopro-eval-pipeline
```

The eval reads:

```text
<SEEDS_ROOT>/<task>/<task_config>.txt
```

| eval | `SEEDS_ROOT` | queue file | seeds/task |
|---|---|---|---|
| clean | `k320_first10_seeds` | `k320_first10_queue.txt` | 10 seeds, `40000`–`40009` |
| clutter `d6`–`d15` | `clutter_seeds_hf` | `clutter_queue_hf_d<N>.txt` | 2 per task/level |
| clean fill-in seeds 5–9 | `k320_seeds5to9` | `k320_first10_queue.txt` | 5 seeds, `40005`–`40009` |

Example:

```text
$BUNDLE/seeds/k320_first10_seeds/put_cup_in_box/bench_demo_study_clean.txt
# 40000 40001 ... 40009
```

For exact reproduction, set `SEEDS_ROOT`, `USE_EVAL_SEEDS=1`, and the same `EVAL_TEST_NUM` used in the reported run.

---

## 7. Worker, lease queue, and resume

A scalable run uses one shared run directory and many workers.

```text
runs/<RUN>/queue.txt          # one scene/task cell per line
runs/<RUN>/queue.txt.leases/  # atomic lock dirs for claimed cells
runs/<RUN>/eval_result/       # raw per-episode metrics
```

Worker loop:

```text
self-stage code/env/checkpoint if needed
start persistent model process
claim queue cell by atomic mkdir lease
run missing seeds only
write metrics/videos
claim next cell
```

Resume behavior:

- `EVAL_RESUME_SEEDS=1` counts existing `_metrics.jsonl` entries and runs only missing seeds.
- Rerunning the same run directory skips completed work.
- To clear stale locks, use:

```bash
rm -rf runs/<RUN>/queue.txt.leases
```

Do not use `rm -f`; `queue.txt.leases` is a directory.

---

## 8. Minimal GB10 run recipe

Example: k320 clean, 5 seeds, node pool.

```bash
source /shared_work/jack/eval_root/env.sh

RUN=k320_clean
mkdir -p runs/$RUN/eval_result
cp $BUNDLE/seeds/k320_first10_queue.txt runs/$RUN/queue.txt
rm -rf runs/$RUN/queue.txt.leases

DXVLA_SAPIEN_B1=1 \
STUDY_RUN=$RUN \
CKPT_NAME=da3-xvla-k320-ckpt90k \
BENCH_CONFIG=clean \
SEEDS_ROOT=$BUNDLE/seeds/k320_first10_seeds \
EVAL_TEST_NUM=5 \
WORKERS_PER_GPU=1 \
srun --partition=gb10 \
     --nodelist=trt-gb10-[3-15] \
     --nodes=13 \
     --ntasks-per-node=1 \
     --gres=gpu:1 \
     --cpus-per-task=18 \
     -t 600 \
     --kill-on-bad-exit=0 \
     bash /shared_work/jack/eval_root/slurm_eval_node.sh

python organize_results.py $RUN k320
```

Resume by rerunning the same block. Increase `EVAL_TEST_NUM` to fill in more seeds; completed seeds are skipped.

For giant/geostack models, additionally set model-specific vars such as:

```bash
GIANT_CODE_DIR=<model_code_dir>
XVLA_DA3_NATIVE_INPUT=1       # K=160 giant variants
DXVLA_GEOSTACK_STEP=20000     # GeoStack variant, if applicable
```

Use each checkpoint's `EVAL_INTERFACE.md` when available.

---

## 9. Output and scoring

Raw metrics stream to:

```text
runs/<RUN>/eval_result/<task>/dxvla/bench_demo_<scene>_<cfg>/.../_metrics.jsonl
```

Each line includes fields like:

```json
{"task": "...", "seed": 40000, "success": true, "hard_success": true}
```

Aggregate with:

```bash
python organize_results.py <RUN> <model_name>
```

Organized outputs:

```text
runs/<model>/<scene>/<task>/seed<N>.mp4
runs/<model>/<scene>/results.txt
runs/<model>/results.txt
```

`organize_results.py` deduplicates latest-per-seed before computing SR/HSR.