# RoboPRO eval — "SIM-as-server, MODEL-connects" variant (GB10) A drop-in **reverse-direction** variant of the RoboPRO client/server eval. The message flow and wire protocol are **identical** to the default; only *who opens the listening socket* is flipped. Default behavior is untouched — both new modes are env-gated and OFF unless you set their flag. ## The two architectures ### Default: MODEL-as-server / SIM-as-client (production, unchanged) ``` MODEL (xvla env) SIM (robotwin env) policy_model_server.py eval_policy_client.py bind/listen/accept <----- connects ---- ModelClient.connect() per-step: recv obs -> predict -> send action ``` - Model loads the policy once, `bind/listen/accept`s, serves action chunks. - Sim runs SAPIEN + the RoboPRO task **in-process** and dials into the model. - Cross-node knobs: `MODEL_SERVER_BIND=0.0.0.0` (model), `MODEL_SERVER_HOST=` (sim). - This is what every `orchestrate_*.sh` / `slurm_eval_node.sh` run uses. See `ROBOPRO_EVAL_CROSSNODE_GB10.md`. ### New: SIM-as-server / MODEL-as-client (this doc) ``` SIM (robotwin env) MODEL (xvla env) eval_policy_client.py policy_model_server.py bind/listen/accept <----- connects ---- connect_and_serve() per-step: SIM sends obs -> MODEL predicts -> MODEL sends action back ``` - The **SIM** binds/listens/accepts and **waits for the model to dial in**. - The **MODEL** loads the policy, then `connect`s to the sim and runs the *same* recv-obs → predict → send-action loop on that connected socket. - The obs→action message flow and framing are **exactly the same** — only the socket-open direction is reversed. Use this when your orchestration prefers the simulator to be the long-lived listening endpoint (e.g. a sim "service" that successive model processes connect into). ## What changed in the code (additive, default-OFF) Two env-gated branches, no change to the default path: | File | Flag | Behavior when set | |---|---|---| | `script/eval_policy_client.py` (`ModelClient._connect`) | `SIM_AS_SERVER=1` | Calls new `_listen_and_accept()`: `bind/listen` on `(SIM_SERVER_BIND\|0.0.0.0, SIM_SERVER_PORT\|--port)`, blocking-`accept` for the model, then uses the accepted socket as `self.sock`. Send-obs/recv-action are unchanged. | | `script/policy_model_server.py` (`main` + new `ModelServer.connect_and_serve`) | `MODEL_AS_CLIENT=1` | Instead of `bind/listen/accept`, `connect`s (with retry) to `(SIM_SERVER_HOST\|localhost, SIM_SERVER_PORT\|--port)`, then runs the **same** `_handle_client` recv→predict→send loop on the connected socket; exits when the sim disconnects. | With neither flag set, both scripts behave exactly as before (model binds, sim connects). ## Env vars / flags / ports | side | env var | meaning | default | |---|---|---|---| | SIM (server) | `SIM_AS_SERVER` | `1` flips the sim to listen/accept | `0` (connect, default mode) | | SIM (server) | `SIM_SERVER_BIND` | interface to bind | `0.0.0.0` | | SIM (server) | `SIM_SERVER_PORT` | port to listen on (falls back to `--port`) | `--port` | | SIM (server) | `SIM_SERVER_RECV_TIMEOUT` | per-call recv timeout (s) on the accepted socket; `0`/empty = block forever | `600` | | MODEL (client) | `MODEL_AS_CLIENT` | `1` flips the model to connect into the sim | `0` (listen, default mode) | | MODEL (client) | `SIM_SERVER_HOST` | sim node hostname to dial | `localhost` | | MODEL (client) | `SIM_SERVER_PORT` | sim port to dial (falls back to `--port`) | `--port` | | MODEL (client) | `MODEL_CLIENT_MAX_ATTEMPTS` | connect retries (5 s apart) before giving up | `1000` | Pick a high port in `29500–29999`. The accept is blocking, so start order doesn't matter: whichever side comes up first waits (sim blocks on `accept`; model retries `connect` every 5 s until the sim's listener is up). All the validated GB10 knobs are unchanged and still apply to **both** processes: `DXVLA_SAPIEN_B1=1`, `WORKERS_PER_GPU=1`, `DXVLA_DENOISER=optix`, node-local staging (`/tmp/dxvla_local`), `USE_EVAL_SEEDS=1` + `BENCH_EVAL_SEEDS_ROOT`, and the model inference flags (`XVLA_POSED_DA3`, `DXVLA_ACTION_SMOOTH`, `DXVLA_FAKE_PROPRIO`, …). ## Wire protocol (identical to the default path) Length-prefixed JSON-with-numpy frames over one TCP socket. The **SIM is always the requester**, the **MODEL is always the responder** — in both architectures. Only the side that *opens* the socket changes. Per control step the sim sends, and the model replies, one frame each: ``` frame = 4-byte big-endian length header + UTF-8 JSON body ``` - **SIM → MODEL** body: `{"cmd": "", "obs": }` - `cmd` is a method name on the policy object (`set_language`, `update_observation_window`, `get_action`, `reset_model`, …). - `obs` is the single arg, or `{"_args":[a,b,...]}` for multi-arg calls, or absent. - **MODEL → SIM** body: `{"res": }`, or `{"error": "...", "traceback": "..."}` on failure. numpy arrays are encoded as `{"__numpy_array__":true,"data":,"dtype":...,"shape":...}` by `numpy_to_json` and rebuilt by `json_to_numpy` (both helpers are duplicated in each script). A typical episode is: `reset_model` → (`set_language` once) → repeated (`update_observation_window`(rgb, proprio, extrinsics, intrinsics) → `get_action` → [num_actions, 20] chunk). ## Copy-paste run commands ### A) Localhost test (both halves, one node) — what we validated Driver script: `test_sim_as_server.sh` (self-stages a cold node, launches sim server then model client, waits, prints the metrics line). Run on a free node: ```bash srun --nodelist=trt-gb10-3 --nodes=1 --gres=gpu:1 --cpus-per-task=20 --overlap -t 30 \ bash /shared_work/jack/eval_root/test_sim_as_server.sh # env knobs it honors: TASK, SCENE, BENCH_CONFIG, PORT, CKPT_NAME, EVAL_TEST_NUM ``` The two processes it launches (abridged, both `cd customized_robotwin`): ```bash # 1) SIM server (robotwin env) — binds, waits for the model SIM_AS_SERVER=1 SIM_SERVER_BIND=0.0.0.0 SIM_SERVER_PORT=29761 \ DXVLA_SAPIEN_B1=1 USE_EVAL_SEEDS=1 EVAL_TEST_NUM=1 \ BENCH_EVAL_SEEDS_ROOT=/shared_work/jack/robopro-eval-pipeline/seeds/k320_first10_seeds \ /tmp/dxvla_local/envs/robotwin/bin/python script/eval_policy_client.py \ --port 29761 --config policy/dxvla/deploy_policy.yml --overrides \ --task_name put_cup_in_box --task_config bench_demo_study_clean \ --ckpt_setting simserver_study_clean --seed 0 --policy_name dxvla & # 2) MODEL client (xvla env) — dials into the sim MODEL_AS_CLIENT=1 SIM_SERVER_HOST=localhost SIM_SERVER_PORT=29761 \ /tmp/dxvla_local/envs/xvla/bin/python script/policy_model_server.py \ --port 29761 --config policy/dxvla/deploy_policy.yml --overrides \ --task_name put_cup_in_box --task_config bench_demo_study_clean \ --ckpt_setting simserver_study_clean --seed 0 --policy_name dxvla \ --checkpoint_path /tmp/dxvla_local/checkpoints/da3-xvla-k320-ckpt90k & ``` Expect on the SIM side: `🚀 SIM server listening on 0.0.0.0:29761` then `🔗 Model connected from (...)`; on the MODEL side: `🔗 Model connected to SIM server at localhost:29761`. ### B) Cross-node (SIM on node A binds, MODEL on node B connects) Allocate **node A = sim**, **node B = model**, port e.g. `29761`. Both nodes must be staged to `/tmp/dxvla_local` (use `stage_k320_node.sh`; cold nodes self-stage). **1. SIM node (A)** — binds `0.0.0.0`, waits for the model: ```bash srun --nodelist=trt-gb10-A --nodes=1 --gres=gpu:1 --cpus-per-task=18 -t 600 bash -c ' L=/tmp/dxvla_local; cd $L/code/RoboPRO/customized_robotwin; source set_env.sh export ROBOTWIN_BENCH_TASK=bench BENCH_ROOT=$L/code/RoboPRO/benchmark ROBOTWIN_ROOT=$PWD unset DISPLAY; export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json export LD_LIBRARY_PATH=/shared_work/jack/vklib:$LD_LIBRARY_PATH export PATH=$L/envs/robotwin/bin:$PATH # node-local ffmpeg for video SIM_AS_SERVER=1 SIM_SERVER_BIND=0.0.0.0 SIM_SERVER_PORT=29761 \ DXVLA_SAPIEN_B1=1 DXVLA_DENOISER=optix WORKERS_PER_GPU=1 \ USE_EVAL_SEEDS=1 EVAL_EXPERT_CHECK=0 BENCH_CONFIG=clean EVAL_TEST_NUM=5 \ BENCH_EVAL_SEEDS_ROOT=/shared_work/jack/robopro-eval-pipeline/seeds/k320_first10_seeds \ DXVLA_PRIMARY_CAM=countertop DXVLA_VIDEO_CAM=countertop_camera \ CUDA_VISIBLE_DEVICES=0 \ $L/envs/robotwin/bin/python script/eval_policy_client.py \ --port 29761 --config policy/dxvla/deploy_policy.yml --overrides \ --task_name put_cup_in_box --task_config bench_demo_study_clean \ --ckpt_setting simserver_study_clean --seed 0 --policy_name dxvla ' # wait for: "🚀 SIM server listening on 0.0.0.0:29761 — waiting for model to connect..." ``` **2. MODEL node (B)** — connects to node A: ```bash srun --nodelist=trt-gb10-B --nodes=1 --gres=gpu:1 --cpus-per-task=20 -t 600 bash -c ' L=/tmp/dxvla_local; cd $L/code/RoboPRO/customized_robotwin; source set_env.sh export PYTHONWARNINGS=ignore::UserWarning PYTHONPATH=$L/code/DA3-XVLA:$PYTHONPATH export HF_HOME=$L/.cache/huggingface CUDA_VISIBLE_DEVICES=0 export XVLA_POSED_DA3=1 XVLA_RGB_INPUT=1 DXVLA_ACTION_SMOOTH=0.7 \ DXVLA_FAKE_PROPRIO=1 DXVLA_PRIMARY_CAM=countertop MODEL_AS_CLIENT=1 SIM_SERVER_HOST=trt-gb10-A SIM_SERVER_PORT=29761 \ $L/envs/xvla/bin/python script/policy_model_server.py \ --port 29761 --config policy/dxvla/deploy_policy.yml --overrides \ --task_name put_cup_in_box --task_config bench_demo_study_clean \ --ckpt_setting simserver_study_clean --seed 0 --policy_name dxvla \ --checkpoint_path $L/checkpoints/da3-xvla-k320-ckpt90k ' # expect: "🔗 Model connected to SIM server at trt-gb10-A:29761" ``` The sim runs the rollout locally on node A; results land in `runs//eval_result//dxvla////_metrics.jsonl`. > Compute nodes share the internal network and can reach each other on high ports. > Use the node **hostname** (`trt-gb10-N`) for `SIM_SERVER_HOST`. Reserve nodes 1, 2. ## Plugging in a different model Same contract as the default server — only the socket direction differs. The model process loads the policy via `get_model()` in `policy/dxvla/deploy_policy.py`. Either: - swap the checkpoint via `--checkpoint_path` / `DXVLA_CKPT` (+ `GIANT_CODE_DIR`/`DA3_XVLA_ROOT` for X-VLA-family ablation ckpts), **or** - implement your own `policy//deploy_policy.py:get_model()` returning an object whose methods match the `cmd`s the sim sends: `reset_model`, `set_language(instr)`, `update_observation_window(rgb_list, proprio, extrinsics, intrinsics)`, `get_action() -> [num_actions, 20]`, and a readable `observation_window` attribute (the proxy mirrors it client-side). Then pass `--policy_name `. The socket/framing layer is provided; you only supply the policy. The per-step obs is 3-view RGB (countertop = primary), proprio, and the language instruction; the sim expects an action chunk back. ## GB10 gotchas (don't skip) - `DXVLA_SAPIEN_B1=1` on the **sim** side; OptiX denoiser (OIDN is a silent no-op on Blackwell). - `WORKERS_PER_GPU=1`; do **not** enable MPS. - Put `$L/envs/robotwin/bin` on `PATH` on the sim side so the node-local **ffmpeg** (video recording) resolves — otherwise the episode crashes at the first recorded clip with `FileNotFoundError: 'ffmpeg'`. - Stage to node-local `/tmp/dxvla_local`; never run from NFS across a fan-out. - Kill background drivers by **PID**; cancel SLURM by jobid/`--name`. Reserve nodes 1, 2 (Mark).