# RoboPRO eval on GB10 — setup guides

How to evaluate VLA policies on the **RoboPRO / RoboTwin** benchmark on an NVIDIA **GB10 (Grace-Blackwell)** SLURM cluster — including running the **model and simulator on separate nodes** (for models too big to co-locate with the sim).

## Architecture
The eval splits into two processes talking over a TCP socket (**4-byte big-endian length header + JSON-with-numpy** payload):
- **Simulator** — SAPIEN + the RoboPRO task, runs **in-process** in the `robotwin` env; produces observations (3-cam RGB + proprio + instruction) and executes the returned actions.
- **Model** — the VLA policy in the `xvla` env; turns each observation into a 30-step action chunk.

The obs→action flow is fixed (sim has the obs, model returns the action); **either side can be the socket server** — choose by env flags:

| pattern | binds/listens | connects | guide |
|---|---|---|---|
| **model-as-server** (pipeline default) | model | sim | `ROBOPRO_EVAL_CROSSNODE_GB10.md` |
| **sim-as-server** | sim | model | `ROBOPRO_EVAL_SIM_AS_SERVER.md` |

Both work **single-node** (localhost) or **cross-node** (set the peer host/bind + a shared port). Use sim-as-server if that's your convention or if your orchestration starts the sim first.

## Documents
- **`EVALUATION_PIPELINE.md`** — **START HERE to recreate the pipeline.** The full end-to-end reference: directory layout, node-local staging, the per-node worker, the work-stealing lease queue (+ resume), every env var, copy-paste run recipes, output/scoring, the orchestrator patterns, and a recreate-elsewhere checklist.
- **`ROBOPRO_EVAL_CROSSNODE_GB10.md`** — model-as-server, split across nodes (the default direction). 2-node SLURM run steps; the `MODEL_SERVER_BIND` (server) / `MODEL_SERVER_HOST` (client) + `--port` knobs; how to plug in your model.
- **`ROBOPRO_EVAL_SIM_AS_SERVER.md`** — sim-as-server alternative (sim binds, model connects). The `SIM_AS_SERVER` / `MODEL_AS_CLIENT` flags + `SIM_SERVER_HOST`/`SIM_SERVER_PORT`. Both modes are additive/flag-gated and validated end-to-end (a full episode completes with success).
- **`GB10_EVAL_PIPELINE.md`** — **read this for the gotchas.** GB10-specific issues + the validated config: SAPIEN **3.0.0b1** (3.0.3 regresses office/kitchen), `WORKERS_PER_GPU=1` (MPS broken on Blackwell), **OptiX** denoiser (OIDN is a silent no-op), node-local staging, ffmpeg PATH, the exit-144 `pkill -f` footgun, etc.

## Quick start
0. **To recreate the whole pipeline, read `EVALUATION_PIPELINE.md`** — it has the layout, the run mechanism, and copy-paste commands. On this GB10 cluster the built setup is group-readable at `/shared_work/jack/eval_root` (run it in place or copy it).
1. Read `GB10_EVAL_PIPELINE.md` and set the validated knobs (`DXVLA_SAPIEN_B1=1`, `WORKERS_PER_GPU=1`, `DXVLA_DENOISER=optix`, node-local `/tmp` staging).
2. Pick a connection pattern (table above) and follow that guide's copy-paste SLURM commands.
3. Smoke-test one task (`EVAL_TEST_NUM=1`) and confirm a `_metrics.jsonl` line with `"success"` appears and both sides log "connected" — then scale up.

## Plugging in your own model
The model side loads the policy via `get_model()` (`policy/dxvla/deploy_policy.py`). Either swap a checkpoint (`CKPT_NAME` + code dir; see each ckpt's `EVAL_INTERFACE.md`), or implement `get_model()` returning an object with
`generate_actions(input_ids, image_input, image_mask, domain_id, proprio, ...) -> [B, num_actions, action_dim]`.
The socket/framing is handled for you; you only provide the policy.