# GB10 RoboPRO / DA3-XVLA Eval Pipeline — Issues & Fixes (canonical)

> Consolidated reference for running the RoboPRO / DA3-XVLA evaluation on the **GB10** cluster.
> Supersedes `GB10_ISSUES.txt`. Companion docs: `SAPIEN_B1_ATTEMPT.md` (full b1 build log),
> `DEBUG_OFFICE.md` (the office-SR root-cause hunt). Last updated 2026-06-25.

## Environment
- 15-node **NVIDIA Grace-Blackwell (GB10)**, **1 GPU/node** (Blackwell **sm_120/121**, ~119 GB unified LPDDR5X).
- **aarch64**, **CUDA 13**, **glibc 2.39**. Node 1 = NFS/login head; nodes 1–2 reserved (Mark).
- `/shared_work` = plain **ext4 on one 3.7 TB NVMe on the head**, NFS-exported (sync) — NOT pooled/BeeGFS.
  Each compute node has its own ~3.6 TB local disk → stage to `/tmp/dxvla_local`.

## TL;DR — config for a correct, comparable eval
| knob | value | why |
|---|---|---|
| `DXVLA_SAPIEN_B1=1` | SAPIEN **3.0.0b1** | 3.0.3 is an office/kitchen regression (§2) |
| `WORKERS_PER_GPU=1` | one CUDA context/GPU | MPS broken; sharing corrupts the sim (§1) |
| `DXVLA_DENOISER=optix` | OptiX denoiser | OIDN is a silent no-op on Blackwell (§3) |
| node-local staging | `/tmp/dxvla_local` | NFS mmap fan-out stalls (§4) |
| `CACHE_LIMIT_GB=999999` | LRU off | LRU was evicting staged giant weights (§5) |
| reserve nodes | exclude from POOL | nodes 1,2 reserved |

---

## 1. GPU concurrency — the big correctness trap
**1a. CUDA MPS is broken on this Blackwell GPU.** Enabling it → CUDA **Error 807** (MPS server-not-ready) then **700** (illegal memory access); corrupted an overnight run. → **MPS disabled** (`ENABLE_MPS=0`); workers go straight to the GPU.

**1b. `WORKERS_PER_GPU>1` corrupts the simulation (no MPS to fall back on).** With 4 sim+render+curobo workers sharing one GPU, episodes go "shakey" — objects jitter/move *before* contact, noisy arm/proprio, precise grasps/inserts fail. **Same seed succeeds at WPG=1, fails at WPG=4 (deterministic flip, not noise).** Concurrent CUDA contexts time-slicing one GPU trigger sporadic memory/compute corruption; the closed perception→action loop amplifies a corrupted RT frame into bad actions. Damage concentrates on precise-insertion tasks (drawers/cabinets); robust pick-place absorbs it. **Impact:** degraded a whole run (office 5%→35% instead of ~58%). → **Run `WORKERS_PER_GPU=1`.** Throughput doesn't scale linearly with WPG anyway (eval is ~half CPU-bound on physics/curobo).

## 2. SAPIEN on aarch64
**2a. No aarch64 SAPIEN wheels on PyPI.** `pip install sapien` pulls unrunnable x86_64. aarch64 wheels exist only as haosulab GitHub-release assets, only for 3.0.3 + a nightly. → installed 3.0.3 cp310 aarch64 wheel (`/shared_work/jack/wheels/`).

**2b. SAPIEN 3.0.3 is a REGRESSION vs the reference's 3.0.0b1 (office/kitchen broke).** Office SR 5% / kitchens 11% vs reference ~40–90% / ~58%, *same checkpoint* — model never descends, closes gripper early. The reference numbers were made on **3.0.0b1**; the 3.0.0b1→3.0.3 delta (79 commits: PhysX defaults + camera/render + 85 Vulkan shaders) makes office/kitchen scenes OOD. `study` (flat) tolerated it; deep shelves/drawers + specular did not. → **Built 3.0.0b1 from source for aarch64 (§3)**, deployed via `DXVLA_SAPIEN_B1=1`. A/B: milktea 0/55 (3.0.3) → 5/6 (b1). **Lesson: pin the simulator version when comparing to reference numbers.**

**2c. librt / GLIBC_PRIVATE** (3.0.3 wheel): vendored librt wants a GLIBC_PRIVATE symbol absent on glibc 2.39. → symlink bundled librt → system `/usr/lib/aarch64-linux-gnu/librt.so.1`.

## 3. Building SAPIEN 3.0.0b1 from source (aarch64) — blockers
- **3a. CUDA 13 dropped old archs** (`Unsupported gpu architecture 'compute_60'`). → set `CUDA_ARCHITECTURES=75;80;86;90;120;121` in CMakeLists, `3rd_party/simsense/CMakeLists.txt`, and the OIDN arch list.
- **3b. PhysX5 prebuilt is x86_64-only.** → repoint `cmake/physx5.cmake` at the aarch64 PhysX asset (105.1-physx-5.3.1) + fix the MD5.
- **3c. KTX astc-encoder NEON fails on gcc13/aarch64.** → `-DISA_NONE=ON` (scalar astcenc).
- **3d. GL/EGL/GLU/DRM dev headers missing, no sudo.** → `apt-get download` + `dpkg-deb -x` the arm64 `*-dev` debs into a local sysroot; point `C(PLUS)_INCLUDE_PATH`/`LIBRARY_PATH` at it.
- **3e. pybind11 smart_holder branch deleted upstream.** → pin to archive/smart_holder commit `68a11bb`.
- **3f. No auditwheel.** → manually bundle `libsapien/libsimsense/libsvulkan2.so` into the wheel, set `$ORIGIN` RUNPATHs via patchelf, add OIDN soname symlinks.
- **Output:** `/shared_work/jack/sapien_b1_build/dist_bundled/sapien-3.0.0b1-cp310-...whl`; env at `envs/robotwin_b1`. Deployed by the `DXVLA_SAPIEN_B1=1` slurm hook (rsyncs the b1 pkg into node-local robotwin; the env has no pip so it's a file copy, not pip install).

## 4. Rendering (ray tracing / OptiX / OIDN)
- **4a. OIDN (Intel Open Image Denoise) CUDA backend is a SILENT NO-OP on Blackwell.** `DXVLA_DENOISER=oidn` → frames identical to no-denoise, no error → noisier RT than reference if relied on. → **Use `DXVLA_DENOISER=optix`** (slurm default).
- **4b. RT render is GPU-heavy** and is the main victim of WPG>1 contention (§1b): under sharing it degrades/corrupts → noisy observations → failed precise tasks.
- **4c. Vulkan ICD / RT setup:** `VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json`, `unset DISPLAY`, `LD_LIBRARY_PATH` includes `/shared_work/jack/vklib`. `DXVLA_RT_SPP=32` (samples/pixel) matches the reference render quality.

## 5. Node-local staging & giant-checkpoint staging
**5a. NFS mmap fan-out stalls.** Loading envs/checkpoints from NFS via mmap across many nodes at once is catastrophically slow. → stage envs+code+ckpt to node-local `/tmp/dxvla_local` (self-staging in `slurm_eval_node.sh`); only bulk one-time rsyncs touch NFS.

**5b. (NEW 2026-06-25) The node-local 50 GB LRU evicted staged giant WEIGHTS.** Running 5 different giants, `stage_giants_node.sh`'s `CACHE_LIMIT_GB=50` LRU evicted a giant's checkpoint *after* staging → at load: `OSError: no file named pytorch_model.bin/model.safetensors found in .../checkpoints/<ckpt>`. → **Disabled the LRU: `CACHE_LIMIT_GB=999999`** in `stage_giants_node.sh` and `slurm_eval_node.sh` (node disk is 3.6 TB; the 50 GB cap was needless).

**5c. (NEW) Giant model code not importable node-local → `ModuleNotFoundError: No module named 'models'`.** The giant's bundled code (`models/modeling_xvla.py`) wasn't reliably present node-local (staging gap / eviction). → **Point `GIANT_CODE_DIR` (→ `DA3_XVLA_ROOT`) at the NFS path** `ablation_ckpts/<ckpt>/{code|overlay}` (dptfused→`code`, spatialexpert→`overlay`). The code is small `.py` — imports fine from NFS; only the *weights* need node-local. Both subdirs are self-sufficient (have `models/` + `datasets/`).

**5d. (NEW) Self-stage skipped refilling evicted/partial weights.** The giant self-stage gate checked only *directory presence* (`[ ! -d $L/checkpoints/$CKPT ]`), so a dir with evicted weights was treated as "staged." → made the giant ckpt self-stage **always run an idempotent rsync** (runs whenever `ablation_ckpts/<ckpt>` exists; rsync `--update` refills missing shards). Giant weights are **sharded** `pytorch_model-0000N-of-0000M.bin` + `index.json`.

**5e. Giant launch env** (per model): `GIANT_CODE_DIR=<nfs code dir>`, `XVLA_DA3_NATIVE_INPUT=1`, `XVLA_DA3_INPUT_HW=252x336` (native aspect), `DXVLA_SKIP_RELOAD=1` (DA3 weights baked into the ckpt — no `reload_pretrained_weights`), `XVLA_SKIP_GEOMETRY_REINIT=1`, `EVAL_FAST_CTRL=1` (else rollouts crawl ~3.6 s/step).

## 6. Motion planning (curobo)
On cluttered/complex scenes `plan_path` is slow and can stall a worker; some scene seeds are physically unstable. → `STALL_TIMEOUT=900` + `CUROBO_IK_FAIL_RETURN=1`; eval seed-advances past unstable seeds (frozen pre-validated seed bank). curobo built node-local against robotwin torch (2.9.1+cu130).

## 7. Node hardware / ops
- **7a.** A node went unresponsive (ssh froze, SLURM showed down) → physical/power-cycle restart (a `sudo reboot` is insufficient when ssh hangs).
- **7b.** Cold/new nodes self-stage base env+code+ckpt from NFS on first job (slurm_eval_node.sh).

## 8. Eval-ops gotchas (not GB10 hardware, but bit us)
- **8a. `queue.txt.leases` is a DIRECTORY** (per-cell lock files). `rm -f queue.txt.leases*` does NOT clear it → stale locks make a re-run SKIP "done" tasks (0 new episodes). → **`rm -rf queue.txt.leases`** before re-running in an existing run dir.
- **8b. Self-kill → exit 144.** `pkill -f <pat>` / `pgrep -f <pat> | kill` where `<pat>` is in the command's OWN command line kills your shell. → **kill drivers by explicit PID; cancel SLURM by `--name`/`--jobid`; never `-f` patterns matching your own command.** (Hit this repeatedly.)
- **8c. slurm exports override passed env** unless made `${VAR:-default}` → per-knob A/B tests were silent no-ops. → make all tunables `${VAR:-default}`.
- **8d. A running orchestrator bakes its node POOL at start.** Editing the script doesn't change the live driver. → stop by PID + relaunch with `POOL` override.
- **8e. Fast cancel→relaunch loses sruns.** Relaunching immediately after `scancel` fails most sruns (nodes still COMPLETING). → poll until jobs=0 AND nodes idle before relaunching.

## 9. Pipeline feature knobs (added during this work)
- `DXVLA_EXEC_ACTIONS=20` — receding-horizon: generate a 30-action chunk, execute the first 20, re-query (0 = full chunk).
- **cross-chunk EMA smoothing** — seeds each chunk's first action from the previous chunk's last *executed* action (fixes the visible chunk-boundary jerk). `DXVLA_ACTION_SMOOTH=0.7`, `DXVLA_SMOOTH_ROT=1`, with `DXVLA_FAKE_PROPRIO=1`.
- `XVLA_VIDEO_FPS=18` — recorded-video playback FPS (default 12).
- `XVLA_POSED_DA3=1` — feed camera extrinsics/intrinsics into the DA3 path (posed depth). Verified the pose comes live from the SAPIEN camera (`extrinsic_cv`), not identity.

## Key takeaways for a correct, comparable GB10 eval
1. **SAPIEN 3.0.0b1** (`DXVLA_SAPIEN_B1=1`), not 3.0.3.
2. **WORKERS_PER_GPU=1** (no MPS; sharing corrupts).
3. **OptiX denoiser** (OIDN is a no-op here).
4. **Node-local staging**, LRU **off** for multi-giant runs; giant **code from NFS**, weights node-local (idempotent rsync).
5. Reserve nodes by **excluding from the POOL** (1 & 2 reserved).
6. Kill by **PID**, cancel by **`--name`** — never `-f` patterns.