README.md

metadata

title: SyncWorld Simulator
emoji: 🦾
colorFrom: pink
colorTo: red
sdk: gradio
sdk_version: 6.17.3
app_file: app.py
short_description: Visually-calibrated zero-shot robot world simulator
python_version: '3.12'
startup_duration_timeout: 1h
models:
  - yyuncong/SyncWorld
datasets:
  - yyuncong/SyncWorld-Evaluation

🦾 SyncWorld — visual calibration turns a world model into a zero-shot simulator

Demo for SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators (UMass Embodied AGI) — code · weights.

A pretrained action-conditioned video world model cannot be dropped into a new scene, because it does not know how this camera sees this robot's action space. SyncWorld's fix is visual calibration: prepend a short clip of the arm sweeping each degree of freedom, paired with the actions that produced it, as extra fully-conditioning context. The model infers the unseen camera↔robot mapping in-context and becomes a usable forward-dynamics simulator — with no per-scene finetuning.

What this Space does

Pick one of the authors' calibrated evaluation episodes, then either

• replay the robot's own actions — re-simulate the expert trajectory and compare against ground truth, or
• drive the arm yourself — command a constant per-step end-effector twist (Δx/Δy/Δz in cm/step, Δroll/Δpitch/Δyaw in °/step, plus the gripper) and watch the model render a counterfactual rollout.

Each press generates one or more autoregressive 16-frame windows at 512 px, closed-loop (each window conditions on the previously generated frames).

Inference path

Ported 1:1 from the authors' reference script examples/eval_gripperhead_fdm_rollout.py, keeping the trained recipe exactly:

Conditioning items	[6 per-DoF calibration segments × 5 frames] + [25 sparse history frames @ stride 3] + [current frame + 16 future frames]
Actions	7-D [Δpos_cm(3), Δeuler_xyz_deg(3), gripper(1)], backward_framewise convention, translation scale 100.0, rotation scale 57.2958
Resolution / fps	512 px square, conditioning fps 15
Sampler	rectified flow, UniPC, 20 steps (adjustable)
Caption	the FDM training constant "A robot arm interacts with the scene."
Backbone	16B Cosmos-3 mixture-of-transformers (Qwen3-VL-8B understanding expert + diffusion expert), Wan2.2 4×16×16 video VAE

Deviations forced by the target hardware: single-process inference (parallel_dims=None), torch.compile disabled (ZeroGPU forks a fresh worker per call), the audio expert disabled (unused by this checkpoint), and the VLM processor served from a local copy of the seven tokenizer files in nvidia/Cosmos3-Nano instead of downloading that repo's 35 GB of weights.

Bundled scenes

Ten episodes (5 ManiSkill + 5 LIBERO) copied verbatim from the authors' own evaluation set, yyuncong/SyncWorld-Evaluation, each with its paired calibration sweep and pose track. Both the code and the data are released under OpenMDW-1.1; see ATTRIBUTIONS.md / NOTICE in the upstream repo.

Citation

@article{syncworld2026,
  title  = {SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators},
  author = {UMass Embodied AGI},
  year   = {2026},
  url    = {https://huggingface.co/papers/2609.09155}
}