Alibaba-DAMO-Academy/RynnWorld-Teleop-Causal

RynnWorld-Teleop-Causal: Real-Time Streaming World Model for Digital Teleoperation

💫 Project Page   |    🤗 Hugging Face    |    🤖 ModelScope  |  🚀 Demo    |    📄 arXiv  

---

🌟 Abstract

We introduce RynnWorld-Teleop, a robot-centric generative world model that instantiates the paradigm of digital teleoperation—decoupling robot data collection from physical hardware constraints. By transforming an operator's real-time hand-pose stream into high-fidelity egocentric robotic videos from a single reference image, RynnWorld-Teleop enables the scaling of expert trajectories in a purely virtual environment. Our framework integrates depth-aware skeletal conditioning with a progressive human-to-robot training curriculum, allowing it to inherit rich manipulation priors from large-scale human datasets. To support interactive use, we distill the model into a causal, autoregressive student capable of real-time streaming. Policies trained exclusively on RynnWorld-Teleop synthetic data achieve effective zero-shot Sim2Real transfer, demonstrating its power as a high-fidelity data engine for scaling dexterous robotic learning.

---

📦 This Repository

This repository hosts the distilled causal (streaming) checkpoint of RynnWorld-Teleop. It is designed for frame-by-frame, real-time generation with a sliding KV cache.

Files

File	Description
ema_weights.bin	EMA weights of the causal transformer (distilled from the SFT teacher)
control_patch_embedding.bin	Conv3d patch embedding that injects hand-pose control video
control_scale.bin	Learnable scalar for control-signal fusion
control_running_stats.bin	Running stats for control normalization

Model Zoo

Model	HuggingFace	ModelScope
SFT (teacher)	Link	Link
Causal (streaming, this repo)	Link	Link

---

🚀 Quick Start

Please refer to the code repository for the full setup, training, and inference pipeline.

🔧 Environment Setup

conda create -n "rynnworld-teleop" python=3.10 -y
conda activate rynnworld-teleop
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

📖 Download the Checkpoint

Our model is developed on top of Wan2.2-TI2V-5B. Download the base model and place it under pretrained/:

RynnWorld-Teleop/
└── pretrained/
    ├── Wan2.2-TI2V-5B-Diffusers/
    │   ├── model_index.json
    │   ├── scheduler/
    │   ├── transformer/
    │   ├── vae/
    │   └── ...
    └── RynnWorld-Teleop-Causal/
        ├── ema_weights.bin
        ├── control_patch_embedding.bin
        ├── control_scale.bin
        └── control_running_stats.bin

Download the causal weights:

mkdir -p pretrained/RynnWorld-Teleop-Causal
huggingface-cli download Alibaba-DAMO-Academy/RynnWorld-Teleop-Causal --local-dir pretrained/RynnWorld-Teleop-Causal

---

🌊 Streaming Inference (Real-time)

For real-time streaming inference with the distilled causal model, use inference_streaming.py, which supports frame-by-frame generation with a KV cache.

Basic Usage

python inference_streaming.py \
  --image first_frame.png \
  --control_video control.mp4 \
  --checkpoint pretrained/RynnWorld-Teleop-Causal \
  --output results/streaming_demo

Advanced Options

FP8 Quantization (Hopper GPUs: H100/H800 only):

python inference_streaming.py \
  --image first_frame.png \
  --control_video control.mp4 \
  --checkpoint pretrained/RynnWorld-Teleop-Causal \
  --output results/streaming_fp8 \
  --fp8

• Automatically detects GPU compute capability
• Skips with warning on non-Hopper GPUs
• Requires torchao package

torch.compile (faster inference):

python inference_streaming.py \
  --image first_frame.png \
  --control_video control.mp4 \
  --checkpoint pretrained/RynnWorld-Teleop-Causal \
  --output results/streaming_compiled \
  --compile

• First sample includes compile overhead (~30-60s)
• Subsequent samples run 1.3–1.6× faster

Batch Inference from Dataset:

python inference_streaming.py \
  --data_json data/sample_data.json \
  --checkpoint pretrained/RynnWorld-Teleop-Causal \
  --output_dir results/batch_demo \
  --num_samples_per_dataset 3 \
  --fp8 --compile

Streaming Model Architecture

The streaming model (core/streaming/ in the code repo) implements:

• Causal Attention — frame-by-frame generation with a sliding KV cache
• Control Patch Embedding — skeleton / hand-pose conditioning
• Dynamic Cache — efficient memory management for long sequences

Key components:

• WanCausalTransformer3DModel — causal transformer with KV-cache support
• DynamicCache — sliding-window KV cache with sink-frame preservation
• WanStreamingPipeline — frame-by-frame inference pipeline

---

📑 Citation

If you find this project useful, please cite:

@article{rynnworld_teleop,
  title  = {RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation},
  author = {DAMO Academy, Alibaba Group},
  year   = {2026},
}

License

Apache License 2.0 — see LICENSE for details.