---
license: mit
pipeline_tag: image-to-video
library_name: diffusers
tags: [kandinsky, image-to-video, physical-ai, world-model]
---

<div align="center">

<img src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/logo.png" width="760" alt="Kandinsky WM"/>

# Kandinsky WM 1.0: A family of models for Physical AI

**Image-to-Video generation for Physical AI: autonomous driving · robotics · general physics**

[🤗 Checkpoints collection](https://huggingface.co/collections/kandinskylab/kandinsky-wm-10-6a67455245052645fcfbd5f3) ·
[🧩 Base model (Kandinsky 5.0)](https://github.com/kandinskylab/kandinsky-5) ·
[💻 GitHub](https://github.com/kandinskylab/kandinsky-wm) ·
[📝 Habr article](https://habr.com/ru/companies/sberbank/articles/1054966/)

</div>

> **This repo holds the 🤖 Robotics checkpoint** of the Kandinsky WM 1.0 family — DiT `robot_rl`, published here in **two layouts**: a diffusers pipeline (`model_index.json`, `transformer/`, `vae/`, `text_encoder*/`) and the GitHub-code DiT at `model/robot_rl.safetensors` (same weights). The other two variants are in the [Kandinsky WM 1.0 collection](https://huggingface.co/collections/kandinskylab/kandinsky-wm-10-6a67455245052645fcfbd5f3).

---

## Contents

- [Overview](#overview)
- [Model Zoo](#model-zoo)
- [Examples](#examples) — [🚗 Autonomous Driving](#🚗-autonomous-driving) · [🤖 Robotics](#🤖-robotics) · [🌍 General Physics](#🌍-general-physics)
- [Results](#results) — [PAI-Bench-G](#pai-bench-g-physical-ai-bench--generation) · [RBench](#rbench) · [Physics-IQ Verified](#physics-iq-verified)
- [Quickstart](#quickstart) — [Path A: GitHub code](#path-a--github-code) · [Path B: Diffusers](#path-b--diffusers)
- [Acknowledgements](#acknowledgements)

---

## Overview

**Kandinsky WM  (World Model) 1.0** is a family of Image-to-Video models that adapt
[Kandinsky 5.0 Video Lite](https://github.com/kandinskylab/kandinsky-5) — a 2B-parameter
latent video diffusion model (a `Kandinsky5Transformer3DModel` DiT paired with a HunyuanVideo VAE
and Qwen2.5-VL + CLIP text encoders, trained with flow matching) — to **Physical AI**: video
generation that is not only visually convincing but *physically plausible* — consistent scene
geometry, object dynamics, interactions and cause-and-effect — so the model can serve as a source
of synthetic training data and as a building block for world models and simulators.

The base model was domain-adapted on large corpora of 5-second scenes (first text-to-video, then a
mixed text/image-to-video regime that preserves first-frame continuation), spanning **autonomous
driving**, **robotics**, and a **general** domain (industrial processes, physical phenomena, and
human–object / human–human interaction). Each checkpoint is then reinforcement-learning post-trained with **GRPO**
against a reward model that scores the physical plausibility of the generated scene, steering the
generator toward more faithful geometry, object behaviour and interactions than domain fine-tuning
alone.

Each checkpoint generates **5-second, 121-frame** clips at **768×512**.

## Model Zoo

Three domain checkpoints, same architecture — only the DiT weights differ (the VAE / text encoders are
byte-identical across all three). Each is published as a one-line diffusers pipeline; a
GitHub-code (DiT-only) copy lives in the **same** Hub repo used purely as file storage.

| Domain | Checkpoint | |
|---|---|---|
| 🚗 Autonomous Driving | [🤗 `Kandinsky-WM-1.0-I2V-5s-AV`](https://huggingface.co/kandinskylab/Kandinsky-WM-1.0-I2V-5s-AV) |  |
| 🤖 Robotics | [🤗 `Kandinsky-WM-1.0-I2V-5s-RO`](https://huggingface.co/kandinskylab/Kandinsky-WM-1.0-I2V-5s-RO) | ← **this repo** |
| 🌍 General Physics | [🤗 `Kandinsky-WM-1.0-I2V-5s-PH`](https://huggingface.co/kandinskylab/Kandinsky-WM-1.0-I2V-5s-PH) |  |

## Examples

Four **Kandinsky WM 1.0** image-to-video generations per domain (top-ranked by an internal visual
review). Each grid shows the first frame of a clip — **click it to play the generated video**
(hosted on the [`static_videos`](https://huggingface.co/datasets/sashakunitsyn/static_videos)
dataset). The first frame and prompt for every clip also live in [`assets/`](https://github.com/kandinskylab/kandinsky-wm/tree/main/assets): under
`assets/<domain>/`, `frame_N.jpg` and `prompt_N.txt` correspond to grid cell _N_ (left→right, top→bottom).

### 🚗 Autonomous Driving

<p align="left">
  <a style="display:inline-block;width:48.5%;max-width:334px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/av/1.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/av/preview_1.webp" alt="▶ AV clip 1"></a>
  <a style="display:inline-block;width:48.5%;max-width:334px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/av/2.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/av/preview_2.webp" alt="▶ AV clip 2"></a>
  <br>
  <a style="display:inline-block;width:48.5%;max-width:334px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/av/3.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/av/preview_3.webp" alt="▶ AV clip 3"></a>
  <a style="display:inline-block;width:48.5%;max-width:334px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/av/4.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/av/preview_4.webp" alt="▶ AV clip 4"></a>
</p>

First frames + prompts: [`assets/av/`](https://github.com/kandinskylab/kandinsky-wm/tree/main/assets/av).

### 🤖 Robotics

<p align="left">
  <a style="display:inline-block;width:58.3%;max-width:400px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/robotics/1.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/robotics/preview_1.webp" alt="▶ Robotics clip 1"></a>
  <a style="display:inline-block;width:38.7%;max-width:266px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/robotics/2.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/robotics/preview_2.webp" alt="▶ Robotics clip 2"></a>
  <br>
  <a style="display:inline-block;width:58.3%;max-width:400px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/robotics/3.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/robotics/preview_3.webp" alt="▶ Robotics clip 3"></a>
  <a style="display:inline-block;width:38.7%;max-width:266px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/robotics/4.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/robotics/preview_4.webp" alt="▶ Robotics clip 4"></a>
</p>

First frames + prompts: [`assets/robotics/`](https://github.com/kandinskylab/kandinsky-wm/tree/main/assets/robotics).

### 🌍 General Physics

<p align="left">
  <a style="display:inline-block;width:48.5%;max-width:400px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/general/1.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/general/preview_1.webp" alt="▶ General clip 1"></a>
  <a style="display:inline-block;width:48.5%;max-width:400px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/general/2.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/general/preview_2.webp" alt="▶ General clip 2"></a>
  <br>
  <a style="display:inline-block;width:48.5%;max-width:400px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/general/3.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/general/preview_3.webp" alt="▶ General clip 3"></a>
  <a style="display:inline-block;width:48.5%;max-width:400px" href="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/general/4.mp4"><img style="width:100%;margin:0 0 4px 0" src="https://huggingface.co/datasets/sashakunitsyn/static_videos/resolve/main/kandinsky-wm-1.0/assets/general/preview_4.webp" alt="▶ General clip 4"></a>
</p>

First frames + prompts: [`assets/general/`](https://github.com/kandinskylab/kandinsky-wm/tree/main/assets/general).

## Results

**Kandinsky WM 1.0** on three physical-AI video benchmarks (our row in **bold**). RBench and
Physics-IQ were run with Qwen3-VL prompt enhancers — noted above each table, with the scripts under
[`prompt_enhancers/`](https://github.com/kandinskylab/kandinsky-wm/tree/main/prompt_enhancers). Sizes are total parameter
counts where publicly disclosed (MoE models note active params; proprietary/undisclosed left as —).

### PAI-Bench-G (Physical AI Bench — Generation)

[Leaderboard](https://huggingface.co/spaces/shi-labs/physical-ai-bench-leaderboard). Column
abbreviations follow the PAI-Bench-G dimensions — see the leaderboard for exact definitions.
Run **without** prompt enhancement.

| Rank | Model | Size | Overall | Domain | Quality | SC | BC | MS | AQ | IQ | OC | IS | IB | CS | AV | RO | IN | PH | HU |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Cosmos3-Super | 64B | 83.9 | 89.5 | 78.2 | 92.7 | 94.1 | 99.2 | 52.7 | 70.8 | 20.5 | 97.7 | 98.1 | 94.4 | 77.5 | 90.0 | 90.7 | 95.0 | 87.6 |
| 2 | Cosmos3-Nano | 16B | 83.7 | 89.4 | 78.1 | 92.3 | 93.8 | 99.2 | 52.7 | 70.1 | 20.4 | 97.9 | 98.3 | 95.0 | 75.4 | 90.2 | 89.7 | 94.5 | 88.0 |
| 3 | Veo-3 | — | 82.1 | 86.7 | 77.6 | 91.4 | 93.1 | 99.2 | 51.9 | 69.8 | 21.7 | 97.0 | 96.9 | 94.4 | 68.7 | 86.9 | 89.7 | 91.6 | 84.4 |
| 4 | **Kandinsky WM 1.0** | **2B** | **81.7** | **86.0** | **77.4** | 91.8 | 94.1 | 99.0 | 53.2 | 64.9 | 21.6 | 97.2 | 97.7 | 93.6 | 72.6 | 82.1 | 88.0 | 90.7 | 86.0 |
| 5 | k5 Lite FT | 2B | 81.4 | 85.8 | 77.1 | 91.3 | 93.9 | 98.7 | 52.7 | 64.3 | 21.7 | 96.6 | 97.3 | 92.9 | 72.6 | 82.1 | 87.7 | 88.8 | 86.4 |
| 6 | Cosmos-Predict2.5-14B | 14B | 81.0 | 83.8 | 78.1 | 93.4 | 94.8 | 99.1 | 52.5 | 70.0 | 20.1 | 97.2 | 97.9 | 94.2 | 67.8 | 79.9 | 87.7 | 93.5 | 80.0 |
| 7 | Cosmos-Predict2.5-2B | 2B | 81.0 | 84.0 | 77.9 | 92.5 | 94.2 | 99.1 | 52.4 | 70.8 | 20.1 | 96.6 | 97.4 | 94.1 | 66.1 | 80.8 | 87.8 | 93.9 | 81.4 |
| 8 | Wan2.2-I2V-A14B | 27B (14B active) | 80.6 | 84.1 | 77.2 | 91.6 | 93.7 | 98.3 | 51.2 | 69.6 | 20.4 | 96.0 | 96.6 | 93.2 | 66.3 | 81.7 | 89.2 | 91.8 | 82.1 |
| 9 | K5 Lite | 2B | 80.5 | 83.0 | 77.9 | 91.7 | 94.4 | 99.3 | 54.1 | 65.6 | 21.7 | 98.1 | 98.6 | 89.2 | 66.3 | 77.3 | 86.3 | 87.2 | 84.6 |
| 10 | Wan2.2-TI2V-5B | 5B | 80.4 | 83.4 | 77.4 | 91.8 | 93.7 | 98.8 | 51.9 | 69.9 | 20.3 | 95.9 | 96.7 | 93.1 | 65.2 | 79.3 | 88.4 | 91.5 | 83.0 |

Physical-AI post-training lifts the base **K5 Lite** by **+1.2 Overall** (80.5 → 81.7) and
**+3.0 on the Domain axis** (83.0 → 86.0) — a 2B model landing between Veo-3 and the
Cosmos-Predict2.5 family.

### RBench

[Leaderboard](https://huggingface.co/spaces/DAGroup-PKU/RBench-Leaderboard), **Qwen** evaluator tab.
RBench evaluates robot-oriented image-to-video generation across five task categories and four
robot embodiments.

> **Prompt enhancer used:** [`prompt_enhancers/rbench/enhance_rbench_prompts_qwen_v3.py`](https://github.com/kandinskylab/kandinsky-wm/blob/main/prompt_enhancers/rbench/enhance_rbench_prompts_qwen_v3.py) — conservative Qwen3-VL image-grounded action canonicalization.

| Rank | Model | Size | Avg. | Common Manipulation | Spatial Relationship | Multi-entity Collaboration | Long-horizon Planning | Visual Reasoning | Single Arm | Dual Arm | Quadruped Robot | Humanoid Robot |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Veo 3 | — | 0.784 | 0.897 | 0.740 | 0.924 | 0.854 | 0.750 | 0.742 | 0.708 | 0.726 | 0.716 |
| 2 | Wan 2.5 | — | 0.781 | 0.888 | 0.825 | 0.916 | 0.719 | 0.738 | 0.742 | 0.744 | 0.711 | 0.742 |
| 3 | Hailuo v2 | — | 0.762 | 0.843 | 0.840 | 0.892 | 0.705 | 0.820 | 0.696 | 0.706 | 0.632 | 0.720 |
| 4 | Seedance 1.0 | — | 0.755 | 0.856 | 0.665 | 0.899 | 0.716 | 0.790 | 0.708 | 0.730 | 0.674 | 0.759 |
| 5 | Wan2.2_A14B | 27B (14B active) | 0.698 | 0.709 | 0.660 | 0.921 | 0.681 | 0.550 | 0.688 | 0.678 | 0.670 | 0.728 |
| 6 | **Kandinsky-WM-1.0** | **2B** | **0.651** | **0.836** | **0.685** | **0.846** | **0.657** | **0.524** | **0.509** | **0.568** | **0.578** | **0.656** |
| 7 | Cosmos 2.5 | 14B | 0.632 | 0.688 | 0.512 | 0.768 | 0.597 | 0.507 | 0.646 | 0.647 | 0.639 | 0.688 |
| 8 | LongCat-Video | 13.6B | 0.609 | 0.678 | 0.465 | 0.814 | 0.490 | 0.354 | 0.698 | 0.602 | 0.666 | 0.710 |
| 9 | DreamGen(gr1) | 14B | 0.575 | 0.507 | 0.500 | 0.848 | 0.353 | 0.405 | 0.660 | 0.632 | 0.611 | 0.656 |
| 10 | Wan2.2_5B | 5B | 0.551 | 0.598 | 0.402 | 0.722 | 0.450 | 0.420 | 0.511 | 0.534 | 0.638 | 0.682 |
| 11 | Wan2.1_14B | 14B | 0.542 | 0.688 | 0.400 | 0.702 | 0.465 | 0.270 | 0.519 | 0.562 | 0.604 | 0.664 |
| 12 | SkyReels | 13B | 0.531 | 0.546 | 0.400 | 0.687 | 0.324 | 0.358 | 0.612 | 0.574 | 0.654 | 0.628 |
| 13 | DreamGen(droid) | 14B | 0.514 | 0.465 | 0.505 | 0.591 | 0.302 | 0.386 | 0.589 | 0.570 | 0.584 | 0.633 |
| 14 | LTX-Video | 2B | 0.486 | 0.450 | 0.382 | 0.734 | 0.358 | 0.286 | 0.487 | 0.487 | 0.588 | 0.603 |
| 15 | FramePack | 13B | 0.453 | 0.455 | 0.240 | 0.630 | 0.197 | 0.345 | 0.403 | 0.500 | 0.670 | 0.635 |
| 16 | CogVideoX_5B | 5B | 0.322 | 0.290 | 0.240 | 0.426 | 0.096 | 0.030 | 0.374 | 0.422 | 0.494 | 0.524 |
| 17 | Vidar | — | 0.207 | 0.118 | 0.140 | 0.082 | 0.019 | 0.030 | 0.344 | 0.390 | 0.380 | 0.364 |
| 18 | UnifoLM-WMA-0 | — | 0.104 | 0.029 | 0.065 | 0.025 | 0.000 | 0.000 | 0.290 | 0.106 | 0.251 | 0.170 |

### Physics-IQ Verified
### Physics-IQ Verified

> **Prompt enhancer used:** [`prompt_enhancers/physics_iq/temporal_expansion.py`](https://github.com/kandinskylab/kandinsky-wm/blob/main/prompt_enhancers/physics_iq/temporal_expansion.py) — Qwen3-VL temporal caption expansion.

| # | Model | Size | Input type | Score | Date added |
|---|---|---|---|---|---|
| 1 | Magi-1 24B + GeoPhys (BoN) (op) | 24B | multiframe (v2v) | 58.2 ± 1.8 | 2026-06-19 |
| 2 | Magi-1 24B (op) | 24B | multiframe (v2v) | 48.4 ± 1.1 | 2026-06-19 |
| 3 | Cosmos3-Super-Image2Video | 64B | i2v | 39.5 ± 0.8 | 2026-06-18 |
| 4 | Grok Imagine Video | — | i2v | 34.8 ± 0.6 | 2026-06-17 |
| 5 | Magi-1 24B + GeoPhys (BoN) (op) | 24B | i2v | 33.7 ± 1.4 | 2026-06-19 |
| 6 | Hunyuan Video 1.5 | 8.3B | i2v | 33.4 ± 0.8 | 2026-06-17 |
| 7 | Wan 2.2 | 27B (14B active) | i2v | 32.2 ± 0.6 | 2026-06-17 |
| 8 | **Kandinsky WM 1.0** | **2B** | i2v | **30.8 ± 0.5** | internal |
| 9 | Cosmos3-Nano | 16B | i2v | 30.3 ± 0.6 | 2026-06-18 |
| 10 | Magi-1 24B (op) | 24B | i2v | 30.2 ± 1.1 | 2026-06-19 |
| 11 | Sora 2 | — | i2v | 26.5 ± 0.8 | 2026-06-17 |
| 12 | K5 Video Lite FT | 2B | i2v | 25.8 ± 1.7 | internal |
| 13 | P-Video | — | i2v | 25.3 ± 1.8 | 2026-06-17 |
| 14 | K5 Video Lite | 2B | i2v | 16.0 ± 1.2 | internal |

## Quickstart

Two ways to run — the checkpoints ship in both formats. The **GitHub-code path** is primary; the
**diffusers path** is a one-liner that already works off the Hub.

### Path A — GitHub code

The base inference code is vendored as a git submodule at [`kandinsky-5/`](https://github.com/kandinskylab/kandinsky-wm/tree/main/kandinsky-5)
(from [`kandinskylab/kandinsky-5`](https://github.com/kandinskylab/kandinsky-5)).

```bash
# 0. get this repo with the submodule
git clone --recurse-submodules https://github.com/kandinskylab/kandinsky-wm.git
cd kandinsky-wm
# (already cloned without it? run: git submodule update --init --recursive)

# 1. install the base code
pip install -r kandinsky-5/requirements.txt

# 2. pull a checkpoint (repo-as-file-storage; DiT + shared VAE/text encoders)
python examples/download_checkpoint.py --domain robotics

# 3. generate (CLI) — run from inside the submodule
cd kandinsky-5
python test.py \
  --config ./configs/k5_lite_i2v_5s_sft_sd.yaml \
  --image  ../assets/robotics/frame_1.jpg \
  --prompt "$(cat ../assets/robotics/prompt_1.txt)" \
  --video_duration 5
```

Python variant: [`examples/run_i2v_github.py`](https://github.com/kandinskylab/kandinsky-wm/blob/main/examples/run_i2v_github.py).

### Path B — Diffusers

```bash
pip install -U diffusers transformers accelerate imageio-ffmpeg
```

```python
import torch
from diffusers import Kandinsky5I2VPipeline
from diffusers.utils import export_to_video, load_image

pipe = Kandinsky5I2VPipeline.from_pretrained(
    "kandinskylab/Kandinsky-WM-1.0-I2V-5s-RO",
    torch_dtype=torch.bfloat16,
).to("cuda")

frames = pipe(
    image=load_image("assets/robotics/frame_1.jpg"),
    prompt=open("assets/robotics/prompt_1.txt").read().strip(),
    negative_prompt=("Static, 2D cartoon, cartoon, 2d animation, paintings, images, "
                     "worst quality, low quality, ugly, deformed, walking backwards"),
    height=512, width=768, num_frames=121,
    num_inference_steps=50, guidance_scale=5.0,
).frames[0]
export_to_video(frames, "robotics_out.mp4", fps=24, quality=9)
```

Script: [`examples/run_i2v_diffusers.py`](https://github.com/kandinskylab/kandinsky-wm/blob/main/examples/run_i2v_diffusers.py).


## Acknowledgements

Built on [Kandinsky 5.0](https://github.com/kandinskylab/kandinsky-5)
([HunyuanVideo VAE](https://huggingface.co/hunyuanvideo-community/HunyuanVideo),
[Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct),
[CLIP](https://huggingface.co/openai/clip-vit-large-patch14)). Evaluated on
[PAI-Bench-G](https://huggingface.co/spaces/shi-labs/physical-ai-bench-leaderboard),
[Physics-IQ](https://github.com/google-deepmind/physics-IQ-benchmark), and
[RBench](https://huggingface.co/spaces/DAGroup-PKU/RBench-Leaderboard).