README.md

metadata

license: mit
pipeline_tag: image-to-video
library_name: diffusers
tags:
  - kandinsky
  - image-to-video
  - physical-ai
  - world-model

Kandinsky WM 1.0: A family of models for Physical AI

Image-to-Video generation for Physical AI: autonomous driving · robotics · general physics

🤗 Checkpoints collection · 🧩 Base model (Kandinsky 5.0) · 💻 GitHub · 📝 Habr article

This repo holds the 🤖 Robotics checkpoint of the Kandinsky WM 1.0 family — DiT robot_rl, published here in two layouts: a diffusers pipeline (model_index.json, transformer/, vae/, text_encoder*/) and the GitHub-code DiT at model/robot_rl.safetensors (same weights). The other two variants are in the Kandinsky WM 1.0 collection.

---

Contents

• Overview
• Model Zoo
• Examples — 🚗 Autonomous Driving · 🤖 Robotics · 🌍 General Physics
• Results — PAI-Bench-G · Physics-IQ Verified
• Quickstart — Path A: GitHub code · Path B: Diffusers
• Acknowledgements

---

Overview

Kandinsky WM (World Model) 1.0 is a family of Image-to-Video models that adapt Kandinsky 5.0 Video Lite — a 2B-parameter latent video diffusion model (a Kandinsky5Transformer3DModel DiT paired with a HunyuanVideo VAE and Qwen2.5-VL + CLIP text encoders, trained with flow matching) — to Physical AI: video generation that is not only visually convincing but physically plausible — consistent scene geometry, object dynamics, interactions and cause-and-effect — so the model can serve as a source of synthetic training data and as a building block for world models and simulators.

The base model was domain-adapted on large corpora of 5-second scenes (first text-to-video, then a mixed text/image-to-video regime that preserves first-frame continuation), spanning autonomous driving, robotics, and a general domain (industrial processes, physical phenomena, and human–object / human–human interaction). Each checkpoint is then reinforcement-learning post-trained with GRPO against a reward model that scores the physical plausibility of the generated scene, steering the generator toward more faithful geometry, object behaviour and interactions than domain fine-tuning alone.

Each checkpoint generates 5-second, 121-frame clips at 768×512.

Model Zoo

Three domain checkpoints, same architecture — only the DiT weights differ (the VAE / text encoders are byte-identical across all three). Each is published as a one-line diffusers pipeline; a GitHub-code (DiT-only) copy lives in the same Hub repo used purely as file storage.

Domain	Checkpoint
🚗 Autonomous Driving	🤗 Kandinsky-WM-1.0-I2V-5s-AV
🤖 Robotics	🤗 Kandinsky-WM-1.0-I2V-5s-RO	← this repo
🌍 General Physics	🤗 Kandinsky-WM-1.0-I2V-5s-PH

Examples

Four Kandinsky WM 1.0 image-to-video generations per domain (top-ranked by an internal visual review). Each grid shows the first frame of a clip — click it to play the generated video (hosted on the static_videos dataset). The first frame and prompt for every clip also live in assets/: under assets/<domain>/, frame_N.jpg and prompt_N.txt correspond to grid cell N (left→right, top→bottom).

🚗 Autonomous Driving

First frames + prompts: assets/av/.

🤖 Robotics

First frames + prompts: assets/robotics/.

🌍 General Physics

First frames + prompts: assets/general/.

Results

Kandinsky WM 1.0 on two physical-AI video benchmarks (our row in bold; rows copied as-is from our internal leaderboard). Physics-IQ was run with a Qwen3-VL prompt enhancer — noted above the table, with the script under prompt_enhancers/. Sizes are total parameter counts where publicly disclosed (MoE models note active params; proprietary/undisclosed left as —).

PAI-Bench-G (Physical AI Bench — Generation)

Leaderboard. Column abbreviations follow the PAI-Bench-G dimensions — see the leaderboard for exact definitions. Run without prompt enhancement.

Rank	Model	Size	Overall	Domain	Quality	SC	BC	MS	AQ	IQ	OC	IS	IB	CS	AV	RO	IN	PH	HU
1	Cosmos3-Super	64B	83.9	89.5	78.2	92.7	94.1	99.2	52.7	70.8	20.5	97.7	98.1	94.4	77.5	90.0	90.7	95.0	87.6
2	Cosmos3-Nano	16B	83.7	89.4	78.1	92.3	93.8	99.2	52.7	70.1	20.4	97.9	98.3	95.0	75.4	90.2	89.7	94.5	88.0
3	Veo-3	—	82.1	86.7	77.6	91.4	93.1	99.2	51.9	69.8	21.7	97.0	96.9	94.4	68.7	86.9	89.7	91.6	84.4
4	Kandinsky WM 1.0	2B	81.7	86.0	77.4	91.8	94.1	99.0	53.2	64.9	21.6	97.2	97.7	93.6	72.6	82.1	88.0	90.7	86.0
5	k5 Lite FT	2B	81.4	85.8	77.1	91.3	93.9	98.7	52.7	64.3	21.7	96.6	97.3	92.9	72.6	82.1	87.7	88.8	86.4
6	Cosmos-Predict2.5-14B	14B	81.0	83.8	78.1	93.4	94.8	99.1	52.5	70.0	20.1	97.2	97.9	94.2	67.8	79.9	87.7	93.5	80.0
7	Cosmos-Predict2.5-2B	2B	81.0	84.0	77.9	92.5	94.2	99.1	52.4	70.8	20.1	96.6	97.4	94.1	66.1	80.8	87.8	93.9	81.4
8	Wan2.2-I2V-A14B	27B (14B active)	80.6	84.1	77.2	91.6	93.7	98.3	51.2	69.6	20.4	96.0	96.6	93.2	66.3	81.7	89.2	91.8	82.1
9	K5 Lite	2B	80.5	83.0	77.9	91.7	94.4	99.3	54.1	65.6	21.7	98.1	98.6	89.2	66.3	77.3	86.3	87.2	84.6
10	Wan2.2-TI2V-5B	5B	80.4	83.4	77.4	91.8	93.7	98.8	51.9	69.9	20.3	95.9	96.7	93.1	65.2	79.3	88.4	91.5	83.0

Physical-AI post-training lifts the base K5 Lite by +1.2 Overall (80.5 → 81.7) and +3.0 on the Domain axis (83.0 → 86.0) — a 2B model landing between Veo-3 and the Cosmos-Predict2.5 family.

Physics-IQ Verified

Prompt enhancer used: prompt_enhancers/physics_iq/temporal_expansion.py — Qwen3-VL temporal caption expansion.

#	Model	Size	Input type	Score	Date added
1	Magi-1 24B + GeoPhys (BoN) (op)	24B	multiframe (v2v)	58.2 ± 1.8	2026-06-19
2	Magi-1 24B (op)	24B	multiframe (v2v)	48.4 ± 1.1	2026-06-19
3	Cosmos3-Super-Image2Video	64B	i2v	39.5 ± 0.8	2026-06-18
4	Grok Imagine Video	—	i2v	34.8 ± 0.6	2026-06-17
5	Magi-1 24B + GeoPhys (BoN) (op)	24B	i2v	33.7 ± 1.4	2026-06-19
6	Hunyuan Video 1.5	8.3B	i2v	33.4 ± 0.8	2026-06-17
7	Wan 2.2	27B (14B active)	i2v	32.2 ± 0.6	2026-06-17
8	Kandinsky WM 1.0	2B	i2v	30.8 ± 0.5	internal
9	Cosmos3-Nano	16B	i2v	30.3 ± 0.6	2026-06-18
10	Magi-1 24B (op)	24B	i2v	30.2 ± 1.1	2026-06-19
11	Sora 2	—	i2v	26.5 ± 0.8	2026-06-17
12	K5 Video Lite FT	2B	i2v	25.8 ± 1.7	internal
13	P-Video	—	i2v	25.3 ± 1.8	2026-06-17
14	K5 Video Lite	2B	i2v	16.0 ± 1.2	internal

Quickstart

Two ways to run — the checkpoints ship in both formats. The GitHub-code path is primary; the diffusers path is a one-liner that already works off the Hub.

Path A — GitHub code

The base inference code is vendored as a git submodule at kandinsky-5/ (from kandinskylab/kandinsky-5).

# 0. get this repo with the submodule
git clone --recurse-submodules https://github.com/kandinskylab/kandinsky-wm.git
cd kandinsky-wm
# (already cloned without it? run: git submodule update --init --recursive)

# 1. install the base code
pip install -r kandinsky-5/requirements.txt

# 2. pull a checkpoint (repo-as-file-storage; DiT + shared VAE/text encoders)
python examples/download_checkpoint.py --domain robotics

# 3. generate (CLI) — run from inside the submodule
cd kandinsky-5
python test.py \
  --config ./configs/k5_lite_i2v_5s_sft_sd.yaml \
  --image  ../assets/robotics/frame_1.jpg \
  --prompt "$(cat ../assets/robotics/prompt_1.txt)" \
  --video_duration 5

Python variant: examples/run_i2v_github.py.

Path B — Diffusers

pip install -U diffusers transformers accelerate imageio-ffmpeg

import torch
from diffusers import Kandinsky5I2VPipeline
from diffusers.utils import export_to_video, load_image

pipe = Kandinsky5I2VPipeline.from_pretrained(
    "kandinskylab/Kandinsky-WM-1.0-I2V-5s-RO",
    torch_dtype=torch.bfloat16,
).to("cuda")

frames = pipe(
    image=load_image("assets/robotics/frame_1.jpg"),
    prompt=open("assets/robotics/prompt_1.txt").read().strip(),
    negative_prompt=("Static, 2D cartoon, cartoon, 2d animation, paintings, images, "
                     "worst quality, low quality, ugly, deformed, walking backwards"),
    height=512, width=768, num_frames=121,
    num_inference_steps=50, guidance_scale=5.0,
).frames[0]
export_to_video(frames, "robotics_out.mp4", fps=24, quality=9)

Script: examples/run_i2v_diffusers.py.

Acknowledgements

Built on Kandinsky 5.0 (HunyuanVideo VAE, Qwen2.5-VL, CLIP). Evaluated on PAI-Bench-G and Physics-IQ.