sashakunitsyn commited on
Commit
e9346ed
Β·
1 Parent(s): 7989d31

docs: add RBench results and prompt enhancer

Browse files
Files changed (1) hide show
  1. README.md +37 -6
README.md CHANGED
@@ -29,7 +29,7 @@ tags: [kandinsky, image-to-video, physical-ai, world-model]
29
  - [Overview](#overview)
30
  - [Model Zoo](#model-zoo)
31
  - [Examples](#examples) β€” [πŸš— Autonomous Driving](#πŸš—-autonomous-driving) Β· [πŸ€– Robotics](#πŸ€–-robotics) Β· [🌍 General Physics](#🌍-general-physics)
32
- - [Results](#results) β€” [PAI-Bench-G](#pai-bench-g-physical-ai-bench--generation) Β· [Physics-IQ Verified](#physics-iq-verified)
33
  - [Quickstart](#quickstart) β€” [Path A: GitHub code](#path-a--github-code) Β· [Path B: Diffusers](#path-b--diffusers)
34
  - [Acknowledgements](#acknowledgements)
35
 
@@ -113,9 +113,9 @@ First frames + prompts: [`assets/general/`](https://github.com/kandinskylab/kand
113
 
114
  ## Results
115
 
116
- **Kandinsky WM 1.0** on two physical-AI video benchmarks (our row in **bold**; rows copied as-is
117
- from our internal leaderboard). Physics-IQ was run with a Qwen3-VL prompt enhancer β€” noted above the
118
- table, with the script under [`prompt_enhancers/`](https://github.com/kandinskylab/kandinsky-wm/tree/main/prompt_enhancers). Sizes are total parameter
119
  counts where publicly disclosed (MoE models note active params; proprietary/undisclosed left as β€”).
120
 
121
  ### PAI-Bench-G (Physical AI Bench β€” Generation)
@@ -141,6 +141,36 @@ Physical-AI post-training lifts the base **K5 Lite** by **+1.2 Overall** (80.5
141
  **+3.0 on the Domain axis** (83.0 β†’ 86.0) β€” a 2B model landing between Veo-3 and the
142
  Cosmos-Predict2.5 family.
143
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
144
  ### Physics-IQ Verified
145
 
146
  > **Prompt enhancer used:** [`prompt_enhancers/physics_iq/temporal_expansion.py`](https://github.com/kandinskylab/kandinsky-wm/blob/main/prompt_enhancers/physics_iq/temporal_expansion.py) β€” Qwen3-VL temporal caption expansion.
@@ -231,5 +261,6 @@ Built on [Kandinsky 5.0](https://github.com/kandinskylab/kandinsky-5)
231
  ([HunyuanVideo VAE](https://huggingface.co/hunyuanvideo-community/HunyuanVideo),
232
  [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct),
233
  [CLIP](https://huggingface.co/openai/clip-vit-large-patch14)). Evaluated on
234
- [PAI-Bench-G](https://huggingface.co/spaces/shi-labs/physical-ai-bench-leaderboard) and
235
- [Physics-IQ](https://github.com/google-deepmind/physics-IQ-benchmark).
 
 
29
  - [Overview](#overview)
30
  - [Model Zoo](#model-zoo)
31
  - [Examples](#examples) β€” [πŸš— Autonomous Driving](#πŸš—-autonomous-driving) Β· [πŸ€– Robotics](#πŸ€–-robotics) Β· [🌍 General Physics](#🌍-general-physics)
32
+ - [Results](#results) β€” [PAI-Bench-G](#pai-bench-g-physical-ai-bench--generation) Β· [RBench](#rbench) Β· [Physics-IQ Verified](#physics-iq-verified)
33
  - [Quickstart](#quickstart) β€” [Path A: GitHub code](#path-a--github-code) Β· [Path B: Diffusers](#path-b--diffusers)
34
  - [Acknowledgements](#acknowledgements)
35
 
 
113
 
114
  ## Results
115
 
116
+ **Kandinsky WM 1.0** on three physical-AI video benchmarks (our row in **bold**). RBench and
117
+ Physics-IQ were run with Qwen3-VL prompt enhancers β€” noted above each table, with the scripts under
118
+ [`prompt_enhancers/`](https://github.com/kandinskylab/kandinsky-wm/tree/main/prompt_enhancers). Sizes are total parameter
119
  counts where publicly disclosed (MoE models note active params; proprietary/undisclosed left as β€”).
120
 
121
  ### PAI-Bench-G (Physical AI Bench β€” Generation)
 
141
  **+3.0 on the Domain axis** (83.0 β†’ 86.0) β€” a 2B model landing between Veo-3 and the
142
  Cosmos-Predict2.5 family.
143
 
144
+ ### RBench
145
+
146
+ [Leaderboard](https://huggingface.co/spaces/DAGroup-PKU/RBench-Leaderboard), **Qwen** evaluator tab.
147
+ RBench evaluates robot-oriented image-to-video generation across five task categories and four
148
+ robot embodiments.
149
+
150
+ > **Prompt enhancer used:** [`prompt_enhancers/rbench/enhance_rbench_prompts_qwen_v3.py`](https://github.com/kandinskylab/kandinsky-wm/blob/main/prompt_enhancers/rbench/enhance_rbench_prompts_qwen_v3.py) β€” conservative Qwen3-VL image-grounded action canonicalization.
151
+
152
+ | Rank | Model | Size | Avg. | Common Manipulation | Spatial Relationship | Multi-entity Collaboration | Long-horizon Planning | Visual Reasoning | Single Arm | Dual Arm | Quadruped Robot | Humanoid Robot |
153
+ |---|---|---|---|---|---|---|---|---|---|---|---|---|
154
+ | 1 | Veo 3 | β€” | 0.784 | 0.897 | 0.740 | 0.924 | 0.854 | 0.750 | 0.742 | 0.708 | 0.726 | 0.716 |
155
+ | 2 | Wan 2.5 | β€” | 0.781 | 0.888 | 0.825 | 0.916 | 0.719 | 0.738 | 0.742 | 0.744 | 0.711 | 0.742 |
156
+ | 3 | Hailuo v2 | β€” | 0.762 | 0.843 | 0.840 | 0.892 | 0.705 | 0.820 | 0.696 | 0.706 | 0.632 | 0.720 |
157
+ | 4 | Seedance 1.0 | β€” | 0.755 | 0.856 | 0.665 | 0.899 | 0.716 | 0.790 | 0.708 | 0.730 | 0.674 | 0.759 |
158
+ | 5 | Wan2.2_A14B | 27B (14B active) | 0.698 | 0.709 | 0.660 | 0.921 | 0.681 | 0.550 | 0.688 | 0.678 | 0.670 | 0.728 |
159
+ | 6 | **Kandinsky-WM-1.0** | **2B** | **0.651** | **0.836** | **0.685** | **0.846** | **0.657** | **0.524** | **0.509** | **0.568** | **0.578** | **0.656** |
160
+ | 7 | Cosmos 2.5 | 14B | 0.632 | 0.688 | 0.512 | 0.768 | 0.597 | 0.507 | 0.646 | 0.647 | 0.639 | 0.688 |
161
+ | 8 | LongCat-Video | 13.6B | 0.609 | 0.678 | 0.465 | 0.814 | 0.490 | 0.354 | 0.698 | 0.602 | 0.666 | 0.710 |
162
+ | 9 | DreamGen(gr1) | 14B | 0.575 | 0.507 | 0.500 | 0.848 | 0.353 | 0.405 | 0.660 | 0.632 | 0.611 | 0.656 |
163
+ | 10 | Wan2.2_5B | 5B | 0.551 | 0.598 | 0.402 | 0.722 | 0.450 | 0.420 | 0.511 | 0.534 | 0.638 | 0.682 |
164
+ | 11 | Wan2.1_14B | 14B | 0.542 | 0.688 | 0.400 | 0.702 | 0.465 | 0.270 | 0.519 | 0.562 | 0.604 | 0.664 |
165
+ | 12 | SkyReels | 13B | 0.531 | 0.546 | 0.400 | 0.687 | 0.324 | 0.358 | 0.612 | 0.574 | 0.654 | 0.628 |
166
+ | 13 | DreamGen(droid) | 14B | 0.514 | 0.465 | 0.505 | 0.591 | 0.302 | 0.386 | 0.589 | 0.570 | 0.584 | 0.633 |
167
+ | 14 | LTX-Video | 2B | 0.486 | 0.450 | 0.382 | 0.734 | 0.358 | 0.286 | 0.487 | 0.487 | 0.588 | 0.603 |
168
+ | 15 | FramePack | 13B | 0.453 | 0.455 | 0.240 | 0.630 | 0.197 | 0.345 | 0.403 | 0.500 | 0.670 | 0.635 |
169
+ | 16 | CogVideoX_5B | 5B | 0.322 | 0.290 | 0.240 | 0.426 | 0.096 | 0.030 | 0.374 | 0.422 | 0.494 | 0.524 |
170
+ | 17 | Vidar | β€” | 0.207 | 0.118 | 0.140 | 0.082 | 0.019 | 0.030 | 0.344 | 0.390 | 0.380 | 0.364 |
171
+ | 18 | UnifoLM-WMA-0 | β€” | 0.104 | 0.029 | 0.065 | 0.025 | 0.000 | 0.000 | 0.290 | 0.106 | 0.251 | 0.170 |
172
+
173
+ ### Physics-IQ Verified
174
  ### Physics-IQ Verified
175
 
176
  > **Prompt enhancer used:** [`prompt_enhancers/physics_iq/temporal_expansion.py`](https://github.com/kandinskylab/kandinsky-wm/blob/main/prompt_enhancers/physics_iq/temporal_expansion.py) β€” Qwen3-VL temporal caption expansion.
 
261
  ([HunyuanVideo VAE](https://huggingface.co/hunyuanvideo-community/HunyuanVideo),
262
  [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct),
263
  [CLIP](https://huggingface.co/openai/clip-vit-large-patch14)). Evaluated on
264
+ [PAI-Bench-G](https://huggingface.co/spaces/shi-labs/physical-ai-bench-leaderboard),
265
+ [Physics-IQ](https://github.com/google-deepmind/physics-IQ-benchmark), and
266
+ [RBench](https://huggingface.co/spaces/DAGroup-PKU/RBench-Leaderboard).