UnityShots Benchmark

A multilingual, multi-cultural k-shot storytelling benchmark for evaluating multi-shot audio-video generation. Each case is a short cinematic story told across several shots, with a consistent cast whose identity, voice, and world must persist across every cut.

This is the evaluation benchmark released with UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating.

• 📄 Paper: arXiv:2606.21661
• 🌐 Project page: jackailab.github.io/Projects/UnityShots
• 💻 Code: github.com/JIA-Lab-research/UnityShots

Overview

Regional & language distribution

• Regions: East Asia (70) · Europe (42) · South/SE Asia (26) · Africa (23) · Latin America (20) · Middle East (19)
• Primary languages: Cantonese (77) · Mandarin (73) · German (51) · English (42) · Spanish (40) · Swahili (30) · Hindi (27) · Arabic (27) · Bengali (25) · Yoruba (19) · Persian (14) · Portuguese (3) · Vietnamese (2)

Dataset structure

UnityShotsBench/
├── index.json                 # all 200 cases: metadata, characters, per-shot scripts
├── seq_list.txt               # ordered list of sequence IDs
├── refs_id/                   # reference identity portraits (for R2V)
│   └── story_XXX/charN.jpg
├── refs_audio/                # reference voice clips, per character (for R2V)
│   └── story_XXX/charN.{wav,mp3}
└── refs_first_frame/          # per-shot first-frame anchors (for I2V)
    └── story_XXX/shot_K.jpg

index.json — per-case schema

{
  "sequence_id": "story_001",
  "type": "storytelling",
  "title": "Blueprints and Sweet Potatoes",
  "global_caption": "On a rainy night in Beijing, a veteran taxi driver and a weary architect ...",
  "_region": "east_asia",
  "_category": "general",
  "characters": [
    { "name": "...", "age": 58, "gender": "M", "language": "zh",
      "voice": "...", "appearance": "..." }
  ],
  "shots": [
    { "idx": 0, "duration_sec": 5.0, "transition": "HARD_CUT",
      "shot_caption": "...", "dialogue": "...", "audio": "..." }
  ]
}

Reference files follow the character order in index.json: the i-th character maps to charI.jpg (identity) and charI.wav / charI.mp3 (voice). I2V first frames map shot k to shot_K.jpg.

Conditioning modes

The same story can be evaluated under three input regimes, all covered by this benchmark:

• T2V — per-shot text captions only.
• I2V — the per-shot first-frame anchors in refs_first_frame/.
• R2V — external identity (refs_id/) + reference voice (refs_audio/) per character.

Intended use & evaluation

Designed for fair evaluation of cross-shot identity preservation, voice/timbre consistency, lip-sync, scene continuity, and audio–text alignment in multi-shot generation. Reference identities are public-domain or AI-generated and are provided for academic, non-commercial research only.

Citation

@article{huang2026unityshots,
  title   = {UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating},
  author  = {Huang, Jiehui and Zhang, Yuechen and Xia, Bin and Wang, Jiahao and
             He, Xu and Tang, Zhenchao and Chu, Meng and Tao, Xin and Wan, Pengfei and Jia, Jiaya},
  journal = {arXiv preprint arXiv:2606.21661},
  year    = {2026}
}

License

Released under CC BY-NC 4.0 (non-commercial). Reference identities and voices are for academic research and benchmarking only.