Release Motius AIST++ music-to-dance artifact

ATTRIBUTIONS.md

@@ -0,0 +1,18 @@
+# Bailando Attribution
+
+This implementation adapts the network architecture and evaluation protocol
+from [lisiyao21/Bailando](https://github.com/lisiyao21/Bailando) at revision
+`cc90b98bff81c9709570db413c9610c2562e27ca`.
+
+Bailando is distributed under the S-Lab License 1.0. The vendored license is
+included as `LICENSE`. The kinetic and geometric feature implementations used
+by the evaluator originate from Meta/Facebook Fairmotion and retain their BSD
+license headers.
+
+The AIST++ reference features are derived from the AIST++ Dance Motion Dataset,
+Copyright 2021 Google LLC and licensed under CC BY 4.0. The evaluator artifact
+contains derived feature vectors and source-sequence audit metadata, not the raw
+AIST++ motion files.
+
+Paper: Li Siyao et al., "Bailando: 3D Dance Generation by Actor-Critic GPT
+With Choreographic Memory," CVPR 2022.

LICENSE

@@ -0,0 +1,35 @@
+S-Lab License 1.0
+
+Copyright 2022 S-Lab
+
+Redistribution and use for non-commercial purpose in source and 
+binary forms, with or without modification, are permitted provided 
+that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright 
+   notice, this list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright 
+   notice, this list of conditions and the following disclaimer in 
+   the documentation and/or other materials provided with the 
+   distribution.
+
+3. Neither the name of the copyright holder nor the names of its 
+   contributors may be used to endorse or promote products derived 
+   from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 
+HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+In the event that redistribution and/or use for commercial purpose in 
+source or binary forms, with or without modification is required, 
+please contact the contributor(s) of the work.

README.md

@@ -0,0 +1,189 @@
+---
+license: other
+license_name: s-lab-license-1.0
+license_link: https://github.com/lisiyao21/Bailando/blob/master/LICENSE
+library_name: motius
+tags:
+  - motion-generation
+  - music-to-dance
+  - aistplusplus
+  - bailando
+datasets:
+  - yeok/danceba
+---
+
+<h1 align="center">Bailando Model Card</h1>
+
+<p align="center">
+  <strong>Actor-critic music-to-dance generation with a learned choreographic memory.</strong>
+</p>
+
+<p align="center">
+  <a href="https://arxiv.org/abs/2203.13055">Paper</a> |
+  <a href="https://www.mmlab-ntu.com/project/bailando/">Project Page</a> |
+  <a href="https://github.com/lisiyao21/Bailando">Original GitHub</a> |
+  <a href="https://huggingface.co/ZeyuLing/Motius-Bailando-AISTPP">Motius Checkpoint</a> |
+  <a href="../tasks/music_to_dance.md">Task Protocol</a>
+</p>
+
+Bailando is the CVPR 2022 oral work *Bailando: 3D Dance Generation by
+Actor-Critic GPT with Choreographic Memory*. It learns upper- and lower-body
+VQ codebooks, then composes those dance units autoregressively from music. The
+Motius release implements the model, audio frontend, AIST++ dataset, pipeline,
+representation bridge, and evaluator without importing an external checkout at
+runtime.
+
+## Preview
+
+<table>
+  <tr>
+    <td width="33%"><img src="https://raw.githubusercontent.com/ZeyuLing/Motius/main/assets/model_zoo/bailando/bailando_aistpp_break_gBR_mBR0_smpl_mesh_512_30fps.gif" alt="Bailando break dance"></td>
+    <td width="33%"><img src="https://raw.githubusercontent.com/ZeyuLing/Motius/main/assets/model_zoo/bailando/bailando_aistpp_krump_gKR_mKR2_smpl_mesh_512_30fps.gif" alt="Bailando krump dance"></td>
+    <td width="33%"><img src="https://raw.githubusercontent.com/ZeyuLing/Motius/main/assets/model_zoo/bailando/bailando_aistpp_waacking_gWA_mWA0_smpl_mesh_512_30fps.gif" alt="Bailando waacking dance"></td>
+  </tr>
+  <tr>
+    <td align="center"><sub>Break / <code>gBR...mBR0</code></sub></td>
+    <td align="center"><sub>Krump / <code>gKR...mKR2</code></sub></td>
+    <td align="center"><sub>Waacking / <code>gWA...mWA0</code></sub></td>
+  </tr>
+</table>
+
+The previews are distinct AIST++ evaluation outputs rendered as neutral SMPL
+meshes at 512x512 and 30 fps. Their position-IK fit errors are 13.75, 13.89,
+and 13.71 mm. MP4 sources and fit reports are stored beside the GIF assets.
+
+## Release Snapshot
+
+| Item | Value |
+| ---- | ----- |
+| Task | Music-to-Dance |
+| Dataset | AIST++ cross-modal split |
+| Music input | 438D features at 7.5 fps, or raw audio through the bundled frontend |
+| Native motion | AIST++ global SMPL-24 joint positions at 60 fps |
+| Parameters | 173,368,139 |
+| Checkpoint | [`ZeyuLing/Motius-Bailando-AISTPP`](https://huggingface.co/ZeyuLing/Motius-Bailando-AISTPP) |
+| Pipeline | `motius.pipelines.bailando.BailandoPipeline` |
+| Upstream revision | `lisiyao21/Bailando@cc90b98bff81c9709570db413c9610c2562e27ca` |
+| License | S-Lab License 1.0, non-commercial use |
+
+The Hugging Face artifact contains both VQ-VAE branches and the actor-critic
+GPT as safetensors, plus the complete architecture config, source hashes,
+license, and attribution. It does not require an upstream repository or a
+second checkpoint download.
+
+## Usage
+
+Install the music frontend dependencies:
+
+```bash
+python -m pip install -e '.[music-to-dance]'
+```
+
+Generate from an audio file:
+
+```python
+from motius.pipelines.bailando import BailandoPipeline
+
+pipe = BailandoPipeline.from_pretrained(
+    "ZeyuLing/Motius-Bailando-AISTPP",
+    device="cuda",
+)
+result = pipe("music.wav")
+
+print(result.joints.shape)       # (batch, frames, 24, 3)
+print(result.music_features.shape)  # (batch, music_frames, 438)
+```
+
+For exact benchmark reproduction, pass the released 438D AIST++ feature stream
+and paired initial motion. Only the first upper/lower VQ token initializes the
+generation, matching the official script:
+
+```python
+result = pipe(
+    music_features=music_features_7p5fps,
+    initial_motion=paired_gt_smpl24,
+)
+```
+
+Without `initial_motion`, the public demo seed `(423, 12)` is used. Raw-audio
+inference is a convenience path; use released precomputed features when exact
+paper parity across audio-library versions matters.
+
+## Evaluation
+
+Motius ran the converted official epoch-500 VQ-VAE and epoch-10 GPT on all 40
+cross-modal validation/test cases. FID and diversity use the 1,320 valid motion
+PKLs in the AIST++ v1 reference archive. Generated features use the first 1,200
+frames; reference features use complete sequences; BeatAlign uses the complete
+generated sequence and the paired 60 fps music-beat stream.
+
+| Result | FID_k | FID_g | Diversity_k | Diversity_g | BeatAlign |
+| ------ | ----: | ----: | ----------: | ----------: | --------: |
+| Motius reproduction | 28.11 | 9.70 | 7.73 | 6.31 | 0.2268 |
+| Bailando paper | 28.16 | 9.62 | 7.83 | 6.34 | 0.2332 |
+| Motius GT | 17.16 | 10.66 | 8.17 | 7.49 | 0.2247 |
+| GT paper | 17.10 | 10.60 | 8.19 | 7.45 | 0.2374 |
+
+Lower is better for FID, higher is better for BeatAlign, and diversity is
+interpreted relative to GT. The paper values are shown as parity targets and
+are not copied into the reproduced row.
+
+### Physical Diagnostics
+
+These Motius joint-level diagnostics use the common SMPL-22 subset. They are
+not metrics from the Bailando paper.
+
+| Result | Jitter | Dynamic | Penetration | Float | Slide |
+| ------ | -----: | ------: | ----------: | ----: | ----: |
+| Bailando | 0.00558 | 0.02183 | 0.00000 | 0.21803 | 0.00428 |
+| Paired GT | 0.00677 | 0.02276 | 0.00000 | 0.10658 | 0.00330 |
+
+`Dynamic` is an expressiveness statistic to compare with GT rather than
+minimize. The floor-dependent diagnostics are reported in native metric units.
+
+## Motion Representation
+
+The public representation name is `aistpp_smpl24_joints`, shape `(T,24,3)` in
+metres with Y up. Joints `0:22` are the standard SMPL body chain and convert
+exactly to `smpl22_joints`:
+
+```python
+from motius.motion import convert_motion
+
+smpl22 = convert_motion(
+    result.joints[0],
+    source="aistpp_smpl24_joints",
+    target="smpl22_joints",
+)
+```
+
+Conversion to `motion135` and SMPL mesh uses position IK because the generated
+tensor stores joint positions rather than local rotations. The three preview
+reports expose the resulting fit errors instead of hiding this lossy step.
+
+## Reproduction Audit
+
+| Check | Result |
+| ----- | ------ |
+| Official checkpoint load | Zero missing and zero unexpected tensors |
+| VQ-VAE source SHA-256 | `35670f42a3b3092438f73f0af3ace7b52e318a8b5c00b2b05c92078176b21716` |
+| GPT source SHA-256 | `903863a4e1cac01fcec30f7939c591eac8ea89f74e9837b93babe3383eecb403` |
+| Generated cases | 40/40, all finite |
+| Full inference time | 80.17 seconds on one H20 |
+| Reference pool | 1,365 PKLs minus the official 45-entry ignore list = 1,320 |
+| SMPL-24 FK calibration | 0.000066 mm MPJPE over 540,384 joint-frames |
+
+The model code remains under the upstream S-Lab License 1.0. AIST++
+annotations are CC BY 4.0. See the artifact `LICENSE` and `ATTRIBUTIONS.md`
+before redistribution or commercial use.
+
+## Citation
+
+```bibtex
+@inproceedings{siyao2022bailando,
+  title={Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory},
+  author={Siyao, Li and Yu, Weijiang and Gu, Tianpei and Lin, Chunze and Wang, Quan and Qian, Chen and Loy, Chen Change and Liu, Ziwei},
+  booktitle={CVPR},
+  year={2022}
+}
+```

bailando_config.json

@@ -0,0 +1,123 @@
+{
+  "artifact_format": "motius-bailando-v1",
+  "model_type": "bailando",
+  "source_repository": "https://github.com/lisiyao21/Bailando",
+  "source_revision": "cc90b98bff81c9709570db413c9610c2562e27ca",
+  "provenance": {
+    "source_repository": "https://github.com/lisiyao21/Bailando",
+    "source_revision": "cc90b98bff81c9709570db413c9610c2562e27ca",
+    "source_checkpoints": {
+      "vqvae": {
+        "filename": "epoch_500.pt",
+        "sha256": "35670f42a3b3092438f73f0af3ace7b52e318a8b5c00b2b05c92078176b21716"
+      },
+      "gpt": {
+        "filename": "epoch_10.pt",
+        "sha256": "903863a4e1cac01fcec30f7939c591eac8ea89f74e9837b93babe3383eecb403"
+      }
+    }
+  },
+  "config": {
+    "fps": 60.0,
+    "code_downsample": 8,
+    "motion_representation": "aistpp_smpl24_joints",
+    "default_initial_codes": [
+      423,
+      12
+    ],
+    "vqvae": {
+      "up_half": {
+        "levels": 1,
+        "downs_t": [
+          3
+        ],
+        "strides_t": [
+          2
+        ],
+        "emb_width": 512,
+        "l_bins": 512,
+        "l_mu": 0.99,
+        "commit": 0.02,
+        "hvqvae_multipliers": [
+          1
+        ],
+        "width": 512,
+        "depth": 3,
+        "m_conv": 1.0,
+        "dilation_growth_rate": 3,
+        "sample_length": 240,
+        "use_bottleneck": true,
+        "joint_channel": 3,
+        "vqvae_reverse_decoder_dilation": true
+      },
+      "down_half": {
+        "levels": 1,
+        "downs_t": [
+          3
+        ],
+        "strides_t": [
+          2
+        ],
+        "emb_width": 512,
+        "l_bins": 512,
+        "l_mu": 0.99,
+        "commit": 0.02,
+        "hvqvae_multipliers": [
+          1
+        ],
+        "width": 512,
+        "depth": 3,
+        "m_conv": 1.0,
+        "dilation_growth_rate": 3,
+        "sample_length": 240,
+        "use_bottleneck": true,
+        "joint_channel": 3,
+        "vqvae_reverse_decoder_dilation": true,
+        "acc": 1.0
+      },
+      "use_bottleneck": true,
+      "joint_channel": 3
+    },
+    "gpt": {
+      "block_size": 29,
+      "base": {
+        "embd_pdrop": 0.1,
+        "resid_pdrop": 0.1,
+        "attn_pdrop": 0.1,
+        "vocab_size_up": 512,
+        "vocab_size_down": 512,
+        "block_size": 29,
+        "n_layer": 6,
+        "n_head": 12,
+        "n_embd": 768,
+        "n_music": 438,
+        "n_music_emb": 768
+      },
+      "head": {
+        "embd_pdrop": 0.1,
+        "resid_pdrop": 0.1,
+        "attn_pdrop": 0.1,
+        "vocab_size": 512,
+        "block_size": 29,
+        "n_layer": 6,
+        "n_head": 12,
+        "n_embd": 768,
+        "vocab_size_up": 512,
+        "vocab_size_down": 512
+      },
+      "critic_net": {
+        "embd_pdrop": 0.0,
+        "resid_pdrop": 0.0,
+        "attn_pdrop": 0.0,
+        "block_size": 29,
+        "n_layer": 3,
+        "n_head": 12,
+        "n_embd": 768,
+        "vocab_size_up": 1,
+        "vocab_size_down": 1
+      },
+      "n_music": 438,
+      "n_music_emb": 768
+    }
+  }
+}

conversion_report.json

@@ -0,0 +1,27 @@
+{
+  "output": "/apdcephfs/AILab_DHA/apdcephfs_cq11/share_1467498/home/zeyuling/Motius/outputs/checkpoints/bailando_aistpp",
+  "load_reports": {
+    "vqvae": {
+      "missing": [],
+      "unexpected": []
+    },
+    "gpt": {
+      "missing": [],
+      "unexpected": []
+    }
+  },
+  "provenance": {
+    "source_repository": "https://github.com/lisiyao21/Bailando",
+    "source_revision": "cc90b98bff81c9709570db413c9610c2562e27ca",
+    "source_checkpoints": {
+      "vqvae": {
+        "filename": "epoch_500.pt",
+        "sha256": "35670f42a3b3092438f73f0af3ace7b52e318a8b5c00b2b05c92078176b21716"
+      },
+      "gpt": {
+        "filename": "epoch_10.pt",
+        "sha256": "903863a4e1cac01fcec30f7939c591eac8ea89f74e9837b93babe3383eecb403"
+      }
+    }
+  }
+}

gpt.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9a5efd15b79d4576ff3a0546614a19b9077248fb49fb5c9117302a1b1c40439
+size 433275812

model_index.json

@@ -0,0 +1,5 @@
+{
+  "_class_name": "BailandoPipeline",
+  "_motius_bundle": "BailandoBundle",
+  "artifact_format": "motius-bailando-v1"
+}

vqvae.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed0af826ef709824b59e3977ce54f9e61056929b8ed069aa33ce9d7b80675c58
+size 262402692