Marionette — weights

Checkpoints for Marionette: Predicting World States, Rendering Geometry, Painting Appearance (arXiv:2608.14530). Code, runnable scripts and a gallery: https://github.com/AlayaLab/Marionette. Project page: https://alayalab.github.io/Marionette/.

▶ Watch the overview video

Marionette factorises an interactive game world model into three stages. Only the first and the third carry weights; the middle one is fixed geometry.

seed pose ──▶ dynamics ──▶ 276D world state ──▶ bridge ──▶ pose-control video ──▶ observation ──▶ RGB
              ActionGPT                          zero-parameter,                  Wan2.2-Fun-5B-Control
              + PoseGPT                          deterministic                    fine-tune, chunk-relay

Files

file	size	stage	what it is
observation/diffusion_pytorch_model.safetensors	10.0 GB	observation	control-conditioned video diffusion transformer, fine-tuned from Wan2.2-Fun-5B-Control
dynamics/pose_gpt.pt	402 MB	dynamics	PoseGPT — predicts the next articulated state
dynamics/action_gpt.pt	61 MB	dynamics	ActionGPT — predicts the next action token

These are not standalone. The observation stage loads the VAE and the umT5-xxl text encoder from the third-party base release, which is not redistributed here:

bash fetch_base_model.sh          # alibaba-pai/Wan2.2-Fun-5B-Control, ~23 GB, Apache-2.0

Seeds, the scanned terrain and the appearance reference are in the code repository, not here — they pack small enough that a separate download would buy nothing.

Use

git clone https://github.com/AlayaLab/Marionette && cd Marionette
bash fetch_weights.sh                      # the weights in this repo
bash fetch_base_model.sh                   # the third-party base model
bash run_demo.sh                           # -> samples_out/.../rollout.mp4

Inference settings the released checkpoint was evaluated at: 704×1280, 40 steps, guidance 6.0, 81-frame chunks, 30 fps.

The prompt is part of the checkpoint

The observation model was trained on captions carrying an explicit appearance id:

Monster Hunter Wilds video game gameplay, stage 101, hunter appearance id 9 wielding weapon type 4, fighting monster id 19.

Pairing this checkpoint with an older caption format — or an older checkpoint with this one — puts the text encoder out of distribution, and the failure is not subtle: monsters drop out of the rollout entirely. Change the checkpoint and the prompt together or neither.

Training

Observation. Fine-tuned from alibaba-pai/Wan2.2-Fun-5B-Control in control_ref mode with the first frame as the appearance reference, at 704×1280, 16×H200 FSDP. The released checkpoint is step 17000. The last 2000 of those steps are on a corpus recorded with a revised capture path in which objects absent from the pose-control signal are also absent from the RGB — mounts in particular — which removes a class of targets the model previously had no way to predict.

Earlier stages of training span 26 monster species; the final segment is a single configuration (one stage, one monster, one weapon class). The checkpoint is correspondingly strongest there, and the multi-species montage on the project page is rendered from an earlier checkpoint for exactly this reason.

Dynamics. ActionGPT and PoseGPT over the 276D articulated state at 20 fps, conditioned on a scanned terrain height field, trained on a single monster species (em19) over 2241 segments. Action vocabularies: 168 for the monster, 977 for the hunter.

Action ids are indices into this checkpoint's vocabulary and are not portable. The vocabulary is built from the training corpus, so the same integer under a model trained on a different corpus is a different animation or nothing at all — an earlier corpus here shares almost none of this one's hunter ids. When the demos name 493 as ATTACK, that is a fact about this checkpoint.

Reproducibility

The action stream is sampled, so the dynamics stage is only reproducible with a pinned seed. run_stage1_render.sh pins TORCH_SEED=43; upstream leaves it unset. With the seed pinned, the repository's verify_reproduction.sh reproduces its recorded pose video byte for byte and its RGB rollout pixel for pixel.

Scope and limitations

• The dynamics model covers one monster species. It is not a general character-motion model.
• Rollout is chunk-relay: each 81-frame chunk is conditioned on the last frame of the previous one, so appearance error compounds with horizon.
• The bridge consumes a terrain height field scanned for specific stages. Novel geometry needs a new scan.
• Research artifact. Not a product, not a game, not a renderer for anything but this pipeline.

Licence and attribution

• Our code: Apache-2.0 (see the code repository's LICENSE).
• These weights: non-commercial research use only, no redistribution — see LICENSE.assets. The restriction comes from the training corpus, not from the base model.
• Base model alibaba-pai/Wan2.2-Fun-5B-Control: Apache-2.0, not redistributed here.
• VideoX-Fun (aigc-apps/VideoX-Fun), which the observation code is vendored from: Apache-2.0.
• Training corpus: the WildWorld dataset — https://github.com/AlayaLab/WildWorld — released for non-commercial research use, redistribution not permitted.

Rights in the game content the corpus was recorded from remain with its publisher, and this project grants no rights in that content.

Two files in this project are unmodified recorded gameplay video, not model output: data/first_frame_ref.mp4 and data/demo/aligned_ref.mp4. The observation model is conditioned on a reference frame, and these supply it. They are short clips from the same recordings the corpus was built from, and they are covered by the research-only terms above. Everything else distributed here — seed states, scanned terrain, pose videos, rollouts and the gallery stills — is numeric derivation or model output.

Citation

@article{meng2026marionette,
  title   = {Marionette: Predicting World States, Rendering Geometry, Painting Appearance},
  author  = {Meng, Zian and Li, Zhen and Li, Chuanhao and Li, Qiang and Zhang, Kaipeng},
  journal = {arXiv preprint arXiv:2608.14530},
  year    = {2026}
}