---
license: mit
library_name: diffusers
pipeline_tag: image-to-image
tags:
- video-vae
- vae
- video-generation
- tokenizer
- pytorch
---
# Motif VAE
Motif VAE is a 3D causal video VAE with 4x temporal and 32x spatial compression and a
128-channel latent space, built as a high-compression tokenizer for latent video diffusion
models. A 256x256x17-frame clip encodes to a `(128, 5, 8, 8)` latent, which is 1/16 the
spatio-temporal tokens of an 8x-spatial VAE, so downstream diffusion-transformer training and
inference cost less. The decoder is much larger than the encoder (about 4:1 in parameters),
which helps it keep fine detail such as small text in documents.
It also runs as a plain image VAE at `T=1`. In its compression class it has the best document
text fidelity, video reconstruction, and diffusability (how well a diffusion model converges on
its latents) of the tokenizers we compared.
## Architecture
| | |
|---|---|
| Type | 3D causal VAE, attention-free (pure convolution) |
| Compression | temporal 4x (T = 4n+1 grammar) / spatial 32x32 |
| Latent | 128 channels, deterministic encoder |
| Parameters | 1.244B total: encoder 242M, decoder 1,002M (1:4.15) |
| Encoder | 5 down stages (3x spatial-only, 2x spatio-temporal), 2 resblocks/stage, parameter-free global skip connections |
| Decoder | 1.5x channel width, 3 resblocks/stage, sub-pixel (PixelShuffle + ICNR) spatial upsampling, trilinear temporal upsampling |
| Norm / act | LayerNorm / SiLU |
## Usage
The model code ships inside this repository (`modeling_motifvae.py`) and loads through the
stock diffusers `AutoModel` — no extra package beyond `torch`, `diffusers`, `einops`:
```python
import torch
from diffusers import AutoModel
vae = AutoModel.from_pretrained("Motif-Technologies/Motif-VAE", trust_remote_code=True)
vae = vae.to("cuda", dtype=torch.bfloat16).eval()
# video: (B, 3, T, H, W) in [-1, 1], T = 4n+1 (1, 5, 9, 13, 17, ...),
# H/W multiples of 32. T=1 works as a plain image VAE.
x = torch.randn(1, 3, 17, 256, 256, device="cuda", dtype=torch.bfloat16)
with torch.no_grad():
z = vae.encode(x).latent_dist.sample() # (1, 128, 5, 8, 8)
rec = vae.decode(z).sample # (1, 3, 17, 256, 256)
```
For diffusion training, whiten the latent to roughly unit variance with the per-channel
statistics shipped in the config — `shift` is the channel mean and `scale` the channel std
(128 values each, measured over Kinetics-400 video (256x256x17) and ImageNet-1K images (256x256)).
Apply per channel, reshaping both to `(1, 128, 1, 1, 1)`:
```python
z = vae.encode(x).latent_dist.sample()
shift = torch.tensor(vae.config.shift, device=z.device).view(1, -1, 1, 1, 1)
scale = torch.tensor(vae.config.scale, device=z.device).view(1, -1, 1, 1, 1)
z = (z - shift) / scale # whiten for the diffusion model
# ... sample/denoise in this space, then invert before decoding:
z = z * scale + shift
rec = vae.decode(z).sample
```
Recompute these statistics on your own data if its distribution differs from natural video.
## Evaluation
Baseline numbers are either the published values cited in each table or measured by us under
the same protocol. MotifVAE is a **32x spatial** tokenizer, so the comparison is against the
other 32x tokenizers (LTX-Video, LTX-2, and the f32 image VAEs below); the lower-compression
16x Wan2.2 appears only as a reference upper bound.
### Image reconstruction — ImageNet-1K val (256x256) & FFHQ (1024x1024)
All models below are **32x (f32) image VAEs**. MotifVAE is measured by us with per-image
PSNR / SSIM (the published-table convention); baseline rows are the published numbers from the
**Qwen-Image-VAE-2.0** technical report (Table 2, same ImageNet-256 / FFHQ protocol). Our own
runs of the baselines reproduce those values to within 0.2 dB, so the comparison is direct.
| Model | Setting | ImageNet PSNR ↑ | SSIM ↑ | FFHQ PSNR ↑ | SSIM ↑ |
|---|---|---|---|---|---|
| **MotifVAE** | **f32c128** | **30.27** | **0.852** | **36.17** | **0.921** |
| Qwen-Image-VAE-2.0 | f32c128 | 29.69 | 0.842 | 35.91 | 0.918 |
| LTX-Video | f32c128 | 29.57 | 0.833 | 35.56 | 0.905 |
| HunyuanImage-2.1 | f32c64 | 28.67 | 0.820 | 35.30 | 0.911 |
| LTX-2 | f32c128 | 26.06 | 0.793 | 33.63 | 0.906 |
| DC-AE (Sana) | f32c32 | 24.82 | 0.690 | 31.35 | 0.830 |
At the f32c128 channel budget MotifVAE leads both benchmarks among 32x image VAEs, above
Qwen-Image-VAE-2.0's f32c128 (29.69 / 35.91 dB), LTX-Video, LTX-2, and the lighter f32c64 /
f32c32 models.
### Qualitative comparison (real photographs)
GT vs frozen-tokenizer reconstruction on ImageNet-1K val photographs; the red box marks the
zoom strip shown below each row.
These rows are high-frequency patterns that high-compression VAEs find hard: an armadillo's
banded shell, a dense solar-panel grid, brickwork, and vehicle markings. LTX-Video and LTX-2
(same 32x class) lose the pattern, and the solar grid collapses for them entirely. MotifVAE
keeps it and stays close to the 16x Wan2.2 reference, which also struggles on some of these.
### Document text fidelity — OmniDoc-TokenBench (256x256)
f32-compression VAEs on OmniDoc-TokenBench (~3K text-rich documents). Baseline rows are the
published numbers from the **Qwen-Image-VAE-2.0** technical report (Table 3, same benchmark
/ resolution / OCR-NED protocol); sorted by NED.
| Model | Setting | SSIM ↑ | PSNR ↑ | LPIPS ↓ | FID ↓ | OCR-NED ↑ |
|---|---|---|---|---|---|---|
| **MotifVAE** | **f32c128** | **0.849** | **22.40** | **0.052** | **3.43** | **0.825** |
| Qwen-Image-VAE-2.0 | f32c128 | 0.844 | 22.13 | 0.064 | 3.36 | 0.707 |
| LTX-Video | f32c128 | 0.806 | 20.92 | 0.119 | 17.10 | 0.565 |
| HunyuanImage-2.1 | f32c64 | 0.781 | 19.85 | 0.096 | 5.19 | 0.490 |
| LTX-2 | f32c128 | 0.735 | 18.41 | 0.119 | 9.94 | 0.357 |
| DC-AE (Sana) | f32c32 | 0.526 | 15.62 | 0.144 | 7.26 | 0.069 |
MotifVAE has the highest OCR-NED of any f32 VAE here: +0.12 over Qwen's f32c128 (same channel
budget) and about 2.3x that of LTX-Video / LTX-2. It also leads every pixel metric in its class.
The figure below covers English print, Korean print, and Korean handwriting; each row shows the
full page and a zoom-in on the red-boxed region. The top two rows are OmniDoc-TokenBench
(English); the bottom two are Korean printed text and handwriting from
[AI-Hub](https://aihub.or.kr/) — Korea's national AI-data hub (operated by NIA, the National
Information Society Agency), which publishes large-scale Korean AI training datasets (the table
metrics above are OmniDoc-only):
### Video reconstruction — Kinetics-400 (256x256) & OpenVid-1M (720p)
The same four frozen tokenizers on both benchmarks, under one pipeline: the three 32x spatial
tokenizers (MotifVAE, LTX-Video, LTX-2) and the 16x Wan2.2 as a reference.
**Kinetics-400 val — 17x256x256**
| Model | compression | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| **MotifVAE** | **4x32x32** | **35.37** | **0.954** | **0.046** |
| LTX-Video | 8x32x32 | 32.58 | 0.919 | 0.101 |
| LTX-2 | 8x32x32 | 28.05 | 0.888 | 0.128 |
| Wan2.2 | 4x16x16 | 37.41 | 0.966 | 0.034 |
**OpenVid val-1000 — native 720p, full-frame**
| Model | compression | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| **MotifVAE** | **4x32x32** | **36.96** | **0.953** | **0.040** |
| LTX-Video | 8x32x32 | 34.40 | 0.925 | 0.055 |
| LTX-2 | 8x32x32 | 32.94 | 0.923 | 0.044 |
| Wan2.2 | 4x16x16 | 38.40 | 0.964 | 0.018 |
Among the 32x spatial tokenizers MotifVAE leads both tables: +2.7 dB over LTX-Video and
+7.3 dB over LTX-2 on K400, and +2.6 / +4.0 dB on 720p. The 16x Wan2.2 reference is a little
ahead; the gap to the other 32x VAEs is largest on high-motion clips (examples below).
> **Protocol.** K400: 256 random val clips. OpenVid: the pinned public
> [`Dev-Jahn/OpenVid-1M-wds`](https://huggingface.co/datasets/Dev-Jahn/OpenVid-1M-wds) val split
> (1000 clips), each resized to height 720 keeping aspect (full-frame). Both use 17 frames/clip
> and report per-frame-averaged PSNR / SSIM / LPIPS under one pipeline; OpenVid LPIPS uses AlexNet.
> Absolute values aren't comparable across papers.
### Video reconstruction examples
Frozen-tokenizer reconstruction on high-motion Kinetics-400 clips. Each clip is one strip:
Input / MotifVAE / LTX-Video / LTX-2 / Wan2.2 (labelled). MotifVAE keeps the fast motion close
to the 16x Wan2.2; LTX-Video and LTX-2 blur it.
### Diffusability — UCF101 class-conditional generation
Latte-XL/1 (patch-1, rectified flow, 250K steps, no-CFG FVD-2048) over each frozen
tokenizer, under the Latte-official **stride-3** protocol. The Wan2.2 control (FVD 204) is
in line with values reported for this setting in the literature, confirming the harness
reproduces the expected scale.
| Tokenizer | FVD @50K | FVD @100K | FVD @250K |
|---|---|---|---|
| **MotifVAE** (4x32x32) | **636** | **385** | **276** |
| LTX-2 (8x32x32) | 952 | 556 | 450 |
| LTX-Video (8x32x32) | 1251 | 732 | 593 |
| Wan2.2 (4x16x16) | 656 | 304 | 204 |
Among the 32x spatial tokenizers MotifVAE leads at every milestone, finishing 39% below LTX-2
and 53% below LTX-Video. It reaches LTX-2's final 250K FVD by about 75K steps (~3x faster) and
stays close to the 16x Wan2.2 reference. Reconstruction quality and diffusability are separate
properties: LTX-Video reconstructs better than LTX-2 but is the worst of the three for diffusion.
## License
MIT