README.md

---
title: JoyAI Video Edit
emoji: 🎬
colorFrom: indigo
colorTo: gray
sdk: gradio
sdk_version: 6.22.0
app_file: app.py
short_description: Streaming instruction-guided video editing
python_version: "3.12"
startup_duration_timeout: 1h
license: apache-2.0
tags:
  - video-editing
  - video-to-video
  - streaming
  - diffusion
models:
  - jdopensource/JoyAI-Video-Edit
  - XiaomiMiMo/MiMo-VL-7B-RL-2508
---

# JoyAI-Video-Edit — streaming video editing

Instruction-guided video editing with [`jdopensource/JoyAI-Video-Edit`](https://huggingface.co/jdopensource/JoyAI-Video-Edit).
Describe an edit in plain language and it is applied to your clip **chunk by chunk**: a
16.3B chunk-causal MMDiT edits 8 frames at a time behind a rolling KV cache, so the
result streams back while it is still being generated.

## How it works

| stage | model |
|---|---|
| condition encoder | `XiaomiMiMo/MiMo-VL-7B-RL-2508` (instruction + anchor frame → 1024 conditioning tokens) |
| editor | 40-block chunk-causal MMDiT, 16.3B params, source-id RoPE, 2 distilled flow-matching steps per chunk |
| video codec | `XVAEChunkCausal` — 16× spatial / 8× temporal causal video VAE with a 1.5× refinement head |

Per 8-frame chunk: VAE-encode the source window → 2 DiT denoise passes over a
`[chunk 0, chunk n-1, chunk n]` attention window → one extra "clean KV" pass that writes
the finished chunk into the cache → VAE-decode (seeded with a pseudo-latent
re-encoded from the previous chunk's last frame) → evict stale cache entries.

## Deviations from the reference deployment

Ported from [`jd-opensource/JoyAI-Video-Edit`](https://github.com/jd-opensource/JoyAI-Video-Edit)
(`deploy/`). The tensor pipeline — chunk windows, cache ids/eviction, temporal-id RoPE,
the pseudo-latent decode trick, scheduler shift, prompt template — matches the reference.
What changed, and why:

- **FlashAttention-4 → SDPA.** FA4/FA3 kernels do not exist for the sm_120 GPU this Space
  runs on; PyTorch SDPA (flash backend) is used with identical, unmasked attention.
- **`sgl-kernel` fused ops → pure PyTorch.** The fused layernorm-modulate / QK-norm-RoPE /
  add-gate kernels are reimplemented with the same math.
- **`sgl-kernel` FP8 GEMMs → `torchao`.** The reference already serves the DiT image stream
  in FP8 (`JOYOMNI_FP8_IMG=1`); here both streams use per-channel FP8 weights with
  per-token dynamic activation scaling so the whole system fits one 48 GB GPU slice.
  The MLLM condition encoder is int8 weight-only.
- **`torch.compile` disabled.** JIT inductor compilation cannot run inside a ZeroGPU worker.
- **Threads → sequential.** The reference runs five pipeline-parallel worker threads feeding
  a WebSocket downlink; a Space request gets one GPU slice, so the stages run in order and
  frames are streamed back per chunk instead.
- **Face-restoration gating and the ONNX detector are not included.**

Sources default to the reference server's settings: 720×1248 (or 1248×720 portrait),
24 fps, 2 steps, seed 42.

Showcase clips under `examples/` are the source videos from the reference repository's
README, with the authors' prompts.