README.md

metadata

title: MMDiff
emoji: 🎨
colorFrom: blue
colorTo: yellow
sdk: gradio
sdk_version: 5.23.0
app_file: app.py
short_description: Multi-modal generation with diffusion transformers
python_version: '3.10'
startup_duration_timeout: 10m

MMDiff: Extending Diffusion Transformers for Multi-Modal Generation

This Space demonstrates MMDiff, a method that extends frozen diffusion transformers (FLUX.1-dev) to generate images alongside dense predictions (saliency maps, segmentation maps, depth maps) in a single forward pass.

How it works

1. A text prompt is used to generate an image with FLUX.1-dev
2. During denoising, intermediate transformer features and concept attention maps are captured
3. Lightweight trained decoder heads (DPT, DeepLabV3+) decode these features into dense predictions:
   • Saliency (DUTS): Binary foreground/background segmentation
   • Segmentation (Pascal VOC): 21-class semantic segmentation
   • Depth (NYU Depth V2): Monocular depth estimation

Model

• Backbone: black-forest-labs/FLUX.1-dev (frozen)
• Decoder weights: yagmurakarken/mmdiff
• Paper: MMDiff: Extending Diffusion Transformers for Multi-Modal Generation