README.md

metadata

library_name: diffusers
tags:
  - modular-diffusers
  - diffusers
  - cosmos3-omni
  - text-to-image

This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

Pipeline Type: Cosmos3DistilledBlocks

Description: Modular pipeline blocks for distilled (few-step) Cosmos3 generation modes.

This pipeline uses a 4-block architecture that can be customized and extended.

Example Usage

[TODO]

Pipeline Architecture

This modular pipeline is composed of the following blocks:

1. text_encoder (Cosmos3DistilledTextEncoderStep)
   • Prepares distilled prompt token IDs. Classifier-free guidance is baked into the weights, so negative_prompt is not exposed and the unconditional branch is derived from an empty prompt.
2. vae_encoder (Cosmos3DistilledAutoVaeEncoderStep)
   • Auto VAE conditioning block for distilled Cosmos3.
3. denoise (Cosmos3DistilledVisionCoreDenoiseStep)
   • Runs the text-and-vision distilled Cosmos3 denoising workflow.
4. decode (Cosmos3VideoDecodeStep)
   • Decodes denoised vision latents into video outputs.

Model Components

1. text_tokenizer (AutoTokenizer)
2. vae (AutoencoderKLWan)
3. video_processor (VideoProcessor)
4. transformer (Cosmos3OmniTransformer)
5. scheduler (FlowMatchEulerDiscreteScheduler)

Configuration Parameters

is_distilled (default: True) distilled_sigmas (default: None)

Workflow Input Specification

text2image

• prompt (str): The text prompt that guides Cosmos3 generation.
• num_frames (int, optional): Number of frames to generate.

text2video

• prompt (str): The text prompt that guides Cosmos3 generation.

image2video

• prompt (str): The text prompt that guides Cosmos3 generation.
• image (None, optional): Reference image for image-to-video conditioning.

video2video

• prompt (str): The text prompt that guides Cosmos3 generation.
• video (None, optional): Reference video for video-to-video conditioning.

Input/Output Specification

Inputs:

• prompt (str): The text prompt that guides Cosmos3 generation.
• num_frames (int, optional): Number of frames to generate.
• height (int, optional): Height of the generated video or image in pixels.
• width (int, optional): Width of the generated video or image in pixels.
• fps (float, optional, defaults to 24.0): Frame rate of the generated video.
• use_system_prompt (bool, optional, defaults to True): Whether to prepend the Cosmos3 system prompt.
• add_resolution_template (bool, optional, defaults to True): Whether to add resolution metadata to the prompt.
• add_duration_template (bool, optional, defaults to True): Whether to add duration metadata to the prompt.
• video (None, optional): Reference video for video-to-video conditioning.
• condition_frame_indexes_vision (tuple | list, optional, defaults to (0, 1)): Latent-frame indexes to preserve from the conditioning video.
• condition_video_keep (str, optional, defaults to first): Which end of a longer conditioning video to use: first or last.
• image (None, optional): Reference image for image-to-video conditioning.
• x0_tokens_vision (Tensor, optional): Vision latents encoded from the conditioning image or video.
• vision_condition_frames (list, optional): Latent-frame indexes fixed by visual conditioning.
• latents (Tensor, optional): Pre-generated noisy vision latents.
• generator (Generator, optional): Torch generator for deterministic generation.
• num_inference_steps (int, optional): The number of denoising steps.
• guidance_scale (float, optional): Unused for distilled checkpoints; classifier-free guidance is baked into the weights and the scale is forced to 1.0. Passing a value other than 1.0 raises an error.
• **denoiser_input_fields (None, optional): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
• output_type (str, optional, defaults to pil): Output format: 'pil', 'np', 'pt'.

Outputs:

• videos (list): The generated videos.