File size: 4,398 Bytes
80d669d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 | ---
library_name: diffusers
tags:
- modular-diffusers
- diffusers
- cosmos3-omni
- text-to-image
---
This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.
**Pipeline Type**: Cosmos3DistilledBlocks
**Description**: Modular pipeline blocks for distilled (few-step) Cosmos3 generation modes.
This pipeline uses a 4-block architecture that can be customized and extended.
## Example Usage
[TODO]
## Pipeline Architecture
This modular pipeline is composed of the following blocks:
1. **text_encoder** (`Cosmos3DistilledTextEncoderStep`)
- Prepares distilled prompt token IDs. Classifier-free guidance is baked into the weights, so `negative_prompt` is not exposed and the unconditional branch is derived from an empty prompt.
2. **vae_encoder** (`Cosmos3DistilledAutoVaeEncoderStep`)
- Auto VAE conditioning block for distilled Cosmos3.
3. **denoise** (`Cosmos3DistilledVisionCoreDenoiseStep`)
- Runs the text-and-vision distilled Cosmos3 denoising workflow.
4. **decode** (`Cosmos3VideoDecodeStep`)
- Decodes denoised vision latents into video outputs.
## Model Components
1. text_tokenizer (`AutoTokenizer`)
2. vae (`AutoencoderKLWan`)
3. video_processor (`VideoProcessor`)
4. transformer (`Cosmos3OmniTransformer`)
5. scheduler (`FlowMatchEulerDiscreteScheduler`)
## Configuration Parameters
is_distilled (default: True)
distilled_sigmas (default: None)
## Workflow Input Specification
<details>
<summary><strong>text2image</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `num_frames` (`int`, *optional*): Number of frames to generate.
</details>
<details>
<summary><strong>text2video</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
</details>
<details>
<summary><strong>image2video</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `image` (`None`, *optional*): Reference image for image-to-video conditioning.
</details>
<details>
<summary><strong>video2video</strong></summary>
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `video` (`None`, *optional*): Reference video for video-to-video conditioning.
</details>
## Input/Output Specification
**Inputs:**
- `prompt` (`str`): The text prompt that guides Cosmos3 generation.
- `num_frames` (`int`, *optional*): Number of frames to generate.
- `height` (`int`, *optional*): Height of the generated video or image in pixels.
- `width` (`int`, *optional*): Width of the generated video or image in pixels.
- `fps` (`float`, *optional*, defaults to `24.0`): Frame rate of the generated video.
- `use_system_prompt` (`bool`, *optional*, defaults to `True`): Whether to prepend the Cosmos3 system prompt.
- `add_resolution_template` (`bool`, *optional*, defaults to `True`): Whether to add resolution metadata to the prompt.
- `add_duration_template` (`bool`, *optional*, defaults to `True`): Whether to add duration metadata to the prompt.
- `video` (`None`, *optional*): Reference video for video-to-video conditioning.
- `condition_frame_indexes_vision` (`tuple | list`, *optional*, defaults to `(0, 1)`): Latent-frame indexes to preserve from the conditioning video.
- `condition_video_keep` (`str`, *optional*, defaults to `first`): Which end of a longer conditioning video to use: `first` or `last`.
- `image` (`None`, *optional*): Reference image for image-to-video conditioning.
- `x0_tokens_vision` (`Tensor`, *optional*): Vision latents encoded from the conditioning image or video.
- `vision_condition_frames` (`list`, *optional*): Latent-frame indexes fixed by visual conditioning.
- `latents` (`Tensor`, *optional*): Pre-generated noisy vision latents.
- `generator` (`Generator`, *optional*): Torch generator for deterministic generation.
- `num_inference_steps` (`int`, *optional*): The number of denoising steps.
- `guidance_scale` (`float`, *optional*): Unused for distilled checkpoints; classifier-free guidance is baked into the weights and the scale is forced to 1.0. Passing a value other than 1.0 raises an error.
- `**denoiser_input_fields` (`None`, *optional*): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
- `output_type` (`str`, *optional*, defaults to `pil`): Output format: 'pil', 'np', 'pt'.
**Outputs:**
- `videos` (`list`): The generated videos.
|