--- base_model: - Lightricks/LTX-2.3 base_model_relation: adapter license: other license_name: ltx-2-community-license license_link: https://github.com/Lightricks/LTX-2/blob/main/LICENSE language: - en tags: - lora - ltx-2.3 - foley - video-to-audio - text-to-audio - audio pipeline_tag: text-to-audio extra_gated_description: By clicking "Agree and Access" you acknowledge the [Privacy Policy](https://static.lightricks.com/legal/Privacy%20Policy%20-%20LTX%20Platform.pdf) and consent to receive offers and updates. You can unsubscribe at any time. extra_gated_button_content: Agree and Access widget: - text: A heavy steel hammer repeatedly strikes glowing iron against a solid steel anvil, each blow landing as a hard metallic clang with a short resonant ring, brief spark spits, and faint forge-room ambience. No speech is present. No music is present. output: url: examples/01_blacksmith_sbs.mp4 - text: Fingers type continuously on low-profile plastic laptop keys, producing rapid light clicks, muted key taps, and occasional soft contact on the aluminum palm rest in a quiet indoor room. No speech is present. No music is present. output: url: examples/02_typing_sbs.mp4 - text: A fighter delivers quick punches and forceful kicks into focus pads, producing tight padded smacks and deeper leather thuds in sequence, with fast foot pivots and soft scuffs on the gym floor. No speech is present. No music is present. output: url: examples/03_focus_pads_sbs.mp4 - text: Shoes walk along weathered wooden boardwalk planks, producing firm hollow wooden footfalls, subtle plank creaks, and light sole scuffs, receding into a quiet forest. No speech is present. No music is present. output: url: examples/04_boardwalk_footsteps_sbs.mp4 - text: Heavy boots step through thin snow and wet icy ground, producing soft snow compression, damp slush squelches, small ice-crust crunches, and muddy sole suction at close range. No speech is present. No music is present. output: url: examples/05_wet_slush_sbs.mp4 - text: Motorcycles ride quickly along smooth mountain asphalt, their engines producing strong throaty revs and rising mechanical roars with tire hiss and steady wind rushing past. No speech is present. No music is present. output: url: examples/06_motorcycles_sbs.mp4 --- # LTX-2.3 Foley V2A LoRA Video-to-audio Foley LoRA for LTX-2.3-22B: generates synchronized, music-free sound effects from silent video (and works for text-to-audio Foley prompts). It is based on the [LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) foundation model. ## Example Outputs The gallery above shows muted → foley comparison examples generated with this LoRA on IP-free stock clips (each panel plays the original silent clip against the same clip with generated Foley). ## Model Files `ltx-2.3-22b-lora-foley-v2a-1.0.safetensors` Recommended checkpoint: **step 500** (selected by validation loudness; later steps tend to over-suppress volume). ## Model Details - **Base Model:** LTX-2.3-22B Video - **Training Type:** LoRA - **Modality:** video-to-audio (primary), text-to-audio - **Recommended checkpoint:** step 500 (`ltx-2.3-22b-lora-foley-v2a-1.0.safetensors`) - **Audio:** Trained for synchronized Foley / SFX audio generation (no speech, no music bed) ## Intended Use & Out-of-Scope **Intended use:** Add or regenerate Foley for silent or muted clips — footsteps, impacts, materials, ambience, tools, kitchen, weather — with prompts that describe on-screen action. Research and creative tooling on top of LTX-2.3. **Out of scope:** Dialogue / speech synthesis, music generation, lip-sync performance, or replacing a full production mix. Not a substitute for licensed commercial SFX libraries when rights matter. ## How It Works The adapter targets the **audio attention / FFN blocks** and **video→audio cross-attention**, so the model keeps the input video and synthesizes a matching audio track. Training used a flexible V2A strategy (video fixed, audio generated) on cleaned Foley clips with strong music/speech suppression in captions and negatives. Prefer **mid-training checkpoints** (here step 500): longer training improved music rejection but often crushed loudness. ## Usage ### ComfyUI 1. Copy `ltx-2.3-22b-lora-foley-v2a-1.0.safetensors` into `models/loras`. 2. Load the **LTX-2.3-22B** base model. 3. Add the Foley LoRA. 4. For **text-to-audio**, use the [T2A workflow](https://github.com/Lightricks/ComfyUI-LTXVideo/blob/master/example_workflows/2.3/LTX-2.3_T2A_Single_Stage_Distilled.json) from [ComfyUI-LTXVideo](https://github.com/Lightricks/ComfyUI-LTXVideo/). 5. For **video-to-audio** (primary), use the dedicated [Foley V2A workflow](ltx-2.3-foley-v2a.json): it encodes and **freezes** the input video and denoises only the audio latent, so the output is the original video frames plus generated Foley. It requires the `ComfyUI-LTXVideo-Internal` audio nodes (`LTXVConcatAVLatent`, `LTXVSetAudioVideoMaskByTime`, `MultimodalGuider`, `LTXVAudioVAEDecode`, …) plus `ComfyUI-VideoHelperSuite`. See Pipeline Details. 6. Start with LoRA strength **0.8–1.0** and adjust per clip. ## Pipeline Details **Primary task is V2A Foley:** mute or strip the source audio, feed the video as conditioning, generate audio with this LoRA. Recommended inference recipe used for the demos: - Guidance scale **6.0** (guidance ~4 often collapses to near-silence on some seeds) - **30** inference steps - STG: scale **1.0**, block **[29]**, mode **`stg_av`** - Resolution / length bucket used in training & demos: **960×544**, **89** frames @ **24 fps** (~3.7 s) - If a result is silent or sounds materially incorrect, try 2–3 generations and choose the one that best matches the visible material and timing—not simply the loudest. ## Recommended Settings - **LoRA strength / weight:** `0.8–1.0` (demos used full strength) - **Inference steps:** `30` - **Guidance scale:** `6.0` (prefer 6–8 over 4 for audible Foley) - **Resolution & length:** Use 960×544 video at 24 fps. The main training length is approximately 3.7 seconds, and the provided workflow supports clips up to approximately 7 seconds. - **Spatial / audio guidance (STG):** scale `1.0`, blocks `[29]`, mode `stg_av` - **Prompting:** Describe the visible sound source, action, contact material, timing, and physical sound character. Avoid using the same generic crackle/crinkle wording for unrelated materials. End with `No speech is present. No music is present.` when you want clean SFX-only audio. - **Multi-seed fallback:** If output is silent or sounds materially wrong, change seed and try 2–3 generations, choosing the one that best matches the visible material and timing—not simply the loudest. - **Checkpoint choice:** Prefer step **500** over later steps for volume; later steps can over-suppress. Example positive prompt: ```text A sledgehammer smashes a glass bottle: a heavy whoosh, a sharp glass shatter and tinkling shards. No speech is present. No music is present. ``` Example negative prompt: ```text music, melody, song, singing, vocals, score, soundtrack, beat, rhythm bed, instrumental backing, speech, dialogue, talking, narration, tinny, thin, harsh, clipped, distorted, low bitrate ``` ## References - **Code:** [GitHub Repository](https://github.com/Lightricks/LTX-2) - **ComfyUI:** [ComfyUI-LTXVideo](https://github.com/Lightricks/ComfyUI-LTXVideo/) ## Tips & Troubleshooting - **Near-silent output:** Raise guidance to 6+, try a new seed, and confirm you are on the step-500 weights (not a late over-suppressed checkpoint). - **Music bleed:** Keep the music/speech negative prompt; reinforce `No music is present` / `No speech is present` in the positive prompt. - **Weak ambience (rain, fire, pottery, typing):** These are seed-sensitive — multi-seed and pick the best; slightly more assertive “loud / close-mic / prominent” wording helps. - **Sync:** Prompt the action you can see (impacts, footsteps, materials). Mismatched prompts produce plausible but unsynced SFX. ## Dataset **\~5,374** short clips (\~3.7 s, 960×544 @ 24 fps): - Curated / cleaned Foley rebuild set (Demucs + HPSS filtering, silence/music rejection, captions) - **[FoleyBench](https://huggingface.co/datasets/FoleyBench/foleybench)** — 5k non-speech / non-music clips (CC BY-NC-SA; academic / non-commercial terms apply to that portion) Captions emphasize on-screen Foley and explicitly exclude speech and music. ## Training - **Technique:** LoRA on audio self-attn, audio cross-attn, video-to-audio attn, and audio FFN; flexible V2A strategy (video latents fixed, audio generated). Warm-started from an earlier Foley LoRA checkpoint. - **Hyperparameters:** rank 32, alpha 32, lr `1e-5`, AdamW, linear schedule (`start_factor` 0.1 → `end_factor` 0.05), batch size 1, grad accumulation 2, `max_grad_norm` 0.5, bf16, gradient checkpointing - **Steps:** 3000 (best validation loudness at **step 500**) - **Infrastructure:** LTX-2 Community Trainer (single-GPU run after multi-GPU NaN instability) ## License See the **[LTX-2-community-license](https://github.com/Lightricks/LTX-2/blob/main/LICENSE)** for full terms. Dataset components may carry additional terms (e.g. FoleyBench CC BY-NC-SA). ## Acknowledgments - Base model by **[Lightricks](https://ltx.io/)** - Training infrastructure: **[LTX-2 Community Trainer](https://github.com/Lightricks/LTX-2/tree/main/packages/ltx-trainer)** - Demo video sources: [Mixkit](https://mixkit.co/) Free License stock (no attribution required)