Lite workflow to 24 fps (accents) + doc math; v1.7 zip refreshed with the fixed Lite graph

1d491ef verified 5 days ago

16.1 kB

license: other
license_name: ltx-2-community-license
license_link: https://huggingface.co/Lightricks/LTX-2/blob/main/LICENSE.txt
tags:
  - comfyui
  - workflow
  - ltx-video
  - ltx-2.3
  - text-to-video
  - talking-head
  - lip-sync
  - audio

LTX-2.3 Multishot Lite

Two-shot talking-character video on LTX-2.3 — shot 2 is a TRUE audio+video extension of shot 1 (not a cut), the character speaks in a reference voice you supply, and both shots are joined and refined into a single final file.

Needs ComfyUI, the models, and ComfyUI-KJNodes (one node powers the extension). Everything else is core.

A word on expectations. This is a bleeding-edge pipeline — a 22B audio+video model running on consumer hardware. It works, but your first clean render will likely take some tuning to YOUR machine (VRAM, system RAM, which model build). Every setting that matters is documented in note blocks inside the workflow, so the guidance travels with the graph. If you get stuck, open a discussion — I answer.

Quick fixes — read this first

Nearly every problem reported with this workflow is one of these.

symptom	what's actually wrong	fix
Second shot doesn't lip-sync — voiceover over a barely-moving face	You are on v1.x, which chained shots on shot 1's decoded last frame. That guide is pixel-continuable, so the sampler copies it instead of animating. Structural — no setting fixes it.	Update to v2.0+. Shot 2 is now a true audio+video latent extension.
`LTXVAudioVideoMask` node is missing (red)	v2 uses one node from ComfyUI-KJNodes for the extension.	Install ComfyUI-KJNodes. Everything else is core ComfyUI.
Burned-in subtitles or captions appear	The graph ships at cfg 1, and at cfg 1 negative prompts are inert on distilled models — your negative is doing nothing.	Raise cfg to about 1.3 on both `CFGGuider` nodes.
Voices come out British or Australian and IGNORE accent wording	Frame rate. LTX-2.3's joint AV prior is 24 fps-native, and this graph shipped at 25 through v2.0 — off-24 fps drifts every voice Commonwealth-ward (25 → southern British, 30 → broad Australian, A/B-verified 2026-07-30) and geometrically overrides accent prose.	Update to v2.1 (ships at 24 fps), or set every fps widget in your graph to 24 — both `LTXVConditioning`, `LTXVPreprocess`, `LTXVEmptyLatentAudio`, all `CreateVideo` nodes, and the AV-extend mask (fps 24, times 3.04 / 13.04). THEN name the accent in the positive: "in a casual American accent".
The model reads your scene description aloud	Nothing told it which words are spoken.	Keep the exclusivity block: "She is the only person speaking, and the only voice on the audio track is hers … nothing else is read aloud."
Audio and video durations drift apart after changing length	In v1 the audio latent did not follow the `length` primitive — a silent desync.	Fixed in v2. Change `length`, then set the AV-extend node's `video_end_time` / `audio_end_time` to `(length + 72) / 24`. At the shipped 241 that is 13.04; for ~20 s use length 265 and 14.04.
Length change errors out	LTX-2.3 hard constraints.	Frames must be 8n+1 (121, 241, 265, 329…) and both dimensions divisible by 32.
Output is mush, or the face wanders	Wrong checkpoint class.	This graph's 8-step sigma ladder assumes a distilled LTX-2.x checkpoint. A non-distilled one will not resolve in 8 steps.
Out of VRAM, or renders crawl	A bf16 LTX-2.3 checkpoint is ~40 GB of weights.	Pick an fp8 checkpoint in `UNETLoader` (or set `weight_dtype` to an fp8 option), and drop resolution before you drop steps.

The one people miss most: cfg 1 makes your negative prompt do nothing. If you are fighting burned-in text, that is why.

v2.1 — 24 fps: your frame rate was choosing your accents

The graph shipped at 25 fps through v2.0. It turns out LTX-2.3's joint audio-video prior is 24 fps-native and render fps is a hidden accent dial: at 25 fps the same prompt and seed render non-rhotic southern British, at 30 fps broad Australian — and any off-24 setting overrides accent wording in the prompt entirely (a few conditioning tokens cannot outvote a geometric signal present in every attention operation at every denoising step). Dose-response verified by A/B on identical configs, 2026-07-30: 24 fps = rhotic General American at full reviewer confidence.

v2.1 changes every fps-carrying widget to 24 (LTXVConditioning ×2, LTXVPreprocess, LTXVEmptyLatentAudio, CreateVideo ×3) and rescales the AV-extend mask times so the extension window stays on the same latent frames (73 / 313 → 3.04 s / 13.04 s at 24 fps). Duration math is now length / 24; the extend end-time formula is (length + 72) / 24.

Flip side, free feature: if you want an authentic British or Australian character, render their scenes at 25 or 30 fps — it is more consistent than any accent wording. And at any fps, still name the accent on every spoken line: with nothing stated, young female voices lean Australian even at 24.

v2.0 — the chained shot is now a TRUE extension (fixes chained lip sync)

If you use two shots, this release matters more than everything before it combined. v1.x's second shot was structurally broken and no setting could fix it. What we found, in code:

LTXVAddGuide does not "set frame 0" — it appends the guide as a competing token at the same timestamp and, at strength 1.0, skips its attention mask entirely. The official Lightricks workflows use 0.7 for exactly this reason.
v1 guided shot 2 with shot 1's own decoded last frame — VAE-round-tripped, at exactly the render size, pixel-continuable. Reproducing it verbatim is a cheaper solution for the sampler than lip-syncing. That is why chained shots came out as "voiceover over a barely-moving face", and why it was WORSE on strongly distilled checkpoints.
Shot 2's audio restarted from pure noise mid-sentence while its video was pinned — two contradictory anchors every render.

v2 replaces the handover with a real audio+video extension (the pattern community extend workflows and Lightricks' own hosted /extend use): the last ~3 s of shot 1 — video AND audio — are encoded as latent context, and the model generates forward from an ongoing utterance. The voice carries over by construction (the chained shot no longer needs a reference-audio node at all), identity holds through the join, and the join is a continuous take instead of a cut.

Measured on the shipped demo config: chained-shot audio/mouth correlation went from ~0.1–0.4 (v1, wildly unstable) to 0.64 — and for the first time the second shot syncs better than the first.

One dependency change: the extension node (LTXVAudioVideoMask) ships in ComfyUI-KJNodes. Everything else is still core. If you work with LTX you almost certainly have KJNodes already; "no custom nodes at all" is now "core + one node from KJNodes", and it buys working multishot.

Also in v2.0: strength 0.7 everywhere (official value — 1.0 disables the guide's attention mask), and every length-dependent value now follows the one length primitive automatically (negative indexing + clamped trims), so a 20-second render is a one-widget change.

Changes in v1.3

v1.2 gave bad tuning advice. This corrects it.

v1.2 called img_compression "the motion dial" and told you to raise it to 50-70 for more movement. Both halves were wrong, and here is the measurement:

That widget is passed straight to x264 as CRF. x264 clamps CRF at 51, so every value from 51 to 100 is byte-identical - if you ran it at 100 you were not at maximum motion, you were at CRF 51 and the rest of the slider did nothing.
At CRF 51, fine high-frequency detail drops to ~83% of source. That is where patterned fabric, curtains, hair and skin texture go: the model receives mush where the pattern was and invents a replacement. It restyles the whole frame, not just the face.

It now ships at 25 (visually near-lossless) and is documented as guide degradation, not a motion dial.

The real first question is your checkpoint. Lip sync does not live in the audio branch - it lives in the video branch's cross-attention to the audio conditioning, and video-side merges rewrite exactly those layers. Comparing one such merge against stock ltx-2.3-22b-distilled-1.1: the audio branch was bit-identical, but only 2 of 24 sampled video cross-attention tensors were unchanged. A checkpoint can be excellent at scenery and markedly worse at mouths with its audio weights untouched.

The workflow now defaults to stock ltx-2.3-22b-distilled-1.1. If your mouths are poor, A/B a stock checkpoint before touching any widget - and compare against a distilled build, never -dev, or the 8-step ladder here will produce garbage and wrongly indict it.

strength returns to 1.0. On a healthy checkpoint a fully pinned frame 0 animates fine and gives the tightest identity hold. Needing to drop below ~0.6 to get any movement is a checkpoint symptom, not a strength setting.

Changes in v1.2

Mouth motion, voice consistency, and the dials to trade them off.

LTXVPreprocess motion dial (new, per shot). LTX is trained on VIDEO frames, which always carry codec artifacts. A pristine photo is out-of-distribution as a "video frame", so the model treats it as a perfect anchor and barely animates. The guide is now round-tripped through an H.264 encode/decode. img_compression 35 by default; raise for more motion, lower (20-25) if the result looks soft or stylized.
The voice chain (new). Shot 2's voice reference is now shot 1's own generated audio, not the external clip. Two shots referencing the same clip still sample independently, so the voice drifted between them; feeding shot 1's output forward is the core-node equivalent of a memory bank.
Identity guidance is now gated to the early steps (end_percent 0.5). Guidance amplifies its effect on the WHOLE denoised tensor - audio AND video - so at full range it restyles the face while it fixes the voice. Identity is decided early; detail forms late. Gating keeps the voice lock without touching the detail passes.

All still 100% core ComfyUI nodes. A MOTION DIALS note block in the graph documents every knob and which way to turn it.

Changes in v1.1

v1.0 had a broken reference-image path — please re-download. LoadImage returns the file's original resolution, not the canvas, so the 1.2x reference zoom cropped a small patch from the top-left corner of large images instead of centring on the subject. The guide was effectively meaningless and renders behaved like text-to-video. Fixed: scale to exactly 1.2x canvas with a centre crop, then centre-crop back down.

Also in v1.1, all aimed at mouth motion and voice consistency:

Guide strength 1.0 -> 0.9. At 1.0 the first frame's noise mask is 0.0 — frame 0 is completely locked to the still, and the model struggles to animate away from a closed mouth.
Head trim: 28 frames on shot 1, 14 on shot 2 (audio trimmed to match). The opening frames morph out of the guide and are where lip sync is worst; shot 1 opens practically on the reference, so it needs more.
Identity guidance on the voice reference. Both builds now use core LTXVReferenceAudio, which patches the model with an extra forward pass without the reference and amplifies the speaker difference. The voice no longer drifts between shots.

What it does

SETUP → SHOT 1 → [last frame] → SHOT 2 → FINISH (join + refine) → FINAL

Chained shots. ImageFromBatch pulls shot 1's final frame into shot 2's LTXVAddGuide at frame 0. The shots genuinely continue each other rather than being two unrelated clips.
Mode 1 (ships enabled) — your reference voice. Point LoadAudio at 3–5 seconds of clean speech; the character speaks your prompt's quoted line in that voice.
Mode 2 — the model invents a voice. Bypass one node per shot (Ctrl+B).
One final file. Both shots joined — video and audio — then refined with a deterministic 1.5× bicubic upscale. Saved as multishot_lite/FINAL.

Quick start

Download LTX23_Multishot_Lite_v2.0.zip (or just the workflow JSON) and open it in ComfyUI.
Set the six loaders in the SETUP group — see the table in INSTRUCTIONS.md. Each reads a different models folder.
Put a 3–5 second voice clip and a first-frame image in ComfyUI/input/.
Queue. Judge multishot_lite/FINAL.

Full walkthrough, per-VRAM settings and troubleshooting: INSTRUCTIONS.md

What you need

thing	where
`ltx-2.3-22b-distilled-1.1` (or another distilled LTX-2.x build)	`models/diffusion_models/`
LTX-2.3 video VAE	`models/vae/`
LTX-2.3 audio VAE + text projection	`models/checkpoints/`
Gemma-3-12B text encoder	`models/text_encoders/`
`LTX-2.3-ID-LoRA-TalkVid-3K.safetensors`	`models/loras/` — Mode 1 needs this

A recent ComfyUI (needs ManualSigmas and ComfySwitchNode in core). Tested on an RTX 5090; smaller cards should drop resolution before anything else.

System RAM matters as much as VRAM — loading copies the whole checkpoint into host memory. On a 64 GB machine set a Windows pagefile of 64–128 GB on an SSD, or the load can be killed by the OS with no Python error.

The three things that most often go wrong

Voice reference length. The entire clip becomes conditioning tokens, so a long sample hands the model extra words to echo — you get your line plus garbled fragments. Keep the trim at 3–5 seconds of clean, solo, continuous speech. This is the single most common cause of bad audio.

Missing ID-LoRA. Reference-audio identity transfer is a trained adapter, not a base-model skill. Without it the model echoes your clip instead of adopting the voice.

The narrator prior. Only text inside "double quotes" should be spoken, but the model will happily read your scene description aloud too. The shipped prompts carry an explicit exclusivity block — "no narration, no voice-over, nothing else is read aloud". Keep it when you rewrite them.

Honest limits

There is no memory bank here. Continuity comes only from the handed-over frame, so identity drifts over many shots. A paired audio+video memory bank cannot run on a stock sampler — that is exactly why the full JoyAI-Echo multishot pack exists. Use Lite for a quick two-shot piece with no dependencies; use the pack when one character has to hold across a whole scene.

Long-form video is heavy: a 1001-frame shot at 960×544 holds tens of GB of frames in system RAM during the join. Start at the shipped defaults.

Also available

GitHub: https://github.com/jlucasmcrell/ltx23-multishot-lite
The full node pack (memory bank, many shots): https://huggingface.co/joeygambino/joyai-echo-multishot-workflow
Models: https://huggingface.co/joeygambino

Credits

Lightricks — LTX-2 / LTX-2.3; the sigma ladder and AV latent conventions here follow their reference workflow.
Google — Gemma 3 12B, the text encoder.

License

LTX-2 Community License. Gemma components are subject to the Gemma Terms of Use. AI-generated content must be disclosed as such. Not affiliated with Lightricks or Google.

Support

Everything here is free and stays free. If it saved you time, you can sponsor me on GitHub, buy me a coffee, or support me on Liberapay.