---
license: other
license_name: ltx-2-community-license
license_link: https://huggingface.co/Lightricks/LTX-2/blob/main/LICENSE.txt
tags:
- comfyui
- workflow
- joyai-echo
- ltx-video
- multishot
- video
- text-to-video
- audio
- patch
---
# JoyAI-Echo Multishot — ComfyUI workflow + node patch
**A ready-to-run ComfyUI workflow for multi-shot talking-character video, plus
the node fixes that make it work.** One recurring character across completely
different locations, with the same face and the same voice — audio generated
together with the picture, not dubbed on afterward.
## Demo
*ONE ROOM - one location, five takes, rendered end-to-end with this workflow and node patch on the Q8_0 GGUF of the surgical merge. AI-generated video and audio.*
## The workflow
`workflow/JoyEcho_Multishot_Workflow_PUBLIC.json` is a complete, grouped and
annotated graph: prompt source -> optional LLM enhance -> encode -> generate
with the cross-shot memory bank -> upscale + audio-synced finish -> save.
In-graph note blocks explain which widgets to set, which to leave alone, and
what to change on bigger cards.
Setup is two steps:
1. Pick your Gemma-3-12B text encoder in the Model Loader's `gemma_file`
dropdown (single-file `.safetensors` or `.gguf`, from `models/text_encoders`
or `models/clip`).
2. Drop `workflow/example_multishot.json` into
`ComfyUI/input/joyecho_prompts/`, pick it in the Prompt Source node, queue.
Defaults are conservative for a 24 GB card. `workflow/README.md` has the full
setup, the prompt-writing rules that make identity hold (repeat the character's
identity sentence **byte-identical** in every shot), and performance notes.
The checkpoint it drives is the echoVid+ltxAud surgical merge
(fp8 23.4 GB / bf16 43 GB):
https://huggingface.co/joeygambino/ltx23-echoVid-ltxAud-surgical
## Under the hood: the node patch
The rest of this repo is a set of bug fixes and features layered on top of the
community `ComfyUI_JoyAI_Echo_GGUF_Nodes` pack (the Rebels GGUF loader stack
around JoyAI-Echo). Everything targets the **multi-shot** path
(`JoyEcho_Generate` + the discrete Rebels loaders / `JoyEcho_ModelLoader`).
The workflow above requires these files.
This is a **patch drop**, not a standalone pack: copy these files over a working
install of the same pack (back up first). The files are interdependent — in
particular `nodes.py` calls new signatures added to the two `libs/` files, so
apply them together.
Tested on an RTX 5090 (32 GB) and a 3090 (24 GB), ComfyUI 0.26–0.27,
torch 2.8–2.11, with the JoyAI-Echo bf16 release and self-built Q8 GGUFs.
---
## Files in this package
```
nodes.py # JoyEcho_TextEncode / _Generate / _ModelLoader / _LLMEnhance
__init__.py # registrations for the new nodes
rebels_loaders.py # discrete GGUF loaders (text-encoder fixes)
joyecho_prompt_source.py (new node) # one dropdown: .txt briefs + .json scripts
joyecho_ref_picker.py (new node) # auto reference-image picker by character name
joyecho_ref_batch.py (new node) # None-tolerant image batcher
joyecho_script_picker.py (new node) # JSON dropdown (superseded by PromptSource)
libs/ltx_distillation/models/ltx_wrapper.py # fp8 quantization passthrough
libs/ltx_core/loader/fuse_loras.py # kohya-LoRA fusion + alpha scaling + fuse telemetry
libs/ltx_core/quantization/policy.py # fp8_scaled_mm_torch policy (+ sm_89 gate)
libs/ltx_core/quantization/fp8_torch_mm.py # native-fp8 Linear forward (torch._scaled_mm)
libs/ltx_distillation/utils.py # tiled VAE decode
libs/ltx_distillation/inference/memory_multishot.py # memory bank TRIM FIX (critical)
libs/ltx_distillation/inference/bidirectional_pipeline.py # dtype hardening vs fp8 params
libs/ltx_distillation/inference/memory_bidirectional_pipeline.py # dtype hardening vs fp8 params
prompts/long_story_writer_system_prompt.md # (optional) de-musicked + character-age edits
```
The files are interdependent - apply the whole set together, never cherry-pick
(a nodes.py newer than its libs/ raises AttributeError at load).
---
## Bug fixes
### 1. `enable_audio_memory=False` silently disabled ALL cross-shot memory
The pack computed `audio_memory_latent=None` when audio memory was off, and the
video **memory-bank save was gated on that latent being non-None** — so with
audio memory off (the standard anti-drone setting) the bank never filled and
cross-shot **identity** silently died (symptom: `memory_size=0` every shot even
with `memory_max_size=7`; a new face each shot).
Fix: memory storage is now unconditional; `enable_audio_memory` gates only the
audio-memory **injection** path. Verify: console `memory_size=` should climb
0,1,2,… capped at your `memory_max_size`. (`nodes.py`)
### 1b. Memory bank trim was a NO-OP whenever `memory_max_size <= num_fix_frames` (CRITICAL)
`PairedAudioVideoMemoryBank._trim()` computed `tail[-keep_tail:]` - and when
`keep_tail == 0` (e.g. the common max_size=3 / num_fix_frames=3 combo),
`tail[-0:]` is the WHOLE list, so the bank grew unbounded: every shot
conditioned on EVERY prior shot. Symptom: console `memory_size=` climbing
0,1,2,...,N-1 past your cap, and severe compounding quality degradation over
long runs (waxy skin, contrast crush, smearing by the late shots - the "gets
worse as it goes" failure). Fixed with a proper zero-tail branch + anchor
clamp; `memory_size=` now freezes at your cap. This one fix eliminated the entire
long-run degradation in our tests. (`libs/.../memory_multishot.py`)
### 2. GGUF text-encoder loader (`RebelsJE_TextEncoder`)
Two fixes so a text-only Gemma-3 GGUF loads cleanly:
- **meta-strip**: drop `vision_tower` / `multi_modal_projector` / `lm_head`
(the text-only GGUF has no weights for them → "Cannot copy out of meta tensor").
- **device-unify**: pin the embeddings-processor to the encoder's actual device
(GGUF Gemma runs on CPU while the connector was on cuda → addmm device mismatch).
- **fp8 gemma scale-key layouts**: the `our_fp8` swap only recognized its own
export layout (bare module names + `.scale_weight`); standard HF/comfy-style
fp8 gemma files (`.weight` + `.weight_scale`, e.g. community
abliterated builds) silently loaded with **zero modules swapped** — the
encoder stayed bf16 with no indication. Both layouts are now accepted
(per-tensor scalar scales; per-channel scales are skipped and those modules
stay bf16), and a loud warning prints if a file matches neither.
(`rebels_loaders.py`)
---
## Features
### 3. Split per-domain negative lever (`JoyEcho_TextEncode`)
The DMD pipeline has no CFG, so the only steering lever is embedding-space.
Instead of one `negative_prompt`/`negative_scale` that steers both branches,
this splits it:
- `negative_prompt_video` / `negative_scale_video` — kills burned-in
captions/subtitles. Working value ~0.5. **Above ~0.8 it over-rotates the
video context and locks every shot to shot 1's composition** (scene-lock).
- `negative_prompt_audio` / `negative_scale_audio` — kills invented
music/score. Keep ≤ ~0.4 or dialogue suffers.
Steering is norm-preserving (RescaleCFG-style): `cond' = renorm(cond + s*(cond − neg))`.
Old single-widget names still work as a fallback. (`nodes.py`)
### 4. Passthrough mode (`JoyEcho_LLMEnhance`)
`mode = "passthrough (raw JSON, skip LLM)"` — feed a finished
`{"prompts":[...]}` script straight through with no LLM call / no API key.
Auto-detects when `story_idea` already parses as that JSON. (`nodes.py`)
### 5. Reference-image conditioning — I2V-as-reference (`JoyEcho_Generate`)
New `reference_image` (IMAGE batch, up to 4). Identity references are prepended
as **video-only conditioning clips** at the memory-encode step — they are
**never** written into the paired audio/video bank. (An earlier attempt that
seeded refs into the bank with zero-filled audio latents injected loud
background noise with 2+ refs; video-only conditioning avoids it entirely.)
Also new: `head_trim_frames` (auto 8 with refs) drops the first N frames of each
shot, where the model morphs out of the reference/memory content. The trim is
applied once right after decode, so the final output, the per-shot preview
files, and any external concat of them stay frame-identical. (`nodes.py`)
### 6. Shot transitions (`JoyEcho_Generate`)
`transition`: `cut` (original) / `dissolve` (overlap cross-dissolve + equal-power
audio crossfade) / `vhs_glitch` (analog static burst at each boundary: snow,
tear bands, dropout lines + a raised-cosine tape-noise audio bed).
`transition_frames`, `glitch_intensity` tune it. (`nodes.py`)
### 7. fp8 transformer quantization (`JoyEcho_ModelLoader`)
New `fp8_transformer` toggle. Quantizes the DiT's attention/FF linear weights to
`float8_e4m3fn` **at load, from the normal bf16 checkpoint** (uses the vendored
`ltx_core.quantization.QuantizationPolicy.fp8_cast()` — upcasts per-layer at
inference). Roughly halves DiT weight memory and halves sequential-offload PCIe
traffic; keeps memory training + all tensors; VAEs/text-encoder/non-linears stay
bf16. Ignored when a GGUF DiT is selected (already quantized).
(`nodes.py` + `libs/ltx_distillation/models/ltx_wrapper.py` — new `quantization`
param; the quantized build path skips the post-load dtype cast that would
otherwise silently upcast fp8 back to bf16.)
### 8. Tiled VAE decode (`JoyEcho_Generate`)
Decoding a long high-res shot (e.g. 241f @ 1280×736) in one pass hard-aborts the
VAE decode on a 24–32 GB card (fatal cuDNN abort mid-conv, not a catchable OOM).
New `decode_tiling` (`auto`/`on`/`off`) routes decode through the vendored
`VideoDecoder.tiled_decode` — **temporal-only** 64-frame chunks with 24-frame
blended overlap (no spatial tiles → no spatial seams), streaming each chunk to
CPU. `auto` engages only above a size threshold, so small renders keep the
original single-pass decode bit-for-bit.
(`nodes.py` + `libs/ltx_distillation/utils.py` — `decode_benchmark_sample` gains
a `video_tiling_config` kwarg + `_decode_video_tiled_uint8`.)
### 9. Model dropdown (`JoyEcho_ModelLoader`)
New `model_file` combo lists every `.safetensors` / `.gguf` under the ComfyUI
`checkpoints` / `diffusion_models` / `unet` dirs. Pick a `.safetensors` → full
checkpoint (replaces `checkpoint_path`); pick a `.gguf` → DiT loaded from GGUF
while `checkpoint_path` still supplies the VAEs / vocoder / text connectors.
`"(use checkpoint_path)"` keeps the old typed-path behavior. A matching
`lora_file` dropdown lists every `.safetensors` under `models/loras`
(applied at `lora_strength` on the safetensors DiT path; ignored for GGUF). Plus a clear
early error if `gemma_path` is a `.gguf`/file/sidecar-less dir (this loader
needs the HF `gemma-3-12b-it` folder; GGUF Gemma only works via
`RebelsJE_TextEncoder`). (`nodes.py`)
### 10. LoRA loading hardening (`JoyEcho_ModelLoader` + `libs/.../fuse_loras.py`)
- A `lora_file` dropdown picks LoRAs from `models/loras` (existing
`lora_strength` widget applies).
- Fusion now supports **kohya naming** (`lora_down`/`lora_up`) in addition to
PEFT (`lora_A`/`lora_B`), with standard `alpha/rank` scaling — previously a
kohya-named LoRA silently did NOTHING (zero keys matched, no warning).
- Fusion prints how many weights fused, and WARNS LOUDLY when a provided LoRA
matched zero keys.
- The loader refuses **ComfyUI-quantized checkpoints** (`.comfy_quant` marker
tensors, e.g. "fp8mixed learned" builds) with a clear error: this loader
never applies their weight scales (the model would silently load mis-scaled)
and LoRA fusion on them crashes with shape errors. Use bf16 checkpoints.
### 11. Automation / batching nodes (new)
- **`JoyEcho_PromptSource`** — one dropdown listing LPFF-style `.txt` briefs
(from the inspire-pack prompts tree) **and** passthrough `.json` scripts
(`input/joyecho_prompts/`). Multi-block briefs fan out like
LoadPromptsFromFile. Emits `story_idea` (→ LLMEnhance) + `character`
(→ RefPicker) + `count`. Replaces the LPFF→UnzipPrompt chain and lets you
switch prompt sources with one dropdown instead of rewiring.
- **`JoyEcho_RefPicker`** — auto-selects a character reference image from a
folder tree keyed by character name (a `character_pick` dropdown of the
folder names, a typed/wired `character` string, or a prompt scan — dialogue
mentions are stripped so only the on-screen subject wins). The dropdown
survives model refreshes, an explicitly named character that matches no
folder refuses to fall back to the prompt scan (a wiped/typo'd name can't
silently become the wrong character's face), and the cache signature
includes the prompt text (without it, ComfyUI could serve a cached pick
from a previous queue item). `on_no_match=no_reference` returns nothing so
a batch keeps running.
- **`JoyEcho_RefBatch`** — None-tolerant image batcher: combines up to 4
optional IMAGE inputs (e.g. two RefPickers for a two-character shot), skips
missing refs, resizes mismatched sizes to the first image, outputs `None` if
all are missing (Generate then just skips identity seeding). The stock KJNodes
`ImageBatchMulti` crashes with `'NoneType' has no attribute 'shape'` on a
missing ref; this replaces it.
- **`JoyEcho_ScriptPicker`** — JSON dropdown (superseded by PromptSource; kept
for compatibility).
### 12. GPU encode hot-swap (`JoyEcho_TextEncode`)
With `low_vram` the Gemma encoder used to encode every shot on CPU (~10s+ per
shot). The encode pass now borrows the (idle) GPU when the encoder fits free
VRAM - with a fits-check, an OOM fallback to CPU, and a move-back before the
denoise phase. 20-shot encodes drop from minutes to seconds. (`nodes.py`)
### 13. `encoder_fp8` (`JoyEcho_ModelLoader`)
Stores the Gemma encoder's linear weights as float8_e4m3fn with per-layer
upcast at encode (encode runs once per item, so the upcast tax that makes
fp8 slow on the DiT is irrelevant here). Wrapper drops ~24GB -> ~21GB and the
GPU hot-swap engages on 32GB cards; JD's connector projections stay bf16.
### 14. `fp8_scaled_mm` (`JoyEcho_ModelLoader`) - native fp8 compute
Stores the DiT's attention/FF linears as fp8 AND runs the matmuls natively
via `torch._scaled_mm` - no per-layer upcast tax (measured x2.8 raw kernel /
x1.5 end-to-end vs bf16 on an RTX 5090). ~22GB resident enables
`sequential_offload=False` at moderate resolutions. REQUIREMENTS: sm_89+
GPU (RTX 40/50 - clear error on older cards, with a per-device runtime
fallback to upcast), and a **bf16 source checkpoint** (an fp8 FILE would load
every tensor fp8 with the cast skipped and crash the noise path - guarded
with a clear error). Tensorwise dynamic activation quant: A/B your content
before adopting.
### 15. `resident_blocks` (`JoyEcho_Generate`)
Sequential offload middle ground: pin the first N of 48 transformer blocks
permanently on GPU, stream the rest. N=24 halves the per-step PCIe traffic;
raise until VRAM is nearly full. Composes with fp8 modes (fp8 blocks are
half the bytes both resident and streamed).
### 16. Hires-fix second pass (`JoyEcho_Generate`)
`hires_factor` (>1.0) + `hires_denoise`: after all shots render, each shot is
bicubic-upscaled, VAE re-encoded, re-noised at a tail sigma and re-denoised
through the DMD ladder at the TARGET resolution - the model synthesizes real
detail (RTX-class upscalers only sharpen what exists). Runs in 65-frame
windows with cross-fade (a 24GB card survives 1920x1088 refines); memory
bank and per-shot previews stay base-res; failures fall back to the base
frames. Audio is untouched.
### 17. Reference scheduling upgrades (`JoyEcho_RefPicker` + `_Generate`)
- Script-carried ref pinning: `{"prompts": [...], "refs": {"zara":
"zara_file.png"}}` pins a scene-matched reference per character (a
full-scene ref SETS the render's scene - match it to the script).
- Re-entry injection: a character returning after a 3+-shot absence gets
their ref re-injected at the return shot automatically (the rolling memory
window is 4; long absences otherwise re-invent the character).
- Generate's ref dedup is schedule-aware (the same image scheduled at two
shots survives; cap 6 scheduled entries).
### 18. Robustness
- Pipelines no longer derive their working dtype from
`next(parameters()).dtype` (an fp8 first-param crashed `torch.randn`);
fp8 dtypes are skipped with a bfloat16 fallback.
- fp8 gemma swap accepts both `.scale_weight` and `.weight_scale` layouts
and warns loudly on zero matches instead of silently staying bf16.
---
## Applying
1. Back up your existing pack folder.
2. Copy each file over the same relative path in
`ComfyUI/custom_nodes/ComfyUI_JoyAI_Echo_GGUF_Nodes/`.
3. Restart ComfyUI. New widgets append at the **end** of existing nodes, so
saved graphs keep their values; the four new nodes appear under the
`JoyAI-Echo` category. Press `R` after adding model files to refresh the
`model_file` dropdown.
The `libs/` files must match the vendored `ltx_core` / `ltx_distillation` in
your pack (same JoyAI-Echo release). If your `libs/` differ substantially,
cherry-pick the changes described above rather than overwriting.
Not included (intentionally): model weights, the `gemma_assets/` tokenizer
binaries, `.bak` snapshots, and `__pycache__`.
---
## Changelog
### 2026-07-20 (v1.2)
- **LLM Enhance: local endpoints no longer require an API key.** When `base_url`
is a local address (localhost, 127.0.0.1, 192.168.x, .local), a placeholder is
sent automatically - Ollama / LM Studio / llama.cpp ignore it anyway. Cloud
providers still raise a clear error naming the endpoint. An empty field
previously errored even against a local server, which was the most common
first-run stumble.
- **Reference Picker: scripted refs now load for non-character scenes.** A
scene's JSON `refs` block (`{"kelpie": "shot.png"}`) is honored directly, so
creature and location references load without the folder name having to appear
in the prompt prose. Previously only names written in the prose matched.
- Reference Picker default refs root is now `input/joyecho_refs/`.
### 2026-07-20 (v1.1)
**Breaking:** `JoyEcho_ModelLoader`'s widgets were reorganized (all inputs are
now optional; pickers sit above their manual-path fallbacks). Saved graphs from
v1.0 will load this node with shifted values - delete and re-add the node, or
start from the bundled workflow.
- **AutoFinish A/V sync fix:** per-shot audio was ~30 ms shorter than its
video and the concat compounded the drift (~120 ms early by shot 5). Each
shot's wav is now padded to its own video duration before concat - sync is
now exact by construction. If multi-shot renders drifted out of sync late in
the piece, this was why.
- **`JoyEcho_AutoFinish` + worker are now included** (they were missing from
v1.0): auto-upscale of per-shot masters and audio-synced final assembly,
triggered automatically after the render.
- `JoyEcho_ModelLoader`: new `gemma_file` dropdown - pick the text encoder
from `models/text_encoders` or `models/clip` instead of typing a path.
- `JoyEcho_PromptSource`: new `manual_path` field - point at any .txt/.json
prompt file anywhere on disk; no folder conventions required.
- Story-writer system prompts: new FRAMING section (use shot-type nouns;
descriptive framing is ignored or mis-read; speaking shots no wider than a
medium close-up - the VAE packs 32 px per latent token, so a wide shot puts
the mouth below one token and lip sync cannot resolve) and HUMAN MOVEMENT
section (state gait/turn/contact mechanics; only describe body parts that
are in frame; speaking shots outrank movement).