README.md

# JoyAI-Echo GGUF nodes — multishot fixes + automation patch


> **Before you dive in - a word on expectations.** This is a community-built,
> bleeding-edge pipeline: a 22B audio+video model with cross-shot memory,
> running on consumer hardware. It is powerful, but it is not one-click - your
> first clean render will likely take some tuning to YOUR machine (VRAM, system
> RAM and pagefile, and which model build: bf16 / fp8 / GGUF / INT8). No two
> rigs behave identically. `INSTRUCTIONS.md` covers install, per-VRAM settings,
> and every failure mode reported so far. **If you get stuck, [open an issue](../../issues)
> - I answer, I troubleshoot, and most of the fixes in this pack exist because
> someone reported something.** You will not be left hanging.


## New in 2.0 — RiftCast

Characters are now portable files. A `.riftcast` cartridge carries a
character's voice anchor, reference stills, canonical description, and
optionally LoRAs, environments, and a CharacterCardV3 persona — drop it in
`input/riftcast/` and the character casts by speaker tag in any script, no
training. Format spec: `RIFTCAST_SPEC.md`; `riftcast.py` packs, inspects,
and cuts cartridges from finished renders; demo cartridges (WREN, HANK)
ship in the repo.

The bundled **RiftCast Studio** workflow is a full character creator: a
Character Designer node for identity and a Style + Wardrobe node for
appearance (**71 styles** across 10 families, plus hair colour, hair shape,
makeup, accessories, demeanor and a wardrobe override) → audition tape →
automatically packed cartridge. A style never injects its own label; each
entry expands into concrete renderable descriptors, with separate masculine
and feminine wardrobe readings. The workflow also carries a source switch
between designer and classic LPFF/JSON prompt files, and a Render Clock node
that drives every fps/frames socket from one place with 8n+1 frame snapping.

Also in 2.0: `video_fps` defaults to 24 per the accent finding (Feature
#22 below — off-24 fps drifts every voice toward British/Australian and
overrides accent wording), widget values persist by NAME across node-layout
changes, `temporal_upscale` 2x latent upsampling to ~48 fps masters, and a
1441-frame (60 s) single-take ceiling.

## Quick fixes — read this first

Nearly every problem reported with this pack is one of these.

| symptom | what's actually wrong | fix |
|---|---|---|
| **Renders crawl; VRAM pinned at 100%** | Without `fp8_scaled_mm` the DiT runs in bf16 — ~40 GB staged. On a 32 GB card that streams over PCIe every step. | Turn **`fp8_scaled_mm` ON** (with a **bf16 or INT8** checkpoint) and **`sequential_offload` OFF** — ~22 GB resident, native fp8 matmul on RTX 40/50-series. It is **off by default** and is the single biggest speed setting in this pack. |
| **`fp8_scaled_mm` errors — "needs the bf16 checkpoint as its source"** | The toggle downcasts the attention/FF linears *itself*, so it must start from bf16. An fp8 FILE would load every tensor as fp8 — norms, tables and adalns included — and crash the denoise pipeline. | Point `model_file` at the **bf16** (or **INT8**) build. Do not pair the fp8 file with this toggle. |
| **The fp8 file didn't speed anything up** | With the toggles off, an fp8 checkpoint **upcasts to bf16 at load**. It saves download size, not memory or time. | Use **bf16 or INT8 + `fp8_scaled_mm`** for the actual win. |
| **The INT8 file is slow** | You ran it without `fp8_scaled_mm`. INT8 reconstructs to bf16 at load, so on its own it is the bf16 path plus ~40 s of reconstruction. | Turn **`fp8_scaled_mm` ON**. INT8 then behaves exactly like bf16 as a source — same ~22 GB resident — at roughly 60 % of the download. |
| **Lip sync drifts apart ~10 s into every shot** | The wrapper's video RoPE clock was hardcoded to 24 fps while audio RoPE runs in true seconds — a 25 fps render drifts ~4 %/s. Looks like a model limit; it is not. | Apply this patch (**Bug fix #0**). No checkpoint can fix it. With the patch, 60–105 s multishot masters hold sync. |
| **`memory_size=0` every shot; a new face each shot** | With `enable_audio_memory` off, the video memory-bank save was gated on the audio latent, so the bank never filled. | Fixed in this patch (**Bug fix #1**). Console `memory_size=` should climb 0,1,2,… up to your cap. |
| **Quality degrades over a long run** — waxy skin, smearing by the late shots | The memory-bank trim was a no-op when `memory_max_size <= num_fix_frames`, so the bank grew unbounded. | Fixed in this patch (**Bug fix #1b**). `memory_size=` now freezes at your cap. |
| **A GGUF errors about VAEs** | A GGUF is **DiT-only**. | Keep a full bf16 checkpoint in `checkpoint_path` — it supplies the VAEs, vocoder and connectors. |
| **Burned-in subtitles or captions** | The DMD pipeline has no CFG, so a plain negative does nothing. | Use `negative_prompt_video` / `negative_scale_video` (~0.5). Above ~0.8 it locks every shot to shot 1's composition. |
| **Voices come out British or Australian and IGNORE accent wording** | `video_fps` is not 24. The joint AV prior is 24 fps-native; any deviation drifts every voice Commonwealth-ward (25 → southern British, 30 → broad Australian — dose-response verified by A/B on identical configs, 2026-07-30) and geometrically overrides accent prose. | Set **`video_fps` = 24** (the v1.7 default) AND name the accent in the *positive*: "in a casual American accent". At 24 fps the wording is obeyed; at 25+ it is not. Full story: **Feature #22**. |
| **ComfyUI dies loading the VAEs, or `AttributeError: SiglipVisionModel` / `KeyError: 'rope_type'`** | transformers version. The pack's `requirements.txt` pins `<4.58`, which is out of date with the code — it ships shims for the 5.x RoPE and Siglip changes — so `pip install -r requirements.txt` silently downgrades a working 5.x install. | Reinstall the requirements into the **embedded** python, then `pip install "transformers==5.3.0"`. Confirmed good: 4.57 and 5.3.0. Confirmed broken: 5.13.1. |
| **Occasional robotic voice on long runs** | JoyAI-Echo's finetune is what suppresses it; the e50 merge keeps half of it. | For long multishot runs prefer the full-Echo surgical merge; use e50 for talking heads. |
| **Second shot won't lip-sync in a hand-built chain** | Guiding shot 2 with shot 1's decoded last frame is broken on *any* checkpoint — the guide is pixel-continuable, so the sampler reproduces it. | Extend with real audio+video latent context. See [Multishot Lite v2](https://huggingface.co/joeygambino/ltx23-multishot-lite). |

## Also here: Multishot Lite (core ComfyUI + one KJNodes node)

A second, much simpler workflow for chained talking shots, with **no node pack
to install**.

**Its own repo:** https://github.com/jlucasmcrell/ltx23-multishot-lite
(standalone docs + issues) · HF: https://huggingface.co/joeygambino/ltx23-multishot-lite

Mirrored here: [MULTISHOT_LITE.md](MULTISHOT_LITE.md) · [Releases](../../releases/latest).

* **Core ComfyUI nodes plus one from ComfyUI-KJNodes** (`LTXVAudioVideoMask`,
  which powers the extension). Nothing else to install.
* **Shot 2 is a true audio+video extension of shot 1**, not a cut: the last ~3 s
  of shot 1's video *and audio* become latent context, so the model generates
  forward from an ongoing utterance and the voice carries over by construction.
* **Mode 1 (default):** character speaks in a **reference voice you supply**.
  **Mode 2:** bypass one node per shot; the model invents a voice.
* Both shots joined and refined into one `FINAL` file.

> **v1.x is superseded.** It chained shots on shot 1's decoded last frame. That
> guide is pixel-continuable, so the sampler reproduced it instead of
> lip-syncing — chained shots came out as voiceover over a barely-moving face,
> and it got *worse* on more strongly distilled checkpoints. No setting fixed
> it. If you are on v1.x, update.

**Not a replacement for the patch below.** Lite has *no memory bank* — identity
continuity comes only from the extension context, so it drifts over many shots.
That is exactly why the node pack exists.

---

## Start here - which download do I need?

**Download the zip from [Releases](../../releases/latest).** That is the whole
thing: every patch file, the workflow, an example prompt file, and step-by-step
instructions. You do not need to clone this repo.

The files in this repo are the same files, unpacked, for browsing and diffing.

**This is a patch, not a standalone node pack.** Install RealRebelAI's
`ComfyUI_JoyAI_Echo_GGUF_Nodes` first, then MERGE these files over it (replace
when prompted) - do not delete or replace the whole folder. The pack prints a
clear startup error if it detects a replace-instead-of-merge install.

Models are not here; they are on Hugging Face: https://huggingface.co/joeygambino

---


> **This is a patch, not a one-click ComfyUI Manager install.**
> It layers on top of an existing `ComfyUI_JoyAI_Echo_GGUF_Nodes` install
> (RealRebelAI's Rebels GGUF loader stack). Install that first, then **merge**
> these files over it - do not replace the folder. The pack prints a clear
> startup error if it detects a replace-instead-of-merge install.
>
> Models are **not** in this repo. They live on Hugging Face:
> https://huggingface.co/joeygambino


A set of bug fixes and features layered on top of the community
`ComfyUI_JoyAI_Echo_GGUF_Nodes` pack (the Rebels GGUF loader stack around
JoyAI-Echo). Everything here targets the **multi-shot** path (`JoyEcho_Generate`
+ the discrete Rebels loaders / `JoyEcho_ModelLoader`).

This is a **patch drop**, not a standalone pack: copy these files over a working
install of the same pack (back up first). The files are interdependent — in
particular `nodes.py` calls new signatures added to the two `libs/` files, so
apply them together.

Tested on an RTX 5090 (32 GB) and a 3090 (24 GB), ComfyUI 0.26–0.27,
torch 2.8–2.11, with the JoyAI-Echo bf16 release and self-built Q8 GGUFs.

---

## Files in this package

```
nodes.py                                     # JoyEcho_TextEncode / _Generate / _ModelLoader / _LLMEnhance
__init__.py                                  # registrations for the new nodes
rebels_loaders.py                            # discrete GGUF loaders (text-encoder fixes)
joyecho_prompt_source.py   (new node)        # one dropdown: .txt briefs + .json scripts
joyecho_ref_picker.py      (new node)        # auto reference-image picker by character name
joyecho_ref_batch.py       (new node)        # None-tolerant image batcher
joyecho_script_picker.py   (new node)        # JSON dropdown (superseded by PromptSource)
libs/ltx_distillation/models/ltx_wrapper.py  # fp8 quantization passthrough
libs/ltx_core/loader/fuse_loras.py           # kohya-LoRA fusion + alpha scaling + fuse telemetry
libs/ltx_core/quantization/policy.py         # fp8_scaled_mm_torch policy (+ sm_89 gate)
libs/ltx_core/quantization/fp8_torch_mm.py   # native-fp8 Linear forward (torch._scaled_mm)
libs/ltx_distillation/utils.py               # tiled VAE decode
libs/ltx_distillation/inference/memory_multishot.py            # memory bank TRIM FIX (critical)
libs/ltx_distillation/inference/bidirectional_pipeline.py      # dtype hardening vs fp8 params
libs/ltx_distillation/inference/memory_bidirectional_pipeline.py # dtype hardening vs fp8 params
prompts/long_story_writer_system_prompt.md   # (optional) de-musicked + character-age edits
```

The files are interdependent - apply the whole set together, never cherry-pick
(a nodes.py newer than its libs/ raises AttributeError at load).

---

## Bug fixes

### 0. RoPE clock hardcoded to 24 fps — the ~10 s lip-sync cliff (CRITICAL)
Lip sync held for the first several seconds of a shot then progressively fell
apart, the mouth running steadily **ahead** of the audio, with the break
crossing visibility around **9.6 s into every shot** regardless of prompt,
model, or reference. This is the reason the pack's practical dialogue limit was
believed to be ~241 frames.

Root cause: `LTX2DiffusionWrapper.VIDEO_FPS` was a hardcoded class constant of
`24.0`, used to convert the video RoPE temporal coordinate from frames into
seconds. The **audio** RoPE is built in true seconds. Rendering at 25 fps
therefore ran the video positional clock 25/24 ≈ **4 % fast** against audio —
a linear divergence of ~0.04 s per second of runtime, i.e. roughly half a frame
of drift per second, accumulating without bound. At ~9–10 s it passes the
threshold where a viewer reads it as "not lip syncing".

Fix: the generate nodes now stamp the actual render fps onto the generator
before sampling — `JoyEcho_Generate` (main pass and the hires refine pass) and
`JoyEcho_SingleShotGenerate`. Any render at a consistent fps is now
rope-coherent end to end.

**Consequence: there is no ~241-frame shot limit.** Verified with 69 s and
105 s multishot masters. This was a pipeline bug, not a model property — no
LTX-2.3 or JoyAI-Echo checkpoint carries a short training-length cap here; the
temporal RoPE range is `positional_embedding_max_pos[0] = 20` (seconds),
identical in JoyAI-Echo, `ltx-2.3-22b-dev`, and `ltx-2.3-22b-distilled-1.1`.
Because the fix is in coordinate math rather than weights, it applies to every
checkpoint loaded through these nodes, merges included.
(`nodes.py` + `libs/ltx_distillation/models/ltx_wrapper.py`)

### 1. `enable_audio_memory=False` silently disabled ALL cross-shot memory
The pack computed `audio_memory_latent=None` when audio memory was off, and the
video **memory-bank save was gated on that latent being non-None** — so with
audio memory off (the standard anti-drone setting) the bank never filled and
cross-shot **identity** silently died (symptom: `memory_size=0` every shot even
with `memory_max_size=7`; a new face each shot).
Fix: memory storage is now unconditional; `enable_audio_memory` gates only the
audio-memory **injection** path. Verify: console `memory_size=` should climb
0,1,2,… capped at your `memory_max_size`. (`nodes.py`)

### 1b. Memory bank trim was a NO-OP whenever `memory_max_size <= num_fix_frames` (CRITICAL)
`PairedAudioVideoMemoryBank._trim()` computed `tail[-keep_tail:]` - and when
`keep_tail == 0` (e.g. the common max_size=3 / num_fix_frames=3 combo),
`tail[-0:]` is the WHOLE list, so the bank grew unbounded: every shot
conditioned on EVERY prior shot. Symptom: console `memory_size=` climbing
0,1,2,...,N-1 past your cap, and severe compounding quality degradation over
long runs (waxy skin, contrast crush, smearing by the late shots - the "gets
worse as it goes" failure). Fixed with a proper zero-tail branch + anchor
clamp; `memory_size=` now freezes at your cap. This one fix eliminated the entire
long-run degradation in our tests. (`libs/.../memory_multishot.py`)

### 2. GGUF text-encoder loader (`RebelsJE_TextEncoder`)
Two fixes so a text-only Gemma-3 GGUF loads cleanly:
- **meta-strip**: drop `vision_tower` / `multi_modal_projector` / `lm_head`
  (the text-only GGUF has no weights for them → "Cannot copy out of meta tensor").
- **device-unify**: pin the embeddings-processor to the encoder's actual device
  (GGUF Gemma runs on CPU while the connector was on cuda → addmm device mismatch).
- **fp8 gemma scale-key layouts**: the `our_fp8` swap only recognized its own
  export layout (bare module names + `.scale_weight`); standard HF/comfy-style
  fp8 gemma files (`<module>.weight` + `.weight_scale`, e.g. community
  abliterated builds) silently loaded with **zero modules swapped** — the
  encoder stayed bf16 with no indication. Both layouts are now accepted
  (per-tensor scalar scales; per-channel scales are skipped and those modules
  stay bf16), and a loud warning prints if a file matches neither.
(`rebels_loaders.py`)

### 3. comfy_quant fp8 checkpoints crashed the state-dict loader (v1.7)
The INT8 ConvRot support intercepted EVERY `.comfy_quant` marker at the
state-dict layer and raised on formats it didn't own — which broke loading of
fp8-mixed single-file gemma encoders ("comfy_quant format 'float8_e4m3fn' is
not supported"). The interception is now scoped to `int8_tensorwise` only;
any other quant format passes through raw for the downstream fp8 swap to
consume, exactly as before INT8 support landed. (`libs/ltx_core/loader/sft_loader.py`)

### 4. Console windows flashing at the end of every render (v1.7)
The AutoFinish worker was spawned with `DETACHED_PROCESS`, i.e. with no
console at all — so every ffmpeg/ffprobe it ran allocated its own visible
console window: a burst of half a dozen focus-stealing flashes as each master
was assembled. The worker now launches with `CREATE_NO_WINDOW`: it gets an
invisible console that all its children inherit silently. (`joyecho_autofinish.py`)

---

## Features

### 3. Split per-domain negative lever (`JoyEcho_TextEncode`)
The DMD pipeline has no CFG, so the only steering lever is embedding-space.
Instead of one `negative_prompt`/`negative_scale` that steers both branches,
this splits it:
- `negative_prompt_video` / `negative_scale_video` — kills burned-in
  captions/subtitles. Working value ~0.5. **Above ~0.8 it over-rotates the
  video context and locks every shot to shot 1's composition** (scene-lock).
- `negative_prompt_audio` / `negative_scale_audio` — kills invented
  music/score. Keep ≤ ~0.4 or dialogue suffers.
Steering is norm-preserving (RescaleCFG-style): `cond' = renorm(cond + s*(cond − neg))`.
Old single-widget names still work as a fallback. (`nodes.py`)

### 4. Passthrough mode (`JoyEcho_LLMEnhance`)
`mode = "passthrough (raw JSON, skip LLM)"` — feed a finished
`{"prompts":[...]}` script straight through with no LLM call / no API key.
Auto-detects when `story_idea` already parses as that JSON. (`nodes.py`)

### 5. Reference-image conditioning — I2V-as-reference (`JoyEcho_Generate`)
New `reference_image` (IMAGE batch, up to 4). Identity references are prepended
as **video-only conditioning clips** at the memory-encode step — they are
**never** written into the paired audio/video bank. (An earlier attempt that
seeded refs into the bank with zero-filled audio latents injected loud
background noise with 2+ refs; video-only conditioning avoids it entirely.)
Also new: `head_trim_frames` (auto 8 with refs) drops the first N frames of each
shot, where the model morphs out of the reference/memory content. The trim is
applied once right after decode, so the final output, the per-shot preview
files, and any external concat of them stay frame-identical. (`nodes.py`)

### 6. Shot transitions (`JoyEcho_Generate`)
`transition`: `cut` (original) / `dissolve` (overlap cross-dissolve + equal-power
audio crossfade) / `vhs_glitch` (analog static burst at each boundary: snow,
tear bands, dropout lines + a raised-cosine tape-noise audio bed).
`transition_frames`, `glitch_intensity` tune it. (`nodes.py`)

### 7. fp8 transformer quantization (`JoyEcho_ModelLoader`)
New `fp8_transformer` toggle. Quantizes the DiT's attention/FF linear weights to
`float8_e4m3fn` **at load, from the normal bf16 checkpoint** (uses the vendored
`ltx_core.quantization.QuantizationPolicy.fp8_cast()` — upcasts per-layer at
inference). Roughly halves DiT weight memory and halves sequential-offload PCIe
traffic; keeps memory training + all tensors; VAEs/text-encoder/non-linears stay
bf16. Ignored when a GGUF DiT is selected (already quantized).
(`nodes.py` + `libs/ltx_distillation/models/ltx_wrapper.py` — new `quantization`
param; the quantized build path skips the post-load dtype cast that would
otherwise silently upcast fp8 back to bf16.)

### 8. Tiled VAE decode (`JoyEcho_Generate`)
Decoding a long high-res shot (e.g. 241f @ 1280×736) in one pass hard-aborts the
VAE decode on a 24–32 GB card (fatal cuDNN abort mid-conv, not a catchable OOM).
New `decode_tiling` (`auto`/`on`/`off`) routes decode through the vendored
`VideoDecoder.tiled_decode` — **temporal-only** 64-frame chunks with 24-frame
blended overlap (no spatial tiles → no spatial seams), streaming each chunk to
CPU. `auto` engages only above a size threshold, so small renders keep the
original single-pass decode bit-for-bit.
(`nodes.py` + `libs/ltx_distillation/utils.py` — `decode_benchmark_sample` gains
a `video_tiling_config` kwarg + `_decode_video_tiled_uint8`.)

### 9. Model dropdown (`JoyEcho_ModelLoader`)
New `model_file` combo lists every `.safetensors` / `.gguf` under the ComfyUI
`checkpoints` / `diffusion_models` / `unet` dirs. Pick a `.safetensors` → full
checkpoint (replaces `checkpoint_path`); pick a `.gguf` → DiT loaded from GGUF
while `checkpoint_path` still supplies the VAEs / vocoder / text connectors.
`"(use checkpoint_path)"` keeps the old typed-path behavior. A matching
`lora_file` dropdown lists every `.safetensors` under `models/loras`
(applied at `lora_strength` on the safetensors DiT path; ignored for GGUF). Plus a clear
early error if `gemma_path` is a `.gguf`/file/sidecar-less dir (this loader
needs the HF `gemma-3-12b-it` folder; GGUF Gemma only works via
`RebelsJE_TextEncoder`). (`nodes.py`)

### 10. LoRA loading hardening (`JoyEcho_ModelLoader` + `libs/.../fuse_loras.py`)
- A `lora_file` dropdown picks LoRAs from `models/loras` (existing
  `lora_strength` widget applies).
- Fusion now supports **kohya naming** (`lora_down`/`lora_up`) in addition to
  PEFT (`lora_A`/`lora_B`), with standard `alpha/rank` scaling — previously a
  kohya-named LoRA silently did NOTHING (zero keys matched, no warning).
- Fusion prints how many weights fused, and WARNS LOUDLY when a provided LoRA
  matched zero keys.
- The loader refuses **ComfyUI-quantized checkpoints** (`.comfy_quant` marker
  tensors, e.g. "fp8mixed learned" builds) with a clear error: this loader
  never applies their weight scales (the model would silently load mis-scaled)
  and LoRA fusion on them crashes with shape errors. Use bf16 checkpoints.

### 11. Automation / batching nodes (new)
- **`JoyEcho_PromptSource`** — one dropdown listing LPFF-style `.txt` briefs
  (from the inspire-pack prompts tree) **and** passthrough `.json` scripts
  (`input/joyecho_prompts/`). Multi-block briefs fan out like
  LoadPromptsFromFile. Emits `story_idea` (→ LLMEnhance) + `character`
  (→ RefPicker) + `count`. Replaces the LPFF→UnzipPrompt chain and lets you
  switch prompt sources with one dropdown instead of rewiring.
- **`JoyEcho_RefPicker`** — auto-selects a character reference image from a
  folder tree keyed by character name (a `character_pick` dropdown of the
  folder names, a typed/wired `character` string, or a prompt scan — dialogue
  mentions are stripped so only the on-screen subject wins). The dropdown
  survives model refreshes, an explicitly named character that matches no
  folder refuses to fall back to the prompt scan (a wiped/typo'd name can't
  silently become the wrong character's face), and the cache signature
  includes the prompt text (without it, ComfyUI could serve a cached pick
  from a previous queue item). `on_no_match=no_reference` returns nothing so
  a batch keeps running.
- **`JoyEcho_RefBatch`** — None-tolerant image batcher: combines up to 4
  optional IMAGE inputs (e.g. two RefPickers for a two-character shot), skips
  missing refs, resizes mismatched sizes to the first image, outputs `None` if
  all are missing (Generate then just skips identity seeding). The stock KJNodes
  `ImageBatchMulti` crashes with `'NoneType' has no attribute 'shape'` on a
  missing ref; this replaces it.
- **`JoyEcho_ScriptPicker`** — JSON dropdown (superseded by PromptSource; kept
  for compatibility).

### 12. GPU encode hot-swap (`JoyEcho_TextEncode`)
With `low_vram` the Gemma encoder used to encode every shot on CPU (~10s+ per
shot). The encode pass now borrows the (idle) GPU when the encoder fits free
VRAM - with a fits-check, an OOM fallback to CPU, and a move-back before the
denoise phase. 20-shot encodes drop from minutes to seconds. (`nodes.py`)

### 13. `encoder_fp8` (`JoyEcho_ModelLoader`)
Stores the Gemma encoder's linear weights as float8_e4m3fn with per-layer
upcast at encode (encode runs once per item, so the upcast tax that makes
fp8 slow on the DiT is irrelevant here). Wrapper drops ~24GB -> ~21GB and the
GPU hot-swap engages on 32GB cards; JD's connector projections stay bf16.

### 14. `fp8_scaled_mm` (`JoyEcho_ModelLoader`) - native fp8 compute
Stores the DiT's attention/FF linears as fp8 AND runs the matmuls natively
via `torch._scaled_mm` - no per-layer upcast tax (measured x2.8 raw kernel /
x1.5 end-to-end vs bf16 on an RTX 5090). ~22GB resident enables
`sequential_offload=False` at moderate resolutions. REQUIREMENTS: sm_89+
GPU (RTX 40/50 - clear error on older cards, with a per-device runtime
fallback to upcast), and a **bf16 source checkpoint** (an fp8 FILE would load
every tensor fp8 with the cast skipped and crash the noise path - guarded
with a clear error). Tensorwise dynamic activation quant: A/B your content
before adopting.

### 15. `resident_blocks` (`JoyEcho_Generate`)
Sequential offload middle ground: pin the first N of 48 transformer blocks
permanently on GPU, stream the rest. N=24 halves the per-step PCIe traffic;
raise until VRAM is nearly full. Composes with fp8 modes (fp8 blocks are
half the bytes both resident and streamed).

### 16. Hires-fix second pass (`JoyEcho_Generate`)
`hires_factor` (>1.0) + `hires_denoise`: after all shots render, each shot is
bicubic-upscaled, VAE re-encoded, re-noised at a tail sigma and re-denoised
through the DMD ladder at the TARGET resolution - the model synthesizes real
detail (RTX-class upscalers only sharpen what exists). Runs in 65-frame
windows with cross-fade (a 24GB card survives 1920x1088 refines); memory
bank and per-shot previews stay base-res; failures fall back to the base
frames. Audio is untouched.

### 17. Reference scheduling upgrades (`JoyEcho_RefPicker` + `_Generate`)
- Script-carried ref pinning: `{"prompts": [...], "refs": {"zara":
  "zara_file.png"}}` pins a scene-matched reference per character (a
  full-scene ref SETS the render's scene - match it to the script).
- Re-entry injection: a character returning after a 3+-shot absence gets
  their ref re-injected at the return shot automatically (the rolling memory
  window is 4; long absences otherwise re-invent the character).
- Generate's ref dedup is schedule-aware (the same image scheduled at two
  shots survives; cap 6 scheduled entries).

### 18. Robustness
- Pipelines no longer derive their working dtype from
  `next(parameters()).dtype` (an fp8 first-param crashed `torch.randn`);
  fp8 dtypes are skipped with a bfloat16 fallback.
- fp8 gemma swap accepts both `.scale_weight` and `.weight_scale` layouts
  and warns loudly on zero matches instead of silently staying bf16.

### 20. Voice casting - per-character voices from files (NEW, 2026-07-29)

The memory bank guarantees voice CONSISTENCY, not correctness: shot 1 rolls
its voice from text conditioning alone, and whatever it rolls, the bank then
carries faithfully. This release makes the voice a CASTING decision instead
of a roll, with zero per-run typing:

- **Folder casting.** Put a clip of the character speaking (>=4 s, mp4 or
  wav) in `ComfyUI/input/joyecho_voices/<speaker-tag-lowercase>/`. Any script
  whose speaker tag matches the folder gets that voice seeded into the memory
  bank as a character-tagged anchor slot BEFORE shot 1 - the first shot
  *continues* the cast voice instead of auditioning a new one, and the same
  file re-casts the same voice in every future render. The pick is
  deterministic (alphabetically first file). Replace the file to recast.
- **Script-carried casting.** A `"voice_refs": {"Alice": "path/clip.mp4"}`
  key in the script JSON overrides the folder scan per character.
- **Audio-only anchors.** A bare wav/flac anchor pairs its voice with the
  character's ref image from `joyecho_refs/<tag>/` so the slot keeps its
  face+voice contract.
- **Speaker order is derived from the script** - an explicit
  `"speakers": [...]` array, or the `"<Name> is talking"` attribution in each
  shot. It now travels INSIDE the conditioning (and its disk cache), so it
  can never go stale or leak between graphs. The `speaker_order` widget
  remains as a manual override only.
- **Anchor + latest policy.** A speaker's audio context is their anchor
  slot(s) plus their most recent shot only - one drifted shot can no longer
  accumulate a majority and take over the rest of the video.
- **Cold-start fix.** A speaker's first line falls back to the unfiltered
  bank instead of a zeroed (silent) one - the regression that previously
  made per-character filtering unusable.

Two-character staging note that saves you a night: a2v cross-attention has
no spatial addressing - audio at time t drives EVERY face in frame, however
small or distant. Stage ONE face per shot (shot-reverse-shot) and put only
the visible character's description in that shot's prompt.

### 21. Correctness fixes from a full sampler-path audit (2026-07-29)

- **Hires refine now respects your seed.** Its re-noise fields were seeded
  from a hardcoded constant - every render's refine detail layer was
  identical regardless of the seed widget, for months.
- **Chained SingleShot graphs no longer condition on the PREVIOUS queue
  run's output.** The memory bank object was mutated in place through
  ComfyUI's output cache; incoming banks are now cloned.
- **The conditioning disk cache key includes the checkpoint** - a model swap
  can no longer be served the previous model's conditioning tensors.
  (Existing cache files are invalidated once; they rebuild on first render.)

### 22. The 24 fps rule — frame rate steers ACCENTS (2026-07-30)

The most consequential finding in this pack, discovered chasing a week of
"why is everyone suddenly Australian":

**LTX-2.3's joint audio-video prior is 24 fps-native, and the render fps is a
hidden accent dial.** The model experiences fps as video-token density per
second of audio RoPE time. Move it off 24 and voices slide monotonically into
the Commonwealth vowel space — same text, same seed, same everything:

| `video_fps` | same prompt, same config | reviewer verdict |
|---|---|---|
| 24 | witness monologue | rhotic **General American**, 5/5 segments |
| 25 | witness monologue | non-rhotic **southern British**, 5/5 segments |
| 30 | witness monologue | broad **General Australian** (rising terminals, raised DRESS), 5/5 |

Two practical consequences:

- **Off-24 fps overrides your prompt.** At 25 fps, even a line carrying
  "speaking in a casual American accent" renders British — a few conditioning
  tokens cannot outvote a geometric signal present in every attention
  operation at every step. At 24 fps the same wording is obeyed.
- **This interacts with Bug fix #0.** Pre-patch, the rope clock was hardcoded
  24, so 25 fps renders had broken lip-sync but American voices. The sync fix
  made the clock honest — and surfaced the accent drift that the bug had been
  masking. If your voices "changed" after applying the patch, this is why:
  you were rendering at 25. Drop to 24 and you get sync AND your accents.

As of v1.7 the node default is 24, the bundled workflow ships at 24, and the
Generate node prints a warning if you dial anything else. The flip side is a
free feature: if you *want* an authentic British or Australian character,
render their scenes at 25 or 30 fps instead of writing accent prose — it is
more consistent than any wording.

Voice prior note: with NO accent stated, young-female characters lean
Australian even at 24 fps (the base model's lean). State the accent on every
spoken line regardless — fps sets whether the model listens; the prompt still
has to do the asking.

### 19. Finishing: who builds your master (READ THIS before touching hires)
`hires_factor` is a ROUTING switch, not a quality slider - it decides which
pipeline builds your final video:

| you want | hires_factor | hires_denoise | master comes from |
|---|---|---|---|
| **default: zero detail-shimmer** | **1.0** | (ignored) | base shots, upscaled by AutoFinish (bicubic + contrast-adaptive sharpen - deterministic, seconds per shot) |
| synthesized detail (pores/hair), accepts slight per-frame texture shimmer | 1.5 | subtle / medium / strong | your refined shots, used as-is (AutoFinish skips its own upscale) |
| deterministic upscale baked into the shot files | 1.5 | spatial | spatial-upscaled shots - EVEN latent grids only (height AND width /32 must be even: 768-height yes, 736 no) |
| the old RTX path | 1.0 | (ignored) | base shots via RTXBatchVideoUpscale (`upscale_mode: rtx (legacy)` on the AutoFinish node) |

- **Judge and publish only the `*_MASTER.mp4`.** The in-graph SaveVideo
  output (prefixed `PREVIEW` in the shipped workflow) is a convenience
  preview: its re-encode is bit-starved and shows artifacts the master does
  not have. This is a ComfyUI limitation, not a render problem.
- Resolutions: everything works at any /32 size; only the `spatial` hires
  mode additionally needs the /32 result EVEN on both axes (1280x768 and
  1344x768 qualify; 1280x736 and 1536x864 do not - spatial will smear one
  edge and warn in the console).
- All other widgets are genuinely free: seed, num_frames (long shots are
  fine - the old ~10s lip-sync limit was the fps bug, now fixed), fps,
  memory sizes, head_trim.
- The master pipeline re-encodes with `bf 0` + `tune grain` end to end, so
  masters never reintroduce B-frame pumping.

---

## Applying

1. Back up your existing pack folder.
2. Copy each file over the same relative path in
   `ComfyUI/custom_nodes/ComfyUI_JoyAI_Echo_GGUF_Nodes/`.
3. Restart ComfyUI. New widgets append at the **end** of existing nodes, so
   saved graphs keep their values; the four new nodes appear under the
   `JoyAI-Echo` category. Press `R` after adding model files to refresh the
   `model_file` dropdown.

The `libs/` files must match the vendored `ltx_core` / `ltx_distillation` in
your pack (same JoyAI-Echo release). If your `libs/` differ substantially,
cherry-pick the changes described above rather than overwriting.

Not included (intentionally): model weights, the `gemma_assets/` tokenizer
binaries, `.bak` snapshots, and `__pycache__`.

---

## Where the models live

Nothing in this repo is a model. Weights are on Hugging Face:

| what | where |
|---|---|
| Surgical merge (bf16 / fp8) | https://huggingface.co/joeygambino/joyai-echo-ltx23-echoVid-ltxAud-surgical |
| Surgical merge GGUF (Q8_0 / Q5_0 / Q4_0) | https://huggingface.co/joeygambino/joyai-echo-ltx23-echoVid-ltxAud-surgical-gguf |
| Surgical merge INT8 ConvRot (stock ComfyUI loaders, **not** this pack) | https://huggingface.co/joeygambino/joyai-echo-ltx23-echoVid-ltxAud-surgical-int8 |
| Gemma-3-12B text encoder GGUF (Q8_0 / Q4_0) | https://huggingface.co/joeygambino/joyai-echo-gemma3-12b-encoder-Q8_0-gguf |
| This pack, mirrored | https://huggingface.co/joeygambino/joyai-echo-multishot-workflow |
| Everything | https://huggingface.co/joeygambino |

Civitai mirrors: [GGUF builds](https://civitai.com/models/2796109) - [workflow](https://civitai.com/models/2780640)

## License and scope

This repository contains **only my patch files** - it is not a redistribution
of the underlying pack or of any model weights.

- The patched pack wraps **JoyAI-Echo**, which is **research / non-commercial**.
  That is the strictest term in the stack and it governs your outputs.
- **LTX-2 / LTX-2.3** is under the
  [LTX-2 Community License](https://huggingface.co/Lightricks/LTX-2/blob/main/LICENSE.txt).
- **Gemma 3** - the text encoder, and the `gemma_assets/` tokenizer sidecars
  bundled here because the loader requires them - is subject to the
  [Gemma Terms of Use](https://ai.google.dev/gemma/terms).
- AI-generated content produced with this stack must be disclosed as such.

Not affiliated with Lightricks, JD, RealRebelAI, Comfy-Org, or Google.

## Credits

This patch stands on other people's work:

- **JD Joy Future Academy** - [JoyAI-Echo](https://huggingface.co/jdopensource/JoyAI-Echo), the multishot memory model this whole stack serves (research / non-commercial license).
- **Lightricks** - [LTX-2 / LTX-2.3](https://huggingface.co/Lightricks/LTX-2.3) (LTX-2 Community License).
- **TenStrip** - the [LTX2.3 DMD LoRAs](https://huggingface.co/TenStrip/LTX2.3_DMD_Lora); the hires `strong (tenstrip 4-step)` mode uses his published upscale sigma ladder verbatim.
- **RealRebelAI** - the Rebels GGUF loader stack this patches, and the [Q6_K_RM GGUF](https://huggingface.co/realrebelai/JoyAI-Echo_GGUF) whose tensor canon the self-built GGUFs mirror.
- **Comfy-Org** - the comfy-quants `int8_tensorwise` + ConvRot export recipe behind the INT8 checkpoint.
- **Google** - Gemma 3 12B, the text encoder (Gemma license).

## Support

Everything here is free and stays free. If it saved you time, you can
[sponsor me on GitHub](https://github.com/sponsors/jlucasmcrell),
[buy me a coffee](https://ko-fi.com/joeygambino), or
[support me on Liberapay](https://liberapay.com/joeygambino).