v1.6: voice casting system + sampler-path audit fixes

ComfyUI_JoyAI_Echo_v1.6_COMPLETE.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9dfda4540dac568972c32f7e66017037caa00d82a40918bb3b1f924cb592fff1
+size 2484894

README.md

@@ -80,7 +80,7 @@ a whole scene.
 
 *(For the node pack — if you want the dependency-free Lite workflow instead, grab `LTX23_Multishot_Lite_v1.0.zip` above.)*
 
-**Just the zip: `ComfyUI_JoyAI_Echo_v1.5_COMPLETE.zip`.** That is the whole thing -
+**Just the zip: `ComfyUI_JoyAI_Echo_v1.6_COMPLETE.zip`.** That is the whole thing -
 every patch file, the workflow, an example prompt file, and step-by-step
 instructions. Nothing else on this page is required.
 
@@ -343,6 +343,54 @@ frames. Audio is untouched.
 - fp8 gemma swap accepts both `.scale_weight` and `.weight_scale` layouts
   and warns loudly on zero matches instead of silently staying bf16.
 
+### 20. Voice casting - per-character voices from files (NEW, 2026-07-29)
+
+The memory bank guarantees voice CONSISTENCY, not correctness: shot 1 rolls
+its voice from text conditioning alone, and whatever it rolls, the bank then
+carries faithfully. This release makes the voice a CASTING decision instead
+of a roll, with zero per-run typing:
+
+- **Folder casting.** Put a clip of the character speaking (>=4 s, mp4 or
+  wav) in `ComfyUI/input/joyecho_voices/<speaker-tag-lowercase>/`. Any script
+  whose speaker tag matches the folder gets that voice seeded into the memory
+  bank as a character-tagged anchor slot BEFORE shot 1 - the first shot
+  *continues* the cast voice instead of auditioning a new one, and the same
+  file re-casts the same voice in every future render. The pick is
+  deterministic (alphabetically first file). Replace the file to recast.
+- **Script-carried casting.** A `"voice_refs": {"Alice": "path/clip.mp4"}`
+  key in the script JSON overrides the folder scan per character.
+- **Audio-only anchors.** A bare wav/flac anchor pairs its voice with the
+  character's ref image from `joyecho_refs/<tag>/` so the slot keeps its
+  face+voice contract.
+- **Speaker order is derived from the script** - an explicit
+  `"speakers": [...]` array, or the `"<Name> is talking"` attribution in each
+  shot. It now travels INSIDE the conditioning (and its disk cache), so it
+  can never go stale or leak between graphs. The `speaker_order` widget
+  remains as a manual override only.
+- **Anchor + latest policy.** A speaker's audio context is their anchor
+  slot(s) plus their most recent shot only - one drifted shot can no longer
+  accumulate a majority and take over the rest of the video.
+- **Cold-start fix.** A speaker's first line falls back to the unfiltered
+  bank instead of a zeroed (silent) one - the regression that previously
+  made per-character filtering unusable.
+
+Two-character staging note that saves you a night: a2v cross-attention has
+no spatial addressing - audio at time t drives EVERY face in frame, however
+small or distant. Stage ONE face per shot (shot-reverse-shot) and put only
+the visible character's description in that shot's prompt.
+
+### 21. Correctness fixes from a full sampler-path audit (2026-07-29)
+
+- **Hires refine now respects your seed.** Its re-noise fields were seeded
+  from a hardcoded constant - every render's refine detail layer was
+  identical regardless of the seed widget, for months.
+- **Chained SingleShot graphs no longer condition on the PREVIOUS queue
+  run's output.** The memory bank object was mutated in place through
+  ComfyUI's output cache; incoming banks are now cloned.
+- **The conditioning disk cache key includes the checkpoint** - a model swap
+  can no longer be served the previous model's conditioning tensors.
+  (Existing cache files are invalidated once; they rebuild on first render.)
+
 ### 19. Finishing: who builds your master (READ THIS before touching hires)
 `hires_factor` is a ROUTING switch, not a quality slider - it decides which
 pipeline builds your final video:

joyecho_prompt_source.py

@@ -60,8 +60,11 @@ def _list_files() -> list[str]:
         for p in sorted(root.rglob("*.txt")):
             out.append(_TXT_PREFIX + str(p.relative_to(root)))
     jroot = _json_root()
-    for p in sorted(jroot.glob("*.json")):
-        out.append(_JSON_PREFIX + p.name)
+    # rglob, matching the .txt branch above: a flat glob hides scripts filed in
+    # subfolders, which is how anyone organises more than a handful of them.
+    # _resolve() already joins the relative path, so nested names round-trip.
+    for p in sorted(jroot.rglob("*.json")):
+        out.append(_JSON_PREFIX + str(p.relative_to(jroot)).replace("\\", "/"))
     return out or [_EMPTY]
 
 
@@ -134,10 +137,39 @@ class JoyEcho_PromptSource:
             arr = data.get("prompts") or data.get("shots")
             if not isinstance(arr, list) or not arr:
                 raise ValueError(f"{p.name} must contain a non-empty 'prompts' (or 'shots') array.")
-            print(f"[JoyEcho] PromptSource: {p.name} (json, {len(arr)} shots, 1 item).", flush=True)
+            # Per-character audio memory: derive the speaker order from the
+            # script (an explicit "speakers" array, else the "<ID> is talking"
+            # attribution in each shot) and stash it for JoyEcho_Generate. The
+            # derivation lives in joyecho_script_picker; both loaders feed the
+            # same stash so it works regardless of which node the graph uses -
+            # the first wiring of this feature went ONLY into ScriptPicker and
+            # the live workflow loads through THIS node, so the 11:23 render
+            # silently ran with per-character memory off.
+            try:
+                from .joyecho_script_picker import (derive_speakers, set_last_speakers,
+                                                    set_last_voice_refs)
+            except ImportError:
+                from joyecho_script_picker import (derive_speakers, set_last_speakers,
+                                                   set_last_voice_refs)
+            speakers = derive_speakers(data, arr)
+            set_last_speakers(speakers)
+            vrefs = data.get("voice_refs") or {}
+            set_last_voice_refs(vrefs)
+            print(f"[JoyEcho] PromptSource: {p.name} (json, {len(arr)} shots, 1 item)"
+                  + (f", speakers: {' '.join(speakers)}" if speakers else ", no speaker tags")
+                  + (f", voice anchors: {', '.join(vrefs)}" if vrefs else "")
+                  + ".", flush=True)
             return ([text], [override], 1)
 
-        # TXT: LPFF-style blocks
+        # TXT: LPFF-style blocks. Clear the speaker stash - it holds whatever the
+        # LAST json load derived, and stale speakers applied to an unrelated
+        # script would filter the audio bank to the wrong characters.
+        try:
+            from .joyecho_script_picker import set_last_speakers, set_last_voice_refs
+        except ImportError:
+            from joyecho_script_picker import set_last_speakers, set_last_voice_refs
+        set_last_speakers([])
+        set_last_voice_refs({})
         items, names = [], []
         for blk in _BLOCK_SPLIT.split(text):
             m = _BLOCK_PATTERN.search(blk)

joyecho_script_picker.py

@@ -12,6 +12,7 @@ automatically (IS_CHANGED tracks mtime) — no need to reselect.
 """
 
 import json
+import re
 from pathlib import Path
 
 import folder_paths
@@ -19,6 +20,71 @@ import folder_paths
 _PROMPTS_SUBDIR = "joyecho_prompts"
 _EMPTY = "(no .json in input/joyecho_prompts)"
 
+# Per-character audio memory needs to know WHO speaks in each shot. That used to
+# be a hand-typed speaker_order widget on the Generate node, which does not
+# survive a queue-driven workflow - nobody is going to retype it per script. The
+# script already states the speaker in every shot, so derive it here and hand it
+# downstream. The widget remains as a manual override.
+#
+# The tags are opaque: save_memory_slot(character=...) and get_memory_audio(
+# speaker=...) only ever compare them for equality, so "ID_A" works as well as
+# "zara" and needs no mapping to a refs folder.
+_SPEAKER_RE = re.compile(r"\b(ID_[A-Z]|[A-Z][a-z]+)\s+is\s+talking\b")
+LAST_SPEAKERS: list[str] = []
+
+# Voice anchors: {"character": "path/to/clip.mp4"} carried by the script's
+# "voice_refs" key. Same stash pattern as LAST_SPEAKERS; the Generate node
+# encodes each clip's audio into a tagged memory-bank slot before shot 1, so
+# the character's voice is CAST from a file instead of rolled from text.
+# Keys must exactly match the script's speaker tags.
+LAST_VOICE_REFS: dict = {}
+
+
+def set_last_voice_refs(refs: dict) -> None:
+    """Stash the current script's voice anchors (empty dict clears)."""
+    global LAST_VOICE_REFS
+    LAST_VOICE_REFS = dict(refs) if isinstance(refs, dict) else {}
+
+
+def set_last_speakers(speakers: list[str]) -> None:
+    """Stash the current script's speaker order for the Generate node.
+
+    Wiring the `speakers` output is the explicit path; this module-level stash is
+    the zero-rewiring fallback so existing saved workflows and queued runs pick
+    it up with no canvas edits. ComfyUI executes a graph's nodes in dependency
+    order within one prompt, so the picker always runs before the generator it
+    feeds, and each execution overwrites the previous value.
+    """
+    global LAST_SPEAKERS
+    LAST_SPEAKERS = list(speakers)
+
+
+def derive_speakers(data: dict, shots: list) -> list[str]:
+    """Speaker tag per shot: an explicit "speakers" array wins, else the prose.
+
+    Returns [] when the script declares nothing and no shot names a speaker -
+    callers then fall back to character-blind memory, i.e. old behaviour.
+    """
+    if isinstance(data, dict):
+        declared = data.get("speakers") or data.get("speaker_order")
+        if isinstance(declared, str):
+            declared = [t for t in re.split(r"[,\s]+", declared.strip()) if t]
+        if isinstance(declared, list) and declared:
+            return [str(declared[i % len(declared)]) for i in range(len(shots))]
+
+    out, seen_any = [], False
+    for shot in shots:
+        m = _SPEAKER_RE.search(str(shot))
+        if m:
+            out.append(m.group(1))
+            seen_any = True
+        else:
+            # Unattributed shot: reuse the previous speaker rather than guessing.
+            # A wrong tag is worse than a repeated one - it would filter the bank
+            # to the wrong character and hand this shot the wrong voice.
+            out.append(out[-1] if out else "")
+    return out if seen_any and all(out) else []
+
 
 def _scripts_dir() -> Path:
     d = Path(folder_paths.get_input_directory()) / _PROMPTS_SUBDIR
@@ -45,8 +111,8 @@ class JoyEcho_ScriptPicker:
     def INPUT_TYPES(cls):
         return {"required": {"script": (_list_scripts(),)}}
 
-    RETURN_TYPES = ("STRING", "STRING",)
-    RETURN_NAMES = ("prompts_json", "path",)
+    RETURN_TYPES = ("STRING", "STRING", "STRING",)
+    RETURN_NAMES = ("prompts_json", "path", "speakers",)
     FUNCTION = "load"
     CATEGORY = "JoyAI-Echo"
 
@@ -81,8 +147,13 @@ class JoyEcho_ScriptPicker:
             arr = data.get("shots")
         if not isinstance(arr, list) or not arr:
             raise ValueError(f"{script} must contain a non-empty 'prompts' (or 'shots') array.")
-        print(f"[JoyEcho] ScriptPicker: {script} ({len(arr)} shots).", flush=True)
-        return (text, str(p),)
+        speakers = derive_speakers(data, arr)
+        set_last_speakers(speakers)
+        set_last_voice_refs(data.get("voice_refs") or {})
+        print(f"[JoyEcho] ScriptPicker: {script} ({len(arr)} shots)"
+              + (f", speakers: {' '.join(speakers)}" if speakers else ", no speaker tags")
+              + ".", flush=True)
+        return (text, str(p), " ".join(speakers),)
 
 
 NODE_CLASS_MAPPINGS = {"JoyEcho_ScriptPicker": JoyEcho_ScriptPicker}

libs/ltx_distillation/inference/memory_multishot.py

@@ -109,11 +109,12 @@ def build_paired_audio_memory_kwargs(
     enable_audio_memory: bool,
     v2a_grad_scale: float = 1.0,
     memory_position_mode: str = "reference",
+    speaker: Optional[str] = None,
 ) -> dict[str, Any]:
     if not enable_audio_memory:
         return {}
 
-    memory_audio = memory_bank.get_memory_audio()
+    memory_audio = memory_bank.get_memory_audio(speaker=speaker)
     if memory_audio is None:
         raise RuntimeError("audio memory was requested but the memory bank contains entries without audio latents")
 
@@ -305,7 +306,47 @@ class PairedAudioVideoMemoryBank:
         fixed = self.memory[:n_fixed]
         tail = self.memory[self.num_fix_frames :]
         keep_tail = self.max_size - len(fixed)
-        self.memory = fixed + (tail[-keep_tail:] if keep_tail > 0 else [])
+        if keep_tail <= 0:
+            self.memory = fixed
+            return
+
+        # Per-character reservation. The plain recency window lets a talkative
+        # character's slots evict a quieter one's ONLY voice exemplar, after
+        # which that character has nothing to be continued from and re-renders
+        # with a new voice. Guarantee every character seen in the tail keeps
+        # their most recent slot, and only evict from characters that still
+        # have another slot left.
+        #
+        # Indices are used throughout rather than entry objects: MemoryEntry is
+        # an eq=True dataclass holding tensors, so `entry in list` would run a
+        # tensor comparison and raise.
+        kept = list(range(len(tail)))[-keep_tail:]
+        last_by_char: dict[str, int] = {}
+        for i, entry in enumerate(tail):
+            owner = entry.metadata.get("character")
+            if owner:
+                last_by_char[owner] = i          # later shot wins = most recent
+
+        for reserved in sorted(last_by_char.values()):
+            if reserved in kept:
+                continue
+            counts: dict[Any, int] = {}
+            for i in kept:
+                owner = tail[i].metadata.get("character")
+                counts[owner] = counts.get(owner, 0) + 1
+            victim = next(
+                (i for i in kept
+                 if tail[i].metadata.get("character") is not None
+                 and counts[tail[i].metadata.get("character")] > 1),
+                None,
+            )
+            if victim is None:
+                break                            # nothing safe to drop; leave as-is
+            kept.remove(victim)
+            kept.append(reserved)
+            kept.sort()
+
+        self.memory = fixed + [tail[i] for i in kept]
 
     def save_memory_slot(
         self,
@@ -324,6 +365,7 @@ class PairedAudioVideoMemoryBank:
         audio_memory_n_fft: int = 1024,
         audio_memory_downsample_factor: int = 4,
         audio_memory_is_causal: bool = True,
+        character: Optional[str] = None,
     ) -> dict[str, Any]:
         audio_latent = self._prepare_audio_latent(audio_latent)
         if audio_latent is None:
@@ -379,7 +421,33 @@ class PairedAudioVideoMemoryBank:
                 video_clip_num_frames=video_clip_num_frames,
             )
 
+        # "last": take the video half of the slot from the END of the shot
+        # instead of from wherever the loudest speech happened.
+        #
+        # The audio window is still chosen by max_response - that is the right
+        # exemplar for the VOICE. But pairing it with a mid-shot video clip
+        # makes the next shot resume from mid-shot, which reads on screen as the
+        # take rewinding a second or two before continuing. Decoupling the two
+        # keeps the best voice reference AND gives the next shot a genuine
+        # continuation point.
+        if str(video_frame_selection_mode).lower() == "last" and frames:
+            n = max(1, int(video_clip_num_frames))
+            clip_start = max(0, len(frames) - n)
+            video_clip = list(frames[clip_start:])
+            if len(video_clip) < n:
+                video_clip.extend([video_clip[-1]] * (n - len(video_clip)))
+            video_metadata = {
+                "video_clip_start": int(clip_start),
+                "video_clip_end": int(len(frames)),
+                "video_clip_length": int(len(video_clip)),
+                "video_clip_center_frame": int(len(frames) - 1),
+                "video_total_frames": int(len(frames)),
+                "video_frame_selection_mode": "last",
+            }
+
         metadata = {"selection_mode": "paired_audio_window", **audio_metadata, **video_metadata}
+        if character:
+            metadata["character"] = str(character)
         entry = MemoryEntry(frame=video_clip, audio_latent=window_latent, metadata=metadata)
         fixed = self.memory[: self.num_fix_frames]
         free = self.memory[self.num_fix_frames :]
@@ -394,8 +462,72 @@ class PairedAudioVideoMemoryBank:
     def get_memory_metadata(self) -> list[dict[str, Any]]:
         return [dict(entry.metadata) for entry in self.memory]
 
-    def get_memory_audio(self) -> Optional[torch.Tensor]:
-        audio_latents = [entry.audio_latent for entry in self.memory]
+    def get_memory_audio(self, speaker: Optional[str] = None) -> Optional[torch.Tensor]:
+        """Concatenated memory audio, optionally scoped to ONE speaker.
+
+        Slots are written one per shot and the bank is character-blind, so with
+        two characters alternating shots the audio lane carries BOTH voices into
+        every shot. The model then has two equally valid voices to continue and
+        picks either - the same-gender voice merge.
+
+        When `speaker` is given, slots belonging to a DIFFERENT named character
+        are zeroed instead of dropped. Zeroing (not dropping) is deliberate: the
+        slot count, the per-slot lengths and therefore
+        `_build_paired_memory_cross_mask`'s positional pairing all stay exactly
+        as they were, so this cannot re-introduce the even-split desync that
+        shredded stored voices. The video half is untouched, so the non-speaker
+        keeps full face continuity - their slot simply pairs their face with
+        silence, which is what was true in that shot anyway.
+
+        Untagged slots (character is None) are always kept: a script that does
+        not declare speakers behaves exactly as before.
+        """
+        # COLD START. If this speaker owns no slot yet, filtering would zero
+        # EVERY slot and hand them a bank of pure silence to continue from -
+        # strictly worse than no filtering at all. Measured on the first attempt
+        # at this feature: Glyph's first line generated against an all-silent
+        # bank and her pitch wandered 219 -> 276 -> 258Hz across three lines,
+        # where with filtering off she was identical on all three. That single
+        # regression is what got per-character memory shelved.
+        #
+        # So: a character's FIRST line falls back to the unfiltered bank, which
+        # is exactly the pre-per-character behaviour for that one shot. From
+        # their second line on they have an exemplar of their own and get full
+        # isolation. The failure mode this removes (silence) is worse than the
+        # one it briefly tolerates (one shot of shared bank).
+        owns_a_slot = speaker is not None and any(
+            entry.metadata.get("character") == speaker for entry in self.memory)
+        effective_speaker = speaker if owns_a_slot else None
+
+        # ANCHOR + LATEST (2026-07-28). A speaker's audio context is their
+        # anchor slot(s) plus their MOST RECENT generated slot - older generated
+        # slots are zeroed like other characters'. Measured motivation: shot 5's
+        # line flipped Zara's accent against a context of three American
+        # exemplars (text pull beat the whole bank), and then shots 6-7 stayed
+        # flipped because the drifted slot outnumbered nothing - consistency
+        # machinery faithfully propagated the poison. With this policy a drifted
+        # shot contributes exactly one slot against the permanent anchor instead
+        # of accumulating a majority, so a single bad roll cannot own the rest
+        # of the video. Untagged slots and no-speaker calls are untouched.
+        latest_gen_idx = -1
+        if effective_speaker is not None:
+            for _i, entry in enumerate(self.memory):
+                if (entry.metadata.get("character") == effective_speaker
+                        and not entry.metadata.get("voice_anchor")):
+                    latest_gen_idx = _i
+
+        audio_latents = []
+        for _i, entry in enumerate(self.memory):
+            audio_latent = entry.audio_latent
+            if audio_latent is not None and effective_speaker is not None:
+                owner = entry.metadata.get("character")
+                if owner is not None and owner != effective_speaker:
+                    audio_latent = torch.zeros_like(audio_latent)
+                elif (owner == effective_speaker
+                        and not entry.metadata.get("voice_anchor")
+                        and _i != latest_gen_idx):
+                    audio_latent = torch.zeros_like(audio_latent)
+            audio_latents.append(audio_latent)
         if not audio_latents or any(audio_latent is None for audio_latent in audio_latents):
             return None
         first = audio_latents[0]
@@ -409,7 +541,13 @@ class PairedAudioVideoMemoryBank:
                     "All memory audio latents must share batch and channel dimensions, "
                     f"got first={tuple(first.shape)} current={tuple(audio_latent.shape)}"
                 )
-        return torch.cat(audio_latents, dim=1).contiguous()
+        # Harmonize dtypes before the cat: a pre-seeded voice-anchor slot may
+        # carry a different dtype than the pipeline's generated latents, and
+        # torch.cat requires one dtype. The LAST slot is the most recently
+        # generated one, so its dtype is the pipeline's - cast everything to it.
+        target_dtype = audio_latents[-1].dtype
+        return torch.cat([al.to(target_dtype) for al in audio_latents],
+                         dim=1).contiguous()
 
     def get_memory_audio_segment_lengths(self) -> tuple[tuple[int, ...], ...]:
         audio_latents = [entry.audio_latent for entry in self.memory]