INSTRUCTIONS.md

# JoyAI-Echo Multishot — Full Instructions

This is the complete, step-by-step guide. If you only read one file, read
this one. It covers install, your first render, where the reference image
goes, LoRAs, performance settings, and every common failure with its fix.

---

## 1. What this workflow actually is (read this first — it prevents confusion)

This is a **multi-shot text-to-video** pipeline with **joint audio** (voice and
picture generated together, not dubbed). You write one or more SHOTS; each shot
renders separately, and a **cross-shot memory bank** carries your character's
face, wardrobe, and voice from shot to shot. The thing that makes it work is
simple but strict:

> **Repeat your character's description sentence WORD-FOR-WORD IDENTICAL in
> every shot.** The memory bank keys on it. Reword it even slightly and the
> face drifts.

It is **not** a classic image-to-video workflow. You do not have to provide an
image at all — text alone holds identity. You *can* add a reference image to
lock the face harder (section 6), but it works differently from normal i2v:
it pre-seeds the memory bank rather than becoming frame one.

---

## 2. What you need

| thing | where it goes | notes |
|---|---|---|
| RealRebelAI's ComfyUI_JoyAI_Echo_GGUF_Nodes | `ComfyUI/custom_nodes/` | the BASE pack — install it first |
| This patch (the folder in this zip) | copied OVER the base pack | replace files when asked |
| A model build (see table below) | `ComfyUI/models/diffusion_models/` (safetensors) or `models/unet/` (GGUF) | one of: bf16, fp8, Q8_0, Q5_0 |
| A FULL checkpoint (bf16 or fp8) | `ComfyUI/models/diffusion_models/` | **required even when using a GGUF** — the GGUF is the transformer only; the VAEs and vocoder load from the full checkpoint |
| Gemma-3-12B-it text encoder | `ComfyUI/models/text_encoders/` | use TRUE BASE Gemma — single-file safetensors or the clean Q8_0 encoder GGUF. **Do not** use the circulating "joyecho" Gemma GGUF: it is a different model and produces duplicated people |
| ComfyUI-Easy-Use + ComfyUI-Custom-Scripts | `custom_nodes/` | two helper packs used by two convenience nodes; or delete those two nodes |

Which model build:

| you have | use |
|---|---|
| RTX 40/50-series, 24 GB+ | fp8 (fast) or Q8_0 GGUF (highest quant fidelity) |
| RTX 30-series or older, 24 GB | Q8_0 GGUF |
| 16 GB card | Q5_0 GGUF |
| 32 GB+ and patience | bf16 |


### System RAM and the Windows pagefile - READ THIS IF COMFYUI DIES WHILE LOADING

**A 24-32 GB GPU is not the binding constraint. System RAM is.** Loading a model
here materializes the whole checkpoint in system RAM as owned copies (not
memory-mapped), and the quantize / LoRA-fuse pass stacks more on top. The bf16
build is 46 GB on disk, so peak system usage runs well past what 64 GB of RAM
alone can provide.

**Symptoms** (all the same root cause):
- RAM climbs to ~100% while **VRAM stays almost idle**, then ComfyUI exits
- `Windows fatal exception: access violation` with a stack ending in
  `sft_loader.py` / `create_vae_wrappers`
- the whole machine hangs or reboots during the first load

There is no catchable Python error for this - the process is killed by the OS.

**Fixes, cheapest first:**

1. **Raise the Windows pagefile. This is the usual fix and costs only disk.**
   Settings > System > About > Advanced system settings > Performance Settings >
   Advanced > Virtual memory > Change. Untick "Automatically manage", select an
   SSD, choose Custom size, and set **65536-131072 MB (64-128 GB)**. Reboot.
   The peak is brief; with a large pagefile it spills to disk instead of dying.
   Machines with 64 GB RAM and a default pagefile are the ones that crash.
2. **Raise `resident_blocks` on the Generate node.** With `sequential_offload`
   on, blocks that are NOT resident live in system RAM. `resident_blocks=1`
   means 47 of 48 blocks sit in RAM while your VRAM goes unused. Push it up
   until VRAM is nearly full - roughly **24-30 on a 32 GB card, 12 on 24 GB**.
   This moves load off RAM and onto the card you bought.
3. **Use a GGUF** in `model_file` - GGUF weights stay packed and memory-mapped
   rather than copied into RAM.
4. **Avoid the fp8 FILE unless you are also using an fp8 toggle.** Loaded on its
   own it is upcast to bf16 at load, so a 25 GB file becomes ~50 GB in RAM. It
   saves disk, not memory.
5. Close other applications, and load no LoRAs on the first successful run
   (fusion allocates extra at the peak).

Recent versions print a system-memory estimate before loading and warn when it
looks too tight, instead of letting Windows fall over silently.

## 3. Install

1. Install the base pack (RealRebelAI's) into `ComfyUI/custom_nodes/` if you
   have not already.
2. Copy this zip's `ComfyUI_JoyAI_Echo_GGUF_Nodes` folder into
   `ComfyUI/custom_nodes/`, REPLACING files when asked.
3. Restart ComfyUI.
4. Open `workflow/JoyEcho_Multishot_Workflow_PUBLIC.json` (drag it onto the
   ComfyUI canvas, or Workflows > Open).

## 4. First render in five minutes

1. Copy `workflow/example_multishot.json` into `ComfyUI/input/joyecho_prompts/`
   (create the folder if it does not exist).
2. In the **Model Loader** node: pick your DiT in `model_file` (GGUF or
   safetensors). If you picked a GGUF, ALSO set `checkpoint_path` to your full
   bf16/fp8 checkpoint. Pick your Gemma in `gemma_file`.
3. In the **Prompt Source** node: pick `example_multishot.json`.
4. Queue. First run encodes the prompts (a few minutes); every later run of the
   same prompts skips encoding entirely (you will see `Conditioning cache HIT`).

Where your output goes — THREE places, by design:

| file | what it is |
|---|---|
| `output/joyecho/shot_000.mp4`, `shot_001.mp4`, ... | each shot, saved the moment it finishes (watch progress live) |
| `output/output_#####.mp4` | the assembled full video from SaveVideo (native resolution, with shot transitions) |
| `output/joyecho/<name>_<timestamp>_MASTER.mp4` | the upscaled master, built automatically in the background a few minutes after the run ends |

## 5. Writing your own shots

Your prompt file is either a `.json` file: `{"prompts": ["shot 1 text",
"shot 2 text", ...]}` — or a `.txt` in the same format the example shows. Rules
that matter:

- **The identity sentence is byte-identical in every shot.** Copy-paste it.
  Include: appearance, wardrobe, and a voice description WITH an explicit
  accent ("in a casual American accent" — the model defaults to British if you
  do not say otherwise).
- One shot is ~12.5 seconds at the default 313 frames. A comfortable spoken
  line for one shot is **about 20-30 words**, in double quotes.
- For talking shots, keep the framing **medium close-up or tighter** and the
  resolution height 864+. In wide shots the mouth is smaller than the model
  can resolve and lip sync suffers no matter what you write.
- Describe sound affirmatively in every shot ("Quiet diegetic sound only:
  rain on the window and a ticking radiator") — otherwise the model invents
  drones and music.
- Do not write "no X" in a prompt — it renders the X. State what IS there.

## 6. THE REFERENCE IMAGE — "where do I insert the source image?"

As of v1.4 the graph ships with a **Reference Image (OPTIONAL)** node just
below the Generate node, deliberately unconnected - unconnected nodes never
run, so it is safe to ignore. To use it: load your image there and drag its
IMAGE output to the `reference_image` input on the left edge of
JoyEcho_Generate. If you are on an older graph without the node:

1. Right-click the canvas > **Add Node > image > Load Image**.
2. Load your character image (a portrait render or photo-style still works
   best — clear face, front or three-quarter angle).
3. Drag from the Load Image node's **IMAGE** output to the **`reference_image`**
   input socket on the left edge of the **JoyEcho_Generate** node.
4. Queue as normal.

What it does: the image is planted into the cross-shot memory bank as a
permanent anchor, so EVERY shot is conditioned on that face and look — on top
of (not instead of) your identity sentence. Keep the identity sentence anyway,
and make it describe the same person as the image.

The `reference_zoom` widget (default 1.2) crops the reference in ~20% before
conditioning. Why: a reference at exactly the render size is treated as
continuable content — the first shot opens ON your image like classic i2v,
which usually is not what you want in a multi-shot piece. 1.2 keeps the
identity and drops that behavior. If you WANT shot 1 to start from your image
(true i2v-style opening), set `reference_zoom` to 1.0.

Advanced: the RefPicker node can auto-load reference folders per character by
name (a folder `ComfyUI/input/joyecho_refs/alice/` is used whenever "alice"
appears in your prompts). For single-character work the Load Image route above
is simpler.

## 7. LoRAs

As of v1.4 the loader takes **multiple LoRAs**, and the workflow ships with
the node for it: the **LoRA Stack** node, already wired into the Model Loader.

- Pick up to three LoRAs in its dropdowns, each with its own strength slider.
  Slots left on `(none)` do nothing.
- Need more than three? Add another LoRA Stack node (Add Node > JoyAI-Echo >
  JoyEcho LoRA Stack) and chain it: first node's `lora_stack` output into the
  second node's `lora_stack` input, second node into the Model Loader.
- Everything stacks with the Model Loader's own `lora_file` pick.

Power users / automation: `lora_path` also accepts a text list — one entry per
line or comma-separated, each with an optional strength suffix
(`my_style.safetensors@0.7`). Same result, no extra nodes.

All LoRAs are fused into the model at load, so there is no per-step cost.
Note: LoRAs apply on the **safetensors DiT path** only — they are ignored when
a GGUF is selected as the DiT (existing limitation, stated in the tooltip).

## 8. Performance settings by VRAM

| widget | what to set |
|---|---|
| `sequential_offload` | ON below 32 GB VRAM. Streams transformer blocks over PCIe. |
| `resident_blocks` | With offload ON: how many of the 48 blocks stay on the GPU. Rough guide: 24 GB card + fp8/GGUF -> ~20-24; 16 GB -> 8-12; raise until VRAM is nearly full. |
| `fp8_scaled_mm` | RTX 40/50-series ONLY. Native fp8 compute, real speedup. Leave OFF on 30-series and older. |
| `low_vram` (encoder) | ON if Gemma encoding OOMs; slower but ~3 GB. |
| resolution | 544x960 is a fast preview. For dialogue, raise height to 864+. |
| `hires_factor` | 1.0 = off (default). The background AutoFinish upscale usually covers you. |
| `head_trim_frames` | Leave at 14 (default). Trims the unstable first ~0.5s of each shot, where identity morph and lip-sync error concentrate. |

## 9. After updating this patch — one-time step

Open your workflow, glance at the LAST few widgets on the Generate node
(reference_zoom / resident_blocks / hires), fix anything that looks wrong, and
SAVE. ComfyUI stores widget values by position; a graph saved under an older
pack version can load a few trailing widgets scrambled until you re-save it
once. If you ever find yourself re-entering the same two or three values after
every restart — this is why, and one save ends it.

## 10. Troubleshooting

| symptom | cause / fix |
|---|---|
| Error: "checkpoint_path is empty" | You picked a GGUF in model_file. GGUFs are the transformer only — set checkpoint_path to a full bf16/fp8 checkpoint too. |
| Two copies of your character / wrong wardrobe / garbled identity | Wrong text encoder. Use TRUE BASE Gemma-3-12B-it (safetensors or the clean Q8_0 encoder GGUF). The "joyecho" Gemma GGUF is a different model — replace it. |
| Face drifts between shots | Your identity sentence is not byte-identical across shots. Copy-paste it; do not reword. |
| Voice is British | State the accent in every shot: "in a casual American accent". |
| Random music / horror drones | Add an explicit sound line to every shot; leave `negative_scale_audio` at 0.3. |
| Lip sync poor in wide shots | Physics of resolution: mouth too small. Medium close-up + height 864+ for talking shots. |
| Widgets reset every restart | Section 9 — re-save the graph once. |
| First run very slow | Normal: prompt encoding + caches build. Subsequent runs with the same prompts skip encoding entirely. |
| Master video missing | It arrives a few minutes AFTER the run ends (background worker). Check `output/joyecho/` for `*_MASTER.mp4` and the `_autofinish_*.log` next to it. |
| OOM during generation | sequential_offload ON, lower resident_blocks, lower resolution, low_vram ON for the encoder. |

---

Model downloads, measurements, and a live demo Space are linked from the
model listings. AI-generated content must be disclosed as such (LTX-2
Community License; the JoyAI-Echo component is research / non-commercial).