Duplicate from Alissonerdx/LTX-Best-Face-ID

Co-authored-by: Alisson Pereira Anjos <Alissonerdx@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,40 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+examples/sample_1.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/sample_2.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/sample_3.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/sample_4.mp4 filter=lfs diff=lfs merge=lfs -text
+examples/sample_5.mp4 filter=lfs diff=lfs merge=lfs -text

Best_FaceID_v1.0_ArcFace_Projector.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17902bed35afc46a20ec183113d431937814e4950f49726578c97702486d9922
+size 69308896

Best_FaceID_v1.0_LoRA.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7aaab2f1bff2af121e0751120ad16a3e443b4223a04b78c51740029d25f17994
+size 2466643384

README.md

@@ -0,0 +1,219 @@
+---
+license: other
+base_model:
+- Lightricks/LTX-2.3
+tags:
+- ltx-video
+- identity-preservation
+- ipt2v
+- reference-to-video
+- lora
+- comfyui
+- ltx2.3
+pipeline_tag: text-to-video
+---
+
+# LTX-Best-Face-ID — LTX-2.3 Identity LoRA (Reference-to-Video / IPT2V)
+
+An identity-preserving **reference-to-video** LoRA for **LTX-2.3 (22B)**. Give it a reference
+photo of a person + a text prompt, and it generates a video that keeps that person's identity.
+
+Built with **overlap reference conditioning + TASS-RoPE (source-phase RoPE)** and a differentiable
+**ArcFace identity loss**. Runs in ComfyUI via the companion **BFS Nodes**.
+
+
+> **Status:** this is the first (overlap/TASS-RoPE) model. Other experimental variants
+> (native Gemma-vision conditioning, timestep-split texture injection) may be released later if
+> they prove out.
+
+---
+
+## 🎬 Examples
+
+<!--
+Drop result videos/gifs in an `assets/` folder in this repo and reference them below.
+On the HF model page, .mp4 in a Markdown <video> tag renders inline; .gif works too.
+Suggested layout: reference image (left) → generated video (right), with the prompt underneath.
+-->
+
+<video src="https://huggingface.co/Alissonerdx/LTX-Best-Face-ID/resolve/main/examples/sample_1.mp4" controls autoplay loop muted></video> 
+<video src="https://huggingface.co/Alissonerdx/LTX-Best-Face-ID/resolve/main/examples/sample_2.mp4" controls autoplay loop muted></video>
+<video src="https://huggingface.co/Alissonerdx/LTX-Best-Face-ID/resolve/main/examples/sample_3.mp4" controls autoplay loop muted></video>
+
+### With 2 Pass (Upscaled)
+<video src="https://huggingface.co/Alissonerdx/LTX-Best-Face-ID/resolve/main/examples/sample_4.mp4" controls autoplay loop muted></video>
+<video src="https://huggingface.co/Alissonerdx/LTX-Best-Face-ID/resolve/main/examples/sample_5.mp4" controls autoplay loop muted></video>
+---
+
+## What it does
+
+- **Reference-to-video (ref_t2v):** one reference image → video of that identity performing the
+  prompt's action.
+- Identity is injected by placing the **reference latent** in the target's frame-0 RoPE grid
+  (overlap) and tagging it with a distinct **source phase** so the model knows it is a *reference*,
+  not the first frame to generate.
+- An auxiliary **ArcFace face-similarity loss** on the decoded prediction sharpens the identity.
+
+---
+
+## How it works (technique)
+
+### Overlap reference + TASS-RoPE (source-phase)
+The reference latent is concatenated to the video sequence sharing the frame-0 grid (classic
+IC-LoRA "overlap"). To stop the reference from leaking into / being confused with the generated
+first frame, each source gets a distinct **multiplicative RoPE phase**:
+
+```
+phase[d] = source_id · phase_scale · θ^(−d/L)      (θ = 10000)
+target tokens: source_id = 0  (phase 0, exact no-op)
+reference:     source_id = 2  (distinct rotary "tag")
+```
+
+This "source tag" lets the model separate *who is who* in the sequence and strongly improves
+identity transfer. Because the tag is positional, the same mechanism generalizes to **multiple
+references** (source_id = 2, 3, 4, …) for multi-subject conditioning.
+
+> **TASS-RoPE** (*Temporal-Adjacent Spatial-Shifted RoPE*) is from **ST-DRC**, Chen et al.,
+> *Spatial-Temporal Decoupled Reference Conditioning for Identity-Preserving Text-to-Video
+> Generation* — [arXiv:2606.02441](https://arxiv.org/abs/2606.02441). This LoRA uses an
+> overlap + per-source RoPE-phase variant of that idea.
+
+### ArcFace identity loss
+During training the flow-matching prediction is decoded to pixels, the face is aligned and passed
+through ArcFace (buffalo_l `w600k_r50`), and a cosine identity loss pulls the generated face toward
+the reference embedding — the same recognition space used to evaluate identity.
+
+---
+
+## Training details
+
+| | |
+|---|---|
+| Base model | LTX-2 (22B) |
+| Method | LoRA (rank 128, alpha 128) |
+| Conditioning | overlap reference latent + TASS-RoPE source-phase (`source_id=2`, `phase_scale=1.0`) |
+| Aux loss | ArcFace identity loss (+ temporal identity consistency) |
+| Data | reference-image ↔ video pairs (OpenS2V subset + HuMoSet), close-up / frontal identities |
+
+---
+
+
+## Reference image — what works best
+
+This model was trained on face-focused reference images. Looking at the training data, every reference follows the same pattern:
+
+- Close-up / bust crop — roughly chest-up, with the face large and clearly visible.
+- A single subject, centered in the frame.
+- Frontal or near-frontal (slight 3/4 angles are fine, but the face is always well exposed).
+- Tightly cropped on the face/upper body — the training crops are about 460×406 (nearly square).
+
+Recommendation: for the best identity transfer, give the model a reference that looks like this — crop your image to the face / upper body. Full-body shots, wide shots, or images where the face is small and far away work noticeably worse, because the model never saw references like that during training.
+
+In short: a clean, frontal, well-lit close-up of the face is the ideal reference.
+
+## Captions (IMPORTANT)
+
+This model was trained on captions in the **`ref_t2v:`** format that **describe the scene and the
+person's action** (not just "a person"). Identity in LTX is strongly prompt-driven — the more the
+prompt describes, the better the identity holds. For best results:
+
+- Prefix prompts with **`ref_t2v:`**.
+- Describe the **action, setting, framing, camera** in present-progressive.
+- Describing **identity attributes** of the reference person (skin tone, hair, eyes, facial hair,
+  glasses, face shape) **noticeably improves** the result.
+
+The shared ComfyUI workflow includes a **Prompt Enhancer** that looks at the reference image (via
+the model's own multimodal Gemma-3 text encoder) and automatically enriches your prompt with the
+person's identity attributes — so you don't have to describe them by hand.
+
+Example (enhanced):
+```
+ref_t2v: A light-skinned man with long dark-brown hair past his shoulders, narrow rectangular
+metal-frame glasses, light blue-gray eyes and light stubble is folding clothes in a laundry room,
+medium-wide shot. He places the folded clothes into a basket, then carries it across the room.
+Neutral indoor lighting, simple domestic environment.
+```
+
+---
+
+## Usage (ComfyUI)
+
+**Requires the BFS Nodes:** https://github.com/alisson-anjos/ComfyUI-BFSNodes
+
+1. Install ComfyUI-BFSNodes (ComfyUI Manager → Install via Git URL, or clone into `custom_nodes/`).
+   Dependencies (`insightface`, `transformers`, …) install automatically.
+2. Load LTX-2 (checkpoint + the Gemma-3 text encoder / CLIP) as usual.
+3. Add the **LTX Identity Transfer** (overlap + source-phase) node; feed it the reference image.
+4. Load **this LoRA** on the MODEL path.
+5. Use the shared workflow (it wires the **Prompt Enhancer** → identity node → sampler).
+
+A ready-made **workflow JSON** is included alongside the model.
+
+### About the ArcFace projector (optional — you don't need it)
+
+The identity node has an optional **ArcFace projector** dropdown (it projects a face embedding into
+extra conditioning tokens). **You can leave it on `None`.** In our experiments the projector's
+effect was **marginal** — identity is carried almost entirely by the overlap reference latent (and
+the prompt), so the projector adds very little on top and is not required for this model. It is kept
+only as an optional extra; skipping it keeps the graph simpler with no meaningful quality loss.
+
+---
+
+## Tips for best results
+
+- Use a **clean, frontal, well-lit close-up** as the reference — identity quality tracks reference
+  quality closely.
+- Let the **Prompt Enhancer** add the identity attributes (or write them yourself).
+- Keep the reference framing simple; busy/occluded reference faces degrade identity.
+
+---
+
+## Limitations
+
+- **First-frame / "mask" tendency:** because identity comes from the raw reference *latent*, the
+  reference appearance (and framing / accessories like glasses) can partially copy into the video —
+  it can look like the reference is "pasted"/masked rather than freshly generated. Using a clean
+  frontal reference and the trim-first-frame node mitigates this.
+- **Prompt dependence:** identity is much stronger when the prompt describes the person. A bare,
+  generic prompt gives weaker identity — use the Prompt Enhancer.
+- **Close-up bias:** the training data skews to close-up / frontal faces, so full-body or
+  large-angle shots hold identity less well.
+- **Metric caveat:** ArcFace similarity is unreliable on small / turned / occluded faces — judge
+  visually, not only by score.
+
+---
+
+## Citation
+
+If you use this model, please cite the base model and the technique:
+
+```bibtex
+@misc{ltx_best_face_id,
+  title  = {LTX-Best-Face-ID: LTX-2.3 Identity LoRA (Reference-to-Video)},
+  author = {Alissonerdx},
+  year   = {2026},
+  howpublished = {\url{https://huggingface.co/Alissonerdx/LTX-Best-Face-ID}}
+}
+```
+
+- **Base model:** LTX-2.3 (Lightricks).
+- **TASS-RoPE / reference conditioning — ST-DRC:**
+
+```bibtex
+@article{chen2026stdrc,
+  title  = {Spatial-Temporal Decoupled Reference Conditioning for Identity-Preserving Text-to-Video Generation},
+  author = {Chen, Yuheng and Hu, Teng and Wang, Yuji and He, Qingdong and Ma, Lizhuang and Zhang, Jiangning},
+  journal = {arXiv preprint arXiv:2606.02441},
+  year   = {2026}
+}
+```
+
+- **Identity loss:** ArcFace (Deng et al., 2019) via InsightFace `buffalo_l`.
+- **Tooling:** [ComfyUI-BFSNodes](https://github.com/alisson-anjos/ComfyUI-BFSNodes).
+
+---
+
+## Acknowledgements
+
+Trained on top of LTX-2. Identity supervision via InsightFace ArcFace. ComfyUI integration via
+the BFS Nodes.

examples/sample_1.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ee90e7def91114800dd9deb24140f759918430c3144ba5844e349cc3e6b5d7d
+size 3187643

examples/sample_2.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:988fb2c1e7453898a7dfbd7e339ce1cd73d5db5555b1cf5cf5903cf8fe7b9531
+size 2030079

examples/sample_3.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98abcb60453fa8222f7e4c3fe4227fd32d385e8781dd6f9d376005c4ed123e95
+size 2622172

examples/sample_4.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:931a6a45ddf620ae2b552f0424756928c5e3005ba3629bd134a6fa93b922f835
+size 1466632

examples/sample_5.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:692cd463afec601975c446e8d840f954c5b143aa9161869a5f7a41b60e36069c
+size 10008651