docs: detailed README with 2D/3D family map + Mermaid flow diagrams

README.md

@@ -1,20 +1,58 @@
-# bytical-talk
+---
+license: apache-2.0
+library_name: pytorch
+tags:
+  - talking-head
+  - lip-sync
+  - synctalk
+  - 2d
+  - bytical
+---
+
+# Bytical 2D Talk — Smart-AI Talking-Head System
 
-**A smart-AI talking-head video system.** It pairs the fast [SyncTalk_2D](https://github.com/ZiqiaoPeng/SyncTalk_2D)
-lip-sync **renderer** with an LLM/embedding-driven **brain** that understands the
-script, adapts to any input video, and critiques its own output — so it *performs* a
-script instead of just reading it.
+> **Dimension: 2D** · Renderer: [SyncTalk_2D](https://github.com/ZiqiaoPeng/SyncTalk_2D) (mouth-inpaint lip-sync)
+> **This repo = the 2D _application_** (renderer + LLM "brain"). Public.
 
-> Research project. The renderer clones a real presenter from a short video; the
-> brain is the layer that makes the system behave like "actual AI" rather than a
-> fixed pipeline. Built to be extended (see the roadmap).
+A talking-head video system that pairs a **fast 2D lip-sync renderer** with an
+**LLM/embedding-driven brain** that understands the script, adapts to any input
+video, and critiques its own output — so it *performs* a script instead of just
+reading it.
+
+---
+
+## Where this sits in the Bytical family
+
+Bytical has **two independent talking-head stacks** — a 2D one and a 3D one.
+This repo is the **2D application**.
+
+```mermaid
+graph TD
+    ROOT["Bytical Talking-Head Systems"]
+    ROOT --> TWOD["2D · SyncTalk_2D<br/>(mouth-inpaint lip-sync)"]
+    ROOT --> THREED["3D · Gaussian Splatting<br/>(TalkingGaussian / InsTaG)"]
+
+    TWOD --> A["bytical-2d-talk<br/>system + LLM brain · public"]
+    TWOD --> B["bytical-2d-synctalk-archive<br/>R&D + trained weights · private"]
+    THREED --> C["bytical-3d-head<br/>R&D checkpoints P1/P2 · private"]
+    THREED --> D["bytical-3d-instag-pretrain<br/>multi-identity pretrain · public"]
+
+    style A fill:#2563eb,color:#ffffff,stroke:#1e3a8a,stroke-width:3px
+```
+
+| Repo | Dim | Role |
+|---|:--:|---|
+| **bytical-2d-talk** ← *you are here* | 2D | System + LLM brain (this repo) |
+| bytical-2d-synctalk-archive | 2D | R&D lab notebook + trained weights/datasets |
+| bytical-3d-head | 3D | Gaussian-Splatting R&D checkpoints (P1/P2) |
+| bytical-3d-instag-pretrain | 3D | Multi-identity InsTaG pretrain weights |
 
 ---
 
 ## Why this exists
 
-Vanilla lip-sync models take `(video, audio) → video`. That's a renderer, not
-intelligence. `bytical-talk` adds a reasoning layer on top:
+Vanilla lip-sync models take `(video, audio) → video`. That's a *renderer*, not
+intelligence. `bytical-2d-talk` adds a reasoning layer on top:
 
 | Brain module | Input → Output | What it means |
 |---|---|---|
@@ -27,31 +65,43 @@ The renderer stays weight-safe and swappable; the brain is provider-agnostic
 
 ---
 
-## Architecture
-
-```
- script ──Director──▶ performance plan (emotion timeline + SSML)
-                           │
-                     [your TTS: SSML ─▶ wav]           (pluggable: Polly / Azure TTS / …)
-                           │
- input video ─AutoConfig─▶ render settings
-                           │
-                       render(video, wav, settings) ─▶ mp4     (SyncTalk_2D + improvements)
-                           │
-                       SelfQC ─▶ pass? ──no──▶ adjust settings, re-render (bounded)
-                           │ yes
-                           ▼
-                       final mp4 + QC report
+## End-to-end flow
+
+```mermaid
+flowchart LR
+    S["script"] --> DIR["Director<br/>(LLM)"]
+    DIR --> PLAN["performance plan<br/>emotion timeline + SSML"]
+    PLAN --> TTS["your TTS<br/>SSML → wav"]
+    V["input video"] --> AC["AutoConfig<br/>(LLM/CV)"]
+    AC --> CFG["render settings"]
+    TTS --> R["render(video, wav, settings)<br/>SyncTalk_2D + improvements"]
+    CFG --> R
+    R --> MP4["mp4"]
+    MP4 --> QC{"SelfQC<br/>pass?"}
+    QC -- "no (bounded retry)" --> AC
+    QC -- "yes" --> OUT["final mp4 + QC report"]
+
+    style DIR fill:#8b5cf6,color:#fff
+    style AC fill:#8b5cf6,color:#fff
+    style QC fill:#f59e0b,color:#111
+    style OUT fill:#16a34a,color:#fff
 ```
 
-- **`bytical_talk/brain/`** — `llm.py`, `director.py`, `autoconfig.py`, `qc.py`
-- **`bytical_talk/render/`** — improved inference: One-Euro crop smoothing, feather
-  paste-back, train/inference resize parity
-- **`bytical_talk/audio/`** — HuBERT features (better generalization to TTS voices)
-- **`bytical_talk/losses/`** — opt-in training upgrades (fixed VGG perceptual,
-  mouth-weighted L1, PatchGAN, LPIPS)
-- **`upstream/synctalk2d/`** — the renderer, fetched by `scripts/fetch_upstream.sh`
-  (not re-hosted)
+**Reading the diagram:** the *brain* nodes (purple) are LLM/CV steps that make
+decisions; the render step is the SyncTalk_2D renderer; **SelfQC** (amber) closes
+the loop by re-driving AutoConfig on failure, up to a bounded number of retries.
+
+---
+
+## Package layout
+
+| Path | What it holds |
+|---|---|
+| `bytical_talk/brain/` | `llm.py`, `director.py`, `autoconfig.py`, `qc.py` |
+| `bytical_talk/render/` | improved inference: One-Euro crop smoothing, feather paste-back, train/inference resize parity |
+| `bytical_talk/audio/` | HuBERT features (better generalization to TTS voices) |
+| `bytical_talk/losses/` | opt-in training upgrades (fixed VGG perceptual, mouth-weighted L1, PatchGAN, LPIPS) |
+| `upstream/synctalk2d/` | the renderer, fetched by `scripts/fetch_upstream.sh` (not re-hosted) |
 
 ---
 
@@ -117,16 +167,17 @@ print(plan.emotion_timeline())   # per-sentence emotion for the renderer
 
 ## Training a presenter (renderer)
 
-Any short, front-facing talking clip works (the Aarav clip and the Docker image in
-the sibling research folder are only test/packaging conveniences — nothing here
-depends on them). Standard SyncTalk_2D flow, then infer with the improvements:
+Any short, front-facing talking clip works. Standard SyncTalk_2D flow, then infer
+with the improvements:
 
 ```bash
-# preprocess + train (upstream), optionally with our opt-in losses / HuBERT audio
 python upstream/synctalk2d/data_utils/process.py dataset/<name>/<name>.mp4
 python bytical_talk/audio/hubert.py --wav_path dataset/<name>/aud.wav --num_frames <N>   # for --asr hubert
 ```
 
+> The trained checkpoints, multi-identity base, and render-ready datasets live in
+> the sibling archive **bytical-2d-synctalk-archive** (private).
+
 ---
 
 ## Roadmap