---
license: apache-2.0
library_name: pytorch
tags:
  - talking-head
  - lip-sync
  - synctalk
  - 2d
  - bytical
---

# Bytical 2D Talk — Smart-AI Talking-Head System

> **Dimension: 2D** · Renderer: [SyncTalk_2D](https://github.com/ZiqiaoPeng/SyncTalk_2D) (mouth-inpaint lip-sync)
> **This repo = the 2D _application_** (renderer + LLM "brain"). Public.

A talking-head video system that pairs a **fast 2D lip-sync renderer** with an
**LLM/embedding-driven brain** that understands the script, adapts to any input
video, and critiques its own output — so it *performs* a script instead of just
reading it.

---

## Where this sits in the Bytical family

Bytical has **two independent talking-head stacks** — a 2D one and a 3D one.
This repo is the **2D application**.

```mermaid
graph TD
    ROOT["Bytical Talking-Head Systems"]
    ROOT --> TWOD["2D · SyncTalk_2D<br/>(mouth-inpaint lip-sync)"]
    ROOT --> THREED["3D · Gaussian Splatting<br/>(TalkingGaussian / InsTaG)"]

    TWOD --> A["bytical-2d-talk<br/>system + LLM brain · public"]
    TWOD --> B["bytical-2d-synctalk-archive<br/>R&D + trained weights · private"]
    THREED --> C["bytical-3d-head<br/>R&D checkpoints P1/P2 · private"]
    THREED --> D["bytical-3d-instag-pretrain<br/>multi-identity pretrain · public"]

    style A fill:#2563eb,color:#ffffff,stroke:#1e3a8a,stroke-width:3px
```

| Repo | Dim | Role |
|---|:--:|---|
| **bytical-2d-talk** ← *you are here* | 2D | System + LLM brain (this repo) |
| bytical-2d-synctalk-archive | 2D | R&D lab notebook + trained weights/datasets |
| bytical-3d-head | 3D | Gaussian-Splatting R&D checkpoints (P1/P2) |
| bytical-3d-instag-pretrain | 3D | Multi-identity InsTaG pretrain weights |

---

## Why this exists

Vanilla lip-sync models take `(video, audio) → video`. That's a *renderer*, not
intelligence. `bytical-2d-talk` adds a reasoning layer on top:

| Brain module | Input → Output | What it means |
|---|---|---|
| **Director** | script → emotion/emphasis/pacing + SSML | reads *meaning* and decides delivery |
| **AutoConfig** | any video → render settings | no manual tuning; works on *any* clip |
| **SelfQC** | rendered video → pass/fail + fix | the system reviews itself and retries smarter |

The renderer stays weight-safe and swappable; the brain is provider-agnostic
(defaults to Azure OpenAI, works with any OpenAI-compatible endpoint).

---

## End-to-end flow

```mermaid
flowchart LR
    S["script"] --> DIR["Director<br/>(LLM)"]
    DIR --> PLAN["performance plan<br/>emotion timeline + SSML"]
    PLAN --> TTS["your TTS<br/>SSML → wav"]
    V["input video"] --> AC["AutoConfig<br/>(LLM/CV)"]
    AC --> CFG["render settings"]
    TTS --> R["render(video, wav, settings)<br/>SyncTalk_2D + improvements"]
    CFG --> R
    R --> MP4["mp4"]
    MP4 --> QC{"SelfQC<br/>pass?"}
    QC -- "no (bounded retry)" --> AC
    QC -- "yes" --> OUT["final mp4 + QC report"]

    style DIR fill:#8b5cf6,color:#fff
    style AC fill:#8b5cf6,color:#fff
    style QC fill:#f59e0b,color:#111
    style OUT fill:#16a34a,color:#fff
```

**Reading the diagram:** the *brain* nodes (purple) are LLM/CV steps that make
decisions; the render step is the SyncTalk_2D renderer; **SelfQC** (amber) closes
the loop by re-driving AutoConfig on failure, up to a bounded number of retries.

---

## Package layout

| Path | What it holds |
|---|---|
| `bytical_talk/brain/` | `llm.py`, `director.py`, `autoconfig.py`, `qc.py` |
| `bytical_talk/render/` | improved inference: One-Euro crop smoothing, feather paste-back, train/inference resize parity |
| `bytical_talk/audio/` | HuBERT features (better generalization to TTS voices) |
| `bytical_talk/losses/` | opt-in training upgrades (fixed VGG perceptual, mouth-weighted L1, PatchGAN, LPIPS) |
| `upstream/synctalk2d/` | the renderer, fetched by `scripts/fetch_upstream.sh` (not re-hosted) |

---

## Install

```bash
git clone https://github.com/piyushptiwari1/bytical-talk.git
cd bytical-talk
pip install -e .            # brain only (light: openai, numpy, pyyaml)
pip install -e ".[render]"  # + renderer/audio/CV deps (torch, cv2, transformers, …)

cp .env.example .env        # then fill in your keys
bash scripts/fetch_upstream.sh   # only needed for rendering
```

### Configure the brain

Edit `.env` (never committed). Default backend is Azure OpenAI:

```
BYTICAL_LLM_PROVIDER=azure
AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com/
AZURE_OPENAI_API_KEY=<key>
AZURE_DEPLOYMENT_NAME=gpt-4o-mini
AZURE_EMBEDDING_MODEL_NAME=text-embedding-3-small
```

Any OpenAI-compatible endpoint works with `BYTICAL_LLM_PROVIDER=openai`.

---

## Use

```bash
# verify credentials + upstream
bytical-talk env-check

# LLM performance plan (no GPU needed)
bytical-talk direct --script "We protect what matters most. Let's find your plan."

# analyze any video -> recommended render settings (needs [render])
bytical-talk autoconfig --video presenter.mp4

# quality review of a rendered clip
bytical-talk qc --video out.mp4

# full pipeline (needs a trained checkpoint + a wav)
bytical-talk generate --script "..." --checkpoint ckpt.pth \
  --dataset dataset/presenter --audio speech.wav --out out.mp4 --reference presenter.mp4
```

Python:

```python
from bytical_talk import Director, auto_config, SelfQC

plan = Director().direct("Hi, I'm here to help you choose the right cover.")
print(plan.ssml)                 # Polly-ready SSML with emphasis + pauses
print(plan.emotion_timeline())   # per-sentence emotion for the renderer
```

---

## Training a presenter (renderer)

Any short, front-facing talking clip works. Standard SyncTalk_2D flow, then infer
with the improvements:

```bash
python upstream/synctalk2d/data_utils/process.py dataset/<name>/<name>.mp4
python bytical_talk/audio/hubert.py --wav_path dataset/<name>/aud.wav --num_frames <N>   # for --asr hubert
```

> The trained checkpoints, multi-identity base, and render-ready datasets live in
> the sibling archive **bytical-2d-synctalk-archive** (private).

---

## Roadmap

Five pillars (see `ROADMAP.md`): **quality** (HuBERT ✓, FiLM multi-scale audio,
attention fusion, temporal loss, super-res), **speed** (ONNX/TensorRT, fp16),
**expressiveness** (emotion conditioning, gestures, prosody), **self-learning**
(auto-QC ✓, hard-example mining, few-shot per-presenter adaptation), and optional,
consent-gated **swaps** (background, face, voice, clothes — all default OFF).

---

## Credits & license

- Renderer: [ZiqiaoPeng/SyncTalk_2D](https://github.com/ZiqiaoPeng/SyncTalk_2D)
  (based on Ultralight-Digital-Human and SyncTalk) — fetched, not re-hosted.
- `bytical_talk/` (the brain + improvements) is licensed **Apache-2.0**.
- Optional swap features must only be used on media you own or have rights to.