Bytical 2D Talk — Smart-AI Talking-Head System
Dimension: 2D · Renderer: SyncTalk_2D (mouth-inpaint lip-sync) This repo = the 2D application (renderer + LLM "brain"). Public.
A talking-head video system that pairs a fast 2D lip-sync renderer with an LLM/embedding-driven brain that understands the script, adapts to any input video, and critiques its own output — so it performs a script instead of just reading it.
Where this sits in the Bytical family
Bytical has two independent talking-head stacks — a 2D one and a 3D one. This repo is the 2D application.
graph TD
ROOT["Bytical Talking-Head Systems"]
ROOT --> TWOD["2D · SyncTalk_2D<br/>(mouth-inpaint lip-sync)"]
ROOT --> THREED["3D · Gaussian Splatting<br/>(TalkingGaussian / InsTaG)"]
TWOD --> A["bytical-2d-talk<br/>system + LLM brain · public"]
TWOD --> B["bytical-2d-synctalk-archive<br/>R&D + trained weights · private"]
THREED --> C["bytical-3d-head<br/>R&D checkpoints P1/P2 · private"]
THREED --> D["bytical-3d-instag-pretrain<br/>multi-identity pretrain · public"]
style A fill:#2563eb,color:#ffffff,stroke:#1e3a8a,stroke-width:3px
| Repo | Dim | Role |
|---|---|---|
| bytical-2d-talk ← you are here | 2D | System + LLM brain (this repo) |
| bytical-2d-synctalk-archive | 2D | R&D lab notebook + trained weights/datasets |
| bytical-3d-head | 3D | Gaussian-Splatting R&D checkpoints (P1/P2) |
| bytical-3d-instag-pretrain | 3D | Multi-identity InsTaG pretrain weights |
Why this exists
Vanilla lip-sync models take (video, audio) → video. That's a renderer, not
intelligence. bytical-2d-talk adds a reasoning layer on top:
| Brain module | Input → Output | What it means |
|---|---|---|
| Director | script → emotion/emphasis/pacing + SSML | reads meaning and decides delivery |
| AutoConfig | any video → render settings | no manual tuning; works on any clip |
| SelfQC | rendered video → pass/fail + fix | the system reviews itself and retries smarter |
The renderer stays weight-safe and swappable; the brain is provider-agnostic (defaults to Azure OpenAI, works with any OpenAI-compatible endpoint).
End-to-end flow
flowchart LR
S["script"] --> DIR["Director<br/>(LLM)"]
DIR --> PLAN["performance plan<br/>emotion timeline + SSML"]
PLAN --> TTS["your TTS<br/>SSML → wav"]
V["input video"] --> AC["AutoConfig<br/>(LLM/CV)"]
AC --> CFG["render settings"]
TTS --> R["render(video, wav, settings)<br/>SyncTalk_2D + improvements"]
CFG --> R
R --> MP4["mp4"]
MP4 --> QC{"SelfQC<br/>pass?"}
QC -- "no (bounded retry)" --> AC
QC -- "yes" --> OUT["final mp4 + QC report"]
style DIR fill:#8b5cf6,color:#fff
style AC fill:#8b5cf6,color:#fff
style QC fill:#f59e0b,color:#111
style OUT fill:#16a34a,color:#fff
Reading the diagram: the brain nodes (purple) are LLM/CV steps that make decisions; the render step is the SyncTalk_2D renderer; SelfQC (amber) closes the loop by re-driving AutoConfig on failure, up to a bounded number of retries.
Package layout
| Path | What it holds |
|---|---|
bytical_talk/brain/ |
llm.py, director.py, autoconfig.py, qc.py |
bytical_talk/render/ |
improved inference: One-Euro crop smoothing, feather paste-back, train/inference resize parity |
bytical_talk/audio/ |
HuBERT features (better generalization to TTS voices) |
bytical_talk/losses/ |
opt-in training upgrades (fixed VGG perceptual, mouth-weighted L1, PatchGAN, LPIPS) |
upstream/synctalk2d/ |
the renderer, fetched by scripts/fetch_upstream.sh (not re-hosted) |
Install
git clone https://github.com/piyushptiwari1/bytical-talk.git
cd bytical-talk
pip install -e . # brain only (light: openai, numpy, pyyaml)
pip install -e ".[render]" # + renderer/audio/CV deps (torch, cv2, transformers, …)
cp .env.example .env # then fill in your keys
bash scripts/fetch_upstream.sh # only needed for rendering
Configure the brain
Edit .env (never committed). Default backend is Azure OpenAI:
BYTICAL_LLM_PROVIDER=azure
AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com/
AZURE_OPENAI_API_KEY=<key>
AZURE_DEPLOYMENT_NAME=gpt-4o-mini
AZURE_EMBEDDING_MODEL_NAME=text-embedding-3-small
Any OpenAI-compatible endpoint works with BYTICAL_LLM_PROVIDER=openai.
Use
# verify credentials + upstream
bytical-talk env-check
# LLM performance plan (no GPU needed)
bytical-talk direct --script "We protect what matters most. Let's find your plan."
# analyze any video -> recommended render settings (needs [render])
bytical-talk autoconfig --video presenter.mp4
# quality review of a rendered clip
bytical-talk qc --video out.mp4
# full pipeline (needs a trained checkpoint + a wav)
bytical-talk generate --script "..." --checkpoint ckpt.pth \
--dataset dataset/presenter --audio speech.wav --out out.mp4 --reference presenter.mp4
Python:
from bytical_talk import Director, auto_config, SelfQC
plan = Director().direct("Hi, I'm here to help you choose the right cover.")
print(plan.ssml) # Polly-ready SSML with emphasis + pauses
print(plan.emotion_timeline()) # per-sentence emotion for the renderer
Training a presenter (renderer)
Any short, front-facing talking clip works. Standard SyncTalk_2D flow, then infer with the improvements:
python upstream/synctalk2d/data_utils/process.py dataset/<name>/<name>.mp4
python bytical_talk/audio/hubert.py --wav_path dataset/<name>/aud.wav --num_frames <N> # for --asr hubert
The trained checkpoints, multi-identity base, and render-ready datasets live in the sibling archive bytical-2d-synctalk-archive (private).
Roadmap
Five pillars (see ROADMAP.md): quality (HuBERT ✓, FiLM multi-scale audio,
attention fusion, temporal loss, super-res), speed (ONNX/TensorRT, fp16),
expressiveness (emotion conditioning, gestures, prosody), self-learning
(auto-QC ✓, hard-example mining, few-shot per-presenter adaptation), and optional,
consent-gated swaps (background, face, voice, clothes — all default OFF).
Credits & license
- Renderer: ZiqiaoPeng/SyncTalk_2D (based on Ultralight-Digital-Human and SyncTalk) — fetched, not re-hosted.
bytical_talk/(the brain + improvements) is licensed Apache-2.0.- Optional swap features must only be used on media you own or have rights to.