# ๐ฌ SynapseMedia โ Multimodal Content Engine
**Turn any long-form video into a vertical reel and a CMS-ready blog post โ fully automated.**
[](https://www.python.org/)
[](https://streamlit.io/)
[](https://ai.google.dev/)
[](multimodal-engine/test/)
[](Dockerfile)
[](LICENSE)
[**Live Demo**](#-demo) ยท [**Quick Start**](#-quick-start) ยท [**Architecture**](#-architecture) ยท [**Docker**](#-docker-deployment)
---
## โจ What It Does
Upload an MP4 โ a tutorial, podcast, or product walkthrough. SynapseMedia runs it through a 4-phase AI pipeline and delivers two production-ready assets:
| Output | Description |
|---|---|
| ๐ **Blog Post** | Full Markdown article with YAML front matter, headers, and tables โ ready for WordPress, Dev.to, or Hugo |
| ๐ฑ **Vertical Reels** | 1080ร1920 MP4 clips at 9:16 โ verified for energy score, auto-cropped, ready for TikTok, Reels, or Shorts |
Everything runs with **zero manual editing**. No timeline scrubbing, no copy-pasting transcripts, no cropping by hand.
---
## ๐ฅ Demo
https://github.com/ajme-abes/RAG-Multimodal-SafeAI/tree/main/multimodal-engine/assets/video.mp4
> Upload a video โ click one button โ get a blog post and vertical reels in minutes.
---
## ๐ธ Screenshots
Main Dashboard
Upload panel, interval slider, layout mode selector
|
Generated Blog Post
YAML front matter + full Markdown rendered in-app
|
Vertical Reels
9:16 clips displayed side-by-side with inline playback
|
---
## ๐๏ธ Architecture
The pipeline runs two tracks in parallel โ one for audio, one for video โ then fuses them.
```
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ Uploaded MP4 Video โ
โโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโ
โ
โโโโโโโโโโโดโโโโโโโโโโโ
โผ โผ
TRACK A TRACK B
audio_processor video_processor
โโโโโโโโโโโโโ โโโโโโโโโโโโโโโ
FFmpeg demux FFmpeg keyframe
16kHz PCM WAV extraction (JPEGs)
โ โ
โผ โผ
Gemini 2.5 Flash Gemini 2.5 Flash
StructuredTranscript ChronologicalVisual
(float timestamps) Timeline (float ts)
โ โ
โโโโโโโโโโโโฌโโโโโโโโโโโโ
โ
โผ
workflow_engine.py
โโโโโโโโโโโโโโโโโโโ
Injects both tracks
into Gemini prompt
โ Markdown Blog Post
(YAML front matter)
โ
โผ
agent_optimizer.py
โโโโโโโโโโโโโโโโโโ
Stage 1: Text filter
(Gemini 2.5 Flash)
Top clip candidates
โ
โผ
reel_generator.py
โโโโโโโโโโโโโโโโโ
Stage 2: Video verify
(Gemini 2.5 Pro)
Energy score > 60
Timestamp fine-tune
โ
โผ
video_processor.py
โโโโโโโโโโโโโโโโโโ
FFmpeg 9:16 render
Blurred Stack or
AI Smart Face Crop
โ
โโโโโโโโโโดโโโโโโโโโ
โผ โผ
๐ Blog Post ๐ฑ Vertical Reels
(.md + YAML) (1080ร1920 MP4)
```
### The 4 Phases
| Phase | Module | What Happens |
|---|---|---|
| **1 โ Hear** | `audio_processor.py` | FFmpeg extracts 16kHz mono WAV โ Gemini 2.5 Flash transcribes into `StructuredTranscript` with per-word float timestamps |
| **2 โ See** | `video_processor.py` | FFmpeg samples keyframes at configurable intervals โ Gemini 2.5 Flash maps each frame to a float timestamp in `ChronologicalVisualTimeline` |
| **3 โ Write** | `workflow_engine.py` | Both data streams are serialised and injected into a Gemini prompt โ structured Markdown blog post with YAML front matter |
| **4 โ Cut** | `agent_optimizer.py` โ `reel_generator.py` | Stage 1 text filter finds hook candidates โ Stage 2 uploads raw clips to Gemini 2.5 Pro for energy scoring โ FFmpeg renders verified clips |
---
## โก Key Features
**๐ฏ Zero-Hallucination Timestamps**
Pydantic models enforce `float` types at every pipeline boundary. No `HH:MM:SS` strings, no off-by-one second errors, no silent type mismatches.
**๐ Dual Render Modes**
- **Blurred Stack** โ Scales the full 16:9 frame to fit 9:16, fills margins with a blurred duplicate. Keeps code and UI text fully readable.
- **AI Smart Face Crop** โ Gemini detects speaker position (`left` / `center` / `right`) and centers the crop frame around them.
**๐ก๏ธ Two-Stage Clip Verification**
Every candidate clip is scored by Gemini 2.5 Pro before rendering. Only clips with `audio_energy_score > 60` and `final_relevance_decision = true` make it to FFmpeg.
**โป๏ธ Automatic Fallback Chain**
If Gemini scene analysis fails โ OpenAI GPT-4o-mini takes over โ if that fails too โ a placeholder timeline keeps the pipeline alive.
**โณ Exponential Backoff with Jitter**
Every Gemini call is wrapped in `retry_with_backoff` โ handles 429 / 503 errors automatically with randomised delay to prevent thundering herd.
**โ๏ธ Cloud Asset Cleanup**
All files uploaded to the Gemini Files API are deleted inside `finally` blocks โ no leaked cloud storage, even on exception.
**๐ CMS-Ready Output**
Blog posts include complete YAML front matter: `title`, `slug`, `date`, `tags`, `category`, `description`. Date-prefixed filenames prevent overwrites.
---
## ๐ ๏ธ Tech Stack
| Layer | Tool | Version |
|---|---|---|
| **UI** | Streamlit | 1.57.0 |
| **Audio extraction** | FFmpeg (`pcm_s16le`) | system |
| **Transcription** | Google Gemini 2.5 Flash | via `google-genai` 1.68.0 |
| **Scene analysis** | Google Gemini 2.5 Flash | via `google-genai` 1.68.0 |
| **Vision fallback** | OpenAI GPT-4o-mini | via `openai` 2.37.0 |
| **Clip verification** | Google Gemini 2.5 Pro | via `google-genai` 1.68.0 |
| **Blog synthesis** | Google Gemini 2.5 Flash | via `google-genai` 1.68.0 |
| **Video rendering** | FFmpeg (`libx264` + `aac`) | system |
| **Schema validation** | Pydantic v2 | 2.12.5 |
| **Testing** | pytest | 72 tests, 0 failures |
---
## ๐ Quick Start
### 1. Prerequisites
Install **FFmpeg** and make sure it's on your system PATH:
```bash
# macOS
brew install ffmpeg
# Ubuntu / Debian
sudo apt install ffmpeg
# Windows
winget install ffmpeg
# Verify
ffmpeg -version
```
### 2. Clone & Install
```bash
git clone https://github.com/ajme-abes/RAG-Multimodal-SafeAI
cd synapsemedia/multimodal-engine
# Create a virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install pinned dependencies
pip install -r requirements.txt
```
### 3. Configure API Keys
Copy the example env file and fill in your keys:
```bash
cp .env.example .env
```
Edit `.env`:
```env
GEMINI_API_KEY="your_google_gemini_api_key"
OPENAI_API_KEY="your_openai_api_key"
```
> Get your Gemini key at [aistudio.google.com](https://aistudio.google.com/app/apikey)
> The OpenAI key is optional โ used only as a fallback if Gemini scene analysis fails.
### 4. Run
```bash
streamlit run app/app.py
```
Open **http://localhost:8501** in your browser.
---
## ๐๏ธ Usage Guide
| Step |
Action |
Tip |
| 1 |
Upload an MP4 in the left sidebar |
Works best with videos 3โ60 minutes long |
| 2 |
Set the Downsampling Interval slider |
Use 5s for code tutorials, 10โ15s for interviews |
| 3 |
Choose a Reel Layout |
Blurred Stack for screen recordings, AI Smart Crop for talking-head |
| 4 |
Click ๐ Process Complete AI Workflow |
A 10-min video takes roughly 2โ4 minutes to process |
| 5 |
Download from the output tabs |
Blog Post tab โ Markdown file ยท Reels tab โ MP4 files |
### Output Tabs
| Tab | Content |
|---|---|
| ๐ **Blog Post** | Rendered Markdown with download button |
| ๐ฑ **Mobile Reels** | Inline video player for each 9:16 clip |
| ๐๏ธ **System Logs** | Raw JSON for the transcript and visual timeline |
---
## ๐ณ Docker Deployment
### Build & Run Locally
```bash
docker build -t synapsemedia .
docker run -p 8501:8501 \
-e GEMINI_API_KEY="your_key" \
-e OPENAI_API_KEY="your_key" \
synapsemedia
```
Open **http://localhost:8501**.
### Docker Compose (Recommended)
```yaml
version: "3.9"
services:
synapsemedia:
build: .
ports:
- "8501:8501"
environment:
- GEMINI_API_KEY=${GEMINI_API_KEY}
- OPENAI_API_KEY=${OPENAI_API_KEY}
volumes:
- ./data:/workspace/data
- ./output:/workspace/output
```
```bash
docker compose up --build
```
---
## ๐งช Tests
The test suite runs without any API keys or video files.
```bash
# From the multimodal-engine/ directory
pytest
```
**72 tests ยท 0 failures ยท ~30s runtime**
| File | What It Covers |
|---|---|
| `test_models.py` | Pydantic schema construction, type coercion, JSON roundtrip |
| `test_utils.py` | Retry on 429/503, immediate raise on 400/401, backoff delay doubling |
| `test_video_processor.py` | All 4 FFmpeg render modes, directory cleanup, base64 encoding, input validation |
| `test_agent_optimizer.py` | Slug sanitisation, layout routing, per-clip render calls, error isolation |
| `test_pipeline_integration.py` | All 4 phases mocked end-to-end, fallback chains, full pipeline smoke test |
---
## ๐ Project Structure
```
multimodal-engine/
โโโ app/
โ โโโ app.py # Streamlit dashboard โ pipeline orchestration & UI
โ โโโ models.py # Shared Pydantic schemas (single source of truth)
โ โโโ utils.py # Exponential backoff with jitter
โ โโโ audio_processor.py # Phase 1 โ FFmpeg WAV demux + Gemini transcription
โ โโโ video_processor.py # Phase 2 โ Keyframe extraction + VLM scene analysis + FFmpeg render
โ โโโ clip_extractor.py # Phase 4a โ Stage 1 text-based semantic filter
โ โโโ reel_generator.py # Phase 4b โ Stage 2 multi-modal video verification
โ โโโ workflow_engine.py # Phase 3 โ Blog synthesis + pipeline orchestration
โโโ assets/
โ โโโ dashboard.png # UI screenshot โ main dashboard
โ โโโ mdpage.png # UI screenshot โ blog output tab
โ โโโ reelpage.png # UI screenshot โ reels output tab
โ โโโ video.mp4 # Demo walkthrough video
โโโ data/ # Runtime: uploaded videos, audio, keyframes (gitignored)
โโโ output/ # Runtime: blog posts and rendered reels (gitignored)
โโโ test/
โ โโโ __init__.py
โ โโโ test_models.py
โ โโโ test_utils.py
โ โโโ test_video_processor.py
โ โโโ test_agent_optimizer.py
โ โโโ test_pipeline_integration.py
โโโ .env.example # API key template
โโโ Dockerfile # Production container
โโโ pytest.ini # Test runner config
โโโ requirements.txt # Pinned Python dependencies
```
---
## ๐ Roadmap
- [ ] **Word-level animated subtitles** โ Burn styled captions into frames using ASS/SSA overlay filters
- [ ] **Frame-by-frame face tracking** โ Replace static `left/center/right` with a landmark model for smooth pan-and-scan
- [ ] **Social publishing webhooks** โ Direct push to YouTube Shorts, Instagram Reels, and TikTok APIs
- [ ] **Natural language video search** โ Query timestamps using the `StructuredTranscript` as a semantic index
- [ ] **Configurable resolutions** โ 1080ร1920 (TikTok/Reels), 1080ร1080 (square), custom aspect ratios
- [ ] **Progress persistence** โ Save pipeline state to disk so a page refresh doesn't lose results
---
## โ ๏ธ Known Limitations
- **Gemini file processing delay** โ Uploaded video files enter a `PROCESSING` state before they can be queried. Large clips can take 30โ60 seconds to process server-side.
- **No persistent storage** โ Streamlit session state resets on page refresh. Results are saved to `output/` on disk but not re-loaded automatically.
- **Rate limits** โ Gemini 2.5 Pro (used in Stage 2) has lower rate limits than Flash. If you process many clips back-to-back, the backoff utility will kick in.
---
## ๐ค Contributing
1. Fork the repo
2. Create a feature branch: `git checkout -b feature/my-feature`
3. Make your changes and add tests
4. Run `pytest` โ all tests must pass
5. Open a pull request
---
Built with โค๏ธ using **Gemini 2.5**, **FFmpeg**, and **Streamlit**
โญ Star this repo if it saved you hours of manual editing