--- title: AI Puppet Theater emoji: 🎭 colorFrom: yellow colorTo: purple sdk: gradio sdk_version: 6.5.1 app_file: app.py python_version: "3.10" pinned: false --- # AI Puppet Theater One premise becomes a tiny AI improv stage, with a Director, puppet actors, audience chaos, voices, and a traceable behind-the-scenes run. AI Puppet Theater is a Hugging Face Gradio Space for the Build Small Hackathon, Track 2: **An Adventure in Thousand Token Wood**. The app turns a short user premise into a compact puppet show: it casts characters, lets an LLM Director manage pacing, gives actors intent and state, accepts audience interruptions, performs simple stage effects, and exports a sanitized trace of the scene. ## Table of Contents - [Live Demo](#live-demo) - [Current Status](#current-status) - [Recommended Demo Settings](#recommended-demo-settings) - [What It Does](#what-it-does) - [Why It Fits Thousand Token Wood](#why-it-fits-thousand-token-wood) - [Architecture](#architecture) - [Agent Architecture](#agent-architecture) - [Features](#features) - [Model and Runtime](#model-and-runtime) - [Hackathon Targets](#hackathon-targets) - [Team](#team) - [Learning path (for contributors)](#learning-path-for-contributors) - [Local Development](#local-development) - [Configuration](#configuration) - [Actor Model Backends](#actor-model-backends) - [Trace Export](#trace-export) - [Fine-Tuning Artifacts](#fine-tuning-artifacts) - [Roadmap](#roadmap) ## Learning path (for contributors) If you are new to Python or LLMs and want to understand how the Gradio UI connects to the Director / actor loop, read **[docs/LEARNING_PATH.md](docs/LEARNING_PATH.md)**. It mirrors the recommended reading order for this repository and includes a UI-to-engine handler map. ## Live Demo - **Hugging Face Space:** [AI Puppet Theater](https://huggingface.co/spaces/build-small-hackathon/AI-Puppet-Theater) - **Demo video:** TBD - **Social post:** TBD - **Field Notes:** TBD - **Trace dataset:** TBD ## Current Status Playable demo. Demo video, Field Notes, and trace dataset links are TBD until those artifacts are published. ## Recommended Demo Settings - **Backend:** HF API, or Deterministic fallback if the model path is unavailable - **Show length:** Standard - **Voice:** Browser TTS, or Edge TTS if available - **Flow:** use **Run One Beat** first, throw a prop, then open **Behind the Curtain** ## What It Does - Creates a short puppet show from a user premise. - Casts distinct puppet actors with goals, moods, memories, secrets, and speaking styles. - Uses a Director agent to choose the next speaker, beat type, pacing, stage effect, and finale timing. - Lets the audience throw props, summon actors, heckle, or request a finale while the show is running. - Renders an animated Gradio stage with active-speaker emphasis, transcript, Director log, Browser TTS, optional Edge TTS, and trace export. - Falls back to deterministic generation whenever model calls fail, keeping the demo runnable without setup. ## Why It Fits Thousand Token Wood - **Delightful and strange interactive story:** the app is a miniature puppet theater where an ordinary premise can become a surreal, fast-moving scene. - **AI is load-bearing:** the LLM Director and actor agents drive character choice, scene structure, timing, dialogue, and theatrical actions. - **Original concept:** the project is not a chat wrapper; it is a staged multi-agent storytelling toy with audience interventions and performance state. - **Polished Gradio app:** the stage, controls, transcript, voice modes, fallback status, and Behind the Curtain trace are designed for a live demo. ## Architecture The diagrams below are rendered PNGs from Mermaid sources in `assets/` so they display correctly on Hugging Face Spaces. Model outputs are treated as validated proposals; deterministic fallbacks keep the show bounded. ### System Flow ![AI Puppet Theater system flow](assets/architecture-system-flow.png) ### Agent Loop ![AI Puppet Theater agent loop](assets/architecture-agent-loop.png) ### Runtime and Fallbacks ![AI Puppet Theater runtime and fallbacks](assets/architecture-runtime-fallbacks.png) ## Agent Architecture **Director agent:** chooses story phase, next speaker, beat type, stage effect, whether to use a prop, whether to reveal a secret, and when to end the scene. The Director can run through HF API, the experimental local OpenBMB runtime path, or deterministic policy. **Actor agents:** respond as individual puppets with persona, goal, mood, memory, intent, secrets, gesture, emotion, and stage-ready short lines. **Theatrical tools:** support show actions such as props, entrances, lights, sounds, secret reveals, stage business, and audience chaos. Tool calls are used to make the performance feel agentic, not to add hidden complexity. **Trace and fallbacks:** every run records sanitized public events for Director decisions, actor outputs, tool results, audience actions, validation, latency, and fallback reasons. The app avoids exposing tokens, private paths, raw tracebacks, hidden reasoning, or actor secrets. ## Features - Dynamic act length and progress-based pacing. - LLM Director for speaker selection, beat planning, and finale control. - Actor intent, memory, mood, state, goals, and secrets. - Audience actions: props, summons, heckles, and finale requests. - Theatrical tools for stage effects and scene business. - Browser TTS with character voices or narrator mode. - Optional Edge TTS character voices with graceful fallback. - HF API backend with deterministic fallback. - Experimental ZeroGPU/local model runtime path. - Behind the Curtain log and downloadable trace JSON. ## Model and Runtime This project targets the hackathon's small-model spirit: model-backed paths should stay within the **32B parameter limit**. - **HF API backend:** `Qwen/Qwen3-4B-Instruct-2507:nscale` - **Experimental local / ZeroGPU runtime path:** optional OpenBMB Transformers backend, prepared for ZeroGPU when enabled - **Fallback:** deterministic Director and actor generation, requiring no token or model download Deterministic mode is the safest demo path. HF API mode is the recommended model-backed path when `HF_TOKEN` is configured. The local OpenBMB / ZeroGPU path is experimental and may be slow or unavailable on CPU Spaces. ## Hackathon Targets - **Track:** An Adventure in Thousand Token Wood - **Best Agent:** Director and actor agents coordinate pacing, intent, state, tools, and scene flow. - **Best Demo:** the app is built as a live Gradio theater, not a notebook or plain chat interface. - **Sharing is Caring:** trace export is designed so runs can be published later as a sanitized dataset. - **Field Notes:** TBD - **Off-Brand:** puppet theater is intentionally playful, strange, and theatrical. - **Well-Tuned / Llama Champion:** Actor SFT dataset, MiniCPM5 Actor LoRA, and Q4_K_M GGUF artifacts are published or prepared for publication; the live Space still defaults to reliable hosted/deterministic runtime paths. ## Team - [Shubham Setia](https://huggingface.co/ShubhamSetia) - [Riya Bagaria](https://huggingface.co/Bagariariya) ## Local Development Install dependencies: ```bash uv sync ``` Run the app locally: ```bash uv run python app.py ``` For live reload while editing `app.py`, use the Gradio CLI: ```bash uv run gradio app.py ``` The Blocks instance is named `app`; Gradio discovers it automatically. If you need to pin the name explicitly: ```bash uv run gradio app.py --demo-name app ``` Check syntax: ```bash uv run python -m py_compile app.py puppet_theater/*.py ``` Generate Hugging Face Space requirements: ```bash uv pip compile pyproject.toml -o requirements.txt ``` If tests are added or changed: ```bash uv run pytest ``` ## Configuration ### Running modes Deterministic mode is the default and requires no model token or local model download. Use it for the fastest, most reliable demo path. The optional local OpenBMB runtime path can generate actor lines or Director decisions locally, but it is experimental and may be slow on CPU Spaces because the model must be loaded into the Space process. The optional Hugging Face API backend uses remote inference through Hugging Face Inference Providers instead of downloading a model locally. It falls back to deterministic generation when a token is missing, the model is unavailable, the API times out, or the model returns invalid JSON. ## Actor Model Backends The **Actor Engine** selector supports: - **Deterministic:** default and recommended for hosted demo reliability. - **Local LoRA Actor model:** loads the published `build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA` adapter on `openbmb/MiniCPM5-1B` with `transformers` and `peft`. - **Local GGUF Actor model:** loads a local quantized GGUF through `llama-cpp-python`; if `ACTOR_GGUF_MODEL_PATH` is not set, it downloads `minicpm5-actor-q4_k_m.gguf` from `build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF`. - **Local OpenBMB:** the existing experimental local OpenBMB Transformers actor backend. - **Hugging Face API:** the existing remote actor backend using Hugging Face Inference Providers. The LoRA runtime libraries are included in the app runtime dependencies. The Space `requirements.txt` also includes `llama-cpp-python` through the upstream prebuilt CUDA wheel index so the GGUF backend can use GPU offload when the Space image has a compatible wheel. Model weights are still loaded lazily: the first selected LoRA or GGUF beat downloads/loads the model and keeps the loaded backend object in memory for the app lifecycle. If a selected backend fails, times out, errors, or returns invalid JSON, runtime output goes through first-JSON extraction, Actor JSON sanitization, validation, and then falls back to the deterministic actor generator. Local model defaults are set for demo reliability: LoRA uses `ACTOR_LORA_DEVICE=auto`, GGUF uses `ACTOR_GGUF_N_GPU_LAYERS=-1`, and both local actor backends default to a 120 second generation timeout. Local LoRA example: ```bash ACTOR_MODEL_BACKEND=local_lora \ ACTOR_LORA_BASE_MODEL=openbmb/MiniCPM5-1B \ ACTOR_LORA_ADAPTER=build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA \ uv run python app.py ``` Local GGUF example: ```bash ACTOR_MODEL_BACKEND=local_gguf \ uv run python app.py ``` `requirements.txt` uses the upstream prebuilt CUDA 12.1 wheel index for `llama-cpp-python` to avoid source builds where possible on the current GPU Space. If the Space CUDA/runtime image changes, replace `/cu121` with the matching CUDA wheel index from the `llama-cpp-python` docs, such as `/cu124`, `/cu125`, `/cu130`, or `/cu132`. For a CPU Space, replace `/cu121` with `/cpu` and set `ACTOR_GGUF_N_GPU_LAYERS=0`. If no compatible prebuilt wheel exists for the Space image, pip may still fall back to a slow native build. To force a local file instead of downloading from the Hugging Face GGUF repo: ```bash ACTOR_MODEL_BACKEND=local_gguf \ ACTOR_GGUF_MODEL_PATH=finetune/outputs/gguf/minicpm5-actor-q4_k_m.gguf \ uv run python app.py ``` To point at a different GGUF repo or filename: ```bash ACTOR_MODEL_BACKEND=local_gguf \ ACTOR_GGUF_REPO_ID=build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF \ ACTOR_GGUF_FILENAME=minicpm5-actor-q4_k_m.gguf \ uv run python app.py ``` Limitations: - Local LoRA needs enough GPU or CPU memory for `openbmb/MiniCPM5-1B` plus the adapter. - Local GGUF needs `llama-cpp-python` and enough CPU/GPU memory for the quantized file; by default it downloads the configured GGUF from Hugging Face Hub, or uses `ACTOR_GGUF_MODEL_PATH` when set. The GGUF file is not committed to git. - Hosted Spaces may use fallback unless a model backend is explicitly configured. - GGUF is prompted with ChatML, matching the best eval format used for the Q4_K_M actor model. ### Hugging Face API Set `HF_API_MODEL_ID` to choose the remote model: ```bash HF_API_MODEL_ID=Qwen/Qwen3-4B-Instruct-2507:nscale ``` The default model string includes the provider suffix used by Hugging Face Inference Providers. If that model is unavailable through your selected provider, use another chat-compatible model id from Hugging Face Inference Providers. Premise casting (title, setting, minimal **backdrop_description**, director + puppets JSON) requests up to **`HF_API_SHOW_BIBLE_MAX_TOKENS`** completion tokens (default **1024**, clamped between 256 and 4096). When **`HF_SETTING_BACKDROP_IMAGE`** is not disabled (default on) and **`HF_TOKEN`** is set, the app generates a **JPEG backdrop from the `setting` sentence** via Hugging Face **text-to-image** (model **`HF_BACKDROP_T2I_MODEL`**, default `black-forest-labs/FLUX.1-schnell`; timeout **`HF_BACKDROP_T2I_TIMEOUT`**, default 90s). If that is off or fails after an LLM cast, a follow-up **chat** completion can still pick a stock **`backdrop_image_url`** (up to **`HF_API_BACKDROP_URL_MAX_TOKENS`**, default **256**). If that JSON was truncated at 200 tokens before, the UI fell back to the default Pip/Mina/Bolt cast; raising this cap fixes incomplete responses. Configure the token as a Hugging Face Space secret named `HF_TOKEN`. `HUGGINGFACEHUB_API_TOKEN` is also supported. Do not commit tokens to the repository, README, app code, logs, or trace output. For **local** runs, put `HF_TOKEN=...` in a file named `.env` at the **repository root** (next to `app.py`). The app loads that file on startup; you do not need to `export` it manually. A template is in [`.env.example`](.env.example). Local deterministic mode and app launch work without `HF_TOKEN`. ### Stage backdrop The show’s **`setting`** string (for example *“A sun-dappled forest clearing with soft grass and distant trees, warm and quiet.”*) is the narrative stage description. When **`HF_TOKEN`** is configured and **`HF_SETTING_BACKDROP_IMAGE`** is not set to `0`/`false`/`off`, the app builds a **text-to-image** prompt from that setting (wide 16:9 puppet-stage framing, calm center, no text/watermarks) and calls the Hugging Face Inference API. The returned image is stored as an **inline JPEG data URL** on the session and used as the CSS stage background. If generation is disabled, fails, or there is no token, the app falls back to an LLM-picked stock **https** URL from **`backdrop_description`**, then to **keyword-matched stock images** in `puppet_theater/session.py` (`_BACKDROP_RULES`). Set **`HF_SETTING_BACKDROP_IMAGE=0`** to skip image generation and use URL/keyword flow only. ### Voice modes Voice is optional. **Off** remains the default and the app works without audio support. Browser TTS modes use the browser's built-in `SpeechSynthesis` API: - **Browser TTS: Character Voices** gives each actor a deterministic browser voice style. - **Browser TTS: Narrator Only** reads every latest line with the narrator style. **Edge TTS: Character Voices** is experimental. It uses the optional `edge-tts` package to generate an MP3 for only the latest actor or tool line, then plays it through the Edge TTS audio player. Actor slots map to stable Edge voices, and repeated lines are cached in a temporary directory so they are not regenerated every time. Old generated files are cleaned up automatically. If `edge-tts` is unavailable, network generation fails, or the service times out, the app shows a short friendly status and keeps the show running. Use Browser TTS or Off as the fallback. Edge TTS does not expose raw tracebacks, local cache paths, tokens, or secrets in the UI or trace output. ### ZeroGPU local inference This Space keeps `sdk: gradio`; it does not require Docker for ZeroGPU. To experiment with the local OpenBMB runtime path on ZeroGPU, set the Space hardware to ZeroGPU in the Hugging Face Space settings and set: ```bash USE_ZEROGPU=true ``` `USE_ZEROGPU` defaults to `false`, so the app still starts locally and on regular CPU Spaces without requesting ZeroGPU. When enabled and the `spaces` package is available, local OpenBMB generation runs through a `@spaces.GPU(duration=30)` function. The app does not load local models at startup. The Space requirements are resolved for Python 3.10 compatibility because ZeroGPU builds may use a Python 3.10 base image. The local project also supports Python 3.11. The runtime pins `torch==2.8.0`, which is the preferred ZeroGPU-compatible Torch version for this Space. `llama-cpp-python` is included in `requirements.txt` through its prebuilt CUDA wheel index for GGUF support on the current GPU Space; for CPU Space hardware, switch that index to `/cpu` and set `ACTOR_GGUF_N_GPU_LAYERS=0`. If the Space image cannot use the selected wheel, pip may fall back to a slow source build. If Hugging Face reports a different supported Torch version for ZeroGPU, update `pyproject.toml`, regenerate `requirements.txt`, and refresh `uv.lock`. If ZeroGPU is not enabled, `spaces` is unavailable, local model loading fails, CUDA is unavailable inside the GPU function, or model output is invalid, the show falls back gracefully. The Hugging Face API backend remains selectable and is the recommended fallback when you have `HF_TOKEN`; deterministic mode remains the no-token safety path. The **Backend** panel reports ZeroGPU enabled status, `spaces` availability, the Torch version, whether CUDA was visible inside the GPU function after a local generation attempt, and the latest fallback reason. ## Trace Export Every show keeps a lightweight agent trace in memory for the current Gradio session. The collapsed **Behind the Curtain** panel shows a compact human-readable trace summary with a copy button, plus the Director log. The collapsed **Trace / Debug** panel shows the full JSON preview. After creating a show, use **Download Trace JSON** to save `ai-puppet-theater-trace-.json`. The file contains a stable envelope with `app_name`, `trace_version`, `session_id`, `created_at`, premise, title, setting, `show_length_mode`, min/target/max beats, actor backend, Director mode, model id when used, and normalized events. Trace events use a consistent public shape with `event_type`, `timestamp` or `step`, beat/story phase metadata, speaker, Director decisions, actor intent, actor memory updates, tool requests and results, audience actions, backend/model metadata, latency, validation status, fallback usage, and short fallback reasons where available. Traces are sanitized so they are safe to publish later as a dataset: they do not include Hugging Face tokens, environment variable values, private local paths, raw tracebacks, hidden reasoning, or private secrets. Errors are reduced to short summaries and model failures are recorded as fallback events. ## Fine-Tuning Artifacts Publishing scripts and model/dataset cards live under `finetune/`. They are dry-run first and are not part of the Space runtime. - **Fine-tuning README:** [finetune/README.md](finetune/README.md) - **Actor SFT dataset repo:** [AI-Puppet-Theater-Actor-SFT](https://huggingface.co/datasets/build-small-hackathon/AI-Puppet-Theater-Actor-SFT) - **MiniCPM5 Actor LoRA repo:** [AI-Puppet-Theater-MiniCPM5-Actor-LoRA](https://huggingface.co/build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA) - **MiniCPM5 Actor GGUF repo:** [AI-Puppet-Theater-MiniCPM5-Actor-GGUF](https://huggingface.co/build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF) The current best adapter candidate is the v0 MiniCPM5 Actor LoRA. It was merged into `openbmb/MiniCPM5-1B`, converted to GGUF, and quantized as `Q4_K_M` for llama.cpp testing. Best GGUF eval used `llama-completion` with `-no-cnv`, reasoning disabled, and `chatml` prompt formatting. On 40 Actor eval prompts, the Q4_K_M GGUF reached 39/40 extracted JSON parse, 39/40 exact top-level schema, 39/40 sanitized Actor JSON usable, and 0/40 interactive markers. The v1 hardening run added useful validation/eval tooling but did not outperform v0 on full eval. Runtime use should still keep first-JSON extraction, Actor JSON validation, tool-request sanitization, and deterministic fallback. ## Roadmap - Add final Space, demo video, social post, Field Notes, and trace dataset links. - Add tested example prompts after live demo validation. - Publish a small sanitized trace dataset. - Improve Field Notes with design decisions, model experiments, and demo learnings. - Evaluate whether the GGUF llama.cpp runtime should be integrated after local metrics stay stable.