Hugging Face's logo

Spaces:
build-small-hackathon
/
AI-Puppet-Theater
Paused

App Files Community
6
AI-Puppet-Theater / README.md
Bagariariya's picture
Bagariariya
chore: generate data from llm
0528b5b
preview code
|
Raw
History Blame
20.4 kB
metadata 
title: AI Puppet Theater
emoji: 🎭
colorFrom: yellow
colorTo: purple
sdk: gradio
sdk_version: 6.5.1
app_file: app.py
python_version: '3.10'
pinned: false

AI Puppet Theater

One premise becomes a tiny AI improv stage, with a Director, puppet actors, audience chaos, voices, and a traceable behind-the-scenes run.

AI Puppet Theater is a Hugging Face Gradio Space for the Build Small Hackathon, Track 2: An Adventure in Thousand Token Wood. The app turns a short user premise into a compact puppet show: it casts characters, lets an LLM Director manage pacing, gives actors intent and state, accepts audience interruptions, performs simple stage effects, and exports a sanitized trace of the scene.

Table of Contents

Learning path (for contributors)

If you are new to Python or LLMs and want to understand how the Gradio UI connects to the Director / actor loop, read docs/LEARNING_PATH.md. It mirrors the recommended reading order for this repository and includes a UI-to-engine handler map.

Live Demo

  • Hugging Face Space: AI Puppet Theater
  • Demo video: TBD
  • Social post: TBD
  • Field Notes: TBD
  • Trace dataset: TBD

Current Status

Playable demo. Demo video, Field Notes, and trace dataset links are TBD until those artifacts are published.

Recommended Demo Settings

  • Backend: HF API, or Deterministic fallback if the model path is unavailable
  • Show length: Standard
  • Voice: Browser TTS, or Edge TTS if available
  • Flow: use Run One Beat first, throw a prop, then open Behind the Curtain

What It Does

  • Creates a short puppet show from a user premise.
  • Casts distinct puppet actors with goals, moods, memories, secrets, and speaking styles.
  • Uses a Director agent to choose the next speaker, beat type, pacing, stage effect, and finale timing.
  • Lets the audience throw props, summon actors, heckle, or request a finale while the show is running.
  • Renders an animated Gradio stage with active-speaker emphasis, transcript, Director log, Browser TTS, optional Edge TTS, and trace export.
  • Falls back to deterministic generation whenever model calls fail, keeping the demo runnable without setup.

Why It Fits Thousand Token Wood

  • Delightful and strange interactive story: the app is a miniature puppet theater where an ordinary premise can become a surreal, fast-moving scene.
  • AI is load-bearing: the LLM Director and actor agents drive character choice, scene structure, timing, dialogue, and theatrical actions.
  • Original concept: the project is not a chat wrapper; it is a staged multi-agent storytelling toy with audience interventions and performance state.
  • Polished Gradio app: the stage, controls, transcript, voice modes, fallback status, and Behind the Curtain trace are designed for a live demo.

Architecture

The diagrams below are rendered PNGs from Mermaid sources in assets/ so they display correctly on Hugging Face Spaces. Model outputs are treated as validated proposals; deterministic fallbacks keep the show bounded.

System Flow

AI Puppet Theater system flow

Agent Loop

AI Puppet Theater agent loop

Runtime and Fallbacks

AI Puppet Theater runtime and fallbacks

Agent Architecture

Director agent: chooses story phase, next speaker, beat type, stage effect, whether to use a prop, whether to reveal a secret, and when to end the scene. The Director can run through HF API, the experimental local OpenBMB runtime path, or deterministic policy.

Actor agents: respond as individual puppets with persona, goal, mood, memory, intent, secrets, gesture, emotion, and stage-ready short lines.

Theatrical tools: support show actions such as props, entrances, lights, sounds, secret reveals, stage business, and audience chaos. Tool calls are used to make the performance feel agentic, not to add hidden complexity.

Trace and fallbacks: every run records sanitized public events for Director decisions, actor outputs, tool results, audience actions, validation, latency, and fallback reasons. The app avoids exposing tokens, private paths, raw tracebacks, hidden reasoning, or actor secrets.

Features

  • Dynamic act length and progress-based pacing.
  • LLM Director for speaker selection, beat planning, and finale control.
  • Actor intent, memory, mood, state, goals, and secrets.
  • Audience actions: props, summons, heckles, and finale requests.
  • Theatrical tools for stage effects and scene business.
  • Browser TTS with character voices or narrator mode.
  • Optional Edge TTS character voices with graceful fallback.
  • HF API backend with deterministic fallback.
  • Experimental ZeroGPU/local model runtime path.
  • Behind the Curtain log and downloadable trace JSON.

Model and Runtime

This project targets the hackathon's small-model spirit: model-backed paths should stay within the 32B parameter limit.

  • HF API backend: Qwen/Qwen3-4B-Instruct-2507:nscale
  • Experimental local / ZeroGPU runtime path: optional OpenBMB Transformers backend, prepared for ZeroGPU when enabled
  • Fallback: deterministic Director and actor generation, requiring no token or model download

Deterministic mode is the safest demo path. HF API mode is the recommended model-backed path when HF_TOKEN is configured. The local OpenBMB / ZeroGPU path is experimental and may be slow or unavailable on CPU Spaces.

Hackathon Targets

  • Track: An Adventure in Thousand Token Wood
  • Best Agent: Director and actor agents coordinate pacing, intent, state, tools, and scene flow.
  • Best Demo: the app is built as a live Gradio theater, not a notebook or plain chat interface.
  • Sharing is Caring: trace export is designed so runs can be published later as a sanitized dataset.
  • Field Notes: TBD
  • Off-Brand: puppet theater is intentionally playful, strange, and theatrical.
  • Well-Tuned / Llama Champion: Actor SFT dataset, MiniCPM5 Actor LoRA, and Q4_K_M GGUF artifacts are published or prepared for publication; the live Space still defaults to reliable hosted/deterministic runtime paths.

Team

Local Development

Install dependencies:

uv sync

Run the app locally:

uv run python app.py

For live reload while editing app.py, use the Gradio CLI:

uv run gradio app.py

The Blocks instance is named app; Gradio discovers it automatically. If you need to pin the name explicitly:

uv run gradio app.py --demo-name app

Check syntax:

uv run python -m py_compile app.py puppet_theater/*.py

Generate Hugging Face Space requirements:

uv pip compile pyproject.toml -o requirements.txt

If tests are added or changed:

uv run pytest

Configuration

Running modes

Deterministic mode is the default and requires no model token or local model download. Use it for the fastest, most reliable demo path.

The optional local OpenBMB runtime path can generate actor lines or Director decisions locally, but it is experimental and may be slow on CPU Spaces because the model must be loaded into the Space process.

The optional Hugging Face API backend uses remote inference through Hugging Face Inference Providers instead of downloading a model locally. It falls back to deterministic generation when a token is missing, the model is unavailable, the API times out, or the model returns invalid JSON.

Actor Model Backends

The Actor Engine selector supports:

  • Deterministic: default and recommended for hosted demo reliability.
  • Local LoRA Actor model: loads the published build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA adapter on openbmb/MiniCPM5-1B with transformers and peft.
  • Local GGUF Actor model: loads a local quantized GGUF through llama-cpp-python; if ACTOR_GGUF_MODEL_PATH is not set, it downloads minicpm5-actor-q4_k_m.gguf from build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF.
  • Local OpenBMB: the existing experimental local OpenBMB Transformers actor backend.
  • Hugging Face API: the existing remote actor backend using Hugging Face Inference Providers.

The LoRA runtime libraries are included in the app runtime dependencies. The Space requirements.txt also includes llama-cpp-python through the upstream prebuilt CUDA wheel index so the GGUF backend can use GPU offload when the Space image has a compatible wheel. Model weights are still loaded lazily: the first selected LoRA or GGUF beat downloads/loads the model and keeps the loaded backend object in memory for the app lifecycle. If a selected backend fails, times out, errors, or returns invalid JSON, runtime output goes through first-JSON extraction, Actor JSON sanitization, validation, and then falls back to the deterministic actor generator.

Local model defaults are set for demo reliability: LoRA uses ACTOR_LORA_DEVICE=auto, GGUF uses ACTOR_GGUF_N_GPU_LAYERS=-1, and both local actor backends default to a 120 second generation timeout.

Local LoRA example:

ACTOR_MODEL_BACKEND=local_lora \
ACTOR_LORA_BASE_MODEL=openbmb/MiniCPM5-1B \
ACTOR_LORA_ADAPTER=build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-LoRA \
uv run python app.py

Local GGUF example:

ACTOR_MODEL_BACKEND=local_gguf \
uv run python app.py

requirements.txt uses the upstream prebuilt CUDA 12.1 wheel index for llama-cpp-python to avoid source builds where possible on the current GPU Space. If the Space CUDA/runtime image changes, replace /cu121 with the matching CUDA wheel index from the llama-cpp-python docs, such as /cu124, /cu125, /cu130, or /cu132. For a CPU Space, replace /cu121 with /cpu and set ACTOR_GGUF_N_GPU_LAYERS=0. If no compatible prebuilt wheel exists for the Space image, pip may still fall back to a slow native build.

To force a local file instead of downloading from the Hugging Face GGUF repo:

ACTOR_MODEL_BACKEND=local_gguf \
ACTOR_GGUF_MODEL_PATH=finetune/outputs/gguf/minicpm5-actor-q4_k_m.gguf \
uv run python app.py

To point at a different GGUF repo or filename:

ACTOR_MODEL_BACKEND=local_gguf \
ACTOR_GGUF_REPO_ID=build-small-hackathon/AI-Puppet-Theater-MiniCPM5-Actor-GGUF \
ACTOR_GGUF_FILENAME=minicpm5-actor-q4_k_m.gguf \
uv run python app.py

Limitations:

  • Local LoRA needs enough GPU or CPU memory for openbmb/MiniCPM5-1B plus the adapter.
  • Local GGUF needs llama-cpp-python and enough CPU/GPU memory for the quantized file; by default it downloads the configured GGUF from Hugging Face Hub, or uses ACTOR_GGUF_MODEL_PATH when set. The GGUF file is not committed to git.
  • Hosted Spaces may use fallback unless a model backend is explicitly configured.
  • GGUF is prompted with ChatML, matching the best eval format used for the Q4_K_M actor model.

Hugging Face API

Set HF_API_MODEL_ID to choose the remote model:

HF_API_MODEL_ID=Qwen/Qwen3-4B-Instruct-2507:nscale

The default model string includes the provider suffix used by Hugging Face Inference Providers. If that model is unavailable through your selected provider, use another chat-compatible model id from Hugging Face Inference Providers.

Premise casting (title, setting, minimal backdrop_description, director + puppets JSON) requests up to HF_API_SHOW_BIBLE_MAX_TOKENS completion tokens (default 1024, clamped between 256 and 4096). When HF_SETTING_BACKDROP_IMAGE is not disabled (default on) and HF_TOKEN is set, the app generates a JPEG backdrop from the setting sentence via Hugging Face text-to-image (model HF_BACKDROP_T2I_MODEL, default black-forest-labs/FLUX.1-schnell; timeout HF_BACKDROP_T2I_TIMEOUT, default 90s). If that is off or fails after an LLM cast, a follow-up chat completion can still pick a stock backdrop_image_url (up to HF_API_BACKDROP_URL_MAX_TOKENS, default 256). If that JSON was truncated at 200 tokens before, the UI fell back to the default Pip/Mina/Bolt cast; raising this cap fixes incomplete responses.

Configure the token as a Hugging Face Space secret named HF_TOKEN. HUGGINGFACEHUB_API_TOKEN is also supported. Do not commit tokens to the repository, README, app code, logs, or trace output.

For local runs, put HF_TOKEN=... in a file named .env at the repository root (next to app.py). The app loads that file on startup; you do not need to export it manually. A template is in .env.example.

Local deterministic mode and app launch work without HF_TOKEN.

Stage backdrop

The show’s setting string (for example “A sun-dappled forest clearing with soft grass and distant trees, warm and quiet.”) is the narrative stage description. When HF_TOKEN is configured and HF_SETTING_BACKDROP_IMAGE is not set to 0/false/off, the app builds a text-to-image prompt from that setting (wide 16:9 puppet-stage framing, calm center, no text/watermarks) and calls the Hugging Face Inference API. The returned image is stored as an inline JPEG data URL on the session and used as the CSS stage background. If generation is disabled, fails, or there is no token, the app falls back to an LLM-picked stock https URL from backdrop_description, then to keyword-matched stock images in puppet_theater/session.py (_BACKDROP_RULES). Set HF_SETTING_BACKDROP_IMAGE=0 to skip image generation and use URL/keyword flow only.

Voice modes

Voice is optional. Off remains the default and the app works without audio support.

Browser TTS modes use the browser's built-in SpeechSynthesis API:

  • Browser TTS: Character Voices gives each actor a deterministic browser voice style.
  • Browser TTS: Narrator Only reads every latest line with the narrator style.

Edge TTS: Character Voices is experimental. It uses the optional edge-tts package to generate an MP3 for only the latest actor or tool line, then plays it through the Edge TTS audio player. Actor slots map to stable Edge voices, and repeated lines are cached in a temporary directory so they are not regenerated every time. Old generated files are cleaned up automatically.

If edge-tts is unavailable, network generation fails, or the service times out, the app shows a short friendly status and keeps the show running. Use Browser TTS or Off as the fallback. Edge TTS does not expose raw tracebacks, local cache paths, tokens, or secrets in the UI or trace output.

ZeroGPU local inference

This Space keeps sdk: gradio; it does not require Docker for ZeroGPU. To experiment with the local OpenBMB runtime path on ZeroGPU, set the Space hardware to ZeroGPU in the Hugging Face Space settings and set:

USE_ZEROGPU=true

USE_ZEROGPU defaults to false, so the app still starts locally and on regular CPU Spaces without requesting ZeroGPU. When enabled and the spaces package is available, local OpenBMB generation runs through a @spaces.GPU(duration=30) function. The app does not load local models at startup.

The Space requirements are resolved for Python 3.10 compatibility because ZeroGPU builds may use a Python 3.10 base image. The local project also supports Python 3.11. The runtime pins torch==2.8.0, which is the preferred ZeroGPU-compatible Torch version for this Space. llama-cpp-python is included in requirements.txt through its prebuilt CUDA wheel index for GGUF support on the current GPU Space; for CPU Space hardware, switch that index to /cpu and set ACTOR_GGUF_N_GPU_LAYERS=0. If the Space image cannot use the selected wheel, pip may fall back to a slow source build. If Hugging Face reports a different supported Torch version for ZeroGPU, update pyproject.toml, regenerate requirements.txt, and refresh uv.lock.

If ZeroGPU is not enabled, spaces is unavailable, local model loading fails, CUDA is unavailable inside the GPU function, or model output is invalid, the show falls back gracefully. The Hugging Face API backend remains selectable and is the recommended fallback when you have HF_TOKEN; deterministic mode remains the no-token safety path.

The Backend panel reports ZeroGPU enabled status, spaces availability, the Torch version, whether CUDA was visible inside the GPU function after a local generation attempt, and the latest fallback reason.

Trace Export

Every show keeps a lightweight agent trace in memory for the current Gradio session. The collapsed Behind the Curtain panel shows a compact human-readable trace summary with a copy button, plus the Director log. The collapsed Trace / Debug panel shows the full JSON preview.

After creating a show, use Download Trace JSON to save ai-puppet-theater-trace-<session_id>.json. The file contains a stable envelope with app_name, trace_version, session_id, created_at, premise, title, setting, show_length_mode, min/target/max beats, actor backend, Director mode, model id when used, and normalized events.

Trace events use a consistent public shape with event_type, timestamp or step, beat/story phase metadata, speaker, Director decisions, actor intent, actor memory updates, tool requests and results, audience actions, backend/model metadata, latency, validation status, fallback usage, and short fallback reasons where available.

Traces are sanitized so they are safe to publish later as a dataset: they do not include Hugging Face tokens, environment variable values, private local paths, raw tracebacks, hidden reasoning, or private secrets. Errors are reduced to short summaries and model failures are recorded as fallback events.

Fine-Tuning Artifacts

Publishing scripts and model/dataset cards live under finetune/. They are dry-run first and are not part of the Space runtime.

The current best adapter candidate is the v0 MiniCPM5 Actor LoRA. It was merged into openbmb/MiniCPM5-1B, converted to GGUF, and quantized as Q4_K_M for llama.cpp testing.

Best GGUF eval used llama-completion with -no-cnv, reasoning disabled, and chatml prompt formatting. On 40 Actor eval prompts, the Q4_K_M GGUF reached 39/40 extracted JSON parse, 39/40 exact top-level schema, 39/40 sanitized Actor JSON usable, and 0/40 interactive markers.

The v1 hardening run added useful validation/eval tooling but did not outperform v0 on full eval. Runtime use should still keep first-JSON extraction, Actor JSON validation, tool-request sanitization, and deterministic fallback.

Roadmap

  • Add final Space, demo video, social post, Field Notes, and trace dataset links.
  • Add tested example prompts after live demo validation.
  • Publish a small sanitized trace dataset.
  • Improve Field Notes with design decisions, model experiments, and demo learnings.
  • Evaluate whether the GGUF llama.cpp runtime should be integrated after local metrics stay stable.