plan.md

Reachy Eleven Agent Plan

Goal

Build a Python Reachy Mini app that uses ElevenLabs ElevenAgents for live voice interaction. The first target is a Wireless Reachy Mini using the system default microphone and speaker.

Technical Approach

• Use the Reachy conversation app template generated by reachy-mini-app-assistant.
• Use uv sync --dev as the default environment workflow and keep uv.lock committed for repeatable installs.
• Make ElevenLabs the default provider via REACHY_AGENT_PROVIDER=elevenlabs.
• Connect to a public ElevenLabs agent with ELEVENLABS_AGENT_ID; use ELEVENLABS_API_KEY only if the agent is private.
• Run the ElevenLabs Python SDK with default system audio and a tapped output stream that feeds Reachy's speech-motion HeadWobbler.
• Register client tools for gestures: reachy_move_head, reachy_set_antennas, reachy_set_body_yaw, reachy_expression, reachy_sleep, and reachy_wake.
• Keep robot motion on the existing MovementManager queue so tool calls do not directly fight the control loop.

Answers Locked In

• App flavor: Python app.
• Robot type: Wireless.
• ElevenLabs auth: public agent ID.
• Audio routing: system default input/output.
• v1 scope: voice plus robot speech motion and simple client-tool gestures.

Notes

• Publishing failed locally because the current Hugging Face token cannot create Spaces under btuttle; implementation continues locally.
• Live audio needs PyAudio/PortAudio on the target machine. If PyAudio is missing, install system PortAudio headers and then run uv sync --dev --extra audio.
• May 16 reconnect fix: if the Reachy Mini daemon websocket drops during ElevenLabs mode, keep the voice loop alive, reconnect the SDK client in the background, and resume the movement manager with the new client instead of stopping the app.
• May 16 version alignment: pin reachy-mini[wireless-version]==1.7.1 because the robot daemon reports version 1.7.1; run both daemon and app through this repo's uv run environment.
• May 16 catalog readiness: dashboard launches now expose a Gradio settings/control UI for ElevenLabs agent ID, optional API key/user ID, and start/stop/restart controls.
• May 16 dashboard route fix: remove the template static GET / route before mounting Gradio so startup shows the ElevenLabs controls, not the unused OpenAI key page.
• May 16 dashboard queue fix: dashboard callbacks use direct Gradio HTTP execution (queue=False) because the Reachy Control app shell can leave queued events spinning indefinitely.