Desmond-Dong commited on
Commit
b10ca0f
·
1 Parent(s): 91cfdeb

docs: translate PROJECT_PLAN.md to English (complete)

Browse files
Files changed (1) hide show
  1. PROJECT_PLAN.md +903 -154
PROJECT_PLAN.md CHANGED
@@ -18,10 +18,10 @@ Integrate Home Assistant voice assistant functionality into Reachy Mini Wi-Fi ro
18
  3. **Home Assistant Centralized Management** - All configuration done on Home Assistant side
19
  4. **Motion Feedback** - Provide head movement and antenna animation feedback during voice interaction
20
  5. **Project Constraints** - Strictly follow [Reachy Mini SDK](reachy_mini) architecture design and constraints
21
- 6. **Code Quality** - Follow Python development standards with consistent code style, clear structure, complete comments, comprehensive documentation
22
  7. **Feature Priority** - Voice conversation with Home Assistant is highest priority; other features are auxiliary and must not affect voice conversation functionality or response speed
23
  8. **No LED Functions** - LEDs are hidden inside the robot; all LED control is ignored
24
- 9. **Preserve Functionality** - Any code modifications should optimize while preserving completed features; do not remove features to solve problems
25
 
26
  ## Technical Architecture
27
 
@@ -91,6 +91,7 @@ Integrate Home Assistant voice assistant functionality into Reachy Mini Wi-Fi ro
91
  - [x] Auto-download sound effect files
92
  - [x] No .env configuration file required
93
 
 
94
  ## File List
95
 
96
  ```
@@ -105,7 +106,7 @@ reachy_mini_ha_voice/
105
  │ ├── camera_server.py # MJPEG camera stream server + face tracking
106
  │ ├── head_tracker.py # YOLO face detector
107
  │ ├── motion.py # Motion control (high-level API)
108
- │ ├── movement_manager.py # Unified movement manager (20Hz control loop)
109
  │ ├── models.py # Data models
110
  │ ├── entity.py # ESPHome base entity
111
  │ ├── entity_extensions.py # Extended entity types
@@ -114,96 +115,322 @@ reachy_mini_ha_voice/
114
  │ ├── zeroconf.py # mDNS discovery
115
  │ └── util.py # Utility functions
116
  ├── wakewords/ # Wake word models (auto-download)
 
 
 
 
 
 
117
  ├── sounds/ # Sound effect files (auto-download)
 
 
118
  ├── pyproject.toml # Project configuration
119
  ├── README.md # Documentation
120
  └── PROJECT_PLAN.md # Project plan
121
  ```
122
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
123
  ## Usage Flow
124
 
125
- 1. **Install App** - Install `reachy-mini-ha-voice` from Reachy Mini App Store
126
- 2. **Start App** - App auto-starts ESPHome server (port 6053), auto-downloads required models and sounds
127
- 3. **Connect Home Assistant** - Home Assistant auto-discovers device (mDNS) or manually add via Settings → Devices & Services → Add Integration → ESPHome
128
- 4. **Use Voice Assistant** - Say "Okay Nabu" to wake, speak command, Reachy Mini provides motion feedback
129
 
130
- ---
 
 
 
 
 
 
131
 
132
- ## ESPHome Entity Implementation
 
 
 
133
 
134
- ### Completed Entities Summary
135
 
136
- **Total: 43+ entities implemented**
137
- - Phase 1-4: Basic controls, motor control, pose control, gaze control
138
- - Phase 5-7: Audio sensors, diagnostics, IMU sensors
139
- - Phase 8-12: Emotion control, microphone volume, camera, audio processing
140
- - Phase 13: Sendspin audio output support
141
 
142
- ### Control Entities (Read/Write)
143
 
144
  | Entity Type | Name | Description |
145
  |-------------|------|-------------|
146
- | `Number` | `speaker_volume` | Speaker volume (0-100) |
147
- | `Select` | `motor_mode` | Motor mode (enabled/disabled/gravity_compensation) |
148
- | `Switch` | `motors_enabled` | Motor torque switch |
149
- | `Button` | `wake_up` / `go_to_sleep` | Wake/sleep robot actions |
150
- | `Number` | `head_x/y/z` | Head position control (±50mm) |
151
- | `Number` | `head_roll/pitch/yaw` | Head angle control |
152
- | `Number` | `body_yaw` | Body yaw angle (-160° ~ +160°) |
153
- | `Number` | `antenna_left/right` | Antenna angle control (±90°) |
154
- | `Number` | `look_at_x/y/z` | Gaze point coordinates |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
155
  | `Select` | `emotion` | Emotion selector (Happy/Sad/Angry/Fear/Surprise/Disgust) |
156
  | `Number` | `microphone_volume` | Microphone volume (0-100%) |
 
 
 
 
 
 
157
  | `Switch` | `agc_enabled` | Auto gain control switch |
158
  | `Number` | `agc_max_gain` | AGC max gain (0-30 dB) |
159
  | `Number` | `noise_suppression` | Noise suppression level (0-100%) |
160
- | `Number` | `tap_sensitivity` | Tap detection sensitivity (0.5-4.0g) |
161
- | `Switch` | `sendspin_enabled` | Sendspin switch |
162
 
163
- ### Sensor Entities (Read-only)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
164
 
165
- | Entity Type | Name | Description |
166
- |-------------|------|-------------|
167
- | `Text Sensor` | `daemon_state` | Daemon status |
168
- | `Binary Sensor` | `backend_ready` | Backend ready status |
169
- | `Text Sensor` | `error_message` | Current error message |
170
- | `Sensor` | `doa_angle` | Sound source direction angle |
171
- | `Binary Sensor` | `speech_detected` | Speech detection status |
172
- | `Sensor` | `control_loop_frequency` | Control loop frequency (Hz) |
173
- | `Text Sensor` | `sdk_version` | SDK version |
174
- | `Text Sensor` | `robot_name` | Robot name |
175
- | `Binary Sensor` | `wireless_version` | Wireless version flag |
176
- | `Binary Sensor` | `simulation_mode` | Simulation mode flag |
177
- | `Text Sensor` | `wlan_ip` | Wireless IP address |
178
- | `Sensor` | `imu_accel_x/y/z` | Accelerometer (m/s²) |
179
- | `Sensor` | `imu_gyro_x/y/z` | Gyroscope (rad/s) |
180
- | `Sensor` | `imu_temperature` | IMU temperature (°C) |
181
- | `Binary Sensor` | `echo_cancellation_converged` | Echo cancellation convergence status |
182
- | `Camera` | `camera` | ESPHome Camera entity |
183
- | `Text Sensor` | `sendspin_url` | Sendspin server URL |
184
- | `Binary Sensor` | `sendspin_connected` | Sendspin connection status |
185
 
186
  ---
187
 
188
- ## Voice Assistant Enhancement Features
189
 
190
- ### Phase 14 - Emotion Action Feedback System 🟡 Partial
191
 
192
- **Status**: Basic infrastructure ready, supports manual trigger, uses voice-driven natural micro-movements during conversation
193
 
194
- **Implemented**:
195
- - ✅ Emotion Selector entity (`emotion`)
196
  - ✅ Basic emotion action playback API (`_play_emotion`)
197
  - ✅ Emotion mapping: Happy/Sad/Angry/Fear/Surprise/Disgust
198
- - ✅ Integration with HuggingFace action library
199
- - ✅ SpeechSway system for natural head micro-movements during conversation
200
 
201
  **Design Decisions**:
202
  - 🎯 No auto-play of full emotion actions during conversation to avoid blocking
203
  - 🎯 Use voice-driven head sway (SpeechSway) for natural motion feedback
204
  - 🎯 Emotion actions retained as manual trigger feature via ESPHome entity
205
 
206
- ### Phase 15 - Face Tracking (Replaces DOA) ✅ Complete
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
207
 
208
  **Goal**: Implement natural face tracking so robot looks at speaker during conversation.
209
 
@@ -213,191 +440,657 @@ reachy_mini_ha_voice/
213
  - Reason: DOA inaccurate at wakeup, frequent queries cause daemon crash
214
 
215
  **Implemented Features**:
216
- - ✅ YOLO face detection using `AdamCodd/YOLOv11n-face-detection` model
217
- - Adaptive frame rate: 15fps during conversation, 3fps when idle without face
218
- - ✅ look_at_image() calculates target pose from face position
219
- - Smooth return to neutral position after face lost (1 second)
220
- - face_tracking_offsets as secondary pose overlay
221
- - Model download retry (3 attempts, 5s interval)
222
- - Conversation mode integration with voice assistant state
 
 
 
 
223
 
224
  **Resource Optimization (v0.5.1)**:
225
  - During conversation (listening/thinking/speaking): High-frequency tracking 15fps
226
  - Idle with face detected: High-frequency tracking 15fps
227
- - Idle without face for 10s: Low-power mode 3fps
228
  - Immediately restore high-frequency tracking when face detected
229
 
230
- ### Phase 16 - Cartoon Style Motion Mode 🟡 Partial
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
231
 
232
  **Goal**: Use SDK interpolation techniques for more expressive robot movements.
233
 
234
- **Implemented**:
235
- - 20Hz unified control loop (reduced from 100Hz to prevent daemon crash)
236
- - Pose change detection - only send commands on significant changes (threshold 0.001)
 
 
 
 
 
 
237
  - ✅ State query caching - 100ms TTL, reduces daemon load
238
  - ✅ Smooth interpolation (ease in-out curve)
239
- - ✅ Breathing animation - idle Z-axis micro-movement + antenna sway
240
- - ✅ Command queue mode - thread-safe external API
241
- - ✅ Error throttling - prevents log explosion
242
- - ✅ Connection health monitoring - auto-detect and recover from connection loss
243
 
244
  **Not Implemented**:
245
  - ❌ Dynamic interpolation technique switching (CARTOON/EASE_IN_OUT etc.)
246
  - ❌ Exaggerated cartoon bounce effects
247
 
248
- ### Phase 17 - Antenna Sync Animation During Speech 🟡 Partial
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
249
 
250
  **Goal**: Antennas sway with audio rhythm during TTS playback, simulating "speaking" effect.
251
 
252
- **Implemented**:
253
  - ✅ Voice-driven head sway (`SpeechSwayGenerator`)
254
  - ✅ VAD detection based on audio loudness
255
  - ✅ Multi-frequency sine wave overlay (Lissajous motion)
256
  - ✅ Smooth envelope transitions
257
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
258
  **Not Implemented**:
259
  - ❌ Antenna sway with audio rhythm (currently only head sway)
260
  - ❌ Audio spectrum analysis driven animation
261
 
262
- ### Phase 18 - Visual Gaze Interaction Not Implemented
263
 
264
  **Goal**: Use camera to detect faces for eye contact.
265
 
266
- ### Phase 19 - Gravity Compensation Interactive Mode 🟡 Partial
 
 
 
 
 
 
 
 
 
 
 
 
267
 
268
- **Implemented**:
269
- - ✅ Gravity compensation mode switch (`motor_mode` Select entity)
 
 
 
 
 
 
 
270
 
271
  **Not Implemented**:
272
- - ❌ Teaching mode - record motion trajectory
273
  - ❌ Save/playback custom actions
 
 
 
 
 
 
 
274
 
275
- ### Phase 20 - Environment Awareness Response 🟡 Partial
276
 
277
- **Implemented**:
278
- - ✅ Tap-to-wake enters continuous conversation mode
279
- - ✅ Second tap exits continuous conversation mode
 
 
 
 
 
 
 
 
 
280
 
281
  **Tap-to-wake vs Voice Wake**:
 
282
  | Wake Method | Conversation Mode | Description |
283
  |-------------|-------------------|-------------|
284
  | Voice wake (Okay Nabu) | Single conversation | Need to say wake word for each conversation |
285
  | Tap-to-wake | Continuous conversation | Auto-continue listening after TTS ends, tap again to exit |
286
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
287
  **Not Implemented**:
288
- - ❌ Shake detection - play dizzy action
289
- - ❌ Tilt/fall detection - play help action
290
- - ❌ Long idle - enter sleep animation
291
 
292
- ### Phase 21 - Home Assistant Scene Integration ❌ Not Implemented
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
293
 
294
  ---
295
 
296
- ## Completion Statistics
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
297
 
298
  | Phase | Status | Completion | Notes |
299
  |-------|--------|------------|-------|
300
  | Phase 1-12 | ✅ Complete | 100% | 40 ESPHome entities implemented (Phase 11 LED disabled) |
301
- | Phase 13 | Complete | 100% | Sendspin audio output support |
302
- | Phase 14 | 🟡 Partial | 30% | API infrastructure ready, missing auto-trigger |
303
- | Phase 15 | Complete | 100% | YOLO face tracking fully implemented |
304
- | Phase 16 | 🟡 Partial | 70% | Control loop + pose detection + breathing animation |
305
- | Phase 17 | 🟡 Partial | 50% | Voice-driven head sway implemented |
306
- | Phase 18 | Not done | 10% | Camera implemented, missing face detection |
307
- | Phase 19 | 🟡 Partial | 40% | Mode switch implemented, missing teaching flow |
308
- | Phase 20 | 🟡 Partial | 30% | Tap-to-wake implemented |
309
- | Phase 21 | ❌ Not done | 0% | Not implemented |
310
-
311
- **Overall Completion**: **Phase 1-13: 100%** | **Phase 14-21: ~45%**
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
312
 
313
  ---
314
 
315
- ## Bug Fixes History
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
316
 
317
- ### v0.5.1 Bug Fixes (2026-01-08)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
318
 
319
- #### Issue 1: Music Not Resuming After Voice Conversation
320
  **Problem**: Music doesn't resume after voice conversation ends.
 
321
  **Root Cause**: Sendspin was incorrectly connected to `tts_player` instead of `music_player`.
322
- **Fix**:
 
323
  - `voice_assistant.py`: Sendspin discovery now connects to `music_player`
324
  - `satellite.py`: `duck()`/`unduck()` now call `music_player.pause_sendspin()`/`resume_sendspin()`
325
 
326
- #### Issue 2: tap_sensitivity Not Persisted
 
327
  **Problem**: tap_sensitivity value set in ESPHome lost after restart.
 
328
  **Fix**:
329
  - `models.py`: Added `tap_sensitivity` field to `Preferences` dataclass
330
  - `entity_registry.py`: Entity setter now saves to `preferences.json`
331
  - Load saved value on startup
332
 
333
- #### Issue 3: Audio Conflict During Voice Assistant Wakeup
 
334
  **Problem**: Audio streaming (Sendspin or ESPHome audio) conflicts when voice assistant wakes up.
 
335
  **Fix**:
336
  - `audio_player.py`: Added `pause_sendspin()` and `resume_sendspin()` methods
337
  - `satellite.py`: `duck()` now pauses Sendspin, `unduck()` resumes it
338
  - Improved `pause()` method to actually stop audio output
339
 
340
- #### Issue 4: AttributeError for _camera_server
 
341
  **Problem**: `_set_conversation_mode()` referenced non-existent `_camera_server` attribute.
 
342
  **Fix**: Changed `self._camera_server` to `self.camera_server` (removed underscore prefix)
343
 
344
- #### Issue 5: tap_sensitivity Default Value Wrong
 
345
  **Problem**: tap_sensitivity default was still 2.0g instead of expected 0.5g.
346
- **Fix**: Use `TAP_THRESHOLD_G_DEFAULT` constant as default value
347
 
348
- #### Issue 6: Sendspin Sample Rate Optimization
349
- **Problem**: ReSpeaker hardware I/O is 16kHz (hardware limitation), but Sendspin might try higher sample rates.
350
- **Fix**: Prioritize 16kHz in Sendspin supported formats list to avoid unnecessary resampling
351
 
352
- ### Daemon Crash Fix (2026-01-07)
353
 
354
- **Problem**: `reachy_mini daemon` crashes during long-term operation.
355
 
356
- **Root Cause Analysis**:
357
- 1. Each `set_target()` sends 3 Zenoh messages
358
- 2. Daemon control loop is 50Hz
359
- 3. Previous 20Hz control loop still too high (20Hz × 3 = 60 msg/s > 50Hz capacity)
360
- 4. Pose change threshold too small (0.002) - almost every loop triggers `set_target()`
361
 
362
- **Fix**:
363
- - Control loop frequency: 20Hz → 10Hz
364
- - Pose change threshold: 0.002 → 0.005
365
- - Camera/face tracking frequency: 15fps → 10fps
366
- - IMU polling frequency: 50Hz → 20Hz
367
- - State cache TTL: 1s → 2s
368
-
369
- **Results**:
370
- | Metric | Before | After | Improvement |
371
- |--------|--------|-------|-------------|
372
- | Control loop frequency | 20 Hz | 10 Hz | ↓ 50% |
373
- | Max Zenoh messages | 60 msg/s | 30 msg/s | ↓ 50% |
374
- | Expected stability | Hours before crash | Stable operation | Significant |
375
-
376
- ### Tap-to-Wake and Microphone Sensitivity Fix (2026-01-07)
377
-
378
- **Problems**:
379
- 1. Tap-to-wake blocking - conversation not working properly after tap wake
380
- 2. Low microphone sensitivity - need to be very close for voice recognition
381
-
382
- **Fixes**:
383
- 1. Removed audio playback in `_tap_continue_feedback()` to avoid blocking
384
- 2. Comprehensive microphone optimization:
385
- - AGC enabled with max gain 30dB
386
- - AGC desired level -18dB
387
- - Base microphone gain 2.0x
388
- - Noise suppression reduced to 0.15
389
- - Echo cancellation and high-pass filter enabled
390
-
391
- **Results**:
392
- | Parameter | Before | After |
393
- |-----------|--------|-------|
394
- | Microphone sensitivity | ~30cm | ~2-3m |
395
- | AGC max gain | ~15dB | 30dB |
396
- | Noise suppression | ~0.5 | 0.15 |
397
 
398
  ---
399
 
400
- ## SDK Data Structure Reference
401
 
402
  ```python
403
  # Motor control mode
@@ -415,6 +1108,23 @@ class DaemonState(Enum):
415
  STOPPED = "stopped"
416
  ERROR = "error"
417
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
418
  # Safety limits
419
  HEAD_PITCH_ROLL_LIMIT = [-40°, +40°]
420
  HEAD_YAW_LIMIT = [-180°, +180°]
@@ -422,6 +1132,45 @@ BODY_YAW_LIMIT = [-160°, +160°]
422
  YAW_DELTA_MAX = 65° # Max difference between head and body yaw
423
  ```
424
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
425
  ## Reference Projects
426
 
427
  - [OHF-Voice/linux-voice-assistant](https://github.com/OHF-Voice/linux-voice-assistant)
 
18
  3. **Home Assistant Centralized Management** - All configuration done on Home Assistant side
19
  4. **Motion Feedback** - Provide head movement and antenna animation feedback during voice interaction
20
  5. **Project Constraints** - Strictly follow [Reachy Mini SDK](reachy_mini) architecture design and constraints
21
+ 6. **Code Quality** - Follow Python development standards with consistent code style, clear structure, complete comments, comprehensive documentation, high test coverage, high code quality, readability, maintainability, extensibility, and reusability
22
  7. **Feature Priority** - Voice conversation with Home Assistant is highest priority; other features are auxiliary and must not affect voice conversation functionality or response speed
23
  8. **No LED Functions** - LEDs are hidden inside the robot; all LED control is ignored
24
+ 9. **Preserve Functionality** - Any code modifications should optimize while preserving completed features; do not remove features to solve problems. When issues occur, prioritize solving problems after referencing examples, not adding various log outputs
25
 
26
  ## Technical Architecture
27
 
 
91
  - [x] Auto-download sound effect files
92
  - [x] No .env configuration file required
93
 
94
+
95
  ## File List
96
 
97
  ```
 
106
  │ ├── camera_server.py # MJPEG camera stream server + face tracking
107
  │ ├── head_tracker.py # YOLO face detector
108
  │ ├── motion.py # Motion control (high-level API)
109
+ │ ├── movement_manager.py # Unified movement manager (20Hz control loop, optimized to prevent daemon crash)
110
  │ ├── models.py # Data models
111
  │ ├── entity.py # ESPHome base entity
112
  │ ├── entity_extensions.py # Extended entity types
 
115
  │ ├── zeroconf.py # mDNS discovery
116
  │ └── util.py # Utility functions
117
  ├── wakewords/ # Wake word models (auto-download)
118
+ │ ├── okay_nabu.json
119
+ │ ├── okay_nabu.tflite
120
+ │ ├── hey_jarvis.json
121
+ │ ├── hey_jarvis.tflite
122
+ │ ├── stop.json
123
+ │ └── stop.tflite
124
  ├── sounds/ # Sound effect files (auto-download)
125
+ │ ├── wake_word_triggered.flac
126
+ │ └── timer_finished.flac
127
  ├── pyproject.toml # Project configuration
128
  ├── README.md # Documentation
129
  └── PROJECT_PLAN.md # Project plan
130
  ```
131
 
132
+ ## Dependencies
133
+
134
+ ```toml
135
+ dependencies = [
136
+ "reachy-mini", # Reachy Mini SDK
137
+ "sounddevice>=0.4.6", # Audio processing (backup)
138
+ "soundfile>=0.12.0", # Audio file reading
139
+ "numpy>=1.24.0", # Numerical computation
140
+ "pymicro-wakeword>=2.0.0,<3.0.0", # Wake word detection
141
+ "pyopen-wakeword>=1.0.0,<2.0.0", # Backup wake word
142
+ "aioesphomeapi>=42.0.0", # ESPHome protocol
143
+ "zeroconf>=0.100.0", # mDNS discovery
144
+ "scipy>=1.10.0", # Motion control
145
+ "pydantic>=2.0.0", # Data validation
146
+ ]
147
+ ```
148
+
149
  ## Usage Flow
150
 
151
+ 1. **Install App**
152
+ - Install `reachy-mini-ha-voice` from Reachy Mini App Store
 
 
153
 
154
+ 2. **Start App**
155
+ - App auto-starts ESPHome server (port 6053)
156
+ - Auto-downloads required models and sounds
157
+
158
+ 3. **Connect Home Assistant**
159
+ - Home Assistant auto-discovers device (mDNS)
160
+ - Or manually add: Settings → Devices & Services → Add Integration → ESPHome
161
 
162
+ 4. **Use Voice Assistant**
163
+ - Say "Okay Nabu" to wake
164
+ - Speak command
165
+ - Reachy Mini provides motion feedback
166
 
167
+ ## ESPHome Entity Planning
168
 
169
+ Based on deep analysis of Reachy Mini SDK, the following entities are exposed to Home Assistant:
 
 
 
 
170
 
171
+ ### Implemented Entities
172
 
173
  | Entity Type | Name | Description |
174
  |-------------|------|-------------|
175
+ | Media Player | `media_player` | Audio playback control |
176
+ | Voice Assistant | `voice_assistant` | Voice assistant pipeline |
177
+
178
+ ### Implemented Control Entities (Read/Write)
179
+
180
+ #### Phase 1-3: Basic Controls and Pose
181
+
182
+ | ESPHome Entity Type | Name | SDK API | Range/Options | Description |
183
+ |---------------------|------|---------|---------------|-------------|
184
+ | `Number` | `speaker_volume` | `AudioPlayer.set_volume()` | 0-100 | Speaker volume |
185
+ | `Select` | `motor_mode` | `set_motor_control_mode()` | enabled/disabled/gravity_compensation | Motor mode selection |
186
+ | `Switch` | `motors_enabled` | `enable_motors()` / `disable_motors()` | on/off | Motor torque switch |
187
+ | `Button` | `wake_up` | `mini.wake_up()` | - | Wake robot action |
188
+ | `Button` | `go_to_sleep` | `mini.goto_sleep()` | - | Sleep robot action |
189
+ | `Number` | `head_x` | `goto_target(head=...)` | ±50mm | Head X position control |
190
+ | `Number` | `head_y` | `goto_target(head=...)` | ±50mm | Head Y position control |
191
+ | `Number` | `head_z` | `goto_target(head=...)` | ±50mm | Head Z position control |
192
+ | `Number` | `head_roll` | `goto_target(head=...)` | -40° ~ +40° | Head roll angle control |
193
+ | `Number` | `head_pitch` | `goto_target(head=...)` | -40° ~ +40° | Head pitch angle control |
194
+ | `Number` | `head_yaw` | `goto_target(head=...)` | -180° ~ +180° | Head yaw angle control |
195
+ | `Number` | `body_yaw` | `goto_target(body_yaw=...)` | -160° ~ +160° | Body yaw angle control |
196
+ | `Number` | `antenna_left` | `goto_target(antennas=...)` | -90° ~ +90° | Left antenna angle control |
197
+ | `Number` | `antenna_right` | `goto_target(antennas=...)` | -90° ~ +90° | Right antenna angle control |
198
+
199
+ #### Phase 4: Gaze Control
200
+
201
+ | ESPHome Entity Type | Name | SDK API | Range/Options | Description |
202
+ |---------------------|------|---------|---------------|-------------|
203
+ | `Number` | `look_at_x` | `look_at_world(x, y, z)` | World coordinates | Gaze point X coordinate |
204
+ | `Number` | `look_at_y` | `look_at_world(x, y, z)` | World coordinates | Gaze point Y coordinate |
205
+ | `Number` | `look_at_z` | `look_at_world(x, y, z)` | World coordinates | Gaze point Z coordinate |
206
+
207
+
208
+ ### Implemented Sensor Entities (Read-only)
209
+
210
+ #### Phase 1 & 5: Basic Status and Audio Sensors
211
+
212
+ | ESPHome Entity Type | Name | SDK API | Description |
213
+ |---------------------|------|---------|-------------|
214
+ | `Text Sensor` | `daemon_state` | `DaemonStatus.state` | Daemon status |
215
+ | `Binary Sensor` | `backend_ready` | `backend_status.ready` | Backend ready status |
216
+ | `Text Sensor` | `error_message` | `DaemonStatus.error` | Current error message |
217
+ | `Sensor` | `doa_angle` | `DoAInfo.angle` | Sound source direction angle (°) |
218
+ | `Binary Sensor` | `speech_detected` | `DoAInfo.speech_detected` | Speech detection status |
219
+
220
+ #### Phase 6: Diagnostic Information
221
+
222
+ | ESPHome Entity Type | Name | SDK API | Description |
223
+ |---------------------|------|---------|-------------|
224
+ | `Sensor` | `control_loop_frequency` | `control_loop_stats` | Control loop frequency (Hz) |
225
+ | `Text Sensor` | `sdk_version` | `DaemonStatus.version` | SDK version |
226
+ | `Text Sensor` | `robot_name` | `DaemonStatus.robot_name` | Robot name |
227
+ | `Binary Sensor` | `wireless_version` | `DaemonStatus.wireless_version` | Wireless version flag |
228
+ | `Binary Sensor` | `simulation_mode` | `DaemonStatus.simulation_enabled` | Simulation mode flag |
229
+ | `Text Sensor` | `wlan_ip` | `DaemonStatus.wlan_ip` | Wireless IP address |
230
+
231
+ #### Phase 7: IMU Sensors (Wireless version only)
232
+
233
+ | ESPHome Entity Type | Name | SDK API | Description |
234
+ |---------------------|------|---------|-------------|
235
+ | `Sensor` | `imu_accel_x` | `mini.imu["accelerometer"][0]` | X-axis acceleration (m/s²) |
236
+ | `Sensor` | `imu_accel_y` | `mini.imu["accelerometer"][1]` | Y-axis acceleration (m/s²) |
237
+ | `Sensor` | `imu_accel_z` | `mini.imu["accelerometer"][2]` | Z-axis acceleration (m/s²) |
238
+ | `Sensor` | `imu_gyro_x` | `mini.imu["gyroscope"][0]` | X-axis angular velocity (rad/s) |
239
+ | `Sensor` | `imu_gyro_y` | `mini.imu["gyroscope"][1]` | Y-axis angular velocity (rad/s) |
240
+ | `Sensor` | `imu_gyro_z` | `mini.imu["gyroscope"][2]` | Z-axis angular velocity (rad/s) |
241
+ | `Sensor` | `imu_temperature` | `mini.imu["temperature"]` | IMU temperature (°C) |
242
+
243
+ #### Phase 8-12: Extended Features
244
+
245
+ | ESPHome Entity Type | Name | Description |
246
+ |---------------------|------|-------------|
247
  | `Select` | `emotion` | Emotion selector (Happy/Sad/Angry/Fear/Surprise/Disgust) |
248
  | `Number` | `microphone_volume` | Microphone volume (0-100%) |
249
+ | `Camera` | `camera` | ESPHome Camera entity (live preview) |
250
+ | `Number` | `led_brightness` | LED brightness (0-100%) |
251
+ | `Select` | `led_effect` | LED effect (off/solid/breathing/rainbow/doa) |
252
+ | `Number` | `led_color_r` | LED red component (0-255) |
253
+ | `Number` | `led_color_g` | LED green component (0-255) |
254
+ | `Number` | `led_color_b` | LED blue component (0-255) |
255
  | `Switch` | `agc_enabled` | Auto gain control switch |
256
  | `Number` | `agc_max_gain` | AGC max gain (0-30 dB) |
257
  | `Number` | `noise_suppression` | Noise suppression level (0-100%) |
258
+ | `Binary Sensor` | `echo_cancellation_converged` | Echo cancellation convergence status |
 
259
 
260
+ > **Note**: Head position (x/y/z) and angles (roll/pitch/yaw), body yaw, antenna angles are all **controllable** entities,
261
+ > using `Number` type for bidirectional control. Call `goto_target()` when setting new values, call `get_current_head_pose()` etc. when reading current values.
262
+
263
+ ### Implementation Priority
264
+
265
+ 1. **Phase 1 - Basic Status and Volume** (High Priority) ✅ **Completed**
266
+ - [x] `daemon_state` - Daemon status sensor
267
+ - [x] `backend_ready` - Backend ready status
268
+ - [x] `error_message` - Error message
269
+ - [x] `speaker_volume` - Speaker volume control
270
+
271
+ 2. **Phase 2 - Motor Control** (High Priority) ✅ **Completed**
272
+ - [x] `motors_enabled` - Motor switch
273
+ - [x] `motor_mode` - Motor mode selection (enabled/disabled/gravity_compensation)
274
+ - [x] `wake_up` / `go_to_sleep` - Wake/sleep buttons
275
+
276
+ 3. **Phase 3 - Pose Control** (Medium Priority) ✅ **Completed**
277
+ - [x] `head_x/y/z` - Head position control
278
+ - [x] `head_roll/pitch/yaw` - Head angle control
279
+ - [x] `body_yaw` - Body yaw angle control
280
+ - [x] `antenna_left/right` - Antenna angle control
281
+
282
+ 4. **Phase 4 - Gaze Control** (Medium Priority) ✅ **Completed**
283
+ - [x] `look_at_x/y/z` - Gaze point coordinate control
284
+
285
+ 5. **Phase 5 - Audio Sensors** (Low Priority) ✅ **Completed**
286
+ - [x] `doa_angle` - Sound source direction
287
+ - [x] `speech_detected` - Speech detection
288
+
289
+ 6. **Phase 6 - Diagnostic Information** (Low Priority) ✅ **Completed**
290
+ - [x] `control_loop_frequency` - Control loop frequency
291
+ - [x] `sdk_version` - SDK version
292
+ - [x] `robot_name` - Robot name
293
+ - [x] `wireless_version` - Wireless version flag
294
+ - [x] `simulation_mode` - Simulation mode flag
295
+ - [x] `wlan_ip` - Wireless IP address
296
+
297
+ 7. **Phase 7 - IMU Sensors** (Optional, wireless version only) ✅ **Completed**
298
+ - [x] `imu_accel_x/y/z` - Accelerometer
299
+ - [x] `imu_gyro_x/y/z` - Gyroscope
300
+ - [x] `imu_temperature` - IMU temperature
301
+
302
+ 8. **Phase 8 - Emotion Control** ✅ **Completed**
303
+ - [x] `emotion` - Emotion selector (Happy/Sad/Angry/Fear/Surprise/Disgust)
304
+
305
+ 9. **Phase 9 - Audio Control** ✅ **Completed**
306
+ - [x] `microphone_volume` - Microphone volume control (0-100%)
307
+
308
+ 10. **Phase 10 - Camera Integration** ✅ **Completed**
309
+ - [x] `camera` - ESPHome Camera entity (live preview)
310
+
311
+ 11. **Phase 11 - LED Control** ❌ **Disabled (LEDs hidden inside robot)**
312
+ - [ ] `led_brightness` - LED brightness (0-100%) - Commented out
313
+ - [ ] `led_effect` - LED effect (off/solid/breathing/rainbow/doa) - Commented out
314
+ - [ ] `led_color_r/g/b` - LED RGB color (0-255) - Commented out
315
+
316
+ 12. **Phase 12 - Audio Processing Parameters** ✅ **Completed**
317
+ - [x] `agc_enabled` - Auto gain control switch
318
+ - [x] `agc_max_gain` - AGC max gain (0-30 dB)
319
+ - [x] `noise_suppression` - Noise suppression level (0-100%)
320
+ - [x] `echo_cancellation_converged` - Echo cancellation convergence status (read-only)
321
+
322
+ 13. **Phase 13 - Sendspin Audio Playback Support** ✅ **Completed**
323
+ - [x] `sendspin_enabled` - Sendspin switch (Switch)
324
+ - [x] `sendspin_url` - Sendspin server URL (Text Sensor)
325
+ - [x] `sendspin_connected` - Sendspin connection status (Binary Sensor)
326
+ - [x] AudioPlayer integrates aiosendspin library
327
+ - [x] TTS audio sent to both local speaker and Sendspin server
328
+
329
+ ---
330
+
331
+ ## 🎉 Phase 1-13 Entities Completed!
332
+
333
+ **Total Completed: 43 entities**
334
+ - Phase 1: 4 entities (Basic status and volume)
335
+ - Phase 2: 4 entities (Motor control)
336
+ - Phase 3: 9 entities (Pose control)
337
+ - Phase 4: 3 entities (Gaze control)
338
+ - Phase 5: 2 entities (Audio sensors)
339
+ - Phase 6: 6 entities (Diagnostic information)
340
+ - Phase 7: 7 entities (IMU sensors)
341
+ - Phase 8: 1 entity (Emotion control)
342
+ - Phase 9: 1 entity (Microphone volume)
343
+ - Phase 10: 1 entity (Camera)
344
+ - Phase 11: 0 entities (LED control - Disabled)
345
+ - Phase 12: 4 entities (Audio processing parameters)
346
+ - Phase 13: 3 entities (Sendspin audio output)
347
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
348
 
349
  ---
350
 
351
+ ## 🚀 Voice Assistant Enhancement Features Implementation Status
352
 
353
+ ### Phase 14 - Emotion Action Feedback System (Partial) 🟡
354
 
355
+ **Implementation Status**: Basic infrastructure ready, supports manual trigger, uses voice-driven natural micro-movements during conversation
356
 
357
+ **Implemented Features**:
358
+ - ✅ Phase 8 Emotion Selector entity (`emotion`)
359
  - ✅ Basic emotion action playback API (`_play_emotion`)
360
  - ✅ Emotion mapping: Happy/Sad/Angry/Fear/Surprise/Disgust
361
+ - ✅ Integration with HuggingFace action library (`pollen-robotics/reachy-mini-emotions-library`)
362
+ - ✅ SpeechSway system for natural head micro-movements during conversation (non-blocking)
363
 
364
  **Design Decisions**:
365
  - 🎯 No auto-play of full emotion actions during conversation to avoid blocking
366
  - 🎯 Use voice-driven head sway (SpeechSway) for natural motion feedback
367
  - 🎯 Emotion actions retained as manual trigger feature via ESPHome entity
368
 
369
+ **Not Implemented**:
370
+ - ❌ Auto-trigger emotion actions based on voice assistant response (decided not to implement to avoid blocking)
371
+ - ❌ Intent recognition and emotion matching
372
+ - ❌ Dance action library integration
373
+ - ❌ Context awareness (e.g., weather query - sunny plays happy, rainy plays sad)
374
+
375
+ **Code Locations**:
376
+ - `entity_registry.py:633-658` - Emotion Selector entity
377
+ - `satellite.py:544-574` - `_play_emotion()` method
378
+ - `motion.py:132-156` - Conversation start motion control (uses SpeechSway)
379
+ - `movement_manager.py:541-595` - Move queue management (allows SpeechSway overlay)
380
+
381
+ **Actual Behavior**:
382
+
383
+ | Voice Assistant Event | Actual Action | Implementation Status |
384
+ |----------------------|---------------|----------------------|
385
+ | Wake word detected | Turn toward sound source + nod confirmation | ✅ Implemented |
386
+ | Conversation start | Voice-driven head micro-movements (SpeechSway) | ✅ Implemented |
387
+ | During conversation | Continuous voice-driven micro-movements + breathing animation | ✅ Implemented |
388
+ | Conversation end | Return to neutral position + breathing animation | ✅ Implemented |
389
+ | Manual emotion trigger | Play via ESPHome `emotion` entity | ✅ Implemented |
390
+
391
+ **Technical Details**:
392
+ ```python
393
+ # motion.py - Use SpeechSway instead of full emotion actions during conversation
394
+ def on_speaking_start(self):
395
+ self._is_speaking = True
396
+ self._movement_manager.set_state(RobotState.SPEAKING)
397
+ # SpeechSway automatically generates natural head micro-movements based on audio loudness
398
+ # No full emotion actions played to avoid blocking conversation experience
399
+
400
+ # movement_manager.py - Motion layering system
401
+ # 1. Move queue (emotion actions) - Sets base pose
402
+ # 2. Action (nod/shake etc.) - Overlays on base pose
403
+ # 3. SpeechSway - Voice-driven micro-movements, can coexist with Move
404
+ # 4. Breathing - Idle breathing animation
405
+ ```
406
+
407
+ **Original Plan** (Decided not to implement to avoid blocking conversation):
408
+
409
+ | Voice Assistant Event | Original Planned Action | Reason Not Implemented |
410
+ |----------------------|------------------------|------------------------|
411
+ | Positive response received | Play "happy" action | Full action would block conversation fluency |
412
+ | Negative response received | Play "sad" action | Full action would block conversation fluency |
413
+ | Play music/entertainment | Play "dance" action | Full action would block conversation fluency |
414
+ | Timer completed | Play "alert" action | Full action would block conversation fluency |
415
+ | Error/cannot understand | Play "confused" action | Full action would block conversation fluency |
416
+
417
+ **Manual Emotion Trigger Example**:
418
+ ```yaml
419
+ # Home Assistant automation example - Manual emotion trigger
420
+ automation:
421
+ - alias: "Reachy Good Morning Greeting"
422
+ trigger:
423
+ - platform: time
424
+ at: "07:00:00"
425
+ action:
426
+ - service: select.select_option
427
+ target:
428
+ entity_id: select.reachy_mini_emotion
429
+ data:
430
+ option: "Happy"
431
+ ```
432
+
433
+ ### Phase 15 - Face Tracking (Replaces DOA Sound Source Tracking) ✅ **Completed**
434
 
435
  **Goal**: Implement natural face tracking so robot looks at speaker during conversation.
436
 
 
440
  - Reason: DOA inaccurate at wakeup, frequent queries cause daemon crash
441
 
442
  **Implemented Features**:
443
+
444
+ | Feature | Description | Implementation Location | Status |
445
+ |---------|-------------|------------------------|--------|
446
+ | YOLO face detection | Uses `AdamCodd/YOLOv11n-face-detection` model | `head_tracker.py` | ✅ Implemented |
447
+ | Adaptive frame rate tracking | 15fps during conversation, 3fps when idle without face | `camera_server.py` | ✅ Implemented |
448
+ | look_at_image() | Calculate target pose from face position | `camera_server.py` | ✅ Implemented |
449
+ | Smooth return to neutral | Smooth return within 1 second after face lost | `camera_server.py` | ✅ Implemented |
450
+ | face_tracking_offsets | As secondary pose overlay to motion control | `movement_manager.py` | ✅ Implemented |
451
+ | Voice activity detection | DOA entity still available for speech detection | `DoAInfo.speech_detected` | ✅ Exposed as entity |
452
+ | Model download retry | 3 retries, 5 second interval | `head_tracker.py` | ✅ Implemented |
453
+ | Conversation mode integration | Auto-switch tracking frequency on voice assistant state change | `satellite.py` | ✅ Implemented |
454
 
455
  **Resource Optimization (v0.5.1)**:
456
  - During conversation (listening/thinking/speaking): High-frequency tracking 15fps
457
  - Idle with face detected: High-frequency tracking 15fps
458
+ - Idle without face for 10s: Low-power mode 3fps (only detect if someone appears)
459
  - Immediately restore high-frequency tracking when face detected
460
 
461
+ **Code Locations**:
462
+ - `head_tracker.py` - YOLO face detector (`HeadTracker` class)
463
+ - `camera_server.py:_capture_frames()` - Adaptive frame rate face tracking
464
+ - `camera_server.py:set_conversation_mode()` - Conversation mode switch API
465
+ - `satellite.py:_set_conversation_mode()` - Voice assistant state integration
466
+ - `movement_manager.py:set_face_tracking_offsets()` - Face tracking offset API
467
+
468
+ **Technical Details**:
469
+ ```python
470
+ # camera_server.py - Adaptive frame rate face tracking
471
+ class MJPEGCameraServer:
472
+ def __init__(self):
473
+ self._fps_high = 15 # During conversation/face detected
474
+ self._fps_low = 3 # Idle without face
475
+ self._low_power_threshold = 10.0 # 10s without face switches to low power
476
+
477
+ def _should_run_face_tracking(self, current_time):
478
+ # Conversation mode: Always high-frequency tracking
479
+ if self._in_conversation:
480
+ return True
481
+ # High-frequency mode: Track every frame
482
+ if self._current_fps == self._fps_high:
483
+ return True
484
+ # Low-power mode: Periodic detection
485
+ return time.since_last_check >= 1/self._fps_low
486
+
487
+ # satellite.py - Voice assistant state integration
488
+ def _reachy_on_listening(self):
489
+ self._set_conversation_mode(True) # Start conversation, high-frequency tracking
490
+
491
+ def _reachy_on_idle(self):
492
+ self._set_conversation_mode(False) # End conversation, adaptive tracking
493
+ ```
494
+
495
+
496
+ ### Phase 16 - Cartoon Style Motion Mode (Partial) 🟡
497
 
498
  **Goal**: Use SDK interpolation techniques for more expressive robot movements.
499
 
500
+ **SDK Support**: `InterpolationTechnique` enum
501
+ - `LINEAR` - Linear, mechanical feel
502
+ - `MIN_JERK` - Minimum jerk, natural and smooth (default)
503
+ - `EASE_IN_OUT` - Ease in-out, elegant
504
+ - `CARTOON` - Cartoon style, with bounce effect, lively and cute
505
+
506
+ **Implemented Features**:
507
+ - ✅ 20Hz unified control loop (`movement_manager.py`) - Reduced from 100Hz to prevent daemon crash
508
+ - ✅ Pose change detection - Only send commands on significant changes (threshold 0.001)
509
  - ✅ State query caching - 100ms TTL, reduces daemon load
510
  - ✅ Smooth interpolation (ease in-out curve)
511
+ - ✅ Breathing animation - Idle Z-axis micro-movement + antenna sway (`BreathingAnimation`)
512
+ - ✅ Command queue mode - Thread-safe external API
513
+ - ✅ Error throttling - Prevents log explosion
514
+ - ✅ Connection health monitoring - Auto-detect and recover from connection loss
515
 
516
  **Not Implemented**:
517
  - ❌ Dynamic interpolation technique switching (CARTOON/EASE_IN_OUT etc.)
518
  - ❌ Exaggerated cartoon bounce effects
519
 
520
+ **Code Locations**:
521
+ - `movement_manager.py:192-243` - BreathingAnimation class
522
+ - `movement_manager.py:246-697` - MovementManager class
523
+
524
+ **Scene Implementation Status**:
525
+
526
+ | Scene | Recommended Interpolation | Effect | Status |
527
+ |-------|--------------------------|--------|--------|
528
+ | Wake nod | `CARTOON` | Lively bounce effect | ❌ Not implemented |
529
+ | Thinking head up | `EASE_IN_OUT` | Elegant transition | ✅ Implemented (smooth interpolation) |
530
+ | Speaking micro-movements | `MIN_JERK` | Natural and fluid | ✅ Implemented (SpeechSway) |
531
+ | Error head shake | `CARTOON` | Exaggerated denial | ❌ Not implemented |
532
+ | Return to neutral | `MIN_JERK` | Smooth return | ✅ Implemented |
533
+ | Idle breathing | - | Subtle sense of life | ✅ Implemented (BreathingAnimation) |
534
+
535
+ ### Phase 17 - Antenna Sync Animation During Speech (Partial) 🟡
536
 
537
  **Goal**: Antennas sway with audio rhythm during TTS playback, simulating "speaking" effect.
538
 
539
+ **Implemented Features**:
540
  - ✅ Voice-driven head sway (`SpeechSwayGenerator`)
541
  - ✅ VAD detection based on audio loudness
542
  - ✅ Multi-frequency sine wave overlay (Lissajous motion)
543
  - ✅ Smooth envelope transitions
544
 
545
+ **Code Locations**:
546
+ - `movement_manager.py:124-189` - SpeechSwayGenerator class
547
+ - `motion.py:212-222` - update_audio_loudness() method
548
+
549
+ **Technical Details**:
550
+ ```python
551
+ # Speech sway parameters
552
+ SWAY_A_PITCH_DEG = 3.0 # Pitch amplitude (degrees)
553
+ SWAY_A_YAW_DEG = 2.0 # Yaw amplitude
554
+ SWAY_A_ROLL_DEG = 2.0 # Roll amplitude
555
+ SWAY_F_PITCH = 0.8 # Pitch frequency Hz
556
+ SWAY_F_YAW = 0.6 # Yaw frequency
557
+ SWAY_F_ROLL = 0.5 # Roll frequency
558
+
559
+ # VAD thresholds
560
+ VAD_DB_ON = -35 # Start detection threshold
561
+ VAD_DB_OFF = -45 # Stop detection threshold
562
+ ```
563
+
564
  **Not Implemented**:
565
  - ❌ Antenna sway with audio rhythm (currently only head sway)
566
  - ❌ Audio spectrum analysis driven animation
567
 
568
+ ### Phase 18 - Visual Gaze Interaction (Not Implemented) ❌
569
 
570
  **Goal**: Use camera to detect faces for eye contact.
571
 
572
+ **SDK Support**:
573
+ - `look_at_image(u, v)` - Look at point in image
574
+ - `look_at_world(x, y, z)` - Look at world coordinate point
575
+ - `media.get_frame()` - Get camera frame (✅ Already implemented in `camera_server.py:146`)
576
+
577
+ **Not Implemented Features**:
578
+
579
+ | Feature | Description | Status |
580
+ |---------|-------------|--------|
581
+ | Face detection | Use OpenCV/MediaPipe to detect faces | ❌ Not implemented |
582
+ | Eye tracking | Look at speaker's face during conversation | ❌ Not implemented |
583
+ | Multi-person switching | When multiple people detected, look at current speaker | ❌ Not implemented |
584
+ | Idle scanning | Randomly look around when idle | ❌ Not implemented |
585
 
586
+ ### Phase 19 - Gravity Compensation Interactive Mode (Partial) 🟡
587
+
588
+ **Goal**: Allow users to physically touch and guide robot head for "teaching" style interaction.
589
+
590
+ **SDK Support**: `enable_gravity_compensation()` - Motors enter gravity compensation mode, can be manually moved
591
+
592
+ **Implemented Features**:
593
+ - ✅ Gravity compensation mode switch (`motor_mode` Select entity, option "gravity_compensation")
594
+ - ✅ `reachy_controller.py:236-237` - Gravity compensation API call
595
 
596
  **Not Implemented**:
597
+ - ❌ Teaching mode - Record motion trajectory
598
  - ❌ Save/playback custom actions
599
+ - ❌ Voice command triggered teaching flow
600
+
601
+ **Application Scenarios**:
602
+ - ❌ User says "Let me teach you a move" → Enter gravity compensation mode
603
+ - ❌ User manually moves head → Record motion trajectory
604
+ - ❌ User says "Remember this" → Save action
605
+ - ❌ User says "Do that action again" → Playback recorded action
606
 
607
+ ### Phase 20 - Environment Awareness Response (Partial) 🟡
608
 
609
+ **Goal**: Use IMU sensors to sense environment changes and respond.
610
+
611
+ **SDK Support**:
612
+ - ✅ `mini.imu["accelerometer"]` - Accelerometer (Phase 7 implemented as entity)
613
+ - ✅ `mini.imu["gyroscope"]` - Gyroscope (Phase 7 implemented as entity)
614
+
615
+ **Implemented Features**:
616
+
617
+ | Detection Event | Response Action | Status |
618
+ |-----------------|-----------------|--------|
619
+ | Tap-to-wake | Enter continuous conversation mode | ✅ Implemented |
620
+ | Second tap | Exit continuous conversation mode | ✅ Implemented |
621
 
622
  **Tap-to-wake vs Voice Wake**:
623
+
624
  | Wake Method | Conversation Mode | Description |
625
  |-------------|-------------------|-------------|
626
  | Voice wake (Okay Nabu) | Single conversation | Need to say wake word for each conversation |
627
  | Tap-to-wake | Continuous conversation | Auto-continue listening after TTS ends, tap again to exit |
628
 
629
+ **Technical Implementation**:
630
+ - `tap_detector.py` - IMU acceleration spike detection
631
+ - `satellite.py:_tap_conversation_mode` - Continuous conversation mode flag
632
+ - Threshold: 2.0g (configurable)
633
+ - Cooldown: 1.0s (prevent repeated triggers)
634
+ - Wireless version only
635
+
636
+ ```python
637
+ # satellite.py - Continuous conversation mode
638
+ def wakeup_from_tap(self):
639
+ if self._tap_conversation_mode:
640
+ # Second tap - Exit continuous conversation
641
+ self._tap_conversation_mode = False
642
+ self._reachy_on_idle()
643
+ else:
644
+ # First tap - Enter continuous conversation
645
+ self._tap_conversation_mode = True
646
+ self.send_messages([VoiceAssistantRequest(start=True)])
647
+
648
+ def _tts_finished(self):
649
+ if self._tap_conversation_mode:
650
+ # Continuous conversation mode: Auto-continue listening
651
+ self.send_messages([VoiceAssistantRequest(start=True)])
652
+ ```
653
+
654
  **Not Implemented**:
 
 
 
655
 
656
+ | Detection Event | Response Action | Status |
657
+ |-----------------|-----------------|--------|
658
+ | Being shaken | Play dizzy action + voice "Don't shake me~" | ❌ Not implemented |
659
+ | Tilted/fallen | Play help action + voice "I fell, help me" | ❌ Not implemented |
660
+ | Long idle | Enter sleep animation | ❌ Not implemented |
661
+
662
+ ### Phase 21 - Home Assistant Scene Integration (Not Implemented) ❌
663
+
664
+ **Goal**: Trigger robot actions based on Home Assistant scenes/automations.
665
+
666
+ **Implementation**: Via ESPHome service calls
667
+
668
+ **Not Implemented Scenes**:
669
+
670
+ | HA Scene | Robot Response | Status |
671
+ |----------|----------------|--------|
672
+ | Good morning scene | Play wake action + "Good morning!" | ❌ Not implemented |
673
+ | Good night scene | Play sleep action + "Good night~" | ❌ Not implemented |
674
+ | Someone home | Turn toward door + wave + "Welcome home!" | ❌ Not implemented |
675
+ | Doorbell rings | Turn toward door + alert action | ❌ Not implemented |
676
+ | Play music | Sway with music rhythm | ❌ Not implemented |
677
+
678
+
679
+ ---
680
+
681
+ ## 📊 Feature Implementation Summary
682
+
683
+ ### ✅ Completed Features
684
+
685
+ #### Core Voice Assistant (Phase 1-12)
686
+ - **45+ ESPHome entities** - All implemented
687
+ - **Basic voice interaction** - Wake word detection, STT/TTS integration
688
+ - **Motion feedback** - Nod, shake, gaze and other basic actions
689
+ - **Audio processing** - AGC, noise suppression, echo cancellation
690
+ - **Camera stream** - MJPEG live preview
691
+
692
+ #### Partially Implemented Features (Phase 14-21)
693
+ - **Phase 14** - Emotion action API infrastructure (manual trigger available)
694
+ - **Phase 19** - Gravity compensation mode switch (teaching flow not implemented)
695
+
696
+ ### ❌ Not Implemented Features
697
+
698
+ #### High Priority
699
+ - ~~**Phase 13** - Sendspin audio playback support~~ ✅ **Completed**
700
+ - **Phase 14** - Auto emotion action feedback (needs voice assistant event association)
701
+ - **Phase 15** - Continuous sound source tracking (only turn toward at wakeup)
702
+
703
+ #### Medium Priority
704
+ - **Phase 16** - Cartoon style motion mode (needs dynamic interpolation switching)
705
+ - **Phase 17** - Antenna sync animation
706
+ - **Phase 18** - Face tracking and eye contact interaction
707
+
708
+ #### Low Priority
709
+ - **Phase 19** - Teaching mode record/playback functionality
710
+ - **Phase 20** - IMU environment awareness response
711
+ - **Phase 21** - Home Assistant scene integration
712
 
713
  ---
714
 
715
+ ## Feature Priority Summary (Updated)
716
+
717
+ ### High Priority (Completed ✅)
718
+ - ✅ **Phase 1-12**: Basic ESPHome entities (45+)
719
+ - ✅ Core voice assistant functionality
720
+ - ✅ Basic motion feedback (nod, shake, gaze)
721
+
722
+ ### High Priority (Partial 🟡)
723
+ - 🟡 **Phase 13**: Emotion action feedback system
724
+ - ✅ Emotion Selector entity and API infrastructure
725
+ - ❌ Auto-trigger emotion actions based on voice assistant response
726
+ - ❌ Intent recognition and emotion matching
727
+ - ❌ Dance action library integration
728
+
729
+ ### High Priority (Not Implemented ❌)
730
+ - ❌ **Phase 14**: Smart sound source tracking enhancement
731
+ - ✅ Turn toward sound source at wakeup
732
+ - ❌ Continuous sound source tracking
733
+ - ❌ Multi-person conversation switching
734
+ - ❌ Sound source visualization
735
+
736
+ ### Medium Priority (Partial 🟡)
737
+ - 🟡 **Phase 15**: Cartoon style motion mode
738
+ - ✅ 20Hz unified control loop architecture (optimized to prevent daemon crash)
739
+ - ✅ Pose change detection + state query caching (reduces daemon load)
740
+ - ✅ Smooth interpolation + breathing animation
741
+ - ❌ Dynamic interpolation technique switching (CARTOON etc.)
742
+ - 🟡 **Phase 16**: Antenna sync during speech
743
+ - ✅ Voice-driven head sway (SpeechSwayGenerator)
744
+ - ❌ Antenna sway with audio rhythm
745
+
746
+ ### Medium Priority (Not Implemented ❌)
747
+ - ❌ **Phase 17**: Visual gaze interaction - Eye contact
748
+
749
+ ### Low Priority (Partial 🟡)
750
+ - 🟡 **Phase 18**: Gravity compensation interactive mode
751
+ - ✅ Gravity compensation mode switch
752
+ - ❌ Teaching style interaction (record/playback functionality)
753
+
754
+ ### Low Priority (Not Implemented ❌)
755
+ - ❌ **Phase 19**: Environment awareness response - IMU triggered actions
756
+ - ❌ **Phase 20**: Home Assistant scene integration - Smart home integration
757
+
758
+ ---
759
+
760
+ ## 📈 Completion Statistics
761
 
762
  | Phase | Status | Completion | Notes |
763
  |-------|--------|------------|-------|
764
  | Phase 1-12 | ✅ Complete | 100% | 40 ESPHome entities implemented (Phase 11 LED disabled) |
765
+ | Phase 13 | 🟡 Partial | 30% | API infrastructure ready, missing auto-trigger |
766
+ | Phase 14 | Not done | 20% | Only turn toward at wakeup implemented |
767
+ | Phase 15 | 🟡 Partial | 70% | 20Hz control loop + pose change detection + state cache + breathing animation implemented |
768
+ | Phase 16 | 🟡 Partial | 50% | Voice-driven head sway implemented |
769
+ | Phase 17 | Not done | 10% | Camera implemented, missing face detection |
770
+ | Phase 18 | 🟡 Partial | 40% | Mode switch implemented, missing teaching flow |
771
+ | Phase 19 | Not done | 10% | IMU data exposed, missing trigger logic |
772
+ | Phase 20 | Not done | 0% | Not implemented |
773
+
774
+ **Overall Completion**: **Phase 1-12: 100%** | **Phase 13-20: ~35%**
775
+
776
+
777
+ ---
778
+
779
+ ## 🔧 Daemon Crash Fix (2025-01-05)
780
+
781
+ ### Problem Description
782
+ During long-term operation, `reachy_mini daemon` would crash, causing robot to become unresponsive.
783
+
784
+ ### Root Cause
785
+ 1. **100Hz control loop too frequent** - Calling `robot.set_target()` every 10ms, even when pose hasn't changed
786
+ 2. **Frequent state queries** - Every entity state read calls `get_status()`, `get_current_head_pose()` etc.
787
+ 3. **Missing change detection** - Even when pose hasn't changed, continues sending same commands
788
+ 4. **Zenoh message queue blocking** - Accumulated 150+ messages per second, daemon cannot process in time
789
+
790
+ ### Fix Solution
791
+
792
+ #### 1. Reduce control loop frequency (movement_manager.py)
793
+ ```python
794
+ # Reduced from 100Hz to 20Hz
795
+ CONTROL_LOOP_FREQUENCY_HZ = 20 # 80% reduction in messages
796
+ ```
797
+
798
+ #### 2. Add pose change detection (movement_manager.py)
799
+ ```python
800
+ # Only send commands on significant pose changes
801
+ if self._last_sent_pose is not None:
802
+ max_diff = max(abs(pose[k] - self._last_sent_pose.get(k, 0.0)) for k in pose.keys())
803
+ if max_diff < 0.001: # Threshold: 0.001 rad or 0.001 m
804
+ return # Skip sending
805
+ ```
806
+
807
+ #### 3. State query caching (reachy_controller.py)
808
+ ```python
809
+ # Cache daemon status query results
810
+ self._cache_ttl = 0.1 # 100ms TTL
811
+ self._last_status_query = 0.0
812
+
813
+ def _get_cached_status(self):
814
+ now = time.time()
815
+ if now - self._last_status_query < self._cache_ttl:
816
+ return self._state_cache.get('status') # Use cache
817
+ # ... query and update cache
818
+ ```
819
+
820
+ #### 4. Head pose query caching (reachy_controller.py)
821
+ ```python
822
+ # Cache get_current_head_pose() and get_current_joint_positions() results
823
+ def _get_cached_head_pose(self):
824
+ # Reuse cached results within 100ms
825
+ ```
826
+
827
+ ### Fix Results
828
+
829
+ | Metric | Before Fix | After Fix | Improvement |
830
+ |--------|------------|-----------|-------------|
831
+ | Control message frequency | ~100 msg/s | ~20 msg/s | ↓ 80% |
832
+ | State query frequency | ~50 msg/s | ~5 msg/s | ↓ 90% |
833
+ | Total Zenoh messages | ~150 msg/s | ~25 msg/s | ↓ 83% |
834
+ | Daemon CPU load | Sustained high load | Normal load | Significantly reduced |
835
+ | Expected stability | Crash within hours | Stable for days | Major improvement |
836
+
837
+ ### Related Files
838
+ - `DAEMON_CRASH_FIX_PLAN.md` - Detailed fix plan and test plan
839
+ - `movement_manager.py` - Control loop optimization
840
+ - `reachy_controller.py` - State query caching
841
+
842
+ ### Future Optimization Suggestions
843
+ 1. ⏳ Dynamic frequency adjustment - 50Hz during motion, 5Hz when idle
844
+ 2. ⏳ Batch state queries - Get all states at once
845
+ 3. ⏳ Performance monitoring and alerts - Real-time daemon health monitoring
846
 
847
  ---
848
 
849
+ ## 🔧 Daemon Crash Deep Fix (2026-01-07)
850
+
851
+ ### Problem Description
852
+ During long-term operation, `reachy_mini daemon` still crashes, previous fix not thorough enough.
853
+
854
+ ### Root Cause Analysis
855
+
856
+ Through deep analysis of SDK source code:
857
+
858
+ 1. **Each `set_target()` sends 3 Zenoh messages**
859
+ - `set_target_head_pose()` - 1 message
860
+ - `set_target_antenna_joint_positions()` - 1 message
861
+ - `set_target_body_yaw()` - 1 message
862
+
863
+ 2. **Daemon control loop is 50Hz**
864
+ - See `reachy_mini/daemon/backend/robot/backend.py`: `control_loop_frequency = 50.0`
865
+ - If message send frequency exceeds 50Hz, daemon may not process in time
866
+
867
+ 3. **Previous 20Hz control loop still too high**
868
+ - 20Hz × 3 messages = 60 messages/second
869
+ - Already exceeds daemon's 50Hz processing capacity
870
+
871
+ 4. **Pose change threshold too small (0.002)**
872
+ - Breathing animation, speech sway, face tracking continuously produce tiny changes
873
+ - Almost every loop triggers `set_target()`
874
+
875
+ ### Fix Solution
876
+
877
+ #### 1. Further reduce control loop frequency (movement_manager.py)
878
+ ```python
879
+ # Reduced from 20Hz to 10Hz
880
+ # 10Hz × 3 messages = 30 messages/second, safely below daemon's 50Hz capacity
881
+ CONTROL_LOOP_FREQUENCY_HZ = 10
882
+ ```
883
+
884
+ #### 2. Increase pose change threshold (movement_manager.py)
885
+ ```python
886
+ # Increased from 0.002 to 0.005
887
+ # 0.005 rad ≈ 0.29 degrees, still smooth enough
888
+ self._pose_change_threshold = 0.005
889
+ ```
890
+
891
+ #### 3. Reduce camera/face tracking frequency (camera_server.py)
892
+ ```python
893
+ # Reduced from 15fps to 10fps
894
+ fps: int = 10
895
+ ```
896
+
897
+ #### 4. Reduce IMU polling frequency (tap_detector.py)
898
+ ```python
899
+ # Reduced from 50Hz to 20Hz
900
+ TAP_DETECTION_RATE_HZ = 20
901
+ ```
902
+
903
+ #### 5. Increase state cache TTL (reachy_controller.py)
904
+ ```python
905
+ # Increased from 1 second to 2 seconds
906
+ self._cache_ttl = 2.0
907
+ ```
908
+
909
+ ### Fix Results
910
+
911
+ | Metric | Before (20Hz) | After (10Hz) | Improvement |
912
+ |--------|---------------|--------------|-------------|
913
+ | Control loop frequency | 20 Hz | 10 Hz | ↓ 50% |
914
+ | Max Zenoh messages | 60 msg/s | 30 msg/s | ↓ 50% |
915
+ | Actual messages (with change detection) | ~40 msg/s | ~15 msg/s | ↓ 62% |
916
+ | Face tracking frequency | 15 Hz | 10 Hz | ↓ 33% |
917
+ | IMU polling frequency | 50 Hz | 20 Hz | ↓ 60% |
918
+ | State cache TTL | 1 second | 2 seconds | ↑ 100% |
919
+ | Expected stability | Crash within hours | Stable operation | Major improvement |
920
+
921
+ ### Key Finding
922
+
923
+ Reference `reachy_mini_conversation_app` uses 100Hz control loop, but it's an official app that may have special optimizations or runs on more powerful hardware. Our app needs more conservative settings.
924
+
925
+ ### Related Files
926
+ - `movement_manager.py` - Control loop frequency and pose threshold
927
+ - `camera_server.py` - Face tracking frequency
928
+ - `tap_detector.py` - IMU polling frequency
929
+ - `reachy_controller.py` - State cache TTL
930
+
931
+
932
+ ---
933
 
934
+ ## 🔧 Tap-to-Wake and Microphone Sensitivity Fix (2026-01-07)
935
+
936
+ ### Problem Description
937
+ 1. **Tap-to-wake blocking** - Conversation not working properly after tap wake, blocking issues
938
+ 2. **Low microphone sensitivity** - Need to be very close for voice recognition
939
+
940
+ ### Root Cause
941
+ 1. **Audio playback blocking** - `_tap_continue_feedback()` plays sound in continuous conversation mode, blocking audio stream processing
942
+ 2. **AGC settings not optimized** - ReSpeaker XVF3800 default settings not suitable for distant voice recognition
943
+
944
+ ### Fix Solution
945
+
946
+ #### 1. Remove audio playback in continuous conversation feedback (satellite.py)
947
+ ```python
948
+ def _tap_continue_feedback(self) -> None:
949
+ """Provide feedback when continuing conversation in tap mode.
950
+
951
+ Triggers a nod to indicate ready for next input.
952
+ Sound is NOT played here to avoid blocking audio streaming.
953
+ """
954
+ # NOTE: Do NOT play sound here - it blocks audio streaming
955
+ if self.state.motion_enabled and self.state.motion:
956
+ self.state.motion.on_continue_listening()
957
+ ```
958
+
959
+ #### 2. Add exception handling to tap callback (voice_assistant.py)
960
+ ```python
961
+ def _on_tap_detected(self) -> None:
962
+ """Callback when tap is detected on the robot.
963
+
964
+ NOTE: This is called from the tap_detector background thread.
965
+ """
966
+ try:
967
+ self._state.satellite.wakeup_from_tap()
968
+ # ... motion feedback
969
+ except Exception as e:
970
+ _LOGGER.error("Error in tap detection callback: %s", e)
971
+ ```
972
+
973
+ #### 3. Comprehensive microphone optimization (voice_assistant.py) - Updated 2026-01-07
974
+ ```python
975
+ def _optimize_microphone_settings(self) -> None:
976
+ """Optimize ReSpeaker XVF3800 microphone settings for voice recognition."""
977
+
978
+ # ========== 1. AGC (Automatic Gain Control) Settings ==========
979
+ # Enable AGC for automatic volume normalization
980
+ respeaker.write("PP_AGCONOFF", [1])
981
+
982
+ # Increase AGC max gain for better distant speech pickup (default ~15dB -> 30dB)
983
+ respeaker.write("PP_AGCMAXGAIN", [30.0])
984
+
985
+ # Set AGC desired output level (default ~-25dB -> -18dB for stronger output)
986
+ respeaker.write("PP_AGCDESIREDLEVEL", [-18.0])
987
+
988
+ # Optimize AGC time constant for voice commands
989
+ respeaker.write("PP_AGCTIME", [0.5])
990
+
991
+ # ========== 2. Base Microphone Gain ==========
992
+ # Increase base microphone gain (default 1.0 -> 2.0)
993
+ respeaker.write("AUDIO_MGR_MIC_GAIN", [2.0])
994
+
995
+ # ========== 3. Noise Suppression Settings ==========
996
+ # Reduce noise suppression to preserve quiet speech (default ~0.5 -> 0.15)
997
+ respeaker.write("PP_MIN_NS", [0.15])
998
+ respeaker.write("PP_MIN_NN", [0.15])
999
+
1000
+ # ========== 4. Echo Cancellation & High-pass Filter ==========
1001
+ respeaker.write("PP_ECHOONOFF", [1])
1002
+ respeaker.write("AEC_HPFONOFF", [1])
1003
+ ```
1004
+
1005
+ ### Fix Results
1006
+
1007
+ | Parameter | Before | After | Notes |
1008
+ |-----------|--------|-------|-------|
1009
+ | Tap continuous conversation | Blocking | Working | Removed blocking audio playback |
1010
+ | Microphone sensitivity | ~30cm | ~2-3m | Comprehensive AGC and gain optimization |
1011
+ | AGC switch | Off | On | Auto volume normalization |
1012
+ | AGC max gain | ~15dB | 30dB | Better distant speech pickup |
1013
+ | AGC target level | -25dB | -18dB | Stronger output signal |
1014
+ | Microphone gain | 1.0x | 2.0x | Base gain doubled |
1015
+ | Noise suppression | ~0.5 | 0.15 | Reduced speech mis-suppression |
1016
+ | Echo cancellation | On | On | Maintain clarity during TTS playback |
1017
+ | High-pass filter | Off | On | Remove low-frequency noise |
1018
+
1019
+ ### XVF3800 Parameter Reference
1020
+
1021
+ | Parameter Name | Type | Range | Description |
1022
+ |----------------|------|-------|-------------|
1023
+ | `PP_AGCONOFF` | int32 | 0/1 | AGC switch |
1024
+ | `PP_AGCMAXGAIN` | float | 0-40 dB | AGC max gain |
1025
+ | `PP_AGCDESIREDLEVEL` | float | dB | AGC target output level |
1026
+ | `PP_AGCTIME` | float | seconds | AGC time constant |
1027
+ | `AUDIO_MGR_MIC_GAIN` | float | 0-4.0 | Microphone gain multiplier |
1028
+ | `PP_MIN_NS` | float | 0-1.0 | Minimum noise suppression (lower = less suppression) |
1029
+ | `PP_MIN_NN` | float | 0-1.0 | Minimum noise estimation |
1030
+ | `PP_ECHOONOFF` | int32 | 0/1 | Echo cancellation switch |
1031
+ | `AEC_HPFONOFF` | int32 | 0/1 | High-pass filter switch |
1032
+
1033
+ ### Related Files
1034
+ - `satellite.py` - Removed blocking audio playback
1035
+ - `voice_assistant.py` - Comprehensive microphone optimization
1036
+ - `reachy_controller.py` - AGC entity default value updates
1037
+ - `entity_registry.py` - AGC max gain range update (0-40dB)
1038
+ - `reachy_mini/src/reachy_mini/media/audio_control_utils.py` - SDK reference
1039
+
1040
+ ---
1041
+
1042
+ ## 🔧 v0.5.1 Bug Fixes (2026-01-08)
1043
+
1044
+ ### Issue 1: Music Not Resuming After Voice Conversation
1045
 
 
1046
  **Problem**: Music doesn't resume after voice conversation ends.
1047
+
1048
  **Root Cause**: Sendspin was incorrectly connected to `tts_player` instead of `music_player`.
1049
+
1050
+ **Fix**:
1051
  - `voice_assistant.py`: Sendspin discovery now connects to `music_player`
1052
  - `satellite.py`: `duck()`/`unduck()` now call `music_player.pause_sendspin()`/`resume_sendspin()`
1053
 
1054
+ ### Issue 2: tap_sensitivity Not Persisted
1055
+
1056
  **Problem**: tap_sensitivity value set in ESPHome lost after restart.
1057
+
1058
  **Fix**:
1059
  - `models.py`: Added `tap_sensitivity` field to `Preferences` dataclass
1060
  - `entity_registry.py`: Entity setter now saves to `preferences.json`
1061
  - Load saved value on startup
1062
 
1063
+ ### Issue 3: Audio Conflict During Voice Assistant Wakeup
1064
+
1065
  **Problem**: Audio streaming (Sendspin or ESPHome audio) conflicts when voice assistant wakes up.
1066
+
1067
  **Fix**:
1068
  - `audio_player.py`: Added `pause_sendspin()` and `resume_sendspin()` methods
1069
  - `satellite.py`: `duck()` now pauses Sendspin, `unduck()` resumes it
1070
  - Improved `pause()` method to actually stop audio output
1071
 
1072
+ ### Issue 4: AttributeError for _camera_server
1073
+
1074
  **Problem**: `_set_conversation_mode()` referenced non-existent `_camera_server` attribute.
1075
+
1076
  **Fix**: Changed `self._camera_server` to `self.camera_server` (removed underscore prefix)
1077
 
1078
+ ### Issue 5: tap_sensitivity Default Value Wrong
1079
+
1080
  **Problem**: tap_sensitivity default was still 2.0g instead of expected 0.5g.
 
1081
 
1082
+ **Fix**: Use `TAP_THRESHOLD_G_DEFAULT` constant as default value
 
 
1083
 
1084
+ ### Issue 6: Sendspin Sample Rate Optimization
1085
 
1086
+ **Problem**: ReSpeaker hardware I/O is 16kHz (hardware limitation), but Sendspin might try higher sample rates.
1087
 
1088
+ **Fix**: Prioritize 16kHz in Sendspin supported formats list to avoid unnecessary resampling
 
 
 
 
1089
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1090
 
1091
  ---
1092
 
1093
+ ### SDK Data Structure Reference
1094
 
1095
  ```python
1096
  # Motor control mode
 
1108
  STOPPED = "stopped"
1109
  ERROR = "error"
1110
 
1111
+ # Full state
1112
+ class FullState:
1113
+ control_mode: MotorControlMode
1114
+ head_pose: XYZRPYPose # x, y, z (m), roll, pitch, yaw (rad)
1115
+ head_joints: list[float] # 7 joint angles
1116
+ body_yaw: float
1117
+ antennas_position: list[float] # [right, left]
1118
+ doa: DoAInfo # angle (rad), speech_detected (bool)
1119
+
1120
+ # IMU data (wireless version only)
1121
+ imu_data = {
1122
+ "accelerometer": [x, y, z], # m/s²
1123
+ "gyroscope": [x, y, z], # rad/s
1124
+ "quaternion": [w, x, y, z], # Attitude quaternion
1125
+ "temperature": float # °C
1126
+ }
1127
+
1128
  # Safety limits
1129
  HEAD_PITCH_ROLL_LIMIT = [-40°, +40°]
1130
  HEAD_YAW_LIMIT = [-180°, +180°]
 
1132
  YAW_DELTA_MAX = 65° # Max difference between head and body yaw
1133
  ```
1134
 
1135
+ ### ESPHome Protocol Implementation Notes
1136
+
1137
+ ESPHome protocol communicates with Home Assistant via protobuf messages. The following message types need to be implemented:
1138
+
1139
+ ```python
1140
+ from aioesphomeapi.api_pb2 import (
1141
+ # Number entity (volume/angle control)
1142
+ ListEntitiesNumberResponse,
1143
+ NumberStateResponse,
1144
+ NumberCommandRequest,
1145
+
1146
+ # Select entity (motor mode)
1147
+ ListEntitiesSelectResponse,
1148
+ SelectStateResponse,
1149
+ SelectCommandRequest,
1150
+
1151
+ # Button entity (wake/sleep)
1152
+ ListEntitiesButtonResponse,
1153
+ ButtonCommandRequest,
1154
+
1155
+ # Switch entity (motor switch)
1156
+ ListEntitiesSwitchResponse,
1157
+ SwitchStateResponse,
1158
+ SwitchCommandRequest,
1159
+
1160
+ # Sensor entity (numeric sensors)
1161
+ ListEntitiesSensorResponse,
1162
+ SensorStateResponse,
1163
+
1164
+ # Binary Sensor entity (boolean sensors)
1165
+ ListEntitiesBinarySensorResponse,
1166
+ BinarySensorStateResponse,
1167
+
1168
+ # Text Sensor entity (text sensors)
1169
+ ListEntitiesTextSensorResponse,
1170
+ TextSensorStateResponse,
1171
+ )
1172
+ ```
1173
+
1174
  ## Reference Projects
1175
 
1176
  - [OHF-Voice/linux-voice-assistant](https://github.com/OHF-Voice/linux-voice-assistant)