Commit ·
b10ca0f
1
Parent(s): 91cfdeb
docs: translate PROJECT_PLAN.md to English (complete)
Browse files- PROJECT_PLAN.md +903 -154
PROJECT_PLAN.md
CHANGED
|
@@ -18,10 +18,10 @@ Integrate Home Assistant voice assistant functionality into Reachy Mini Wi-Fi ro
|
|
| 18 |
3. **Home Assistant Centralized Management** - All configuration done on Home Assistant side
|
| 19 |
4. **Motion Feedback** - Provide head movement and antenna animation feedback during voice interaction
|
| 20 |
5. **Project Constraints** - Strictly follow [Reachy Mini SDK](reachy_mini) architecture design and constraints
|
| 21 |
-
6. **Code Quality** - Follow Python development standards with consistent code style, clear structure, complete comments, comprehensive documentation
|
| 22 |
7. **Feature Priority** - Voice conversation with Home Assistant is highest priority; other features are auxiliary and must not affect voice conversation functionality or response speed
|
| 23 |
8. **No LED Functions** - LEDs are hidden inside the robot; all LED control is ignored
|
| 24 |
-
9. **Preserve Functionality** - Any code modifications should optimize while preserving completed features; do not remove features to solve problems
|
| 25 |
|
| 26 |
## Technical Architecture
|
| 27 |
|
|
@@ -91,6 +91,7 @@ Integrate Home Assistant voice assistant functionality into Reachy Mini Wi-Fi ro
|
|
| 91 |
- [x] Auto-download sound effect files
|
| 92 |
- [x] No .env configuration file required
|
| 93 |
|
|
|
|
| 94 |
## File List
|
| 95 |
|
| 96 |
```
|
|
@@ -105,7 +106,7 @@ reachy_mini_ha_voice/
|
|
| 105 |
│ ├── camera_server.py # MJPEG camera stream server + face tracking
|
| 106 |
│ ├── head_tracker.py # YOLO face detector
|
| 107 |
│ ├── motion.py # Motion control (high-level API)
|
| 108 |
-
│ ├── movement_manager.py # Unified movement manager (20Hz control loop)
|
| 109 |
│ ├── models.py # Data models
|
| 110 |
│ ├── entity.py # ESPHome base entity
|
| 111 |
│ ├── entity_extensions.py # Extended entity types
|
|
@@ -114,96 +115,322 @@ reachy_mini_ha_voice/
|
|
| 114 |
│ ├── zeroconf.py # mDNS discovery
|
| 115 |
│ └── util.py # Utility functions
|
| 116 |
├── wakewords/ # Wake word models (auto-download)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
├── sounds/ # Sound effect files (auto-download)
|
|
|
|
|
|
|
| 118 |
├── pyproject.toml # Project configuration
|
| 119 |
├── README.md # Documentation
|
| 120 |
└── PROJECT_PLAN.md # Project plan
|
| 121 |
```
|
| 122 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 123 |
## Usage Flow
|
| 124 |
|
| 125 |
-
1. **Install App**
|
| 126 |
-
|
| 127 |
-
3. **Connect Home Assistant** - Home Assistant auto-discovers device (mDNS) or manually add via Settings → Devices & Services → Add Integration → ESPHome
|
| 128 |
-
4. **Use Voice Assistant** - Say "Okay Nabu" to wake, speak command, Reachy Mini provides motion feedback
|
| 129 |
|
| 130 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 131 |
|
| 132 |
-
|
|
|
|
|
|
|
|
|
|
| 133 |
|
| 134 |
-
##
|
| 135 |
|
| 136 |
-
|
| 137 |
-
- Phase 1-4: Basic controls, motor control, pose control, gaze control
|
| 138 |
-
- Phase 5-7: Audio sensors, diagnostics, IMU sensors
|
| 139 |
-
- Phase 8-12: Emotion control, microphone volume, camera, audio processing
|
| 140 |
-
- Phase 13: Sendspin audio output support
|
| 141 |
|
| 142 |
-
###
|
| 143 |
|
| 144 |
| Entity Type | Name | Description |
|
| 145 |
|-------------|------|-------------|
|
| 146 |
-
|
|
| 147 |
-
|
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
|
| 154 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 155 |
| `Select` | `emotion` | Emotion selector (Happy/Sad/Angry/Fear/Surprise/Disgust) |
|
| 156 |
| `Number` | `microphone_volume` | Microphone volume (0-100%) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 157 |
| `Switch` | `agc_enabled` | Auto gain control switch |
|
| 158 |
| `Number` | `agc_max_gain` | AGC max gain (0-30 dB) |
|
| 159 |
| `Number` | `noise_suppression` | Noise suppression level (0-100%) |
|
| 160 |
-
| `
|
| 161 |
-
| `Switch` | `sendspin_enabled` | Sendspin switch |
|
| 162 |
|
| 163 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
|
| 165 |
-
| Entity Type | Name | Description |
|
| 166 |
-
|-------------|------|-------------|
|
| 167 |
-
| `Text Sensor` | `daemon_state` | Daemon status |
|
| 168 |
-
| `Binary Sensor` | `backend_ready` | Backend ready status |
|
| 169 |
-
| `Text Sensor` | `error_message` | Current error message |
|
| 170 |
-
| `Sensor` | `doa_angle` | Sound source direction angle |
|
| 171 |
-
| `Binary Sensor` | `speech_detected` | Speech detection status |
|
| 172 |
-
| `Sensor` | `control_loop_frequency` | Control loop frequency (Hz) |
|
| 173 |
-
| `Text Sensor` | `sdk_version` | SDK version |
|
| 174 |
-
| `Text Sensor` | `robot_name` | Robot name |
|
| 175 |
-
| `Binary Sensor` | `wireless_version` | Wireless version flag |
|
| 176 |
-
| `Binary Sensor` | `simulation_mode` | Simulation mode flag |
|
| 177 |
-
| `Text Sensor` | `wlan_ip` | Wireless IP address |
|
| 178 |
-
| `Sensor` | `imu_accel_x/y/z` | Accelerometer (m/s²) |
|
| 179 |
-
| `Sensor` | `imu_gyro_x/y/z` | Gyroscope (rad/s) |
|
| 180 |
-
| `Sensor` | `imu_temperature` | IMU temperature (°C) |
|
| 181 |
-
| `Binary Sensor` | `echo_cancellation_converged` | Echo cancellation convergence status |
|
| 182 |
-
| `Camera` | `camera` | ESPHome Camera entity |
|
| 183 |
-
| `Text Sensor` | `sendspin_url` | Sendspin server URL |
|
| 184 |
-
| `Binary Sensor` | `sendspin_connected` | Sendspin connection status |
|
| 185 |
|
| 186 |
---
|
| 187 |
|
| 188 |
-
## Voice Assistant Enhancement Features
|
| 189 |
|
| 190 |
-
### Phase 14 - Emotion Action Feedback System
|
| 191 |
|
| 192 |
-
**Status**: Basic infrastructure ready, supports manual trigger, uses voice-driven natural micro-movements during conversation
|
| 193 |
|
| 194 |
-
**Implemented**:
|
| 195 |
-
- ✅ Emotion Selector entity (`emotion`)
|
| 196 |
- ✅ Basic emotion action playback API (`_play_emotion`)
|
| 197 |
- ✅ Emotion mapping: Happy/Sad/Angry/Fear/Surprise/Disgust
|
| 198 |
-
- ✅ Integration with HuggingFace action library
|
| 199 |
-
- ✅ SpeechSway system for natural head micro-movements during conversation
|
| 200 |
|
| 201 |
**Design Decisions**:
|
| 202 |
- 🎯 No auto-play of full emotion actions during conversation to avoid blocking
|
| 203 |
- 🎯 Use voice-driven head sway (SpeechSway) for natural motion feedback
|
| 204 |
- 🎯 Emotion actions retained as manual trigger feature via ESPHome entity
|
| 205 |
|
| 206 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 207 |
|
| 208 |
**Goal**: Implement natural face tracking so robot looks at speaker during conversation.
|
| 209 |
|
|
@@ -213,191 +440,657 @@ reachy_mini_ha_voice/
|
|
| 213 |
- Reason: DOA inaccurate at wakeup, frequent queries cause daemon crash
|
| 214 |
|
| 215 |
**Implemented Features**:
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 223 |
|
| 224 |
**Resource Optimization (v0.5.1)**:
|
| 225 |
- During conversation (listening/thinking/speaking): High-frequency tracking 15fps
|
| 226 |
- Idle with face detected: High-frequency tracking 15fps
|
| 227 |
-
- Idle without face for 10s: Low-power mode 3fps
|
| 228 |
- Immediately restore high-frequency tracking when face detected
|
| 229 |
|
| 230 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 231 |
|
| 232 |
**Goal**: Use SDK interpolation techniques for more expressive robot movements.
|
| 233 |
|
| 234 |
-
**
|
| 235 |
-
-
|
| 236 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 237 |
- ✅ State query caching - 100ms TTL, reduces daemon load
|
| 238 |
- ✅ Smooth interpolation (ease in-out curve)
|
| 239 |
-
- ✅ Breathing animation -
|
| 240 |
-
- ✅ Command queue mode -
|
| 241 |
-
- ✅ Error throttling -
|
| 242 |
-
- ✅ Connection health monitoring -
|
| 243 |
|
| 244 |
**Not Implemented**:
|
| 245 |
- ❌ Dynamic interpolation technique switching (CARTOON/EASE_IN_OUT etc.)
|
| 246 |
- ❌ Exaggerated cartoon bounce effects
|
| 247 |
|
| 248 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 249 |
|
| 250 |
**Goal**: Antennas sway with audio rhythm during TTS playback, simulating "speaking" effect.
|
| 251 |
|
| 252 |
-
**Implemented**:
|
| 253 |
- ✅ Voice-driven head sway (`SpeechSwayGenerator`)
|
| 254 |
- ✅ VAD detection based on audio loudness
|
| 255 |
- ✅ Multi-frequency sine wave overlay (Lissajous motion)
|
| 256 |
- ✅ Smooth envelope transitions
|
| 257 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 258 |
**Not Implemented**:
|
| 259 |
- ❌ Antenna sway with audio rhythm (currently only head sway)
|
| 260 |
- ❌ Audio spectrum analysis driven animation
|
| 261 |
|
| 262 |
-
### Phase 18 - Visual Gaze Interaction
|
| 263 |
|
| 264 |
**Goal**: Use camera to detect faces for eye contact.
|
| 265 |
|
| 266 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 267 |
|
| 268 |
-
|
| 269 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 270 |
|
| 271 |
**Not Implemented**:
|
| 272 |
-
- ❌ Teaching mode -
|
| 273 |
- ❌ Save/playback custom actions
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 274 |
|
| 275 |
-
### Phase 20 - Environment Awareness Response
|
| 276 |
|
| 277 |
-
**
|
| 278 |
-
|
| 279 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 280 |
|
| 281 |
**Tap-to-wake vs Voice Wake**:
|
|
|
|
| 282 |
| Wake Method | Conversation Mode | Description |
|
| 283 |
|-------------|-------------------|-------------|
|
| 284 |
| Voice wake (Okay Nabu) | Single conversation | Need to say wake word for each conversation |
|
| 285 |
| Tap-to-wake | Continuous conversation | Auto-continue listening after TTS ends, tap again to exit |
|
| 286 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 287 |
**Not Implemented**:
|
| 288 |
-
- ❌ Shake detection - play dizzy action
|
| 289 |
-
- ❌ Tilt/fall detection - play help action
|
| 290 |
-
- ❌ Long idle - enter sleep animation
|
| 291 |
|
| 292 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 293 |
|
| 294 |
---
|
| 295 |
|
| 296 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 297 |
|
| 298 |
| Phase | Status | Completion | Notes |
|
| 299 |
|-------|--------|------------|-------|
|
| 300 |
| Phase 1-12 | ✅ Complete | 100% | 40 ESPHome entities implemented (Phase 11 LED disabled) |
|
| 301 |
-
| Phase 13 |
|
| 302 |
-
| Phase 14 |
|
| 303 |
-
| Phase 15 |
|
| 304 |
-
| Phase 16 | 🟡 Partial |
|
| 305 |
-
| Phase 17 |
|
| 306 |
-
| Phase 18 |
|
| 307 |
-
| Phase 19 |
|
| 308 |
-
| Phase 20 |
|
| 309 |
-
|
| 310 |
-
|
| 311 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 312 |
|
| 313 |
---
|
| 314 |
|
| 315 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 316 |
|
| 317 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 318 |
|
| 319 |
-
#### Issue 1: Music Not Resuming After Voice Conversation
|
| 320 |
**Problem**: Music doesn't resume after voice conversation ends.
|
|
|
|
| 321 |
**Root Cause**: Sendspin was incorrectly connected to `tts_player` instead of `music_player`.
|
| 322 |
-
|
|
|
|
| 323 |
- `voice_assistant.py`: Sendspin discovery now connects to `music_player`
|
| 324 |
- `satellite.py`: `duck()`/`unduck()` now call `music_player.pause_sendspin()`/`resume_sendspin()`
|
| 325 |
|
| 326 |
-
###
|
|
|
|
| 327 |
**Problem**: tap_sensitivity value set in ESPHome lost after restart.
|
|
|
|
| 328 |
**Fix**:
|
| 329 |
- `models.py`: Added `tap_sensitivity` field to `Preferences` dataclass
|
| 330 |
- `entity_registry.py`: Entity setter now saves to `preferences.json`
|
| 331 |
- Load saved value on startup
|
| 332 |
|
| 333 |
-
###
|
|
|
|
| 334 |
**Problem**: Audio streaming (Sendspin or ESPHome audio) conflicts when voice assistant wakes up.
|
|
|
|
| 335 |
**Fix**:
|
| 336 |
- `audio_player.py`: Added `pause_sendspin()` and `resume_sendspin()` methods
|
| 337 |
- `satellite.py`: `duck()` now pauses Sendspin, `unduck()` resumes it
|
| 338 |
- Improved `pause()` method to actually stop audio output
|
| 339 |
|
| 340 |
-
###
|
|
|
|
| 341 |
**Problem**: `_set_conversation_mode()` referenced non-existent `_camera_server` attribute.
|
|
|
|
| 342 |
**Fix**: Changed `self._camera_server` to `self.camera_server` (removed underscore prefix)
|
| 343 |
|
| 344 |
-
###
|
|
|
|
| 345 |
**Problem**: tap_sensitivity default was still 2.0g instead of expected 0.5g.
|
| 346 |
-
**Fix**: Use `TAP_THRESHOLD_G_DEFAULT` constant as default value
|
| 347 |
|
| 348 |
-
|
| 349 |
-
**Problem**: ReSpeaker hardware I/O is 16kHz (hardware limitation), but Sendspin might try higher sample rates.
|
| 350 |
-
**Fix**: Prioritize 16kHz in Sendspin supported formats list to avoid unnecessary resampling
|
| 351 |
|
| 352 |
-
###
|
| 353 |
|
| 354 |
-
**Problem**:
|
| 355 |
|
| 356 |
-
**
|
| 357 |
-
1. Each `set_target()` sends 3 Zenoh messages
|
| 358 |
-
2. Daemon control loop is 50Hz
|
| 359 |
-
3. Previous 20Hz control loop still too high (20Hz × 3 = 60 msg/s > 50Hz capacity)
|
| 360 |
-
4. Pose change threshold too small (0.002) - almost every loop triggers `set_target()`
|
| 361 |
|
| 362 |
-
**Fix**:
|
| 363 |
-
- Control loop frequency: 20Hz → 10Hz
|
| 364 |
-
- Pose change threshold: 0.002 → 0.005
|
| 365 |
-
- Camera/face tracking frequency: 15fps → 10fps
|
| 366 |
-
- IMU polling frequency: 50Hz → 20Hz
|
| 367 |
-
- State cache TTL: 1s → 2s
|
| 368 |
-
|
| 369 |
-
**Results**:
|
| 370 |
-
| Metric | Before | After | Improvement |
|
| 371 |
-
|--------|--------|-------|-------------|
|
| 372 |
-
| Control loop frequency | 20 Hz | 10 Hz | ↓ 50% |
|
| 373 |
-
| Max Zenoh messages | 60 msg/s | 30 msg/s | ↓ 50% |
|
| 374 |
-
| Expected stability | Hours before crash | Stable operation | Significant |
|
| 375 |
-
|
| 376 |
-
### Tap-to-Wake and Microphone Sensitivity Fix (2026-01-07)
|
| 377 |
-
|
| 378 |
-
**Problems**:
|
| 379 |
-
1. Tap-to-wake blocking - conversation not working properly after tap wake
|
| 380 |
-
2. Low microphone sensitivity - need to be very close for voice recognition
|
| 381 |
-
|
| 382 |
-
**Fixes**:
|
| 383 |
-
1. Removed audio playback in `_tap_continue_feedback()` to avoid blocking
|
| 384 |
-
2. Comprehensive microphone optimization:
|
| 385 |
-
- AGC enabled with max gain 30dB
|
| 386 |
-
- AGC desired level -18dB
|
| 387 |
-
- Base microphone gain 2.0x
|
| 388 |
-
- Noise suppression reduced to 0.15
|
| 389 |
-
- Echo cancellation and high-pass filter enabled
|
| 390 |
-
|
| 391 |
-
**Results**:
|
| 392 |
-
| Parameter | Before | After |
|
| 393 |
-
|-----------|--------|-------|
|
| 394 |
-
| Microphone sensitivity | ~30cm | ~2-3m |
|
| 395 |
-
| AGC max gain | ~15dB | 30dB |
|
| 396 |
-
| Noise suppression | ~0.5 | 0.15 |
|
| 397 |
|
| 398 |
---
|
| 399 |
|
| 400 |
-
## SDK Data Structure Reference
|
| 401 |
|
| 402 |
```python
|
| 403 |
# Motor control mode
|
|
@@ -415,6 +1108,23 @@ class DaemonState(Enum):
|
|
| 415 |
STOPPED = "stopped"
|
| 416 |
ERROR = "error"
|
| 417 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 418 |
# Safety limits
|
| 419 |
HEAD_PITCH_ROLL_LIMIT = [-40°, +40°]
|
| 420 |
HEAD_YAW_LIMIT = [-180°, +180°]
|
|
@@ -422,6 +1132,45 @@ BODY_YAW_LIMIT = [-160°, +160°]
|
|
| 422 |
YAW_DELTA_MAX = 65° # Max difference between head and body yaw
|
| 423 |
```
|
| 424 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 425 |
## Reference Projects
|
| 426 |
|
| 427 |
- [OHF-Voice/linux-voice-assistant](https://github.com/OHF-Voice/linux-voice-assistant)
|
|
|
|
| 18 |
3. **Home Assistant Centralized Management** - All configuration done on Home Assistant side
|
| 19 |
4. **Motion Feedback** - Provide head movement and antenna animation feedback during voice interaction
|
| 20 |
5. **Project Constraints** - Strictly follow [Reachy Mini SDK](reachy_mini) architecture design and constraints
|
| 21 |
+
6. **Code Quality** - Follow Python development standards with consistent code style, clear structure, complete comments, comprehensive documentation, high test coverage, high code quality, readability, maintainability, extensibility, and reusability
|
| 22 |
7. **Feature Priority** - Voice conversation with Home Assistant is highest priority; other features are auxiliary and must not affect voice conversation functionality or response speed
|
| 23 |
8. **No LED Functions** - LEDs are hidden inside the robot; all LED control is ignored
|
| 24 |
+
9. **Preserve Functionality** - Any code modifications should optimize while preserving completed features; do not remove features to solve problems. When issues occur, prioritize solving problems after referencing examples, not adding various log outputs
|
| 25 |
|
| 26 |
## Technical Architecture
|
| 27 |
|
|
|
|
| 91 |
- [x] Auto-download sound effect files
|
| 92 |
- [x] No .env configuration file required
|
| 93 |
|
| 94 |
+
|
| 95 |
## File List
|
| 96 |
|
| 97 |
```
|
|
|
|
| 106 |
│ ├── camera_server.py # MJPEG camera stream server + face tracking
|
| 107 |
│ ├── head_tracker.py # YOLO face detector
|
| 108 |
│ ├── motion.py # Motion control (high-level API)
|
| 109 |
+
│ ├── movement_manager.py # Unified movement manager (20Hz control loop, optimized to prevent daemon crash)
|
| 110 |
│ ├── models.py # Data models
|
| 111 |
│ ├── entity.py # ESPHome base entity
|
| 112 |
│ ├── entity_extensions.py # Extended entity types
|
|
|
|
| 115 |
│ ├── zeroconf.py # mDNS discovery
|
| 116 |
│ └── util.py # Utility functions
|
| 117 |
├── wakewords/ # Wake word models (auto-download)
|
| 118 |
+
│ ├── okay_nabu.json
|
| 119 |
+
│ ├── okay_nabu.tflite
|
| 120 |
+
│ ├── hey_jarvis.json
|
| 121 |
+
│ ├── hey_jarvis.tflite
|
| 122 |
+
│ ├── stop.json
|
| 123 |
+
│ └── stop.tflite
|
| 124 |
├── sounds/ # Sound effect files (auto-download)
|
| 125 |
+
│ ├── wake_word_triggered.flac
|
| 126 |
+
│ └── timer_finished.flac
|
| 127 |
├── pyproject.toml # Project configuration
|
| 128 |
├── README.md # Documentation
|
| 129 |
└── PROJECT_PLAN.md # Project plan
|
| 130 |
```
|
| 131 |
|
| 132 |
+
## Dependencies
|
| 133 |
+
|
| 134 |
+
```toml
|
| 135 |
+
dependencies = [
|
| 136 |
+
"reachy-mini", # Reachy Mini SDK
|
| 137 |
+
"sounddevice>=0.4.6", # Audio processing (backup)
|
| 138 |
+
"soundfile>=0.12.0", # Audio file reading
|
| 139 |
+
"numpy>=1.24.0", # Numerical computation
|
| 140 |
+
"pymicro-wakeword>=2.0.0,<3.0.0", # Wake word detection
|
| 141 |
+
"pyopen-wakeword>=1.0.0,<2.0.0", # Backup wake word
|
| 142 |
+
"aioesphomeapi>=42.0.0", # ESPHome protocol
|
| 143 |
+
"zeroconf>=0.100.0", # mDNS discovery
|
| 144 |
+
"scipy>=1.10.0", # Motion control
|
| 145 |
+
"pydantic>=2.0.0", # Data validation
|
| 146 |
+
]
|
| 147 |
+
```
|
| 148 |
+
|
| 149 |
## Usage Flow
|
| 150 |
|
| 151 |
+
1. **Install App**
|
| 152 |
+
- Install `reachy-mini-ha-voice` from Reachy Mini App Store
|
|
|
|
|
|
|
| 153 |
|
| 154 |
+
2. **Start App**
|
| 155 |
+
- App auto-starts ESPHome server (port 6053)
|
| 156 |
+
- Auto-downloads required models and sounds
|
| 157 |
+
|
| 158 |
+
3. **Connect Home Assistant**
|
| 159 |
+
- Home Assistant auto-discovers device (mDNS)
|
| 160 |
+
- Or manually add: Settings → Devices & Services → Add Integration → ESPHome
|
| 161 |
|
| 162 |
+
4. **Use Voice Assistant**
|
| 163 |
+
- Say "Okay Nabu" to wake
|
| 164 |
+
- Speak command
|
| 165 |
+
- Reachy Mini provides motion feedback
|
| 166 |
|
| 167 |
+
## ESPHome Entity Planning
|
| 168 |
|
| 169 |
+
Based on deep analysis of Reachy Mini SDK, the following entities are exposed to Home Assistant:
|
|
|
|
|
|
|
|
|
|
|
|
|
| 170 |
|
| 171 |
+
### Implemented Entities
|
| 172 |
|
| 173 |
| Entity Type | Name | Description |
|
| 174 |
|-------------|------|-------------|
|
| 175 |
+
| Media Player | `media_player` | Audio playback control |
|
| 176 |
+
| Voice Assistant | `voice_assistant` | Voice assistant pipeline |
|
| 177 |
+
|
| 178 |
+
### Implemented Control Entities (Read/Write)
|
| 179 |
+
|
| 180 |
+
#### Phase 1-3: Basic Controls and Pose
|
| 181 |
+
|
| 182 |
+
| ESPHome Entity Type | Name | SDK API | Range/Options | Description |
|
| 183 |
+
|---------------------|------|---------|---------------|-------------|
|
| 184 |
+
| `Number` | `speaker_volume` | `AudioPlayer.set_volume()` | 0-100 | Speaker volume |
|
| 185 |
+
| `Select` | `motor_mode` | `set_motor_control_mode()` | enabled/disabled/gravity_compensation | Motor mode selection |
|
| 186 |
+
| `Switch` | `motors_enabled` | `enable_motors()` / `disable_motors()` | on/off | Motor torque switch |
|
| 187 |
+
| `Button` | `wake_up` | `mini.wake_up()` | - | Wake robot action |
|
| 188 |
+
| `Button` | `go_to_sleep` | `mini.goto_sleep()` | - | Sleep robot action |
|
| 189 |
+
| `Number` | `head_x` | `goto_target(head=...)` | ±50mm | Head X position control |
|
| 190 |
+
| `Number` | `head_y` | `goto_target(head=...)` | ±50mm | Head Y position control |
|
| 191 |
+
| `Number` | `head_z` | `goto_target(head=...)` | ±50mm | Head Z position control |
|
| 192 |
+
| `Number` | `head_roll` | `goto_target(head=...)` | -40° ~ +40° | Head roll angle control |
|
| 193 |
+
| `Number` | `head_pitch` | `goto_target(head=...)` | -40° ~ +40° | Head pitch angle control |
|
| 194 |
+
| `Number` | `head_yaw` | `goto_target(head=...)` | -180° ~ +180° | Head yaw angle control |
|
| 195 |
+
| `Number` | `body_yaw` | `goto_target(body_yaw=...)` | -160° ~ +160° | Body yaw angle control |
|
| 196 |
+
| `Number` | `antenna_left` | `goto_target(antennas=...)` | -90° ~ +90° | Left antenna angle control |
|
| 197 |
+
| `Number` | `antenna_right` | `goto_target(antennas=...)` | -90° ~ +90° | Right antenna angle control |
|
| 198 |
+
|
| 199 |
+
#### Phase 4: Gaze Control
|
| 200 |
+
|
| 201 |
+
| ESPHome Entity Type | Name | SDK API | Range/Options | Description |
|
| 202 |
+
|---------------------|------|---------|---------------|-------------|
|
| 203 |
+
| `Number` | `look_at_x` | `look_at_world(x, y, z)` | World coordinates | Gaze point X coordinate |
|
| 204 |
+
| `Number` | `look_at_y` | `look_at_world(x, y, z)` | World coordinates | Gaze point Y coordinate |
|
| 205 |
+
| `Number` | `look_at_z` | `look_at_world(x, y, z)` | World coordinates | Gaze point Z coordinate |
|
| 206 |
+
|
| 207 |
+
|
| 208 |
+
### Implemented Sensor Entities (Read-only)
|
| 209 |
+
|
| 210 |
+
#### Phase 1 & 5: Basic Status and Audio Sensors
|
| 211 |
+
|
| 212 |
+
| ESPHome Entity Type | Name | SDK API | Description |
|
| 213 |
+
|---------------------|------|---------|-------------|
|
| 214 |
+
| `Text Sensor` | `daemon_state` | `DaemonStatus.state` | Daemon status |
|
| 215 |
+
| `Binary Sensor` | `backend_ready` | `backend_status.ready` | Backend ready status |
|
| 216 |
+
| `Text Sensor` | `error_message` | `DaemonStatus.error` | Current error message |
|
| 217 |
+
| `Sensor` | `doa_angle` | `DoAInfo.angle` | Sound source direction angle (°) |
|
| 218 |
+
| `Binary Sensor` | `speech_detected` | `DoAInfo.speech_detected` | Speech detection status |
|
| 219 |
+
|
| 220 |
+
#### Phase 6: Diagnostic Information
|
| 221 |
+
|
| 222 |
+
| ESPHome Entity Type | Name | SDK API | Description |
|
| 223 |
+
|---------------------|------|---------|-------------|
|
| 224 |
+
| `Sensor` | `control_loop_frequency` | `control_loop_stats` | Control loop frequency (Hz) |
|
| 225 |
+
| `Text Sensor` | `sdk_version` | `DaemonStatus.version` | SDK version |
|
| 226 |
+
| `Text Sensor` | `robot_name` | `DaemonStatus.robot_name` | Robot name |
|
| 227 |
+
| `Binary Sensor` | `wireless_version` | `DaemonStatus.wireless_version` | Wireless version flag |
|
| 228 |
+
| `Binary Sensor` | `simulation_mode` | `DaemonStatus.simulation_enabled` | Simulation mode flag |
|
| 229 |
+
| `Text Sensor` | `wlan_ip` | `DaemonStatus.wlan_ip` | Wireless IP address |
|
| 230 |
+
|
| 231 |
+
#### Phase 7: IMU Sensors (Wireless version only)
|
| 232 |
+
|
| 233 |
+
| ESPHome Entity Type | Name | SDK API | Description |
|
| 234 |
+
|---------------------|------|---------|-------------|
|
| 235 |
+
| `Sensor` | `imu_accel_x` | `mini.imu["accelerometer"][0]` | X-axis acceleration (m/s²) |
|
| 236 |
+
| `Sensor` | `imu_accel_y` | `mini.imu["accelerometer"][1]` | Y-axis acceleration (m/s²) |
|
| 237 |
+
| `Sensor` | `imu_accel_z` | `mini.imu["accelerometer"][2]` | Z-axis acceleration (m/s²) |
|
| 238 |
+
| `Sensor` | `imu_gyro_x` | `mini.imu["gyroscope"][0]` | X-axis angular velocity (rad/s) |
|
| 239 |
+
| `Sensor` | `imu_gyro_y` | `mini.imu["gyroscope"][1]` | Y-axis angular velocity (rad/s) |
|
| 240 |
+
| `Sensor` | `imu_gyro_z` | `mini.imu["gyroscope"][2]` | Z-axis angular velocity (rad/s) |
|
| 241 |
+
| `Sensor` | `imu_temperature` | `mini.imu["temperature"]` | IMU temperature (°C) |
|
| 242 |
+
|
| 243 |
+
#### Phase 8-12: Extended Features
|
| 244 |
+
|
| 245 |
+
| ESPHome Entity Type | Name | Description |
|
| 246 |
+
|---------------------|------|-------------|
|
| 247 |
| `Select` | `emotion` | Emotion selector (Happy/Sad/Angry/Fear/Surprise/Disgust) |
|
| 248 |
| `Number` | `microphone_volume` | Microphone volume (0-100%) |
|
| 249 |
+
| `Camera` | `camera` | ESPHome Camera entity (live preview) |
|
| 250 |
+
| `Number` | `led_brightness` | LED brightness (0-100%) |
|
| 251 |
+
| `Select` | `led_effect` | LED effect (off/solid/breathing/rainbow/doa) |
|
| 252 |
+
| `Number` | `led_color_r` | LED red component (0-255) |
|
| 253 |
+
| `Number` | `led_color_g` | LED green component (0-255) |
|
| 254 |
+
| `Number` | `led_color_b` | LED blue component (0-255) |
|
| 255 |
| `Switch` | `agc_enabled` | Auto gain control switch |
|
| 256 |
| `Number` | `agc_max_gain` | AGC max gain (0-30 dB) |
|
| 257 |
| `Number` | `noise_suppression` | Noise suppression level (0-100%) |
|
| 258 |
+
| `Binary Sensor` | `echo_cancellation_converged` | Echo cancellation convergence status |
|
|
|
|
| 259 |
|
| 260 |
+
> **Note**: Head position (x/y/z) and angles (roll/pitch/yaw), body yaw, antenna angles are all **controllable** entities,
|
| 261 |
+
> using `Number` type for bidirectional control. Call `goto_target()` when setting new values, call `get_current_head_pose()` etc. when reading current values.
|
| 262 |
+
|
| 263 |
+
### Implementation Priority
|
| 264 |
+
|
| 265 |
+
1. **Phase 1 - Basic Status and Volume** (High Priority) ✅ **Completed**
|
| 266 |
+
- [x] `daemon_state` - Daemon status sensor
|
| 267 |
+
- [x] `backend_ready` - Backend ready status
|
| 268 |
+
- [x] `error_message` - Error message
|
| 269 |
+
- [x] `speaker_volume` - Speaker volume control
|
| 270 |
+
|
| 271 |
+
2. **Phase 2 - Motor Control** (High Priority) ✅ **Completed**
|
| 272 |
+
- [x] `motors_enabled` - Motor switch
|
| 273 |
+
- [x] `motor_mode` - Motor mode selection (enabled/disabled/gravity_compensation)
|
| 274 |
+
- [x] `wake_up` / `go_to_sleep` - Wake/sleep buttons
|
| 275 |
+
|
| 276 |
+
3. **Phase 3 - Pose Control** (Medium Priority) ✅ **Completed**
|
| 277 |
+
- [x] `head_x/y/z` - Head position control
|
| 278 |
+
- [x] `head_roll/pitch/yaw` - Head angle control
|
| 279 |
+
- [x] `body_yaw` - Body yaw angle control
|
| 280 |
+
- [x] `antenna_left/right` - Antenna angle control
|
| 281 |
+
|
| 282 |
+
4. **Phase 4 - Gaze Control** (Medium Priority) ✅ **Completed**
|
| 283 |
+
- [x] `look_at_x/y/z` - Gaze point coordinate control
|
| 284 |
+
|
| 285 |
+
5. **Phase 5 - Audio Sensors** (Low Priority) ✅ **Completed**
|
| 286 |
+
- [x] `doa_angle` - Sound source direction
|
| 287 |
+
- [x] `speech_detected` - Speech detection
|
| 288 |
+
|
| 289 |
+
6. **Phase 6 - Diagnostic Information** (Low Priority) ✅ **Completed**
|
| 290 |
+
- [x] `control_loop_frequency` - Control loop frequency
|
| 291 |
+
- [x] `sdk_version` - SDK version
|
| 292 |
+
- [x] `robot_name` - Robot name
|
| 293 |
+
- [x] `wireless_version` - Wireless version flag
|
| 294 |
+
- [x] `simulation_mode` - Simulation mode flag
|
| 295 |
+
- [x] `wlan_ip` - Wireless IP address
|
| 296 |
+
|
| 297 |
+
7. **Phase 7 - IMU Sensors** (Optional, wireless version only) ✅ **Completed**
|
| 298 |
+
- [x] `imu_accel_x/y/z` - Accelerometer
|
| 299 |
+
- [x] `imu_gyro_x/y/z` - Gyroscope
|
| 300 |
+
- [x] `imu_temperature` - IMU temperature
|
| 301 |
+
|
| 302 |
+
8. **Phase 8 - Emotion Control** ✅ **Completed**
|
| 303 |
+
- [x] `emotion` - Emotion selector (Happy/Sad/Angry/Fear/Surprise/Disgust)
|
| 304 |
+
|
| 305 |
+
9. **Phase 9 - Audio Control** ✅ **Completed**
|
| 306 |
+
- [x] `microphone_volume` - Microphone volume control (0-100%)
|
| 307 |
+
|
| 308 |
+
10. **Phase 10 - Camera Integration** ✅ **Completed**
|
| 309 |
+
- [x] `camera` - ESPHome Camera entity (live preview)
|
| 310 |
+
|
| 311 |
+
11. **Phase 11 - LED Control** ❌ **Disabled (LEDs hidden inside robot)**
|
| 312 |
+
- [ ] `led_brightness` - LED brightness (0-100%) - Commented out
|
| 313 |
+
- [ ] `led_effect` - LED effect (off/solid/breathing/rainbow/doa) - Commented out
|
| 314 |
+
- [ ] `led_color_r/g/b` - LED RGB color (0-255) - Commented out
|
| 315 |
+
|
| 316 |
+
12. **Phase 12 - Audio Processing Parameters** ✅ **Completed**
|
| 317 |
+
- [x] `agc_enabled` - Auto gain control switch
|
| 318 |
+
- [x] `agc_max_gain` - AGC max gain (0-30 dB)
|
| 319 |
+
- [x] `noise_suppression` - Noise suppression level (0-100%)
|
| 320 |
+
- [x] `echo_cancellation_converged` - Echo cancellation convergence status (read-only)
|
| 321 |
+
|
| 322 |
+
13. **Phase 13 - Sendspin Audio Playback Support** ✅ **Completed**
|
| 323 |
+
- [x] `sendspin_enabled` - Sendspin switch (Switch)
|
| 324 |
+
- [x] `sendspin_url` - Sendspin server URL (Text Sensor)
|
| 325 |
+
- [x] `sendspin_connected` - Sendspin connection status (Binary Sensor)
|
| 326 |
+
- [x] AudioPlayer integrates aiosendspin library
|
| 327 |
+
- [x] TTS audio sent to both local speaker and Sendspin server
|
| 328 |
+
|
| 329 |
+
---
|
| 330 |
+
|
| 331 |
+
## 🎉 Phase 1-13 Entities Completed!
|
| 332 |
+
|
| 333 |
+
**Total Completed: 43 entities**
|
| 334 |
+
- Phase 1: 4 entities (Basic status and volume)
|
| 335 |
+
- Phase 2: 4 entities (Motor control)
|
| 336 |
+
- Phase 3: 9 entities (Pose control)
|
| 337 |
+
- Phase 4: 3 entities (Gaze control)
|
| 338 |
+
- Phase 5: 2 entities (Audio sensors)
|
| 339 |
+
- Phase 6: 6 entities (Diagnostic information)
|
| 340 |
+
- Phase 7: 7 entities (IMU sensors)
|
| 341 |
+
- Phase 8: 1 entity (Emotion control)
|
| 342 |
+
- Phase 9: 1 entity (Microphone volume)
|
| 343 |
+
- Phase 10: 1 entity (Camera)
|
| 344 |
+
- Phase 11: 0 entities (LED control - Disabled)
|
| 345 |
+
- Phase 12: 4 entities (Audio processing parameters)
|
| 346 |
+
- Phase 13: 3 entities (Sendspin audio output)
|
| 347 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 348 |
|
| 349 |
---
|
| 350 |
|
| 351 |
+
## 🚀 Voice Assistant Enhancement Features Implementation Status
|
| 352 |
|
| 353 |
+
### Phase 14 - Emotion Action Feedback System (Partial) 🟡
|
| 354 |
|
| 355 |
+
**Implementation Status**: Basic infrastructure ready, supports manual trigger, uses voice-driven natural micro-movements during conversation
|
| 356 |
|
| 357 |
+
**Implemented Features**:
|
| 358 |
+
- ✅ Phase 8 Emotion Selector entity (`emotion`)
|
| 359 |
- ✅ Basic emotion action playback API (`_play_emotion`)
|
| 360 |
- ✅ Emotion mapping: Happy/Sad/Angry/Fear/Surprise/Disgust
|
| 361 |
+
- ✅ Integration with HuggingFace action library (`pollen-robotics/reachy-mini-emotions-library`)
|
| 362 |
+
- ✅ SpeechSway system for natural head micro-movements during conversation (non-blocking)
|
| 363 |
|
| 364 |
**Design Decisions**:
|
| 365 |
- 🎯 No auto-play of full emotion actions during conversation to avoid blocking
|
| 366 |
- 🎯 Use voice-driven head sway (SpeechSway) for natural motion feedback
|
| 367 |
- 🎯 Emotion actions retained as manual trigger feature via ESPHome entity
|
| 368 |
|
| 369 |
+
**Not Implemented**:
|
| 370 |
+
- ❌ Auto-trigger emotion actions based on voice assistant response (decided not to implement to avoid blocking)
|
| 371 |
+
- ❌ Intent recognition and emotion matching
|
| 372 |
+
- ❌ Dance action library integration
|
| 373 |
+
- ❌ Context awareness (e.g., weather query - sunny plays happy, rainy plays sad)
|
| 374 |
+
|
| 375 |
+
**Code Locations**:
|
| 376 |
+
- `entity_registry.py:633-658` - Emotion Selector entity
|
| 377 |
+
- `satellite.py:544-574` - `_play_emotion()` method
|
| 378 |
+
- `motion.py:132-156` - Conversation start motion control (uses SpeechSway)
|
| 379 |
+
- `movement_manager.py:541-595` - Move queue management (allows SpeechSway overlay)
|
| 380 |
+
|
| 381 |
+
**Actual Behavior**:
|
| 382 |
+
|
| 383 |
+
| Voice Assistant Event | Actual Action | Implementation Status |
|
| 384 |
+
|----------------------|---------------|----------------------|
|
| 385 |
+
| Wake word detected | Turn toward sound source + nod confirmation | ✅ Implemented |
|
| 386 |
+
| Conversation start | Voice-driven head micro-movements (SpeechSway) | ✅ Implemented |
|
| 387 |
+
| During conversation | Continuous voice-driven micro-movements + breathing animation | ✅ Implemented |
|
| 388 |
+
| Conversation end | Return to neutral position + breathing animation | ✅ Implemented |
|
| 389 |
+
| Manual emotion trigger | Play via ESPHome `emotion` entity | ✅ Implemented |
|
| 390 |
+
|
| 391 |
+
**Technical Details**:
|
| 392 |
+
```python
|
| 393 |
+
# motion.py - Use SpeechSway instead of full emotion actions during conversation
|
| 394 |
+
def on_speaking_start(self):
|
| 395 |
+
self._is_speaking = True
|
| 396 |
+
self._movement_manager.set_state(RobotState.SPEAKING)
|
| 397 |
+
# SpeechSway automatically generates natural head micro-movements based on audio loudness
|
| 398 |
+
# No full emotion actions played to avoid blocking conversation experience
|
| 399 |
+
|
| 400 |
+
# movement_manager.py - Motion layering system
|
| 401 |
+
# 1. Move queue (emotion actions) - Sets base pose
|
| 402 |
+
# 2. Action (nod/shake etc.) - Overlays on base pose
|
| 403 |
+
# 3. SpeechSway - Voice-driven micro-movements, can coexist with Move
|
| 404 |
+
# 4. Breathing - Idle breathing animation
|
| 405 |
+
```
|
| 406 |
+
|
| 407 |
+
**Original Plan** (Decided not to implement to avoid blocking conversation):
|
| 408 |
+
|
| 409 |
+
| Voice Assistant Event | Original Planned Action | Reason Not Implemented |
|
| 410 |
+
|----------------------|------------------------|------------------------|
|
| 411 |
+
| Positive response received | Play "happy" action | Full action would block conversation fluency |
|
| 412 |
+
| Negative response received | Play "sad" action | Full action would block conversation fluency |
|
| 413 |
+
| Play music/entertainment | Play "dance" action | Full action would block conversation fluency |
|
| 414 |
+
| Timer completed | Play "alert" action | Full action would block conversation fluency |
|
| 415 |
+
| Error/cannot understand | Play "confused" action | Full action would block conversation fluency |
|
| 416 |
+
|
| 417 |
+
**Manual Emotion Trigger Example**:
|
| 418 |
+
```yaml
|
| 419 |
+
# Home Assistant automation example - Manual emotion trigger
|
| 420 |
+
automation:
|
| 421 |
+
- alias: "Reachy Good Morning Greeting"
|
| 422 |
+
trigger:
|
| 423 |
+
- platform: time
|
| 424 |
+
at: "07:00:00"
|
| 425 |
+
action:
|
| 426 |
+
- service: select.select_option
|
| 427 |
+
target:
|
| 428 |
+
entity_id: select.reachy_mini_emotion
|
| 429 |
+
data:
|
| 430 |
+
option: "Happy"
|
| 431 |
+
```
|
| 432 |
+
|
| 433 |
+
### Phase 15 - Face Tracking (Replaces DOA Sound Source Tracking) ✅ **Completed**
|
| 434 |
|
| 435 |
**Goal**: Implement natural face tracking so robot looks at speaker during conversation.
|
| 436 |
|
|
|
|
| 440 |
- Reason: DOA inaccurate at wakeup, frequent queries cause daemon crash
|
| 441 |
|
| 442 |
**Implemented Features**:
|
| 443 |
+
|
| 444 |
+
| Feature | Description | Implementation Location | Status |
|
| 445 |
+
|---------|-------------|------------------------|--------|
|
| 446 |
+
| YOLO face detection | Uses `AdamCodd/YOLOv11n-face-detection` model | `head_tracker.py` | ✅ Implemented |
|
| 447 |
+
| Adaptive frame rate tracking | 15fps during conversation, 3fps when idle without face | `camera_server.py` | ✅ Implemented |
|
| 448 |
+
| look_at_image() | Calculate target pose from face position | `camera_server.py` | ✅ Implemented |
|
| 449 |
+
| Smooth return to neutral | Smooth return within 1 second after face lost | `camera_server.py` | ✅ Implemented |
|
| 450 |
+
| face_tracking_offsets | As secondary pose overlay to motion control | `movement_manager.py` | ✅ Implemented |
|
| 451 |
+
| Voice activity detection | DOA entity still available for speech detection | `DoAInfo.speech_detected` | ✅ Exposed as entity |
|
| 452 |
+
| Model download retry | 3 retries, 5 second interval | `head_tracker.py` | ✅ Implemented |
|
| 453 |
+
| Conversation mode integration | Auto-switch tracking frequency on voice assistant state change | `satellite.py` | ✅ Implemented |
|
| 454 |
|
| 455 |
**Resource Optimization (v0.5.1)**:
|
| 456 |
- During conversation (listening/thinking/speaking): High-frequency tracking 15fps
|
| 457 |
- Idle with face detected: High-frequency tracking 15fps
|
| 458 |
+
- Idle without face for 10s: Low-power mode 3fps (only detect if someone appears)
|
| 459 |
- Immediately restore high-frequency tracking when face detected
|
| 460 |
|
| 461 |
+
**Code Locations**:
|
| 462 |
+
- `head_tracker.py` - YOLO face detector (`HeadTracker` class)
|
| 463 |
+
- `camera_server.py:_capture_frames()` - Adaptive frame rate face tracking
|
| 464 |
+
- `camera_server.py:set_conversation_mode()` - Conversation mode switch API
|
| 465 |
+
- `satellite.py:_set_conversation_mode()` - Voice assistant state integration
|
| 466 |
+
- `movement_manager.py:set_face_tracking_offsets()` - Face tracking offset API
|
| 467 |
+
|
| 468 |
+
**Technical Details**:
|
| 469 |
+
```python
|
| 470 |
+
# camera_server.py - Adaptive frame rate face tracking
|
| 471 |
+
class MJPEGCameraServer:
|
| 472 |
+
def __init__(self):
|
| 473 |
+
self._fps_high = 15 # During conversation/face detected
|
| 474 |
+
self._fps_low = 3 # Idle without face
|
| 475 |
+
self._low_power_threshold = 10.0 # 10s without face switches to low power
|
| 476 |
+
|
| 477 |
+
def _should_run_face_tracking(self, current_time):
|
| 478 |
+
# Conversation mode: Always high-frequency tracking
|
| 479 |
+
if self._in_conversation:
|
| 480 |
+
return True
|
| 481 |
+
# High-frequency mode: Track every frame
|
| 482 |
+
if self._current_fps == self._fps_high:
|
| 483 |
+
return True
|
| 484 |
+
# Low-power mode: Periodic detection
|
| 485 |
+
return time.since_last_check >= 1/self._fps_low
|
| 486 |
+
|
| 487 |
+
# satellite.py - Voice assistant state integration
|
| 488 |
+
def _reachy_on_listening(self):
|
| 489 |
+
self._set_conversation_mode(True) # Start conversation, high-frequency tracking
|
| 490 |
+
|
| 491 |
+
def _reachy_on_idle(self):
|
| 492 |
+
self._set_conversation_mode(False) # End conversation, adaptive tracking
|
| 493 |
+
```
|
| 494 |
+
|
| 495 |
+
|
| 496 |
+
### Phase 16 - Cartoon Style Motion Mode (Partial) 🟡
|
| 497 |
|
| 498 |
**Goal**: Use SDK interpolation techniques for more expressive robot movements.
|
| 499 |
|
| 500 |
+
**SDK Support**: `InterpolationTechnique` enum
|
| 501 |
+
- `LINEAR` - Linear, mechanical feel
|
| 502 |
+
- `MIN_JERK` - Minimum jerk, natural and smooth (default)
|
| 503 |
+
- `EASE_IN_OUT` - Ease in-out, elegant
|
| 504 |
+
- `CARTOON` - Cartoon style, with bounce effect, lively and cute
|
| 505 |
+
|
| 506 |
+
**Implemented Features**:
|
| 507 |
+
- ✅ 20Hz unified control loop (`movement_manager.py`) - Reduced from 100Hz to prevent daemon crash
|
| 508 |
+
- ✅ Pose change detection - Only send commands on significant changes (threshold 0.001)
|
| 509 |
- ✅ State query caching - 100ms TTL, reduces daemon load
|
| 510 |
- ✅ Smooth interpolation (ease in-out curve)
|
| 511 |
+
- ✅ Breathing animation - Idle Z-axis micro-movement + antenna sway (`BreathingAnimation`)
|
| 512 |
+
- ✅ Command queue mode - Thread-safe external API
|
| 513 |
+
- ✅ Error throttling - Prevents log explosion
|
| 514 |
+
- ✅ Connection health monitoring - Auto-detect and recover from connection loss
|
| 515 |
|
| 516 |
**Not Implemented**:
|
| 517 |
- ❌ Dynamic interpolation technique switching (CARTOON/EASE_IN_OUT etc.)
|
| 518 |
- ❌ Exaggerated cartoon bounce effects
|
| 519 |
|
| 520 |
+
**Code Locations**:
|
| 521 |
+
- `movement_manager.py:192-243` - BreathingAnimation class
|
| 522 |
+
- `movement_manager.py:246-697` - MovementManager class
|
| 523 |
+
|
| 524 |
+
**Scene Implementation Status**:
|
| 525 |
+
|
| 526 |
+
| Scene | Recommended Interpolation | Effect | Status |
|
| 527 |
+
|-------|--------------------------|--------|--------|
|
| 528 |
+
| Wake nod | `CARTOON` | Lively bounce effect | ❌ Not implemented |
|
| 529 |
+
| Thinking head up | `EASE_IN_OUT` | Elegant transition | ✅ Implemented (smooth interpolation) |
|
| 530 |
+
| Speaking micro-movements | `MIN_JERK` | Natural and fluid | ✅ Implemented (SpeechSway) |
|
| 531 |
+
| Error head shake | `CARTOON` | Exaggerated denial | ❌ Not implemented |
|
| 532 |
+
| Return to neutral | `MIN_JERK` | Smooth return | ✅ Implemented |
|
| 533 |
+
| Idle breathing | - | Subtle sense of life | ✅ Implemented (BreathingAnimation) |
|
| 534 |
+
|
| 535 |
+
### Phase 17 - Antenna Sync Animation During Speech (Partial) 🟡
|
| 536 |
|
| 537 |
**Goal**: Antennas sway with audio rhythm during TTS playback, simulating "speaking" effect.
|
| 538 |
|
| 539 |
+
**Implemented Features**:
|
| 540 |
- ✅ Voice-driven head sway (`SpeechSwayGenerator`)
|
| 541 |
- ✅ VAD detection based on audio loudness
|
| 542 |
- ✅ Multi-frequency sine wave overlay (Lissajous motion)
|
| 543 |
- ✅ Smooth envelope transitions
|
| 544 |
|
| 545 |
+
**Code Locations**:
|
| 546 |
+
- `movement_manager.py:124-189` - SpeechSwayGenerator class
|
| 547 |
+
- `motion.py:212-222` - update_audio_loudness() method
|
| 548 |
+
|
| 549 |
+
**Technical Details**:
|
| 550 |
+
```python
|
| 551 |
+
# Speech sway parameters
|
| 552 |
+
SWAY_A_PITCH_DEG = 3.0 # Pitch amplitude (degrees)
|
| 553 |
+
SWAY_A_YAW_DEG = 2.0 # Yaw amplitude
|
| 554 |
+
SWAY_A_ROLL_DEG = 2.0 # Roll amplitude
|
| 555 |
+
SWAY_F_PITCH = 0.8 # Pitch frequency Hz
|
| 556 |
+
SWAY_F_YAW = 0.6 # Yaw frequency
|
| 557 |
+
SWAY_F_ROLL = 0.5 # Roll frequency
|
| 558 |
+
|
| 559 |
+
# VAD thresholds
|
| 560 |
+
VAD_DB_ON = -35 # Start detection threshold
|
| 561 |
+
VAD_DB_OFF = -45 # Stop detection threshold
|
| 562 |
+
```
|
| 563 |
+
|
| 564 |
**Not Implemented**:
|
| 565 |
- ❌ Antenna sway with audio rhythm (currently only head sway)
|
| 566 |
- ❌ Audio spectrum analysis driven animation
|
| 567 |
|
| 568 |
+
### Phase 18 - Visual Gaze Interaction (Not Implemented) ❌
|
| 569 |
|
| 570 |
**Goal**: Use camera to detect faces for eye contact.
|
| 571 |
|
| 572 |
+
**SDK Support**:
|
| 573 |
+
- `look_at_image(u, v)` - Look at point in image
|
| 574 |
+
- `look_at_world(x, y, z)` - Look at world coordinate point
|
| 575 |
+
- `media.get_frame()` - Get camera frame (✅ Already implemented in `camera_server.py:146`)
|
| 576 |
+
|
| 577 |
+
**Not Implemented Features**:
|
| 578 |
+
|
| 579 |
+
| Feature | Description | Status |
|
| 580 |
+
|---------|-------------|--------|
|
| 581 |
+
| Face detection | Use OpenCV/MediaPipe to detect faces | ❌ Not implemented |
|
| 582 |
+
| Eye tracking | Look at speaker's face during conversation | ❌ Not implemented |
|
| 583 |
+
| Multi-person switching | When multiple people detected, look at current speaker | ❌ Not implemented |
|
| 584 |
+
| Idle scanning | Randomly look around when idle | ❌ Not implemented |
|
| 585 |
|
| 586 |
+
### Phase 19 - Gravity Compensation Interactive Mode (Partial) 🟡
|
| 587 |
+
|
| 588 |
+
**Goal**: Allow users to physically touch and guide robot head for "teaching" style interaction.
|
| 589 |
+
|
| 590 |
+
**SDK Support**: `enable_gravity_compensation()` - Motors enter gravity compensation mode, can be manually moved
|
| 591 |
+
|
| 592 |
+
**Implemented Features**:
|
| 593 |
+
- ✅ Gravity compensation mode switch (`motor_mode` Select entity, option "gravity_compensation")
|
| 594 |
+
- ✅ `reachy_controller.py:236-237` - Gravity compensation API call
|
| 595 |
|
| 596 |
**Not Implemented**:
|
| 597 |
+
- ❌ Teaching mode - Record motion trajectory
|
| 598 |
- ❌ Save/playback custom actions
|
| 599 |
+
- ❌ Voice command triggered teaching flow
|
| 600 |
+
|
| 601 |
+
**Application Scenarios**:
|
| 602 |
+
- ❌ User says "Let me teach you a move" → Enter gravity compensation mode
|
| 603 |
+
- ❌ User manually moves head → Record motion trajectory
|
| 604 |
+
- ❌ User says "Remember this" → Save action
|
| 605 |
+
- ❌ User says "Do that action again" → Playback recorded action
|
| 606 |
|
| 607 |
+
### Phase 20 - Environment Awareness Response (Partial) 🟡
|
| 608 |
|
| 609 |
+
**Goal**: Use IMU sensors to sense environment changes and respond.
|
| 610 |
+
|
| 611 |
+
**SDK Support**:
|
| 612 |
+
- ✅ `mini.imu["accelerometer"]` - Accelerometer (Phase 7 implemented as entity)
|
| 613 |
+
- ✅ `mini.imu["gyroscope"]` - Gyroscope (Phase 7 implemented as entity)
|
| 614 |
+
|
| 615 |
+
**Implemented Features**:
|
| 616 |
+
|
| 617 |
+
| Detection Event | Response Action | Status |
|
| 618 |
+
|-----------------|-----------------|--------|
|
| 619 |
+
| Tap-to-wake | Enter continuous conversation mode | ✅ Implemented |
|
| 620 |
+
| Second tap | Exit continuous conversation mode | ✅ Implemented |
|
| 621 |
|
| 622 |
**Tap-to-wake vs Voice Wake**:
|
| 623 |
+
|
| 624 |
| Wake Method | Conversation Mode | Description |
|
| 625 |
|-------------|-------------------|-------------|
|
| 626 |
| Voice wake (Okay Nabu) | Single conversation | Need to say wake word for each conversation |
|
| 627 |
| Tap-to-wake | Continuous conversation | Auto-continue listening after TTS ends, tap again to exit |
|
| 628 |
|
| 629 |
+
**Technical Implementation**:
|
| 630 |
+
- `tap_detector.py` - IMU acceleration spike detection
|
| 631 |
+
- `satellite.py:_tap_conversation_mode` - Continuous conversation mode flag
|
| 632 |
+
- Threshold: 2.0g (configurable)
|
| 633 |
+
- Cooldown: 1.0s (prevent repeated triggers)
|
| 634 |
+
- Wireless version only
|
| 635 |
+
|
| 636 |
+
```python
|
| 637 |
+
# satellite.py - Continuous conversation mode
|
| 638 |
+
def wakeup_from_tap(self):
|
| 639 |
+
if self._tap_conversation_mode:
|
| 640 |
+
# Second tap - Exit continuous conversation
|
| 641 |
+
self._tap_conversation_mode = False
|
| 642 |
+
self._reachy_on_idle()
|
| 643 |
+
else:
|
| 644 |
+
# First tap - Enter continuous conversation
|
| 645 |
+
self._tap_conversation_mode = True
|
| 646 |
+
self.send_messages([VoiceAssistantRequest(start=True)])
|
| 647 |
+
|
| 648 |
+
def _tts_finished(self):
|
| 649 |
+
if self._tap_conversation_mode:
|
| 650 |
+
# Continuous conversation mode: Auto-continue listening
|
| 651 |
+
self.send_messages([VoiceAssistantRequest(start=True)])
|
| 652 |
+
```
|
| 653 |
+
|
| 654 |
**Not Implemented**:
|
|
|
|
|
|
|
|
|
|
| 655 |
|
| 656 |
+
| Detection Event | Response Action | Status |
|
| 657 |
+
|-----------------|-----------------|--------|
|
| 658 |
+
| Being shaken | Play dizzy action + voice "Don't shake me~" | ❌ Not implemented |
|
| 659 |
+
| Tilted/fallen | Play help action + voice "I fell, help me" | ❌ Not implemented |
|
| 660 |
+
| Long idle | Enter sleep animation | ❌ Not implemented |
|
| 661 |
+
|
| 662 |
+
### Phase 21 - Home Assistant Scene Integration (Not Implemented) ❌
|
| 663 |
+
|
| 664 |
+
**Goal**: Trigger robot actions based on Home Assistant scenes/automations.
|
| 665 |
+
|
| 666 |
+
**Implementation**: Via ESPHome service calls
|
| 667 |
+
|
| 668 |
+
**Not Implemented Scenes**:
|
| 669 |
+
|
| 670 |
+
| HA Scene | Robot Response | Status |
|
| 671 |
+
|----------|----------------|--------|
|
| 672 |
+
| Good morning scene | Play wake action + "Good morning!" | ❌ Not implemented |
|
| 673 |
+
| Good night scene | Play sleep action + "Good night~" | ❌ Not implemented |
|
| 674 |
+
| Someone home | Turn toward door + wave + "Welcome home!" | ❌ Not implemented |
|
| 675 |
+
| Doorbell rings | Turn toward door + alert action | ❌ Not implemented |
|
| 676 |
+
| Play music | Sway with music rhythm | ❌ Not implemented |
|
| 677 |
+
|
| 678 |
+
|
| 679 |
+
---
|
| 680 |
+
|
| 681 |
+
## 📊 Feature Implementation Summary
|
| 682 |
+
|
| 683 |
+
### ✅ Completed Features
|
| 684 |
+
|
| 685 |
+
#### Core Voice Assistant (Phase 1-12)
|
| 686 |
+
- **45+ ESPHome entities** - All implemented
|
| 687 |
+
- **Basic voice interaction** - Wake word detection, STT/TTS integration
|
| 688 |
+
- **Motion feedback** - Nod, shake, gaze and other basic actions
|
| 689 |
+
- **Audio processing** - AGC, noise suppression, echo cancellation
|
| 690 |
+
- **Camera stream** - MJPEG live preview
|
| 691 |
+
|
| 692 |
+
#### Partially Implemented Features (Phase 14-21)
|
| 693 |
+
- **Phase 14** - Emotion action API infrastructure (manual trigger available)
|
| 694 |
+
- **Phase 19** - Gravity compensation mode switch (teaching flow not implemented)
|
| 695 |
+
|
| 696 |
+
### ❌ Not Implemented Features
|
| 697 |
+
|
| 698 |
+
#### High Priority
|
| 699 |
+
- ~~**Phase 13** - Sendspin audio playback support~~ ✅ **Completed**
|
| 700 |
+
- **Phase 14** - Auto emotion action feedback (needs voice assistant event association)
|
| 701 |
+
- **Phase 15** - Continuous sound source tracking (only turn toward at wakeup)
|
| 702 |
+
|
| 703 |
+
#### Medium Priority
|
| 704 |
+
- **Phase 16** - Cartoon style motion mode (needs dynamic interpolation switching)
|
| 705 |
+
- **Phase 17** - Antenna sync animation
|
| 706 |
+
- **Phase 18** - Face tracking and eye contact interaction
|
| 707 |
+
|
| 708 |
+
#### Low Priority
|
| 709 |
+
- **Phase 19** - Teaching mode record/playback functionality
|
| 710 |
+
- **Phase 20** - IMU environment awareness response
|
| 711 |
+
- **Phase 21** - Home Assistant scene integration
|
| 712 |
|
| 713 |
---
|
| 714 |
|
| 715 |
+
## Feature Priority Summary (Updated)
|
| 716 |
+
|
| 717 |
+
### High Priority (Completed ✅)
|
| 718 |
+
- ✅ **Phase 1-12**: Basic ESPHome entities (45+)
|
| 719 |
+
- ✅ Core voice assistant functionality
|
| 720 |
+
- ✅ Basic motion feedback (nod, shake, gaze)
|
| 721 |
+
|
| 722 |
+
### High Priority (Partial 🟡)
|
| 723 |
+
- 🟡 **Phase 13**: Emotion action feedback system
|
| 724 |
+
- ✅ Emotion Selector entity and API infrastructure
|
| 725 |
+
- ❌ Auto-trigger emotion actions based on voice assistant response
|
| 726 |
+
- ❌ Intent recognition and emotion matching
|
| 727 |
+
- ❌ Dance action library integration
|
| 728 |
+
|
| 729 |
+
### High Priority (Not Implemented ❌)
|
| 730 |
+
- ❌ **Phase 14**: Smart sound source tracking enhancement
|
| 731 |
+
- ✅ Turn toward sound source at wakeup
|
| 732 |
+
- ❌ Continuous sound source tracking
|
| 733 |
+
- ❌ Multi-person conversation switching
|
| 734 |
+
- ❌ Sound source visualization
|
| 735 |
+
|
| 736 |
+
### Medium Priority (Partial 🟡)
|
| 737 |
+
- 🟡 **Phase 15**: Cartoon style motion mode
|
| 738 |
+
- ✅ 20Hz unified control loop architecture (optimized to prevent daemon crash)
|
| 739 |
+
- ✅ Pose change detection + state query caching (reduces daemon load)
|
| 740 |
+
- ✅ Smooth interpolation + breathing animation
|
| 741 |
+
- ❌ Dynamic interpolation technique switching (CARTOON etc.)
|
| 742 |
+
- 🟡 **Phase 16**: Antenna sync during speech
|
| 743 |
+
- ✅ Voice-driven head sway (SpeechSwayGenerator)
|
| 744 |
+
- ❌ Antenna sway with audio rhythm
|
| 745 |
+
|
| 746 |
+
### Medium Priority (Not Implemented ❌)
|
| 747 |
+
- ❌ **Phase 17**: Visual gaze interaction - Eye contact
|
| 748 |
+
|
| 749 |
+
### Low Priority (Partial 🟡)
|
| 750 |
+
- 🟡 **Phase 18**: Gravity compensation interactive mode
|
| 751 |
+
- ✅ Gravity compensation mode switch
|
| 752 |
+
- ❌ Teaching style interaction (record/playback functionality)
|
| 753 |
+
|
| 754 |
+
### Low Priority (Not Implemented ❌)
|
| 755 |
+
- ❌ **Phase 19**: Environment awareness response - IMU triggered actions
|
| 756 |
+
- ❌ **Phase 20**: Home Assistant scene integration - Smart home integration
|
| 757 |
+
|
| 758 |
+
---
|
| 759 |
+
|
| 760 |
+
## 📈 Completion Statistics
|
| 761 |
|
| 762 |
| Phase | Status | Completion | Notes |
|
| 763 |
|-------|--------|------------|-------|
|
| 764 |
| Phase 1-12 | ✅ Complete | 100% | 40 ESPHome entities implemented (Phase 11 LED disabled) |
|
| 765 |
+
| Phase 13 | 🟡 Partial | 30% | API infrastructure ready, missing auto-trigger |
|
| 766 |
+
| Phase 14 | ❌ Not done | 20% | Only turn toward at wakeup implemented |
|
| 767 |
+
| Phase 15 | 🟡 Partial | 70% | 20Hz control loop + pose change detection + state cache + breathing animation implemented |
|
| 768 |
+
| Phase 16 | 🟡 Partial | 50% | Voice-driven head sway implemented |
|
| 769 |
+
| Phase 17 | ❌ Not done | 10% | Camera implemented, missing face detection |
|
| 770 |
+
| Phase 18 | 🟡 Partial | 40% | Mode switch implemented, missing teaching flow |
|
| 771 |
+
| Phase 19 | ❌ Not done | 10% | IMU data exposed, missing trigger logic |
|
| 772 |
+
| Phase 20 | ❌ Not done | 0% | Not implemented |
|
| 773 |
+
|
| 774 |
+
**Overall Completion**: **Phase 1-12: 100%** | **Phase 13-20: ~35%**
|
| 775 |
+
|
| 776 |
+
|
| 777 |
+
---
|
| 778 |
+
|
| 779 |
+
## 🔧 Daemon Crash Fix (2025-01-05)
|
| 780 |
+
|
| 781 |
+
### Problem Description
|
| 782 |
+
During long-term operation, `reachy_mini daemon` would crash, causing robot to become unresponsive.
|
| 783 |
+
|
| 784 |
+
### Root Cause
|
| 785 |
+
1. **100Hz control loop too frequent** - Calling `robot.set_target()` every 10ms, even when pose hasn't changed
|
| 786 |
+
2. **Frequent state queries** - Every entity state read calls `get_status()`, `get_current_head_pose()` etc.
|
| 787 |
+
3. **Missing change detection** - Even when pose hasn't changed, continues sending same commands
|
| 788 |
+
4. **Zenoh message queue blocking** - Accumulated 150+ messages per second, daemon cannot process in time
|
| 789 |
+
|
| 790 |
+
### Fix Solution
|
| 791 |
+
|
| 792 |
+
#### 1. Reduce control loop frequency (movement_manager.py)
|
| 793 |
+
```python
|
| 794 |
+
# Reduced from 100Hz to 20Hz
|
| 795 |
+
CONTROL_LOOP_FREQUENCY_HZ = 20 # 80% reduction in messages
|
| 796 |
+
```
|
| 797 |
+
|
| 798 |
+
#### 2. Add pose change detection (movement_manager.py)
|
| 799 |
+
```python
|
| 800 |
+
# Only send commands on significant pose changes
|
| 801 |
+
if self._last_sent_pose is not None:
|
| 802 |
+
max_diff = max(abs(pose[k] - self._last_sent_pose.get(k, 0.0)) for k in pose.keys())
|
| 803 |
+
if max_diff < 0.001: # Threshold: 0.001 rad or 0.001 m
|
| 804 |
+
return # Skip sending
|
| 805 |
+
```
|
| 806 |
+
|
| 807 |
+
#### 3. State query caching (reachy_controller.py)
|
| 808 |
+
```python
|
| 809 |
+
# Cache daemon status query results
|
| 810 |
+
self._cache_ttl = 0.1 # 100ms TTL
|
| 811 |
+
self._last_status_query = 0.0
|
| 812 |
+
|
| 813 |
+
def _get_cached_status(self):
|
| 814 |
+
now = time.time()
|
| 815 |
+
if now - self._last_status_query < self._cache_ttl:
|
| 816 |
+
return self._state_cache.get('status') # Use cache
|
| 817 |
+
# ... query and update cache
|
| 818 |
+
```
|
| 819 |
+
|
| 820 |
+
#### 4. Head pose query caching (reachy_controller.py)
|
| 821 |
+
```python
|
| 822 |
+
# Cache get_current_head_pose() and get_current_joint_positions() results
|
| 823 |
+
def _get_cached_head_pose(self):
|
| 824 |
+
# Reuse cached results within 100ms
|
| 825 |
+
```
|
| 826 |
+
|
| 827 |
+
### Fix Results
|
| 828 |
+
|
| 829 |
+
| Metric | Before Fix | After Fix | Improvement |
|
| 830 |
+
|--------|------------|-----------|-------------|
|
| 831 |
+
| Control message frequency | ~100 msg/s | ~20 msg/s | ↓ 80% |
|
| 832 |
+
| State query frequency | ~50 msg/s | ~5 msg/s | ↓ 90% |
|
| 833 |
+
| Total Zenoh messages | ~150 msg/s | ~25 msg/s | ↓ 83% |
|
| 834 |
+
| Daemon CPU load | Sustained high load | Normal load | Significantly reduced |
|
| 835 |
+
| Expected stability | Crash within hours | Stable for days | Major improvement |
|
| 836 |
+
|
| 837 |
+
### Related Files
|
| 838 |
+
- `DAEMON_CRASH_FIX_PLAN.md` - Detailed fix plan and test plan
|
| 839 |
+
- `movement_manager.py` - Control loop optimization
|
| 840 |
+
- `reachy_controller.py` - State query caching
|
| 841 |
+
|
| 842 |
+
### Future Optimization Suggestions
|
| 843 |
+
1. ⏳ Dynamic frequency adjustment - 50Hz during motion, 5Hz when idle
|
| 844 |
+
2. ⏳ Batch state queries - Get all states at once
|
| 845 |
+
3. ⏳ Performance monitoring and alerts - Real-time daemon health monitoring
|
| 846 |
|
| 847 |
---
|
| 848 |
|
| 849 |
+
## 🔧 Daemon Crash Deep Fix (2026-01-07)
|
| 850 |
+
|
| 851 |
+
### Problem Description
|
| 852 |
+
During long-term operation, `reachy_mini daemon` still crashes, previous fix not thorough enough.
|
| 853 |
+
|
| 854 |
+
### Root Cause Analysis
|
| 855 |
+
|
| 856 |
+
Through deep analysis of SDK source code:
|
| 857 |
+
|
| 858 |
+
1. **Each `set_target()` sends 3 Zenoh messages**
|
| 859 |
+
- `set_target_head_pose()` - 1 message
|
| 860 |
+
- `set_target_antenna_joint_positions()` - 1 message
|
| 861 |
+
- `set_target_body_yaw()` - 1 message
|
| 862 |
+
|
| 863 |
+
2. **Daemon control loop is 50Hz**
|
| 864 |
+
- See `reachy_mini/daemon/backend/robot/backend.py`: `control_loop_frequency = 50.0`
|
| 865 |
+
- If message send frequency exceeds 50Hz, daemon may not process in time
|
| 866 |
+
|
| 867 |
+
3. **Previous 20Hz control loop still too high**
|
| 868 |
+
- 20Hz × 3 messages = 60 messages/second
|
| 869 |
+
- Already exceeds daemon's 50Hz processing capacity
|
| 870 |
+
|
| 871 |
+
4. **Pose change threshold too small (0.002)**
|
| 872 |
+
- Breathing animation, speech sway, face tracking continuously produce tiny changes
|
| 873 |
+
- Almost every loop triggers `set_target()`
|
| 874 |
+
|
| 875 |
+
### Fix Solution
|
| 876 |
+
|
| 877 |
+
#### 1. Further reduce control loop frequency (movement_manager.py)
|
| 878 |
+
```python
|
| 879 |
+
# Reduced from 20Hz to 10Hz
|
| 880 |
+
# 10Hz × 3 messages = 30 messages/second, safely below daemon's 50Hz capacity
|
| 881 |
+
CONTROL_LOOP_FREQUENCY_HZ = 10
|
| 882 |
+
```
|
| 883 |
+
|
| 884 |
+
#### 2. Increase pose change threshold (movement_manager.py)
|
| 885 |
+
```python
|
| 886 |
+
# Increased from 0.002 to 0.005
|
| 887 |
+
# 0.005 rad ≈ 0.29 degrees, still smooth enough
|
| 888 |
+
self._pose_change_threshold = 0.005
|
| 889 |
+
```
|
| 890 |
+
|
| 891 |
+
#### 3. Reduce camera/face tracking frequency (camera_server.py)
|
| 892 |
+
```python
|
| 893 |
+
# Reduced from 15fps to 10fps
|
| 894 |
+
fps: int = 10
|
| 895 |
+
```
|
| 896 |
+
|
| 897 |
+
#### 4. Reduce IMU polling frequency (tap_detector.py)
|
| 898 |
+
```python
|
| 899 |
+
# Reduced from 50Hz to 20Hz
|
| 900 |
+
TAP_DETECTION_RATE_HZ = 20
|
| 901 |
+
```
|
| 902 |
+
|
| 903 |
+
#### 5. Increase state cache TTL (reachy_controller.py)
|
| 904 |
+
```python
|
| 905 |
+
# Increased from 1 second to 2 seconds
|
| 906 |
+
self._cache_ttl = 2.0
|
| 907 |
+
```
|
| 908 |
+
|
| 909 |
+
### Fix Results
|
| 910 |
+
|
| 911 |
+
| Metric | Before (20Hz) | After (10Hz) | Improvement |
|
| 912 |
+
|--------|---------------|--------------|-------------|
|
| 913 |
+
| Control loop frequency | 20 Hz | 10 Hz | ↓ 50% |
|
| 914 |
+
| Max Zenoh messages | 60 msg/s | 30 msg/s | ↓ 50% |
|
| 915 |
+
| Actual messages (with change detection) | ~40 msg/s | ~15 msg/s | ↓ 62% |
|
| 916 |
+
| Face tracking frequency | 15 Hz | 10 Hz | ↓ 33% |
|
| 917 |
+
| IMU polling frequency | 50 Hz | 20 Hz | ↓ 60% |
|
| 918 |
+
| State cache TTL | 1 second | 2 seconds | ↑ 100% |
|
| 919 |
+
| Expected stability | Crash within hours | Stable operation | Major improvement |
|
| 920 |
+
|
| 921 |
+
### Key Finding
|
| 922 |
+
|
| 923 |
+
Reference `reachy_mini_conversation_app` uses 100Hz control loop, but it's an official app that may have special optimizations or runs on more powerful hardware. Our app needs more conservative settings.
|
| 924 |
+
|
| 925 |
+
### Related Files
|
| 926 |
+
- `movement_manager.py` - Control loop frequency and pose threshold
|
| 927 |
+
- `camera_server.py` - Face tracking frequency
|
| 928 |
+
- `tap_detector.py` - IMU polling frequency
|
| 929 |
+
- `reachy_controller.py` - State cache TTL
|
| 930 |
+
|
| 931 |
+
|
| 932 |
+
---
|
| 933 |
|
| 934 |
+
## 🔧 Tap-to-Wake and Microphone Sensitivity Fix (2026-01-07)
|
| 935 |
+
|
| 936 |
+
### Problem Description
|
| 937 |
+
1. **Tap-to-wake blocking** - Conversation not working properly after tap wake, blocking issues
|
| 938 |
+
2. **Low microphone sensitivity** - Need to be very close for voice recognition
|
| 939 |
+
|
| 940 |
+
### Root Cause
|
| 941 |
+
1. **Audio playback blocking** - `_tap_continue_feedback()` plays sound in continuous conversation mode, blocking audio stream processing
|
| 942 |
+
2. **AGC settings not optimized** - ReSpeaker XVF3800 default settings not suitable for distant voice recognition
|
| 943 |
+
|
| 944 |
+
### Fix Solution
|
| 945 |
+
|
| 946 |
+
#### 1. Remove audio playback in continuous conversation feedback (satellite.py)
|
| 947 |
+
```python
|
| 948 |
+
def _tap_continue_feedback(self) -> None:
|
| 949 |
+
"""Provide feedback when continuing conversation in tap mode.
|
| 950 |
+
|
| 951 |
+
Triggers a nod to indicate ready for next input.
|
| 952 |
+
Sound is NOT played here to avoid blocking audio streaming.
|
| 953 |
+
"""
|
| 954 |
+
# NOTE: Do NOT play sound here - it blocks audio streaming
|
| 955 |
+
if self.state.motion_enabled and self.state.motion:
|
| 956 |
+
self.state.motion.on_continue_listening()
|
| 957 |
+
```
|
| 958 |
+
|
| 959 |
+
#### 2. Add exception handling to tap callback (voice_assistant.py)
|
| 960 |
+
```python
|
| 961 |
+
def _on_tap_detected(self) -> None:
|
| 962 |
+
"""Callback when tap is detected on the robot.
|
| 963 |
+
|
| 964 |
+
NOTE: This is called from the tap_detector background thread.
|
| 965 |
+
"""
|
| 966 |
+
try:
|
| 967 |
+
self._state.satellite.wakeup_from_tap()
|
| 968 |
+
# ... motion feedback
|
| 969 |
+
except Exception as e:
|
| 970 |
+
_LOGGER.error("Error in tap detection callback: %s", e)
|
| 971 |
+
```
|
| 972 |
+
|
| 973 |
+
#### 3. Comprehensive microphone optimization (voice_assistant.py) - Updated 2026-01-07
|
| 974 |
+
```python
|
| 975 |
+
def _optimize_microphone_settings(self) -> None:
|
| 976 |
+
"""Optimize ReSpeaker XVF3800 microphone settings for voice recognition."""
|
| 977 |
+
|
| 978 |
+
# ========== 1. AGC (Automatic Gain Control) Settings ==========
|
| 979 |
+
# Enable AGC for automatic volume normalization
|
| 980 |
+
respeaker.write("PP_AGCONOFF", [1])
|
| 981 |
+
|
| 982 |
+
# Increase AGC max gain for better distant speech pickup (default ~15dB -> 30dB)
|
| 983 |
+
respeaker.write("PP_AGCMAXGAIN", [30.0])
|
| 984 |
+
|
| 985 |
+
# Set AGC desired output level (default ~-25dB -> -18dB for stronger output)
|
| 986 |
+
respeaker.write("PP_AGCDESIREDLEVEL", [-18.0])
|
| 987 |
+
|
| 988 |
+
# Optimize AGC time constant for voice commands
|
| 989 |
+
respeaker.write("PP_AGCTIME", [0.5])
|
| 990 |
+
|
| 991 |
+
# ========== 2. Base Microphone Gain ==========
|
| 992 |
+
# Increase base microphone gain (default 1.0 -> 2.0)
|
| 993 |
+
respeaker.write("AUDIO_MGR_MIC_GAIN", [2.0])
|
| 994 |
+
|
| 995 |
+
# ========== 3. Noise Suppression Settings ==========
|
| 996 |
+
# Reduce noise suppression to preserve quiet speech (default ~0.5 -> 0.15)
|
| 997 |
+
respeaker.write("PP_MIN_NS", [0.15])
|
| 998 |
+
respeaker.write("PP_MIN_NN", [0.15])
|
| 999 |
+
|
| 1000 |
+
# ========== 4. Echo Cancellation & High-pass Filter ==========
|
| 1001 |
+
respeaker.write("PP_ECHOONOFF", [1])
|
| 1002 |
+
respeaker.write("AEC_HPFONOFF", [1])
|
| 1003 |
+
```
|
| 1004 |
+
|
| 1005 |
+
### Fix Results
|
| 1006 |
+
|
| 1007 |
+
| Parameter | Before | After | Notes |
|
| 1008 |
+
|-----------|--------|-------|-------|
|
| 1009 |
+
| Tap continuous conversation | Blocking | Working | Removed blocking audio playback |
|
| 1010 |
+
| Microphone sensitivity | ~30cm | ~2-3m | Comprehensive AGC and gain optimization |
|
| 1011 |
+
| AGC switch | Off | On | Auto volume normalization |
|
| 1012 |
+
| AGC max gain | ~15dB | 30dB | Better distant speech pickup |
|
| 1013 |
+
| AGC target level | -25dB | -18dB | Stronger output signal |
|
| 1014 |
+
| Microphone gain | 1.0x | 2.0x | Base gain doubled |
|
| 1015 |
+
| Noise suppression | ~0.5 | 0.15 | Reduced speech mis-suppression |
|
| 1016 |
+
| Echo cancellation | On | On | Maintain clarity during TTS playback |
|
| 1017 |
+
| High-pass filter | Off | On | Remove low-frequency noise |
|
| 1018 |
+
|
| 1019 |
+
### XVF3800 Parameter Reference
|
| 1020 |
+
|
| 1021 |
+
| Parameter Name | Type | Range | Description |
|
| 1022 |
+
|----------------|------|-------|-------------|
|
| 1023 |
+
| `PP_AGCONOFF` | int32 | 0/1 | AGC switch |
|
| 1024 |
+
| `PP_AGCMAXGAIN` | float | 0-40 dB | AGC max gain |
|
| 1025 |
+
| `PP_AGCDESIREDLEVEL` | float | dB | AGC target output level |
|
| 1026 |
+
| `PP_AGCTIME` | float | seconds | AGC time constant |
|
| 1027 |
+
| `AUDIO_MGR_MIC_GAIN` | float | 0-4.0 | Microphone gain multiplier |
|
| 1028 |
+
| `PP_MIN_NS` | float | 0-1.0 | Minimum noise suppression (lower = less suppression) |
|
| 1029 |
+
| `PP_MIN_NN` | float | 0-1.0 | Minimum noise estimation |
|
| 1030 |
+
| `PP_ECHOONOFF` | int32 | 0/1 | Echo cancellation switch |
|
| 1031 |
+
| `AEC_HPFONOFF` | int32 | 0/1 | High-pass filter switch |
|
| 1032 |
+
|
| 1033 |
+
### Related Files
|
| 1034 |
+
- `satellite.py` - Removed blocking audio playback
|
| 1035 |
+
- `voice_assistant.py` - Comprehensive microphone optimization
|
| 1036 |
+
- `reachy_controller.py` - AGC entity default value updates
|
| 1037 |
+
- `entity_registry.py` - AGC max gain range update (0-40dB)
|
| 1038 |
+
- `reachy_mini/src/reachy_mini/media/audio_control_utils.py` - SDK reference
|
| 1039 |
+
|
| 1040 |
+
---
|
| 1041 |
+
|
| 1042 |
+
## 🔧 v0.5.1 Bug Fixes (2026-01-08)
|
| 1043 |
+
|
| 1044 |
+
### Issue 1: Music Not Resuming After Voice Conversation
|
| 1045 |
|
|
|
|
| 1046 |
**Problem**: Music doesn't resume after voice conversation ends.
|
| 1047 |
+
|
| 1048 |
**Root Cause**: Sendspin was incorrectly connected to `tts_player` instead of `music_player`.
|
| 1049 |
+
|
| 1050 |
+
**Fix**:
|
| 1051 |
- `voice_assistant.py`: Sendspin discovery now connects to `music_player`
|
| 1052 |
- `satellite.py`: `duck()`/`unduck()` now call `music_player.pause_sendspin()`/`resume_sendspin()`
|
| 1053 |
|
| 1054 |
+
### Issue 2: tap_sensitivity Not Persisted
|
| 1055 |
+
|
| 1056 |
**Problem**: tap_sensitivity value set in ESPHome lost after restart.
|
| 1057 |
+
|
| 1058 |
**Fix**:
|
| 1059 |
- `models.py`: Added `tap_sensitivity` field to `Preferences` dataclass
|
| 1060 |
- `entity_registry.py`: Entity setter now saves to `preferences.json`
|
| 1061 |
- Load saved value on startup
|
| 1062 |
|
| 1063 |
+
### Issue 3: Audio Conflict During Voice Assistant Wakeup
|
| 1064 |
+
|
| 1065 |
**Problem**: Audio streaming (Sendspin or ESPHome audio) conflicts when voice assistant wakes up.
|
| 1066 |
+
|
| 1067 |
**Fix**:
|
| 1068 |
- `audio_player.py`: Added `pause_sendspin()` and `resume_sendspin()` methods
|
| 1069 |
- `satellite.py`: `duck()` now pauses Sendspin, `unduck()` resumes it
|
| 1070 |
- Improved `pause()` method to actually stop audio output
|
| 1071 |
|
| 1072 |
+
### Issue 4: AttributeError for _camera_server
|
| 1073 |
+
|
| 1074 |
**Problem**: `_set_conversation_mode()` referenced non-existent `_camera_server` attribute.
|
| 1075 |
+
|
| 1076 |
**Fix**: Changed `self._camera_server` to `self.camera_server` (removed underscore prefix)
|
| 1077 |
|
| 1078 |
+
### Issue 5: tap_sensitivity Default Value Wrong
|
| 1079 |
+
|
| 1080 |
**Problem**: tap_sensitivity default was still 2.0g instead of expected 0.5g.
|
|
|
|
| 1081 |
|
| 1082 |
+
**Fix**: Use `TAP_THRESHOLD_G_DEFAULT` constant as default value
|
|
|
|
|
|
|
| 1083 |
|
| 1084 |
+
### Issue 6: Sendspin Sample Rate Optimization
|
| 1085 |
|
| 1086 |
+
**Problem**: ReSpeaker hardware I/O is 16kHz (hardware limitation), but Sendspin might try higher sample rates.
|
| 1087 |
|
| 1088 |
+
**Fix**: Prioritize 16kHz in Sendspin supported formats list to avoid unnecessary resampling
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1089 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1090 |
|
| 1091 |
---
|
| 1092 |
|
| 1093 |
+
### SDK Data Structure Reference
|
| 1094 |
|
| 1095 |
```python
|
| 1096 |
# Motor control mode
|
|
|
|
| 1108 |
STOPPED = "stopped"
|
| 1109 |
ERROR = "error"
|
| 1110 |
|
| 1111 |
+
# Full state
|
| 1112 |
+
class FullState:
|
| 1113 |
+
control_mode: MotorControlMode
|
| 1114 |
+
head_pose: XYZRPYPose # x, y, z (m), roll, pitch, yaw (rad)
|
| 1115 |
+
head_joints: list[float] # 7 joint angles
|
| 1116 |
+
body_yaw: float
|
| 1117 |
+
antennas_position: list[float] # [right, left]
|
| 1118 |
+
doa: DoAInfo # angle (rad), speech_detected (bool)
|
| 1119 |
+
|
| 1120 |
+
# IMU data (wireless version only)
|
| 1121 |
+
imu_data = {
|
| 1122 |
+
"accelerometer": [x, y, z], # m/s²
|
| 1123 |
+
"gyroscope": [x, y, z], # rad/s
|
| 1124 |
+
"quaternion": [w, x, y, z], # Attitude quaternion
|
| 1125 |
+
"temperature": float # °C
|
| 1126 |
+
}
|
| 1127 |
+
|
| 1128 |
# Safety limits
|
| 1129 |
HEAD_PITCH_ROLL_LIMIT = [-40°, +40°]
|
| 1130 |
HEAD_YAW_LIMIT = [-180°, +180°]
|
|
|
|
| 1132 |
YAW_DELTA_MAX = 65° # Max difference between head and body yaw
|
| 1133 |
```
|
| 1134 |
|
| 1135 |
+
### ESPHome Protocol Implementation Notes
|
| 1136 |
+
|
| 1137 |
+
ESPHome protocol communicates with Home Assistant via protobuf messages. The following message types need to be implemented:
|
| 1138 |
+
|
| 1139 |
+
```python
|
| 1140 |
+
from aioesphomeapi.api_pb2 import (
|
| 1141 |
+
# Number entity (volume/angle control)
|
| 1142 |
+
ListEntitiesNumberResponse,
|
| 1143 |
+
NumberStateResponse,
|
| 1144 |
+
NumberCommandRequest,
|
| 1145 |
+
|
| 1146 |
+
# Select entity (motor mode)
|
| 1147 |
+
ListEntitiesSelectResponse,
|
| 1148 |
+
SelectStateResponse,
|
| 1149 |
+
SelectCommandRequest,
|
| 1150 |
+
|
| 1151 |
+
# Button entity (wake/sleep)
|
| 1152 |
+
ListEntitiesButtonResponse,
|
| 1153 |
+
ButtonCommandRequest,
|
| 1154 |
+
|
| 1155 |
+
# Switch entity (motor switch)
|
| 1156 |
+
ListEntitiesSwitchResponse,
|
| 1157 |
+
SwitchStateResponse,
|
| 1158 |
+
SwitchCommandRequest,
|
| 1159 |
+
|
| 1160 |
+
# Sensor entity (numeric sensors)
|
| 1161 |
+
ListEntitiesSensorResponse,
|
| 1162 |
+
SensorStateResponse,
|
| 1163 |
+
|
| 1164 |
+
# Binary Sensor entity (boolean sensors)
|
| 1165 |
+
ListEntitiesBinarySensorResponse,
|
| 1166 |
+
BinarySensorStateResponse,
|
| 1167 |
+
|
| 1168 |
+
# Text Sensor entity (text sensors)
|
| 1169 |
+
ListEntitiesTextSensorResponse,
|
| 1170 |
+
TextSensorStateResponse,
|
| 1171 |
+
)
|
| 1172 |
+
```
|
| 1173 |
+
|
| 1174 |
## Reference Projects
|
| 1175 |
|
| 1176 |
- [OHF-Voice/linux-voice-assistant](https://github.com/OHF-Voice/linux-voice-assistant)
|