Reachy Mini's voice pipeline uses local voice activity detection and Whisper speech-to-text running on a Mac mini to detect speech before checking for the wake word.

4:52via the local SST, the speech to text that's done being Whisper and it's happening on my Mac mini. From there, it looks to see if the word "Hey Marshmallow" has been uttered. Marshmallow is the name of this robot. My kids named it because just look at the guy. From then, Richie would then acknowledge the conversation and then start conversation mode. Richie basically just listens for the next sentence and then

5:20after 1.1 seconds of silence, which by all means is a value that I am still tweaking and I've had to tweak based on the way kids speak to this thing, the way I speak to this thing. It's taken a lot of trial and error here. It then transcribes that text locally. That text is basically sent to Hermes in a prompt. Hermes does its thing, returns the prompt, then our software generates text

5:45to speech using Cocoro using MLX on the Mac, which actually does take a bit of time and one of the things that I'm looking to improve upon the general speed and latency in conversation with this thing. We then send that audio to the robot and the robot outputs it directly so we can hear it. It then begins listening for the next prompt for me. So what you can see is it's

6:09basically just a conversational prompt. Now we also have some stop word detection where we say go to sleep and Reggie will stop the conversation. Again, this is still things that I'm tweaking and I set out to create a whole host of profiling tools so that I could understand where the latency is. Now one of the reasons why Hermes is less good for this is because by default a Hermes

6:30agents have a ton of additional tokens in here so the responses take a long time because we're sending a lot of information in each prompt rather than quick little fast prompts. It also depends of course which model you're using to have it respond to. No, I just have this connected via my Codex subscription so it is using some chunky models rather than some highly tuned fast models. That's something that I