The hardest engineering problem was getting wake word detection and conversation timing right, since it depends on voice activity detection reading audio levels to decide when speech starts and stops.
Scott explains that the real difficulty wasn't connecting to Hermes but tuning the voice activity detection that governs when the robot listens, transcribes, and responds. ✦ AI generated
Scott · Syntax · 2026-07-17 · original ↗
starts at this moment · 4:21
the tough stuff here was actually getting the wake word timing correct as well as conversation timing. Basically, the Richie software has voice activity detection. It decides when speech is actually starting and ending based on the audio level that's coming in.
verbatim transcript · starts at 4:21
4:21machines since they're all connected via SSH. So simply getting those things working was super easy, but the tough stuff here was actually getting the wake word timing correct as well as conversation timing. Basically, the Richie software has voice activity detection. It decides when speech is actually starting and ending based on the audio level that's coming in. If the voice activity detection detects speech or silence, it then starts to transcribe
4:52via the local SST, the speech to text that's done being Whisper and it's happening on my Mac mini. From there, it looks to see if the word "Hey Marshmallow" has been uttered. Marshmallow is the name of this robot. My kids named it because just look at the guy. From then, Richie would then acknowledge the conversation and then start conversation mode. Richie basically just listens for the next sentence and then
5:20after 1.1 seconds of silence, which by all means is a value that I am still tweaking and I've had to tweak based on the way kids speak to this thing, the way I speak to this thing. It's taken a lot of trial and error here. It then transcribes that text locally. That text is basically sent to Hermes in a prompt. Hermes does its thing, returns the prompt, then our software generates text
5:45to speech using Cocoro using MLX on the Mac, which actually does take a bit of time and one of the things that I'm looking to improve upon the general speed and latency in conversation with this thing. We then send that audio to the robot and the robot outputs it directly so we can hear it. It then begins listening for the next prompt for me. So what you can see is it's
6:09basically just a conversational prompt. Now we also have some stop word detection where we say go to sleep and Reggie will stop the conversation. Again, this is still things that I'm tweaking and I set out to create a whole host of profiling tools so that I could understand where the latency is. Now one of the reasons why Hermes is less good for this is because by default a Hermes