Swapping OpenAI API calls for Hermes API calls in the Reachy conversation app instantly provided memory, tool calling across all connected profiles, and coding capability via SSH, with minimal extra work.
Because Scott already had Hermes set up with many integrations, simply pointing the robot's software at Hermes instead of OpenAI gave it memory, tool use, and coding abilities for free. ✦ AI generated
Scott · Syntax · 2026-07-17 · original ↗
starts at this moment · 3:35
This actually gave me a ton of features out of the box considering I use My Hermes for a ton of stuff already. So just simply by connecting it to Hermes, I was able to get memory via Hancho, tool calling of all the tools that my various profiles have set up, coding tasks so it can actually do coding on any of my machines since they're all connected via SSH.
verbatim transcript · starts at 3:35
3:35mini that has a built-in Raspberry Pi. However, I think the light version, if you have a long-running computer, makes a ton of sense, right? So, what did I do? Well, I started off with a fork of the existing conversation app. From there, I used the prompt in the docs to really get started. I then replaced all of the calls to OpenAI with calls to My Hermes, which was really simple because
3:59Hermes has a great API. And this actually gave me a ton of features out of the box considering I use My Hermes for a ton of stuff already. So just simply by connecting it to Hermes, I was able to get memory via Hancho, tool calling of all the tools that my various profiles have set up, coding tasks so it can actually do coding on any of my
4:21machines since they're all connected via SSH. So simply getting those things working was super easy, but the tough stuff here was actually getting the wake word timing correct as well as conversation timing. Basically, the Richie software has voice activity detection. It decides when speech is actually starting and ending based on the audio level that's coming in. If the voice activity detection detects speech or silence, it then starts to transcribe
4:52via the local SST, the speech to text that's done being Whisper and it's happening on my Mac mini. From there, it looks to see if the word "Hey Marshmallow" has been uttered. Marshmallow is the name of this robot. My kids named it because just look at the guy. From then, Richie would then acknowledge the conversation and then start conversation mode. Richie basically just listens for the next sentence and then
5:20after 1.1 seconds of silence, which by all means is a value that I am still tweaking and I've had to tweak based on the way kids speak to this thing, the way I speak to this thing. It's taken a lot of trial and error here. It then transcribes that text locally. That text is basically sent to Hermes in a prompt. Hermes does its thing, returns the prompt, then our software generates text