Video · 2026-07-17 · 14m · 6 moments

Can This Robot Be My Personal Assistant

✦ AI generated

timeline · colored by role

Claim◆

To turn Reachy Mini into a true personal assistant, it needs to be connected to Hermes with wake word detection, back-and-forth conversation, memory, tool calling, simple coding tasks, Home Assistant control, and calendar reading.

Scott lays out the full feature checklist required to turn the Reachy Mini robot into a genuine personal assistant powered by his Hermes AI system.

transcript

Scott: to make Richie a true personal assistant and claim success here, I need this robot to do a few things. I need to, one, connect it to my Hermes. I need to get wake word detection, so that way I can talk to it directly starting out of nowhere. I need to have conversation going back and forth.

extends · 1gives example · 1

Mechanism

Swapping OpenAI API calls for Hermes API calls in the Reachy conversation app instantly provided memory, tool calling across all connected profiles, and coding capability via SSH, with minimal extra work.

Because Scott already had Hermes set up with many integrations, simply pointing the robot's software at Hermes instead of OpenAI gave it memory, tool use, and coding abilities for free.

transcript

Scott: This actually gave me a ton of features out of the box considering I use My Hermes for a ton of stuff already. So just simply by connecting it to Hermes, I was able to get memory via Hancho, tool calling of all the tools that my various profiles have set up, coding tasks so it can actually do coding on any of my machines since they're all connected via SSH.

Mechanism

The hardest engineering problem was getting wake word detection and conversation timing right, since it depends on voice activity detection reading audio levels to decide when speech starts and stops.

Scott explains that the real difficulty wasn't connecting to Hermes but tuning the voice activity detection that governs when the robot listens, transcribes, and responds.

transcript

Scott: the tough stuff here was actually getting the wake word timing correct as well as conversation timing. Basically, the Richie software has voice activity detection. It decides when speech is actually starting and ending based on the audio level that's coming in.

gives example · 1

Mechanism

Hermes-based agents respond more slowly than a direct model call because by default they carry a lot of extra context tokens in every prompt, adding latency.

Scott identifies that Hermes's default agent overhead—extra tokens sent with every prompt—along with using a heavier model, is a key source of the robot's response latency.

transcript

Scott: one of the reasons why Hermes is less good for this is because by default a Hermes agents have a ton of additional tokens in here so the responses take a long time because we're sending a lot of information in each prompt rather than quick little fast prompts.

supports · 1

Anecdote

The first version of the spelling test app failed because the listening window was unclear, causing the robot to consistently mark correct spelling as wrong and frustrate the tester.

In its first live test with Scott's 9-year-old, the spelling app kept scoring correctly spelled words as wrong because the robot's listening window timing was off, leaving the child upset.

transcript

Scott: what you didn't see was him spelling the words correctly over and over again and the robot consistently getting them wrong. Specifically because the window for when he was supposed to talk wasn't clear.

supports · 1

Mechanism◆

Putting the robot in front of a computer that could take the spelling test itself, using OpenAI models to check debug info and iteratively fix and retest, made the app self-heal into a well-working version.

Scott built an automated feedback loop where a computer conducted the spelling test against the robot and used OpenAI models to debug and iteratively fix timing issues, ultimately producing a smoothly working app.

transcript

Scott: what I actually ended up doing was setting the Richie Mini in front of a computer, having that computer take the spelling test on the Richie Mini back and forth with audio and microphones, and then that way the system was able to self-heal and improve the latency using Open AI models to run the test, check the debugging information, and then fix it, test it, reiterate until now we have an app that is, I got to say, working pretty well.

provides context · 1supports · 1

Highlight slides

Making Reachy Mini a Real Personal Assistant✦ from: To turn Reachy Mini into a true personal assistant, it needs to be connected to Hermes with wake word detection, back-and-forth conversation, memory, tool calling, simple coding tasks, Home Assistant control, and calendar reading.Full Feature Checklist✦ from: To turn Reachy Mini into a true personal assistant, it needs to be connected to Hermes with wake word detection, back-and-forth conversation, memory, tool calling, simple coding tasks, Home Assistant control, and calendar reading.Robot Faces Its Own Spelling Test✦ from: Putting the robot in front of a computer that could take the spelling test itself, using OpenAI models to check debug info and iteratively fix and retest, made the app self-heal into a well-working version.Self-Healing via OpenAI Models✦ from: Putting the robot in front of a computer that could take the spelling test itself, using OpenAI models to check debug info and iteratively fix and retest, made the app self-heal into a well-working version.

Related episodes

Video · a16z Podcast

Ben Horowitz on the Global Race for Tech, Power, and Influence

→

Video · Lenny's Podcast

The rise of taste, human authenticity and judgment in an AI world | Adam Mosseri (Head of IG)

→

Audio · All-In Podcast

SpaceX-Cursor Deal, SaaS Debt Bomb, New Apple CEO, SPLC Indictment, Colon Cancer Spike

→