Hermes-based agents respond more slowly than a direct model call because by default they carry a lot of extra context tokens in every prompt, adding latency.
Scott identifies that Hermes's default agent overhead—extra tokens sent with every prompt—along with using a heavier model, is a key source of the robot's response latency. ✦ AI generated
Scott · Syntax · 2026-07-17 · original ↗
starts at this moment · 6:09
one of the reasons why Hermes is less good for this is because by default a Hermes agents have a ton of additional tokens in here so the responses take a long time because we're sending a lot of information in each prompt rather than quick little fast prompts.
verbatim transcript · starts at 6:09
6:09basically just a conversational prompt. Now we also have some stop word detection where we say go to sleep and Reggie will stop the conversation. Again, this is still things that I'm tweaking and I set out to create a whole host of profiling tools so that I could understand where the latency is. Now one of the reasons why Hermes is less good for this is because by default a Hermes
6:30agents have a ton of additional tokens in here so the responses take a long time because we're sending a lot of information in each prompt rather than quick little fast prompts. It also depends of course which model you're using to have it respond to. No, I just have this connected via my Codex subscription so it is using some chunky models rather than some highly tuned fast models. That's something that I
6:54will be experimenting with more in the future. So, did this work? Well, let's start a wake listening and we're going to see if all of these things work and if we can check them off our list. I have Marshmallow here. We have wake word listening. We have conversation and we're going to first give this a try with memory and see if Marshmallow can understand. By the way, this is the
7:19custom UI that I've been building. Hey Marshmallow. >> Hey, I'm listening. >> What do you know about me? >> You're Scott. You like concise, direct technical progress, live checks, and family-friendly Reiki mini apps. You also prefer short robot prompts, and you've got two kids who may use me for spelling practice. I am listening. >> Go to sleep. >> Okay, I'll listen for hey Marshmallow. >> So, you can see that the conversation
8:08definitely works. Now, the latency is certainly there. It's something that I'm working on improving. Okay, now let's try some of the other things we had in mind. Now, I have several different profiles in Hermes that all have different abilities. And the Marshmallow profile is kid-friendly and kid-safe. However, I have a default profile that has access to things like my calendar, as well as can do coding. So, we're