Sidhant argues agents.md files break down at enterprise scale, describing an internal case where a rule to use fakes instead of mocks conflicted with existing code patterns, leaving the agent to inconsistently pick one or the other. ✦ AI generated
Sidhant Pardeshi · The TWIML AI Podcast · 2026-03-10 · original ↗
starts at this moment · 39:10
“The other thing we're saying recently is research that says that agent MD can actually be counterproductive. Do you have any experience or or insights into into that?”
we have a rule that says in in Python, uh only use fakes and not mocks for writing tests, all right? Like you can think of that as a agents.md. Um but then in the code base, we've extensively used mocks, right? And we have another instruction that says always mimic the patterns that we've already used in the code base, right? Now, what do you expect the agent to do, right?
verbatim transcript · starts at 39:10
39:10So, it's really hard for the agent to prior to know what to prioritize. And especially when it leads to a conflict, right? So, for example, um we had this situation internally. So, we use uh we use busy little busy, right? And uh we have a rule that says in in Python, uh only use fakes and not mocks for writing tests, all right? Like you can think of that as a agents.md.
39:34Um but then in the code base, we've extensively used mocks, right? And we have another instruction that says always mimic the patterns that we've already used in the code base, right? Now, what do you expect the agent to do, right? So, what what is it going to happen? I think it's going to use fakes sometimes and mocks sometimes, and it's all on you, right? So, that So, those
39:55are some of the challenges why agents.md is not effective and you know, as as someone pointed out rightfully, like maybe even counterproductive in many cases. But in most of the vast majority of the smaller scale use cases, it's a pretty effective technique. Yeah, it's interesting in that context to reflect on how much of working with agents is task and context dependent. Like a lot of you know, a lot of we yeah, we we throw
40:26around a lot of directives like you should thou shalt, you know, prompt like this thou shalt prompt like that, but I guess it really just comes back to the importance of evals. Like, you know, just because you see something out on, you know, on X or whatever doesn't mean it necessarily applies to your case. Maybe you should test it, but run it through your eval suite. Yeah, and I think you hit a very
40:50important point, one that's very close to my heart. Evals I think have been consistently underperforming and I'm not going to So just today or just yesterday I believe OpenAI released an article a memo where they said we've stopped testing on SweepBench verified because the problems are not well defined and they're the they're the they contributed in creating SweepBench verified, right? They realized that gap. So they're now