Onyx trains small, specialized models that act as a triage layer—capable only of deciding whether a more expensive, smarter agent needs to intervene—solving the cost and latency problem that would otherwise make agent oversight economically impossible.

(00:00:00) As you're exponentially doing more things with AIs, you're going to start having really bad actions happen. (00:00:05) And we've seen some of that happen lately with agents accidentally publishing code and tokens that they weren't supposed to. (00:00:11) Like definitely enterprise are starting to realize that that risk has grown exponentially and that they don't have any way to stop the adoption. (00:00:19) They just now have to do something to reduce the chance of these agent actions being illegitimate or incorrect. (00:00:26) But we're allowed to look at a lot of historical data of how these agents have behaved, but enterprise (00:00:30) that are not willing to have Anthropic or OpenAI keep that historical data because they know these are very data-hungry companies that will want to train on that data. (00:00:45) Hi, listeners. (00:00:46) Welcome back to Know Priors. (00:00:47) Today, I'm here with Max and Bar Kogan, the co-founder and CEO of Onyx Security, an Israel-based startup of researchers, mathematicians, and engineers, building agents to watch the AI agents. (00:00:59) We talk about specialized model training, mythos, alignment research, and the Israeli ecosystem in security and now AI. (00:01:06) Welcome. (00:01:07) Maxim, thanks so much for doing this. (00:01:09) Thank you. (00:01:09) Pleasure to be here. (00:01:11) Everyone is much more concerned about security and the impact of AI on security than they were certainly a few months ago. (00:01:19) The consensus risk story two years ago when you started the company was basically like DLP for chatbots, like what are employees putting into ChatGPT? (00:01:29) Now we have clearly something that is not quite panic, but close to market-wide panic. (00:01:36) How did you decide to bet on agent actions when you started? (00:01:41) Look, I think for us, the pivotal point was AutoGPT. (00:01:44) I think AutoGPT kind of let everyone's imagination, including ours, run wild because it was a... (00:01:51) Can you remind listeners what that was? (00:01:53) Sure. (00:01:54) AutoGPT, and I'm sorry if I don't know the guy behind it, but a huge fan, they created the first, as far as I know, first really autonomous agent running on LLMs, right? (00:02:07) So an agent that, you know, would let LLM not generate text, but decide what to do, and then give that agent an API access to do that thing a tool to do it, and then would do that in a loop. (00:02:20) So basically, (00:02:21) in theory, could let agents do very complicated things, anything a person could do on the computer. (00:02:27) Now, granted, it didn't work that well. (00:02:29) It was too early. (00:02:30) The models were not good enough. (00:02:32) GPT-4 was not good enough. (00:02:34) But I think it did give everyone a glimpse into the future of what if the models were good enough, and then basically using that same structure, we could have very capable agents doing stuff for us. (00:02:49) I think that was, in many ways, (00:02:51) Cloud code today is not dissimilar to AutoGPT back then. (00:02:55) I think they were a bit early on, again, before the models were ready, but the concept was right. (00:03:00) And the thought that stick with me was I was very high pilled even back then. (00:03:04) So I was thinking, oh my God, models are going to be way smarter than us. (00:03:10) When that happens, how do we oversee these very smart agents that are, they're smarter than us, they're very capable, (00:03:20) how we're going to feel easy about them doing stuff for us, especially when they start managing really important stuff. (00:03:25) Then one day they're managing your water supply and your electricity, your power grid, right? (00:03:31) How do you control them? (00:03:32) And that was like the thing I was kind of obsessed about that thought. (00:03:36) I was also too early. (00:03:37) So I think at the time, enterprises were not using any agents. (00:03:42) There were hardly any agents out there. (00:03:45) And talking with a lot of security buyers at the time, they were like, oh, dude, you're way too early. (00:03:50) This is not something that's going to happen. (00:03:52) I asked you the same question. (00:03:54) I said, is anyone going to do this before you run out of money? (00:03:58) And I think there was a good chance that I would have run out of money before, because I think you were right. (00:04:04) I think there was an element of chance here. (00:04:07) But then I think the market did happen. (00:04:10) So we had suddenly reasoning models that could do long horizon tasks. (00:04:14) We had Cloud Code, which became like the really first widely used autonomous agent. (00:04:21) And then we had Cowork and OpenClaw. (00:04:25) And I think we're starting to see now that these types of agents that are very autonomous, even though they're like everyone was afraid to build them. (00:04:34) So everyone started building these low-code platforms that were much more limited, much more based on connectors. (00:04:41) Those platforms ended up being quite limited, so we didn't get the productivity gains from those limited platforms. (00:04:47) But when we started getting the crazy benefits from these very unleashed agents that could do everything, that had much less controls baked into them, and even very large enterprises decided they're going to adopt it, you know, like Anthropix revenue is coming from... (00:05:02) enterprises that are paying for cloud code to do a lot of the work that developers used to do. (00:05:07) That was a bit about kind of how we started, and we definitely were in luck that very autonomous agents appeared before it was too late. (00:05:17) So can you describe a little bit, just because it's, I think, both... (00:05:23) close to impossible, and then very useful in this period of AI to think about what is deployment right now, and then what's changing about capability? (00:05:31) What's the one-liner on what the Onyx product does today, and then how you think about long-term vision? (00:05:37) Today, Onyx is really does two things. (00:05:42) Number one is we train models and build agents that can oversee other agents. (00:05:49) And the goal of that is to say, okay, we need someone to be able to tell that all of these actions that are now happening by these AIs that we're adopting are legitimate because the number of these actions is growing exponentially. (00:06:03) And so things that we thought might be useful in the past, like a human in the loop, now that you're going to have 100X, 1,000X, a million X of these actions, that's not going to work. (00:06:14) And then (00:06:15) We take that capability and we basically productize it in the product that we call the control plane or the secure control plane, where we come to the present and say, Hey, let's find all of your AIs and autonomous agents and hook them up to Onyx, to this system where we can oversee what your AIs are doing so that you don't run into the risk of as you're exponentially doing more things with AIs. (00:06:39) you're going to start having really bad actions happen. (00:06:41) And we've seen some of that happen lately with downtimes that were caused by agents doing their own thing, agents accidentally publishing code and tokens that they weren't supposed to, and so on. (00:06:53) So definitely enterprises are starting to realize that that risk has grown exponentially and that they don't have any way to stop the adoption. (00:07:02) So they just now have to do something to (00:07:06) reduce the chance of these agent actions being illegitimate or incorrect. (00:07:12) Yeah, I think one of the core reasons, obviously, the... (00:07:17) foundation model labs are going after code is because it is very powerful in general and can do, you know, in theory, all things software can over time. (00:07:28) The flip side of that is it can do all things software can, right? (00:07:31) And so I joyously am already in the camp of having allowed a, having been over permissive with my agents such that it deleted data permanently and caused rework. (00:07:41) So I'm like, oh, (00:07:42) Okay. (00:07:42) I think I need some guardian spirits around it. (00:07:47) Given your deployments today and talking to large enterprises, what is the state of deployment, right? (00:07:53) Like, how much do you see that's within these more scoped, like studio-like platforms versus, you know, free-riding coding agents? (00:08:05) You know, how much are you actually seeing in large enterprises and in different sectors? (00:08:09) Yeah, so I think (00:08:10) Right now, in our typical enterprise, we're going to see if we break it down to three categories. (00:08:16) So we break it down to various SaaS platforms that are typically more low-code, where people build agents in this drag-and-drop way. (00:08:25) And they're not really autonomous agents, right? (00:08:27) They're kind of the same kind of, I would think of them more as AI automations. (00:08:31) And then there are (00:08:33) and first party agents, people are building in their cloud, potentially because it's an application they want inside the company or even a product they're planning to release to the customers that is Agentic. (00:08:44) And then the third category is very autonomous coding agents and assistants of these categories, I would say roughly at this point, over 50% is the autonomous coding agents and assistants in the average enterprise, then probably (00:09:02) 45% is those low-code automations, and the last 2% are really the first-party ones that they're building themselves, because obviously it's much harder to build effective agents, so, and it's much easier to adopt agents off the shelf or build them with low-code. (00:09:20) So, and that's what we're seeing, and we do see that the autonomous is also the fastest growing category. (00:09:26) So, it used to be that only developers (00:09:29) and we would see cloud code growing like fire in our customer base, and now we're seeing a cloud cowork growing even faster. (00:09:36) We're starting to see, to our own surprise, actually, people adopting OpenCloud as a legitimate sanction tool in the company because the CEO is very driven to adopt AI. (00:09:49) So I think that today, Autonomous AI is by far the fastest growing category and (00:09:56) Today typically comes without any controls. (00:09:59) So enterprises already buy, let's say, $100 billion of security today. (00:10:07) They have lots of different protections at the endpoint and network and cloud and identity, domains. (00:10:17) What's relevant here for securing agents or is none of it? (00:10:21) Like, how do you think about the existing protection side? (00:10:24) Security is always a space where you have some overlap between different tooling, but in this, and you have the concept of defensive debt as well. (00:10:34) So you want to have defenses at different levels of your technology stack to solve the problem. (00:10:40) And that said, I think in this space, we're kind of, a lot of enterprises are kind of helpless. (00:10:47) Because I'll take an example, the identity approach. (00:10:51) Like traditionally, if we have a software system that's running in our company, our first and most important control will be to limit what permission it has, right? (00:11:03) And then no matter what, even if it goes wrong, even if it's compromised, it can't typically do stuff that was originally allowed to do. (00:11:12) But with these autonomous AIs, with these assistants, with these coding agents, (00:11:17) We kind of want them to have our permissions because we want to tell Cloud Code to do something or Cloud Cowork to do something, and we want to then go have lunch and we want to come back and see that it's done. (00:11:29) And we want to give it so many diverse tasks as well that we kind of can't find the right set of permissions to do. (00:11:37) So suddenly our identity security software is not very useful. (00:11:41) Then if you think about endpoint security, right, or API security, (00:11:48) if we tell our cloud code that we want to recreate a database and it should delete it and recreate it, that's great. (00:11:56) That's going to save our DevOps team and our platform teams a lot of time. (00:12:00) It's a great benefit of Cloud Code, but if Cloud Code is working on an unrelated task and suddenly thinks that maybe the right thing to do is to delete our database and recreate it, maybe we don't want that to happen. (00:12:12) And unfortunately, our endpoint providers or API security tools, they don't know what Cloud was thinking, why is it doing what it's doing, right? (00:12:20) So a lot of these existing tools, they don't have the context to understand what these very flexible, unpredictable systems are doing if you're not (00:12:29) building some kind of controls that are built for these systems, then you're either gonna end up there limiting them a lot, making them almost much less useful to the enterprise, or you're gonna miss a lot of pretty dangerous things that they might be doing. (00:12:45) As somebody who has worked in security for a long time, my first very traditional instinct on a problem like this is like, (00:12:53) That sounds like a problem for a proxy with a policy engine. (00:12:57) We make some rules, we make the rules smarter. (00:13:00) Why doesn't that work? (00:13:01) Or did you try it? (00:13:03) There are a few things that, I mean, proxy's integration method, I would say. (00:13:08) So there are some AI systems where you would want to integrate with a proxy if that's the easiest way to do it. (00:13:16) But number one, there's a lot of systems where that's just not viable technically because AI today runs (00:13:22) on the cloud, on someone else's infrastructure, on your endpoint, and just proxy is not always an option. (00:13:28) And the second thing is the question, okay, great, you're proxying, so you're seeing the data, you're seeing it, but that's not the hard problem. (00:13:36) The hard problem is understanding if what I should do now. (00:13:39) It turns out that in the case of AI systems, that is the hard question. (00:13:43) Like, what is the engine that needs to underwrite these different actions and say if they're okay or not? (00:13:50) And because (00:13:52) We need to be able to understand what another AI system is thinking, what is it planning to do, and then have our own opinion on that. (00:14:00) And consider we're trying to understand some of the smartest models in the world are doing the right thing. (00:14:06) So who are we to do it? (00:14:06) How are we going to do it correctly, right? (00:14:08) And so that turns out to be a really difficult technical question. (00:14:12) Part of the solution for Onyx has been training its own models. (00:14:17) Like what can you say about that? (00:14:19) If you try today, let's say we're trying to build a solution to oversee and kind of control how other agents are operating. (00:14:27) Maybe the first thing a lot of our listeners might think is say, well, I'll just ask Cloud Code to do it. (00:14:34) And in a sense, they would be right because Cloud Code is great. (00:14:38) And maybe we can ask it to spawn a version of itself for every agent that we have and kind of keep monitoring everything that agent is planning to do. (00:14:48) And if you think that there's a problem, intervene. (00:14:52) So that approach, it has, obviously it's pretty naive and there are some ways in which it totally fails we could talk about, but it has some merit to it, right? (00:15:01) So it does seem intuitive that it's a good idea to have capable agents reviewing what other agents are doing, same as we have capable humans reviewing what other humans are doing, right? (00:15:14) But then the problems that you're going to run into is, (00:15:17) how do I make this work from a cost, latency, reliability perspective? (00:15:25) Because if I need to run an agent for every agent you're running as your security vendor, you're gonna be paying for me more than you're paying for your AI, right? (00:15:35) So it's not, it's pretty much a deal breaker. (00:15:38) And also it's gonna be so slow, so you're not gonna be happy with whatever latency you're gonna get. (00:15:43) And so the challenge then becomes, (00:15:47) How do I know what are the times where I need to interject with these smart agents to look at what's happening? (00:15:56) And that's when actually what you want to do is you want to try to train very smart models that are, actually, let me correct that, very not smart models, but models that are just good at one thing. (00:16:10) They're very small. (00:16:11) They almost can't do anything else other than be able to say, (00:16:15) Should I have a smarter agent look at this? (00:16:17) And if you manage to bake in that intuition into those small models, well, in the sense that they don't miss a lot of stuff and they don't call that other agent too much, then you can get to a really good balance of, we're very performant, we have smart agents overseeing things when needed, but we're not, but our costs are low and our latency is low. (00:16:42) And then that becomes the challenge because you need to make sure that as the frontier models get smarter and the hardness has become more evolved, you need to be able to have models that are on your side that are small and effective at continuously being able to say, now's the time, this is the action where I think someone should take a closer look. (00:17:07) And that's why Onyx trains and models for this purpose and it's, you know, most of (00:17:12) the hard things that we're doing in this space. (00:17:14) Yeah, you and I actually both love to play blitz chess. (00:17:18) And I look at Guardian as a system that's a little bit analogous. (00:17:23) Like, it's not clear either of us is going to be competitive with Magnus in a real game. (00:17:28) But if you play enough times with the right data, then all you have to do is make intuitive decisions under time pressure very, very quickly. (00:17:38) It's actually a different game, right? (00:17:40) And do you think that makes (00:17:42) sense? (00:17:42) Or am I reaching here? (00:17:43) Yeah, actually, I didn't think about it, but yeah, there's a lot of analogies because I think if you look at top chess players in the world, most of the moves that they make are intuitive. (00:17:53) They don't calculate forward. (00:17:54) They've seen so much games and they've played so much games that they already have a good sense of what is the right move and that they're not taking too much risk here by taking this move without calculating. (00:18:05) Then if you look at those games, every once in a while, they do stop for suddenly (00:18:11) a really long period of time to just calculate forward a lot of options because they know this is a critical move in the game. (00:18:17) There's risk. (00:18:18) You need to think through what you're doing and you need to decide correctly. (00:18:22) I think that's very similar. (00:18:23) It's the efficient way to run computation, right? (00:18:26) You don't want to spend too much intelligence where you don't have to, and you want to spend a lot of intelligence, overwhelmingly a lot, in situations where there's high risk. (00:18:38) You guys are a team mostly based in Israel today. (00:18:41) I think the world has accepted that there is a cohort of amazing Israeli security talent that comes out of the military and offensive security and then repeat entrepreneurs like you guys. (00:18:55) I think the DNA at Onyx is a little bit different here. (00:18:58) Your co-founder, Gil, came out of building synthetic data and working in NVIDIA. (00:19:03) What would you characterize the talent at Onyx as particularly (00:19:08) good at? (00:19:08) And then, are people actually training interesting frontier models in Israel now? (00:19:15) So first of all, I think Israel is a bit, started maybe a bit late in the game, but it's catching up quickly. (00:19:21) So I think there's now amazing companies in Israel building world models, building AI infrastructure that's top of its glass, building chips. (00:19:31) So I think Israel in general is becoming very strong in AI. (00:19:36) And we're proud to be a part of that movement. (00:19:39) And I think you're right, our company has a very mixed DNA between cyber and AI, which kind of reflects mine and Gail's backgrounds. (00:19:48) Most of the people in our company, most of our research engineering come from a unit in the Israeli intelligence where we actually deal with math and cyber and the intersection thereof. (00:20:02) And so I think it is also reflected in kind of the type of talent that we bring in. (00:20:06) I think it's important for a few reasons. (00:20:09) The first and foremost is that we want it to be more than just a security company long term. (00:20:13) We think that to solve this problem well, it's going to require deep AI expertise, but then that the problem is not just cybersecurity. (00:20:22) The problem is how do we control advanced AI long term? (00:20:27) And the (00:20:29) That problem, even if you just forget about enterprise security and the different gaps in various controls that they currently have, first principles, that problem just sounds very important to me. (00:20:40) So I think it will be crucially important if you have AI companies that are $10 trillion companies, we think you want a company that is not the vendor of the AI itself to oversee and help you control what AI is doing. (00:20:55) And we think that's an opening that's $100 billion plus opening for a really important company. (00:21:03) And then if you think about what it's going to take to control advanced data long term, then we're just scratching the surface because long term, you're going to have to also understand much better what models are thinking, what models are, what's happening on the internals of these models as they're operating. (00:21:21) And that's also a lot of where our research is focused. (00:21:25) So the industry is quite divided on this issue. (00:21:29) I mean, amongst the people who think about whether or not mechanistic interpretability or research into better understanding models is possible. (00:21:39) Like that's a question. (00:21:41) And so it's something you believe in. (00:21:43) We believe that there's been a lot of strong progress in that direction. (00:21:49) We believe that understanding the internal weights and activations (00:21:55) is the internal structure, the mathematical structure of these systems is going to be at least part of the solution. (00:22:02) And in many ways we think that, and this is maybe, you know, we'll only know when we get there, but we think that for our level of intelligence, it's kind of difficult to understand very quickly what is the internal structure of (00:22:22) A large language model. (00:22:23) What is the internal structure of the weights and activations? (00:22:27) Our level, like human intelligence or our level of your models? (00:22:30) Okay, human intelligence. (00:22:31) Oh yeah, I think like, yeah, exactly. (00:22:33) I think as humans, it might still be very difficult to understand what weights and activations mean and maybe mechanistic interpretability. (00:22:42) It seems like, oh, maybe that's too hard or shouldn't be possible. (00:22:44) But as we're starting to have models that are much smarter than us, at least in some important ways, we think that we'll be able to start cracking mechanistic capability much more effectively. (00:22:56) And I think it's going to be extremely rewarding, by the way, long-term for understanding intelligence in general, like not just overseeing, but just understanding what intelligence is. (00:23:10) how it works? (00:23:10) What's the difference between the smarter model and the less smart model? (00:23:13) I completely agree that the opportunity to understand and trust and secure and govern these super intelligent AIs is a very large opportunity. (00:23:27) If we just scroll back today, the security person in me says, Well, then I have to give you all the permissions and understanding that I have to give these companies to. (00:23:35) Like, how do you get (00:23:37) customers or the Fortune 100s you're working with already, or tech natives, everybody cares about their own security and business to trust you now as a, you're like less than 100 people, right? (00:23:51) I think it's one of those things that should not be possible. (00:23:56) So in theory,