The standalone inference business (Fireworks, Together, etc.) is exploding because the open-weight model trend is exploding, and these companies are seeing massive growth with expanding gross margins.
Rory and Jason discuss Fireworks' $1.5B round at 17.5B valuation, noting it's doing over $1B ARR in 3.5 years with 40T tokens/day, and the key insight is that inference providers benefit from the open-weight trend and can charge high margins on committed compute. ✦ AI generated
Rory · 20VC · 2026-07-23 · original ↗
starts at this moment · 32:02
the story is inference. Yes. First of all I agree. Yeah inference is a hu it goes back ironically to the prior comment on openweight models. it this kind of standalone inference is a big business right and you know obviously you know inference is both something that's done within the the frontier model companies where they do their own inference and people like Microsoft and Google provide the capex and provide the compute for that but people like fireworks and base 10 they and file they all make their money offering a variety of these open openweight models to third party developers and enterprises that want to use open source models to do AI, right? And it, as I said, the two trends go together. They're exploding because the open source trend is exploding. So if you're base 10, if you're foul more media, if you're fireworks, if you're together, this is your marker than your moment, right?
verbatim transcript · starts at 32:02
32:02they're incredible. Uh thank you very that they're really [ __ ] good. Lynn is amazing. Uh doing over a billion in AR got there in three and a half years and they announced around 40 trillion tokens a day up from 15. I mean I think the story is inference. Yes. First of all I agree. Yeah inference is a hu it goes back ironically to the prior comment on openweight models. it this
32:25kind of standalone inference is a big business right and you know obviously you know inference is both something that's done within the the frontier model companies where they do their own inference and people like Microsoft and Google provide the capex and provide the compute for that but people like fireworks and base 10 they and file they all make their money offering a variety of these open openweight models to third
32:49party developers and enterprises that want to use open source models to do AI, right? And it, as I said, the two trends go together. They're exploding because the open source trend is exploding. So if you're base 10, if you're foul more media, if you're fireworks, if you're together, this is your marker than your moment, right? So yeah, I this is cuz this is how you access those because I
33:14can tell you one thing. We can going back to the discussion about open weight models from China. It's one thing to decide to use an openweight model on fireworks in the US. What you're not going to do is be using the API back to China even if they'd let you. Right? So this is a onetoone linkage between the open source the open way trend. Right? These are the
33:35companies that are benefiting massively from that trend and it's not the only kind of route for inference. There are you know inference for US-based models etc etc. But the vast bulk of it is oh my god I'm sourcing Quen. I'm sourcing Kim. I'm hosting Quen Kimmy. I want to use it as Cursor. I want to get someone to provide me some inference. These guys exist and you know they have lots of
33:56customer skew at the high end. I believe you know companies like cursor are probably big customers of all these guys. At least they were until they were acquired by um acquired and probably still are right. So yeah, I mean it's a great candidly. I think you know it actually goes back to the point I made earlier. some businesses where and I think you know you can look at and
34:16say oh my gosh the cost con you know you have margin compression in your future because you're buying your compute from the neoclouds and you know you're offering this product and are you going to be scrunched and margins you know were probably slow for a while but now the beauty of it is demand is massive so whatever compute you have today whatever compute you've already signed up for and these guys sign up for
34:39compute from the neo clouds in general are bu are starting to build their own. Whatever compute you own now, you can charge way more, which means what looked like a grow a lowish gross margin business has now probably become a very attractive business. So, not only are they probably going grow growing 5x to a billion, but they're probably growing 5x to a billion with expanding gross margins. And just just to add some