Article · 2026-07-22 · 6 moments

Open models recap: more on Kimi K3, Qwen 3.8, Xi's WAIC speech, distillation, the open-closed gap, and what's next

A podcast with Florian Brand. ✦ AI generated

timeline · colored by role

Claim◆

The number of months open models are behind closed frontier models is impossible to pin down because different benchmarks give different answers.

Florian argues that measuring the open-closed gap in months is meaningless because benchmark providers cherry-pick metrics to support their preferred narrative.

transcript

Florian Brand: The biggest thing at every model release, at least at every open model release, is how much or how many months it is behind the closed frontier. People love to put a definite number onto this, which is really muddying because we have so many different benchmark providers and such so many different benchmarks that every site pulls up their favorite benchmarks to show that the current model is at the frontier, which is then countered by the other side pulling up another benchmark and showing it's actually a year behind.

Data

Kimi K3 produces significantly simpler, more readable code than Codex, but misses edge cases that Codex catches, making it roughly equivalent to a GPT-54 or 55 level for coding tasks.

Florian shares his hands-on experience using Kimi K3 for coding at Prime Intellect — the code is more readable and simpler, but it misses niche edge cases that Codex handles, landing it around GPT-54/55 level and usable for supervised runs and experiments.

transcript

Florian Brand: The main thing I found with Kimi is its code is a lot simpler which makes it way more readable, but it misses some things that Codex would be on those levels. I would say Kimi K3 is like 54-55 level for these kind of tasks. I read the code and say that's really good code, then I give it a pass over with Codex and it finds all these niche cases where it doesn't excel. But for supervising runs or running experiments it is actually really usable.

Mechanism

The open model ecosystem has professionalized significantly — model weights now ship with vLLM patches and partner integrations, unlike a year ago when the community had to figure everything out from scratch.

Florian notes that the ecosystem has matured: model releases now come with pre-built inference patches and partner support, dramatically shortening the time from release to usable deployment compared to the ad-hoc approach of a year ago.

transcript

Florian Brand: The open ecosystem has professionalized quite a lot in the last few months. During your initial rollout, they all come with some partners which have the weights beforehand. They have the vLLM patches out days or even weeks before these days, which is completely different from a year ago where basically weights got dropped and the model makers were like all right, you got to figure this out. So I expect general availability on day one will be pretty okay.

Claim◆

Chinese AI labs are more capital-efficient than US labs, and this structural advantage is a big deal as model costs scale up.

Nathan suggests Chinese labs can turn capital into compute, data, and talent more efficiently, possibly due to lower costs, better-trained talent, or a mindset focused on catching up rather than pushing the frontier.

transcript

Nathan Lambert: I think the Chinese labs are more capital efficient and you can turn capital into compute data and talent in a way that makes the models better. This is super important if it actually is some structural advantage. If a next generation model costs $10 billion for Anthropic but only $4 billion for Kimi, this could be very huge. Big Eagle replied to my tweet and said it helps because they're not trying to push the frontier, they're just trying to catch up.

supports · 1

Anecdote

The US cybersecurity community was forced to use a weaker Chinese model (GLM) to analyze a hacking attempt because frontier models had guardrails blocking defensive analysis.

Florian describes a Hugging Face report where an agent tried to hack their system, and they had to use GLM instead of GPT or Claude because the frontier models' guardrails blocked the defensive analysis, creating a perverse incentive to rely on less capable models for security work.

transcript

Florian Brand: There was that report from Hugging Face two or three days ago that they had some agent trying to hack their system. They tried to analyze it with GPT and with Claude but were unable because all the guardrails blocked them. So they had to use GLM, a lesser capable model, but it had no guardrails for this kind of defensive action. They had to use a worse model to defend themselves, which is a horrible state to be in — US-based companies relying on lesser models because the closed frontier is inaccessible.

gives example · 3supports · 1

Claim

Distillation is becoming less impactful over time as training shifts to RL, and Ben Thompson's contrary claim that distillation helps more during RL is wrong.

Nathan argues that distillation via SFT was more impactful in earlier generations, but now that RL dominates post-training, the marginal benefit of distilling from frontier models has diminished. He pushes back on Ben Thompson's claim that RL makes distillation more important.

transcript

Nathan Lambert: Distillation has become less and less impactful over time as the Chinese models get closer to the frontier and the training regime shifts to RL. Ben Thompson very strongly proclaimed that distillation is getting more impactful as you do RL. But distillation during the RL stage is a lot harder. Big RL runs are millions of rollouts — to do this on an API like Fable would be insanely expensive and might not even give you a performance uplift versus using your own tailored grader model.

rebuts · 1

Highlight slides

The open-closed gap is unknowable✦ from: The number of months open models are behind closed frontier models is impossible to pin down because different benchmarks give different answers.Conflicting benchmarks cancel each other out✦ from: The number of months open models are behind closed frontier models is impossible to pin down because different benchmarks give different answers.Chinese AI labs are more capital-efficient✦ from: Chinese AI labs are more capital-efficient than US labs, and this structural advantage is a big deal as model costs scale up.Cost gap could reshape competitive dynamics✦ from: Chinese AI labs are more capital-efficient than US labs, and this structural advantage is a big deal as model costs scale up.

Related episodes

Video · a16z Podcast

Ben Horowitz on the Global Race for Tech, Power, and Influence

→

Audio · All-In Podcast

OpenAI Misses Targets, Codex vs Claude, Elon vs Sam Trial, Big Hyperscaler Beats, Peptide Craze

→

Video · The TWIML AI Podcast

AI Factories Explained: What’s Actually New?

→