Article · 2026-06-15 · 6 moments

Import AI 461: "Alignment is not on track"; FrontierCode; and synthetic research interns

Where are your agents right now? ✦ AI generated

GUEST Sequent GUEST Xiaomi HOST Jack Clark

Claim◆

Artificial superintelligence may arrive within a few years, and it is unclear whether alignment research is on track to be ready in that same timeframe, since current empirical programs at AI labs are unlikely to deliver a priori confidence that training an ASI will go well.

Sequent, a new nonprofit formed from UK AI Security Institute and Timaeus researchers, argues alignment work is not keeping pace with the likely timeline for superintelligent AI, motivating its launch.

transcript

Sequent: Artificial superintelligence (ASI) may be developed in the next few years. It is unclear whether alignment is on track to be ready on the same timeframe. At a minimum, the empirical programs at AI labs are unlikely to deliver a priori confidence, before training ASI, that things will go well.

explains mechanism · 1supports · 2

Context

As AI systems get smarter and take over more of the research enterprise, potentially undergoing recursive self-improvement, we need much better alignment techniques, and independent organizations like Sequent are valuable because they can raise the alarm if frontier labs act dangerously.

Import AI's author argues that stronger, independently-developed alignment techniques are essential as AI takes on more autonomous research and self-improvement work, and that watchdog-style organizations can help sound the alarm on frontier labs.

transcript

Jack Clark: We definitely need better alignment techniques to be confident of things like RSI. Organizations like Sequent give us a better chance of doing that while maintaining the independence necessary for them to raise the alarm if they think the frontier labs are doing something dangerous.

Mechanism

Frontier AI labs' alignment methods are essentially reactive and functional but do not yield principled insight into if or when they will fail, unlike the portfolio of theory-driven bets Sequent intends to pursue.

Sequent positions its research strategy as seeking principled confidence that alignment generalizes to uncontrolled, real-world situations, contrasting this with what it sees as the reactive methods used by major AI labs.

transcript

Sequent: This is in contrast to the approach of most frontier AI labs, which Sequent describes as "essentially reactive, resulting in methods that, while functional, do not yield principled insight into if or when they will fail."

supports · 1

Data◆

FrontierCode is hard enough that even Claude Opus 4.8 scores only 13.4% on its hardest 'Diamond' tier, which gives confidence the benchmark will remain a useful measure of AI coding progress for years to come.

Cognition's new FrontierCode benchmark proves extremely difficult for top models — Claude Opus 4.8 manages just 13.4% on the Diamond tier — suggesting it will resist saturation longer than prior coding benchmarks like SWE-Bench.

transcript

Jack Clark: The best part about the benchmark is how hard it is - Claude Opus 4.8 gets a score of 13.4% on the hardest ("Diamond") component of the benchmark, giving me some confidence that FrontierCode will be a useful way to assess progress of AI systems in the coming years.

Definition

The AARR benchmark suite tests whether AI agents can emulate the professionalism, thoroughness, and nuanced reasoning of human researchers, specifically whether an agent can perform entry-level research tasks with appropriate diligence and methodology.

The AARRI-Bench benchmark evaluates AI agents on research-intern-level tasks—like spotting fabricated data or refusing to falsify results under pressure—with the top model, Claude-Opus-4.7, scoring 68.3%, suggesting AI is starting to be useful as a research assistant.

transcript

AARR researchers (Xi'an Jiaotong University and Xidian University): "AARR focuses on whether agents can emulate the professionalism, thoroughness, and nuanced reasoning that characterize human researchers in granular research scenarios," they write. AARRI-Bench studies "the ability of an agent to perform entry-level research tasks with appropriate diligence and methodology".

Mechanism

Xiaomi reached 1000 tokens/second on a 1-trillion-parameter model by codesigning the model with its software stack—FP4 quantization, DFlash speculative decoding, and TileRT-optimized inference—running on an ordinary 8-GPU commodity node rather than specialized hardware.

Xiaomi's MiMo-V2.5-Pro-UltraSpeed model hits 1000 tokens per second through co-designed quantization and speculative decoding techniques running on commodity 8-GPU nodes, reflecting a broader push by Chinese firms to maximize efficiency amid export controls.

transcript

Xiaomi: Xiaomi was able to do this by codesigning the model with the software stack around it, including obvious things like FP4 quantization, as well as using DFlash (a "speculative decoding method based on block-level masked parallel prediction"), and also working closely with TileRT, software from startup Tile AI which speeds up LLM inference on commodity hardware.

Highlight slides

ASI May Arrive Within Years✦ from: Artificial superintelligence may arrive within a few years, and it is unclear whether alignment research is on track to be ready in that same timeframe, since current empirical programs at AI labs are unlikely to deliver a priori confidence that training an ASI will go well.FrontierCode: Built to Resist Saturation✦ from: FrontierCode is hard enough that even Claude Opus 4.8 scores only 13.4% on its hardest 'Diamond' tier, which gives confidence the benchmark will remain a useful measure of AI coding progress for years to come.Why Alignment May Fall Short✦ from: Artificial superintelligence may arrive within a few years, and it is unclear whether alignment research is on track to be ready in that same timeframe, since current empirical programs at AI labs are unlikely to deliver a priori confidence that training an ASI will go well.Why This Matters✦ from: FrontierCode is hard enough that even Claude Opus 4.8 scores only 13.4% on its hardest 'Diamond' tier, which gives confidence the benchmark will remain a useful measure of AI coding progress for years to come.Jack Clark's Take✦ from: FrontierCode is hard enough that even Claude Opus 4.8 scores only 13.4% on its hardest 'Diamond' tier, which gives confidence the benchmark will remain a useful measure of AI coding progress for years to come.

Related episodes

Article · Latent Space

Artificial superintelligence may arrive within a few years, and it is unclear whether alignment research is on track to be ready in that same timeframe, since current empirical programs at AI labs are unlikely to deliver a priori confidence that training an ASI will go well.

As AI systems get smarter and take over more of the research enterprise, potentially undergoing recursive self-improvement, we need much better alignment techniques, and independent organizations like Sequent are valuable because they can raise the alarm if frontier labs act dangerously.

Frontier AI labs' alignment methods are essentially reactive and functional but do not yield principled insight into if or when they will fail, unlike the portfolio of theory-driven bets Sequent intends to pursue.

FrontierCode is hard enough that even Claude Opus 4.8 scores only 13.4% on its hardest 'Diamond' tier, which gives confidence the benchmark will remain a useful measure of AI coding progress for years to come.

The AARR benchmark suite tests whether AI agents can emulate the professionalism, thoroughness, and nuanced reasoning of human researchers, specifically whether an agent can perform entry-level research tasks with appropriate diligence and methodology.

Xiaomi reached 1000 tokens/second on a 1-trillion-parameter model by codesigning the model with its software stack—FP4 quantization, DFlash speculative decoding, and TileRT-optimized inference—running on an ordinary 8-GPU commodity node rather than specialized hardware.

[AINews] OpenAI GPT-5.6 Sol / Terra / Luna — restricted to trusted partners

Using Local Coding Agents

Banning Open Source AI Would Be A Mistake