Current LLM coding agents are already very good at open-ended hyperparameter and architecture search — rewriting data loaders, spotting small gradients, tuning constraints — turning what used to be grid search into flexible, grad-student-like experimentation.

Eric Jang says the models he used (Opus 4.6/4.7) go well beyond traditional grid search, autonomously diagnosing issues like small gradients and rewriting code (data augmentation, optimization constraints) to squeeze out performance gains, though they still struggle to choose which experiment to run next. ✦ AI generated

Eric Jang · Dwarkesh Podcast · 2026-05-15 · original ↗

The really cool thing that automated coding can do now is search a much more open-ended set of problems. It can say, “I’ve identified that the gradients are small in this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with. Let’s try to find the best way to fit the constraints of the optimization problem.” You end up with this much more flexible, high-level, almost grad-student-like ability to just grind a performance metric.

verbatim transcript · starts at 142:16