Video · 2026-08-12 · 56m · 6 moments

Why Image Generation Needs More Than Bigger Models

✦ AI generated

timeline · colored by role

Context◆

Image generation has progressed to producing plausible-looking images, but a significant gap remains between plausible outputs and precise, controllable results that faithfully match user prompts.

Text-to-image models now produce visually impressive images, but they struggle with specific compositions, distinct identities, higher resolutions, and reliable controllability. The next frontier is closing this gap.

transcript

Sam Sharenton: Once a frontier research problem in computer vision, text to image generation has reached a point where almost anyone can ask an AI system for a picture and get something that looks remarkably good. But looking good and being correct are not the same thing. Ask for several different people and the model may generate variations of the same face. Ask for a specific composition, identity or number of subjects, and it may ignore those details. Push towards higher resolution or local generation, and quality, speed, and memory quickly become constraints. The next frontier in image generation is closing this gap between plausible images and precise results. It's making these systems controllable, efficient, and reliable enough to consistently produce a high-quality rendition of the image you actually ask for.

explains mechanism · 1

Claim◆

Text-to-image models fail to generate distinct identities because their training objectives focus on realism and prompt matching without explicitly encouraging diversity between people.

When generating multiple people, T2I models produce nearly identical faces despite realistic pixel quality. The root cause is that training objectives prioritize realism and prompt alignment but lack explicit diversity incentives.

transcript

Fati Periqi: this is not an image quality problem the problem that I mentioned before like we are asking the model to generate faces and and certain number of faces and it keeps generating same faces identical almost identical faces over and over again. So quality wise, image quality wise, if you look at the pixels and noise and everything, it looks realistic. But the missing piece was that those models, base models, amazing models had not really learned to create truly distinct identities because existing training objectives focus heavily on the realism and matching the user prompt, but they don't explicitly encourage diversity between people and that is very important.

provides context · 1supports · 1

Claim

A single generative model may be overloaded when asked to simultaneously handle prompt understanding, subject counting, composition, interaction reasoning, identity preservation, and rendering.

When generating a scene with multiple people, a single model must handle prompt comprehension, subject counting, composition, interaction reasoning, identity preservation, and final rendering all at once—potentially too many concurrent objectives.

transcript

Fati Periqi: maybe I mean these are models genetic models. Maybe we are also asking a single model to solve too many difficult problems at once. Think about what happens when you generate a scene with several people. Going back to that core example now the model has to understand the prompt but what I'm asking to model and then decide how many people should appear determine where they should be placed like the composition of the scene reason about their interactions because if there's a person if there's another person most likely there there is some connection preserve and you also I didn't mention about this thing identity of the person we can give okay this is my daughter this is my son and I want them to be in the picture not like any random person and finally render everything together in all a single process. maybe this is too much

supports · 2

Mechanism◆

Separating image generation into a planning stage (architect) and a rendering stage (artist) mirrors how human artists work and produces more natural compositions.

R2CAN divides image generation into an architect that plans scene structure and an artist that renders photorealistic images, inspired by how human artists approach composition before details. This separation enables better control and more natural-looking results.

transcript

Fati Periqi: maybe it's easier to approach some of these red AI challenges like a human being. We do not like try to solve everything ourselves. Well, not just ourselves, but even like even in this domain, you know, art like you an artist approaching this problem wouldn't necessarily think about it pixel by pixel. They think about like if you know what's the subject, what's the background and there's a planning, right? In this paper, we build on it. Build on that idea. Instead of asking model to do everything, we separate planning from rendering similar to how a human artist would work. So we have two components there. Architect which is it doesn't generate pixels but instead it creates the structure or composition for the scene. Then kind of it decides where for inance if there's a person is going to be in this image where it should appear and if there are multiple people how they should be arranged you know that it should look natural and you know realistic then artist starts with that composition structure and generates the final photorealistic image while of course from one objective also preserve identities if we provide identity.

extends · 1supports · 3

Mechanism

Generating high-resolution images (4-16 megapixels) on memory-constrained devices requires a cascade approach: base generation, upsampling, then latent-space patch refinement with semantic noise injection.

Pixel Rush enables 35x faster high-resolution generation by producing a base 1K image, upsampling it, patchifying into latent space, refining with semantic noise, and blending patches with boundary-aware noise injection—all without modifying the base model.

transcript

Fati Periqi: it again starts with a prompt and there is this base generation like any model it could be let's say flat model and then it generates let's say a base image what base image what I mean by that let's say 1k image. Then we have this cascade upsampling stage. That is the part that is no about this paper. This cascade upsample what it takes it takes this image which is like no RGB pixels not latent space and then it case for instance any using any OP image super resolution solution. It could be by cubic up sampling or you know kind of it could be something smarter it generates let's say higher resolution image. So when we do that okay we have now let's say 16 megapixel in the image not one megapix we have like a lot of pixels and then we take that image sam and then we apply an encoder a va then we go into a know kind of latent space in that latent space of course I mentioned that we want to we are concerned about the memory we now divide that latent space into manageable chunks we pify them.

provides context · 2supports · 1

Mechanism

Inverting a clean image into its corresponding noise representation—rather than starting from random noise—enables seamless inpainting without boundary artifacts.

InverField achieves artifact-free inpainting by inverting the clean image into its specific noise representation, then combining that inverted noise with new noise in the mask region, allowing seamless harmonization without visible boundaries.

transcript

Fati Periqi: Those models when we train we start with the clean image and then add noise noise and at the end it becomes like noisy image. So think about the reverse process where we actually train the model. So we can take this input image and map into noise. So we are progressively inverting a real let's say clean image into noisy versions. So kind of this thing you know very well studied understood and it's very fast like 60 mcond we can take a large image and then end up create going through the reverse den noising and a noisy version of it. So this noise is not random noise anymore. It is specific to the input image. So if I change the input image the noise is going to be different. So this is about entire image and then we have this mask of the bird. Now I added noise to there for the bird because I allow I want to allow algorithm to generate a new bird compliant with my field prompt text prompt.

provides context · 1supports · 2

Highlight slides

Text-to-Image: Plausibility Achieved✦ from: Image generation has progressed to producing plausible-looking images, but a significant gap remains between plausible outputs and precise, controllable results that faithfully match user prompts.The Next Frontier: Closing the Gap✦ from: Image generation has progressed to producing plausible-looking images, but a significant gap remains between plausible outputs and precise, controllable results that faithfully match user prompts.The Next Frontier: Closing the Gap✦ from: Image generation has progressed to producing plausible-looking images, but a significant gap remains between plausible outputs and precise, controllable results that faithfully match user prompts.The Identity Problem in T2I Models✦ from: Text-to-image models fail to generate distinct identities because their training objectives focus on realism and prompt matching without explicitly encouraging diversity between people.Why Models Lack Identity Diversity✦ from: Text-to-image models fail to generate distinct identities because their training objectives focus on realism and prompt matching without explicitly encouraging diversity between people.Quality ≠ Identity Diversity✦ from: Text-to-image models fail to generate distinct identities because their training objectives focus on realism and prompt matching without explicitly encouraging diversity between people.Human-Like Image Generation: Separating Planning from Rendering✦ from: Separating image generation into a planning stage (architect) and a rendering stage (artist) mirrors how human artists work and produces more natural compositions.R2CAN Architecture: Two-Stage Approach✦ from: Separating image generation into a planning stage (architect) and a rendering stage (artist) mirrors how human artists work and produces more natural compositions.Artist: The Rendering Stage✦ from: Separating image generation into a planning stage (architect) and a rendering stage (artist) mirrors how human artists work and produces more natural compositions.

Related episodes

Video · The Cognitive Revolution

Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

→

Article · ByteByteGo Newsletter

The New American AI Model Designed to be Customized

→

Video · Machine Learning Street Talk

He won a Nobel here for AlphaFold. Then he left. - John Jumper

→