Video · 2026-08-12 · 56m · 6 moments

Can We Make AI Image Generation More Controllable?

✦ AI generated

timeline · colored by role

Claim◆

The core limitation of current text-to-image models is not image quality but controllability — they fail to generate distinct identities when asked for multiple people, producing near-identical faces instead.

Despite photorealistic outputs, current T2I models struggle with identity preservation across multiple subjects. Training objectives optimize for realism and prompt matching but do not explicitly encourage facial diversity, leading to the same face duplicated across groups.

transcript

Fati Periqi: This is not an image quality problem the problem that I mentioned before like we are asking the model to generate faces and and certain number of faces and it keeps generating same faces identical almost identical faces over and over again. So quality wise, image quality wise, if you look at the pixels and noise and everything, it looks realistic. But the missing piece was that those models, base models, amazing models had not really learned to create truly distinct identities because existing training objectives focus heavily on the realism and matching the user prompt, but they don't explicitly encourage diversity between people.

explains mechanism · 2provides context · 4supports · 2

Mechanism◆

Explicitly optimizing for identity diversity as a reinforcement learning objective — including both intra-image diversity (distinct faces within one image) and inter-image diversity (different faces across runs) — dramatically improves facial distinctness without requiring large datasets.

The DISCO paper fine-tunes existing T2I models using GRPO with reward signals for intra-image and inter-image face diversity plus correct subject count. This raises unique-face accuracy from very low baseline scores to 98–99%, with efficient training via curriculum learning.

transcript

Fati Periqi: Instead of creating a completely new T2I model, we kept the underlying model and fine-tune it with reinforcement learning. We designed rewards that encourage several things simultaneously. For instance, different people within an image should have distinct identities. You don't want to duplicate create duplicate faces. That's something we call in the paper as intra image diversity and across different runs of the same model with similar prompts same prompt we should not keep producing generating the same faces. So this is inter run inter image diversity. So these are explicit new objectives when we train fine-tune the model and also we want model to generate the correct number of people if I'm asking generate two people that should be two not three. We see that when we explicitly impose such objective diversity objective the score for instance unique face accuracy deduction score significantly improves. Disco is around 98 99 but the models that we started that doesn't have such explicit diversity objective they are very low there is maybe more than 10 20 percentage gap.

provides context · 1

Claim◆

A single generative model is being asked to solve too many difficult sub-problems simultaneously — understanding prompts, determining composition, preserving identities, reasoning about interactions, and rendering — which may exceed what one model can do well.

Generating a multi-person scene requires simultaneous prompt comprehension, subject count determination, spatial composition, interaction reasoning, identity preservation, and rendering. Fati argues this load is too heavy for a single end-to-end model, motivating modular architectures.

transcript

Fati Periqi: Maybe we are also asking a single model to solve too many difficult problems at once. Think about what happens when you generate a scene with several people. Going back to that core example now the model has to understand the prompt but what I'm asking to model and then decide how many people should appear determine where they should be placed like the composition of the scene reason about their interactions because if there's a person if there's another person most likely there there is some connection preserve and you also I didn't mention about this thing identity of the person we can give okay this is my daughter this is my son and I want them to be in the picture not like any random person and finally render everything together in all a single process. maybe this is too much.

gives example · 1provides context · 1supports · 3

Mechanism

Separating image generation into an Architect (planning composition) and an Artist (rendering pixels) — mirroring how human artists work — makes complex multi-subject scenes more tractable than end-to-end generation.

The RT-CAM paper splits generation into two components: an Architect that outputs spatial composition (face locations as XY coordinates) without generating pixels, and an Artist (a fine-tuned diffusion model) that renders photorealistic output from that structure. Both are trained with GRPO, using rewards for composition accuracy, pose, identity, and count.

transcript

Fati Periqi: Instead of asking model to do everything, we separate planning from rendering similar to how a human artist would work. So we have two components there. Architect which is it doesn't generate pixels but instead it creates the structure or composition for the scene. Then it decides where for instance if there's a person is going to be in this image where it should appear and if there are multiple people how they should be arranged you know that it should look natural and realistic then artist starts with that composition structure and generates the final photorealistic image while of course from one objective also preserve identities if we provide identity. So yeah this is like a planning an architect then an artist type of framework.

supports · 3

Mechanism

High-resolution image generation (4–16 megapixels) on memory-constrained mobile devices is achievable by generating a base image, upsampling, patchifying in latent space, and applying semantic noise injection to refine texture and seamlessly blend patch boundaries.

Pixel Rush generates a base 1K image, upsamples it with off-the-shelf super-resolution, then encodes to latent space and divides into patches. Each patch gets guided semantic noise to add high-resolution texture, and boundary-aware noise blending resolves seam artifacts — all within the original latent space dimensions. This yields ~35× speedup (from ~10 minutes to ~20 seconds) while keeping memory footprints mobile-compatible.

transcript

Fati Periqi: It starts with a prompt and there is this base generation like any model... and then it generates let's say a base image... 1k image. Then we have this cascade upsampling stage... it generates let's say higher resolution image. So when we do that we have now let's say 16 megapixel in the image... then we take that image and then we apply an encoder, a VAE, then we go into latent space in that latent space... we divide that latent space into manageable chunks we patchify them... when we are doing blending across such latent space data representation of for these 16 patches we like to again do another noise injection when we combine them together and but in this case it is not like all over the patch same in the latent space... we kind of allow algorithm to generate more noise towards the boundary but in the center of the patches be maybe less at a lesser degree... it's maybe 35 times faster. You know, from let's say 10 minutes to around 20 seconds type of acceleration.

provides context · 3supports · 1

Mechanism

Image inpainting quality can be dramatically improved by inverting the input image into its corresponding noise representation and blending it with mask-region noise, so the background semantically influences foreground generation and eliminates boundary artifacts.

InverField solves the boundary-artifact problem in image inpainting by first inverting the clean input image into its noise representation (a fast ~60ms process), then initializing the diffusion with that inverted noise everywhere except the mask region, which gets new noise. Because both noise distributions share the same latent space from the start, the background seamlessly harmonizes with the newly generated foreground — no visible seams.

transcript

Fati Periqi: We can take this input image and map into noise. So we are progressively inverting a real clean image into noisy versions... this noise is not random noise anymore. It is specific to the input image. So if I change the input image the noise is going to be different... Now I changed the way that we create this image... when we start with this inverted noise plus the mask and new noise within the mask. Now first of all we can retain background but we also allow background to slightly impact the foreground like the mask itself allows seamless harmonization and it generates high quality images but most importantly there's no boundary artifacts anymore because we have two noise like no like random noise and inverted noise within the same when we started the image we are not starting from just noise within the mask.

provides context · 2supports · 1

Highlight slides

The Real Limitation: Controllability, Not Quality✦ from: The core limitation of current text-to-image models is not image quality but controllability — they fail to generate distinct identities when asked for multiple people, producing near-identical faces instead.What's Missing in Training✦ from: The core limitation of current text-to-image models is not image quality but controllability — they fail to generate distinct identities when asked for multiple people, producing near-identical faces instead.解决方案方向✦ from: The core limitation of current text-to-image models is not image quality but controllability — they fail to generate distinct identities when asked for multiple people, producing near-identical faces instead.DISCO: Identity Diversity via RL✦ from: Explicitly optimizing for identity diversity as a reinforcement learning objective — including both intra-image diversity (distinct faces within one image) and inter-image diversity (different faces across runs) — dramatically improves facial distinctness without requiring large datasets.Method & Training✦ from: Explicitly optimizing for identity diversity as a reinforcement learning objective — including both intra-image diversity (distinct faces within one image) and inter-image diversity (different faces across runs) — dramatically improves facial distinctness without requiring large datasets.Efficient Training Approach✦ from: Explicitly optimizing for identity diversity as a reinforcement learning objective — including both intra-image diversity (distinct faces within one image) and inter-image diversity (different faces across runs) — dramatically improves facial distinctness without requiring large datasets.The Single-Model Overload Problem✦ from: A single generative model is being asked to solve too many difficult sub-problems simultaneously — understanding prompts, determining composition, preserving identities, reasoning about interactions, and rendering — which may exceed what one model can do well.What Multi-Person Generation Demands✦ from: A single generative model is being asked to solve too many difficult sub-problems simultaneously — understanding prompts, determining composition, preserving identities, reasoning about interactions, and rendering — which may exceed what one model can do well.Implications for Model Architecture✦ from: A single generative model is being asked to solve too many difficult sub-problems simultaneously — understanding prompts, determining composition, preserving identities, reasoning about interactions, and rendering — which may exceed what one model can do well.

Related episodes

Video · The Cognitive Revolution

The core limitation of current text-to-image models is not image quality but controllability — they fail to generate distinct identities when asked for multiple people, producing near-identical faces instead.

Explicitly optimizing for identity diversity as a reinforcement learning objective — including both intra-image diversity (distinct faces within one image) and inter-image diversity (different faces across runs) — dramatically improves facial distinctness without requiring large datasets.

A single generative model is being asked to solve too many difficult sub-problems simultaneously — understanding prompts, determining composition, preserving identities, reasoning about interactions, and rendering — which may exceed what one model can do well.

Separating image generation into an Architect (planning composition) and an Artist (rendering pixels) — mirroring how human artists work — makes complex multi-subject scenes more tractable than end-to-end generation.

High-resolution image generation (4–16 megapixels) on memory-constrained mobile devices is achievable by generating a base image, upsampling, patchifying in latent space, and applying semantic noise injection to refine texture and seamlessly blend patch boundaries.

Image inpainting quality can be dramatically improved by inverting the input image into its corresponding noise representation and blending it with mask-region noise, so the background semantically influences foreground generation and eliminates boundary artifacts.

LTX CEO on Video Gen + Can we beat AI superforecasters?

Britain Is Losing The AI Race To America - Alistair Pullen (Cosine AI)

Watching America Run Away With AI - Alistair Pullen (Cosine AI)