Distillation depends on a realistic prompt distribution

John Schulman says a realistic prompt distribution can help a distilled model match a larger model. He contrasts this with easily verifiable tasks, which may produce strong benchmark results but weaker performance across a broader distribution.

John Schulman·Published Sep 11, 2026

Behind the viewpoint

Translations are for reading; original excerpts remain the evidence.

AI researchers debate how close we are to recursive self-improvement

Original excerpt

If you have a really good realistic prompt distribution for distillation, you can match the big model really well. But if you only have this distribution of easily verifiable tasks, then you can match the big model on all the benchmarks, but you do worse on this broader distribution

Additional context is not included in this excerpt. Read the original conversation.

Open original · 26:48 View source page

Open the episode and seek to 26:48.

Start time comes from the supplied transcript. Playback alignment is awaiting review.