Three AI researchers discuss why we might not see superintelligence by 2036, citing bottlenecks in generalization, objective-setting, and sim-to-real transfer. They debate how distillation, RL scaling, and data quality shape progress, with timelines for ASI ranging from 3 to 10 years.
Summarized by Podsumo
Bottlenecks to recursive self-improvement: The panel agrees that even if AIs dominate coding and benchmark tasks, bottlenecks like judgment, taste, and defining objectives will slow recursive self-improvement.
Distillation democratizes progress: Distillation from frontier models, especially via router data, allows Chinese labs and others to catch up quickly, reducing centralization.
Sim-to-real gap: Many real-world tasks (e.g., law, business) are hard to simulate, and current models struggle with sample efficiency compared to humans.
RL success explained: RL works not because it learns many bits, but because it amplifies high-signal bits from correct answers, while mid-training provides a warm start.
Timelines for ASI: Estimates range from 3–10 years for an AI that dominates all cognitive computer-based work, with AI research automation seen as particularly hard.
"Alignment is the final job [for humans]."
"[Distillation] fights against the centralizing force [of model providers]."
"I think there's going to be a long tail of miscellaneous stuff which some human can do, which will take the models quite a while to do."