42章经

少有的同时做过 Agent、多模态和具身的人 | 对谈前混元多模态 RL Infra 负责人淏楠

54 min

This podcast episode features a conversation with Haonan, a former head of multimodal RL infrastructure at Tencent's Hunyuan team, who also worked on early Agent systems at MirrorMind. The discussion traces the evolution of AI from Agents to multimodal generation and finally to embodied AI (robotics). Key insights include the current limitations of multimodal video generation, the growing importance of real-time interaction in video models, and the disruptive impact of the Sora model on the embodied AI field. Haonan argues that the path to embodied AI success lies in combining strong small models for control with large foundational models for understanding, a hybrid approach that is now more viable thanks to models like Sora.

Summarized by Podsumo

🎧 Listen 🎙️ Ask about this episode

✨ Key Takeaways

1

The guest argues that current video generation models are about 60% of the way to ideal, with significant room for improvement in consistency, controllability, and especially interactive capabilities.
2

Haonan predicts that the real breakthrough for embodied AI will come not from a single giant model, but from a hybrid system: a small, specialized model for precise control, paired with a large general model (like Sora) for high-level understanding and planning.
3

A contrarian view on embodied AI data is presented: while synthetic data from simulation is valuable, real-world data remains crucial for fine-grained control (e.g., grasping an apple with five fingers versus a pinch), especially for complex tasks.
4

The episode highlights a key shift in thinking: Sora's success suggests that the 'brain' for embodied AI (high-level reasoning) might be solved by large language models, leaving startups to focus on the 'cerebellum' (low-level motor control and hardware).
5

Haonan identifies hospitality (e.g., hotel service robots) as a promising early use case for embodied AI, due to its semi-structured environments where wheeled robots can be effective.

💬 Notable Quotes

"I felt that the biggest difference between doing Agents and multimodal generation is that in Agents, you're on the main line of human-computer interaction. If you can't find a job at one company, you can just go to another, but you're still on the main line."
"For embodied AI, a large model like GPT-4 writing '0.296' for a joint angle is a bit insane for the model. But Sora can actually output a plausible control sequence. That's why this is so shocking to the whole field."
"I think a key moment for the entire embodied AI industry is that Sora made me feel for the first time that this thing can actually land, it can really be done. Their ability is so strong, and it's not even GPT-3.5 level yet."