This episode features KaiChao You, co-founder of Infract, discussing the journey of the open-source vLLM project from a Berkeley algorithm to a $150 million seed-funded startup. Key insights include the transition from AI research to systems engineering, the challenges of maintaining open-source ideals during commercialization, and the importance of model–Infra co-design for efficient AI infrastructure.
Summarized by Podsumo
KaiChao You transitioned from AI algorithm research to systems engineering due to the degradation of academic peer review and the realization that infrastructure scalability outweighs algorithmic tweaks in impact.
vLLM underwent a major V0 to V1 architectural overhaul in 2024, likened to transitioning from thermal to renewable energy, to keep pace with evolving models and hardware.
The vLLM team faced significant challenges from AI-generated PR spam, forcing them to prioritize contributions from recognized institutions and implement stricter verification processes.
Infract's founding was delayed by over two years due to co-founders' decisions, with offers of $20 million annual salaries from big tech firms, but they ultimately incorporated to ensure vLLM's long-term viability.
Model–Infra co-design is critical: model architecture (e.g., RoPE) must align with infrastructure to efficiently leverage specialized hardware, as demonstrated by DeepSeek's exceptional Infra team.
"If we don't start this company together, you might make a lot of money, but in ten years, if vLLM fails, will you be happy or unhappy? — KaiChao You (paraphrasing the decisive question)"
"We hope that in ten years, large model inference is a solved problem—people just think of Infract when they think of inference. — KaiChao You"
"The hardware lottery is over; now, your algorithm must win the specialized hardware lottery. — KaiChao You (paraphrasing the concept)"