Latent Space: The AI Engineer Podcast

Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

58 min

Modal CTO Akshat Bubna discusses how the cloud platform evolved from a serverless runtime for bursty workloads into ideal infrastructure for AI agents, shifting focus from developer experience to agent experience. Key innovations include elastic inference across 17 cloud providers, speculative decoding for 2-4x speedups, and sandboxes as a core primitive for production agents. The conversation covers infrastructure trends like GPU/CPU co-location, RL burstiness, and why Modal avoids the managed agent API market in favor of lower-level compute primitives.

Summarized by Podsumo

🎧 Listen 🎙️ Ask about this episode

✨ Key Takeaways

1

Modal pivoted from a serverless runtime for bursty data workloads to become a leading AI infrastructure platform, now focusing on 'agent experience' (AX) alongside developer experience (DX).
2

The company's core differentiator is elastic inference: auto-scaling from 1,000 to 10,000+ GPUs across 17 cloud providers, with region-aware scaling for unpredictable traffic patterns.
3

Speculative decoding (D Flash) provides 2-4x inference speedups without quality loss, and Modal open-sourced it while upstreaming improvements to frameworks like SGLang.
4

Sandboxes, built in May 2023, became a key primitive for agents—used for production RL rollouts, CI/CD, and background agents like Ramp’s accounting agent.
5

Modal’s IPv6 overlay network (i6pn) enables RDMA-tier inter-node networking for smaller-scale post-training runs, with 3 Tbps internode bandwidth.

💬 Notable Quotes

"The thing we first found product-market fit with was inference for custom models. We stayed away from the LLM space and served companies like Suno for audio, Runway for video, robotics, and comp bio."
"We’ve actually changed our SDK team to think about agent experience instead of developer experience. The same benefits that apply for DX also apply for AX."
"The biggest use case is elastic inference. For custom models, traffic is diurnal and unpredictable—you need to scale up and down in different regions at different times. That’s our sweet spot."