Latent Space: The AI Engineer Podcast

Notion’s Token Town: 5 Rebuilds, 100+ Tools, MCP vs CLIs and the Software Factory Future — Simon Last & Sarah Sachs of Notion

1h 17min

Notion's Simon Last and Sarah Sachs discuss their journey building AI agents, highlighting five major rebuilds and the evolution to over 100 tools. They delve into the trade-offs between MCPs and CLIs, Notion's vision for a software factory where agents build and maintain software, and their unique low-ego, prototype-driven culture. The conversation also covers their sophisticated evaluation systems and a strategic approach to pricing and model selection to ensure value for customers.

Summarized by Podsumo

🎧 Listen 🎙️ Ask about this episode

✨ Key Takeaways

1

Notion's agent system underwent five major rebuilds, learning to cater to model capabilities (e.g., Markdown, SQLite) rather than internal data models, moving from few-shot to goal-driven prompting for scalability.
2

Simon Last is bullish on CLIs for their bootstrapping and debugging power in terminal environments, while Sarah Sachs emphasizes MCPs for narrow, lightweight, and tightly permissioned agents, and Notion's commitment to supporting them for cost-effective, deterministic tasks.
3

Notion envisions a future where coding agents are the 'kernel of AGI,' capable of bootstrapping, debugging, and maintaining their own software within an automated 'software factory' workflow, minimizing human intervention.
4

They employ a multi-layered evaluation system, including 'frontier evals' with a 30% pass rate to push model boundaries, and have established a new career path for 'Model Behavior Engineers' (MBEs) to build agent-driven evaluation harnesses.
5

Notion uses usage-based pricing for agentic features to manage costs and focuses on providing the 'right tool for the job,' actively investing in open-source models to fill market gaps in the intelligence-price-latency triangle.

💬 Notable Quotes

"Broadly speaking, I'm really bullish on CLI's. I'm still bullish on MCPs in a certain environment. I think it's really great for when you want a narrow lightweight agent."

— Simon Last
"I think the coding agents are the kernel of the GI. Everything is a coding agent."

— Simon Last
"Our job is to not make the best harness for agent work. Our job is to be the best place where people collaborate."

— Sarah Sachs