The Pragmatic Engineer

Designing Data-intensive Applications with Martin Kleppmann

1h 25min

Martin Kleppmann discusses the second edition of his seminal book, "Designing Data-intensive Applications," detailing how his work on Kafka at LinkedIn shaped the first edition and the new focus on cloud-native architectures and AI-related data systems in the updated version. He also shares insights from his career transition from industry to academia, emphasizing the importance of understanding system fundamentals, ethical engineering, and tackling long-term, non-commercial challenges like local-first software and formal verification.

Summarized by Podsumo

🎧 Listen 🎙️ Ask about this episode

✨ Key Takeaways

1

DDIA's Evolution: The second edition updates the book to reflect cloud-native architectures (e.g., building databases on object stores) and AI-related data systems (e.g., vector indexes, data frames), while removing outdated topics like MapReduce.
2

Industry to Academia: Kleppmann's career path from successful startups (Reportive acquired by LinkedIn) to working on Kafka, then transitioning to academia, highlights the value of long-term research and non-commercial problem-solving (e.g., local-first software, decentralized access control).
3

Understanding System Internals: Even with the rise of managed cloud services, engineers benefit from knowing how data systems work internally to make informed trade-offs (cost vs. performance vs. resilience) and effectively debug issues.
4

Formal Verification & AI: Kleppmann believes formal verification will become increasingly crucial, especially with AI-generated code, to ensure correctness and security in high-stakes systems, despite its current complexity.
5

Engineer's Ethical Responsibility: The book's final chapter emphasizes engineers' responsibility to consider the societal impact of their technologies and make intentional decisions about the kind of world they are building.

💬 Notable Quotes

"The moral of this chapter is really that actually know if you want to make things reliable You really do have to worry about a whole bunch of weird unusual, but but certainly possible edge cases."
"If you want to change the world, then thinking about the impact that your technologies have on the world is part of your job."
"Sometimes in order to learn something you just have to struggle with it a bit."