This episode of Latent Space features Philip Kiely and Ali Taha from Baseten discussing the intricacies of inference engineering for large language models. They cover the entire pipeline from routing a long query to optimizing model serving through techniques like quantization, speculative decoding, and disaggregated prefill/decode, emphasizing the shift towards training-aware inference and the growing importance of infrastructure-level solutions.
Summarized by Podsumo
Baseten's inference pipeline for a long query (e.g., 200,000 tokens) first checks for cached inputs to skip prefill, then uses disaggregated prefill and decode GPUs, and may leverage a traffic-specific speculative decoder to boost speed.
Quantization (e.g., to NVFP4) is a key performance lever, but it is lossy; Ali describes a method where carefully chosen layers are quantized so that errors cancel out, achieving 20% more quantization than competitors with better fidelity measured by KL divergence.
Achieving 10x speed improvements over a baseline is possible by stacking optimizations: quantization (~2x), speculative decoding (~2x), disaggregation (~2x), and better runtimes/kernels—moving from ~30-40 tokens/s to 300-400 tokens/s on top hardware.
Training and inference are increasingly converging: inference engineers now train model-specific speculators, perform post-training quantization recovery, and even use models like GLM-5.2 to write GPU kernels that optimize their own inference.
Ali predicts future GPUs (like NVIDIA Rubin) will become more ASIC-like with specialized instructions, shifting inference engineering from kernel-level hacking to an infrastructure problem focused on KV cache routing, offloading, and high-speed interconnects.
"If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that 100% fidelity of the model."
"The challenge with tool calling more and more seems to be that the companies want certain tool calling, which is a very sensitive thing to train. And because you're dealing with all of the JSON outputs, if it doesn't close the end of the request in a very certain manner, you end up with a model that did the tool calling and like the thinking."
"If you look at highly optimized domains, like say finance, you measure how much better you got in basis points... When we publish optimizations, it's 20%, it's a hundred percent, it's two hundred percent. So there's still probably like a lot further to go, honestly."