This episode explores the evolution of recommender systems, moving beyond simple accuracy metrics like RMSE to more complex goals such as user engagement, fairness, and diversity. The discussion highlights how optimizing for engagement can lead to issues like filter bubbles and popularity bias, and examines new approaches like embeddings, large language models, and hybrid systems to create more equitable and effective recommendations.
Summarized by Podsumo
Recommender systems are increasingly optimized for engagement (clicks, watch time) rather than accuracy, leading to phenomena like doom scrolling and filter bubbles.
Popularity bias is amplified by algorithms, making popular content even more dominant while niche content becomes harder to discover.
Fairness concerns involve multiple stakeholders, including content providers (artists, creators) whose livelihoods depend on algorithmic exposure.
The shift from matrix factorization to embeddings and large language models is transforming how systems represent and recommend content, but brings new challenges like hallucinations and lack of interpretability.
Hybrid approaches combining language models with traditional retrieval systems may offer a balanced path forward, allowing user steering while maintaining reliability.
"The ratings provided by Netflix represented how someone felt about the movie after watching it. This is not the same question as what movie would you most like to watch next. — Host"
"For the recommender system, this is not really different from people that are liking a new song because they are going there, commenting, and interacting and spending time there. — Grigor Dunneber"
"If there is a bad embedding, it harms both unpopular and popular artists. There's an incentive for all people involved to have good embeddings. — David Liu"