Software Engineering Daily

Foundation Models for Structured Data

45 min

This episode discusses the application of deep learning and foundation models to structured, tabular data, an area that has lagged behind advances in computer vision and NLP. Jure Leskovec explains that by treating databases as graphs and using graph transformer architectures, it is possible to build predictive models that require significantly less manual feature engineering. This approach, called relational deep learning, promises to revolutionize predictive modeling for enterprise data, much like transformers have revolutionized text and images.

Summarized by Podsumo

🎧 Listen 🎙️ Ask about this episode

✨ Key Takeaways

1

Relational deep learning treats databases as graphs and applies attention mechanisms directly over structured relational data, eliminating the need for extensive manual feature engineering.
2

Jure Leskovec argues that tabular data is a distinct modality requiring its own specialized neural network architectures, similar to how images need CNNs and text needs transformers.
3

Foundation models for tabular data, like Kumo RFM, are pre-trained and can be used for predictive tasks (churn, fraud, etc.) without task-specific training, analogous to using ChatGPT for text.
4

The approach leverages graph transformers, which are more flexible and generalize better than traditional graph neural networks for large-scale, multi-table enterprise databases.
5

Jure emphasizes that even a 1-2% improvement in predictive accuracy can have massive business impact, and relational deep learning can achieve superhuman performance by learning directly from raw data.

💬 Notable Quotes

"The point is that every percentage improvement in accuracy of these models means humongous business impact, right? Even one, two percent improvement can have humongous business impact."
"So the approach we invented... is this notion of relational deep learning, where we basically take the transformer architecture and generalize it so that it can attend over this structured relational enterprise data."
"Actually, the structured data world that has humongous business impact is ready for the AI revolution to actually take place there as well."