A collection of production-ready recommendation system papers
Recommendation system is a pivotal technology for contemporary internet business. It helps on-line business to algorithmically identifies items that potentially draw interests of users, and thus boost stickiness of users to the site or service.
Academic research in this realm has been active for ages. Many innovative ideas are witnessed during the past a few years which incorporate the cutting-edge techniques (e.g., deep learning, etc.) into a recommendation system. However, many of such proposals stopped at prototyping stage owing to issues in many different aspects.
A production-ready recommendation system is not just algorithm implementations. It should be a systematic integral of models and relevant peripheral components that servers to performing recommendation tasks in a pragmatic and efficient way. It should be easily integrated into an enterprise-grade production pipeline such that collectively the system resolves a real-world business problem under architectural and engineering constraints such as scalability, latency, etc.
As follows criteria of a production-ready recommendation system are listed.
A clear definition of real-world scenarios guide developers and architects to understand business problems well and translate them into modeling problems.
A good recommender excels in modeling performance. Performance can be measured by offline metrics like root mean squared error (RMSE), precision and recall at top-k recommended items, etc., and/or online business metrics as click-through rate (CTR), impression rate, etc.
It is worth noting that there is usually no correlation between offline evaluation performance and online ones. Apparent reason for this is that usually an offline evaluation is based on a sample dataset which is biased. A production-ready requires online testing framework to evaluate recommender performance in actual application.
It is not rare to see a recommendation system is deployed to serve millions or even billions of users. To this end, the recommendation system must be scalable such that an increasing demand of user size and item size is still within its capacity.
Latency is another critical criterion for the modern recommendation system, especially for particular on-line services that require immediate responsive feedbacks to engage service subscribers.
On top of the model creation pipeline, a whole architecture that stream model inputs/outputs is needed.
Online testing is essential to guarantee a recommender works as expected when it is deployed for actual use.
A successful deployment of a recommendation in industry-grade solution or product proves the effectiveness and efficiency.
The list contains research work in production-ready recommendation systems in the past 10 years. The list is not exhaustive. Resources in the list are either from publicly available research articles or on-line resources.
| Things to recommend | Author | Year | Text resources | Codes | Remarks |
|---|---|---|---|---|---|
| Music | Spotify | 2018 | Link | - | - |
| Retail products | Overstock | 2018 | Link | - | - |
| Online items | 2018 | Link | - | - | |
| Retail products | JD.com | 2017 | Link | - | - |
| Social network user interaction | 2017 | Link | - | - | |
| Retail products | Sears | 2017 | Link | - | - |
| Online items | 2017 | Link | - | - | |
| Fashion products | Zalando | 2017 | Link | - | - |
| Product size | Amazon | 2017 | Link | - | - |
| Ads | Microsoft | 2017 | Link | - | - |
| Music | Clova | 2017 | Link | - | - |
| News article | Yahoo! Japan | 2017 | Link | - | - |
| Display Ads | Alibaba | 2017 | Link | - | - |
| Mobile App | 2016 | Link | - | - | |
| Meals | CornelTech | 2016 | Link | - | - |
| Videos | 2016 | Link | - | - | |
| Movie | Netflix | 2015 | Link | - | - |
| Social media | Yahoo! | 2011 | Link | - | - |
| News article | Yahoo! | 2010 | Link | - | - |
| Ads | Microsoft | 2010 | Link | - | - |
| Movie | Microsoft | 2009 | Link | - | - |
| News article | 2007 | Link | - | - | |
| News article | 2007 | Link | - | - |