reczoo/Critical-Review-Papers

A awesome list of critical review papers that criticize existing work

Open-Critical-Reviews

Recommendation

• [Arxiv'2020] FuxiCTR: An Open Benchmark for Click-Through Rate Prediction, by Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, Xiuqiang He.

  Argument: Many new models for CTR prediction do not perform as expected compared to some classic ones.

• [RecSys'2020] Neural Collaborative Filtering vs. Matrix Factorization Revisited, by Steffen Rendle, Walid Krichene, Li Zhang, John Anderson.

  Argument: MLP in NCF does not outperform inner products in MF as reported.

• [Arxiv'2020] Empirical Analysis of Session-Based Recommendation Algorithms, by Malte Ludewig, Noemi Mauro, Sara Latifi, Dietmar Jannach.

  Argument: The progress in terms of prediction accuracy that is achieved with neural methods is still limited. In most cases, our experiments show that simple heuristic methods based on nearest-neighbors schemes are preferable.

• [Arxiv'2020] A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research, by Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, Dietmar Jannach.
• [CIKM'2020] Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems, by Maurizio Ferrari Dacrema, Federico Parroni, Paolo Cremonesi, Dietmar Jannach.

  Argument: Convolutions over user-item embedding maps do not outperform traditional baselines as reported.

• [SIGIR'2020] How Useful are Reviews for Recommendation? A Critical Review and Potential Improvements, by Noveen Sachdeva, Julian McAuley.

  Argument: Whether reviews are helpful for recommendation is questionable. Some state-of-the-art methods fail to outperform existing baselines.

• [RecSys'2019] Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches, by Maurizio Ferrari Dacrema, Paolo Cremonesi, Dietmar Jannach.

  Argument: Many deep models, including CMN, MCRec, CVAE, CDL, NCF and MVAE, do not perform better than simple baselines such as MostPopular, KNN baselines and SLIM.

NLP

• [AAAI'2020] A Critique of the Smooth Inverse Frequency Sentence Embeddings

• [ACL'2018] Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms

• [ICLR'2017] A Simple but Tough-to-Beat Baseline for Sentence Embeddings

CV

• [Arxiv'2020] A Metric Learning Reality Check

• [ECCV'2020] GDumb: A Simple Approach that Questions Our Progress in Continual Learning