MohitDhungana/duplicate_question_detection

BCT 4th year major project

Duplicate Question Detection

Dataset drive link(use "Add to my drive")

• https://drive.google.com/open?id=1ZN9l2s_e8GIfTxhzAQjIsD1sbRQ3F2N0

Trained model file link (85DQD2.h5)

• https://drive.google.com/open?id=1-3A9AvUrS_RHqDVy9SmVAEJPsU-Uoz0d

Vocabulary dictionary json file link (vocabulary_dictionary.json)

• https://drive.google.com/open?id=1sDxz-5MVGtEBJGO_Hsby0SKSnJBP2KOb

Locally trained model (model_for_gui.h5):

• https://drive.google.com/open?id=1aIXvw_-GnqqBk9HiIdQiGO-mNp2aeYzq

RNN and LSTM

• http://karpathy.github.io/2015/05/21/rnn-effectiveness/
• https://peterroelants.github.io/posts/rnn-implementation-part01/
• https://en.wikipedia.org/wiki/Long_short-term_memory
• http://colah.github.io/posts/2015-08-Understanding-LSTMs/?fbclid=IwAR1y1uK2rldpNTWv4VAIDzb45ZkBOJEq8Fg1W_Aau0g8RLNwNLmaBfK-OvU

Other resources:

• https://hub.packtpub.com/use-tensorflow-and-nlp-to-detect-duplicate-quora-questions-tutorial/?fbclid=IwAR01TdBmlHfOR_lUfbL7YvGbPRhlut7SuDLISgq6EI8Xj8msZtfJXKSq5pk

• https://github.com/aerdem4/kaggle-quora-dup?fbclid=IwAR25Re-WszL0CtkiTorsWpOJg7XiAn8gzk_jeAWTJnSHzH-CbgAOZ3GVO2c

• https://medium.com/mlreview/implementing-malstm-on-kaggles-quora-question-pairs-competition-8b31b0b16a07?fbclid=IwAR0SPEFO9qaLACO-xRkBnEcSTySJESr1l1j1Ol8xnX56py2SiuGEw945dro

• http://cs230.stanford.edu/projects_fall_2018/reports/12447287.pdf?fbclid=IwAR3JlOV-ifVFu0-xaal2Ugg1adfWA6Uhf39-mTPg11WW-4AJ4YVR6GethPE

• https://towardsdatascience.com/identifying-duplicate-questions-on-quora-top-12-on-kaggle-4c1cf93f1c30?fbclid=IwAR3fvuN2i0qx9i6ysSME9YFaIh3kxbL2haJxFJJKQofpX8t3E8FP0DqH-EE

• https://www.geeksforgeeks.org/tf-idf-model-for-page-ranking/?fbclid=IwAR3LS3faRwFpu-tGK4DGxvQqrrjsi7_aY4RSVsQn9-aKM-syNQ0kewKCb2w

by Mamata

• https://medium.springboard.com/identifying-duplicate-questions-a-machine-learning-case-study-37117723844 (use transfer learning on word2vec in dataset?)

• https://github.com/minsuk-heo/python_tutorial/blob/master/data_science/nlp/word2vec_tensorflow.ipynb

by Mohit

• https://colab.research.google.com/drive/1CXdKsk9mvHiuFMRowELp0NDP5__bsuVy#scrollTo=wYhaXSvYEcju (sample code .ipynb)
• https://www.researchgate.net/publication/307558687_Siamese_Recurrent_Architectures_for_Learning_Sentence_Similarity (Siamese Recurrent Architectures for Learning Sentence Similarity research paper)
• https://machinelearningmastery.com/keras-functional-api-deep-learning/ keras functional API usage example(THIS IS DOPE)

by Subin

• need to add Project and Project/Pickle in My drive if you want to use SubindQd (note:the name should be in the specified combination of upper and lower case characters) also need to change id
• http://ruder.io/optimizing-gradient-descent/index.html#whichoptimizertochoose
• https://ai.stackexchange.com/questions/3156/how-to-select-number-of-hidden-layers-and-number-of-memory-cells-in-an-lstm
• https://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw
• https://medium.com/data-science-101/detecting-duplicate-questions-on-quora-beating-stanfords-accuracy-6f18b3634d06 (code that has beaten stanfords accuracy and ours till date)