tdiggelm/nn-experiments

README

• General

  • translational inveriance: https://stats.stackexchange.com/questions/208936/what-is-translation-invariance-in-computer-vision-and-convolutional-neural-netwo
  • deep learning book: http://www.deeplearningbook.org/contents/convnets.html
  • classification: best practice: https://machinelearningmastery.com/best-practices-document-classification-deep-learning/

• RNN seq-2-seq learning

  • https://www.analyticsvidhya.com/blog/2018/03/essentials-of-deep-learning-sequence-to-sequence-modelling-with-attention-part-i/

• Instead of RNN, implement temporal convolution:

  • The fall of rnn: https://towardsdatascience.com/the-fall-of-rnn-lstm-2d1594c74ce0
  • https://github.com/philipperemy/keras-tcn

• Attention mechanism: https://github.com/philipperemy/keras-attention-mechanism

• Papers (to implement):

  • 2017, VDCNN Very Deep Convolutional Networks for Text Classification: https://arxiv.org/pdf/1606.01781.pdf (implementation at https://github.com/zonetrooper32/VDCNN)
  • 2018, Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction: https://arxiv.org/pdf/1808.03867.pdf
  • 2016, HAN Hierarchical Attention Networks for Document Classification, http://www.aclweb.org/anthology/N16-1174 (blog: https://medium.com/analytics-vidhya/hierarchical-attention-networks-d220318cf87e, implementation: https://github.com/richliao/textClassifier)