May 13, 2019. Guorui Shen, guorui233@outlook.com
Recurrent Neural Networks (RNNs) is actually a first-order Hidden Markov Model (HMM) which has the form
h_t=f(x_t,h_{t-1};\theta),\\
y_t = g(h_t;\theta),
where
\begin{align}
h_t&=f(W_{hx}x_t+W_{hh}h_{t-1}+b_{h}),\cr
x_{t+1}&= W_{xh}h_t+b_{x}.
\end{align}
RNNs have many variants according to various RNN cells, including Vanilla RNN (or an Elman RNN after Prof. Jeffrey Elman, the most simple kind), Long Short Term Memory (LSTM, 1997), Bidirectional LSTM, Gated Recurrent Units (GRU, 2014), Encoder-Decoder sequence-to-sequence architecture, Reservoir Computing or Echo State Network (RC or ESN) and etc.
RNNs are mainly used in time series analysis, automatic image captioning, sentiment classification, machine traslation, chatbot and etc.
As for how to implement a RNN in real situation, here we have a list of hands-on examples, based on tensorflow.
- Vanilla unit -
tf.contrib.rnn.BasicRNNCell(hidden_dim)
- LSTM unit -
tf.contrib.rnn.BasicLSTMCell(hidden_dim)
,tf.contrib.rnn.LSTMCell(hidden_dim)
, - GRU -
tf.contrib.rnn.GRUCell(hidden_dim)
Date | Codes | Data |
---|---|---|
Jul 21, 2019 | KdV-attention-GRU-seq2seq-single-model.py | twoSolitonKdV.mat |
Jul 5, 2019 | seq2seq-tf-time-series-kaggle-web-traffic-prediction-1st place solution | |
Jun 30, 2019 | Attention-based-Seq2seq-Eng2Jap-Keras.ipynb | eng2jap.csv |
Jun 24, 2019 | English2French-keras-tf-seq2seq-tensorboard.ipynb | fra.txt |
Jun 16, 2019 | seq2seq-vs-GPR-Lorenz-prediction-tensorflow-xyz2xyz.ipynb | Lorenz.mat |
Jun 12, 2019 | wildml-define-rnn-using-numpy | |
Jun 11, 2019 | copytask-seq2seq-on-tensorflow-for-beginner.ipynb | |
Jun 11, 2019 | Intro-to-LSTM-and-Seq2seq.ipynb | |
May 15, 2019 | Predict-KS-via-Reservoir-Computing-all2all.ipynb | KS.mat |
May 15, 2019 | lstm-Lorenz-prediction-xyz2xyz-batch-training-on-tensorflow.ipynb | Lorenz.mat |
May 13, 2019 | rnn-mnist.ipynb | |
May 13, 2019 | define_lstm_unit_from_scratch_tensorflow.ipynb | |
May 13, 2019 | seq2seq-sine-cosine-prediction-tensorflow.ipynb | |
May 13, 2019 | Predict-Lorenz-using-Reservoir-Computing-simple-demo.ipynb | Lorenz.mat |
Websites
- wikipedia-Recurrent_neural_network
- Tensorflow based coding guideline for RNN - easy tensorflow: RNNs
Slides
- Lecture given by Fei-Fei Li & Justin Johnson & Serena Yeung - stanford cs231n slides
Literatures
- Reservoir Computing for chaotic KS prediction. Pathak, Jaideep, et al. "Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach." Physical review letters 120.2 (2018): 024102.
- Echo States Networks for time series prediction. Jaeger, Herbert, and Harald Haas. "Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication." science 304.5667 (2004): 78-80.
- Chapter 10 Sequence Modeling: Recurrent and Recursive Nets. LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." nature 521.7553 (2015): 436.
- NIPS Time Series Workshop 2017. Yu, Rose, et al. "Long-term forecasting using tensor-train rnns." arXiv preprint arXiv:1711.00073 (2017).