Our goal is to design an architecutre for generating a music sequence gven another sequence correspnding in time. For example, generating a melody track given a drum beat, or generating the righthand of a piano, given the left hand. At training time, the network will be provided with two sequences which match temporally. We will refer to these as the melody and harmony sequences. At test time, only the harmony sequence will be provided, and a melody sequence will be generated by the network.
#####[RNN RBM] (https://github.com/tensorflow/magenta/blob/master/magenta/reviews/rnnrbm.md) This blog post describes how to do polyphonic music generations using RBMs whose input is provided at each step by the RNN. Perhaps, we will incorporate RBMs once we have a basic architecute working. The sample code for the blog post also contains some convenient code for reading MIDI into vectors, which me might borrow.
#####Recurrent Neural Network (RNN) basics and the Long Short Term Memory (LSTM) Learn about RNN & LSTM basics in python
#####The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy Good resource for learning about RNN's
#####Understanding LSTM Networks by colah's blog Suggested by Benedict
#####Wavenet: A generative model for raw audio/Deepmind
Explains idea of dilated multilayer RNN's
- Add MIDI to vector code
- data processing pipeline - ensure propagation melody_input time t+1 to all nodes
- Implement basic 1 layer LSTM network
- design a cost function - use L2 norm to start
- Add RBM/NADE/Deep generative model layer.
- Add multiple (partially?) dilated RNN layers in model
- Recognize and modify styles?
DI: code skeleton
LA Deliverable: Look at Udacity TensorFlow/ DeepLearning course for code resources and understanding how the code is setup. See Deep Learning for Text and Sequences.https://www.udacity.com/course/viewer#!/c-ud730/l-6452084188/m-6560586345
BM: insert pretty pictures of RNN skeleton.