I've been using Kaggle as an excuse to learn techniques in machine learning/artificial intelligence.
Here are some primary resources I've been learning from (in rough chronological order). For reference, I started from an extensive programming background, a decent but rusty math background, and a rudimentary background in machine learning.
-
http://karpathy.github.io/2015/05/21/rnn-effectiveness/: It was fun to play with the released code, even though I didn't yet know what many of the parameters meant.
-
http://www.pyimagesearch.com/2014/09/22/getting-started-deep-learning-python/: Didn't worry too much about the details of Deep Belief Networks, since I've been told those aren't behind the most recent advances in deep learning. However, I found a lot of value in actually getting a not-completely-black-boxed neural network up and running.
-
http://neuralnetworksanddeeplearning.com/: It wasn't until I worked through this book that I really felt like I understood what was going on (or at least knew enough to have a sense of what I didn't know). I highly recommend going through the entire thing. I was particularly impressed by the author's ability to anticipate my confusions and objections, and also convey intuitions and motivations.
-
https://class.coursera.org/ml-005/lecture: Andrew Ng's Machine Learning course. Contains helpful details on a number of topics I hadn't seen before. However, the course moves slower than I was hoping, but with the right cherry-picking it felt pretty useful.
-
http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf: Useful for learning some practical tricks for getting better performance out of your neural network. The first half is very useful and readable (though I didn't work through all of the math). The second half seems less so: as they conclude the paper, "Classical second-order methods are impractical in almost all useful cases".
-
http://deeplearning.net/tutorial/lenet.html: Decent introduction to convolutional neural networks. I wasn't previously familiar with convolutions and didn't fully understand it until I'd read http://www.songho.ca/dsp/convolution/convolution.html.
-
http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf: Contains thorough background on recurrent neural networks, with many experiments and tricks for training your own.
-
http://andrew.gibiansky.com/blog/machine-learning/conjugate-gradient/: This whole blog is great. It has good exposition on some of the more mathematically-involved techniques.
-
http://arxiv.org/pdf/1211.5063v2.pdf: A better choice of activation function and initialization scheme.
-
http://deepdish.io/2015/02/24/network-initialization/: More recent overview of initialization techniques.
-
http://yyue.blogspot.com/2015/01/a-brief-overview-of-deep-learning.html: Great overview of how to think about deep neural networks, and how to train them in practice.
-
https://christopherolah.wordpress.com/: Many amazing blog posts which explore deep concepts in accessible ways.