deeplaying/codes_fastai

My codes for http://course.fast.ai/lessons/lessons.html

fast ai

Week1 - Introduction

• Course page
• Course notes
• Notebook for lesson 1
• Lesson 1 Overview
• Lesson 1: Practical Deep Learning for Coders

Useful commands:

wget http://www.platform.ai/files/nbs/lesson1.ipynb
wget http://www.platform.ai/files/nbs/utils.zip
wget http://www.platform.ai/files/nbs/vgg16.zip
wget http://www.platform.ai/data/dogscats.zip

unzip utils.zip
unzip vgg16.zip
unzip dogscats.zip

rm dogscats.zip
rm utils.zip
rm vgg16.zip

Errors for week1

Do not use the wget for utils.zip and vgg16.zip.

Instead, please download from github repo and also include vgg16bn.py.

Week2

• Course page

• Discussion page

• Lesson 2: Practical Deep Learning for Coders

• Lesson 0 - recommended to watch after lesson 2

• Notebooks

  • convolution-intro.ipynb - The convolution tutorial notebook used in the introductory lesson presented during the Data Institute launch
  • lesson2.ipynb - the main notebook for lesson 2
  • redux.ipynb - how to enter the Dogs vs Cats Redux competition, and how to visualize your models correct and incorrect predictions
  • sgd-intro.ipynb - the simple SGD tutorial

• Excel files used

• Stanford Cs231 reading

  • Optimization: Stochastic Gradient Descent
  • Backpropagation, Intuitions
  • Neural Networks Part 1: Setting up the Architecture

• Introduction to Deep Learning by Michael Nielsen

  • Chapter 1 - Using neural nets to recognize handwritten digits
  • Chapter 2 - How the backpropagation algorithm works
  • Chapter 3 - Improving the way neural networks learn

Week2 Extras

Suggested Readings from notebooks

• Chapter 4 - A visual proof that neural nets can compute any function

Errors

Error one

val_data = get_data(val_batches)
trn_data = get_data(batches)

should be

val_data = get_data(path + 'valid')
trn_data = get_data(path + 'train')

More over, using get_data would cause your memory to be inefficient (on p2.x on aws), it is suggested on the fourms to use batch generator instead.

I overcame the problem by running the code as it is, save using bcolz. Afterwards i restart the notebook, and only ran the load command without get_data.

As an aside, you can use sys.getsizeof() to understand the memory usage of each python object.

This link is also a good read to understand how to monitor your instance memory stance.

Error two

def fit_model(model, batches, val_batches, nb_epoch=1):
    model.fit_generator(batches, samples_per_epoch=batches.N, nb_epoch=nb_epoch, 
                        validation_data=val_batches, nb_val_samples=val_batches.N)

model.evaluate_generator(get_batches('valid', gen, False, batch_size*2), val_batches.N)

The batches.N and val_batches.N should be batches.n and val_batches.n instead based on the util functions.

Week 3

• Course page
  • Note that there are also articles on Matrix Product, Convolutions (and max pooling), Activations, SGD, Backprop.
• Lesson 3 video
• Lesson 3 notebook
• convolution-intro.ipynb - The convolution tutorial notebook used in the introductory lesson presented during the Data Institute launch. This is covered in week2 as well.
• Mnist walkthrough notebook
• Suggested Extra Readings
  • Stanford CNN
  • Chapter 6 of Michael Nielsen's Book
• To do:
  • Understand Batch-normalization
  • Understand difference between Correlation and Convolution.

Errors

• width_zoom_range does not exists in image.ImageDataGenerator

 gen = image.ImageDataGenerator(rotation_range=10,
 width_shift_range=0.1,height_shift_range=0.1,
  width_zoom_range=0.2, shear_range=0.15, zoom_range=0.1, 
  channel_shift_range=10., horizontal_flip=True, dim_ordering='tf')

• Weights in ('/data/jhoward/ILSVRC2012_img/bn_do3_1.h5') is not provided.
• Tried using fc_model weights, but the val_loss explodes upwards.
• Tried using vggbn16 weights, similar case.

Further notes for reference:

• adjusting learning rates - most promising, but have to wait for week4 lessons
• Enquiry about bn_do3_1.h5
• Proposed solution - but does not work for me

Week4

• Course page
• Lesson 4: COLLABORATIVE FILTERING, EMBEDDINGS, AND MORE
• Lesson 4 notebook

Download data

wget http://files.grouplens.org/datasets/movielens/ml-latest-small.zip
unzip ml-latest-small.zip
mv ml-latest-small.zip/ ml-small/ 

Week5

• Course page
• Lesson 5 - INTRO TO NLP AND RNNS
• Lesson 5 notebook
• Lesson 5 sub notebook - glove

Download data / model

other models are available here.

wget http://www.platform.ai/models/glove/6B.50d.tgz
tar -zxf 6B.50d.tgz

Errors

1. If you have the error 'The following error happened while compiling the node'

   refer to here

   basically, go to your keras packages, and edit theano_backend.py

   def round(x):
       return T.round(x, mode='half_to_even')
   

   to

   def round(x):
       return T.round(x, mode='half_away_from_zero')
   
   

2. The seed is very important in the training of neural nets, you might not get the same results as the author.

   • Additional notes here.

Week 6

• Course page
• Lesson 6 - BUILDING RNN
• Lesson 6 - Notebook

Misc

Kaggle cli