Goal

To train a neural network to segregate good images and bad images.

Programs used

`cd path/to/this/folder'
sh setup.sh. - This is a one time event. This will install python packages needed for functioning and make folder structure.

1 Images (User has to put images)

2 hottiedata (Nothing to be done by user. Auto-generated)

input - Place to store train / test data in CSV / txt format. (Nothing to be done by the user. Automatically generated.)
output - Place to store output data of false positives and false negatives on test data.

`cd path/to/this/folder'
Put your images in the Images folder as shown in the Images folder description. The description is self explanatory.
Run sh preprocess.sh from commandline.
This will generate all the text files / CSV files in hottiedata folder. (Extracted images and put their crux data in CSV / txt).
Step 1 to 3 are one time event. Unless you want to change images, you don't have to do this again.
Open Octave CLI.
write `cd path/to/this/folder' and click enter
Write 'hottie1' and press enter.
Keep on pushing enter whenever asked.
The console will show you the training / testing accuracy and also list of false positive / false negative files (places where the code failed).
Examine the image and try to figure out the cause of error.
run sh postprocess.sh from commandline. This will segregate the good and bad images from Images/ToBeSorted folder into Images/sorted/positives and Images/sorted/negatives folders.
Check the Images/sorted/positives and Images/sorted/negatives for segregated images.

The next sections are for developer notes. Ignore if you are a user.

resize.py does the preprocessing.
Images are usually of different sizes. Therefore, it is necessary that they are converted to uniform data for machine learning.
We resize every image into 10X10 pixel size (i.e. make it very low resolution).
Every pixel has 4 features CYMK which take values from 0-255 (i.e. 256 possible values).
Therefore our input data has 10X10X4 = 400 input features.
These 400 input features are written in a CSV file. One line represents one image file.
All files from the input folders are converted to the corresponding CSV files by this process.
The file names are also stored as txt files, for later retrieval (which line of data belongs to which file).
After preprocessing, we don't use images any more. We do our training etc with the CSV files only.

As mentoned in preprocessing section, our data has values from 0-255, and a total of 400 columns for each image.
If we keep the numbers as they are, it would be difficult to train neural networks. Therefore, we need to normalize data.
Python can normalize, but as the data size is higher, a Linear Algebra program like octave does it faster. So I have chosen octave to do this.
We take average mean for each column and normalize according to the following formula (observation-mean)/256. Note that mean varies for each column.
Thus, each value is converted to (-1,1) range.
If the input images are not changed, the normalized values would also not change. Therefore, we have stored the normalized data in hottiedata/input/normalized/traindata.mat.
This will ensure that the data is loaded directly from this file as long as there is such a file.
This will help us run the octave code for training neural network a bit faster.
If you change the images, please delete the traindata.mat file.
Code is resilient enough and will regenerate a new testdata.mat with the new images you have put in folders.

We train the neural network with the training data.
We display training classification accuracy.
We display test classification accuracy.
If the accuracy on test classification is 80% (or greater than some user defined threshold), the parameters are stored in hottiedata/input/learntparameters.mat.
This way we can store the neural network we found OK, and then apply it to some other test cases too.

Code of Coursera Machine learning course ex4 has been extensively used to create neural network on octave.