/HDF5-dataset-implimentation-for-deep-learning-training

Primary LanguageJupyter Notebook

HDF5-dataset-implimentation-for-deep-learning-training

Motivation:

i am always facinated by the idea of creating an end to end Deep learning framework such as Tensorflow, Pytorch, caffe and so on. Among various components of a DL framework, the data loading part plays an important role when it comes to training a model. Most of the training process involves repeated modifications to the model, hyper parameter which ends up retraining the model over and over again. The data feeding method will affect the training time overall making it one of the crutial part to keep in mind regardless of what DL architecture is trained. This motivated me towards making an tutorial on data saving and loading formats that support fast read and write from the disk.

Intended topics to be covered:

  1. What to know about HDF5 file format and its advantages.
  2. Converting a dataset into HDF5 file format and store it.
  3. Read and using the HDF5 data and training a model.
What is HDF5 format?

TODO : finish README TODO : tensorflow basedataiterator to boost the speed of loading