This repo contains a series of algorithm, data structure, and system design exercises.
To run this code, you'll need Make, Java 7, Python 3.5, and numpy installed installed.
All code is executable from a single command. Simply type:
make all_solutions
This will make a new data directory, and download the required dataset. All solutions are outputted to solutions.
This project uses GNU Make to orchestrate a data pipeline, which is defined in Makefile.
If you're new to Make, don't worry! It's simple. A Makefile consists of a series of tasks that take input files and product output files. For example:
clean_data/product.csv: raw_data/product.csv clean_dataset.py
cat raw_data/product.csv | python clean_dataset.py clean_data/product.pklraw_data/product.csv, clean_dataset.py, and clean_data are the input files.
clean_data/product.csv is the output file.
The shell command starting with cat is the task - it takes the first input file, raw_data/product.csv,
and passes it to the python module clean_dataset.py to produce our output file, clean_data/product.pkl.
To execute the task, you type:
make clean_data/product.csv
If the file doesn't already exist, Make will... make it for you!
(The reason that clean_dataset.py was passed in as an input file to the task, even though it doesn't contain
any data for the task to process, has to due with the way Make works. If you try to execute the task, but
clean_data/product.csv already exists, Make will do nothing - unless the last updated timestamp of any of
the input files is newer than the output file! Perhaps the input data changed, or the source code that converts
the input data to the output. In either case, we want Make to re-execute the task, and create a new output for us.)
If a task depends on an input file, and this input file doesn't exist on disk yet, but it is defined as the output file of another task, Make will execute that task first. By specifying dependencies in this way, you can execute an entire workflow simply by executing the final task.
That's pretty much it!