A logical, reasonably standardized, but flexible project structure for doing and sharing data science work.
- Python 2.7 or 3.5+
- Cookiecutter Python package >= 1.4.0: This can be installed with pip by or conda depending on how you manage your Python packages:
$ pip install cookiecutteror
$ conda config --add channels conda-forge
$ conda install cookiecuttercookiecutter -c v1 https://github.com/drivendata/cookiecutter-data-science
Cookiecutter data science is moving to v2 soon, which will entail using
the command ccds ... rather than cookiecutter .... The cookiecutter command
will continue to work, and this version of the template will still be available.
To use the legacy template, you will need to explicitly use -c v1 to select it.
Please update any scripts/automation you have to append the -c v1 option (as above),
which is available now.
The directory structure of your new project looks like this:
├── LICENSE
├── AUTHORS.md <- Authors of this project and their affiliations.
├── Makefile <- Makefile with commands like `make data` or `make train`
├── README.md <- The top-level README for developers using this project.
├── data
│ ├── external <- Data from third party sources.
│ ├── interim <- Intermediate data that has been transformed.
│ ├── processed <- The final, canonical data sets for modeling.
│ └── raw <- The original, immutable data dump.
│
├── docker <- All the relevant files regarding the creation of a docker container (e.g. Dockerfile)
│
├── docs <- A default Sphinx project; see sphinx-doc.org for details
│
├── misc <- Miscellaneous files like any bash scripts, e.g. to run on some cluster
│
├── models <- Trained and serialized models, model predictions, or model summaries
│ ├── selection <- Selected models from all session runs
│ └── sessions <- All session runs where each run contains the metrics and info for this run
│
├── notebooks <- Jupyter notebooks. Naming convention is a number (for ordering),
│ │ the creator's initials, and a short `-` delimited description, e.g.
│ │ `1.0-jqp-initial-data-exploration`.
│ ├── experimental <- Any experimental notebook to explore the data characteristics/attributes
│ └── explanatory <- Any explanatory notebook to explain why specific decisions (e.g. regarding architecture,
│ preprocessing) were made
│
├── references <- Data dictionaries, manuals, and other explanatory materials.
│
├── reports <- Generated analysis as HTML, PDF, LaTeX, etc.
│ ├── figures <- Generated graphics and figures to be used in reporting
│ └── manuscripts <- Any manuscript that was created within the project/repository
│
├── requirements.txt <- The requirements file for reproducing the analysis environment, e.g.
│ generated with `pip freeze > requirements.txt`
│
├── setup.py <- makes project pip installable (pip install -e .) so src can be imported
├── src <- Source code for use in this project.
│ ├── __init__.py <- Makes src a Python module
│ │
│ ├── data <- Scripts to download or generate data
│ │ └── make_dataset.py
│ │
│ ├── features <- Scripts to turn raw data into features for modeling
│ │ └── build_features.py
│ │
│ ├── optimization <- Scripts to execute a grid search for hyperparameter optimization
│ │ ├── gridpoint.py
│ │ ├── gridspace.yml
│ │ └── optimize.py
│ │
│ ├── models <- Scripts to train models and then use trained models to make
│ │ │ predictions
│ │ ├── predict_model.py
│ │ └── train_model.py
│ │
│ └── visualization <- Scripts to create exploratory and results oriented visualizations
│ └── visualize.py
│
└── tox.ini <- tox file with settings for running tox; see tox.readthedocs.io