Description
Towards a NAS Benchmark for Classification in Earth Observation
Quickstart
Create the pipeline environment and install the tum_dlr_automl_for_eo package
Before using the template, one needs to install the project as a package.
- First, create a virtual environment.
You can either do it with conda (preferred) or venv.
- Then, activate the environment
- Install the Naslib with the command below:
pip install -e git+https://github.com/emreds/NASLib.git#egg=naslib
- Then cd into the project's folder:
cd tum-dlr-automl-for-eo
- Finally, install the rest of the dependencies Run:
pip install -e .
Run the MNIST example
This pipeline comes with a toy example (MNIST dataset with a simple feedforward neural network). To run the training (resp. testing) pipeline, simply run:
python scripts/train.py
# or python scripts/test.py
Or, if you want to submit the training job to a submit (resp. interactive) cluster node via slurm, run:
sbatch job_submission.sbatch
# or sbatch job_submission_interactive.sbatch
- The experiments, evaluations, etc., are stored under the
logs
directory.- The default experiments tracking system is mlflow. The
mlruns
directory is contained inlogs
. To view a user friendly view of the experiments, run:# make sure you are inside logs (where mlruns is located) mlflow ui --host 0000
- When evaluating (running
test.py
), make sure you give the correct checkpoint path inconfigs/test.yaml
Project Organization
├── configs <- Hydra configuration files
│ ├── callbacks <- Callbacks configs
│ ├── datamodule <- Datamodule configs
│ ├── debug <- Debugging configs
│ ├── experiment <- Experiment configs
│ ├── hparams_search <- Hyperparameter search configs
│ ├── local <- Local configs
│ ├── log_dir <- Logging directory configs
│ ├── logger <- Logger configs
│ ├── model <- Model configs
│ ├── trainer <- Trainer configs
│ │
│ ├── test.yaml <- Main config for testing
│ └── train.yaml <- Main config for training
│
├── data <- Project data
│ └── MNIST
│ ├── processed <- Processed data
│ └── raw <- Raw data
│
├── docs <- Directory for Sphinx documentation in rst or md.
├── models <- Trained and serialized models, model predictions
├── notebooks <- Jupyter notebooks.
├── reports <- Generated analysis as HTML, PDF, LaTeX, etc.
│ └── figures <- Generated plots and figures for reports.
├── scripts <- Scripts used in project
│ ├── job_submission.sbatch <- Submit training job to slurm
│ ├── job_submission_interactive.sbatch <- Submit training job to slurm (interactive node)
│ ├── test.py <- Run testing
│ └── train.py <- Run training
│
├── src/tum_dlr_automl_for_eo <- Source code
│ ├── datamodules <- Lightning datamodules
│ ├── models <- Lightning models
│ ├── utils <- Utility scripts
│ │
│ ├── testing_pipeline.py <- Model evaluation workflow
│ └── training_pipeline.py <- Model training workflow
│
├── tests <- Tests of any kind
│ ├── helpers <- A couple of testing utilities
│ ├── shell <- Shell/command based tests
│ └── unit <- Unit tests
│
├── .gitignore <- List of files/folders ignored by git
├── .pre-commit-config.yaml <- Configuration of pre-commit hooks for code formatting
├── requirements.txt <- File for installing python dependencies
├── setup.cfg <- Configuration of linters and pytest
├── LICENSE.txt <- License as chosen on the command-line.
├── pyproject.toml <- Build configuration. Don't change! Use `pip install -e .`
│ to install for development or to build `tox -e build`.
├── setup.cfg <- Declarative configuration of your project.
├── setup.py <- [DEPRECATED] Use `python setup.py develop` to install for
│ development or `python setup.py bdist_wheel` to build.
└── README.md