Decide what to do with train.py

[mdraw]

The file scripts/train.py is currently used as a provisional entry point for quickly testing changes in elektronn3. It supports some configuration via CLI arguments, but there are still many limiting hard-coded values in it, so editing its source code is often necessary. What do we do with it? The following alternatives came to my mind:

1. Keep it as is, with its small set of options, and declare it as a "template for custom training scripts".
2. Reduce its options to a bare minimum again and declare it as a minimum training script template. The standard usage scenario of elektronn3 would then be to make a copy of train.py and customize it for your own needs, always running your trainings from your own version of train.py. We could place multiple alternative versions of it in an examples directory, making them take the place of ELEKTRONN2's example configs.
3. Go the ELEKTRONN{,2} way and create an "official" elektronn3-train entry-point that abstracts common training initialization and where all configurable stuff is placed in training config" .py files. (I don't think this can be implemented in a sensible way.)
4. Just keep adding more CLI options.

IMO alternative 3 and 4 would lead to too much bloat and overengineering eventually, because there are just too many possible configurations that need to be supported. I would go for 2.
Any other suggestions or opinions?

[pschubert]

I also favor option 2 because it provides an easy to understand example while maintaining a high degree of customization.
We should keep in mind to support that all settings/parameters of a training are stored within the training output folder.

[mdraw]

I made a new issue #11 for your second point, because I think it's a very important feature.