Refactor the code to use Pytorch Lightning & Hydra
MohammedAljahdali opened this issue · 5 comments
Hey, I am currently working on a NN pruning project. In short I would like to see if you're open to the idea of using Libraries such as Pytorch Lightning and Hydra. Hydra would make it very easy to setup configuration for experiments in a clean way. Pytorch Lightning, is a good refactor that will clean the code, but also would provide support to multi-gpu training, early stopping, checkpointing, the use of multiple supported loggers, and more.
Addressing each proposal separately:
-
Hydra: I'm not really a fan of Hydra, I tried it a year or so ago and while great for combining configs, I found it to be quite cumbersome and lacking when working on interactive environment such as jupyter notebooks. I do like structured configs and use them on my current workflow, but I'd like something leaner than Hydra. I think something like OmegaConf, which is a dependency of Hydra, would be a better fit whilst proving pretty much the same advantages, without removing flexibility from the code.
-
PyTorch Lightning - I'd be happy to have a pytorch lightning version of the pipeline. I'm under the impression that this would be orthogonal to the pruning utils and that it would replace the experiment code. Feel free to submit a PR implementing an alternative experiment (ie. lightning module) pipeline to the current one. I say alternative since I'd like to keep the old code around for reproducibility of the original paper.
To be honest I really like hydra, I think it is difficult to do work in an interactive environment, when your code base uses hydra.Although, I didn't have to use code that I built with it in jupyter notebooks much, so I can not say much
Yeah adding PyTorch Lightning (PL) would orthogonal to the pruning utils. And, it would replace the experiment, data handling, and some of the boiler plates such as metrics, logging, early stopping and checkpointing would come for free, when using PL. Also refactoring the data part would solve this #22 issue.
I have already started working on moving your current code on top of template repo that utilizes PL and hydra, if would like we could arrange something so that you could oversee the development of this new pipeline let say. Regarding hydra I could provide an alternative down the line, but it is easier for me to use since I am used to handling configs with it.
I am currently using a fork of this repo to do the work, I don't have experience with using git to contribute to other projects. So if there is a better way please let me know!
Hey @JJGO,
I would like to ask you can the pruning utlis be used for iterative pruning techniques, specifically can it prune only from parameters that are not pruned?
Moreover, why not adopt the available PyTorch pruning? Are there any reasons that make them not appropriate for benchmarking?
They can be used with iterative pruning techniques if memory serves right. You just have to schedule the amount of pruning correctly.
On the second topic, the project predates torch pruning utils and serves a slightly different purpose. This might have changed from the last time I looked at it, but torch pruning utils seemed more focused on providing an API to prune networks using existing strategies and not as much in providing an interface to develop and benchmark novel pruning strategies which was the aim of shrinkbench.
Closing due to inactivity