Code to reproduce the ICLR 2024 Paper : "Implicit regularization of deep residual networks towards neural ODEs"
This package has been developed and tested with python3.11
. It is therefore not guaranteed to work with earlier versions of python.
This package can easily be installed using pip
, with the following command:
pip install -e .
This will install the package and all its dependencies, listed in requirements.txt
. To test that the installation has been successful, you can install pytest
and run the test suite using
pip install pytest
pytest
We train the model as described in the paragraph Large-depth limit, with reduced number of training epochs. Feel free to adjust it.
To reproduce the left-hand figure:
python finite_training_time_lipschitz.py
The plot is saved in the folder figures.
To reproduce the right-hand figure:
python finite_training_time_convergence.py
The plot is saved in the folder figures.
We train the model as described in the paragraph Long-time limit, with reduced number of training epochs. Feel free to adjust it.
To reproduce the figure:
python infinite_training_time.py
You can directly visualize the results using the notebsook learned_weights.ipynb using the pretrained models available in the folder checkpoints.
Alternatively, you can train models from scratch using the following.
python one_expe_cifar.py --lr 4e-2 --depth 256 --seed 1 --smooth_init SMOOTH_INIT --non_lin NON_LIN
Where SMOOTH_INIT is in [True, False]. When True, the weights are initialized smoothly. NON_LIN is the non-linearity used and can be relu, gelu, or linear.
If you use this code in your research, please consider citing:
@inproceedings{marion2024implicit, title={Implicit regularization of deep residual networks towards neural ODEs}, author={Marion, Pierre and Wu, Yu-Han and Sander, Michael E and Biau, G{'e}rard}, booktitle={International Conference on Learning Representations}, year={2024} }