In the recent past a certain property of neural training trajectories in weight-space had been isolated, that of "local elasticity" (
Link to our paper can be found here
Model | Result | Hypothesis That it Corroborates | Example of Exact |
---|---|---|---|
Classification by features | Theorem 3.2 | Hypothesis 1.2 (b) | Theorem 3.2 |
A single ReLU gate | Lemma 4.3 | Hypothesis 1.2 (b) | (implicitly contained in the discussion) |
Weight d-homogeneous, feature linear predictors |
Theorem 5.1 Theorem 5.3 |
Hypothesis 1.1 (b) Hypothesis 1.2 (b) |
Theorem 5.2 |
We trained a RESNET-18 on SVHN, CIFAR-10 and CIFAR-100 to track the dynamics of
The images at the left are for our definition of
-
Assets: Contains all pickle files for the python lists used to store all the accuracy metrics, loss and
$S_{\rm rel}$ throughout training. Users would only need to load the appropriate pickle files using the following code to reproduce the plots without training:
with open(path+'pair_iters_cifar_100_kl.pkl','rb') as f:
pair_iters=pickle.load(f)
where path
would be a string defining the directory to access the Assets
folder
-
Plots: Contains all the relevant plots that have been used in the paper. The directory contains subdirectories corresponding to each dataset which contain further subdirectories corresponding to the two definitions of
$S_{\rm rel}$ . -
Scripts: Contains all the
.ipynb
files which were used to conduct the experiments on the datasets and generate the plots. Users need to make sure that the script namedutils.py
is in the same directory as the scripts.