This repository presents the code corresponding to the paper Cup Curriculum: Curriculum Learning on Model Capacity. As highlighted in the paper, the cup curriculum is a curriculum learning approach focussing on the model capacity. The analysis of different variations of the cup curriculum is done in a natural language processing setting. Our empirical evaluation of these different strategies show that the cup curriculum outperforms early stopping reliably with a median relative performance increase of 1.2% to 1.7% (see Fig.1 right). In addition to this result, we observe from both the cup curriculum and early stopping a similar resiliance to overfitting.
The following gives an overview of the cup curriculum and tries to explain which functions you need to manipulate to adjust my code propperly.
A formal definition of the cup curriculum can be found in the corresponding paper.
The cup curriculum consists of two training phases, one reduces the model capacity via pruning and the other increases it via reintroduction of model capacity.
The first phase of training currently uses a variation of iterative magnitude pruning to reduce model capacity.
The pruned weights are reintroduced in a second phase, resulting in the model capacity to show a cup-shaped curve over the training iterations (see Fig.1 left).
CupCurriculum.py implements the cup curriculum based on the Lottery Ticket Hypothesis Paper, this repository, and this transformer tutorial of pytorch. Main.py is used to execute CupCurriculum.py's main(). Utilities.py defines simple helper functions used for file management and plots. Lastly plot_generator.py provides functions for different visualization of the achieved results.