Seed repo for the CVPR'21 Continual Learning Challenge, both for the Supervised Learning and Reinforcement Learning tracks.
Join the community on the ContinualAI slack (the #cl-workshop channel)
Remember to update, using the instructions below!
- Install Docker (or Docker Desktop if you're on Mac/Win) and make sure it's running
- Make sure you can run the
docker
command without root. This is needed for the evalai submission tool. - Install Unix command line tools (like
rm
, andmake
)- On Linux, you already have everything
- On Mac, install
homebrew
and through homebrew, themake
package - On Windows, not 100% sure, but try using Cygwin or Windows Subsystem for Linux. PowerShell alone is NOT sufficient.
- Install Miniconda/Anaconda and have the conda/python/pip binaries for that available on your
$PATH
and in your shell. - Create an account at http://eval.ai, create a participant team and click through the steps on the challenge submission site (select your team, click next, agree to the terms & conditions)
-
Get your API Authentication token from you profile on the EvalAI Website.
-
From the root directory of the repository, run the following commands:
./install.sh <EVALAI_TOKEN>
conda activate clcomp21
Submissions should be contained in the submission/
folder. You can draw inspiration from the following examples.
-
submission/dummy_method.py: Model-free example that outputs random predictions/actions. Applicable to all tracks (RL and SL).
-
-
Simple Classifier: Standard neural net classifier without any CL-related mechanism. Works in the SL track, but has very poor performance.
-
CL Regularized Classifier: Adds a simple CL regularization loss to the classifier above. Still exhibits poor performance.
-
Multi-Head / Task Inference Classifier: Performs multi-head prediction, and a simple form of task inference. Gets better results that the two previous methods.
-
-
- A2C Example: Example where A2C is implemented from scratch as a Method for the RL track. The code for A2C was adapted from this blogpost.
-
NOTE: See the Sequoia repo for more examples!
-
- Customizing the BaselineMethod Example where we extend the BaselineMethod from Sequoia, which is applicable for both RL and SL.
Make sure to change the contents of submission/submission.py
, so that the various get_method
actually return your method to use for each task (get_method_sl
-> SL track, get_method_rl
-> RL Track, get_method
-> Both/Bonus track)
For more information on the various CL Settings and Methods available in Sequoia, please check out the Sequoia repository.
This is already part of the install.sh
script, so you don't have to run this if you only just installed everything. This is best run once every few weeks or if you notice the readme changed or there was announcement in the challenge Slack.
make update
- Supervised Learning track:
make sl
- Reinforcement Learning track:
make rl
- "Combined" track:
make both
Pick the right option for the track that you want to submit to (...-sl/-rl/-both):
make [upload-sl|upload-rl|upload-both]
Q: When I'm trying to make a submission, it says "Error: You have not participated in this challenge"
A: You do actually have to participate in the challenge. Please see the last step of the "Prerequisites" section above. On eval.ai, create a participant team, go to the submission page of this challenge (https://eval.ai/web/challenges/challenge-page/829/submission), click on your team, click next, agree to the terms & conditions. Then you are officially a participant and can make submissions. To verify that this worked, please run evalai challenges --participant
on a terminal and make sure that it has our challenge listed there (ID: 829, Title: "Challenge track of "Workshop on Continual Learning in Computer Vision 2021")
Q: I'm getting an error a la Connection aborted. FileNotFoundError(2, No such file or directory)
A: Make sure your current user has access rights to the docker cli without sudo. To test this, run docker run hello-world
(without sudo obvs).