💻🔥🎁 Kaggle compute power from your terminal

🆓 Access to free kaggle compute power from your command line.

Push notebooks through Kaggle API.
Version everything under github.
Save training metrics (+option to log to weights and biases)

Here's what your runs could look like on Weights and biases...

🆘 How to use?

pip install kaggle
Create a kaggle account, get a kaggle API token.
Copy paste this repo as a template and start customizing, check that you can train locally...
Fill the __kaggle_login.py ⚠️ do not push it to git
Push your code to github
Use command line to push your notebook to Kaggle.

💡 What to customize?

Customize configuration.py

NB_ID = "training-notebook" # This will be the name which appears on Kaggle.
GIT_USER = "balthazarneveu" # Your git user name
GIT_REPO = "mva_pepites" # Your current git repo
KAGGLE_DATASET_LIST = [] # Keep free unless you need to acess kaggle datasets. You'll need to modify the remote_training_template.ipynb.

Note: you can add kaggle datasets (if you need to put 4Gb of data, it's possible to host it with Kaggle datasets). Fill the KAGGLE_DATASET_LIST. You'll also have to customize the remote_training_template.ipynb to unzip and acess datasets.

You can run several experiments in a row using -e 1 2 3. If initialization is long (decompress datasets, preprocess etc...), it may be worth running several experiments in a row.

📉 Training

🆔 Keep track of experiments by an integer id.

Each experiment is defined by:

📜 Dataloader configuration (data, augmentations)
⚙️ Model (architecture, sizes)
💹 Optimizer configuration (hyperparameters)

🧪 Code to define new experiments

Remote training

Retrive your kaggle token from the website.
Several accounts mean simply more GPU power. As of 2024, Kaggle allows 30 hours per week, limited to 12hours of execution per notebook.
🔓 Create a __kaggle_login.py file locally.

kaggle_users = {
    "user1": {
        "username": "user1_kaggle_name",
        "key": "user1_kaggle_key"
    },
    "user2": {
        "username": "user2_kaggle_name",
        "key": "user2_kaggle_key"
    },
}

⚠️ Do not push these secret tokens to github and leak it publicly 🤦

Run python remote_training.py -u user1 -e X -nowb This will create a dedicated folder for training a specific experiment with a dedicated notebook.

use -p (--push) will upload/push the notebook and run it.
use -d (--download) to download the training results and save it to disk. This is not automatic

🟢 First time setup

python remote_training.py -u user1 -e 0 --cpu --push -nowb

use --cpu to setup at the begining (avoid using GPU when you set up ⚠️ )

Go to kaggle and check your notifications to access your notebook.

Edit notebook manually

allow internet requires your permission (internet is required to clone the git)

☎️ a verified kaggle account is required

🔑 Allow Kaggle secrets to access wandb:

wandb_api_key: weights and biases API key.

You'll need to manually edit the notebook under kaggle web page to allow secrets.

Quick save your notebook.

Now run the remote training script again, this should execute.

❤️ Don't be scared, the provided experiments will go very fast (less than 2 minutes to run on kaggle).

Local training

python train.py -e 0 1

⚠️ This is fully experimental, there are probably much better ways to wrap an existing training script.

🔍 Want to contribute, new features, spotted a bug under your OS? file an issue here

🔑 It is possible to work with private github repositories but it will require your github token to be inserted into kaggle secrets.

⭐ Give a star to this repo if you're planning using it.

Jovillios/mva_pepites

💻🔥🎁 Kaggle compute power from your terminal

📉 Training

Remote training

🟢 First time setup

Local training