Nested Transformer: A Solution to Kaggle AI4Code Competition
We called our network architecture "Nested Transformers", since it contains two levels of transformers:
- Cell transformer: A transformer encoder to encode cell tokens, this can be a typical transformer for NLP.
- Notebook transformer: A transformer encoder with cell*cell self-attention to learn the interaction among cells.
A brief description of the solution is provided here.
- Check and modify env.sh
- Create relevant directories and place datasets to the RAW_DIR specified in env.sh
- Check configs/train.sh to adjust the hyperparameters
- Next, follow the steps below:
# install the src module
pip install -e .
# set environment variables
source env.sh
# data preprocessing
python preprocess.py
# training
sh configs/train.sh