The Tab Transformer is a novel model for tabular data that applies the self-attention mechanism from Transformer models to learn complex relationships between categorical features. It encodes categorical variables into embeddings and uses multiple layers of self-attention to dynamically understand the context of each feature within a row.
The FT Transformer extends the concepts from the Tab Transformer by incorporating feature tokenization, allowing it to effectively handle both categorical and continuous features. It improves upon the Tab Transformer by using feature-wise transformations which lead to better performance on a range of tabular datasets.
The conditionally independent point process transformer is similar to a GPT Neo-X transformer. Measurements are aggregated together within an event to form event embeddings, which are then processed via an autoregressive transformer.
This code has been tested on:
- Ubuntu 22.04.4 LTS
- Python 3.10.12
- PyTorch 2.2.1 compiled for CUDA 12.1 and cuDNN 8.9.7 (Instructions for Pytorch 2 and CUDA 11.8) (CUDA 12.1) (cuDNN v8.9.7) (cuDNN instructions) or PyTorch 2.3.1 compiled for CUDA 12.4 and cuDNN 9.2.0.
Note: it is recommended to install PyTorch in a python virtual environment (see Getting started).
NVIDIA Driver Version: 550.54.15 or 550.90.07
CUDA Version: 12.1 or 12.4
GPUs: GeForce RTX 2080 (x2) or NVIDIA RTX 6000 Ada Generation (x3)
-
model_utils.py
- Set of utility functions and classes for training and validating machine learning models in PyTorch, including support for data loading, model training with techniques like MixUp and CutMix, model validation, and custom dataset handling.
-
data_loader.py
[static analyses]- Loads and merges data from .txt files. Randomly splits the data into training (70%), validation (20%), and test (10%) sets. Preprocesses datasets, converts datasets into PyTorch Tensors, and saves them to file.
-
event_stream.py
[temporal analyses]- Preprocess and generate "Event Stream" dataset. Make sure to set the appropriate file paths and configurations within the script. The script will generate the necessary data files, including the outcomes, diagnoses, procedures, and labs data.
-
build_task.py
[temporal analyses]- Defines the specific task --- this case, the task is binary classification on A1cGreaterThan7.
-
finetune.py
[temporal analyses]- Fine-tunes a transformer (from scratch) on the binary classification task. Utilizes the
fine_config.yaml
config file.
- Fine-tunes a transformer (from scratch) on the binary classification task. Utilizes the
-
tune_finetune.py
[temporal analyses]- Fine-tuning script called by
tune_temporal.py
.
- Fine-tuning script called by
-
data_analysis.py
[static analyses]- Visualizes one-hot encoded feature sparsity and generates training dataset summary statistics.
-
tune_static.py
[static analyses]- Hyperparameter optimization for static transfomer models using Ray Tune.
-
tune_temporal.py
[temporal analyses]- Hyperparameter optimization for temporal transfomer models using Ray Tune.
-
hp_sweep.py
[temporal analyses]- Perform hyperparameter tuning for the temporal analyses by loading the dataset, creating the model, and training it.
-
train.py
[static analyses]- Trains transformer model, supporting Tab Transformer and FT Transformer. Optional pretraining with CutMix and Mixup.
Getting started (credit)
- Check if
pip
is installed:
$ pip3 --version
#If `pip` is not installed, follow steps below:
$ cd ~
$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ python3 get-pip.py
- Install virtual environment first & then activate:
$ python3 -m pip install --user virtualenv #Install virtualenv if not installed in your system
$ python3 -m virtualenv env10 #Create virtualenv for your project
$ source env10/bin/activate #Activate virtualenv for linux/MacOS
- Install PyTorch via pip by running following command:
# CUDA 12.1
$ pip3 install torch==2.2.1 torchvision==0.17.1 torchaudio==2.2.1 --index-url https://download.pytorch.org/whl/cu121
# CUDA 12.4: https://github.com/pytorch/pytorch#from-source
- Clone project repo and install other dependencies from
requirements.txt
file:
$ git clone https://github.com/jvpoulos/diabetes_pred.git
$ pip3 install -r diabetes_pred/requirements.txt
- (Optional, for static analyses) Install git repo TabTransformer, forked from tab-transformer-pytorch:
$ pip3 install git+https://github.com/jvpoulos/TabTransformer.git
Optionally, follow instructions for installing flash attention. Note: FlashAttention only supports Ampere GPUs or newer.
- Clone forked version of git repo EventStreamGPT, outside of project directory:
$ git clone https://github.com/jvpoulos/EventStreamGPT.git
touch EventStreamGPT/__init__.py
touch EventStreamGPT/EventStream/__init__.py
touch EventStreamGPT/EventStream/data/__init__.py
- Install Dask (optional):
$ python3 -m pip install "dask[complete]" --upgrade
- Load data:
$ cd diabetes_pred
$ python3 src/data_loader.py
- (Optional) Create plots and summary statistics for the training dataset (static analyses):
$ python3 src/data_analysis.py
- (Optional) Hyperparameter optimization for transfomer model. Arguments:
--model_type
('TabTransformer', 'FTTransformer', or 'ResNet')--epochs
.
$ python3 src/tune_static.py --model_type FTTransformer --epochs 25
- Train and evaluate transformer. Arguments:
--model_type
(required)--dim
--depth
--heads
--ff_dropout
--attn_dropout
--batch_size
--learning_rate
--scheduler
--weight_decay
--epochs
--early_stopping_patience
--use_cutmix
--cutmix_prob
--cutmix_alpha
--use_mixup
--mixup_alpha
--clipping
use_batch_accumulation
--max_norm
--mixup_alpha
--model_path
.
$ python3 src/train.py --model_type FTTransformer --dim 128 --depth 3 --heads 16 --ff_dropout 0 --attn_dropout 0 --use_batch_accumulation --clipping --max_norm 5 --batch_size 8 --epochs 200 --early_stopping_patience 10 --scheduler 'cosine'
or
$ python3 src/train.py --model_type ResNet --dim 128 --depth 3 --dropout 0.2 --batch_size 8 --epochs 200 --early_stopping_patience 10 --use_batch_accumulation --clipping --max_norm 5 --scheduler 'cosine' --learning_rate 0.01 --normalization layernorm --use_mixup --use_cutmix --weight_decay 0.1 --d_hidden_factor 4
- Create Python path for ESGPT
$ echo 'export PYTHONPATH=$PYTHONPATH:../EventStreamGPT' >> ~/.bashrc
$ source ~/.bashrc
$ echo $PYTHONPATH
# :../EventStreamGPT
- Create data files (arguments:
--use_dask
,--debug
):
$ python3 src/event_stream.py --use_labs
- (Optional) Hyperparameter optimization for transfomer model:
$ python3 src/tune_temporal.py --epochs 200
- Train the transformer from scratch:
$ python3 src/finetune.py use_labs=true