This project was done during my semester at the MVA in the Deep Learning for Imaging course teached by O. COLLIOT (CNRS), M. VAKALOPOULOU.
Lymphocytosis is characterized by an incerease in the number of lymphocytes, a type of white blood cell, in the bloodstream. It can indicate the body’s immune system is fighting off pathogens (reactive) or it may result from chronic patologies including blood cancers (tumoral). The diagnosis relies on two stages : a microscopic examination and clinical tests. We tackle the lack of reproducibility of the first stage to better filter patients which should be referred for flow cytometry analysis. We approached this problem through the prism of Multi-Instance Learning, given that we have several bags of images per patient. To do this, we coupled the ability of Vision Transformers to capture long-distance relationships with self-attention mechanism between images of a patient we have adapted to medical imaging using self-supervision methods. We incorporated clinical data to create a trainable end-to- end model with low computational resources using the distillation mechanism and ingenious finetuning methods.
If you are intrested by some interactive we invite you to navigate in our Wandb projet here !!
Code is available within notebooks which is kind of main.py but as we run it on kaggle (to get acces to GPU) it is easier. You will need to fill cfg configuration dictionnaries, you give the explaination of the key in the notebooks/Configuration.md
in src
Our project right now support some models from timm:
- VIT based models
- ResNet based models
- EfficientNet based models
First, you will find the source code in src
.
- We provide in
data.py
andmodel.py
and zoo of DataLoader and Models that you can access by the Factory. If you want to add another one, you just need to register it. - We also provide in
adapter.py
some generic code for finetuning VIT from timm. - Finally, the code for the MAE self-supervised pre-training is availale in :
mae_pretraining.py
. - The file
utils.py
contains some usefull functions.
One data split is provide if you want to launch our code quicly in data/split.py
. However, we encourage you to create yours. We have created some usefull function to do it interactively.
code : self_supervised_MAE.ipynb
It is compatible with VIT timm based models. you just need to fill the configuration _cfg
and create your own dataloader and it will run !
code : training_using_MAE.ipynb
You will need to define a cfg
dictionary.
You may need to create a wandb account and link it. If you don't want to, you can simply set the key no_wandb
: True.
Pay attention to change download you weight from Self-supervised pre-traning (MAE) (otherwise, our models weight are available check the `MAE pretraining``section for some models).
code : final-notebook-experiments.ipynb
You will need to define two cfg
dictionaries :
- the first one corresponds to the pretraining, it will just train the encoder
- the second one correponds to the finetuning using adapters and you can pick classifier available from
model.py
The weights for the MAE section are available here for the model vit_tiny_patch16_224.augreg_in21k
and vit_small_patch16_224.augreg_in21k
model from timm
The models were trained with limited resources from 250 to 800 epochs and different datasets, and convergence was not achieved (lack of computing time).
We proposed to train an end-to-end model using pre-trained models : MAE, DinoV2, ResNet, EfficientNet. For the VIT based models, we implemented adapter finetuning : LoRA, prompttuning, bottleneck adapter and adaptformer.
Finally, we have proposed several classifiers that use the self-attention mechanism.
We have also proposed a model that uses clinical data in the form of cross attention, as a doctor might do.
We adapted the really great code of : https://github.com/IcarusWizard/MAE/tree/main. We also re-use some code from DLMI course practical works.