/BERT-pretraining-techniques-on-TPU-pods

Primary LanguagePython

A Pipeline Of Bert pretraining techniques On Google TPU-pods

A pipeline of various pertaining Bert approaches on text dataset using google TPU.

Introduction

This repo introduces various pre-training approaches to enhance generic BERT models. All engineering proceses were designed for TPU devices. Pre-training methods were implemented using NPL library from TF2 model garden.

  1. Pre-training from scratch
  2. Contiunal pre-training
  3. Simultaneous pre-training

Cite

The study will be published as a publication...

Bibtex

References

[1]Wada, S., Takeda, T., Manabe, S., Konishi, S., Kamohara, J., & Matsumura, Y. (2020). Pre-training technique to localize medical bert and enhance biomedical bert. arXiv preprint arXiv:2005.07202.

Acknowledgements

We would like to acknowledge the TPU Research Cloud program (TRC) and the Google’s CURe program in providing access to TPU-pods and GCP credits, respectively.