Code and data for working with huggingface transformers: https://github.com/huggingface/transformers
Running the code in this repo requires installation of the huggingface transformers library. Everything here was created from within the transformers/examples directory.
Model fine-tuning scripts and Jupyter notebooks require data sources not included in this repo because they exceed GitHub's 100MB file size limit. See the huggingface documentation for information about where to download the sample datasets the original examples reference: https://huggingface.co/transformers/examples.html#. Where Common Crawl (CC) data is mentioned, training and test files sampled from the January 2020 Common Crawl dataset (https://commoncrawl.org/2020/02/january-2020-crawl-archive-now-available/) were used.