/albert_pytorch

A Lite Bert For Self-Supervised Learning Language Representations

Primary LanguagePython

English Version | 中文版说明

albert_pytorch

This repository contains a PyTorch implementation of the albert model from the paper

A Lite Bert For Self-Supervised Learning Language Representations

by Zhenzhong Lan. Mingda Chen....

Dependencies

  • pytorch=1.10
  • cuda=9.0
  • cudnn=7.5
  • scikit-learn
  • sentencepiece

Download Pre-trained Models of English

Official download links: google albert

Adapt to this version，download pytorch model (google drive):

v1

v2

Fine-tuning

１. Place config.json and 30k-clean.model into the prev_trained_model/albert_base_v2 directory. example:

├── prev_trained_model
|  └── albert_base_v2
|  |  └── pytorch_model.bin
|  |  └── config.json
|  |  └── 30k-clean.model

2．convert albert tf checkpoint to pytorch

python convert_albert_tf_checkpoint_to_pytorch.py \
    --tf_checkpoint_path=./prev_trained_model/albert_base_tf_v2 \
    --bert_config_file=./prev_trained_model/albert_base_v2/config.json \
    --pytorch_dump_path=./prev_trained_model/albert_base_v2/pytorch_model.bin

3．run sh run_classifier_sst2.shto fine tuning albert model

Result

Performance of ALBERT on GLUE benchmark results using a single-model setup on dev:

Cola Sst-2 Mnli Sts-b
metric matthews_corrcoef accuracy accuracy pearson
model Cola Sst-2 Mnli Sts-b
albert_base_v2 0.5756 0.926 0.8418 0.9091
albert_large_v2 0.5851 0.9507 0.9151
albert_xlarge_v2 0.6023 0.9221