syslot/albert-zh-for-pytorch-transformers

轉換好的 Albert 中文模型 (for pytorch-transformers)

Albert-zh for pytorch-transformers

• 僅僅是基於參考進行轉換，然後踩踩雷
• Albert zh for pytorch-transformers
• 測試支援繁體中文

可用模型

• albert_tiny_zh
• albert_base_zh
• albert_large_zh
• albert_xlarge_zh

API

from albert_zh import ...

AlbertConfig
AlbertTokenizer
AlbertModel
AlbertForMaskedLM
AlbertForQuestionAnswering
AlbertForSequenceClassification

https://huggingface.co/transformers/v2.3.0/model_doc/albert.html

使用方法

• 請參見usage_example.py
• 測試在 transformers 2.3.0 正常運作

常見問題

loss 降不下來，訓練出來變垃圾

確保 model class 與 model config 由 albert_zh 引入，而非 transformers

lonePatient/albert_pytorch#35

AttributeError: 'BertConfig' object has no attribute 'share_type'

config.json增加"share_type":"all"

訓練時模型亂印東西

請用log()代替print()，並且在程式開始的時候先執行一次blockPrint()

import os,sys
def log(*logs):
    enablePrint()
    print(*logs)
    blockPrint()

# Disable
def blockPrint():
    sys.stdout = open(os.devnull, 'w')

# Restore
def enablePrint():
    sys.stdout = sys.__stdout__

測試環境

• python 3.6.4
• pyotrch 1.3 (with cuda 10)
• transformers 2.3.0

參考

albert zh

• https://github.com/brightmart/albert_zh

albert tf to pytorch

• https://github.com/lonePatient/albert_pytorch