Would share more details for using BERT?

[ybdesire]

Would you share more detailed steps for using BERT as word embedding ?

It seems bert has not been suppoted ?
https://github.com/allanj/pytorch_lstmcrf/blob/master/trainer.py#L56

[allanj]

Since I'm actually working/debugging on another version (in another branch) that uses BERT (from Huggingface) as a direct encoder which allows us to fine-tune BERT, I did not provide more details on using BERT as static embedding.

The quick answer I will give is to use the BERT As Service repo: https://github.com/hanxiao/bert-as-service

You can extract the representations for your sentence and save them using pickle.
One thing you should keep in mind is the BERT-As-Service package gives you workpiece token representation instead of the word representation.

You have three options:

1. Use the first workpiece representation of a word to represent the word
2. Use the last workpiece representation of a word to represent the word
3. Use the average workpiece representation of a word to represent the word

Since you have such a requirement, I will try to provide the script later on as well.

[allanj]

Please check out the latest commit 9afe2ed .
You will be able to process bert representation now.

Script can be found in preprocess/get_bert_vec.py

I use BERT-As-Service for now.
Alternatively, you can just use the one provided by HuggingFace.

[ybdesire]

@allanj wow, thank you very much for the quickly help. I will try it latter. : )