tree-park/bert-pretrainer

BERT paper implementation from scratch- Pytorch

Python

BERT Paper Implementation

BERT implementation pytorch code

Architecture

Embedding

Positional Encoding
Word Embedding
Segment Embedding

Transformer Encoder

Multi-head Attention
Position-wise Fead Forward Network
ResNet + NormLayer

Pre-train for Two Task

Next Sentence Prediction
Masked Language Model

Refer

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
(https://arxiv.org/abs/1810.04805)