/bert-predicting-movie-reviews

Following https://colab.research.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb

Primary LanguagePython

Predicting movie reviews with BERT

Following https://colab.research.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb

Installation

virtualenv --python=python3.7 env
source env/bin/activate
pip install -r requirements.txt

Synchronizing data and model (optionally)

Via dvc:

dvc pull

Downloading train and test data

(env) ➜  bert-predicting-movie-reviews git:(master) python ./download_and_load_datasets.py
>>> import fastparquet
>>> from fastparquet import ParquetFile
>>> pf = ParquetFile('data/train.parq')
>>> train_df = pf.to_pandas()
>>> train_df.head()
                                            sentence sentiment  polarity
0  In Panic In The Streets Richard Widmark plays ...         8         1
1  If you ask me the first one was really better ...         1         0
2  I am a big fan a Faerie Tale Theatre and I've ...        10         1
3  I just finished reading a book about Dillinger...         1         0
4  Greg Davis and Bryan Daly take some crazed sta...         2         0

Preprocessing

(env) ➜  bert-predicting-movie-reviews git:(master) python preprocessing.py
>>> import preprocessing
>>> tokenizer = preprocessing.create_tokenizer_from_hub_module()
>>> print(tokenizer.tokenize("This here's an example of using the BERT tokenizer"))
['this', 'here', "'", 's', 'an', 'example', 'of', 'using', 'the', 'bert', 'token', '##izer']
>>> import pickle
>>> train_features, test_features = (
...     pickle.load(open(f"data/{mode}_features.pkl", "rb"))
...     for mode in ["train", "test"]
... )
>>> len(train_features), len(test_features)
(5000, 5000)
>>> train_features[0].input_ids
[101, 16357, 3854, 25487, 2015, 12124, 2003, 2019, 5186, 2204, 2143, 1012, 2129, 1005, 1055, 2008, 1029, 3854, 25487, 2038, 1037, 2200, 19551, 2126, 
2000, 2425, 1037, 2466, 1010, 2012, 2560, 2002, 2106, 2031, 2007, 12124, 1012, 2000, 2033, 1996, 2878, 2143, 2001, 2066, 2019, 3325, 2130, 2065, 1045
, 2106, 2156, 2009, 2006, 1037, 2235, 2547, 3898, 1012, 2130, 2007, 2035, 1996, 12225, 1010, 1999, 2026, 5448, 1010, 2023, 2143, 2003, 1996, 2087, 31
43, 1010, 2613, 1998, 3048, 3538, 1997, 5988, 2059, 2087, 1997, 1996, 3152, 2006, 1996, 2327, 5539, 2862, 1012, 1045, 2036, 2228, 2009, 2003, 3383, 1
996, 11228, 10458, 1010, 1996, 2087, 7788, 1998, 3143, 2143, 2105, 1012, 2035, 2157, 2045, 2024, 2060, 2204, 3924, 2205, 1010, 2021, 2023, 2028, 2003
, 2026, 5440, 1012, 1996, 102]
>>> train_features[0].label_id
1
>>> train_features[0].is_real_example
True
>>> train_features[0].input_mask
[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
>>> train_features[0].segment_ids
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Model

>>> from defaults import BATCH_SIZE, LABEL_LIST, LEARNING_RATE, NUM_TRAIN_EPOCHS, WARMUP_PROPORTION
>>> from model import model_fn_builder
>>> num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)
>>> num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)
>>> model_fn = model_fn_builder(len(LABEL_LIST), LEARNING_RATE, num_train_steps, num_warmup_steps)

Training

(env) ➜  bert-predicting-movie-reviews git:(master) ✗ BATCH_SIZE=16 python train.py
Training took time  <Period [2019-05-31T08:55:19.991552+02:00 -> 2019-05-31T09:01:59.765379+02:00]>

Evaluation

(env) ➜  bert-predicting-movie-reviews git:(master) ✗ BATCH_SIZE=16 python evaluate.py
{'auc': 0.85695416, 'eval_accuracy': 0.857, 'f1_score': 0.8551164, 'false_negatives': 366.0, 'false_positives': 349.0, 'loss': 0.6504969, 'precision': 0.8580724, 'recall': 0.85218096, 'true_negatives': 2175.0, 'true_positives': 2110.0, 'global_step': 937}

Prediction

(env) ➜  bert-predicting-movie-reviews git:(master) ✗ BATCH_SIZE=16 python predict.py \
  "That movie was absolutely awful" \                                                                                                                  "The acting was a bit lacking" \                                                                       
  "The film was creative and surprising" \
  "Absolutely fantastic"
[{'sentence': 'That movie was absolutely awful', 'probabilities': array([-1.3002045e-03, -6.6458936e+00], dtype=float32), 'label': 'Negative'}, {'sen
tence': 'The acting was a bit lacking', 'probabilities': array([-2.7289076e-03, -5.9052234e+00], dtype=float32), 'label': 'Negative'}, {'sentence': '
The film was creative and surprising', 'probabilities': array([-6.6402445e+00, -1.3075859e-03], dtype=float32), 'label': 'Positive'}, {'sentence': 'A
bsolutely fantastic', 'probabilities': array([-5.9880304e+00, -2.5118012e-03], dtype=float32), 'label': 'Positive'}]