keras-team/keras-nlp

Cannot reproduce results from notebook on Colab

Opened this issue · 3 comments

jespernwulff commented

Describe the bug
Running the KerasNLP guide in Colab, I get more or less 50% classification accuracy, i.e. no better than a random classifier. It get the same when fine-tuning the classifier. I am making no changes to any code in the notebook. Just running the code.

To Reproduce
Link to the Colab Notebook, which reproduces the bug.

Expected behavior
I expected to get the same accuracy results as reported in the text, i.e., around 78% accuracy for the first example.

SuryanarayanaY commented

Hi @jespernwulff ,

I have replicated the issue and attached gist here.

In the generated documentation guide the sparse_categorical_accuracy touced to 0.8443 in 2nd epoch but when trained now it is staying at 0.5 only even in epoch 3.

SamanehSaadat commented

Hi @jespernwulff!

Thanks for reporting this issue! I'll look into it!

SamanehSaadat commented

I think the issue is related to the dataset. I also got around 50% accuracy when I ran the fine-tuning with the IMDB dataset downloaded in the guide but when I switched to loading the IMDB dataset from tensorflow_datasets, I got 87% accuracy.

I'll update the guide to get the IMDB dataset from tensorflow_datasets for now.

  • 👍1