songlab-cal/gpn

Data used for computing perplexity

liuyq123 opened this issue · 4 comments

liuyq123 commented

Hi Gonzalo,

I'd like to compute perplexity to see if my retrained model is of the same quality as your model. But I don't have the data/mlm/windows/five_prime_UTR.test/512/128/seqs.txt file you used.
Can you tell me how to download it? Thank you!

gonzalobenegas commented

Hello, we didn't end up using that file. We computed perplexity on the validation_file argument in the training script (data/mlm/dataset/test/Arabidopsis_thaliana.test.512.256.parquet).

liuyq123 commented

Thank you!

Is the perplexity just the eval loss in weight & biases? But my eval loss started at 1.15, and I expected it to be bigger than the 3.01 reported in your paper.

gonzalobenegas commented

It's just $e^{\text{eval loss}}$. For example $e^{1.15}=3.16$. Have you tried running some final steps with a lower learning rate?

liuyq123 commented

I see, thank you!

I haven't yet. The parameters I'm using now are the same as the ones in train_512_convnet_only_athaliana.sh. I've finished 700,000 steps, and the loss is 1.12. I will try a lower learning rate later. Thank you!