About results on GPT2XL

Question

About results on GPT2XL

littlefive5 opened this issue a year ago · 8 comments

I'm curious about the hyper-parameters for running GPT2-XL.
I tried the default setting and achieved loss 0.04.
The output before editing is the same as the output after editing.
Is there anything I have overlooked?

Answer 1 · 2023-09-26T18:31:38.000Z

Hi there! Could you be more specific about what you actually ran? For this experiment there should be a training phase where a new GPT2-XL model is trained, did this happen?

Answer 2 · 2023-09-27T01:57:47.000Z

Well, I didn't train a new model.
I just want do a simple test and I modified the example.ipynb as the following:

I find that the output before editing is the same as the output after editing.
I think that pre-training the model can make the editing get a better performance?
But without pre-training, it is odd that nothing happens after the edit.

Answer 3 · 2023-09-27T12:25:02.000Z

Thanks for the extra details and for pointing this out! I will try to reproduce this error this week and will follow up. It's possible but unlikely that the training of the model's the problem. Instead, the token replacement method in grace/editors/grace_barebones.py is the one that works best for QA models, not GPT2-XL. The point of the barebones version was just for a QA example so this might be a case where that doesn't work as is.

Answer 4 · 2023-09-27T14:28:57.000Z

Thanks, I also build a Grace model from the grace.py using the setting in the config files.
But I encounter the same problem.
I think that maybe the default setting in config/editor/grace.yaml is not for GPT-style model?

Answer 5 · 2023-09-27T14:32:00.000Z

I'll look into this this week. You could also try setting replacement to replace_last When using grace.py this time, did you train the GPT2-XL model like in main.py?

Answer 6 · 2023-09-27T15:22:59.000Z

I did not train it. I would try to train the model this time. By the way, can you share me a checkpoint for the trained model since pretraining is time-consuming?

Answer 7 · 2023-09-27T16:03:04.000Z

Yes I am working to upload the checkpoint now, will let you know when it's up!

Answer 8 · 2023-10-09T14:30:11.000Z

@littlefive5 I've confirmed that the issue is that we evaluate solely on the log likelihoods of the correct outputs, not based on decoding and checking. This is actually expected behavior! I'll try to add in another version of GRACE training for GPT models that will fix this (it's easy to fix: just don't turn labels off from the prompt), but we didn't conduct our experiments this way.