xiye17/ComplementaryExpl

Is there an issue with the provided dataset?

Closed this issue · 2 comments

Dear Author,

Thank you very much for open-sourcing your code and dataset. I encountered an issue with a missing cache file while running the GSM experiment. Your code requires reading BERT scores between examples from 'misc/gsm--tr0-512-dv0-500--bertscoreapi-question--scores.json'. However, the provided cache folder does not contain this score file. Upon checking the provided files, I found that they contain results with keys dict_keys(['text', 'index', 'finish_reason', 'prompt', 'completion_offset']), where prompt includes the final example list input to the model. Therefore, I am curious if there might be some issues with the dataset. I look forward to your response.

Best regards

Thanks for the question. Yes the cache file for bertscores is not there. I think the code is set to compute bertscores using bertscore package (line 278 in selective.py) in that case. (this works on my end). Can you try again and let me know if this happens. Thanks

Thank you for your patience!