A useful link for USPTO dataset

Question

A useful link for USPTO dataset

csyanghan opened this issue a year ago · 7 comments

https://deepchem.readthedocs.io/en/latest/api_reference/moleculenet.html#uspto-datasets

Answer 1 · 2023-06-20T01:24:31.000Z

At the same time, could you explain the meaning of Suzuki, or tell me where you collect them.

Answer 2 · 2023-06-28T23:09:52.000Z

Thanks for your attention for this project! The suzuki is a specific type of chemical reaction and we have updated the data source in our README: https://github.com/ChemFoundationModels/ChemLLMBench#dataset

Answer 3 · 2023-06-30T07:35:33.000Z

Thanks a lot, I find your paper report the BLEU-2 of MolT6-Large in Molecule Captioning is 0.482, however, it is reported as 0.594 in the original paper.

Answer 4 · 2023-07-01T12:08:24.000Z

This is because the test set are different. In the original paper, they use the whole test set to evaluate. In our work, due the API requesting time and money consumption, we sample 100 examples from the test set to evaluate all models. The 0.482 here is the result for the 100 examples.

Answer 5 · 2023-07-03T14:14:27.000Z

Thanks, I see. 100 samples are a bit small (as ~3300 samples in the test set), this may cause bias, and I also suggest you should indicate that in the paper.

Answer 6 · 2023-07-04T09:26:24.000Z

Thanks for your suggestion. Actually we have indicated the test set in Table 2 and the "Experiment setup strategy" part in Section 2 (The Evaluation Process and Setting). Due to the high time consumption and fee of the API requesting, we have to reduce the test set to 100. We're currently preparing more tasks and dataset to evaluate LLMs. Once all tasks are finished, we will consider increase the size of the test set.

Answer 7 · 2023-07-08T02:30:44.000Z

I am confused by the Scaffold and the Random option in the tables, is it mean that Scaffold is the certain examples (carefully picked) and Random is the random examples from the train set?