Develop a benchmark set (or sets) for realistic comparison of performance?
Opened this issue · 2 comments
There's been some discussion, if we are going to compare performance, that we have a community organized set of simulations for benchmarks. This would probably actually be a paper developed by authors, but the journal editorial board might need to be involved to help it along.
We should start with the density benchmark set from John Chodera and Kyle Beauchamp. I believe Kyle has also done some work on modernizing it/making it easy to repeat with protocol modifications, etc. As it is, they ran it with GAFF and then I ran it with SMIRNOFF, so there will be at least multiple force fields on the same set. Would be good to extend out for other codes and force fields. I can get Kyle to update us on where things stand if it's of interest.
This is on roughly 42 different pure solvents at a range of temperatures and pressures, IIRC. The data is from ThermoML. I can give a reference if needed, but not at the moment as I'm headed to a meeting.
I think there are actually multiple benchmarks. At least two: one for correctness, one for speed.