acmsigsoft/artifact-evaluation

Artifact Evaluation Process: Communication

jon-bell opened this issue · 3 comments

jon-bell commented

How much communication should be allowed between evaluators and authors during the review period?

Examples range from:

  • No communication other than the authors submit an artifact and eventually get reviews
  • A "kick-the-tires" phase, in which evaluators can do some sanity checking, ask questions, and get a response (one time).
  • Open discussion, where evaluators can freely discuss with authors (blind or not)
sbaltes commented

I very much prefer the latter, which I experienced during the FSE 2018 artifact evaluation. In that case, they used GitHub issues to discuss the artifacts using anonymized user accounts.

  • 👍1
piskachev commented

At least "kick-the-tires" phase should be there.

wagnerst commented

Fully open, in my opinion. There are always clarifications necessary.