codecheckers/codecheck

Add automated checking of code repository to help codecheckers or (CODECHECK) editors

Opened this issue · 0 comments

nuest commented

Let's try to capture some steps of the human evaluation process in a tool so that we can "pre-check" the contents of a workflow repository:

  • Is there a README?
  • Is there a license?
  • Does the README mention the figures and tables from the paper (clear connection between outputs and code files)?
  • Is there an environment documentation or file (cf. repo2docker REES spec)
  • Do all links (and DOIs) resolve?
  • Are execution times mentioned in the README?
  • If HPC is mentioned, is a sample dataset also mentioned?
  • Are any full paths in documentation or code files (/home/.., C:\Users\.., etc.)
  • ...

Tools that show how the interaction could work: