Add automated checking of code repository to help codecheckers or (CODECHECK) editors
Opened this issue · 0 comments
nuest commented
Let's try to capture some steps of the human evaluation process in a tool so that we can "pre-check" the contents of a workflow repository:
- Is there a README?
- Is there a license?
- Does the README mention the figures and tables from the paper (clear connection between outputs and code files)?
- Is there an environment documentation or file (cf. repo2docker REES spec)
- Do all links (and DOIs) resolve?
- Are execution times mentioned in the README?
- If HPC is mentioned, is a sample dataset also mentioned?
- Are any full paths in documentation or code files (
/home/..
,C:\Users\..
, etc.) - ...
Tools that show how the interaction could work: