acmsigsoft/EmpiricalStandards

Add "list of unreasonable deviations" to .txt export

drpaulralph opened this issue · 4 comments

This is related to issue 31. If issue 31 is done, put the list of unreasonable deviations, described below, AFTER the recommendation.

The .txt export currently looks like this:

Review Checklist

Essential

Y states a purpose, problem, objective, or research question
Y explains why the problem, objective, or research question is important
Y defines jargon, acronyms and key concepts
Y methodology is appropriate for stated purpose or questions
Y describes in detail what, where, when and how data were collected
R describes in detail how the data were analyzed
R discusses and validates assumptions of any statistical tests used
R presents results
R results directly address research questions
R supports main claims or conclusions with explicit evidence or arguments
U discusses implications of the results
U discusses the study's limitations and threats to validity
U contributes in some way to the collective body of knowledge
U language is not misleading; any grammatical problems do not substantially hinder understanding
Y balances the study's anticipated benefits with its potential risks or harms, minimizing risk or harm wherever possible
Y visualizations/graphs are not misleading
... [Decision, Legend, standards used, etc.]

We want to split up the "essential" list to bad and good, as follows. Take note that "R's" stay in the essential list. Only "U's" go in the new list. The name of the new list depends on the decision. If the decision is "reject", the new list should be called "Reasons for Rejection." If the name of the decision is "accept," the new list should be called "Unreasonable Deviations Requiring Revision." Note that items from the Desirable and Extraordinary lists never go in the new Unreasonable Deviations / Reasons for Rejection list.

Review Checklist

Unreasonable Deviations Requiring Revision [for Accept or revision decisions] OR

Reasons for Rejection [for Reject decisions]

U discusses implications of the results
U discusses the study's limitations and threats to validity
U contributes in some way to the collective body of knowledge
U language is not misleading; any grammatical problems do not substantially hinder understanding

Essential

Y states a purpose, problem, objective, or research question
Y explains why the problem, objective, or research question is important
Y defines jargon, acronyms and key concepts
Y methodology is appropriate for stated purpose or questions
Y describes in detail what, where, when and how data were collected
R describes in detail how the data were analyzed
R discusses and validates assumptions of any statistical tests used
R presents results
R results directly address research questions
R supports main claims or conclusions with explicit evidence or arguments
Y balances the study's anticipated benefits with its potential risks or harms, minimizing risk or harm wherever possible
Y visualizations/graphs are not misleading

... [Decision, Legend, standards used, etc.]

Hi, is this completely done? It was previously based on the old (yes/no) deviations, but now we have four types of deviations.

Hi Taher, I've updated the new deviation code and the list should work for different types of deviations now.

OK, sounds good. Thanks, @eschltz.

I just have some comments:

  • Do we still need to keep F and U in the generated file?
  • Is there a case where an attribute gets N?
  • Maybe we need an appropriate heading in the generated file for the attributes a paper satisfies, rather than just saying Essential.
  • On the website we say manuscript, but in the generated file we say paper.

Perhaps @drpaulralph can give input on this.

Thank you.