Add section on paper scope to author instructions
Closed this issue · 5 comments
Currently, the author instructions on the webpage don't really specify what the scope of a ReScience C paper should be, specifically whether the authors should explain the model they are replicating in detail or whether it's enough to refer to the original article. I've seen both versions in submitted papers. As discussed here, I think referring to the original article is especially problematic if it is behind a paywall, since not all readers may have institutional access.
Generally, I think it would be good to have a section in the author instructions that explains what the scope of a paper should be. I'd be happy to draft one, but I'm not sure what should go in there (actually, I thought I saw a section like that in the author instructions once, but it doesn't seem to be there anymore. Am I missing something?) Specifically:
- Should the article explain the model in detail (preferably including equations unless there's copyright problems)?
- Should there be explanations of the code?
- How much should be left open for the authors to do as they like?
My opinion is that the article should contain an explanation of the model (with equations) and some motivation why this article was chosen for replication, but not necessarily explanations of the code. Are there other opinions (especially by the EICs, @oliviaguest, @benoit-girard, @khinsen, @rougier)?
You are raising an important point, and this should indeed be covered by the author instructions, but only once we have a good idea of what works best. Here are my current thoughts on this - they may change, and you are all very welcome to disagree!
In an ideal world of Open Science, repeating material from a replicated article is not useful for anyone. What is useful, however, is a high-level summary of the models and methods being re-implemented. It is helpful for reviewers and readers who can read the replication paper more easily. Moreover, it serves as evidence for correct (or incorrect) understanding by the replicating scientists. Whether or not such a high-level overview should include equations depends on the particularities of the models/methods, so it's best left to reviewers to judge.
The same idea can be applied to explanations of the code. The paper should provide a high-level overview that facilitates reading of the code itself. Details are better explained in comments in the source code files. What matters most is that the paper + code combo is understandable, which is again up to the reviewers to judge. I doubt it makes sense to be more specific in the author instructions at this point, given how little experience we (the scientific community, not just ReScience) have with publishing executable papers.
This leaves the question of how to deal with replicated articles hidden behind paywalls. I'd say it's not our mission to solve the paywall problem. Asking ReScience authors to explain someone else's work because it's behind a paywall punishes the wrong person. But if authors volunteer to do it, that's fine with me as well.
I mostly agree. A high-level summary of the model and / or the code is certainly more useful and appropriate for a replication article than a detailed repetition of the original work. You're also right that the reviewers are probably best positioned to judge this in each particular case.
I also agree that ReScience should not punish the replication authors for the fact that the original work is behind a paywall by requiring additional work. However, I think we need to address at least part of the paywall problem (even if we can't solve it as a whole) for the simple reason that it's especially problematic in the context of replications. In a paper reporting a novel result, if one of the references is unavailable that's inconvenient but does not necessarily keep the reader from understanding the core points. In the case of a replication, if the original is unavailable and the explanations in the replication paper are insufficient, the reader can give up.
Of course, a possible counterargument could be that someone who is interested in reading a replication is probably familiar with the original paper, but I wouldn't want to assume that as the default case.
Since the problem is exacerbated in the case of a replication, the authors need to "solve" it somehow. I think adding some advice to the author instructions would help them with this and as long as it is phrased as advice and not as fixed demands, we're also not punishing the authors by increasing their workload. However, I think it might already help to just bring the issue to every author's awareness - if one has institutional access to most things, it's easy to forget that many people don't. The instructions could go along the following lines:
- Check whether the article is paywalled.
- If it's green OA, maybe include a link to the free version in your manuscript.
- If it is completely closed, make sure your manuscript can be understood without reading the original.
If we add a general recommendation to include a high-level summary of the model in the manuscript anyway, then this also does not punish the authors, I think. Wether the explanations are sufficient is then judged by the reviewers.
The way I see it is that replication (creating an implementation de novo) — via ascertaining if the original publication can be replicated — can play the explicit role of evaluating if the original work was comprehensively specified, e.g., see [1] and [2]. This process is both scientific and meta-scientific because it allows us to then also evaluate the theory itself embodied by the computational account. As such, I do not think it makes sense to have strict rules, but (as others have mentioned above) rough guidelines. Not (only) because it's a nascent concept (I would push back that this aspect is that new methodologically), but because it's genuinely that diverse as an endeavour and warrants an evaluation on a case-by-case basis.
I think it's a complex (although very useful and required for scientific progress generally) undertaking to ask somebody who re-implemented a model to also "evaluate the theory itself embodied by the computational account" but something that possibly is outside the scope of ReScience C. What is within scope is to facilitate the above via promoting alternative open source codebases (implementations) for computational accounts. With that mind, I agree with @cJarvers that authors should "make sure [their] manuscript can be understood without reading the original." But this can be very hard to do without essentially rewriting the original article in one form or another. So the question becomes again what I mentioned: a matter of a case-by-case basis evaluation. Perhaps the rules could be guidelines for reviewers and editors too? Especially since authors can be often too zoomed in to know what is enough detail.
1: Cooper, R. P. & Guest, O. (2014). Implementations are not specifications: specification, replication and experimentation in computational cognitive modeling. Cognitive Systems Research. PDF. DOI: https://dx.doi.org/10.1016/j.cogsys.2013.05.001
2: Guest, O. & Andrea E. Martin (2020). How computational modeling can force theory building in psychological science. PDF. DOI: https://dx.doi.org/10.31234/osf.io/rybh9
Thanks for the broader perspective, @oliviaguest. Especially your 2014 paper is very helpful to understand the role/potential of replications better.
I take it the general consensus is that there should only be rough guidelines and no strict rules. The goal would be to alert authors to these issues: that their article should be readable on its own and that the original may be paywalled. I like @oliviaguest's suggestion to add something similar to the reviewer instructions as well. I guess this would fall under point "4. Clarity and completeness of the accompanying article".
@rougier: yes, I'll make a draft PR. That may take me a few days, though.