[REVIEW]: The Riffomonas Reproducible Research Tutorial Series

Question

[REVIEW]: The Riffomonas Reproducible Research Tutorial Series

Closed this issue 6 years ago · 43 comments

Submitting author: @pschloss (Patrick D Schloss)
Repository: https://github.com/riffomonas/reproducible_research
Version: v1.0.0
Editor: @tracykteal
Reviewer: @jhollist
Archive: 10.5281/zenodo.1404230

Status

Status badge code:

HTML: <a href="http://jose.theoj.org/papers/62515e587e897f867e0746e79bc3ab36"><img src="http://jose.theoj.org/papers/62515e587e897f867e0746e79bc3ab36/status.svg"></a>
Markdown: [![status](http://jose.theoj.org/papers/62515e587e897f867e0746e79bc3ab36/status.svg)](http://jose.theoj.org/papers/62515e587e897f867e0746e79bc3ab36)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jhollist , please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://jose.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @tracykteal know.

Review checklist for @jhollist

Conflict of interest

As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

I confirm that I read and will adhere to the JOSE code of conduct.

General checks

Repository: Is the source for this learning module available at the repository url?
License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
Version: Does the release version given match the repository release (v1.0.0)?
Authorship: Has the submitting author (@pschloss) made substantial contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

A statement of need: Do the authors clearly state the need for this module and who the target audience is?
Installation instructions: Is there a clearly stated list of dependencies?
Example usage: Do the authors include examples of how to use the module?
Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Pedagogy / Instructional design

Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

Authors: Does the paper.md file include a list of authors with their affiliations?
A statement of need: Do the authors clearly state the need for this module and who the target audience is?
References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Answer 1 · 2018-06-11T23:00:27.000Z

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks. @jhollist it looks like you're currently assigned as the reviewer for this paper 🎉.

⭐ Important ⭐

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

Answer 2 · 2018-06-11T23:00:28.000Z

Attempting PDF compilation. Reticulating splines etc...

Answer 3 · 2018-06-11T23:00:52.000Z

--> Check article proof 📄 <--

Answer 4 · 2018-06-11T23:02:48.000Z

👋 @jhollist — here's where the review happens! Feel free to ask any questions, tagging the author or handling editor, as needed. You can also open issues on the repository of the submission.

Answer 5 · 2018-06-13T16:18:53.000Z

@tracykteal I'm hoping you can secure a second reviewer for this submission. Thanks!

Answer 6 · 2018-07-06T13:17:54.000Z

@labarba and @tracykteal, I have a couple of questions about the review checklist.

I assume the "statement of need" in Documentation and JOSE Paper are the same?
How should the Documentation be presented in the submission? I am still going through the review but am pretty sure all of the required things will be there. Didn't know if you were also looking for a more concise doc (i.e. a README) that had all of this in it.

Hope to finish this up today, but if not it'll be week after next (on vacation next week).

Answer 7 · 2018-07-06T13:24:20.000Z

Actually, we've been having a conversation in another review that tends to leaving the Statement of Need in the paper only. That is, not requiring it in the documentation.

Answer 8 · 2018-07-06T13:42:04.000Z

It looks like the submitted paper reproduces verbatim the contents of the repository's README. Hmm...

Answer 9 · 2018-07-06T14:17:31.000Z

I find it unsatisfying that the paper and the README have the same contents. These documents have different goals., in my opinion

About the paper, the Author Guide says:

The goal is that someone reading the JOSE paper has enough information to decide if they'd be interested in adoping the learnig module or software.

The paper is a form of "scholarly advertisement" of the learning module. Readers want to know who the target audience, the design behind the learning module, how it could be used by another instructor or by an independent learner, how it has already been used in the classroom or elsewhere, and of course a description of the contents.

The README is the entry point to the repository, and it might be too lengthy to include all of the above. It should have installation instructions, list of dependencies, maybe statements about the license and how to contribute, and include an index of the lessons, with links.

Answer 10 · 2018-07-06T14:27:05.000Z

Looking now at the Introduction tutorial, I find a statement that succinctly says what this is:

this is a series of tutorials on improving the reproducibility of data analysis for those doing microbial ecology research. [...] the data set that we're going to be working with [...] is from the human microbiome research.

I wish this to-the-point statement appeared at the top of the paper!

Answer 11 · 2018-07-06T14:31:56.000Z

Thanks for the guidance! I was leaning in the same direction on the README. Will include in my review as well.

Answer 12 · 2018-07-06T14:45:55.000Z

@pschloss : I would like to see the bullet lists on slides 5 & 6 of the Introduction lesson, just like that, on the paper. It gives the reader an immediate picture of the contents of the module!

Also, in the Statement of Need (in the paper), you give general statements about reproducibility, about the field of microbiome research, and so on, and only in the final sentence refer to this tutorial. What we really want in this section is a statement about how this tutorial satisfies a particular need, and perhaps how it differs (or is similar) to other tutorials on the subject.

From the Author Guide:

Authors make the case for their submission's contribution in the paper, under the heading "Statement of Need."

@tracykteal : I'm stepping in here, in this review that you're editing, because with the first handful of papers in JOSE we are in fact defining our genre.

Answer 13 · 2018-07-06T15:37:24.000Z

Didn't get finished on review today. Will finish up week of July 16th.

Answer 14 · 2018-07-06T16:24:25.000Z

Thanks for the feedback and my apologies for missing the desired distinction between the paper and the README. Part of the "problem" as you all have identified in the thread here is that it's early days for the journal and I didn't have a lot to go off of for examples.

Should I go ahead and fix things up now as the reviews come in or wait until @jhollist (and @tracykteal?) has had a chance to go through it?

Answer 15 · 2018-07-06T16:27:02.000Z

If you're going to re-structure the writing in the paper and README, it's probably a good idea to start now, so when the reviewer and managing editor come back to it, they can comment on your updated version.

Answer 16 · 2018-07-09T13:42:49.000Z

@whedon generate pdf

Answer 17 · 2018-07-09T13:42:51.000Z

Attempting PDF compilation. Reticulating splines etc...

Answer 18 · 2018-07-09T13:43:17.000Z

--> Check article proof 📄 <--

Answer 19 · 2018-07-09T13:43:20.000Z

FWIW, I've updated the README and paper.md files to reflect @labarba's comments

Answer 20 · 2018-07-25T20:13:48.000Z

All, done with my review (better late than never and apologies for taking so long). I have updated the checklist above and am providing a more detailed review in the target repo: riffomonas/reproducible_research#12

Any questions about the review or if you need me to place items here as opposed to the Riffomonas repo, let me know.

Thanks for asking me to do the review.

Answer 21 · 2018-07-26T00:25:35.000Z

Thanks so much @jhollist for doing this thorough review and the more detailed review notes. Just reading through your comments so I can come back and provide some recommendations.

JOSE reviews are discussions rather than a yes/no type of decision. Since we are just developing the pedagogy / instructional design section, that in particular is more of a discussion about suggestions. @jhollist made some nice suggestions, but @pschloss you don't necessarily have to implement all of the recommendations. It would be great to continue discussion on that issue that Jeff started.

One major point Jeff mentioned was that some of the sections had potentially a bit too much content. Given that the slide decks have accompanying video, I can see that it would be difficult to make that change however.

One point for the checklist is the Version. @pschloss it looks like there hasn't been a release yet. Before publication you can do a release and add the version.

Answer 22 · 2018-07-26T14:01:15.000Z

Was happy to do this review, @tracykteal and happy to be asked so early in the life of JOSE. I have some follow up on breaking the tutorials up but will put that over on riffomonas/reproducible_research#12

Answer 23 · 2018-08-15T15:26:20.000Z

@whedon generate pdf

Answer 24 · 2018-08-15T15:26:22.000Z

Attempting PDF compilation. Reticulating splines etc...

Answer 25 · 2018-08-15T15:26:52.000Z

--> Check article proof 📄 <--

Answer 26 · 2018-08-15T15:27:49.000Z

Thanks again for the review @jhollist and @tracykteal - I have addressed the comments over at riffomonas/reproducible_research as separate issues. I'm happy to follow up on any of the comments if you have further suggestions or comments.

Answer 27 · 2018-08-18T17:31:36.000Z

@jhollist and @tracykteal — Is this submission ready for acceptance?

Answer 28 · 2018-08-20T12:15:42.000Z

Issues have been resolved. @pschloss will create version 1.0 and then we'll get it in for acceptance.

Answer 29 · 2018-08-20T14:58:59.000Z

I concur! The issues have been resolved. I look forward to recommending this material and think it will be a great first pub for JOSE!

Answer 30 · 2018-08-24T17:13:51.000Z

Everything should be all set - let me know what else you need me to do at this point.

Thanks!

Answer 31 · 2018-08-27T13:35:17.000Z

You're all set! We'll get it published!

Answer 32 · 2018-08-27T13:35:43.000Z

Thank you so much for your submission. We're very excited to have this in JOSE.

Answer 33 · 2018-08-27T13:37:35.000Z

Sorry, one more thing. Can you create an archive (on Zenodo, figshare, or other) and post the archive DOI here. Then we link to that DOI for the publication.

Answer 34 · 2018-08-27T17:35:23.000Z

Wonderful - here you go:
https://doi.org/10.5281/zenodo.1404230

Answer 35 · 2018-08-27T19:44:42.000Z

@whedon set 10.5281/zenodo.1404230 as archive

Answer 36 · 2018-08-27T19:44:45.000Z

OK. 10.5281/zenodo.1404230 is the archive.

Answer 37 · 2018-08-27T19:50:05.000Z

@jhollist Not all review-checklist items are checked, but you commented above that you are satisfied with the revision. Can you complete the checklist?

Answer 38 · 2018-08-29T11:01:53.000Z

On vacation now with very flaky connection! Will finish this up on my return this Saturday.

Answer 39 · 2018-08-29T22:43:52.000Z

@jhollist had conveyed in the project repository that those items have been addressed, so I updated the checklist. @labarba this should be officially accepted and published. Thanks!

Answer 40 · 2018-08-30T19:01:59.000Z

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://jose.theoj.org/papers/10.21105/jose.00013/status.svg)](https://doi.org/10.21105/jose.00013)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/jose.00013">
  <img src="https://jose.theoj.org/papers/10.21105/jose.00013/status.svg" alt="DOI badge" >
</a>

This is how it will look in your documentation:

We need your help!

Journal of Open Source Education is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://jose.theoj.org/reviewer-signup.html
Making a small donation to support our running costs here: https://www.flipcause.com/secure/cause_pdetails/Mjk3ODA=

Answer 41 · 2018-08-30T19:04:18.000Z

Yay! @pschloss —Your JOSE paper is now published and you can add the lovely green badge to your repo!

Tremendous thanks to @jhollist for reviewing and @tracykteal for handling the submission. Your volunteer efforts are helping us trail-blaze in new-wave publishing 🎉

Answer 42 · 2018-08-30T20:12:32.000Z

Wonderful - thanks everyone!

Answer 43 · 2018-09-03T11:19:43.000Z

Congratulations! And thanks to @tracykteal for taking care of the checklist for me.

Enjoyed reviewing this one.