benmarwick/rrtools

Currently lacks support for GitLab repositories

ntrlshrp opened this issue · 10 comments

ntrlshrp commented

I'm curious to see if others think this would be valuable to this package. Thanks!

Is your feature request related to a problem? Please describe.
I want to use rrtools and host my remote repository on GitLab, not GitHub. Currently, there are no convenience functions to make this happen (nor to change all instances of github.com to gitlab.com (e.g., in the README.Rmd)).

Describe the solution you'd like
Offer users a convenience function, rrtools::use_gitlab that is analogous to usethis::use_github(credentials = git2r::cred_ssh_key(), auth_token = "xxxx", protocol = "https", private = FALSE) doing the following:

  • git push --set-upstream
  • git remote add origin
  • set up .gitlab-ci.yml (GitLab does their own Continuous Integration and Continuous Development (CI/CD))
  • use default Dockerfile in that .gitlab-ci.yml (so you can reproduce the outputs using docker rather than gitlab-runner)
  • update the ~dozen lines that refer to GitHub or github.com

Outside this convenience function, other changes may be necessary:

  • update the rrtools::use_travis() function to alert the user to possible duplication and allow an off-ramp if GitLab CI/CD already covers the user's needs
  • update the rrtools::create_compendium() function to ask if GitHub or GitLab
  • update the README.Rmd on rrtools to alert people that you can do rrtools::use_gitlab() to do in a single step what is currently represented as steps (3b) usethis::use_github, (6) rrtools::use_dockerfile(), and (7) rrtools::use_travis()

Describe alternatives you've considered
I could attempt to request this feature in the usethis package rather than here, i.e., usethis::use_gitlab. My sense, however, is that this is fairly simple to start a new repository by simply calling some system() functions although, at the same time, I see that usethis::use_github is ~70 lines with at least 6 checks that the function can appropriately succeed.

Also, if rrtools uses the hypothetical usethis::use_gitlab(), it may be difficult to wrap into a single step what is currently represented as steps (3b) usethis::use_github, (6) rrtools::use_dockerfile(), and (7) rrtools::use_travis().

Additional context
NA

benmarwick commented

Thanks, yes, we've previous considered this in #69, did you take a look at that? Checking again just now, I see that the developers of usethis are not interested to support gitlab at the moment: r-lib/usethis#567

ntrlshrp commented

Thanks, @benmarwick, for alerting me to the very similar #69. Nonetheless, that issue seemed to be resolved by a solution that does not resolve the issue I've posted here.

Solution to #69 [bold added by @ntrlshrp ]:

In the meantime, my suggestion to integrate rrtools with GitLab would be to use git commit/push/pull from the terminal in RStudio (rather than the R code functions from usethis), and set the git repository URL to your gitlab repo rather than your github repo.

The main difference at the setup will be to skip [(3b)] usethis::use_github(auth_token = "xxxx", protocol = "https", private = FALSE) and instead go to the terminal to set the remote to your gitlab URL and commit/push/pull from the terminal.

My new issue #91:

I would like the convenience function, rrtools::use_gitlab, to instead:

  • (a): automatically take the user all the way to demonstrably reproducible research, i.e.,

wrap into a single step what is currently represented as steps (3b) usethis::use_github, (6) rrtools::use_dockerfile(), and (7) rrtools::use_travis().

  • (b): automatically help the user point colleagues to the correct repository, i.e.,

update the ~dozen lines that refer to GitHub or github.com

This convenience function strikes me as easy and allows maximum benefit from rrtools usage with minimum user resources (i.e., time, decisionmaking, recoding, etc. within and across different services).

benmarwick commented

Yes, it would be useful to see this function you propose appear in your fork where we can explore and test it. Given the challenges that the usethis developers had with the GitHub functions, I wonder it may take some time to get functions for GitLab just right.

Your proposal (a) may need further discussion. We decided with create_compendium to capture only the first 5 steps because very few people are using the subsequent steps (docker, travis, tests), and they usually can't fit into a short workshop on rrtools.

So it might be more consistent with our current approach to have an argument in those docker and travis functions that has a default value of GitHub, which the user can change to GitLab.

ntrlshrp commented

Finally with some free time, I've pushed an MWE of rrtools::use_gitlab() to my fork where others can explore and test.

Recipe
Part 1

  1. Make sure you can use feature/gitlab branch here: https://github.com/ntrlshrp/rrtools/tree/feature/gitlab
  2. Run rrtools::create_compendium("myNewPackage")
  3. [Now in myNewPackage project] Run `token <- readLines("path/location/of/gitlab/token")
  4. Run rrtools::use_gitlab(auth_token = token)
    1. Say yes to "Overwrite README.Rmd", "Overwrite CONDUCT.md", and "Overwrite CONTRIBUTING.md"

Part 2

  1. Commit everything and push.

GitLab will run two CI/CD jobs, check-package and render-paper. These should both succeed. The user has now produced a reproducible paper.html!

Part 3

  1. Add a dummy function to R/ (e.g., hello()) [so that coverage report works]
    • Don't forget to document it (e.g., devtools::load_all() and devtools::document)
  2. Change myNewPackage/.gitlab-ci.yml so that lines 8, 10, and 12 say "yes" instead of "no"
  3. Commit everything and push.

GitLab will run five CI/CD jobs, the two jobs above plus pages, push-image-to-registry, and code-coverage-report. These should all succeed and produce the following outputs, respectively.

Outputs

I'm curious to see if these changes work for those who primarily use github and, more generally, on other machines. There is no rigorous testing of the changes yet, simply an MWE to get things off the ground.

Details
Part 1
rrtools has now:

  • pushed the rrtools' "Initial commit" of myNewPackage to gitlab.com/username/pkgname.git master
    • username is the value from git config user.name in Terminal and pkgname comes from DESCRIPTION file, line 1
  • added .gitlab-ci.yml to myNewPackage that, once pushed to gitlab.com, will:
    • run devtools::check() within Docker image rocker/{{{rocker}}}:{{{r_version}}}
    • render paper.html using Dockerfile
  • adjusted CONTRIBUTING.md and README.Rmd/md to point to gitlab.com instead of github.com
  • added standard Dockerfile to myNewPackage that can make a Docker container using image rocker/{{{rocker}}}:{{{r_version}}}
  • done minor stuff:
    • added analysis, ^\.gitlab-ci\.yml$, and Dockerfile to .Rbuildignore so the R package builds correctly
    • added analysis/ for template paper
    • deleted .Rprofile that rrtools wrote
nevrome commented

When I tried to test this workflow I got stuck because my git user name does not equal my gitlab user name.

username <- system("git config user.name", intern = TRUE)

is not a sufficient solution, I guess. usethis solves this problem for use_github() by getting the user name from the API (git_credentials(protocol, auth_token)). Maybe you could use something like this as well? Maybe gitlabr could help you solve this.

Maybe you could also add some documentation for use_gitlab mirroring the documentation of usethis::use_github. I consider it especially important to add a sentence about the settings of the API token. I always wonder which checkboxes to click.

ntrlshrp commented

Thanks, @nevrome ! This is awesome. I don't have time presently, but will attempt to resolve this issue %1 where gitlab username does not equal the value returned by git config user.name. I will also attempt documentation similar to usethis::use_github.

  • 👍1
  • 🚀1
ntrlshrp commented

In response to @nevrome, who pointed out a simple, but critical bug (and the need for more documentation), I've whipped up a new branch on my fork, feature/gitlab-02.

Recipe
Part 1

  1. Run remotes::install_github('ntrlshrp/rrtools', ref = 'feature/gitlab-02')
  2. Run rrtools::create_compendium("myNewPackage")
  3. [Now in myNewPackage project] Run rrtools::use_gitlab(auth_token = readLines("path/location/of/gitlab/personal-access-token"))
    • For help setting up your personal access token, try ?rrtools::use_gitlab
    • Note: use_gitlab() now interactively requests your GitLab LOGIN ("GitLab username", NOT your full name). For programmatic deployment, you can set options(gitlab.username = '<GITLAB_USER_LOGIN>')

Part 2

  1. Commit everything and push.

GitLab will run two CI/CD jobs, check-package and render-paper. These should both succeed. The user has now produced a reproducible paper.docx!

Part 3 is as above.

NOTES:

  • Use case (1): user creates new compendium, immediately runs rrtools::use_gitlab()
    • Checks:
      • did user set auth_token?
      • did user run use_gitlab() inside a git repo?
      • did user already associate an origin remote with this local repo?
  • rrtools::use_gitlab() alters README to be GitLab specific (this needs refinement before deployment, e.g., Binder link still says gh instead of gl, but shows the method I have in mind to accomplish this alteration).
  • I've created a .gitlab-ci.yml for the rrtools package as a sort of external unit test for this (and other) use cases and scenarios for rrtools::use_gitlab().
    • E.g., if this rrtools branch is on a GitLab project (and you've added your personal access token as a variable, RRTOOLS_TOKEN, to this project) and you trigger a CI/CD pipeline (e.g., Click "Edit" on .gitlab-ci.yml and then click "Commit"), the pipeline runs a rocker/verse container, installs an rrtools branch that includes use_gitlab(), creates a new compendium as if it were a user, and runs rrtools::use_gitlab() within that compendium. This associates the local repo (inside rocker/verse) with a new remote repo on GitLab.com, https://gitlab.com/$GITLAB_USER_LOGIN/newRRtoolsCompendium.
      • It then adds all files, commits, and pushes to this GitLab.com remote, which triggers (on the remote repo) the .gitlab-ci.yml derived from the rrtools/inst/gitlab-ci.yml template. This runs two CI/CD jobs (see Part 2 above), check-package and render-paper. These should both succeed. The user has now produced a reproducible paper.docx!
      • It also checks the package on three versions of rocker/verse (which probably makes more sense in the .travis.yml).
  • *** I have not automated checking the other three CI/CD jobs: pages, push-image-to-registry, and code-coverage-report.
  • *** I have not automated the "Checks" above to see that they work properly.

Feedback:

  • What other unit tests or use cases need development?

I hope this is helpful. Thanks.

PS: Running Parts 1 and 2 inside a Docker container (replace <GITLAB_USER_LOGIN> and <your-personal-access-token> appropriately).

Dockerfile

FROM rocker/verse:3.5.3
RUN . /etc/environment \
    && R -e "remotes::install_github('ntrlshrp/rrtools', ref = 'feature/gitlab-02')" \
    && cd / \
    && git config --global user.name "MyGitUSERNAME" \
    && git config --global user.email "MyGitUSEREMAIL" \
    && R -e "rrtools::create_compendium('newRRtoolsCompendiumDocker')" \
    && cd newRRtoolsCompendiumDocker \
    && R -e "options(gitlab.username = '<GITLAB_USER_LOGIN>'); rrtools::use_gitlab(auth_token = '<your-personal-access-token>')" \
    && git add . \
    && git commit -m "Added all new files since initial commit" \
    && git push https://oauth2:<your-personal-access-token>@gitlab.com/<GITLAB_USER_LOGIN>/newRRtoolsCompendiumDocker.git
nevrome commented

Thank you for your patience, @ntrlshrp. I finally came around to test this fixed implementation. Everything seems to work and the documentation is fine. You did an excellent job here and I'm sorry that it took my so long to come back to this.

What do you think @benmarwick?

I suggest you, @ntrlshrp, create PR to the current rrtools master and we review it formally. You don't have to add any message in the PR, just a link to this issue. If more tests and features are required it will become clear there.

I'm for example not sure if these backup .bak-files (?) are necessary. I also get the warning

Warning messages:
1: In file.rename("CONDUCT.md", "CONDUCT.md.bak") :
  cannot rename file 'CONDUCT.md' to 'CONDUCT.md.bak', reason 'No such file or directory'
2: In file.rename("CONTRIBUTING.md", "CONTRIBUTING.md.bak") :
  cannot rename file 'CONTRIBUTING.md' to 'CONTRIBUTING.md.bak', reason 'No such file or directory'

when I apply use_gitlab() in a new repo. Probably this backup creation should be wrapped in if (file.exists(...)) {}. But these are minor details we can easily discuss/add later.

ntrlshrp commented

No need for apologies, @nevrome , glad to hear that it looks ready for a more formal review. Please see the suggested PR. Thanks again!

Also, your solution to the backup .bak files sounds ideal. My use case started with rrtools::create_compendium() but if the user has not directly or indirectly run the rrtools::use_readme_rmd() function before rrtools::use_gitlab(), the CONDUCT and CONTRIBUTING files likely aren't there and thus don't need backing up. In any event, I'll wait for more issues to pop out of the formal review before making new commits. Thanks again!

  • 👍1
benmarwick commented

Closing due to lack of activity.