Shallow git clones can fail on refs

[mikepurvis]

For a lot of our upstream dependencies, we have locked ourselves to specific hash versions, especially if we've needed to fork from a tag and patch on a fix.

Unfortunately with vcs import --shallow, a hash version can error out. It would be nice if vcs had fallback logic to detect this case and retry with a full clone, and then after that a git fetch --all if the ref still can't be found.

[dirk-thomas]

Please provide reproducible steps.

[mikepurvis]

Okay, so having looked a bit further, it seems this is git host dependent. For example, the following succeeds with Github and a public mirror that I created just now on GitLab.com:

repositories:
  ros_comm:
    type: git
    url: http://github.com/ros/ros_comm.git
    #url: https://gitlab.com/mikepurvis/ros_comm.git
    #url: http://gitlab.clearpathrobotics.com/forks/ros_comm.git
    version: 133b510d79cd4702b34c14cbfc18e00a0d8e0d51

However, it fails with --shallow on our internal GitLab instance, which is version 11.5.5, released early 2019:

$ vcs import --input repos.txt --shallow
=== ./ros_comm (git) ===
error: Server does not allow request for unadvertised object 133b510d79cd4702b34c14cbfc18e00a0d8e0d51

I don't know what the story here would be for gitea, git-over-ssh, etc. But because it does actually work in non-shallow mode, I think probably the only change I could reasonably ask for here would be a fallback from shallow to full clone on this specific failure.

The further fallback to doing git fetch --all would only be necessary for some extremely niche cases, such as if you were referencing a commit that was only reachable by a hidden ref that wasn't a branch or tag. So I don't think that's worth implementing; anyone using hidden refs in their repos should know what they're getting themselves into.

[dirk-thomas]

it fails with --shallow on our internal GitLab instance, which is version 11.5.5, released early 2019:

Can you check if updating to a newer version would resolve the problem? Maybe GitLab support can confirm this?

I think probably the only change I could reasonably ask for here would be a fallback from shallow to full clone on this specific failure.

I assume you use this in an automated process? Otherwise I would suggest the user just calls the command again without --shallow.

[mikepurvis]

It does look like some later GitLab version has added support for this, based on their hosted instance doing the right thing with it. So it's probably fine to just close this, and we'll get updated on our side. If a user comes along who wants to use gitea or something else, this can be reopened at that point.

Partly it's automation and also just a nuisance as a user to track down which repos failed in the giant list— if you're cloning hundreds of repos, it's not at all obvious which ones are the problem when vcs dumps all the output from all of them.

[dirk-thomas]

It does look like some later GitLab version has added support for this, based on their hosted instance doing the right thing with it. So it's probably fine to just close this, and we'll get updated on our side. If a user comes along who wants to use gitea or something else, this can be reopened at that point.

Sounds good to me. Thanks for checking.