golang/go

cmd/go: provide a way to resolve an import path to a vcs, url, etc.

josharian opened this issue · 12 comments

josharian commented

'go get' contains non-trivial logic to figure out how to obtain a package based on its import path, including consulting a list of well-known hosting providers and checking for html meta tags. There is no way to get this information other than actually doing the 'go get', or maintaining a non-trivial fork of a subset of cmd/go.

I'd like a way to get this information from cmd/go. Ideally it would be a library, a la package go/build, but a command line invocation would work as well, perhaps something accepting a format flag:

$ go get -resolve '{{.ImportPath}} {{.URL}} {{.IsCustom}} {{.VCS}} {{.Root}} and probably some vcs commands...' custom.xyz/pkg

or perhaps emitting JSON, with one entry per import path.

I asked for this originally at #18119 (comment). I noted there that @rogpeppe also wants this; see https://go-review.googlesource.com/8725. Another partial copy of this bit of cmd/go can be found at https://github.com/sdboyer/gps (e.g. https://github.com/sdboyer/gps/blob/master/discovery.go).

rakyll commented

I'd love to see a go/get package a la go/build. It is a major blocker for custom tools that want to support go-get and not willing to shell out.

  • 👍7
dmitshur commented

Edit in 2023: Please be mindful that this comment was written in 2017, when GOPATH mode was only mode that existed, before module support was added in late 2018. Its advice doesn't apply as is today.

a way to resolve an import path to a vcs, url, etc.

There is no way to get this information other than actually doing the 'go get', or maintaining a non-trivial fork of a subset of cmd/go. ... Ideally it would be a library

Please correct me if I'm misunderstanding what this issue is about (I'm surprised no one has said anything by now).

This very functionality already exists, and has existed for a long time, in golang.org/x/tools/go/vcs package, under the func named RepoRootForImportPath.

For example, here's output from running vcs.RepoRootForImportPath("rsc.io/pdf/pdfpasswd", false):

&vcs.RepoRoot{
	VCS:  (*vcs.Cmd)(vcs.vcsGit),
	Repo: (string)("https://github.com/rsc/pdf"),
	Root: (string)("rsc.io/pdf"),
}

Note that it correctly resolves a vanity import path to "https://github.com/rsc/pdf", and the repo root to "rsc.io/pdf". It's the same code that cmd/go uses for go get, just copied to another library where it can be imported.

I know this as someone who has sent multiple CLs in order to keep the two in sync (people sometimes send a fix to one, but forget the other).

There's also issue #11490 (/cc @adg) that tracks a possible unification of the two identical sets of code. It wasn't possible before because it was hard to have something both in cmd/go and another library. But I believe that may be changing as of #18653.

  • ❤️2
sdboyer commented

Just seeing this - thanks @shurcooL for the pointer to it.

I'd also like very much to see a package like this. In fact, that's kinda what i wrote gps to be. Though, obviously, it can't be that in any official way unless/until dep makes it into the toolchain.

bcmills commented

This very functionality already exists, and has existed for a long time, in golang.org/x/tools/go/vcs package, under the func named RepoRootForImportPath.

I would rather consolidate this functionality into go list than maintain a fork in x/tools.

We already resolve new modules in some go list commands (for example, go list -m -versions), so it seems relatively harmless to add origin information there too.

  • 👍5
mvdan commented

As per #18387 (comment), do we all agree that it's okay for go list -m to gain a way to show the VCS type and URL? Intuitively, I imagine that could be some new fields in go list -m -json, but I don't know if it would need to be opted into via some flag or require extra work.

I would love for us to converge around a specific design, so we can look at implementing it for 1.18 once the tree reopens.

Also, I'm moving this thread out of the proposal milestone, because it is not a proposal as far as I can tell.

bcmills commented

This probably requires #44742. If and when that is addressed, we can separately consider adding the metadata to go list.

mvdan commented

Interesting: I can get the @v/$version.info file from a module, which includes VCS information, but there's no easy way for me to tell what the current VCS information is. The closest I can use is @v/latest.info, but that may lag behind if the latest semver tag happened weeks or months ago.

Similar to how we have @v/list, perhaps there could be @v/info. Otherwise the new command described in #18387 (comment) give out of date information.

  • 👍1
hyangah commented

Can someone summarize what is the remaining work to close this?

With go list:

@dmitshur wrote a detailed instruction in #57051

$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }

I wish I knew an easy way to resolve the mapping from a package path to its module path (ideally without creating a work module). However, that's doable.

With module proxy api:

If you know the module path and the module proxy fetched the latest version after go 1.19, you can access the similar info from /@latest endpoint.

https://proxy.golang.org/<the_module_path>/@latest
bcmills commented

I believe that recipe only works for module paths, whereas this issue is requesting a mechanism for package import paths.

I think it is still true that “[t]here is no way to get this information other than actually doing the 'go get', or maintaining a non-trivial fork of a subset of cmd/go”, although what's left is probably a smaller change in go list (to add support for package queries) or similar.

jeffwidman commented

While this issue is clearly asking for "import path" which maps to go-import tag, I suspect some folks subscribed to this are actually looking for the "source code URL" which may be mapped to a different location via the go-source tag:

andydotxyz commented

While this issue is clearly asking for "import path" which maps to go-import tag, I suspect some folks subscribed to this are actually looking for the "source code URL" which may be mapped to a different location via the go-source tag:

That's the case for what I am trying to do. The deprecated library suggests a go list which doesn't actually sort the use-case for what is the source root for a named package.

  • 👍2
andydotxyz commented

just to add a little output to summarise the challenge I am facing - given an import (/main) path the proposed solution won't work:

><> go list -m -f '{{.Origin}}' fyne.io/fyne/v2/cmd/fyne@latest
go: module fyne.io/fyne/v2/cmd/fyne: no matching versions for query "latest"

It only maps at the module root