Masterminds/glide

Issue with glide dependencies containing local submodules

Opened this issue · 5 comments

clarsonneur commented

Hi glide team!

I found something about glide dependencies detection, which is, from my point of view, annoying.

I have 2 git repositories cloned in $GOPATH/src.

  1. glide-submodule-demo containing main.go
  2. subparentdemo containing a parent.go, refering a local go subpackage subdemo and the subdemo subdir with the lib.go

So shortly, in term of import, I have:
main.go: In glide-submodule-demo (repo1)

package main

import "github.com/clarsonneur/subparentdemo"

[...]

parent.go: In subparentdemo (repo2)

package subparentdemo

import "subparentdemo/subdemo"

[...]

and in lib.go: subparentdemo/subdemo (repo2)

package subdemo

[...]

If I do a glide up from glide-submodule-demo, I got the following:

$ glide up
[WARN]  The name listed in the config file (github.com/clarsonneur/glide-submodule-demo) does not match the current location (test)
[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching github.com/clarsonneur/subparentdemo.
[INFO]  Resolving imports
[INFO]  --> Fetching subparentdemo/subdemo.
[WARN]  Unable to checkout subparentdemo/subdemo
[INFO]  Downloading dependencies. Please wait...
[INFO]  Setting references for remaining imports
[ERROR] Failed to set version on subparentdemo/subdemo to : Cannot detect VCS
[ERROR] Failed to set references: Cannot detect VCS (Skip to cleanup)
[INFO]  Exporting resolved dependencies...
[INFO]  --> Exporting github.com/clarsonneur/subparentdemo
[ERROR] Cannot detect VCS

The issue is easily reproducible from https://github.com/clarsonneur/glide-submodule-demo

glide detects subparentdemo/subdemo dependency, and try to fetch it. But it should not, because it has already been fetched by subparentdemo. So the subdir is there.

To fix this issue, I must update parent.go import as follow:

import "subparentdemo/subdemo" => import "github.com/clarsonneur/subparentdemo/subdemo"

The thing which is annoying me here with this fix, is when I build/test the subparentdemo module, I must have github.com/clarsonneur/subparentdemo in GOPATH or in vendor (with glide if I want...)
So, there is 2 different copy of the same project that I'm developping.
The fix I uses, is to create a link like that:
$GOPATH/src/github.com/clarsonneur/subparentdemo -> $GOPATH/src/subparentdemo

Not really nice. but it works.

I like notion of sub-modules in glide, which enhance gracefully go development.
And using a local import reference is really nice in go.
But those 2 features applied, fails glide.

From my point of view, I believe glide should take care of that as well in dependencies detection.
But if there is another way to structure my dependencies, I'm opened on that as well!

Any hint? Fixes?

Thank you

mh-cbon commented

yes, i agree, i experience a similar error, glide is stuck now.

$ go env
GOARCH="amd64"
GOBIN="/home/mh-cbon/gow/bin"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/mh-cbon/gow"
GORACE=""
GOROOT="/home/mh-cbon/.gvm/gos/go1.8"
GOTOOLDIR="/home/mh-cbon/.gvm/gos/go1.8/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build865382226=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
$ go version
go version go1.8 linux/amd64

It won t fetch github.com/gorilla/websocket,
because it failed to fetch a local package.

$ pwd
/home/mh-cbon/gow/src/github.com/mh-cbon/rclone-parser
$ pwd
/home/mh-cbon/gow/src/github.com/mh-cbon/backup
$ cat glide.yaml 
package: github.com/mh-cbon/backup
import:
- package: github.com/gorilla/mux
  version: ^1.3.0
- package: github.com/mh-cbon/gssc
  version: ^0.0.1
- package: github.com/mitchellh/go-homedir
- package: github.com/gorilla/websocket
  version: ^1.1.0
$ cd app
$ go list -f '{{join .Deps "\n"}}' | grep rclo
github.com/mh-cbon/rclone-parser/parser
github.com/mh-cbon/rclone-parser/rclone
$ cd ..
$ go run main.go 
api/tasks.go:6:2: cannot find package "github.com/gorilla/websocket" in any of:
	/home/mh-cbon/gow/src/github.com/mh-cbon/backup/vendor/github.com/gorilla/websocket (vendor tree)
	/home/mh-cbon/.gvm/gos/go1.8/src/github.com/gorilla/websocket (from $GOROOT)
	/home/mh-cbon/gow/src/github.com/gorilla/websocket (from $GOPATH)
$ glide up
[INFO]	Downloading dependencies. Please wait...
[INFO]	--> Fetching updates for github.com/gorilla/mux.
[INFO]	--> Fetching updates for github.com/mh-cbon/gssc.
[INFO]	--> Fetching updates for github.com/mitchellh/go-homedir.
[INFO]	--> Fetching updates for github.com/gorilla/websocket.
[INFO]	--> Detected semantic version. Setting version for github.com/mh-cbon/gssc to 0.0.1.
[INFO]	--> Detected semantic version. Setting version for github.com/gorilla/websocket to v1.1.0.
[INFO]	--> Detected semantic version. Setting version for github.com/gorilla/mux to v1.3.0.
[INFO]	Resolving imports
[INFO]	--> Fetching updates for github.com/gorilla/context.
[INFO]	--> Fetching updates for github.com/pkg/errors.
[INFO]	--> Detected semantic version. Setting version for github.com/pkg/errors to v0.8.0.
[INFO]	--> Fetching github.com/mh-cbon/disksinfo.
[WARN]	Unable to checkout github.com/mh-cbon/disksinfo
[ERROR]	Error looking for github.com/mh-cbon/disksinfo/diskinfo: Unable to get repository
[INFO]	--> Fetching github.com/mh-cbon/rclone-parser.
[WARN]	Unable to checkout github.com/mh-cbon/rclone-parser
[ERROR]	Error looking for github.com/mh-cbon/rclone-parser/parser: Unable to get repository
[WARN]	Unable to set version on github.com/mh-cbon/rclone-parser to . Err: Unable to retrieve checked out version
[ERROR]	Error scanning github.com/mh-cbon/rclone-parser/rclone: open /home/mh-cbon/.glide/cache/src/https-github.com-mh-cbon-rclone-parser/rclone: no such file or directory
[ERROR]	This error means the referenced package was not found.
[ERROR]	Missing file or directory errors usually occur when multiple packages
[ERROR]	share a common dependency and the first reference encountered by the scanner
[ERROR]	sets the version to one that does not contain a subpackage needed required
[ERROR]	by another package that uses the shared dependency. Try setting a
[ERROR]	version in your glide.yaml that works for all packages that share this
[ERROR]	dependency.
[ERROR]	Failed to retrieve a list of dependencies: Error resolving imports
$ cd ../rclone-parser/
$ git remote -v
$
$ cd ../backup/
$ glide install --force
[WARN]	Lock file may be out of date. Hash check of YAML failed. You may need to run 'update'
[INFO]	Downloading dependencies. Please wait...
[INFO]	--> Found desired version locally github.com/gorilla/context 08b5f424b9271eedf6f9f0ce86cb9396ed337a42!
[INFO]	--> Found desired version locally github.com/gorilla/mux 392c28fe23e1c45ddba891b0320b3b5df220beea!
[INFO]	--> Found desired version locally github.com/mh-cbon/gssc 62a18b7291f080081e4dff8dd694ac07d16a18b4!
[INFO]	--> Found desired version locally github.com/mitchellh/go-homedir b8bc1bf767474819792c23f32d8286a45736f1c6!
[INFO]	--> Found desired version locally github.com/pkg/errors 645ef00459ed84a119197bfb8d8205042c6df63d!
[INFO]	Setting references.
[INFO]	--> Setting version for github.com/gorilla/mux to 392c28fe23e1c45ddba891b0320b3b5df220beea.
[INFO]	--> Setting version for github.com/mh-cbon/gssc to 62a18b7291f080081e4dff8dd694ac07d16a18b4.
[INFO]	--> Setting version for github.com/mitchellh/go-homedir to b8bc1bf767474819792c23f32d8286a45736f1c6.
[INFO]	--> Setting version for github.com/gorilla/context to 08b5f424b9271eedf6f9f0ce86cb9396ed337a42.
[INFO]	--> Setting version for github.com/pkg/errors to 645ef00459ed84a119197bfb8d8205042c6df63d.
[INFO]	Exporting resolved dependencies...
[INFO]	--> Exporting github.com/pkg/errors
[INFO]	--> Exporting github.com/gorilla/mux
[INFO]	--> Exporting github.com/mitchellh/go-homedir
[INFO]	--> Exporting github.com/mh-cbon/gssc
[INFO]	--> Exporting github.com/gorilla/context
[INFO]	Replacing existing vendor dependencies
$ go run main.go 
api/tasks.go:6:2: cannot find package "github.com/gorilla/websocket" in any of:
	/home/mh-cbon/gow/src/github.com/mh-cbon/backup/vendor/github.com/gorilla/websocket (vendor tree)
	/home/mh-cbon/.gvm/gos/go1.8/src/github.com/gorilla/websocket (from $GOROOT)
	/home/mh-cbon/gow/src/github.com/gorilla/websocket (from $GOPATH)

Note also, if i do glide up --quick, it will fail on github.com/pkg/errors

$ glide up --quick
[INFO]	Downloading dependencies. Please wait...
[INFO]	--> Fetching updates for github.com/gorilla/mux.
[INFO]	--> Fetching updates for github.com/mitchellh/go-homedir.
[INFO]	--> Fetching updates for github.com/mh-cbon/gssc.
[INFO]	--> Fetching updates for github.com/gorilla/websocket.
[INFO]	--> Detected semantic version. Setting version for github.com/mh-cbon/gssc to 0.0.1.
[INFO]	--> Detected semantic version. Setting version for github.com/gorilla/websocket to v1.1.0.
[INFO]	--> Detected semantic version. Setting version for github.com/gorilla/mux to v1.3.0.
[INFO]	Exporting resolved dependencies...
[INFO]	--> Exporting github.com/mh-cbon/gssc
[INFO]	--> Exporting github.com/gorilla/websocket
[INFO]	--> Exporting github.com/mitchellh/go-homedir
[INFO]	--> Exporting github.com/gorilla/mux
[INFO]	Replacing existing vendor dependencies
[WARN]	Skipping lockfile generation because full dependency tree is not being calculated
 $ go run main.go
vendor/github.com/mh-cbon/gssc/gssc.go:11:2: cannot find package "github.com/pkg/errors" in any of:
	/home/mh-cbon/gow/src/github.com/mh-cbon/backup/vendor/github.com/pkg/errors (vendor tree)
	/home/mh-cbon/.gvm/gos/go1.8/src/github.com/pkg/errors (from $GOROOT)
	/home/mh-cbon/gow/src/github.com/pkg/errors (from $GOPATH)
$ ll vendor/github.com/gorilla/
total 8
drwxr-xr-x 2 mh-cbon mh-cbon 4096 10 mars  01:17 mux
drwxr-xr-x 3 mh-cbon mh-cbon 4096 10 mars  01:17 websocket
mh-cbon commented

meanwhile, i added those dep to ignore key,

ignore:
  - github.com/mh-cbon/rclone-parser
  - github.com/mh-cbon/disksinfo/diskinfo
  • 👍8
  • 😄1
  • 🎉1
  • ❤️1
clarsonneur commented

Ha, cool. I didn't know I can add those 'ignore' tags. thank you! I will try that.

sh0umik commented

Ignore key solves the problem if you are compiling locally . But if you are using docker then the local package needs to be in the vendor directory . Struggling with that .

mh-cbon commented

imu, when you do docker things, it should be against a finished package.
I mean package with a fixed set of dependencies.
In our case we are dealing with unfinished sets as the version of the packages we want to work with exist only in the workspace (not in vendor).

So in your case, in your docker, when you do glide up, all dependencies must exist with a version and a reachable remote.