cmd/go: using golang.org/x/* from mainland China is awkward with modules
hellojukay opened this issue · 46 comments
What version of Go are you using (go version
)?
[jukay@mac-pro]~/log$ go version go version go1.11 darwin/amd64 [jukay@mac-pro]~/log$
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env
)?
go env
Output
[jukay@mac-pro]~/log$ go env GOARCH="amd64" GOBIN="" GOCACHE="/Users/jukay/Library/Caches/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="darwin" GOOS="darwin" GOPATH="/Users/jukay/Code/Go" GOPROXY="" GORACE="" GOROOT="/usr/local/Cellar/go/1.11/libexec" GOTMPDIR="" GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64" GCCGO="gccgo" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="/Users/jukay/log/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build712144984=/tmp/go-build -gno-record-gcc-switches -fno-common" [jukay@mac-pro]~/log$
What did you do?
just compile this project outside GOPATH
[jukay@mac-pro]~/log$ go build
What did you expect to see?
[jukay@mac-pro]~/log$ time go build
real 0m0.316s
user 0m0.195s
sys 0m0.232s
What did you see instead?
[jukay@mac-pro]~/log$ time go build
real 1m0.316s
user 0m0.195s
sys 0m0.232s
Unable to understand such a simple code, it took a minute to compile it.
when i usego build -v
,it not use cache.
[jukay@mac-pro]~/log$ go build -v
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
If the cache isn't warm, the first build may take longer if it needs to download and build the dependencies. Have you tried running time go build
a second time? It should take a few seconds at most.
@mvdan I run time go build
many times, it cast much time every time.
Is your GOCACHE directory /Users/jukay/Library/Caches/go-build
writeable? Does it contain anything? What does go build -x
show?
Also, note that your go build -v
above shows HTTP errors. Are the build succeeding or failing? If the build fails because it can't download some modules or dependencies, they will of course not be cached.
Doesn't china block URLs with x
in them? Someone mentioned it on go-nuts.
@mvdan go build
alway success, it just very slow。i can go mod vendor
,it create vendor
directory,and download all dependencies. this is my source code
https://github.com/hellojukay/vgo-issue
Hi @hellojukay - to help us investigate the issue faster, it would be great if you can answer the questions asked above. Especially -
What does your GOCACHE
directory contain ?
What is the output of go build -x -v
?
Thank you.
@agnivade yes ,yestoday i try to upgrade my golang version to 1.11.2
, this is output of go build -x -v
[jukay@mac-pro]~/log$ go build -x -v
WORK=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build601851662
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
mkdir -p $WORK/b001/
cat >$WORK/b001/importcfg.link << 'EOF' # internal
packagefile main=/Users/jukay/Library/Caches/go-build/27/27043f87ca77211db1b891dd2bc2209ebd069dfe894370d2782c42512cb35969-d
packagefile github.com/sirupsen/logrus=/Users/jukay/Library/Caches/go-build/e7/e7275b538c9195808fa163ca3c56c5d63281712d3d135165cce1aac98cc02bf8-d
packagefile runtime=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime.a
packagefile bufio=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/bufio.a
packagefile bytes=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/bytes.a
packagefile encoding/json=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/json.a
packagefile fmt=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/fmt.a
packagefile golang.org/x/crypto/ssh/terminal=/Users/jukay/Library/Caches/go-build/0e/0ebfbc4e93e20cd471ded2ecdc73841dc36970144e7957043cdc75f65e0099f8-d
packagefile io=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/io.a
packagefile log=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/log.a
packagefile os=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/os.a
packagefile reflect=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/reflect.a
packagefile sort=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sort.a
packagefile strings=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/strings.a
packagefile sync=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sync.a
packagefile sync/atomic=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/sync/atomic.a
packagefile time=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/time.a
packagefile internal/bytealg=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/bytealg.a
packagefile internal/cpu=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/cpu.a
packagefile runtime/internal/atomic=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime/internal/atomic.a
packagefile runtime/internal/sys=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/runtime/internal/sys.a
packagefile errors=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/errors.a
packagefile unicode/utf8=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode/utf8.a
packagefile unicode=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode.a
packagefile encoding=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding.a
packagefile encoding/base64=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/base64.a
packagefile math=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/math.a
packagefile strconv=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/strconv.a
packagefile unicode/utf16=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/unicode/utf16.a
packagefile golang.org/x/sys/unix=/Users/jukay/Library/Caches/go-build/7f/7f9284a7bb944dc9533f22fbc3ba192e51145ab0792f17b55a151eb6411981f5-d
packagefile internal/poll=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/poll.a
packagefile internal/syscall/unix=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/syscall/unix.a
packagefile internal/testlog=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/testlog.a
packagefile syscall=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/syscall.a
packagefile internal/race=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/internal/race.a
packagefile encoding/binary=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/encoding/binary.a
packagefile math/bits=/usr/local/Cellar/go/1.11.2/libexec/pkg/darwin_amd64/math/bits.a
EOF
mkdir -p $WORK/b001/exe/
cd .
/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64/link -o $WORK/b001/exe/a.out -importcfg $WORK/b001/importcfg.link -buildmode=exe -buildid=7l5ZuOc-Ol-x9AyQCky7/WlX-HbvTd8yVTtnfYvzf/J45TO0-nMm2qI1TaHJKq/7l5ZuOc-Ol-x9AyQCky7 -extld=clang /Users/jukay/Library/Caches/go-build/27/27043f87ca77211db1b891dd2bc2209ebd069dfe894370d2782c42512cb35969-d
/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64/buildid -w $WORK/b001/exe/a.out # internal
mv $WORK/b001/exe/a.out main
rm -r $WORK/b001/
This is my GOCACHE
directory
[jukay@mac-pro]~/log$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/jukay/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/jukay/Code/Go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/jukay/log/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/yp/_44_30ts2_xf43zpz2j72k200000gn/T/go-build984680892=/tmp/go-build -gno-record-gcc-switches -fno-common"
[jukay@mac-pro]~/log$ cd /Users/jukay/Library/Caches/go-build
[jukay@mac-pro]~/Library/Caches/go-build$ ls
00 14 28 3c 50 64 78 8c README b3 c7 db ef
01 15 29 3d 51 65 79 8d a0 b4 c8 dc f0
02 16 2a 3e 52 66 7a 8e a1 b5 c9 dd f1
03 17 2b 3f 53 67 7b 8f a2 b6 ca de f2
04 18 2c 40 54 68 7c 90 a3 b7 cb df f3
05 19 2d 41 55 69 7d 91 a4 b8 cc e0 f4
06 1a 2e 42 56 6a 7e 92 a5 b9 cd e1 f5
07 1b 2f 43 57 6b 7f 93 a6 ba ce e2 f6
08 1c 30 44 58 6c 80 94 a7 bb cf e3 f7
09 1d 31 45 59 6d 81 95 a8 bc d0 e4 f8
0a 1e 32 46 5a 6e 82 96 a9 bd d1 e5 f9
0b 1f 33 47 5b 6f 83 97 aa be d2 e6 fa
0c 20 34 48 5c 70 84 98 ab bf d3 e7 fb
0d 21 35 49 5d 71 85 99 ac c0 d4 e8 fc
0e 22 36 4a 5e 72 86 9a ad c1 d5 e9 fd
0f 23 37 4b 5f 73 87 9b ae c2 d6 ea fe
10 24 38 4c 60 74 88 9c af c3 d7 eb ff
11 25 39 4d 61 75 89 9d b0 c4 d8 ec log.txt
12 26 3a 4e 62 76 8a 9e b1 c5 d9 ed trim.txt
13 27 3b 4f 63 77 8b 9f b2 c6 da ee
[jukay@mac-pro]~/Library/Caches/go-build$
[jukay@mac-pro]~/Library/Caches/go-build/27$ ls
27043f87ca77211db1b891dd2bc2209ebd069dfe894370d2782c42512cb35969-d
275c760696e3a1f52ee8966821a649f4ee3a720eb6a37639cb7259c82b53825f-a
27afd4f6c43d566036c18b19d4eb17b801039ebb1692781323ce1274baaebbfb-a
27c160e9dd5b2354628a8bb5a4a70aaac2197a1d70ab57c66e8c8e2287bbb3e2-a
[jukay@mac-pro]~/Library/Caches/go-build/27$
After upgrade go go version go1.11.2 darwin/amd64
,
[jukay@mac-pro]~/log$ time go build
real 1m0.434s
user 0m0.212s
sys 0m0.272s
Even if I use go mod edit replace
, vgo alway try to connect to golang.org , when i run go build,until the connection is time out。it didn't seem to use the cache。
There seems to be various oddities-
-
The build tries to connect to https://golang.org/x/crypto and https://golang.org/x/crypto everytime.
-
Your go.mod seems to replace
replace golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33 => github.com/golang/sys v0.0.0-20180905080454-ebe1bf3edb33
Yet, it is trying to connect to golang.org.
/cc @myitcv @bcmills for further insight.
And FYI - I think you mean go
when you say vgo
, because module functionality is integrated into the go toolchain.
@agnivade Not only on mac, but also on centos7, compiler will try to connect golang. org until connection timeout
[root@iZ2zefvnzjphkalt990sg0Z vgo-issue]# go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/root/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/vgo-issue/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build194230206=/tmp/go-build -gno-record-gcc-switches"
[root@iZ2zefvnzjphkalt990sg0Z vgo-issue]# time go build -x -v
WORK=/tmp/go-build973675453
Fetching https://golang.org/x/sys?go-get=1
https fetch failed: Get https://golang.org/x/sys?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
Fetching https://golang.org/x/crypto?go-get=1
https fetch failed: Get https://golang.org/x/crypto?go-get=1: dial tcp 216.239.37.1:443: i/o timeout
real 1m0.108s
user 0m0.068s
sys 0m0.041s
GOCACHE
/root/.cache/go-build
[root@iZ2zefvnzjphkalt990sg0Z go-build]# ls
00 09 12 1b 24 2d 36 3f 48 51 5a 63 6c 75 7e 87 90 99 a2 ab b4 bd c6 cf d8 e1 ea f3 fc
01 0a 13 1c 25 2e 37 40 49 52 5b 64 6d 76 7f 88 91 9a a3 ac b5 be c7 d0 d9 e2 eb f4 fd
02 0b 14 1d 26 2f 38 41 4a 53 5c 65 6e 77 80 89 92 9b a4 ad b6 bf c8 d1 da e3 ec f5 fe
03 0c 15 1e 27 30 39 42 4b 54 5d 66 6f 78 81 8a 93 9c a5 ae b7 c0 c9 d2 db e4 ed f6 ff
04 0d 16 1f 28 31 3a 43 4c 55 5e 67 70 79 82 8b 94 9d a6 af b8 c1 ca d3 dc e5 ee f7 log.txt
05 0e 17 20 29 32 3b 44 4d 56 5f 68 71 7a 83 8c 95 9e a7 b0 b9 c2 cb d4 dd e6 ef f8 README
06 0f 18 21 2a 33 3c 45 4e 57 60 69 72 7b 84 8d 96 9f a8 b1 ba c3 cc d5 de e7 f0 f9 trim.txt
07 10 19 22 2b 34 3d 46 4f 58 61 6a 73 7c 85 8e 97 a0 a9 b2 bb c4 cd d6 df e8 f1 fa
08 11 1a 23 2c 35 3e 47 50 59 62 6b 74 7d 86 8f 98 a1 aa b3 bc c5 ce d7 e0 e9 f2 fb
[root@iZ2zefvnzjphkalt990sg0Z go-build]#
A couple of observations
- If the replacements need to be systematic (as they appear to be, potentially for reasons of certain URLs being blocked) then some sort of proxy is going to ultimately be a cleaner solution
- The
go
tool checkinggolang.org
regardless of thereplace
is likely related to #26241
@hellojukay, could you see if you can reproduce the issue with a go
toolchain built from source¹ at HEAD
? I'd like to figure out whether this is a duplicate of #27859 or if there is something more to it.
¹ See https://golang.org/doc/install/source for instructions.
Note another instance of effectively the same problem in #28716.
Stepping back for one second, if golang.org
remains blocked in China (ref https://blog.golang.org/hello-china), then I wonder whether we need to think of a different approach here.
Even though in Go 1.12 #26241 will give us comprehensive support for the concept of "wherever you see X use Y", it strikes me that users from mainland China will always need to add a replace
statements for any of the golang.org/x/*
modules, in every one of their projects where they are a (transitive) dependency.
This seems less than ideal. I would guess that this would impact a significant portion of users' projects.
Is there anything we can do here in the go
tool itself?
cc @andybons
@myitcv I think this is a bug of the code of go modules, If I set up replace for golang.org
, then it should not connect to golang.org
。Isn't it? wait for me , i am building go compiler from the master branch golang 。
@hellojukay I'm going to re-open this issue if that's ok. I've also retitled it to better reflect the question I asked @andybons and @bcmills in #28652 (comment)
@myitcv I build go from master branch source, it works well, build very fast。Thanks !!!
@myitcv I am using goproxy.io to resolve this problem .
but as you say we use golang.org/x/*
packages always download zip from github.com/golang/*
,
go get
was blocked. so we always can not use dep
govendor
to manage our project . so sadly !
or if I have a vps outside of china and has shadowsocks-server on this.
we can export https_proxy=http://127.0.0.1:1080
to go get
. but it was too slowly . and there was so much people that don't has vps . them does not know what was happened to golang.org
if them was new gopher .
so I hope there was a better way to solve this problem which site has been blcoked from our area.
or using domain alias
Thanks for the further details @clearcodecn
so I hope there was a better way to solve this problem which site has been blcoked from our area.
or using domain alias
Yes indeed, that was the motivation behind my question.
@hellojukay is it ok if we keep this issue open to track the question I asked in #28652 (comment)?
@myitcv Well, it depends on you.
Thank you @hellojukay - I'll keep this open for @andybons to comment.
May I know why close #29014?? Both issue not resolved.
I try add replace golang.org/x/net/trace v0.0.0-20181129055619-fae4c4e3ad76 => github.com/golang/net/trace v0.0.0-20181129055619-fae4c4e3ad76 //indirect
also not work.
Let me restate issue again:
I am plan import "github.com/dgraph-io/badger"
It indirect require golang.org/x/net/trace
I know need try use replace methodology to resolve this kind of issue
But still not work with below pain points:
a. I have to try find out the version. Can we let it blank or use * to mean match any version?
b. Even I try go to github to capture your latest commit and put in mod file, it still not work.
c. I try git clone github.com/golang/net and put in cache folder, also useless.
Any better and easy way? Or else, I have to abandon mod way and go back to traditional way again.
And, same as #28652 issue, each go run/build take long time and finally say:
"cannot find module for path golang.org/x/net/trace"
It is waste time.
I try add replace golang.org/x/net/trace v0.0.0-20181129055619-fae4c4e3ad76 => github.com/golang/net/trace v0.0.0-20181129055619-fae4c4e3ad76 //indirect
also not work.
Hi @ScottHuangZL - As mentioned above in this comment, this is fixed in the master branch.
So it won't work until you try the master branch. Please let us know if you still face the issue if you build from master.
@agnivade
I try below, and finally work!
go mod edit -require=golang.org/x/net@v0.0.0
go mod edit -replace=golang.org/x/net@v0.0.0=github.com/golang/net@latest
But the "go build" take a long time to complete build. ==> I have concern for the long build time even it finally success complete, how to resolve it?? Thanks.
If I try "go build -x -v", it will be failed with below message:
Fetching https://golang.org/x/net?go-get=1
https fetch failed: Get https://golang.org/x/net?go-get=1: dial tcp 216.239.37.1:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
It means every build actually it try fetch from online again for x/net. Why? thanks.
It means every build actually it try fetch from online again for x/net. Why?
That's #27859.
a. I have to try find out the version. Can we let it blank or use * to mean match any version?
You can already leave the source of a replacement blank to match any version.
That behavior is tested explicitly here:
The reason you have to specify any version in the first place is #26241.
@ScottHuangZL build golang compiler master source, it works.
@hellojukay @agnivade No plan to build go from master source branch. Will wait until the go1.11.3 version ready.
@agnivade @bcmills May I know when go.1.11.3 release? thanks.
That fix (#27859) is not going to be backported. If you don't want to build from master, please wait for the 1.12 beta release which should be out soon (within this week).
@agnivade sounds good, thanks.
@panjiang wanted a wildcard replace shortcut so that one could replace all the golang.org/x repos to their github.com equivalent in one-shot. #29076 (comment)
A quick way:
go mod edit -require=golang.org/x/net@v0.0.0
go mod edit -replace=golang.org/x/net@v0.0.0=github.com/golang/net@latest
go mod tidy
go mod vendor
go build -mod=vendor
Then the build speed still fast!!!
I just landed the fix for #26241, which should remove the need for the go mod edit -require
step (you can skip straight to the go mod edit -replace
and omit the arbitrary v0.0.0
version).
go module is a nightmare in China. Even if I have a very good VPN, go get -u will never complete in my life.
go mod edit
won't work because you need to have the correct go.mod at first. It's like "which came first: the chicken or the egg?".
Please kill me I am sad and mad.
@sgon00 https://gocenter.io/ can help you.
@hellojukay Thanks a lot for letting me know gocenter.io. I have't tried it yet and will try it later. Sorry that I completely collapsed yesterday. I was trying go get -u simplePackage
inside module directory from 11:00am to 03:00am without a luck. It can never complete successfully. But running the same command outside the module directory, it can be completed quickly without any problems. This is just crazy.
linux
GOPROXY=https://mirrors.aliyun.com/goproxy/ go build
or win
env GOPROXY=https://mirrors.aliyun.com/goproxy/ go build
or use https://goproxy.io
Given goproxy.io
and other proxies with high availability in China, is there anything left to do here?
It's never just golang.org
, github.com
is also sometimes/always unreachable in some special provinces, such as Beijing and Xinjiang. In fact, people in mainland China are already used to solving this kind of problem by using a public hosted GOPROXY.
Given that Go modules are nearly mature and widely used in production, I believe most people are aware of the benefits of using a GOPROXY (not just solve this kind of problem). So I see no reason for this issue to remain open.