fioconfig compile failure behind corporate proxy...
Closed this issue · 12 comments
I am seeing the compile failure below. As far as I know my http_proxy and htts_proxy environment variables are good and the proxy is working fine. If I edit the fioconfig recipe and insert GOPROXY=direct infront of the compilation command I get a similar error but only after a good couple of minutes of compilation. Any idea's what's wrong there?
Build Configuration:
BB_VERSION = "1.48.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "ubuntu-18.04"
TARGET_SYS = "aarch64-lmp-linux"
MACHINE = "raspberrypi4-64"
DISTRO = "lmp"
DISTRO_VERSION = "3.2"
TUNE_FEATURES = "aarch64 armv8a crc crypto cortexa72"
TARGET_FPU = ""
meta-lmp-base = "HEAD:3beb1d4791967ea4f1123dea519fb29574d102c2"
meta-oe
meta-networking
meta-filesystems
meta-perl
meta-python = "HEAD:b9dcf17700014dec679307dc1247882dccd439ca"
meta-virtualization = "HEAD:9fe997733d9bad4ac24dfb41e91a0e06b9e82791"
meta-clang = "HEAD:1dfc3a905ad0fed7a3b300aa51153365f9950fe7"
meta-updater = "HEAD:075ca14b8800dcae1c2532389e5dd219f6e7c2dd"
meta-security = "HEAD:4583ab9b08c1c582a9d7e8ab70d6685ed4720799"
meta-arm
meta-arm-toolchain
meta-arm-bsp = "HEAD:1a0a9700f2e667752e985f57087272553b239aa4"
meta-freescale = "HEAD:b9870755b89b8bc81d0fee60e3050155e94a9c46"
meta-freescale-3rdparty = "HEAD:d23ec3443d2a2fb36f29d0ef04d2806dab286676"
meta-raspberrypi = "HEAD:09a3c11696c75593f8e475da5dfb401016c6aaca"
meta-riscv = "HEAD:9a75ffbb21ebee61782576d524a454d04a9f91e2"
meta-rtlwifi = "HEAD:ccf8cdc0e2282cf3989d35504989dc404bbe1370"
meta-intel = "HEAD:5492270c796daf8b7a7cc9cd93880c2bb25523c0"
meta-yocto-bsp = "HEAD:269f81c2b2ac64219c65953bc56bd41317a8db0f"
meta-xilinx-bsp = "HEAD:c4df43e8b4b965a6deb13ed108fb6da96cdaffb4"
meta-xilinx-tools = "HEAD:6f15ebaddb86898d3d34053283ca9d0a5dc8d67b"
meta-lmp-bsp = "HEAD:3beb1d4791967ea4f1123dea519fb29574d102c2"
meta = "HEAD:af428fa2432279d24cdf2a62f9dee91b30d46c3a"
Initialising tasks: 100% |#######################################################################################################################################################################################| Time: 0:00:02
Sstate summary: Wanted 0 Found 0 Missed 0 Current 40 (0% match, 100% complete)
NOTE: Executing Tasks
ERROR: fioconfig-git-r0 do_compile: Execution of '/build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191' failed with exit code 1:
go: github.com/doanac/crypto11@v1.2.2-0.20200715151421-f3d2e17ac497: Get "https://proxy.golang.org/github.com/doanac/crypto11/@v/v1.2.2-0.20200715151421-f3d2e17ac497.mod": dial tcp: lookup proxy.golang.org: no such host
WARNING: /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191:167 exit 1 from 'aarch64-lmp-linux-go build -tags vpn -ldflags="${FIOCONFIG_LDFLAGS}" -o /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/build/bin/linux_arm64/fioconfig main.go'
WARNING: Backtrace (BB generated script):
#1: do_compile, /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191, line 167
#2: main, /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191, line 172
Backtrace (metadata-relative locations):
#1: do_compile, /build1/peter/PE2/LMP/build/conf/../../layers/meta-lmp/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb, line 24
ERROR: Logfile of failure stored in: /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/log.do_compile.69191
Log data follows:
| DEBUG: Executing shell function do_compile
| go: github.com/doanac/crypto11@v1.2.2-0.20200715151421-f3d2e17ac497: Get "https://proxy.golang.org/github.com/doanac/crypto11/@v/v1.2.2-0.20200715151421-f3d2e17ac497.mod": dial tcp: lookup proxy.golang.org: no such host
| WARNING: /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191:167 exit 1 from 'aarch64-lmp-linux-go build -tags vpn -ldflags="${FIOCONFIG_LDFLAGS}" -o /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/build/bin/linux_arm64/fioconfig main.go'
| WARNING: Backtrace (BB generated script):
| #1: do_compile, /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191, line 167
| #2: main, /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191, line 172
| ERROR: Execution of '/build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191' failed with exit code 1:
| go: github.com/doanac/crypto11@v1.2.2-0.20200715151421-f3d2e17ac497: Get "https://proxy.golang.org/github.com/doanac/crypto11/@v/v1.2.2-0.20200715151421-f3d2e17ac497.mod": dial tcp: lookup proxy.golang.org: no such host
| WARNING: /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191:167 exit 1 from 'aarch64-lmp-linux-go build -tags vpn -ldflags="${FIOCONFIG_LDFLAGS}" -o /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/build/bin/linux_arm64/fioconfig main.go'
| WARNING: Backtrace (BB generated script):
| #1: do_compile, /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191, line 167
| #2: main, /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/run.do_compile.69191, line 172
|
| Backtrace (metadata-relative locations):
| #1: do_compile, /build1/peter/PE2/LMP/build/conf/../../layers/meta-lmp/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb, line 24
ERROR: Task (/build1/peter/PE2/LMP/build/conf/../../layers/meta-lmp/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb:do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 312 tasks of which 306 didn't need to be rerun and 1 failed.
NOTE: Writing buildhistory
NOTE: Writing buildhistory took: 4 seconds
Summary: 1 task failed:
/build1/peter/PE2/LMP/build/conf/../../layers/meta-lmp/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb:do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Thanks for reporting, will have a look.
This issue seems to be related how go uses GOPROXY for fetching the modules that are dependencies for fioconfig. By default GOPROXY is set to “https://proxy.golang.org,direct”, which forces the google proxy as default option, and direct in case it is not found via proxy. In your case there seems to be a problem resolving proxy.golang.org, so that could be internal to your network.
The interesting part is that GOPROXY=direct would force it to use git directly, and that should make it work because git understands the http_proxy/https_proxy settings, but this doesn't work in OE because that logic is only part of the fetcher, and here the compile process is the one fetching all dependencies.
This seems to be a more generic golang issue in OE/Yocto (for projects using go mods), so will take a look at how we can fix this properly, but you can probably get it to work locally by changing the fioconfig recipe with the following:
meta-lmp$ git diff
diff --git a/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb b/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb
index fdb177e..4b7d65c 100644
--- a/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb
+++ b/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb
@@ -21,7 +21,7 @@ do_compile() {
BUILD_COMMIT=`git rev-parse --short HEAD`
FIOCONFIG_LDFLAGS="-X ${GO_IMPORT}/internal.Commit=${BUILD_COMMIT}"
mkdir -p ${B}/${GO_BUILD_BINDIR}
- ${GO} build -tags vpn -ldflags="${FIOCONFIG_LDFLAGS}" -o ${B}/${GO_BUILD_BINDIR}/fioconfig main.go
+ http_proxy="<your_proxy>" https_proxy="<your proxy>" GOPROXY=direct ${GO} build -tags vpn -ldflags="${FIOCONFIG_LDFLAGS}" -o ${B}/${GO_BUILD_BINDIR}/fioconfig main.go
chmod u+w -R ${B}
}
Thanks for the detailed answer, I will give it a go. I did think our proxy might be not be resolving the default Go proxy, but I can get the offending module with wget, so it seems it might be an OE problem.
Thanks for the detailed answer, I will give it a go. I did think our proxy might be not be resolving the default Go proxy, but I can get the offending module with wget, so it seems it might be an OE problem.
Can you test building fioconfig outside OE?
git clone https://github.com/foundriesio/fioconfig.git
cd fioconfig
go build -o fioconfig main.go
Assuming you got a recent go version installed at your host, it should also try to use the default settings for GOPROXY (and you can also set direct).
I can build outside of OE, see below
peter@ubuntu:~/fioconfig$ go build -o fioconfig main.go
go: downloading github.com/urfave/cli v1.22.1
go: downloading github.com/doanac/crypto11 v1.2.2-0.20200715151421-f3d2e17ac497
go: downloading github.com/foundriesio/go-toml v1.8.1-0.20200721033514-2232fec316b9
go: downloading github.com/ethereum/go-ethereum v1.9.11
go: downloading github.com/urfave/cli/v2 v2.2.0
go: downloading github.com/miekg/pkcs11 v1.0.3-0.20190429190417-a667d056470f
go: downloading github.com/thales-e-security/pool v0.0.1
go: downloading github.com/cpuguy83/go-md2man/v2 v2.0.0-20190314233015-f79a8a8ca69d
go: downloading github.com/russross/blackfriday/v2 v2.0.1
go: downloading github.com/shurcooL/sanitized_anchor_name v1.0.0
peter@ubuntu:~/fioconfig$
I also tried the patch to the recipe above and that works too :)
Having got past the compilation phase I now have another QA issue. I don't know if this is related to changing the recipe.
ERROR: fioconfig-git-r0 do_package_qa: QA Issue: /usr/lib/go/pkg/mod/cache/vcs/dbb1616a13223a75321e21a8150b1a7781650b73e761213ecc0ab67568c38ac2/hooks/fsmonitor-watchman.sample contained in package fioconfig-staticdev requires /usr/bin/perl, but no providers found in RDEPENDS_fioconfig-staticdev? [file-rdeps]
ERROR: fioconfig-git-r0 do_package_qa: QA run found fatal errors. Please consider fixing them.
ERROR: Logfile of failure stored in: /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/log.do_package_qa.114171
ERROR: Task (/build1/peter/PE2/LMP/build/conf/../../layers/meta-lmp/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb:do_package_qa) failed with exit code '1'
I also tried the patch to the recipe above and that works too :)
Great, so that is indeed a problem in OE (fetching during compile, which doesn't include the http_proxy variables).
Looking at how to fix this properly now.
Having got past the compilation phase I now have another QA issue. I don't know if this is related to changing the recipe.
ERROR: fioconfig-git-r0 do_package_qa: QA Issue: /usr/lib/go/pkg/mod/cache/vcs/dbb1616a13223a75321e21a8150b1a7781650b73e761213ecc0ab67568c38ac2/hooks/fsmonitor-watchman.sample contained in package fioconfig-staticdev requires /usr/bin/perl, but no providers found in RDEPENDS_fioconfig-staticdev? [file-rdeps] ERROR: fioconfig-git-r0 do_package_qa: QA run found fatal errors. Please consider fixing them. ERROR: Logfile of failure stored in: /build1/peter/PE2/LMP/build/tmp-lmp/work/cortexa72-lmp-linux/fioconfig/git-r0/temp/log.do_package_qa.114171 ERROR: Task (/build1/peter/PE2/LMP/build/conf/../../layers/meta-lmp/meta-lmp-base/recipes-support/fioconfig/fioconfig_git.bb:do_package_qa) failed with exit code '1'
This one is interesting because in theory the same modules (and revs) would be used when downloading from scratch via proxy, so unclear what happened (maybe it got more source files to parse now?). Will check as well.
Adding the following should fix your QA warning:
RDEPENDS_${PN}-staticdev += "perl"
Changes merged on lmp-manifest, please sync to the latest on master and let me know if it fixes your issue.
All good now, I'll leave you to close the issue?