Cannot build wolfi-dev/os datadog-agent on a chainguard workstation
smoser opened this issue · 7 comments
smoser commented
On a workstation I currently cannot build datadog-agent with --runner=docker or --runner=bubblewrap.
Full logs of my failed attemt here here:
$ git log --oneline HEAD^..
87d8a659c (HEAD, upstream/main, upstream/HEAD) gitaly setup_cgroups (#24320)
$ git remote -v | grep upst
upstream git@github.com:wolfi-dev/os (fetch)
upstream git@github.com:wolfi-dev/os (push)
$ melange version | grep ^[A-Z]
GitVersion: v0.11.1
GitCommit: a52edcc075ebf1dc89aea87893e3821944171ee3
GitTreeState: clean
BuildDate: '2024-07-19T02:44:39Z'
GoVersion: go1.22.5
Compiler: gc
Platform: linux/amd64
Failures:
-
--runner=bubblewrap$ make MELANGE_EXTRA_OPTS=--runner=bwrap package/datadog-agent 2024/07/19 20:14:30 WARN 8700K .......... .......... .......... .......... .......... 97% 673M 0s 2024/07/19 20:14:30 WARN 8750K .......... .......... .......... .......... .......... 97% 114M 0s 2024/07/19 20:14:30 WARN 8800K .......... .......... .......... .......... .......... 98% 68.7M 0s 2024/07/19 20:14:30 WARN 8850K .......... .......... .......... .......... .......... 98% 131M 0s 2024/07/19 20:14:30 WARN 8900K .......... .......... .......... .......... .......... 99% 184M 0s 2024/07/19 20:14:30 WARN 8950K .......... .......... .......... .......... .......... 99% 402M 0s 2024/07/19 20:14:30 WARN 9000K .... 100% 7.79T=0.3s 2024/07/19 20:14:30 WARN 2024/07/19 20:14:30 WARN 2024-07-19 20:14:30 (27.3 MB/s) - 'common.deb' saved [9220280/9220280] 2024/07/19 20:14:30 WARN 2024/07/19 20:14:55 INFO clang --version 2024/07/19 20:14:55 INFO clang version 12.0.1 2024/07/19 20:14:55 INFO Target: x86_64-unknown-linux-gnu 2024/07/19 20:14:55 INFO Thread model: posix 2024/07/19 20:14:55 INFO InstalledDir: /opt/clang/bin 2024/07/19 20:14:55 WARN /bin/bash: line 1: sudo: command not found 2024/07/19 20:14:55 WARN /bin/bash: line 1: sudo: command not found 2024/07/19 20:14:55 WARN /bin/bash: line 1: sudo: command not found 2024/07/19 20:14:55 INFO sudo /opt/datadog-agent/embedded/bin/clang-bpf --version 2024/07/19 20:14:55 INFO sudo /opt/datadog-agent/embedded/bin/llc-bpf --version 2024/07/19 20:14:55 INFO sudo mkdir -p /opt/datadog-agent/embedded/bin 2024/07/19 20:14:55 ERRO ERROR: failed to build package. the build environment has been preserved: 2024/07/19 20:14:55 INFO workspace dir: /home/user/tmp/melange-workspace-403892385 2024/07/19 20:14:55 INFO guest dir: /home/user/tmp/melange-guest-3929106491 2024/07/19 20:14:55 ERRO failed to build package: unable to run package datadog-agent pipeline: unable to run pipeline: exit status 127 make[1]: *** [Makefile:111: packages/x86_64/datadog-agent-7.55.1-r0.apk] Error 1 make[1]: Leaving directory '/home/user/src/wolfi-os' make: *** [Makefile:101: package/datadog-agent] Error 2 -
--runner=docker:$ echo $TMPDIR /home/usr/tmp $ ls -altrd /home/user/tmp drwxr-sr-x 226 user docker 40960 Jul 19 20:27 /home/user/tmp 2024/07/19 20:25:13 WARN + cp Dockerfiles/agent/init-stage3-host-pid /home/build/melange-out/datadog-agent/etc/s6/init/init-stage3-host-pid 2024/07/19 20:25:13 WARN + exit 0 2024/07/19 20:25:13 INFO running step "Strip binaries" 2024/07/19 20:25:13 WARN + '[' -d /home/build ] 2024/07/19 20:25:13 WARN + cd /home/build 2024/07/19 20:25:13 WARN + exit 0 2024/07/19 20:25:13 WARN + '[' -d /home/build/melange-out/datadog-agent ] 2024/07/19 20:25:13 WARN + cd /home/build/melange-out/datadog-agent 2024/07/19 20:25:13 WARN + scanelf --recursive --nobanner --osabi --etype ET_DYN,ET_EXEC . 2024/07/19 20:25:13 WARN + read type osabi filename 2024/07/19 20:25:13 WARN + '[' NONE '!=' STANDALONE ] 2024/07/19 20:25:13 WARN + strip -g ./usr/bin/agent 2024/07/19 20:25:16 WARN + read type osabi filename 2024/07/19 20:25:16 WARN + '[' NONE '!=' STANDALONE ] 2024/07/19 20:25:16 WARN + strip -g ./usr/lib/libdatadog-agent-rtloader.so.0.1.0 2024/07/19 20:25:16 WARN + read type osabi filename 2024/07/19 20:25:16 WARN + '[' NONE '!=' STANDALONE ] 2024/07/19 20:25:16 WARN + strip -g ./usr/lib/libdatadog-agent-three.so 2024/07/19 20:25:16 WARN + read type osabi filename 2024/07/19 20:25:16 WARN + exit 0 2024/07/19 20:25:16 INFO running pipeline for subpackage datadog-agent-jmx 2024/07/19 20:25:26 INFO pod 6d99476d0e0ba3e4507c2c22e201b1442f07bb419b7babc3598b8c3dfe055198 terminated 2024/07/19 20:25:26 ERRO ERROR: failed to build package. the build environment has been preserved: 2024/07/19 20:25:26 INFO workspace dir: /home/user/tmp/melange-workspace-150168302 2024/07/19 20:25:26 INFO guest dir: /home/user/tmp/melange-guest-2009475398 2024/07/19 20:25:26 ERRO failed to build package: mkdir /home/user/tmp/melange-workspace-150168302/melange-out: permission denied make[1]: *** [Makefile:111: packages/x86_64/datadog-agent-7.55.1-r0.apk] Error 1 make[1]: Leaving directory '/home/user/src/wolfi-os' make: *** [Makefile:101: package/datadog-agent] Error 2
smoser commented
Note that on a workstation you get TMPDIR set to /home/user/tmp/ and
$ ls -ld /home/user/tmp/
drwxr-sr-x 226 user docker 40960 Jul 19 20:33 /home/user/tmp/
I've also tried unsetting TMPDIR (thus using /tmp) (dies the same way).
$ make ....
2024/07/19 20:36:19 INFO running pipeline for subpackage datadog-agent-jmx
2024/07/19 20:36:30 INFO pod e526f79266b40c1dba9eeb27775b82217a088ac964d4c134afd51144ecd4e9aa terminated
2024/07/19 20:36:30 ERRO ERROR: failed to build package. the build environment has been preserved:
2024/07/19 20:36:30 INFO workspace dir: /tmp/melange-workspace-2565688441
2024/07/19 20:36:30 INFO guest dir: /tmp/melange-guest-4221147986
2024/07/19 20:36:30 ERRO failed to build package: mkdir /tmp/melange-workspace-2565688441/melange-out: permission denied
make[1]: *** [Makefile:111: packages/x86_64/datadog-agent-7.55.1-r0.apk] Error 1
$ ls -ld /tmp/
drwxrwxrwt 11 root root 300 Jul 19 20:32 /tmp/
hbh7 commented
I'm having the same sort of issue I think. Looks like the melange workspace folder is getting created as root and then can't be accessed again. This only happens sometimes.
hbh7 commented
Also of note, if I run with MELANGE_EXTRA_OPTS=--runner=bubblewrap, it'll succeed. Although it often will take 10x longer...
philroche commented
wiki page documenting what is required to workaround these issues @ https://eng.inky.wtf/docs/squads/sustaining/playbooks/package-updates/datadog_agent/
philroche commented
@smoser Can we close this now with your changes @ wolfi-dev/os#33387 ?
philroche commented
https://github.com/chainguard-dev/wiki/pull/914 PR to remove playbook now that building datadog-agent without build patches is possible
smoser commented
@smoser Can we close this now with your changes @ wolfi-dev/os#33387 ?
yes. closing. thanks.