Gdn failed to run on ubuntu bionic

Question

Gdn failed to run on ubuntu bionic

xtremerui opened this issue 2 years ago · 33 comments

We noticed in Concourse CI testing, workers failed to start by error

gdn: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by gdn)
gdn: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by gdn)

it is running on ubuntu bionic.

version: 1.21.0

Answer 1 · 2022-09-02T15:46:32.000Z

Hi @xtremerui ,

Are you in a position where you can install glibc?

Thanks,
Marc

Answer 2 · 2022-09-06T14:24:10.000Z

yes we control the dependency installation for the env where we build our binary.

Answer 3 · 2022-09-06T14:40:20.000Z

@xtremerui
After installing glibc does the problem resolve?

Answer 4 · 2022-09-06T15:14:27.000Z

Do i need to install glibc by a specific version like 2.32 or 2.34 in this case?

Answer 5 · 2022-09-06T15:23:02.000Z

@MarcPaquette i stand corrected here. For the docker image we build, we have control. But for the test that our binary runs on google cloud, we dont have control over the OS image that google provides (which is a standard google jammy jellyfish OS image family).

Answer 6 · 2022-09-07T13:57:31.000Z

Hi @xtremerui

We just released Garden-runc-release v1.22.2, which statically compiles gdn.

Can you test it out and see if it fixes your issue?

Thanks,
@MarcPaquette

Answer 7 · 2022-09-07T20:51:44.000Z

@MarcPaquette Thx for the effort.

Now we are seeing

/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)

in a GCP VM where gdn is ran

ruiya@smoke-flexible-primate:/home/concourse$ ldd --version
ldd (Ubuntu GLIBC 2.27-3ubuntu1.6) 2.27
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
ruiya@smoke-flexible-primate:/home/concourse$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"
ruiya@smoke-flexible-primate:/home/concourse$

Answer 8 · 2022-10-04T13:04:08.000Z

Hi @xtremerui
Can you confirm that the latest release resolves this issue for you?

Answer 9 · 2022-10-04T19:56:56.000Z

@MarcPaquette nope we are still seeing the same error

Answer 10 · 2022-12-06T11:14:02.000Z

Hi all,
any news on this?
I'm still having the same issues as @xtremerui.

/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)

GLIBC is expected to be at least 2.28, what I find in bionic/1.150 is 2.27:

ldd --version
ldd (Ubuntu GLIBC 2.27-3ubuntu1.6) 2.27
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

Answer 11 · 2022-12-06T15:09:41.000Z

We have this story in our internal backlog: https://www.pivotaltracker.com/story/show/183466259

I've been out of sabbatical for a couple months. let me ping the team to see where we have it prioritized.

CC @ameowlia

Answer 12 · 2022-12-06T18:51:38.000Z

We started making the garden job use the system iptables (/sbin) by default in v1.22.0.

What version of garden + the stemcell are you seeing this on?
What is your garden.iptables_bin_dir property set to?
Where is this error appearing for you?

Answer 13 · 2023-01-09T16:36:57.000Z

Hi @xtremerui

Is this issue still outstanding for you? It's been fairly quite for a bit of time and I'm wondering if we can close it out?

Answer 14 · 2023-01-09T19:34:57.000Z

@geofffranks @MarcPaquette we are still getting tickets from Customer about this issue.

What version of garden + the stemcell are you seeing this on?

1.22.7 that released with Concourse v7.9.0

What is your garden.iptables_bin_dir property set to?

we don't set this property in Concourse

Where is this error appearing for you?

The above errors (both in mine and @schindlersebastian comments) show up when Concourse runs gdn binary here. It seems like gdn binary doesn't equiped with needed GLIBC so it still looks for that lib from OS.

Answer 15 · 2023-01-18T16:04:02.000Z

@xtremerui and @schindlersebastian

Can you provide us with a reproduction steps?

FWIW, Bionic support ends in April.

Answer 16 · 2023-02-06T20:38:55.000Z

@xtremerui @schindlersebastian 1.23.0 version of the gdn is now shipped without dependency on GLIBC and build with musl. Please try them on bionic and let us know if that will solve this issue.

Answer 17 · 2023-02-09T20:56:29.000Z

I am going to close this issue, please re-open if you are still having a problem.

Answer 18 · 2023-02-09T22:31:54.000Z

We have built the latest Concourse image with gdn 1.23 and it still fails on GCP VM with ubuntu-1804-lts os image, please find the detailed error below.

PS: I can't reopen this issue. Should I create a new issue instead?

error log

```console

Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: {"timestamp":"2023-02-09T22:08:38.287387443Z","level":"error","source":"guardian","message":"guardian.starting-guardian-backend","data":{"error":"bulk starter: setting up default chains: iptables: setup-global-chains: + set -o nounset\n+ set -o errexit\n+ shopt -s nullglob\n+ filter_input_chain=w--input\n+ filter_forward_chain=w--forward\n+ filter_default_chain=w--default\n+ filter_instance_prefix=w--instance-\n+ nat_prerouting_chain=w--prerouting\n+ nat_postrouting_chain=w--postrouting\n+ nat_instance_prefix=w--instance-\n+ iptables_bin=/var/gdn/assets/linux/sbin/iptables\n+ case "${ACTION}" in\n+ setup_filter\n+ teardown_filter\n+ teardown_deprecated_rules\n++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT\n+ rules=\n+ true\n+ echo ''\n+ grep ' -j garden-dispatch'\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD\n+ rules=\n+ true\n+ grep ' -j garden-dispatch'\n+ echo ''\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ /var/gdn/assets/linux/sbin/iptables -w -F garden-dispatch\n+ true\n+ /var/gdn/assets/linux/sbin/iptables -w -X garden-dispatch\n+ true\n++ /var/gdn/assets/linux/sbin/iptables -w -S w--forward\n+ rules=\n+ true\n+ echo ''\n+ grep '\-g w--instance-'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n++ /var/gdn/assets/linux/sbin/iptables -w -S\n+ rules=\n+ true\n+ echo ''\n+ grep '^-A w--instance-'\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n++ /var/gdn/assets/linux/sbin/iptables -w -S\n+ rules=\n+ true\n+ echo ''\n+ grep '^-N w--instance-'\n+ sed -e s/-N/-X/ -e 's/\s\+$//'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD\n+ rules=\n+ true\n+ echo ''\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ grep ' -j w--forward'\n+ /var/gdn/assets/linux/sbin/iptables -w -F w--forward\n+ true\n+ /var/gdn/assets/linux/sbin/iptables -w -F w--default\n+ true\n++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT\n+ rules=\n+ true\n+ echo ''\n+ grep ' -j w--input'\n+ sed -e 's/--icmp-type any/--icmp-type 255\/255/'\n+ sed -e s/-A/-D/ -e 's/\s\+$//'\n+ xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w\n+ /var/gdn/assets/linux/sbin/iptables -w -F w--input\n+ true\n+ /var/gdn/assets/linux/sbin/iptables -w -X w--input\n+ true\n++ ip route show\n++ grep default\n++ head -1\n++ cut '-d ' -f5\n+ default_interface=ens4\n+ /var/gdn/assets/linux/sbin/iptables -w -N w--input\n+ /var/gdn/assets/linux/sbin/iptables -w -F w--input\n/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables)\n/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)\n/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables)\n/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)\n"}}
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: bulk starter: setting up default chains: iptables: setup-global-chains: + set -o nounset
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + set -o errexit
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + shopt -s nullglob
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_input_chain=w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_forward_chain=w--forward
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_default_chain=w--default
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_instance_prefix=w--instance-
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_prerouting_chain=w--prerouting
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_postrouting_chain=w--postrouting
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_instance_prefix=w--instance-
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + iptables_bin=/var/gdn/assets/linux/sbin/iptables
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + case "${ACTION}" in
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + setup_filter
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + teardown_filter
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + teardown_deprecated_rules
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j garden-dispatch'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j garden-dispatch'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F garden-dispatch
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -X garden-dispatch
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S w--forward
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '-g w--instance-'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '^-A w--instance-'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '^-N w--instance-'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-N/-X/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j w--forward'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--forward
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--default
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j w--input'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -X w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ ip route show
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ grep default
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ head -1
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ cut '-d ' -f5
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + default_interface=ens4
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -N w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: bulk starter: setting up default chains: iptables: setup-global-chains: + set -o nounset
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + set -o errexit
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + shopt -s nullglob
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_input_chain=w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_forward_chain=w--forward
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_default_chain=w--default
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + filter_instance_prefix=w--instance-
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_prerouting_chain=w--prerouting
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_postrouting_chain=w--postrouting
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + nat_instance_prefix=w--instance-
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + iptables_bin=/var/gdn/assets/linux/sbin/iptables
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + case "${ACTION}" in
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + setup_filter
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + teardown_filter
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + teardown_deprecated_rules
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j garden-dispatch'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j garden-dispatch'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F garden-dispatch
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -X garden-dispatch
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S w--forward
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '-g w--instance-'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '^-A w--instance-'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep '^-N w--instance-'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-N/-X/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S FORWARD
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j w--forward'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--forward
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--default
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ /var/gdn/assets/linux/sbin/iptables -w -S INPUT
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + rules=
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + echo ''
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + grep ' -j w--input'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e 's/--icmp-type any/--icmp-type 255/255/'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + sed -e s/-A/-D/ -e 's/\s+$//'
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + xargs --no-run-if-empty --max-lines=1 /var/gdn/assets/linux/sbin/iptables -w
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -X w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + true
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ ip route show
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ grep default
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ head -1
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: ++ cut '-d ' -f5
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + default_interface=ens4
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -N w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: + /var/gdn/assets/linux/sbin/iptables -w -F w--input
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables) Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: /var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)
Feb 9 22:08:38 smoke-sunny-mammal concourse[7102]: {"timestamp":"2023-02-09T22:08:38.292747265Z","level":"error","source":"worker","message":"worker.garden.gdn-runner.logging-runner-exited","data":{"error":"exit status 1","session":"1.2"}}

</details>

Answer 19 · 2023-02-13T14:07:41.000Z

@xtremerui i am going to re-open this issue instead. I think the problem is that we need to build iptables with musl as well.

Answer 20 · 2023-02-13T16:14:19.000Z

@xtremerui With the above commit, we are now building iptables with musl as well, which should solve the issue you were seeing last. That said, we are using x86_64 libraries to build these binaries and that probably means that the arm64 binaries for (#238) wouldn't work as expected. We have no way of verifying arm64 binaries for gdn. If it's broken, we are happy to take them off publishing list so that only x86_64 is published.

Answer 21 · 2023-02-13T18:00:02.000Z

@winkingturtle-vmw thank you so much! We will soon update the CI to build arm64 binary for Concourse. Once that is done we should be able to verify it.

Answer 22 · 2023-02-22T21:18:49.000Z

@xtremerui We just released v1.25.0 with the fix for this issue. Please let us know if that solves the issues.

Answer 23 · 2023-02-22T23:46:16.000Z

@winkingturtle-vmw we are now seeing this error with latest gdn release

concourse-worker-1  | {"timestamp":"2023-02-22T23:41:59.829149215Z","level":"error","source":"guardian","message":"guardian.starting-guardian-backend","data":{"error":"bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted"}}
concourse-worker-1  | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse-worker-1  | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse-worker-1  | {"timestamp":"2023-02-22T23:41:59.831115448Z","level":"error","source":"worker","message":"worker.garden.gdn-runner.logging-runner-exited","data":{"error":"exit status 1","session":"1.2"}}

Here is the gdn command if it helps:

concourse-worker-1  | gdn args: [server --bind-ip 0.0.0.0 --bind-port 7777 --depot /worker-state/depot --properties-path /worker-state/garden-properties.json --time-format rfc3339 --no-image-plugin --max-containers 250 --network-pool 10.80.0.0/16]

Answer 24 · 2023-02-23T14:07:24.000Z

@xtremerui Is this error happening now with both bionic and jammy? Is there a way for us to test out the gdn binary manually in a concourse worker?

Answer 25 · 2023-02-23T15:53:16.000Z

@winkingturtle-vmw if you follow this https://github.com/concourse/concourse#quick-start in a bionic or jammy, it should pulling the latest concourse/dev docker image that built with gdn 1.25.

After your local concourse is up, you can docker exec the worker container (if it doesn't started you can exec on the web container since they are using the same base concourse/dev image).

Answer 26 · 2023-02-23T15:55:34.000Z

@xtremerui Is this error happening now with both bionic and jammy?

Answer 27 · 2023-02-23T16:06:49.000Z

I have only tested in Jammy as our CI haven't reached to the phase to fan out on bionic OS testing.

Answer 28 · 2023-02-28T21:30:21.000Z

@xtremerui We ended up building the iptables on bionic so that it will continue to work on both Jammy and Bionic. I've verified that the resulting gdn binary works on a bionic workstation, and it will be release in the next version of garden-runc

@winkingturtle-vmw we are now seeing this error with latest gdn release

concourse-worker-1  | {"timestamp":"2023-02-22T23:41:59.829149215Z","level":"error","source":"guardian","message":"guardian.starting-guardian-backend","data":{"error":"bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted"}}
concourse-worker-1  | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse-worker-1  | bulk starter: mounting subsystem 'cpuset' in '/sys/fs/cgroup/cpuset': operation not permitted
concourse-worker-1  | {"timestamp":"2023-02-22T23:41:59.831115448Z","level":"error","source":"worker","message":"worker.garden.gdn-runner.logging-runner-exited","data":{"error":"exit status 1","session":"1.2"}}

Here is the gdn command if it helps:

concourse-worker-1  | gdn args: [server --bind-ip 0.0.0.0 --bind-port 7777 --depot /worker-state/depot --properties-path /worker-state/garden-properties.json --time-format rfc3339 --no-image-plugin --max-containers 250 --network-pool 10.80.0.0/16]

In regards to this issue that you referred to. I believe this is the result of Jammy docker images using cgroups-v2 by default. In Jammy bosh stemcells, we actually turn off cfgroups-v2 and use cgroups-v1 and that's why guardian still works. At this time, there hasn't been a plan to support cgroups-v2 yet. We hope to introduce this support soon.

Answer 29 · 2023-03-08T22:47:43.000Z

@winkingturtle-vmw is there an ETA on next runc release? Thank you!

Answer 30 · 2023-03-16T12:43:15.000Z

@xtremerui v1.26.0 is released now.

Answer 31 · 2023-03-17T09:05:10.000Z

@xtremerui manually replacing /var/vcap/packages/concourse/bin/gdn with the newly released 1.26.0 works for us (concourse 7.9.1)

(hmm, the worker is running now, but we have a resource-checking failure. Might not be related, checking...)

Answer 32 · 2023-03-18T02:39:48.000Z

@beeender in our CI we also notice worker started fine but smoke test failed with checking error:

    stderr: '',

    stdout: `checking some-pipeline/mockery in build 2␊

    initializing check: mockery␊

    selected worker: smoke-sacred-foxhound␊

    run check: start process: backend error: Exit status: 500, message: {"Type":"","Message":"EOF","Handle":"","ProcessID":"","Binary":""}␊

    ␊

    errored␊

    `,

    message: `Command failed: fly -t wats-target-efbf6e47-1c2b-43a3-97d9-21c5acafda1e check-resource -r some-pipeline/mockery␊

     \`fly -t wats-target-efbf6e47-1c2b-43a3-97d9-21c5acafda1e check-resource -r some-pipeline/mockery\` (exited with error code 2)`

Will investigate later to get some backend logs about the failure.

Answer 33 · 2023-03-28T16:43:56.000Z

@beeender here is the log output from the GCP vm that runs Concourse with gdn v1.26

error log

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.333442718Z","level":"info","source":"guardian","message":"guardian.create.start","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.333547607Z","level":"info","source":"guardian","message":"guardian.create.network-depot-setup-bindmounts.start","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.333692387Z","level":"info","source":"guardian","message":"guardian.create.network-depot-setup-bindmounts.finished","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.333718990Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.start","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.3"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.336064292Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.depot-create.started","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.3.1"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.337222417Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.depot-create.finished","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.3.1"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.337286212Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.creating","data":{"bundle":"/etc/concourse/work-dir/depot/d47bf858-3602-47bb-4a42-0f1d7d2a1c85","bundlePath":"/etc/concourse/work-dir/depot/d47bf858-3602-47bb-4a42-0f1d7d2a1c85","handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","id":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","logPath":"/etc/concourse/work-dir/depot/d47bf858-3602-47bb-4a42-0f1d7d2a1c85/create.log","pidFilePath":"/etc/concourse/work-dir/depot/d47bf858-3602-47bb-4a42-0f1d7d2a1c85/pidfile","runc":"goci.RuncBinary{Path:"/var/gdn/assets/linux/bin/runc", Root:"/run/runc"}","session":"38.3.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 142.811787] cgroup: cgroup: disabling cgroup2 socket matching due to net_prio or net_cls activation

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.511133319Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.completing","data":{"bundle":"/etc/concourse/work-dir/depot/d47bf858-3602-47bb-4a42-0f1d7d2a1c85","handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.3.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.511746397Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.finished","data":{"bundle":"/etc/concourse/work-dir/depot/d47bf858-3602-47bb-4a42-0f1d7d2a1c85","handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.3.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.511783106Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.finished","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.3"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.512209409Z","level":"info","source":"guardian","message":"guardian.create.containerizer-create.create.watch.watching","data":{"bundle":"/etc/concourse/work-dir/depot/d47bf858-3602-47bb-4a42-0f1d7d2a1c85","handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.3.2.1"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.525048376Z","level":"info","source":"guardian","message":"guardian.create.network.started","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.7","spec":""}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.525115738Z","level":"info","source":"guardian","message":"guardian.create.network.config-create","data":{"config":{"ContainerHandle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","HostIntf":"wk51d659a9hg-0","ContainerIntf":"wk51d659a9hg-1","IPTablePrefix":"w--","IPTableInstance":"k51d659a9hg","BridgeName":"wbrdg-0a500000","BridgeIP":"10.80.0.1","ContainerIP":"10.80.0.2","ExternalIP":"10.128.0.62","Subnet":{"IP":"10.80.0.0","Mask":"/////A=="},"Mtu":1460,"PluginNameservers":null,"OperatorNameservers":["8.8.8.8","8.8.4.4"],"AdditionalNameservers":[],"AdditionalHostEntries":null,"PluginSearchDomains":null},"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.7","spec":""}}

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 142.959243] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.

Mar 28 16:15:59 smoke-sacred-foxhound systemd-networkd[827]: wbrdg-0a500000: Link UP

Mar 28 16:15:59 smoke-sacred-foxhound systemd-udevd[7154]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.

Mar 28 16:15:59 smoke-sacred-foxhound networkd-dispatcher[1241]: WARNING:Unknown index 3 seen, reloading interface list

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 142.968878] wbrdg-0a500000: port 1(wk51d659a9hg-0) entered blocking state

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 142.968881] wbrdg-0a500000: port 1(wk51d659a9hg-0) entered disabled state

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 142.969033] device wk51d659a9hg-0 entered promiscuous mode

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 142.969723] wbrdg-0a500000: port 1(wk51d659a9hg-0) entered blocking state

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 142.969725] wbrdg-0a500000: port 1(wk51d659a9hg-0) entered forwarding state

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 142.969801] wbrdg-0a500000: port 1(wk51d659a9hg-0) entered disabled state

Mar 28 16:15:59 smoke-sacred-foxhound systemd-networkd[827]: wk51d659a9hg-0: Link UP

Mar 28 16:15:59 smoke-sacred-foxhound systemd-udevd[7153]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.

Mar 28 16:15:59 smoke-sacred-foxhound systemd-udevd[7153]: Could not generate persistent MAC address for wk51d659a9hg-1: No such file or directory

Mar 28 16:15:59 smoke-sacred-foxhound systemd-udevd[7160]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.

Mar 28 16:15:59 smoke-sacred-foxhound systemd-udevd[7160]: Could not generate persistent MAC address for wk51d659a9hg-0: No such file or directory

Mar 28 16:15:59 smoke-sacred-foxhound systemd-networkd[827]: wk51d659a9hg-0: Gained carrier

Mar 28 16:15:59 smoke-sacred-foxhound systemd-networkd[827]: wbrdg-0a500000: Gained carrier

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 143.045230] IPv6: ADDRCONF(NETDEV_CHANGE): wk51d659a9hg-0: link becomes ready

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 143.045276] wbrdg-0a500000: port 1(wk51d659a9hg-0) entered blocking state

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 143.045279] wbrdg-0a500000: port 1(wk51d659a9hg-0) entered forwarding state

Mar 28 16:15:59 smoke-sacred-foxhound kernel: [ 143.045319] IPv6: ADDRCONF(NETDEV_CHANGE): wbrdg-0a500000: link becomes ready

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.627321522Z","level":"info","source":"guardian","message":"guardian.create.network.finished","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38.7","spec":""}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.627408943Z","level":"info","source":"guardian","message":"guardian.create.created","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"38"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.627511053Z","level":"info","source":"guardian","message":"guardian.api.garden-server.create.created","data":{"request":{"Handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","GraceTime":0,"RootFSPath":"raw:///etc/concourse/work-dir/volumes/live/933dc33b-1580-4bdd-652a-ed2a4be9ec29/volume","BindMounts":[{"src_path":"/etc/concourse/work-dir/volumes/live/fec0bf72-8576-45f0-4eab-0d5c0542587a/volume","dst_path":"/scratch","mode":1}],"Network":"","Privileged":false,"Limits":{"bandwidth_limits":{},"cpu_limits":{},"disk_limits":{},"memory_limits":{},"pid_limits":{}}},"session":"3.1.3"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.645462056Z","level":"info","source":"guardian","message":"guardian.api.garden-server.get-properties.got-properties","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"3.1.4"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.647386807Z","level":"info","source":"guardian","message":"guardian.run.started","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","path":"/opt/resource/check","session":"39"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.647635445Z","level":"info","source":"guardian","message":"guardian.run.exec.start","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","path":"/opt/resource/check","session":"39.1"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.648605027Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.start","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","path":"/opt/resource/check","session":"39.1.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.660151821Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.execrunner.start","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","id":"c30ded01-4d7b-4e3e-7109-9cfb17dca9a8","path":"/opt/resource/check","session":"39.1.2.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.661658509Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.execrunner.done","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","id":"c30ded01-4d7b-4e3e-7109-9cfb17dca9a8","path":"/opt/resource/check","session":"39.1.2.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.661717741Z","level":"info","source":"guardian","message":"guardian.run.exec.exec-with-bndl.finished","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","path":"/opt/resource/check","session":"39.1.2"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.661733894Z","level":"info","source":"guardian","message":"guardian.run.exec.finished","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","path":"/opt/resource/check","session":"39.1"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.661749414Z","level":"info","source":"guardian","message":"guardian.run.finished","data":{"handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","path":"/opt/resource/check","session":"39"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.661764677Z","level":"error","source":"guardian","message":"guardian.api.garden-server.run.failed","data":{"error":"EOF","handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"3.1.5"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6822]: {"timestamp":"2023-03-28T16:15:59.665402792Z","level":"info","source":"atc","message":"atc.tracker-imb.run.errored","data":{"build":"check","error":"run check: start process: backend error: Exit status: 500, message: {"Type":"","Message":"EOF","Handle":"","ProcessID":"","Binary":""}\n","pipeline":"some-pipeline","pre_build_id":1,"resource":"mockery","session":"23.1","team":"watsjs-team-9c095818-38d6-4f24-9ab9-ea2b723477a8"}}

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6822]: {"timestamp":"2023-03-28T16:15:59.672070276Z","level":"info","source":"atc","message":"atc.tracker-imb.run.finish.errored","data":{"build":"check","build_id":1,"error":"run check: start process: backend error: Exit status: 500, message: {"Type":"","Message":"EOF","Handle":"","ProcessID":"","Binary":""}\n","pipeline":"some-pipeline","pre_build_id":1,"resource":"mockery","session":"23.1.5","team":"watsjs-team-9c095818-38d6-4f24-9ab9-ea2b723477a8"}}

It seems garden runs the check process with an unknown error.

Mar 28 16:15:59 smoke-sacred-foxhound concourse[6876]: {"timestamp":"2023-03-28T16:15:59.661764677Z","level":"error","source":"guardian","message":"guardian.api.garden-server.run.failed","data":{"error":"EOF","handle":"d47bf858-3602-47bb-4a42-0f1d7d2a1c85","session":"3.1.5"}}

Given the logs of garden setting up network guardian.create.network.config-create above, i feel like there is some issues about the network configuration so the concourse image checking error by networking failure.

The error EOF is similar to an issue we just noticed recently when running a local Concourse with guardian runtime, where the image checking in a garden container failed by verifying the host public key due to Github's public key rotation (after flush out the host public key the issue is resolved).