Build Testing
Closed this issue · 20 comments
So nice that they cut us off on the forum... Appreciate the work you have been doing in documenting.
I was going through testing this evening and got to the point of building the first package (dropbear). It is failing to pull the custom container images from the local registry.
22:15:06 Cloning repository https://github.com/vyos/vyos-build.git
[Pipeline] sh
22:14:39 + docker pull 10.1.1.94:5000/vyos/vyos-build:sagitta
22:14:39 Error response from daemon: Get "http://10.1.1.94:5000/v2/": dial tcp 10.1.1.94:5000: connect: connection refused
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withDockerRegistry
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
22:14:39 Failed in branch amd64
Is there a step missing for setting up docker to listen on network port.
I may just try pushing to my private repo on Docker Hub tomorrow.
Suggestions?
isaac @ behrenshome . com
Yeah, seems like overreaction is their motto.
let's focus focus only on equuleus. Skip sagitta references for now.
Jenkins doesn't have access to your local registry. This is related to part:
Build patched vyos-build docker image and create local registry
https://github.com/dd010101/vyos-jenkins#build-patched-vyos-build-docker-image-and-create-local-registry
and
Declarative Pipeline (Docker)
https://github.com/dd010101/vyos-jenkins?tab=readme-ov-file#configure-dev_packages_vyos_net_host-variable-and-add-global-vyos-build-jenkins-library
You need start local registry, otherwise jenkins will just use dockerhub that's the docker run -d -p 5000:5000 --restart always --name registry registry:2.7 command, and this is where you have problem, you specified Declarative Pipeline (Docker) and that's seems ok but jenkins can't connect for some reason to the registry, is the local registry running? docker ps -a? Do you have port 5000 on 10.1.1.94 open?
The port 5000 acts like webserver and returns brackets if you probe curl 10.1.1.94:5000/v2/ && echo:
root@vyos-jenkins:~# curl 172.17.17.17:5000/v2/ && echo
{}
Make sure registry is listening where it should.
Private docker hub isn't needed, I did succeed just by using the local registry alone.
EDIT: I did verify how this works and it should work as said, there is one additional step but that comes after so you didn't get there yet, with the original instructions you may end-up with error Error response from daemon: manifest for 172.17.17.17:5000/vyos/vyos-build:equuleus not found: manifest unknown: manifest unknown because I omitted the part where you push the created images to the local registry, see readme for details https://github.com/dd010101/vyos-jenkins?tab=readme-ov-file#build-patched-vyos-build-docker-image-and-create-local-registry
I was just about to send the PR about pushing the container to the registry.
I focused on sagitta packages, and I think I was able to build most of them using the official docker image:
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/t/telegraf/telegraf_1.28.3-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libn/libnftnl/libnftnl11_1.2.6-2_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libn/libnftnl/libnftnl11-dbgsym_1.2.6-2_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libn/libnftnl/libnftnl-dev-doc_1.2.6-2_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libn/libnftnl/libnftnl-dev_1.2.6-2_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libr/librtr0/librtr-doc_0.8.0_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libr/librtr0/librtr-dbg_0.8.0_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libr/librtr0/rtr-tools-dbg_0.8.0_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libr/librtr0/rtr-tools_0.8.0_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libr/librtr0/librtr-dev_0.8.0_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/libr/librtr0/librtr0_0.8.0_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/f/frr/frr-snmp_9.1-160-g488c106e7_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/f/frr/frr-rpki-rtrlib_9.1-160-g488c106e7_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/f/frr/frr-doc_9.1-160-g488c106e7_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/f/frr/frr-dbgsym_9.1-160-g488c106e7_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/f/frr/frr-pythontools_9.1-160-g488c106e7_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/f/frr/frr-snmp-dbgsym_9.1-160-g488c106e7_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/f/frr/frr_9.1-160-g488c106e7_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/f/frr/frr-rpki-rtrlib-dbgsym_9.1-160-g488c106e7_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/v/vyos-linux-firmware/vyos-linux-firmware_20231211_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/v/vyos-intel-qat/vyos-intel-qat_4.24.0-00005-0_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/v/vyos-intel-ixgbe/vyos-intel-ixgbe_5.20.3_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/v/vyos-intel-ixgbevf/vyos-intel-ixgbevf_4.18.9_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/v/vyos-1x/vyos-1x-vmware_1.4.0-epa2-374-gef31f7025_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/v/vyos-1x/vyos-1x-dbgsym_1.4.0-epa2-374-gef31f7025_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/v/vyos-1x/vyos-1x-smoketest_1.4.0-epa2-374-gef31f7025_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/v/vyos-1x/vyos-1x_1.4.0-epa2-374-gef31f7025_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/a/accel-ppp/accel-ppp_1.12.0-260-g19c36e5_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/l/linux-upstream/linux-libc-dev_6.6.31-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/l/linux-upstream/linux-headers-6.6.31-amd64-vyos_6.6.31-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/l/linux-upstream/linux-image-6.6.31-amd64-vyos_6.6.31-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/o/openvpn-otp/openvpn-otp_1.0-4-g47f8ccf_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/o/opennhrp/opennhrp_0.14-20-g613277f_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/o/openvpn-dco/openvpn-dco_0.2.20231117_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/r/radvd/radvd_2.20-rc1-23-gf2de476_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/n/ndppd/ndppd-dbgsym_0.2.5-6_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/n/ndppd/ndppd_0.2.5-6_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/n/nftables/libnftables1_1.0.9-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/n/nftables/libnftables1-dbgsym_1.0.9-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/n/nftables/libnftables-dev_1.0.9-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/n/nftables/python3-nftables_1.0.9-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/n/nftables/nftables_1.0.9-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/n/nftables/nftables-dbgsym_1.0.9-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/e/ethtool/ethtool-dbgsym_6.6-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/e/ethtool/ethtool_6.6-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/p/pyhumps/python3-pyhumps_3.8.0-1_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/p/pmacct/pmacct_1.7.7-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/p/pmacct/pmacct-dbgsym_1.7.7-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/d/ddclient/ddclient_3.11.2+vyos0_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/d/dropbear/dropbear-bin-dbgsym_2022.83-1+deb12u1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/d/dropbear/dropbear-bin_2022.83-1+deb12u1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/d/dropbear/dropbear-run_2022.83-1+deb12u1_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/d/dropbear/dropbear_2022.83-1+deb12u1_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/d/dropbear/dropbear-initramfs_2022.83-1+deb12u1_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/liby/libyang2/libyang-tools_2.1.148-1_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/liby/libyang2/libyang2-dev_2.1.148-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/liby/libyang2/libyang2-dbgsym_2.1.148-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/liby/libyang2/libyang2-tools-dbgsym_2.1.148-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/liby/libyang2/libyang2-tools_2.1.148-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/liby/libyang2/libyang2_2.1.148-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wpa/wpasupplicant-dbgsym_2.10-1028-g6b9c86466_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wpa/eapoltest-dbgsym_2.10-1028-g6b9c86466_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wpa/hostapd_2.10-1028-g6b9c86466_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wpa/wpasupplicant_2.10-1028-g6b9c86466_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wpa/libwpa-client-dev_2.10-1028-g6b9c86466_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wpa/eapoltest_2.10-1028-g6b9c86466_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wpa/hostapd-dbgsym_2.10-1028-g6b9c86466_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wide-dhcpv6/wide-dhcpv6-client_20080615-23_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wide-dhcpv6/wide-dhcpv6-server-dbgsym_20080615-23_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wide-dhcpv6/wide-dhcpv6-relay-dbgsym_20080615-23_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wide-dhcpv6/wide-dhcpv6-client-dbgsym_20080615-23_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wide-dhcpv6/wide-dhcpv6-server_20080615-23_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/w/wide-dhcpv6/wide-dhcpv6-relay_20080615-23_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/h/hsflowd/hsflowd_2.0.55-1_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/k/keepalived/keepalived-dbgsym_2.2.8-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/k/keepalived/keepalived_2.2.8-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-relay_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-client-ddns_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-server_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-dev_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-client_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-server-ldap_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-server-dbgsym_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-relay-dbgsym_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-relay-dbgsym_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-client-ddns-dbgsym_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-common_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-server-ldap-dbgsym_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-keama_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-keama-dbgsym_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/i/isc-dhcp/isc-dhcp-client-dbgsym_4.4.3-P1-4_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/j/jool/jool-dbgsym_4.1.9+bf4c7e3669-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/pool/main/j/jool/jool_4.1.9+bf4c7e3669-1_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/equuleus/pool/main/d/dropbear/dropbear-bin_2019.78-2_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/equuleus/pool/main/d/dropbear/dropbear-bin-dbgsym_2019.78-2_amd64.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/equuleus/pool/main/d/dropbear/dropbear_2019.78-2_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/equuleus/pool/main/d/dropbear/dropbear-initramfs_2019.78-2_all.deb
/home/sentrium/web/dev.packages.vyos.net/public_html/repositories/equuleus/pool/main/d/dropbear/dropbear-run_2019.78-2_all.deb
Will try to build the ISO later to check out what is missing.
Indeed push was omitted but that isn't where Issac ended, seems like he gets connection issue before docker even tries to pull.
I did update readme with bunch info about sagitta few moments ago including about packages that are missing. I managed to build 171 of 190 .debs for sagitta (with the current info
ISO build for sagitta fails now with random error:
Setting up libc6:amd64 (2.36-9+deb12u7) ...
/usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: No such file or directory
dpkg: error processing package libc6:amd64 (--configure):
installed libc6:amd64 package post-installation script subprocess returned error exit status 127
Seem like vyos-build from vyos is broken right now? Doesn't even start to build ISO and fails right away on preparation. Installing libcrypt1 doesn't help.
Building my own vyos-build resolves the libcrypto nonsense, so indeed the vyos-build from vyos is broken. But 171 is not enough obviously, ISO stops at libnss-tacplus and python3-vici and I don't know where vyos did get those, see issues.md for details. Those missing don't look like vyos specific, perhaps they are just pulled from other places like debian or other repositories without any vyos changes?
EDIT: in last hours lot of progress happened thanks to work of other people it's now possible to build all equuleus and sagitta packages, sagitta requires some that aren't even in vyos repositories, will update tomorrow with details
Yeah, seems like overreaction is their motto.
let's focus focus only on equuleus. Skip sagitta references for now.
Jenkins doesn't have access to your local registry. This is related to part: Build patched vyos-build docker image and create local registry https://github.com/dd010101/vyos-jenkins#build-patched-vyos-build-docker-image-and-create-local-registry and Declarative Pipeline (Docker) https://github.com/dd010101/vyos-jenkins?tab=readme-ov-file#configure-dev_packages_vyos_net_host-variable-and-add-global-vyos-build-jenkins-library
You need start local registry, otherwise jenkins will just use dockerhub that's the
docker run -d -p 5000:5000 --restart always --name registry registry:2.7command, and this is where you have problem, you specified Declarative Pipeline (Docker) and that's seems ok but jenkins can't connect for some reason to the registry, is the localregistryrunning? docker ps -a? Do you have port 5000 on 10.1.1.94 open?The port 5000 acts like webserver and returns brackets if you probe
curl 10.1.1.94:5000/v2/ && echo:root@vyos-jenkins:~# curl 172.17.17.17:5000/v2/ && echo {}Make sure registry is listening where it should.
Private docker hub isn't needed, I did succeed just by using the local registry alone.
EDIT: I did verify how this works and it should work as said, there is one additional step but that comes after so you didn't get there yet, with the original instructions you may end-up with error
Error response from daemon: manifest for 172.17.17.17:5000/vyos/vyos-build:equuleus not found: manifest unknown: manifest unknownbecause I omitted the part where you push the created images to the local registry, see readme for details https://github.com/dd010101/vyos-jenkins?tab=readme-ov-file#build-patched-vyos-build-docker-image-and-create-local-registry
Thanks for the pointers... I missed that last docker run command. Ran that and have the registry running and pushed images. Was able to build dropbear before bed last night. Will push on later today...
I did need to add a step to have Docker access insecure registries. Is that just me or should it be added?
ibehr@ubujnk01:~$ cat /etc/docker/daemon.json
{
"insecure-registries": [
"localhost:5000",
"10.1.1.94:5000",
"ubujnk01.iac.lan:5000"
]
}Insecure registry was missing. Added. It's required but I had it already configured from other time.
Continued.... making good progress. Couple of packages that are not building.
Seems like these built for you.
OWAMP sagitta
14:53:16 + cd ..
14:53:16 + ./build.sh
14:53:16 + pwd
14:53:16 + CWD=/var/lib/jenkins/workspace/vyos-jenkins_owamp_sagitta/build-amd64/packages/owamp
14:53:16 + set -e
14:53:16 + SRC=owamp
14:53:16 + [ ! -d owamp ]
14:53:16 + cd owamp
14:53:16 + echo I: Retrieve version information from Git
14:53:16 I: Retrieve version information from Git
14:53:16 + echo I: Build VyOS owamp Packages
14:53:16 I: Build VyOS owamp Packages
14:53:16 + dpkg-buildpackage -us -uc -tc -b
14:53:16 dpkg-buildpackage: info: source package owamp
14:53:16 dpkg-buildpackage: info: source version 4.4.6-1
14:53:16 dpkg-buildpackage: info: source distribution perfsonar-4.4
14:53:16 dpkg-buildpackage: info: source changed by perfSONAR developers <debian@perfsonar.net>
14:53:16 dpkg-source --before-build .
14:53:16 dpkg-buildpackage: info: host architecture amd64
14:53:17 dpkg-source: info: using patch list from debian/patches/series
14:53:17 dpkg-source: info: applying autotools.diff
14:53:17 dpkg-source: info: applying man.diff
14:53:17 dpkg-checkbuilddeps: error: Unmet build dependencies: dh-apparmor dh-exec libcap-dev
14:53:17 dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
14:53:17 dpkg-buildpackage: warning: (Use -d flag to override.)
vyos-cloud-init equuleus
10:32:48 Commit message: "Merge pull request #72 from zdc/T6007-equuleus"
[Pipeline] sh
10:32:49 + git rev-parse --short=11 HEAD
[Pipeline] sh
10:32:49 + ./packages/bddeb
10:32:50 Traceback (most recent call last):
10:32:50 File "./packages/bddeb", line 315, in <module>
10:32:50 sys.exit(main())
10:32:50 File "./packages/bddeb", line 242, in main
10:32:50 ver_data = read_version()
10:32:50 File "./packages/bddeb", line 136, in read_version
10:32:50 return json.loads(run_helper('read-version', ['--json']))
10:32:50 File "./packages/bddeb", line 58, in run_helper
10:32:50 (stdout, _stderr) = subp.subp(cmd)
10:32:50 File "/var/lib/jenkins/workspace/jenkins_vyos-cloud-init_equuleus/build-amd64/cloudinit/subp.py", line 333, in subp
10:32:50 stdout=out, stderr=err, exit_code=rc, cmd=args
10:32:50 cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
10:32:50 Command: ['/var/lib/jenkins/workspace/jenkins_vyos-cloud-init_equuleus/build-amd64/tools/read-version', '--json']
10:32:50 Exit code: 1
10:32:50 Reason: -
10:32:50 Stdout:
10:32:50 Stderr: git describe version (None) differs from cloudinit.version (22.1)
10:32:50 Please get the latest upstream tags.
10:32:50 As an example, this can be done with the following:
10:32:50 $ git remote add upstream https://git.launchpad.net/cloud-init
10:32:50 $ git fetch upstream --tags
vyos-cloud-init sagitta
12:42:50 Commit message: "Merge pull request #75 from zdc/T6038-sagitta"
[Pipeline] sh
12:42:51 + git rev-parse --short=11 HEAD
[Pipeline] sh
12:42:51 + ./packages/bddeb
12:42:52 Traceback (most recent call last):
12:42:52 File "/var/lib/jenkins/workspace/_jenkins_vyos-cloud-init_sagitta/build-amd64/./packages/bddeb", line 315, in <module>
12:42:52 sys.exit(main())
12:42:52 ^^^^^^
12:42:52 File "/var/lib/jenkins/workspace/_jenkins_vyos-cloud-init_sagitta/build-amd64/./packages/bddeb", line 242, in main
12:42:52 ver_data = read_version()
12:42:52 ^^^^^^^^^^^^^^
12:42:52 File "/var/lib/jenkins/workspace/_jenkins_vyos-cloud-init_sagitta/build-amd64/./packages/bddeb", line 136, in read_version
12:42:52 return json.loads(run_helper('read-version', ['--json']))
12:42:52 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
12:42:52 File "/var/lib/jenkins/workspace/_jenkins_vyos-cloud-init_sagitta/build-amd64/./packages/bddeb", line 58, in run_helper
12:42:52 (stdout, _stderr) = subp.subp(cmd)
12:42:52 ^^^^^^^^^^^^^^
12:42:52 File "/var/lib/jenkins/workspace/_jenkins_vyos-cloud-init_sagitta/build-amd64/cloudinit/subp.py", line 332, in subp
12:42:52 raise ProcessExecutionError(
12:42:52 cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
12:42:52 Command: ['/var/lib/jenkins/workspace/_jenkins_vyos-cloud-init_sagitta/build-amd64/tools/read-version', '--json']
12:42:52 Exit code: 1
12:42:52 Reason: -
12:42:52 Stdout:
12:42:52 Stderr: git describe version (None) differs from cloudinit.version (22.1)
12:42:52 Please get the latest upstream tags.
12:42:52 As an example, this can be done with the following:
12:42:52 $ git remote add upstream https://git.launchpad.net/cloud-init
12:42:52 $ git fetch upstream --tags
All three are known and resolved.
The first one needs patched sagitta vyos-build, see readme to build and push both sagitta and current image.
The last two need extra jenkins pipeline configuration, see Readme - Pipeline - Behaviours -> Add -> Advanced clone behaviours.
The first one needs patched sagitta vyos-build, see readme to build and push both sagitta and current image.
I assume these are changes since last night? I am rebuilding the customer containers now....
The last two need extra jenkins pipeline configuration, see Readme - Pipeline - Behaviours -> Add -> Advanced clone behaviours.
Crap -- I had selected Advance checkout behaviors. -- Correcting that fixed!
Thanks for all the help!!!
I assume these are changes since last night
It's possible, the development is very rapid currently, in last few hours a lot discovered thanks to other people so the readme is already outdated but in form of additional packages so the current packages and technique stays the same.
I have packages build and nginx up and running.
When I try to build against the new repo I get:
P: Configuring file /etc/apt/sources.list
Ign:1 http://ubujnk01.iac.lan:8000/sagitta current InRelease
Err:2 http://ubujnk01.iac.lan:8000/sagitta current Release
404 Not Found [IP: 10.1.1.94 8000]
Frustrated because all seems well when I browse the repo or use curl.
Did you see such at all?
Maybe there is difference between how you are accessing it and how docker is (from different subnet?). I didn't have such issue but my nginx was very plain so it may change if your configuration is more complex. What is your nginx configuration?
It for sure seems like nginx configuration issue - like the docker is seeing different vhost than you do and that's why the difference.
Pretty much what you had, just put it in a container. The part that confuses me is that I can browse and curl from it... but apt throws the issue.
ibehr@ubujnk01:~/nginx$ ls
docker-compose.yml nginx.conf
ibehr@ubujnk01:~/nginx$ cat docker-compose.yml
version: '3'
services:
web:
image: nginx
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- /home/sentrium/web/dev.packages.vyos.net/public_html/repositories:/repositories:ro
ports:
- "8000:80"
environment:
- NGINX_HOST=ubujnk01.iac.lan
- NGINX_PORT=8000
ibehr@ubujnk01:~/nginx$ cat nginx.conf
events {
}
http {
server {
listen 80;
listen [::]:80;
root /repositories;
autoindex on;
server_name _;
location / {
try_files $uri $uri/ =404;
}
location ~ /(.*)/conf {
deny all;
}
location ~ /(.*)/db {
deny all;
}
}
}
That seems ok.
Do you see the files apt tried to access in /home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta?
root@vyos-jenkins:~# ls -alh /home/sentrium/web/dev.packages.vyos.net/public_html/repositories/sagitta/dists/sagitta/
total 28K
drwxr-xr-x 4 jenkins jenkins 4.0K May 20 21:12 .
drwxr-xr-x 3 jenkins jenkins 4.0K May 15 13:39 ..
drwxr-xr-x 4 jenkins jenkins 4.0K May 15 13:39 contrib
-rw-r--r-- 1 jenkins jenkins 2.3K May 20 21:12 InRelease
drwxr-xr-x 4 jenkins jenkins 4.0K May 15 13:39 main
-rw-r--r-- 1 jenkins jenkins 1.6K May 20 21:12 Release
-rw-r--r-- 1 jenkins jenkins 659 May 20 21:12 Release.gpg
EDIT: I maybe see the issue sagitta current Release.
APT tries to access current release, that's not right.
Are you building from vyos-build sagitta docker?
I hoppen into the container where I am running the build and get this, which I think looks right...
vyos_bld@4fa387be9a3a:/etc/apt/sources.list.d$ curl http://ubujnk01.iac.lan:8000/sagitta/dists/sagitta/Release
Origin: sagitta
Label: sagitta
Codename: sagitta
Date: Wed, 22 May 2024 05:53:07 UTC
Architectures: amd64
Components: main
Description: sagitta
MD5Sum:
cefd70e1ae0fb800c51b02a0631aa60e 39788 main/binary-amd64/Packages
5976cfe90f943dcf90263beac8a42157 12882 main/binary-amd64/Packages.gz
80c3123ddb1b29ff856e42001ebbf70e 88 main/binary-amd64/Release
89d50b0f60a42834963d769b57f7c698 12005 main/source/Sources
94a92fdbdf678baf5f8d9ccae599f120 3834 main/source/Sources.gz
a3d6ad8e7deaeaa14055b2e87feba128 89 main/source/Release
SHA1:
7ad18727cc1bd7795cc3836e098bfd35a61b56c8 39788 main/binary-amd64/Packages
ea48b0753659572287a454e8223547622195e041 12882 main/binary-amd64/Packages.gz
31c57d51a273d634c86cdde57ffe19988d4b3f29 88 main/binary-amd64/Release
e53426c2a75cc3d58fc64020bc4f33bcad5a826b 12005 main/source/Sources
70d0f53b0782108c88daa92ae01694c7870c3237 3834 main/source/Sources.gz
2b6fe52f5913ff59578bb2906fe70ab4203e0872 89 main/source/Release
SHA256:
002b28a10949cda6b3ec83f3d55ef913ff254df836c312e60064e206328dd623 39788 main/binary-amd64/Packages
581661bfd975a68c75451856b6011ccb3ce219bf360a78ec620465fbbe4129a8 12882 main/binary-amd64/Packages.gz
af2ea8bbacdd4098868857cf55735020e8845e751e86fd4e0bbf4a1172f84a3f 88 main/binary-amd64/Release
b839d23043dfb22463a39bd3fab6cdaa21e531aa154e7184d9c76c4f394ab1f4 12005 main/source/Sources
7013178a1baa8282689ef7dd4ddd961452c5263745017420936afeec4eafddf6 3834 main/source/Sources.gz
aa983e4159c95205a4c38d10db3b7990f637e1a9f6e58fbf3a1499160bad6b7a 89 main/source/Release
My build scripts are this:
ibehr@ubufoc01:~/vyos-build$ cat autobuild.sh
#!/bin/bash
# VyOS Build Branch (Rolling Release=current, Stable=equuleus, 1.2=crux)
#BUILD_BRANCH="current"
BUILD_BRANCH="sagitta"
# This should be your cloned vyos/vyos-build git repo
PROJECT_DIR="/home/ibehr/vyos-build"
# Your home directory
HOME_DIR="/home/ibehr"
# Final .iso file destination: Set this to where you'd like
ISO_DEST="/home/ibehr/vyos-iso"
# Check if ISO_DEST exists, if not, then create.
if [ ! -d "$ISO_DEST" ] ; then
mkdir -p $ISO_DEST
fi
##
## Do the Work
##
# Change to $PROJECT_DIR
cd $PROJECT_DIR
# Ensure we have the latest updates to the build scripts
/usr/bin/git pull
# Pull & start the vyos-build:current container, starting isobuild.sh at launch
#/usr/bin/docker pull vyos/vyos-build:$BUILD_BRANCH && /usr/bin/docker run --rm -t \
/usr/bin/docker pull ubujnk01.iac.lan:5000/vyos/vyos-build:$BUILD_BRANCH && /usr/bin/docker run --rm -t \
-v "$PROJECT_DIR":/vyos \
-v "$HOME_DIR/.gitconfig":/etc/gitconfig \
-v "$HOME_DIR/.bash_aliases":/home/vyos_bld/.bash_aliases \
-v "$HOME_DIR/.bashrc":/home/vyos_bld/.bashrc \
-v "$PROJECT_DIR/apt.gpg.key":/opt/apt.gpg.key \
-w /vyos --privileged --sysctl net.ipv6.conf.lo.disable_ipv6=0 \
-e GOSU_UID=$(id -u) -e GOSU_GID=$(id -g) \
vyos/vyos-build:$BUILD_BRANCH bash ./isobuild.sh $BUILD_BRANCH
Then here is the isobuild that runs in the container.
ibehr@ubufoc01:~/vyos-build$ cat isobuild.sh
#!/bin/bash
BUILD_BRANCH=$1
# Build Information
BUILD_BY="isaac@behrenshome.com"
BUILD_TYPE="release"
VERSION="$(TZ='UTC' date +%Y.%m.%d)-$BUILD_BRANCH"
ARCH="amd64"
BUILD_COMMENT="Integrated packages: bat, dnsutils, git, python3-pip"
# Custom Packages
PKG1="bat"
PKG2="dnsutils"
PKG3="git"
PKG4="python3-pip"
PKG5="bind9"
PKG6="sysstat"
PKG7="qemu-guest-agent"
PKG8="awscli"
PKG9="prometheus-node-exporter"
# Configure, build & create ISO
sudo ./build-vyos-image \
--build-by "$BUILD_BY" \
--build-type "$BUILD_TYPE" \
--version "$VERSION" \
--architecture "$ARCH" \
--custom-package "$PKG1" \
--custom-package "$PKG2" \
--custom-package "$PKG3" \
--custom-package "$PKG4" \
--custom-package "$PKG5" \
--custom-package "$PKG6" \
--custom-package "$PKG7" \
--custom-package "$PKG8" \
--custom-package "$PKG9" \
--build-comment "$BUILD_COMMENT" \
--vyos-mirror http://ubujnk01.iac.lan:8000/sagitta \
--custom-apt-key /opt/apt.gpg.key \
iso
Yep, the webserver stuff is all correct but the docker container is trying to access current release instead of sagitta. I would try to rebuild the sagitta docker container to be sure it's sagitta since I don't have any other explanation why would sagitta access current release.
Will keep digging... thanks for the sanity checks!
Do you ever sleep? :-) Guess you could ask me the same given the time...
Do you ever sleep? :-) Guess you could ask me the same given the time...
It's illusion because I'm 7 hours ahead - and it just happens you work on it in time that matches when I wakeup - it's 8AM here.
Have you seen this error?
I got all of the build running and successfully build ISOs.
When booting the ISO I get to Grub and select either option.
Next I get this output and the VM just hangs.
Likewise, I can use the ISO on an existing install and complete an add system image. All is good until rebooting into the new environment and boot hangs.
I imagine it is something to do with the kernel or initramfs.
No, I didn't see anything like that. It's hard to say what is wrong since you don't see anything... No display generally means issue with GPU driver - even if there was crash/freeze you should see something right?
I didn't see any issues with the ISOs I created, once the ISO did built then it just worked.
Did you try the fail-safe mode option to force basic GPU driver? This also just freezes without any output? You can try attach serial console to the VM and watch if the OS writes something there as replacement for display. If I recall correctly the VyOS image is configured to use first serial port with 115200 baud rate.
How do you build the ISO? Do you delete the build directory each run (in the vyos-build repo, from where you starting the build)? If you don't then please try. I did see issue with failing ISO build before that was caused by make clean not doing it's job correctly.
As generic debugging steps you should make sure you have everything up to date - you have fresh vyos-build docker container image, you are using latest vyos-build repository, there are no failing builds of packages in Jenkins...
I tried my latest sagitta ISO (two weeks old) and that one boots as expected.
So I tried to build another fresh sagitta ISO and this one also boots.
So I tried to delete build cache and build another 100% fresh sagitta ISO and this one also boots.
I don't have much to suggest since there are no pointers - we need more information - blank screen alone doesn't tell much.
As always, thanks for your feedback! I was able to get my images to boot by changing the VM to basic VGA.
Interesting thing on the updates to existing systems. The 2 that I was testing with were running rolling 1.5 release. Seems that while you can add an image for Sagitta to them it will not boot well. I find this interesting as in the past I had gone back and forth between rolling 1.5 and back to 1.4.
Thank you sooooo much for all the work that you have put in to (re)creating the VyOS build process for the community!!!