ARMv7 not working
ptoump opened this issue · 13 comments
Hi,
on ARMv7 (raspberry pi 3b) the docker image keeps restarting since the last update.
on amd64 it works fine.
Log:
Can you please show me how the container is started? What command or docker-compose yaml?
It would be very interesting to see a ls -lashi on the folder /var/lib/exim4 in the container - basically you need to start the container like this:
$ docker run --rm -it registry.gitlab.com/ix.ai/smtp:latest ls -lashi /var/lib/exim4
Unable to find image 'registry.gitlab.com/ix.ai/smtp:latest' locally
latest: Pulling from ix.ai/smtp
39cf15d1b231: Already exists
1cff220308df: Pull complete
0098406c9585: Pull complete
Digest: sha256:837e05432a92c89814c3d95d95f1029650918f31eb4523dd28ed93e8ce3f217b
Status: Downloaded newer image for registry.gitlab.com/ix.ai/smtp:latest
+ sed -ri '
s/^#?(dc_local_interfaces)=.*/\1='\''[0.0.0.0]:25 ; [::0]:25'\''/;
s/^#?(dc_other_hostnames)=.*/\1='\'''\''/;
s/^#?(dc_relay_nets)=.*/\1='\''172.17.0.2\/16'\''/;
s/^#?(dc_eximconfig_configtype)=.*/\1='\''internet'\''/;
' /etc/exim4/update-exim4.conf.conf
+ update-exim4.conf -v
using non-split configuration scheme from /etc/exim4/exim4.conf.template
total 48K
10232991 8.0K drwxr-xr-x 1 root root 4.0K Sep 21 05:15 .
10232878 8.0K drwxr-xr-x 1 root root 4.0K Sep 8 20:44 ..
10232992 4.0K -rw-r--r-- 1 root root 4 Sep 8 20:44 berkeleydbvers.txt
10233062 28K -rw-r--r-- 1 root Debian-exim 26K Sep 21 05:15 config.autogenerated
Edit: however, in case it doesn't work, you need to run it with:
$ docker run --rm -it --entrypoint /bin/ls registry.gitlab.com/ix.ai/smtp:latest -lashi /var/lib/exim4
total 40K
10232991 4.0K drwxr-xr-x 2 root root 4.0K Sep 8 20:44 .
10232878 4.0K drwxr-xr-x 1 root root 4.0K Sep 8 20:44 ..
10232992 4.0K -rw-r--r-- 1 root root 4 Sep 8 20:44 berkeleydbvers.txt
10232993 28K -rw-r--r-- 1 root Debian-exim 26K Sep 8 20:44 config.autogenerated
Unfortunately, I don't have any rpi 3 to test - and all my experiments starting ARMv7 with qemu were a complete failure - however I can't understand how, using the same build process, on different archs the path isn't consistently available.
Here you go:
without the entrypoint it generates the same error..
I have pinned the older version either way, as the SSL verification breaks things for gmail in my case..
After researching it a bit, I found that the script /usr/sbin/update-exim4.conf is generating the error. This script is installed as part of the exim4-config package.
However, since the error points at "setting times", I am thinking that it has something to do with a noatime option in the fstab file of the rpi-server. But in order to verify this, I will need a few more pieces of information.
I've prepared a new dev image, that should display two things in addition to what we currently see: the user that runs the update-exim4.conf script and the mounts.
To run it, please use:
docker run --rm -it --env VERBOSE=true registry.gitlab.com/ix.ai/smtp:dev-branch
I'm very curious about the output.
In parallel, I'll look into the upstream changes made to /usr/sbin/update-exim4.conf between v0.0.5 and v0.0.6.
Here you go:
panos@rpi-server:~ $ sudo docker run --rm -it --env VERBOSE=true registry.gitlab.com/ix.ai/smtp:dev-branch
[sudo] password for panos:
Unable to find image 'registry.gitlab.com/ix.ai/smtp:dev-branch' locally
dev-branch: Pulling from ix.ai/smtp
c82ec4692e56: Already exists
3296c0c70fcc: Pull complete
0a8b1689d540: Pull complete
Digest: sha256:74085624e002d4e714a73ce56226227c74deb48f63b047dfd25b7ef1c97b8737
Status: Downloaded newer image for registry.gitlab.com/ix.ai/smtp:dev-branch
Running as user 'root'
+ mount
+ grep -vE 'sys|proc|devpts|mqueue|tmpfs'
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/L3H6L4TH4UYSA6CDALH6KD6NTT:/var/lib/docker/overlay2/l/FCZEHOXCKIGXIQX7Z4ODPD52VL:/var/lib/docker/overlay2/l/I7GQMYYBYDYNY6SHOSMOFESQ5L:/var/lib/docker/overlay2/l/NBUENRZQY4K5EIAVVWLDQVOZ23,upperdir=/var/lib/docker/overlay2/ebb313b88863b35be0aa4bbdf5c7c843349cc9b1510809e47eed92fc74611f02/diff,workdir=/var/lib/docker/overlay2/ebb313b88863b35be0aa4bbdf5c7c843349cc9b1510809e47eed92fc74611f02/work)
/dev/root on /etc/resolv.conf type ext4 (rw,noatime,commit=600)
/dev/root on /etc/hostname type ext4 (rw,noatime,commit=600)
/dev/root on /etc/hosts type ext4 (rw,noatime,commit=600)
+ sed -ri '
s/^#?(dc_local_interfaces)=.*/\1='\''[0.0.0.0]:25 ; [::0]:25'\''/;
s/^#?(dc_other_hostnames)=.*/\1='\'''\''/;
s/^#?(dc_relay_nets)=.*/\1='\''172.17.0.4\/16'\''/;
s/^#?(dc_eximconfig_configtype)=.*/\1='\''internet'\''/;
' /etc/exim4/update-exim4.conf.conf
+ update-exim4.conf -v
using non-split configuration scheme from /etc/exim4/exim4.conf.template
touch: setting times of '/var/lib/exim4/config.autogenerated.tmp': Operation not permitted
noatime and delayed commit are critical for the longevity of the sdcard (if you don't want to end up with corrupted SDs every couple of months).
No idea why they need to set the access time though..
Thanks for your efforts!
So, the diff between the two is:
20c20
< UPEX4C_version="4.92-8+deb10u4"
---
> UPEX4C_version="4.94.2-7"
109c109
< UPEX4C_tmp="$(tempfile -m600 -p ex4)"
---
> UPEX4C_tmp="$(mktemp)"
So, my theory is that, for whatever reason, tempfile and mktemp create different types of files. And this only for ARMv7, since on ARM64 and on AMD64 it works flawlessly (yes, I also have noatime everywhere).
I guess I'll need to expand the verbosity to inject a set -x in /usr/sbin/update-exim4.conf.
As soon as the pipeline is completed for the latest commit, please pull the dev-branch image again and run it again :-)
Thanks
The pipeline: https://gitlab.com/ix.ai/smtp/-/pipelines/375163937
Here you go:
panos@rpi-server:~ $ sudo docker run --rm -it --env VERBOSE=true registry.gitlab.com/ix.ai/smtp:dev-branch
Running as user 'root'
+ mount
+ grep -vE 'sys|proc|devpts|mqueue|tmpfs'
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/UII5UAN6MBAY2H4SZJCLQRO4LK:/var/lib/docker/overlay2/l/A3L7OMSQO6XM43M7QWTQCFNK7L:/var/lib/docker/overlay2/l/6GX6IJYXXMUURW43BCMD7ZUJJ2:/var/lib/docker/overlay2/l/NBUENRZQY4K5EIAVVWLDQVOZ23,upperdir=/var/lib/docker/overlay2/9222a872a4726aba831349ef4987b66c830f7a93999c3eaba645638f1f9ddb31/diff,workdir=/var/lib/docker/overlay2/9222a872a4726aba831349ef4987b66c830f7a93999c3eaba645638f1f9ddb31/work)
/dev/root on /etc/resolv.conf type ext4 (rw,noatime,commit=600)
/dev/root on /etc/hostname type ext4 (rw,noatime,commit=600)
/dev/root on /etc/hosts type ext4 (rw,noatime,commit=600)
+ sed -ri '
s/^#?(dc_local_interfaces)=.*/\1='\''[0.0.0.0]:25 ; [::0]:25'\''/;
s/^#?(dc_other_hostnames)=.*/\1='\'''\''/;
s/^#?(dc_relay_nets)=.*/\1='\''172.17.0.4\/16'\''/;
s/^#?(dc_eximconfig_configtype)=.*/\1='\''internet'\''/;
' /etc/exim4/update-exim4.conf.conf
+ update-exim4.conf.debug
+ . /usr/sbin/update-exim4.conf
+ set -e
+ set -C
+ set -f
+ UPEX4C_confdir=/etc/exim4
+ UPEX4C_sections=main acl router transport retry rewrite auth
+ UPEX4C_semicolon=dc_local_interfaces dc_relay_nets dc_relay_domains
+ EXIM=/usr/sbin/exim4
+ UPEX4C_verbose=no
+ UPEX4C_autoconfigfile=/var/lib/exim4/config.autogenerated
+ UPEX4C_outputfile=/var/lib/exim4/config.autogenerated
+ UPEX4C_version=4.94.2-7
+ getopt -n update-exim4.conf -l check,keepcomments,removecomments,output:,confdir:,help,verbose -- +o:d:vh
+ TEMP= --
+ test 0 != 0
+ eval set -- --
+ set -- --
+ test -- != --
+ shift
+ [ 0 -ne 0 ]
+ [ -e /etc/exim4/exim4.conf ]
+ UE4CC=/etc/exim4/update-exim4.conf.conf
+ UPEX4C_confd=/etc/exim4/conf.d
+ dirname /var/lib/exim4/config.autogenerated
+ [ -d /var/lib/exim4 ]
+ [ -f /etc/exim4/update-exim4.conf.conf ]
+ . /etc/exim4/update-exim4.conf.conf
+ dc_eximconfig_configtype=internet
+ dc_other_hostnames=
+ dc_local_interfaces=[0.0.0.0]:25 ; [::0]:25
+ dc_readhost=
+ dc_relay_domains=
+ dc_minimaldns=false
+ dc_relay_nets=172.17.0.4/16
+ dc_smarthost=
+ CFILEMODE=644
+ dc_use_split_config=false
+ dc_hide_mailname=
+ dc_mailname_in_oh=true
+ dc_localdelivery=mail_spool
+ UPEX4C_autoconfigfile=/var/lib/exim4/config.autogenerated
+ dirname /var/lib/exim4/config.autogenerated
+ [ /var/lib/exim4 = /var/lib/exim4 ]
+ UPEX4C_tmp=/var/lib/exim4/config.autogenerated.tmp
+ [ 644 = ]
+ [ false = ]
+ [ mail_spool = ]
+ [ = ]
+ UPEX4C_comments=no
+ TEMPLATEFILE=/etc/exim4/exim4.conf.template
+ lowercase false
+ echo false
+ lowerpipe
+ tr A-Z a-z
+ dc_use_split_config=false
+ lowercase no
+ echo no
+ lowerpipe
+ tr A-Z a-z
+ UPEX4C_verbose=no
+ [ false = true ]
+ [ no = yes ]
+ lowerpipe
+ tr A-Z a-z
+ sed -n s/\([-[:alnum:]@\.]\+\).*/\1/;p;q
+ check_ascii_pipe
+ cat
+ IN=localhost
+ echo localhost
+ sed s/[^-0-9ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\/\.!*@_~:;< \[\]]/_/g
+ OUT=localhost
+ [ localhost != localhost ]
+ echo localhost
+ mailname=localhost
+ echo [0.0.0.0]:25 ; [::0]:25 172.17.0.4/16
+ grep -q [[:space:]]\(partial-\)\?\(cdb\|dbm\|dbmnz\|\(d\|ipl\|\(n\?wild\)\?l\)search\|nis\)\([\*@]\)\?[[:space:]]*;
+ lowercase
+ check_ascii_pipe
+ echo
+ lowerpipe
+ tr A-Z a-z
+ cat
+ IN=
+ echo
+ sed s/[^-0-9ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz\/\.!*@_~:;< \[\]]/_/g
+ OUT=
+ [ != ]
+ echo
+ dc_other_hostnames=
+ echo @:localhost:
+ sed -e s/[;: ]*$// -e s/ *// -e s/;/:/g
+ local_domains=@:localhost
+ gentmpconf
+ rm -f /var/lib/exim4/config.autogenerated.tmp
+ touch /var/lib/exim4/config.autogenerated.tmp
touch: setting times of '/var/lib/exim4/config.autogenerated.tmp': Operation not permitted
also, if that helps:
panos@rpi-server:~/test $ stat test
File: test
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: b302h/45826d Inode: 257822 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1001/ panos) Gid: ( 1001/ panos)
Access: 2021-09-22 14:38:06.795980877 +0300
Modify: 2021-09-22 14:38:06.795980877 +0300
Change: 2021-09-22 14:38:06.795980877 +0300
Birth: -
panos@rpi-server:~/test $ touch -a test
panos@rpi-server:~/test $ stat test
File: test
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: b302h/45826d Inode: 257822 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1001/ panos) Gid: ( 1001/ panos)
Access: 2021-09-22 14:38:40.685782744 +0300
Modify: 2021-09-22 14:38:06.795980877 +0300
Change: 2021-09-22 14:38:40.685782744 +0300
Birth: -
panos@rpi-server:~/test $ touch test
panos@rpi-server:~/test $ stat test
File: test
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: b302h/45826d Inode: 257822 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1001/ panos) Gid: ( 1001/ panos)
Access: 2021-09-22 14:39:06.455632135 +0300
Modify: 2021-09-22 14:39:06.455632135 +0300
Change: 2021-09-22 14:39:06.455632135 +0300
Birth: -
Well, I'm a bit at loss here. I've seen that coreutils on arm64 and armhf is exactly the same version (8.32-4).
What would make sense is also to run touch manually in the docker container - to see if it works:
$ docker run --rm -it --entrypoint bash registry.gitlab.com/ix.ai/smtp:dev-branch
Unable to find image 'registry.gitlab.com/ix.ai/smtp:dev-branch' locally
dev-branch: Pulling from ix.ai/smtp
1a5fc71c2293: Already exists
cdd5ba83ae35: Pull complete
23f8124ff7c6: Pull complete
Digest: sha256:deab968983f885e6e3a660070a7c2a28be893a868c8980fb7e1f1aa1b916bf1f
Status: Downloaded newer image for registry.gitlab.com/ix.ai/smtp:dev-branch
root@dc065d9d92a9:/# touch /tmp/foo
root@dc065d9d92a9:/# stat /tmp/foo
File: /tmp/foo
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 5eh/94d Inode: 3271135 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-09-22 17:07:30.461986341 +0000
Modify: 2021-09-22 17:07:30.461986341 +0000
Change: 2021-09-22 17:07:30.461986341 +0000
Birth: 2021-09-22 17:07:30.461986341 +0000
Next, if it doesn't work, I suggest doing the same with the Debian stable and stable-slim images (output on arm64):
docker run --rm -it --entrypoint bash debian:stable
Unable to find image 'debian:stable' locally
stable: Pulling from library/debian
97d29e55b1af: Pull complete
Digest: sha256:a9cb4a9ddf9f28bc17fc390baba42ac7eb067ae54d20b55720ed9ff3323b1d87
Status: Downloaded newer image for debian:stable
root@8ef18109ceb8:/# touch /foo
root@8ef18109ceb8:/# stat /foo
File: /foo
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 5eh/94d Inode: 3536227 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-09-22 17:09:59.239923100 +0000
Modify: 2021-09-22 17:09:59.239923100 +0000
Change: 2021-09-22 17:09:59.239923100 +0000
Birth: 2021-09-22 17:09:59.239923100 +0000
root@8ef18109ceb8:/#
If I were to have a RPI 3, I would start running touch with strace next and open up a ticket with Debian (https://bugs.debian.org/cgi-bin/pkgreport.cgi?package=coreutils)
Yeap, that settles it, it's a debian bug.
I can't even install strace to get something useful.
Tested with alpine and touch works fine, as well as the oldstable debian..
panos@rpi-server:~ $ sudo docker run --rm -it --entrypoint bash debian:unstable-slim
Unable to find image 'debian:unstable-slim' locally
unstable-slim: Pulling from library/debian
f66a36b205dd: Pull complete
Digest: sha256:3a5ad7e8053c28e2a35ec243f854b7bc4420b163c4ffffcf9cfc753496b5e726
Status: Downloaded newer image for debian:unstable-slim
root@feceaa39ff50:/# cd tmp
root@feceaa39ff50:/tmp# touch foo
touch: setting times of 'foo': Operation not permitted
root@feceaa39ff50:/tmp# exit
exit
^C
panos@rpi-server:~ $ sudo docker run --rm -it --entrypoint bash debian:oldstable-slim
Unable to find image 'debian:oldstable-slim' locally
oldstable-slim: Pulling from library/debian
e62a1d82a4e6: Pull complete
Digest: sha256:1dc7ae745df559de6006f9048633723a1c28175f10d5f7b1a778323e4fa4e3b1
Status: Downloaded newer image for debian:oldstable-slim
root@a12f4049a7a6:/# cd tmp
root@a12f4049a7a6:/tmp# touch foo
root@a12f4049a7a6:/tmp# exit
exit
panos@rpi-server:~ $ sudo docker run --rm -it --entrypoint bash debian:stable-slim
Unable to find image 'debian:stable-slim' locally
stable-slim: Pulling from library/debian
c82ec4692e56: Already exists
Digest: sha256:709a5b8e7d08d14bb1fcce67f4f9c23ea7ad79442cbc81b6f8800ed492d90d77
Status: Downloaded newer image for debian:stable-slim
root@6d43155fbba5:/# cd tmp
root@6d43155fbba5:/tmp# touch foo
touch: setting times of 'foo': Operation not permitted
root@6d43155fbba5:/tmp#
Thanks for your time to go through this!
Can you please tell me details about the host OS?
Which flavour is it and what kernel is it running. I plan on opening a ticket with Debian for this.
Thanks
Hi
sorry for the long delay, I lost my SDCard (lost as in got tired and went read only), so I was waiting for a replacement..
Fresh install : Linux rpi3 5.10.60-v7+ #1449 SMP Wed Aug 25 15:00:01 BST 2021 armv7l (Raspbian)
I have just rebuilt the image with the tag v0.1.3. If the problem still persists, please give it a try.
This issue has been open long enough. Closing it 🙂