Bug: stacker internal unmount cmd fails in CI runs
Closed this issue · 3 comments
rchincha commented
stacker version
v1.0.0-rc7
Describe the bug
https://github.com/project-stacker/stacker/actions/runs/6806024931/job/18506608262
# 838 23 0:64 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/c rw,relatime shared:349 - overlay overlay ro,lowerdir=/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756,xino=on,userxattr
# mountinfo after umount a^
# Debug mode:
# stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
# stacker subcommand: [/home/runner/work/stacker/stacker/stacker --debug internal-go atomfs umount c]
# device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
# device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
# device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
# device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
# device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
# device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
To reproduce
- Configuration
- Client tool used
- Seen error
Expected behavior
No response
Screenshots
No response
Additional context
No response
smoser commented
For more reference, a full log of the failed test case test/atomfs.bats "mount + umount + mount a tree of images works" is attached here.
2023-11-09T01:18:52.0917015Z not ok 8 mount + umount + mount a tree of images works in 42sec
2023-11-09T01:18:52.0918633Z # (from function `stacker' in file test/helpers.bash, line 87,
2023-11-09T01:18:52.0919943Z # in test file test/atomfs.bats, line 109)
2023-11-09T01:18:52.1045434Z # `stacker internal-go atomfs umount c' failed
2023-11-09T01:18:52.1046080Z # Debug mode:
2023-11-09T01:18:52.1046615Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1048090Z # nsexec-ing [-- /home/runner/work/stacker/stacker/stacker --internal-userns --debug build --layer-type=squashfs]
2023-11-09T01:18:52.1049423Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1050149Z # no previous storage type detected
2023-11-09T01:18:52.1123146Z # initializing stacker recipe: stacker.yaml
2023-11-09T01:18:52.1126195Z # substituting $STACKER_ROOTFS_DIR to /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots
2023-11-09T01:18:52.1129959Z # substituting $STACKER_STACKER_DIR to /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker
2023-11-09T01:18:52.1133729Z # substituting $STACKER_OCI_DIR to /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci
2023-11-09T01:18:52.1136259Z # substituting $STACKER_WORK_DIR to
2023-11-09T01:18:52.1137849Z # stacker build order:
2023-11-09T01:18:52.1140515Z # 0 build /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/stacker.yaml: requires: []
2023-11-09T01:18:52.1145110Z # building: 0 /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/stacker.yaml
2023-11-09T01:18:52.1150622Z # substituting $STACKER_ROOTFS_DIR to /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots
2023-11-09T01:18:52.1154414Z # substituting $STACKER_STACKER_DIR to /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker
2023-11-09T01:18:52.1158943Z # substituting $STACKER_OCI_DIR to /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci
2023-11-09T01:18:52.1161822Z # substituting $STACKER_WORK_DIR to
2023-11-09T01:18:52.1163527Z # Dependency Order [base a b c]
2023-11-09T01:18:52.1165172Z # preparing image base...
2023-11-09T01:18:52.1166009Z # overlay-dirs, possibly modified after import: []
2023-11-09T01:18:52.1167172Z # loading oci:/home/runner/work/stacker/stacker/test/centos:latest
2023-11-09T01:18:52.1172047Z # unpacking to /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/base
2023-11-09T01:18:52.1176449Z # lxc rootfs overlay arg overlayfs:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/base/overlay
2023-11-09T01:18:52.1179770Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1185255Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --oci-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci --roots-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots --stacker-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker --storage-type overlay --debug internal-go check-aa-profile lxc-container-default-cgns]
2023-11-09T01:18:52.1191145Z # bind mounting /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker/imports/base into container
2023-11-09T01:18:52.1192681Z # + touch /base
2023-11-09T01:18:52.1193597Z # creating oci image base-squashfs (type=squashfs+verity) by converting base (type=tar)
... [mksquashfs output]
2023-11-09T01:18:52.1241709Z # new oci layer sha256:2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756 [squashfs+verity] created from path /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1 as part of base:latest
2023-11-09T01:18:52.1247632Z # Generating overlay_dirs layers for base
... [mksquashfs output]
2023-11-09T01:18:52.1301894Z # generated squashfs+verity layer sha256:29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea from /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/base/overlay
2023-11-09T01:18:52.1307617Z # renaming /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/base/overlay -> /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea/overlay
2023-11-09T01:18:52.1310855Z # setting git version annotation to 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1311732Z # filesystem base built successfully
2023-11-09T01:18:52.1312283Z # preparing image a...
2023-11-09T01:18:52.1312949Z # overlay-dirs, possibly modified after import: []
2023-11-09T01:18:52.1319734Z # lxc rootfs overlay arg overlayfs:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/base/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/a/overlay
2023-11-09T01:18:52.1325948Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1335606Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --oci-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci --roots-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots --stacker-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker --storage-type overlay --debug internal-go check-aa-profile lxc-container-default-cgns]
2023-11-09T01:18:52.1342229Z # bind mounting /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker/imports/a into container
2023-11-09T01:18:52.1343713Z # + touch /a
2023-11-09T01:18:52.1345360Z # Running image copy to oci:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci:a-squashfs
2023-11-09T01:18:52.1377963Z # Generating overlay_dirs layers for a
... [mksquashfs output]
2023-11-09T01:18:52.1400787Z # generated squashfs+verity layer sha256:7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb from /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/a/overlay
2023-11-09T01:18:52.1405845Z # renaming /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/a/overlay -> /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb/overlay
2023-11-09T01:18:52.1409438Z # setting git version annotation to 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1410299Z # filesystem a built successfully
2023-11-09T01:18:52.1410823Z # preparing image b...
2023-11-09T01:18:52.1411467Z # overlay-dirs, possibly modified after import: []
2023-11-09T01:18:52.1418204Z # lxc rootfs overlay arg overlayfs:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/base/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/b/overlay
2023-11-09T01:18:52.1424032Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1429507Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --oci-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci --roots-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots --stacker-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker --storage-type overlay --debug internal-go check-aa-profile lxc-container-default-cgns]
2023-11-09T01:18:52.1435345Z # bind mounting /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker/imports/b into container
2023-11-09T01:18:52.1436882Z # + touch /b
2023-11-09T01:18:52.1438422Z # Running image copy to oci:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci:b-squashfs
2023-11-09T01:18:52.1440398Z # Generating overlay_dirs layers for b
... [mksquashfs output]
2023-11-09T01:18:52.1462343Z # generated squashfs+verity layer sha256:54b150d6de23e4c058cf54d731807449a8749664449e210babc0f89099e124ad from /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/b/overlay
2023-11-09T01:18:52.1467319Z # renaming /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/b/overlay -> /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_54b150d6de23e4c058cf54d731807449a8749664449e210babc0f89099e124ad/overlay
2023-11-09T01:18:52.1470534Z # setting git version annotation to 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1471385Z # filesystem b built successfully
2023-11-09T01:18:52.1471907Z # preparing image c...
2023-11-09T01:18:52.1472523Z # overlay-dirs, possibly modified after import: []
2023-11-09T01:18:52.1479277Z # lxc rootfs overlay arg overlayfs:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/base/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756/overlay:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/c/overlay
2023-11-09T01:18:52.1485244Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1492363Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --oci-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci --roots-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots --stacker-dir /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker --storage-type overlay --debug internal-go check-aa-profile lxc-container-default-cgns]
2023-11-09T01:18:52.1498356Z # bind mounting /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/.stacker/imports/c into container
2023-11-09T01:18:52.1499865Z # + touch /c
2023-11-09T01:18:52.1501415Z # Running image copy to oci:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci:c-squashfs
2023-11-09T01:18:52.1502994Z # Generating overlay_dirs layers for c
... [mksquashfs output]
2023-11-09T01:18:52.1524462Z # generated squashfs+verity layer sha256:f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a from /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/c/overlay
2023-11-09T01:18:52.1529532Z # renaming /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/c/overlay -> /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/roots/sha256_f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a/overlay
2023-11-09T01:18:52.1532595Z # setting git version annotation to 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1533485Z # filesystem c built successfully
2023-11-09T01:18:52.1534004Z # Debug mode:
2023-11-09T01:18:52.1534536Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1536050Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --debug internal-go atomfs mount a-squashfs a]
2023-11-09T01:18:52.1547576Z # about to mount {[{application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb 8192 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:99af6cde1a67fa3bd96f5c11ffca3b7a9762feca5225fa9e9e68d19cb0f5d67a] [] <nil> } {application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea 8192 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:32d0774e17db2977ef2aead422b3d850c0302c1b02e0084e869804c084ecd6ed] [] <nil> } {application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756 73691136 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:15ba479c5dd1a5dc2f3eff7035a206afc070f9dfa41f0e8d904e7287ac62bd60] [] <nil> }] {/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata a-squashfs a true}}
2023-11-09T01:18:52.1556575Z # Debug mode:
2023-11-09T01:18:52.1557159Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1558687Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --debug internal-go atomfs mount b-squashfs b]
2023-11-09T01:18:52.1569904Z # about to mount {[{application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:54b150d6de23e4c058cf54d731807449a8749664449e210babc0f89099e124ad 8192 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:55ebc5ea0ad86a3fca903b0456df2166463fa133a8d52bfec174f0e083a56a86] [] <nil> } {application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea 8192 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:32d0774e17db2977ef2aead422b3d850c0302c1b02e0084e869804c084ecd6ed] [] <nil> } {application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756 73691136 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:15ba479c5dd1a5dc2f3eff7035a206afc070f9dfa41f0e8d904e7287ac62bd60] [] <nil> }] {/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata b-squashfs b true}}
2023-11-09T01:18:52.1579550Z # 23 1 8:17 / / rw,relatime shared:1 - ext4 /dev/root rw,discard,errors=remount-ro
2023-11-09T01:18:52.1581197Z # 24 23 0:5 / /dev rw,nosuid,noexec,relatime shared:2 - devtmpfs devtmpfs rw,size=8180192k,nr_inodes=2045048,mode=755,inode64
2023-11-09T01:18:52.1582741Z # 25 23 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
2023-11-09T01:18:52.1583958Z # 26 23 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:6 - sysfs sysfs rw
2023-11-09T01:18:52.1585417Z # 27 26 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:7 - securityfs securityfs rw
2023-11-09T01:18:52.1586860Z # 28 24 0:23 / /dev/shm rw,nosuid,nodev shared:3 - tmpfs tmpfs rw,inode64
2023-11-09T01:18:52.1588491Z # 29 24 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
2023-11-09T01:18:52.1590434Z # 30 23 0:25 / /run rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,size=3273008k,nr_inodes=819200,mode=755,inode64
2023-11-09T01:18:52.1592574Z # 31 30 0:26 / /run/lock rw,nosuid,nodev,noexec,relatime shared:12 - tmpfs tmpfs rw,size=5120k,inode64
2023-11-09T01:18:52.1649759Z # 32 26 0:27 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:8 - cgroup2 cgroup2 rw
2023-11-09T01:18:52.1651273Z # 33 26 0:28 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:9 - pstore pstore rw
2023-11-09T01:18:52.1652683Z # 34 26 0:29 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:10 - bpf bpf rw,mode=700
2023-11-09T01:18:52.1654639Z # 35 25 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2335
2023-11-09T01:18:52.1656524Z # 36 24 0:31 / /dev/hugepages rw,relatime shared:14 - hugetlbfs hugetlbfs rw,pagesize=2M
2023-11-09T01:18:52.1659977Z # 37 24 0:19 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:15 - mqueue mqueue rw
2023-11-09T01:18:52.1661526Z # 38 26 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:16 - debugfs debugfs rw
2023-11-09T01:18:52.1663437Z # 39 26 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw
2023-11-09T01:18:52.1665231Z # 40 26 0:32 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:18 - fusectl fusectl rw
2023-11-09T01:18:52.1666909Z # 41 26 0:20 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:19 - configfs configfs rw
2023-11-09T01:18:52.1668998Z # 63 30 0:33 / /run/credentials/systemd-sysusers.service ro,nosuid,nodev,noexec,relatime shared:20 - ramfs ramfs rw,mode=700
2023-11-09T01:18:52.1671196Z # 88 23 7:0 / /snap/core20/2015 ro,nodev,relatime shared:43 - squashfs /dev/loop0 ro,errors=continue,threads=single
2023-11-09T01:18:52.1673054Z # 91 23 7:1 / /snap/lxd/24322 ro,nodev,relatime shared:45 - squashfs /dev/loop1 ro,errors=continue,threads=single
2023-11-09T01:18:52.1674904Z # 94 23 7:2 / /snap/snapd/20290 ro,nodev,relatime shared:47 - squashfs /dev/loop2 ro,errors=continue,threads=single
2023-11-09T01:18:52.1677159Z # 97 23 8:31 / /boot/efi rw,relatime shared:49 - vfat /dev/sdb15 rw,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
2023-11-09T01:18:52.1679283Z # 100 35 0:35 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:51 - binfmt_misc binfmt_misc rw
2023-11-09T01:18:52.1680997Z # 294 23 8:1 / /mnt rw,relatime shared:134 - ext4 /dev/sda1 rw
2023-11-09T01:18:52.1682511Z # 365 30 0:25 /snapd/ns /run/snapd/ns rw,nosuid,nodev - tmpfs tmpfs rw,size=3273008k,nr_inodes=819200,mode=755,inode64
2023-11-09T01:18:52.1684810Z # 646 365 0:4 mnt:[4026532219] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
2023-11-09T01:18:52.1686683Z # 647 30 0:45 / /run/user/1001 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=1636500k,nr_inodes=409125,mode=700,uid=1001,gid=127,inode64
2023-11-09T01:18:52.1695519Z # 392 23 0:44 / /var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/merged rw,relatime shared:277 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/EZEZS3LLL3RKCZARCUQTLOQSDX:/var/lib/docker/overlay2/l/IKVX3LCU3S73PFXJN4PITNOZZ6:/var/lib/docker/overlay2/l/PBJEKG2SNNVWY2AK6BH6WZ6QBA:/var/lib/docker/overlay2/l/UWNPK2CVL7WD76CRDXKFFFZGLG:/var/lib/docker/overlay2/l/PUWPXDNCIS2YUQOTC5XC6WE7V5:/var/lib/docker/overlay2/l/TZGQFKLW2ANQ7CEWOX57KI3SEY,upperdir=/var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/diff,workdir=/var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/work,nouserxattr
2023-11-09T01:18:52.1703119Z # 640 30 0:4 net:[4026532230] /run/docker/netns/c6dd3f79365e rw shared:285 - nsfs nsfs rw
2023-11-09T01:18:52.1704863Z # 609 23 0:57 / /var/lib/lxcfs rw,nosuid,nodev,relatime shared:293 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2023-11-09T01:18:52.1709324Z # 742 23 253:0 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb ro,relatime shared:302 - squashfs /dev/mapper/7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.1715779Z # 760 23 253:2 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea ro,relatime shared:310 - squashfs /dev/mapper/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.1722513Z # 778 23 253:3 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756 ro,relatime shared:318 - squashfs /dev/mapper/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.1733328Z # 799 23 0:59 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/a rw,relatime shared:333 - overlay overlay ro,lowerdir=/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756,xino=on,userxattr
2023-11-09T01:18:52.1743534Z # 817 23 253:4 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/54b150d6de23e4c058cf54d731807449a8749664449e210babc0f89099e124ad ro,relatime shared:341 - squashfs /dev/mapper/54b150d6de23e4c058cf54d731807449a8749664449e210babc0f89099e124ad-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.1754427Z # 838 23 0:64 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/b rw,relatime shared:349 - overlay overlay ro,lowerdir=/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/54b150d6de23e4c058cf54d731807449a8749664449e210babc0f89099e124ad:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756,xino=on,userxattr
2023-11-09T01:18:52.1761296Z # mountinfo after b^
2023-11-09T01:18:52.1761746Z # Debug mode:
2023-11-09T01:18:52.1762292Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1764063Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --debug internal-go atomfs umount b]
2023-11-09T01:18:52.1765136Z # Debug mode:
2023-11-09T01:18:52.1765693Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1767294Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --debug internal-go atomfs mount c-squashfs c]
2023-11-09T01:18:52.1778069Z # about to mount {[{application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a 8192 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:5cbc39ac27255ed990d58b85e14d43b2838c4d99a5f4750a626ce998d76a9ff0] [] <nil> } {application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea 8192 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:32d0774e17db2977ef2aead422b3d850c0302c1b02e0084e869804c084ecd6ed] [] <nil> } {application/vnd.stacker.image.layer.squashfs+zstd+verity sha256:2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756 73691136 [] map[io.stackeroci.stacker.squashfs_verity_root_hash:15ba479c5dd1a5dc2f3eff7035a206afc070f9dfa41f0e8d904e7287ac62bd60] [] <nil> }] {/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/oci /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata c-squashfs c true}}
2023-11-09T01:18:52.1787652Z # 23 1 8:17 / / rw,relatime shared:1 - ext4 /dev/root rw,discard,errors=remount-ro
2023-11-09T01:18:52.1789336Z # 24 23 0:5 / /dev rw,nosuid,noexec,relatime shared:2 - devtmpfs devtmpfs rw,size=8180192k,nr_inodes=2045048,mode=755,inode64
2023-11-09T01:18:52.1790920Z # 25 23 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
2023-11-09T01:18:52.1792170Z # 26 23 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:6 - sysfs sysfs rw
2023-11-09T01:18:52.1793940Z # 27 26 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:7 - securityfs securityfs rw
2023-11-09T01:18:52.1795392Z # 28 24 0:23 / /dev/shm rw,nosuid,nodev shared:3 - tmpfs tmpfs rw,inode64
2023-11-09T01:18:52.1796836Z # 29 24 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
2023-11-09T01:18:52.1798545Z # 30 23 0:25 / /run rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,size=3273008k,nr_inodes=819200,mode=755,inode64
2023-11-09T01:18:52.1800405Z # 31 30 0:26 / /run/lock rw,nosuid,nodev,noexec,relatime shared:12 - tmpfs tmpfs rw,size=5120k,inode64
2023-11-09T01:18:52.1801998Z # 32 26 0:27 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:8 - cgroup2 cgroup2 rw
2023-11-09T01:18:52.1803472Z # 33 26 0:28 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:9 - pstore pstore rw
2023-11-09T01:18:52.1804937Z # 34 26 0:29 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:10 - bpf bpf rw,mode=700
2023-11-09T01:18:52.1806954Z # 35 25 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2335
2023-11-09T01:18:52.1808830Z # 36 24 0:31 / /dev/hugepages rw,relatime shared:14 - hugetlbfs hugetlbfs rw,pagesize=2M
2023-11-09T01:18:52.1810257Z # 37 24 0:19 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:15 - mqueue mqueue rw
2023-11-09T01:18:52.1811755Z # 38 26 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:16 - debugfs debugfs rw
2023-11-09T01:18:52.1813338Z # 39 26 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw
2023-11-09T01:18:52.1814915Z # 40 26 0:32 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:18 - fusectl fusectl rw
2023-11-09T01:18:52.1816462Z # 41 26 0:20 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:19 - configfs configfs rw
2023-11-09T01:18:52.1818315Z # 63 30 0:33 / /run/credentials/systemd-sysusers.service ro,nosuid,nodev,noexec,relatime shared:20 - ramfs ramfs rw,mode=700
2023-11-09T01:18:52.1820697Z # 88 23 7:0 / /snap/core20/2015 ro,nodev,relatime shared:43 - squashfs /dev/loop0 ro,errors=continue,threads=single
2023-11-09T01:18:52.1822523Z # 91 23 7:1 / /snap/lxd/24322 ro,nodev,relatime shared:45 - squashfs /dev/loop1 ro,errors=continue,threads=single
2023-11-09T01:18:52.1824350Z # 94 23 7:2 / /snap/snapd/20290 ro,nodev,relatime shared:47 - squashfs /dev/loop2 ro,errors=continue,threads=single
2023-11-09T01:18:52.1826511Z # 97 23 8:31 / /boot/efi rw,relatime shared:49 - vfat /dev/sdb15 rw,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
2023-11-09T01:18:52.1828599Z # 100 35 0:35 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:51 - binfmt_misc binfmt_misc rw
2023-11-09T01:18:52.1829947Z # 294 23 8:1 / /mnt rw,relatime shared:134 - ext4 /dev/sda1 rw
2023-11-09T01:18:52.1831492Z # 365 30 0:25 /snapd/ns /run/snapd/ns rw,nosuid,nodev - tmpfs tmpfs rw,size=3273008k,nr_inodes=819200,mode=755,inode64
2023-11-09T01:18:52.1832967Z # 646 365 0:4 mnt:[4026532219] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
2023-11-09T01:18:52.1834745Z # 647 30 0:45 / /run/user/1001 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=1636500k,nr_inodes=409125,mode=700,uid=1001,gid=127,inode64
2023-11-09T01:18:52.1843460Z # 392 23 0:44 / /var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/merged rw,relatime shared:277 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/EZEZS3LLL3RKCZARCUQTLOQSDX:/var/lib/docker/overlay2/l/IKVX3LCU3S73PFXJN4PITNOZZ6:/var/lib/docker/overlay2/l/PBJEKG2SNNVWY2AK6BH6WZ6QBA:/var/lib/docker/overlay2/l/UWNPK2CVL7WD76CRDXKFFFZGLG:/var/lib/docker/overlay2/l/PUWPXDNCIS2YUQOTC5XC6WE7V5:/var/lib/docker/overlay2/l/TZGQFKLW2ANQ7CEWOX57KI3SEY,upperdir=/var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/diff,workdir=/var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/work,nouserxattr
2023-11-09T01:18:52.1851101Z # 640 30 0:4 net:[4026532230] /run/docker/netns/c6dd3f79365e rw shared:285 - nsfs nsfs rw
2023-11-09T01:18:52.1852852Z # 609 23 0:57 / /var/lib/lxcfs rw,nosuid,nodev,relatime shared:293 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2023-11-09T01:18:52.1857134Z # 742 23 253:0 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb ro,relatime shared:302 - squashfs /dev/mapper/7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.1863456Z # 760 23 253:2 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea ro,relatime shared:310 - squashfs /dev/mapper/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.1869922Z # 778 23 253:3 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756 ro,relatime shared:318 - squashfs /dev/mapper/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.1880556Z # 799 23 0:59 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/a rw,relatime shared:333 - overlay overlay ro,lowerdir=/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/7285c5907bedc3076b0ec6167e91868c9c8c1b088a7308d13b268dca78dd7bcb:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756,xino=on,userxattr
2023-11-09T01:18:52.1893624Z # 817 23 253:4 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a ro,relatime shared:341 - squashfs /dev/mapper/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.1906943Z # 838 23 0:64 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/c rw,relatime shared:349 - overlay overlay ro,lowerdir=/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756,xino=on,userxattr
2023-11-09T01:18:52.1915793Z # mountinfo after c^
2023-11-09T01:18:52.1916450Z # Debug mode:
2023-11-09T01:18:52.1917265Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.1919036Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --debug internal-go atomfs umount a]
2023-11-09T01:18:52.1920907Z # 23 1 8:17 / / rw,relatime shared:1 - ext4 /dev/root rw,discard,errors=remount-ro
2023-11-09T01:18:52.1922606Z # 24 23 0:5 / /dev rw,nosuid,noexec,relatime shared:2 - devtmpfs devtmpfs rw,size=8180192k,nr_inodes=2045048,mode=755,inode64
2023-11-09T01:18:52.1924231Z # 25 23 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:5 - proc proc rw
2023-11-09T01:18:52.1926193Z # 26 23 0:22 / /sys rw,nosuid,nodev,noexec,relatime shared:6 - sysfs sysfs rw
2023-11-09T01:18:52.1927674Z # 27 26 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:7 - securityfs securityfs rw
2023-11-09T01:18:52.1929110Z # 28 24 0:23 / /dev/shm rw,nosuid,nodev shared:3 - tmpfs tmpfs rw,inode64
2023-11-09T01:18:52.1930483Z # 29 24 0:24 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
2023-11-09T01:18:52.1932124Z # 30 23 0:25 / /run rw,nosuid,nodev shared:11 - tmpfs tmpfs rw,size=3273008k,nr_inodes=819200,mode=755,inode64
2023-11-09T01:18:52.1933706Z # 31 30 0:26 / /run/lock rw,nosuid,nodev,noexec,relatime shared:12 - tmpfs tmpfs rw,size=5120k,inode64
2023-11-09T01:18:52.1935204Z # 32 26 0:27 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:8 - cgroup2 cgroup2 rw
2023-11-09T01:18:52.1936641Z # 33 26 0:28 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:9 - pstore pstore rw
2023-11-09T01:18:52.1938080Z # 34 26 0:29 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:10 - bpf bpf rw,mode=700
2023-11-09T01:18:52.1939971Z # 35 25 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:13 - autofs systemd-1 rw,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=2335
2023-11-09T01:18:52.1941762Z # 36 24 0:31 / /dev/hugepages rw,relatime shared:14 - hugetlbfs hugetlbfs rw,pagesize=2M
2023-11-09T01:18:52.1943458Z # 37 24 0:19 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:15 - mqueue mqueue rw
2023-11-09T01:18:52.1944952Z # 38 26 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:16 - debugfs debugfs rw
2023-11-09T01:18:52.1946948Z # 39 26 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:17 - tracefs tracefs rw
2023-11-09T01:18:52.1948688Z # 40 26 0:32 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:18 - fusectl fusectl rw
2023-11-09T01:18:52.1950300Z # 41 26 0:20 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:19 - configfs configfs rw
2023-11-09T01:18:52.1966264Z # 63 30 0:33 / /run/credentials/systemd-sysusers.service ro,nosuid,nodev,noexec,relatime shared:20 - ramfs ramfs rw,mode=700
2023-11-09T01:18:52.1968980Z # 88 23 7:0 / /snap/core20/2015 ro,nodev,relatime shared:43 - squashfs /dev/loop0 ro,errors=continue,threads=single
2023-11-09T01:18:52.1971138Z # 91 23 7:1 / /snap/lxd/24322 ro,nodev,relatime shared:45 - squashfs /dev/loop1 ro,errors=continue,threads=single
2023-11-09T01:18:52.1973600Z # 94 23 7:2 / /snap/snapd/20290 ro,nodev,relatime shared:47 - squashfs /dev/loop2 ro,errors=continue,threads=single
2023-11-09T01:18:52.1976402Z # 97 23 8:31 / /boot/efi rw,relatime shared:49 - vfat /dev/sdb15 rw,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
2023-11-09T01:18:52.1979085Z # 100 35 0:35 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:51 - binfmt_misc binfmt_misc rw
2023-11-09T01:18:52.1980851Z # 294 23 8:1 / /mnt rw,relatime shared:134 - ext4 /dev/sda1 rw
2023-11-09T01:18:52.1982842Z # 365 30 0:25 /snapd/ns /run/snapd/ns rw,nosuid,nodev - tmpfs tmpfs rw,size=3273008k,nr_inodes=819200,mode=755,inode64
2023-11-09T01:18:52.1984575Z # 646 365 0:4 mnt:[4026532219] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
2023-11-09T01:18:52.1986938Z # 647 30 0:45 / /run/user/1001 rw,nosuid,nodev,relatime shared:297 - tmpfs tmpfs rw,size=1636500k,nr_inodes=409125,mode=700,uid=1001,gid=127,inode64
2023-11-09T01:18:52.1998047Z # 392 23 0:44 / /var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/merged rw,relatime shared:277 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/EZEZS3LLL3RKCZARCUQTLOQSDX:/var/lib/docker/overlay2/l/IKVX3LCU3S73PFXJN4PITNOZZ6:/var/lib/docker/overlay2/l/PBJEKG2SNNVWY2AK6BH6WZ6QBA:/var/lib/docker/overlay2/l/UWNPK2CVL7WD76CRDXKFFFZGLG:/var/lib/docker/overlay2/l/PUWPXDNCIS2YUQOTC5XC6WE7V5:/var/lib/docker/overlay2/l/TZGQFKLW2ANQ7CEWOX57KI3SEY,upperdir=/var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/diff,workdir=/var/lib/docker/overlay2/09ac1e05120091f8c2f87c5c72fa0413ddd61d549fea20f7a12f6a252d61d2dd/work,nouserxattr
2023-11-09T01:18:52.2007304Z # 640 30 0:4 net:[4026532230] /run/docker/netns/c6dd3f79365e rw shared:285 - nsfs nsfs rw
2023-11-09T01:18:52.2009221Z # 609 23 0:57 / /var/lib/lxcfs rw,nosuid,nodev,relatime shared:293 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
2023-11-09T01:18:52.2013760Z # 760 23 253:2 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea ro,relatime shared:310 - squashfs /dev/mapper/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.2044148Z # 778 23 253:3 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756 ro,relatime shared:318 - squashfs /dev/mapper/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.2051159Z # 817 23 253:4 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a ro,relatime shared:341 - squashfs /dev/mapper/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity ro,errors=continue,threads=single
2023-11-09T01:18:52.2061793Z # 838 23 0:64 / /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/c rw,relatime shared:349 - overlay overlay ro,lowerdir=/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/29782e958f744613968a83f29c9ea17ff0122a716bc7db99deed5659da396cea:/home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/atomfs-metadata/mounts/2c49fa22fb6cf5930ead7e6b6e1db7636174537de0e1cd9806c2dbee6dd2e756,xino=on,userxattr
2023-11-09T01:18:52.2068982Z # mountinfo after umount a^
2023-11-09T01:18:52.2069477Z # Debug mode:
2023-11-09T01:18:52.2070271Z # stacker version 589a6486aeb2a5fc157edf043d0310795cd2c90b
2023-11-09T01:18:52.2071671Z # stacker subcommand: [/home/runner/work/stacker/stacker/stacker --debug internal-go atomfs umount c]
2023-11-09T01:18:52.2073676Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2075864Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2078225Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2080637Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2082840Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2084945Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2087139Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2089375Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2091565Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2094036Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2096259Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2098408Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2100524Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2102729Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2104929Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2107093Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2109270Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2111390Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2113559Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2115806Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2117961Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2120363Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2135464Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2137644Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2139911Z # device-mapper: remove ioctl on f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity failed: Resource busy
2023-11-09T01:18:52.2142031Z # Device /dev/mapper/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity is still in use.
2023-11-09T01:18:52.2143742Z # error: libcryptsetup function 'crypt_deactivate' returned error with code '-16'.
2023-11-09T01:18:52.2144772Z # stackerbuild.io/stacker/pkg/squashfs.Umount
2023-11-09T01:18:52.2149716Z # /stacker-tree/pkg/squashfs/verity.go:471
2023-11-09T01:18:52.2150441Z # stackerbuild.io/stacker/pkg/atomfs.Umount
2023-11-09T01:18:52.2151311Z # /stacker-tree/pkg/atomfs/molecule.go:244
2023-11-09T01:18:52.2151953Z # main.doAtomfsUmount
2023-11-09T01:18:52.2152602Z # /stacker-tree/cmd/stacker/internal_go.go:209
2023-11-09T01:18:52.2153261Z # github.com/urfave/cli/v2.(*Command).Run
2023-11-09T01:18:52.2154125Z # /build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:273
2023-11-09T01:18:52.2155002Z # github.com/urfave/cli/v2.(*Command).Run
2023-11-09T01:18:52.2156290Z # /build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:266
2023-11-09T01:18:52.2157185Z # github.com/urfave/cli/v2.(*Command).Run
2023-11-09T01:18:52.2158054Z # /build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:266
2023-11-09T01:18:52.2158918Z # github.com/urfave/cli/v2.(*Command).Run
2023-11-09T01:18:52.2159804Z # /build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/command.go:266
2023-11-09T01:18:52.2160898Z # github.com/urfave/cli/v2.(*App).RunContext
2023-11-09T01:18:52.2161801Z # /build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:332
2023-11-09T01:18:52.2162950Z # github.com/urfave/cli/v2.(*App).Run
2023-11-09T01:18:52.2163827Z # /build/gopath/pkg/mod/github.com/urfave/cli/v2@v2.25.0/app.go:309
2023-11-09T01:18:52.2164629Z # main.main
2023-11-09T01:18:52.2165235Z # /stacker-tree/cmd/stacker/main.go:328
2023-11-09T01:18:52.2165794Z # runtime.main
2023-11-09T01:18:52.2166242Z # /usr/lib/go/src/runtime/proc.go:267
2023-11-09T01:18:52.2166799Z # runtime.goexit
2023-11-09T01:18:52.2167252Z # /usr/lib/go/src/runtime/asm_amd64.s:1650
raharper commented
This is hard to reproduce (locally). A couple of comments.
The squashfs image (layer c, f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity) is used in two places:
- the actual squashfs mount of the layer (using a loopdev + dm-verity) to create /dev/mapper/f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a-verity
- the combined molucule mountpoint of (c) via overlay: /home/runner/work/stacker/stacker/stackertest-test_mount_-2b_umount_-2b_mount_a_tree_of_images_works.LIZrhM/c
The unmount code checks if any of the lower layers are in use (ie combined into another overlay mountpoint), after the unmount of /a; the only overlayfs mount left is /c, and Umount will unmount the overlayfs mount for /c
Re-reading mountinfo, what remains is the 3 underlying squashfs-verity mounts. for each of these the squashfs.Unmount() is called which handles finding the backing loopdev of the layer, and calling into devicemapper to close the -verity device.
The error we see here shows that something else was accessing/using the f3ecbca987fb6509fa389ffa3e50afb99d27c3a8342253e2ebe485e23a71f49a verity device.
It's possible something udev/script related, or something else.
libdevicemapper (which the go-cryptsetup library uses) does support retry and lazy unmount; however this is not exposed in the golang api.
So I think we'll likely need to add a retry() wrapper inside squashfs.Umount().
Long term, we might suggest that the go-cryptsetup library expose the c-library retry parameters and detach flags.
https://github.com/martinjungblut/go-cryptsetup/blob/master/device.go#L297
https://github.com/mhfan/cryptsetup/blob/master/lib/libdevmapper.c#L449
smoser commented
This is hard to reproduce (locally). A couple of comments.
hard as in "i have never reproduced" or "i have reproduced, but rarely".