RPM to container migration fails due to sparse files

[sbluhm]

Problem description

Running mgradm migrate podman uyuni.example.com --logLevel debug fails with

[...]
Copying /var/spacewalk...
receiving incremental file list

sent 9,673 bytes  received 1,306 bytes  7,319.33 bytes/sec
total size is 3,187,026  speedup is 290.28
Copying /var/log...
receiving incremental file list
./
lastlog
rsync: [receiver] write failed on "/var/log/lastlog": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [generator] write error: Broken pipe (32)
Error: cannot run migration script: cannot run uyuni migration container: failed to run uyuni-migration container: exit status 11

On source server:

# ll -h /var/log/last*
-rw-rw-r--. 1 root utmp 379G May 28 08:13 /var/log/lastlog
# du /var/log/lastlog
32	/var/log/lastlog
# ls -s  /var/log/lastlog
32 /var/log/lastlog

Steps to reproduce

Note: Source and destination is on Enterprise Linux 9

1. Make sure /var/log/lastlog appears large on the source server.
2. Run mgradm migrate podman uyuni.example.com --logLevel debug and wait

Uyuni version

Installed Packages
Name         : Uyuni-Server-release
Version      : 2023.12
Release      : 230900.210.2.uyuni3
Architecture : x86_64
Size         : 1.4 k
Source       : Uyuni-Server-release-2023.12-230900.210.2.uyuni3.src.rpm
Repository   : @System
From repo    : copr:copr.fedorainfracloud.org:sbluhm:uyuni-release
Summary      : Uyuni Server
License      : GPL-2.0-or-later
Description  : Uyuni lets you efficiently manage physical, virtual,
             : and cloud-based Linux systems. It provides automated and cost-effective
             : configuration and software management, asset management, and system
             : provisioning.

Uyuni proxy version (if used)

No response

Useful logs

# mgradm migrate podman uyuni.example.com --logLevel debug
9:56AM INF mgradm/cmd/cmd.go:53 > Welcome to mgradm
9:56AM INF mgradm/cmd/cmd.go:54 > Executing command: podman
9:56AM DBG shared/utils/utils.go:157 > Computed image name is registry.opensuse.org/uyuni/server:latest
9:56AM DBG shared/utils/exec.go:50 > Running: /tmp/mgradm-2613065558/inspect.sh 
9:56AM DBG shared/utils/utils.go:290 > Trying to read /tmp/mgradm-2613065558/data
9:56AM INF shared/podman/images.go:35 > Ensure image registry.opensuse.org/uyuni/server:latest is available
9:56AM DBG shared/podman/images.go:165 > Checking for registry.opensuse.org/uyuni/server:latest
9:56AM DBG shared/utils/exec.go:67 > Running: podman images --format={{ .Repository }} registry.opensuse.org/uyuni/server:latest
9:56AM DBG shared/podman/images.go:43 > Image registry.opensuse.org/uyuni/server:latest already present
9:56AM INF mgradm/shared/podman/podman.go:201 > Migrating server
9:56AM DBG shared/utils/exec.go:50 > Running: podman run --name uyuni-migration --rm --cap-add NET_RAW --tmpfs /run -v cgroup:/sys/fs/cgroup:rw --security-opt label:disable -e SSH_AUTH_SOCK -v /tmp/ssh-XXXXXXHMVP49:/tmp/ssh-XXXXXXHMVP49 -v /tmp/mgradm-3509868227:/var/lib/uyuni-tools/ -v /root/.ssh/known_hosts:/etc/ssh/ssh_known_hosts -v var-cobbler:/var/lib/cobbler -v var-salt:/var/lib/salt -v var-cache:/var/cache -v var-spacewalk:/var/spacewalk -v var-log:/var/log -v srv-salt:/srv/salt -v srv-www:/srv/www/ -v srv-tftpboot:/srv/tftpboot -v srv-formulametadata:/srv/formula_metadata -v srv-pillar:/srv/pillar -v srv-susemanager:/srv/susemanager -v srv-spacewalk:/srv/spacewalk -v root:/root -v ca-cert:/etc/pki/trust/anchors -v etc-tls:/etc/pki/tls -v var-pgsql:/var/lib/pgsql -v etc-rhn:/etc/rhn -v tls-key:/etc/pki/spacewalk-tls -v etc-apache2:/etc/apache2 -v etc-systemd-multi:/etc/systemd/system/multi-user.target.wants -v etc-systemd-sockets:/etc/systemd/system/sockets.target.wants -v etc-salt:/etc/salt -v etc-rhn:/etc/rhn -v etc-tomcat:/etc/tomcat -v etc-cobbler:/etc/cobbler -v etc-sysconfig:/etc/sysconfig -v etc-postfix:/etc/postfix -v etc-sssd:/etc/sssd registry.opensuse.org/uyuni/server:latest /var/lib/uyuni-tools/migrate.sh
Stopping spacewalk service...
Shutting down spacewalk services...
Done.
Stopping posgresql service...
Copying /var/lib/cobbler...
receiving incremental file list

sent 9,962 bytes  received 16,147 bytes  52,218.00 bytes/sec
total size is 654,715  speedup is 25.08
Copying /var/lib/salt...
receiving incremental file list

sent 9,656 bytes  received 81 bytes  6,491.33 bytes/sec
total size is 0  speedup is 0.00
Copying /var/cache...
receiving incremental file list

sent 10,203 bytes  received 61,580 bytes  143,566.00 bytes/sec
total size is 361,318,823  speedup is 5,033.49
Copying /var/spacewalk...
receiving incremental file list

sent 9,673 bytes  received 1,306 bytes  7,319.33 bytes/sec
total size is 3,187,026  speedup is 290.28
Copying /var/log...
receiving incremental file list
./
lastlog
rsync: [receiver] write failed on "/var/log/lastlog": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]
rsync: [generator] write error: Broken pipe (32)
Error: cannot run migration script: cannot run uyuni migration container: failed to run uyuni-migration container: exit status 11

Additional information

No response

[rjmateus]

@sbluhm
As you can see @nadvornik already opened a PR to skip that file, since it's not needed to migrate.

[avshiliaev]

Seems to be solved for now, the PR is merged

[sbluhm]

Thank you. Will close this issue as I can't test this anymore.