rickysarraf/apt-offline

`apt-offline set` sometimes fails to parse command-line when using `--`

mtalexan opened this issue · 7 comments

mtalexan commented

Running apt-offline set --install-packages {package names} -- wanted.asc silently fails to include some packages in the output, and fails with an explict error only when those packages that were silently skipped are the only packages listed, despite all the packages being available in the apt database.

Problem 1: A package that's available in the apt database isn't working with apt-offline
Problem 2: Packages that fail in this way fail silently when included in a list of packages

Steps to Reproduce

Ubuntu 22.04.2, with apt-offline version 1.8.4 installed from the repos. For reproduction purposes, a single online machine can be used to try to generate the signature.

The following packages have been found to fail (so far): emacs, emacs-gtk

  1. sudo apt-get update
  2. Confirm all database updates succeed
  3. sudo apt-cache policy emacs
  4. Confirm a version is listed as a Candidate
  5. sudo apt-cache show emacs
  6. Confirm a version is listed and a Filename is included.
  7. sudo apt-get install --dry-run parallel | grep "^Inst " to see what would be needed by an install
  8. sudo apt-offline set --apt-backend apt-get --install-packages emacs -- wanted.asc
  9. Notice Error
Step 1 Output: sudo apt-get update 

Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:4 http://packages.microsoft.com/repos/code stable InRelease
Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease
Reading package lists... Done

Step 3 Output: sudo apt-cache policy emacs 

emacs:
  Installed: (none)
  Candidate: 1:27.1+1-3ubuntu5
  Version table:
     1:27.1+1-3ubuntu5 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu jammy/universe i386 Packages

Step 5 Output: sudo apt-cache show emacs 

Package: emacs
Architecture: all
Version: 1:27.1+1-3ubuntu5
Priority: optional
Section: universe/editors
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Rob Browning <rlb@defaultvalue.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 77
Depends: emacs-gtk (>= 1:27.1) | emacs-lucid (>= 1:27.1) | emacs-nox (>= 1:27.1)
Filename: pool/universe/e/emacs/emacs_27.1+1-3ubuntu5_all.deb
Size: 13284
MD5sum: 72cdaf6570f1d8a0a2973eeb9f517872
SHA1: 507b22697d96cbfc001020e085f0c4f804a6dcf1
SHA256: 1e186c5436d9d83de73d6deabc4f63229f373ccaa30fa10a22b1e39f1f7036c6
SHA512: 415d6bf24bc031f3fc7b2d6dd1009bfb7959ce92eef843ecbbf0b4a1cdd44a3d8678a5902323798982b3b2fe30aaf8d66cf8ef556ee31082a2fc58d0459d18d1
Homepage: https://www.gnu.org/software/emacs/
Description-en: GNU Emacs editor (metapackage)
 GNU Emacs is the extensible self-documenting text editor.
 This is a metapackage that will always depend on the latest
 recommended Emacs variant (currently emacs-gtk).
Description-md5: f991b22b712b84422dc583d29a7b2171

Step 7 Output: sudo apt-get install --dry-run emacs | grep "^Inst " 

Inst emacs-common (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [all])
Inst emacs-bin-common (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [amd64])
Inst m17n-db (1.8.0-3 Ubuntu:22.04/jammy [all])
Inst libotf1 (0.9.16-3build1 Ubuntu:22.04/jammy [amd64])
Inst libm17n-0 (1.8.0-4 Ubuntu:22.04/jammy [amd64])
Inst emacs-gtk (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [amd64])
Inst emacs (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [all])
Inst emacs-el (1:27.1+1-3ubuntu5 Ubuntu:22.04/jammy [all])

Step 8 Output: sudo apt-offline set --apt-backend apt-get --install-packages emacs -- wanted.asc 

Gathering installation details for package ['emacs']
ERROR: Generated signature file wanted.asc is of 0 bytes
ERROR: This is usually the case when the underneath apt system has no payload to download

Steps to Reproduce Hidden Error

  1. sudo apt-get update
  2. Confirm all database updates succeed
  3. sudo apt-cache policy parallel
  4. Confirm a version is listed as a Candidate
  5. sudo apt-cache policy emacs
  6. Confirm a version is listed as a Candidate
  7. sudo apt-cache show parallel
  8. Confirm a version is listed and a Filename is included.
  9. sudo apt-cache show emacs
  10. Confirm a version is listed and a Filename is included.
  11. sudo apt-get install --dry-run parallel | grep "^Inst " to see what would be needed by an install
  12. sudo apt-offline set --apt-backend apt-get --install-packages parallel -- wanted.asc
  13. Confirm this step succeeds
  14. cat wanted.asc confirm the contents include everything the dry-run would have installed
  15. sudo apt-get install --dry-run emacs | grep "^Inst " to see what would be needed by an install
  16. sudo apt-offline set --apt-backend apt-get --install-packages emacs -- wanted.asc
  17. Notice this step fails
  18. sudo apt-offline set --apt-backend apt-get --install-packages emacs parallel -- wanted.asc
  19. Notice this step succeeds
  20. cat wanted.asc see that only parallel and its dependencies are included
Step 3 Output: sudo apt-cache policy parallel 

parallel:
  Installed: (none)
  Candidate: 20210822+ds-2
  Version table:
     20210822+ds-2 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu jammy/universe i386 Packages

Step 7 Output: sudo apt-cache show parallel 

Package: parallel
Architecture: all
Version: 20210822+ds-2
Priority: extra
Section: universe/utils
Origin: Ubuntu
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Original-Maintainer: Debian Med Packaging Team <debian-med-packaging@lists.alioth.debian.org>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 2943
Depends: procps, sysstat, perl:any
Suggests: ash, csh, fish, ksh, tcsh, zsh
Filename: pool/universe/p/parallel/parallel_20210822+ds-2_all.deb
Size: 1946526
MD5sum: 1670f08bf20379ba4d00f61614e7a40e
SHA1: 7ffd10d06d2f32c1f6e340797d0627169d733b87
SHA256: 97cfb71545c60e510d3b8b854cbe7e2b905bd1bf3656b79ea260038237bbcbb1
SHA512: f272766263d8f28914bf858e5edd5dab3cc4d8bc0ea5b58d5ef3ab66310585f2455d85691ceaa8299b50b12afc824df6570ab0f22b48b544f16dea66bf9cd0ea
Homepage: https://www.gnu.org/software/parallel/
Description-en: build and execute command lines from standard input in parallel
 GNU Parallel is a shell tool for executing jobs in parallel using one
 or more machines. A job is typically a single command or a small
 script that has to be run for each of the lines in the input. The
 typical input is a list of files, a list of hosts, a list of users, or
 a list of tables.
 .
 If you use xargs today you will find GNU Parallel very easy to use. If
 you write loops in shell, you will find GNU Parallel may be able to
 replace most of the loops and make them run faster by running jobs in
 parallel. If you use ppss or pexec you will find GNU Parallel will
 often make the command easier to read.
 .
 GNU Parallel also makes sure output from the commands is the same
 output as you would get had you run the commands sequentially. This
 makes it possible to use output from GNU Parallel as input for other
 programs.
Description-md5: 8a851a81a172dd16a4bcbeb1f8f3c4ab

Step 11 Output: sudo apt-get install --dry-run parallel | grep "^Inst " 

Inst sysstat (12.5.2-2ubuntu0.1 Ubuntu:22.04/jammy-updates, Ubuntu:22.04/jammy-security [amd64])
Inst parallel (20210822+ds-2 Ubuntu:22.04/jammy [all])

Step 12 Output: sudo apt-offline set --apt-backend apt-get --install-packages parallel -- wanted.asc 

Gathering installation details for package ['parallel']

Step 14 Output: cat wanted.asc 

'http://us.archive.ubuntu.com/ubuntu/pool/main/s/sysstat/sysstat_12.5.2-2ubuntu0.1_amd64.deb' sysstat_12.5.2-2ubuntu0.1_amd64.deb 487044 MD5Sum:68f0f873cb63af7c4a1b63cb80027dd1
'http://us.archive.ubuntu.com/ubuntu/pool/universe/p/parallel/parallel_20210822%2bds-2_all.deb' parallel_20210822+ds-2_all.deb 1946526 MD5Sum:1670f08bf20379ba4d00f61614e7a40e

Step 18 Output: sudo apt-offline set --apt-backend apt-get --install-packages emacs parallel -- wanted.asc 

Gathering installation details for package ['emacs', 'parallel']

Step 20 Output: cat wanted.asc 

'http://us.archive.ubuntu.com/ubuntu/pool/main/s/sysstat/sysstat_12.5.2-2ubuntu0.1_amd64.deb' sysstat_12.5.2-2ubuntu0.1_amd64.deb 487044 MD5Sum:68f0f873cb63af7c4a1b63cb80027dd1
'http://us.archive.ubuntu.com/ubuntu/pool/universe/p/parallel/parallel_20210822%2bds-2_all.deb' parallel_20210822+ds-2_all.deb 1946526 MD5Sum:1670f08bf20379ba4d00f61614e7a40e

Other Notes

The emacs package is a group-package that has options of emacs-gtk, emacs-nox, or emacs-lucid. However the emacs-gtk package, which is the default selection for the group, produces the same errors/results even though it's a regular package. Neither the emacs-nox nor the emacs-lucid have this problem though.

rickysarraf commented

What exactly is a group package ? Do you mean a meta package ?

I tried with your example above and I have:

rrs@priyasi:~/.../apt-offline (master)$ sudo apt-offline set /tmp/emacs.uris --install-packages emacs
[sudo] password for rrs: 
Gathering installation details for package ['emacs']
20:03 ♒ ॐ ♅ ♄ ⛢     ☺ 😄    


rrs@priyasi:~/.../apt-offline (master)$ cat /tmp/emacs.uris 
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-el_28.2%2b1-10_all.deb' emacs-el_1%3a28.2+1-10_all.deb 16872928 MD5Sum:4220a0f76d24cb81971d03fd6df4d965
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-common_28.2%2b1-10_all.deb' emacs-common_1%3a28.2+1-10_all.deb 14004332 MD5Sum:374a384eec3c9eabb1b97ef2bae84d88
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-bin-common_28.2%2b1-10_amd64.deb' emacs-bin-common_1%3a28.2+1-10_amd64.deb 101852 MD5Sum:fb3254fac1732c713addd782a408663c
'http://deb.debian.org/debian/pool/main/g/gcc-12/libgccjit0_12.2.0-14_amd64.deb' libgccjit0_12.2.0-14_amd64.deb 8782528 MD5Sum:0af91771f4dcf247ddffd33c0bcde993
'http://deb.debian.org/debian/pool/main/m/m17n-db/m17n-db_1.8.0-5_all.deb' m17n-db_1.8.0-5_all.deb 1297224 MD5Sum:d727a0795d335d2cc76f8ebe074998aa
'http://deb.debian.org/debian/pool/main/libo/libotf/libotf1_0.9.16-4_amd64.deb' libotf1_0.9.16-4_amd64.deb 55484 MD5Sum:a5be4717ef0201ff1f9760ed74181fbc
'http://deb.debian.org/debian/pool/main/m/m17n-lib/libm17n-0_1.8.0-6_amd64.deb' libm17n-0_1.8.0-6_amd64.deb 253528 MD5Sum:67320e7a44df16e3f0622a8d2814deb6
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-gtk_28.2%2b1-10_amd64.deb' emacs-gtk_1%3a28.2+1-10_amd64.deb 5997860 MD5Sum:d35f587db91d0b946835d24c6e4b778b
'http://deb.debian.org/debian/pool/main/e/emacs/emacs_28.2%2b1-10_all.deb' emacs_1%3a28.2+1-10_all.deb 15780 MD5Sum:c14b4d686bf10dd393503cee6c08d27b
20:04 ♒ ॐ ♅ ♄ ⛢     ☺ 😄
rickysarraf commented

Same with 2 packages

rrs@priyasi:~/.../apt-offline (master)$ sudo apt-offline set /tmp/emacs.uris --install-packages emacs parallel
Gathering installation details for package ['emacs', 'parallel']
20:06 ♒ ॐ ♅ ♄ ⛢     ☺ 😄    

rrs@priyasi:~/.../apt-offline (master)$ cat /tmp/emacs.uris 
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-el_28.2%2b1-10_all.deb' emacs-el_1%3a28.2+1-10_all.deb 16872928 MD5Sum:4220a0f76d24cb81971d03fd6df4d965
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-common_28.2%2b1-10_all.deb' emacs-common_1%3a28.2+1-10_all.deb 14004332 MD5Sum:374a384eec3c9eabb1b97ef2bae84d88
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-bin-common_28.2%2b1-10_amd64.deb' emacs-bin-common_1%3a28.2+1-10_amd64.deb 101852 MD5Sum:fb3254fac1732c713addd782a408663c
'http://deb.debian.org/debian/pool/main/g/gcc-12/libgccjit0_12.2.0-14_amd64.deb' libgccjit0_12.2.0-14_amd64.deb 8782528 MD5Sum:0af91771f4dcf247ddffd33c0bcde993
'http://deb.debian.org/debian/pool/main/m/m17n-db/m17n-db_1.8.0-5_all.deb' m17n-db_1.8.0-5_all.deb 1297224 MD5Sum:d727a0795d335d2cc76f8ebe074998aa
'http://deb.debian.org/debian/pool/main/libo/libotf/libotf1_0.9.16-4_amd64.deb' libotf1_0.9.16-4_amd64.deb 55484 MD5Sum:a5be4717ef0201ff1f9760ed74181fbc
'http://deb.debian.org/debian/pool/main/m/m17n-lib/libm17n-0_1.8.0-6_amd64.deb' libm17n-0_1.8.0-6_amd64.deb 253528 MD5Sum:67320e7a44df16e3f0622a8d2814deb6
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-gtk_28.2%2b1-10_amd64.deb' emacs-gtk_1%3a28.2+1-10_amd64.deb 5997860 MD5Sum:d35f587db91d0b946835d24c6e4b778b
'http://deb.debian.org/debian/pool/main/e/emacs/emacs_28.2%2b1-10_all.deb' emacs_1%3a28.2+1-10_all.deb 15780 MD5Sum:c14b4d686bf10dd393503cee6c08d27b
'http://deb.debian.org/debian/pool/main/p/parallel/parallel_20221122%2bds-2_all.deb' parallel_20221122+ds-2_all.deb 1873196 MD5Sum:904fb973c23ca9d92e413a975e1ed667
20:06 ♒ ॐ ♅ ♄ ⛢     ☺ 😄
mtalexan commented

What exactly is a group package ? Do you mean a meta package ?

Yes, sorry that's exactly what I meant.

I tried with your example above and I have:
...snip...

Ah ha, it appears to be an issue with the command-line parser (I think?).

On a completely fresh Ubuntu 22.04 desktop minimal install (new VM), this:

sudo apt-get update
sudo apt-get install apt-offline
sudo apt-offline set --install-packages emacs -- emacs.uris

produces an error. However on a separate completely fresh Ubuntu 22.04 desktop minimal install (new VM), this:

sudo apt-get update
sudo apt-get install apt-offline
sudo apt-offline set emacs.uris --install-packages emacs

works as expected.

In both cases the initial output printed is:

Gathering installation details for package ['emacs']

But in the case with an error it immediately prints the error I noted above.

It looks like there's an error with the command-line parsing mismatching the documented support. From the man page:

apt-offline relies on argparse for argument/option parsing. To explicitly instruct apt-offline about an argument, you can pass it with the -- delimiter.

Ex. apt-offline set --update --upgrade --install-packages wm2 -- foo.sig

By specifying the -- delimiter, we instruct apt-offline that foo.sig is an argument to the apt-offline command and not to the --install-packages option.

Otherwise, you could also use it positionally next to the set command

Ex. apt-offline set foo.sig --update --upgrade --install-packages wm2

But when I try to use the format from the first example for the emacs package, or a list that includes the emacs package, it fails, while the format of the second example works. Strangely though, both work with the parallel package by itself.

rickysarraf commented

produces an error. However on a separate completely fresh Ubuntu 22.04 desktop minimal install (new VM), this:

Given that the issue is not reproducible on 2 different machine with the same version of the OS, it should be fair to say that the issue is with the machine in particular. Possibly, you may have other local python modules installed ?

On the problematic machine, could you run the same command with --verbose switch ? Maybe that'd shed some more light.

mtalexan commented

Given that the issue is not reproducible on 2 different machine with the same version of the OS, it should be fair to say that the issue is with the machine in particular. Possibly, you may have other local python modules installed ?

I'm actually saying the problem is easily reproducible on any machine, but it's a problem only when certain command-line syntax is used.

Putting the file name at the end like in the manpage example fails:

apt-offline set --install-packages emacs -- emacs.uris

But putting it earlier like the alternative in the manpage works:

apt-online set emacs.uris --install-packages emacs
rickysarraf commented

@mtalexan

I'm actually saying the problem is easily reproducible on any machine, but it's a problem only when certain command-line syntax is used.

It seems to be working fine for me here.

rrs@priyasi:~$ sudo apt-offline set --install-packages emacs parallel -- /tmp/emacs.uris
[sudo] password for rrs: 
Gathering installation details for package ['emacs', 'parallel']
13:56 ♒ ॐ ♅ ♄ ⛢     ☺ 😄    


rrs@priyasi:~$ cat /tmp/emacs.uris 
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-el_28.2%2b1-10_all.deb' emacs-el_1%3a28.2+1-10_all.deb 16872928 MD5Sum:4220a0f76d24cb81971d03fd6df4d965
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-common_28.2%2b1-10_all.deb' emacs-common_1%3a28.2+1-10_all.deb 14004332 MD5Sum:374a384eec3c9eabb1b97ef2bae84d88
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-bin-common_28.2%2b1-10_amd64.deb' emacs-bin-common_1%3a28.2+1-10_amd64.deb 101852 MD5Sum:fb3254fac1732c713addd782a408663c
'http://deb.debian.org/debian/pool/main/g/gcc-12/libgccjit0_12.2.0-14_amd64.deb' libgccjit0_12.2.0-14_amd64.deb 8782528 MD5Sum:0af91771f4dcf247ddffd33c0bcde993
'http://deb.debian.org/debian/pool/main/m/m17n-db/m17n-db_1.8.0-5_all.deb' m17n-db_1.8.0-5_all.deb 1297224 MD5Sum:d727a0795d335d2cc76f8ebe074998aa
'http://deb.debian.org/debian/pool/main/libo/libotf/libotf1_0.9.16-4_amd64.deb' libotf1_0.9.16-4_amd64.deb 55484 MD5Sum:a5be4717ef0201ff1f9760ed74181fbc
'http://deb.debian.org/debian/pool/main/m/m17n-lib/libm17n-0_1.8.0-6_amd64.deb' libm17n-0_1.8.0-6_amd64.deb 253528 MD5Sum:67320e7a44df16e3f0622a8d2814deb6
'http://deb.debian.org/debian/pool/main/e/emacs/emacs-gtk_28.2%2b1-10_amd64.deb' emacs-gtk_1%3a28.2+1-10_amd64.deb 5997860 MD5Sum:d35f587db91d0b946835d24c6e4b778b
'http://deb.debian.org/debian/pool/main/e/emacs/emacs_28.2%2b1-10_all.deb' emacs_1%3a28.2+1-10_all.deb 15780 MD5Sum:c14b4d686bf10dd393503cee6c08d27b
'http://deb.debian.org/debian/pool/main/p/parallel/parallel_20221122%2bds-2_all.deb' parallel_20221122+ds-2_all.deb 1873196 MD5Sum:904fb973c23ca9d92e413a975e1ed667
13:56 ♒ ॐ ♅ ♄ ⛢     ☺ 😄
mtalexan commented

I was just trying to reproduce the issue with the --verbose flag, and now it's working with both syntaxes. I created a fresh VM again for this test since it was reproducible with multiple fresh VMs last week, so I don't have the old one to reexamine. Weird.