Recursive dependency resolution

Question

Recursive dependency resolution

Closed this issue a year ago · 10 comments

According to #47 & the feature list, ahriman should handle AUR dependencies recursively. Whenever I'm building a package requiring packages not yet available in the repo, I'm getting:

[DEBUG 2023-08-13 14:48:27,244] [ahriman.core.build_tools.task.Task]: ==> Making package: nwg-panel 0.9.11-1 (Sun Aug 13 14:48:27 2023)
[DEBUG 2023-08-13 14:48:27,246] [ahriman.core.build_tools.task.Task]: ==> Checking runtime dependencies...
[DEBUG 2023-08-13 14:48:27,258] [ahriman.core.build_tools.task.Task]: ==> Installing missing dependencies...
[DEBUG 2023-08-13 14:48:27,387] [ahriman.core.build_tools.task.Task]: error: target not found: python-dasbus
[DEBUG 2023-08-13 14:48:27,390] [ahriman.core.build_tools.task.Task]: error: target not found: nwg-icon-picker
[DEBUG 2023-08-13 14:48:27,411] [ahriman.core.build_tools.task.Task]: ==> ERROR: 'pacman' failed to install missing dependencies.
[DEBUG 2023-08-13 14:48:27,414] [ahriman.core.build_tools.task.Task]: ==> Missing dependencies:
[DEBUG 2023-08-13 14:48:27,415] [ahriman.core.build_tools.task.Task]:   -> python
[DEBUG 2023-08-13 14:48:27,416] [ahriman.core.build_tools.task.Task]:   -> python-gobject
[DEBUG 2023-08-13 14:48:27,428] [ahriman.core.build_tools.task.Task]:   -> python-i3ipc
[DEBUG 2023-08-13 14:48:27,430] [ahriman.core.build_tools.task.Task]:   -> python-cairo
[DEBUG 2023-08-13 14:48:27,432] [ahriman.core.build_tools.task.Task]:   -> python-dasbus
[DEBUG 2023-08-13 14:48:27,433] [ahriman.core.build_tools.task.Task]:   -> python-netifaces
[DEBUG 2023-08-13 14:48:27,434] [ahriman.core.build_tools.task.Task]:   -> python-psutil
[DEBUG 2023-08-13 14:48:27,435] [ahriman.core.build_tools.task.Task]:   -> python-requests
[DEBUG 2023-08-13 14:48:27,436] [ahriman.core.build_tools.task.Task]:   -> bluez-utils
[DEBUG 2023-08-13 14:48:27,438] [ahriman.core.build_tools.task.Task]:   -> gtk3
[DEBUG 2023-08-13 14:48:27,439] [ahriman.core.build_tools.task.Task]:   -> gtk-layer-shell
[DEBUG 2023-08-13 14:48:27,440] [ahriman.core.build_tools.task.Task]:   -> light
[DEBUG 2023-08-13 14:48:27,441] [ahriman.core.build_tools.task.Task]:   -> playerctl
[DEBUG 2023-08-13 14:48:27,442] [ahriman.core.build_tools.task.Task]:   -> nwg-icon-picker
[DEBUG 2023-08-13 14:48:27,443] [ahriman.core.build_tools.task.Task]: ==> Checking buildtime dependencies...
[DEBUG 2023-08-13 14:48:27,444] [ahriman.core.build_tools.task.Task]: ==> Installing missing dependencies...

followed by

[DEBUG 2023-08-13 14:48:28,463] [ahriman.core.build_tools.task.Task]: ==> ERROR: Could not resolve all dependencies.
[DEBUG 2023-08-13 14:48:28,480] [ahriman.core.build_tools.task.Task]: Attempted to remove disk file system under "/run/systemd/nspawn/propagate/ahriman", and we can't allow that.
[DEBUG 2023-08-13 14:48:28,482] [ahriman.core.build_tools.task.Task]: ==> ERROR: Build failed, check /var/lib/ahriman/ahriman/chroot/ahriman-x86_64/ahriman/build

According to #47 (comment), the repo needs to be available in order to profit. I see that a file properly got created:

[ahriman@61fd61d7dd25 ~]$ cat /usr/share/devtools/pacman.conf.d/ahriman-x86_64.conf
[options]
HoldPkg = pacman glibc
Architecture = x86_64
NoProgressBar
VerbosePkgLists
ParallelDownloads = 5
SigLevel = Required DatabaseOptional
LocalFileSigLevel = Optional

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

[multilib]
Include = /etc/pacman.d/mirrorlist

[chaotic-aur]
SigLevel = Never
Server = file:///var/lib/ahriman/ahriman/repository/x86_64

Full setup can be found here, full log output of the build here.

The question is - what am I missing here for proper dependency resolution? 👀 Thank you very much :)

Answer 1 · 2023-08-13T20:58:16.000Z

well as mentioned in docs, it should work out of box 😄 however:

[DEBUG 2023-08-13 14:48:27,387] [ahriman.core.build_tools.task.Task]: error: target not found: python-dasbus
[DEBUG 2023-08-13 14:48:27,390] [ahriman.core.build_tools.task.Task]: error: target not found: nwg-icon-picker

in this case there are no such packages in aur, I guess they are something which come from separated repos? For now dependencies lookup in local packages aren't supported. In the other hand from quick check I'd say it should generate another error (unless --no-dependencies flag has been applied)

According to #47 (comment), the repo needs to be available in order to profit. I see that a file properly got created:

it is only about the case in which packages were already built before (in the mentioned issue, the initial service setup has been done with errors)

TL;DR let me check case with non-AUR packages as dependencies

Answer 2 · 2023-08-13T23:13:06.000Z

so okay, from quick check

[remote-pull]
target = local-packages

[local-packages]
pull_url = https://git.arcanis.me/arcanis/ahriman-local-packages.git

the repository contains two packages, one depends on another

$ docker run ... ahriman package-add package2 --now
...
[ERROR 2023-08-13 23:07:44,991] [stderr]: process exception
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/remote/aur.py", line 148, in package_info
    return next(package for package in packages if package.name == package_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ahriman/application/handlers/handler.py", line 100, in call
    cls.run(args, architecture, configuration, report=args.report)
  File "/usr/lib/python3.11/site-packages/ahriman/application/handlers/add.py", line 51, in run
    packages = application.with_dependencies(packages, process_dependencies=args.dependencies)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/application/application/application.py", line 145, in with_dependencies
    package = Package.from_aur(package_name, self.repository.pacman, username)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/models/package.py", line 229, in from_aur
    package = AUR.info(name, pacman=pacman)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/remote/remote.py", line 60, in info
    return cls().package_info(package_name, pacman=pacman)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/remote/aur.py", line 150, in package_info
    raise UnknownPackageError(package_name)
ahriman.core.exceptions.UnknownPackageError: Package base package1 is unknown
[ERROR 2023-08-13 23:07:44,991] [stderr]: process exception
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/remote/aur.py", line 148, in package_info
    return next(package for package in packages if package.name == package_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/ahriman/application/handlers/handler.py", line 100, in call
    cls.run(args, architecture, configuration, report=args.report)
  File "/usr/lib/python3.11/site-packages/ahriman/application/handlers/add.py", line 51, in run
    packages = application.with_dependencies(packages, process_dependencies=args.dependencies)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/application/application/application.py", line 145, in with_dependencies
    package = Package.from_aur(package_name, self.repository.pacman, username)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/models/package.py", line 229, in from_aur
    package = AUR.info(name, pacman=pacman)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/remote/remote.py", line 60, in info
    return cls().package_info(package_name, pacman=pacman)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/remote/aur.py", line 150, in package_info
    raise UnknownPackageError(package_name)
ahriman.core.exceptions.UnknownPackageError: Package base package1 is unknown

This is correct trace produced in case if dependency is non-AUR package. However:

$ docker run ... ahriman package-add package2 --now --no-dependencies
...
[DEBUG 2023-08-13 23:08:16,842] [ahriman.core.build_tools.task.Task]: Attempted to remove disk file system under "/run/systemd/nspawn/propagate/root", and we can't allow that.
[DEBUG 2023-08-13 23:08:18,656] [ahriman.core.build_tools.task.Task]: ==> Synchronizing chroot copy [/var/lib/ahriman/ahriman/chroot/ahriman-x86_64/root] -> [ahriman]...done
[DEBUG 2023-08-13 23:08:19,154] [ahriman.core.build_tools.task.Task]: ==> Making package: package2 20230814-1 (Sun Aug 13 23:08:19 2023)
[DEBUG 2023-08-13 23:08:19,157] [ahriman.core.build_tools.task.Task]: ==> Retrieving sources...
[DEBUG 2023-08-13 23:08:21,446] [ahriman.core.build_tools.task.Task]: ==> Making package: package2 20230814-1 (Sun Aug 13 23:08:21 2023)
[DEBUG 2023-08-13 23:08:21,448] [ahriman.core.build_tools.task.Task]: ==> Checking runtime dependencies...
[DEBUG 2023-08-13 23:08:21,497] [ahriman.core.build_tools.task.Task]: ==> Installing missing dependencies...
[DEBUG 2023-08-13 23:08:21,657] [ahriman.core.build_tools.task.Task]: error: target not found: package1
[DEBUG 2023-08-13 23:08:21,688] [ahriman.core.build_tools.task.Task]: ==> ERROR: 'pacman' failed to install missing dependencies.
[DEBUG 2023-08-13 23:08:21,691] [ahriman.core.build_tools.task.Task]: ==> Missing dependencies:
[DEBUG 2023-08-13 23:08:21,693] [ahriman.core.build_tools.task.Task]:   -> package1
[DEBUG 2023-08-13 23:08:21,695] [ahriman.core.build_tools.task.Task]: ==> Checking buildtime dependencies...
[DEBUG 2023-08-13 23:08:21,700] [ahriman.core.build_tools.task.Task]: ==> ERROR: Could not resolve all dependencies.
[DEBUG 2023-08-13 23:08:21,744] [ahriman.core.build_tools.task.Task]: Attempted to remove disk file system under "/run/systemd/nspawn/propagate/ahriman", and we can't allow that.
[DEBUG 2023-08-13 23:08:21,746] [ahriman.core.build_tools.task.Task]: ==> ERROR: Build failed, check /var/lib/ahriman/ahriman/chroot/ahriman-x86_64/ahriman/build

what do I miss in this test case?

Answer 3 · 2023-08-13T23:32:59.000Z

For now dependencies lookup in local packages aren't supported

this part has been implemented in a1d0e99 (at least the test case above has passed) and available in edge image

Answer 4 · 2023-08-14T05:46:04.000Z

I sadly can't test it right now, but let me add some information: the log is from the build of an AUR package and not a local one. It's nwg-panel and all dependencies are actually in AUR. Also, local dependency resolution is great to have too! 🎉

Answer 5 · 2023-08-14T12:49:10.000Z

oh, true. Any chance to see full log?

Answer 6 · 2023-08-15T10:09:52.000Z

Of course! I used ahriman --log-handler console add --now nwg-panel to generate the following log.

Answer 7 · 2023-08-15T21:31:44.000Z

it looks like dependencies were already built, but smh were not registered with database 🤔 in this particular case something like

ahriman shell 'from ahriman.core.sign.gpg import GPG; from ahriman.core.alpm.repo import Repo; sign = GPG(configuration); repo = Repo(configuration.repository_name, configuration.repository_paths, sign.repository_sign_args); [repo.add(pkg) for pkg in configuration.repository_paths.repository.glob("*.pkg.*")]'

should help (it is literally re-adding each found package to database)

Answer 8 · 2023-08-16T17:37:08.000Z

The given command gives interesting results (all of those are available) 👀

[ahriman@dff9745f8c4d /]$ ahriman shell 'from ahriman.core.sign.gpg import GPG; from ahriman.core.alpm.repo import Repo; sign = GPG(configuration); repo = Repo(configuration.repository_name, configuration.repository_paths, sign.repository_sign_args); [repo.add(pkg) for pkg in configuration.repository_paths.repository.glob("*.pkg.*")]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <listcomp>
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/repo.py", line 72, in add
    Repo._check_output(
  File "/usr/lib/python3.11/site-packages/ahriman/core/util.py", line 149, in check_output
    raise exception
ahriman.core.exceptions.BuildError: Package sweet-kde-theme-git-r26.5fc01aa-1-x86_64.pkg.tar.zst.sig build failed, check logs for details
[ahriman@dff9745f8c4d /]$ packag^C
[ahriman@dff9745f8c4d /]$ ahriman remove sweet-kde-theme-git
[ahriman@dff9745f8c4d /]$ ahriman shell 'from ahriman.core.sign.gpg import GPG; from ahriman.core.alpm.repo import Repo; sign = GPG(configuration); repo = Repo(configuration.repository_name, configuration.repository_paths, sign.repository_sign_args); [repo.add(pkg) for pkg in configuration.repository_paths.repository.glob("*.pkg.*")]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <listcomp>
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/repo.py", line 72, in add
    Repo._check_output(
  File "/usr/lib/python3.11/site-packages/ahriman/core/util.py", line 149, in check_output
    raise exception
ahriman.core.exceptions.BuildError: Package resvg-0.35.0-1-x86_64.pkg.tar.zst.sig build failed, check logs for details
[ahriman@dff9745f8c4d /]$ ahriman remove resvg
[ahriman@dff9745f8c4d /]$ ahriman shell 'from ahriman.core.sign.gpg import GPG; from ahriman.core.alpm.repo import Repo; sign = GPG(configuration); repo = Repo(configuration.repository_name, configuration.repository_paths, sign.repository_sign_args); [repo.add(pkg) for pkg in configuration.repository_paths.repository.glob("*.pkg.*")]'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <listcomp>
  File "/usr/lib/python3.11/site-packages/ahriman/core/alpm/repo.py", line 72, in add
    Repo._check_output(
  File "/usr/lib/python3.11/site-packages/ahriman/core/util.py", line 149, in check_output
    raise exception
ahriman.core.exceptions.BuildError: Package ananicy-rules-git-2.2.1.r320.g62771a8-1-any.pkg.tar.zst.sig build failed, check logs for details

I could also confirm that the dependencies did in fact not build by looking at the repository folders content. Something else seems to be going on 🤔

Answer 9 · 2023-08-16T22:26:21.000Z

The given command gives interesting results (all of those are available)

eeh, well, my glob also handles signatures, sorry didn't think about it.

ok then i got another idea (or question like). In your configurations

...
[alpm]
database = /var/lib/pacman
mirror = https://geo.mirror.pkgbuild.com/$repo/os/$arch
repositories = core extra multilib chaotic-aur
...

in this specific configuration I believe you might have configured chaotic-aur system-wide. At the start of the container, there will be setup stage in which databases will be copied from /var/lib/pacman to application home (because use_ahriman_cache = yes), later it will use these databases for packages lookup

[INFO 2023-08-15 10:08:04,710] [ahriman.core.alpm.pacman.Pacman]: loading pacman database core
[INFO 2023-08-15 10:08:04,710] [ahriman.core.alpm.pacman.Pacman]: loading pacman database extra
[INFO 2023-08-15 10:08:04,710] [ahriman.core.alpm.pacman.Pacman]: loading pacman database multilib
[INFO 2023-08-15 10:08:04,710] [ahriman.core.alpm.pacman.Pacman]: loading pacman database chaotic-aur

but this repository isn't mentioned in devtools configuration (it refers to own database). So basically packages exist during dependency check, but devtools don't see them.

So you need to either:

remove chaotic-aur reference from the application config
or
append generated /usr/share/devtools/pacman.conf.d/ahriman-x86_64.conf with chaotic-aur (be aware of section names collision though)

Answer 10 · 2023-08-17T11:09:10.000Z

Wow, I feel like a fool 😅

in this specific configuration I believe you might have configured chaotic-aur system-wide. At the start of the container, there will be setup stage in which databases will be copied from /var/lib/pacman to application home (because use_ahriman_cache = yes), later it will use these databases for packages lookup

Did what you proposed (removing the reference) and now every package resolves dependencies correctly! Sorry for wasting your time.