Support building on multiple builders

[dr460nf1r3]

Feature summary

The proposal is to support additional builders who are building packages for a main node, which handles the database and everything else but the building.

Cause of the feature request

Bigger repositories might need multiple machines, or use 3rd party supplied resources, which makes a distribution of build processes needed.

Ahriman already supports pushing packages to another location, which could be leveraged to implement pushing to the main node, triggering an action that adds the package to the main ahriman instance.

Proposed changes and/or features

Implement a feature in which ahriman can be used in a multi-node setup.

[arcan1s]

makes sense to me

[arcan1s]

lets try to implement this feature in two steps:

1. Add flags to web api for update action, which replicates --no-aur and --no-local logic. Add special trigger which works like calling remote url on success; in case of current feature, it should call update url with specific flags

Having this, you will be able to configure remote workers, which are responsible for updating the part of repository. However you have to be worried about dependencies yourself. These workers should first upload packages (e.g. by rsync) and finally call remote url.

Alternatively this part can be done by instead watching directory, however, it has some issues, e.g. we will need to separate smh upload start (when inode information changes) and when file is uploaded

2. Think about starting workers manually and delegating tasks to them. This part, however, requires some assumptions, e.g. the easiest way is to just spawn containers with delegating tasks to them, however, it requires orchestrations system (kuber?); in order to make it system independent, the only way is to register workers in config and call them by web api

[arcan1s]

so as #105 has been merged, the first part of the feature has been implemented. Major changes from original concept include special remote upload trigger (instead of rsync) because I found it a little bit tricky to configure ssh access for service user (plus ssh is not available by default in container). Details about tested setup can be found here

[dr460nf1r3]

Alright, so I finally got to test the new version. Initial observations:

• The remote upload trigger seems to work fairly well. I was building a few packages on the worker node and all of them were showing up correctly on the master node.
• Log files are uploaded to the master node as well from what I observed? A pretty neat feature!
• It does not seem to be possible to utilize both unix socket + webinterface and remote trigger at the same time? Ahriman was failing to upload the package due to an invalid path when having set the socket to active on the worker node.
• Packages built by the worker node would not be added to the worker node's database and only to the master node from what I observed. Is that correct? I'm asking because the docs refer to removing packages as follows:

  Remove package on worker.
  Remove package on master node.
  

• What format does AHRIMAN_REPOSITORY_SERVER have to follow?

[arcan1s]

Log files are uploaded to the master node as well from what I observed? A pretty neat feature!

It does not seem to be possible to utilize both unix socket + webinterface and remote trigger at the same time? Ahriman was failing to upload the package due to an invalid path when having set the socket to active on the worker node.

Packages built by the worker node would not be added to the worker node's database and only to the master node from what I observed. Is that correct?

yes, correct. In fact any reports are go through master node, including packages registration and so on. Currently worker cannot have its own interface, because some settings interfere there, like it seems reasonable to store logs on master, but there is no ability to configure logger explicitly, etc.

More likely I will extend settings later, because of master -> worker interaction requirements.

I'm asking because the docs refer to removing packages as follows

yes, docs are correct also. At the moment some features mostly depend on filesystem, rather than database. Thus worker nodes still know about their packages (unless you start them on clear filesystem each time)

What format does AHRIMAN_REPOSITORY_SERVER have to follow?

same as pacman.conf's Server directive

[arcan1s]

I've implemented another part of the feature (#118), which allows to delegate build process automatically based on the amount of configured workers. Yet, it seems that some features are unavailable (like pkgrel bump) and updates check is run on the single node.

Here is a small faq and (working) docker compose configuration

[arcan1s]

So, finally, auto discovery feature has been also implemented in #121 and was tested on the configuration described here and planned to be released soon.

For now it seems that the most of the features have been implemented and, thus, I'm closing this issue. Yet, feel free to continue discussion (and/or provide any feedback) here