Deploy inotifywait script to modify repo definitions when they change
Opened this issue · 6 comments
A limitation of cacher today is that it stops qubesos to use its appvm to report updates status to dom0. Meaning users have to check manually for all templates package updates in their hygiene.
Another limitation is that a lot of people install apps in qubes for a spontaneous need without installing those apps in the repo. This fails miserably, and is the same reason of first limitation and a blocker for massive adoption/upstreaming.
À solution to this would be to have inotify script deployed and running as a service in the templates, therefore also running in the qubes.
That service could detect (qubesdb /type) if we are in a template and make sure all repos are apt-cacher compliant, or if not, make the urls transformed back to the standard format.
Using inotifywait here would assure that anyone wanting to install whatever they want, not to have to even consider adapting for apt-cacher-ng. Just drop the standard repo url, bypass network restrictions to download pubkey (still a problem but I give up on that one: I guess extrepo is the expected project usage for mitigation, or have the user expected to know how to export proxy address to use wget/curl, not this issue solution proposal) and you're good to go.
Updating/reinstallimg sys-cacher rpm, or recalling salt recipe on restored newly installed template would be the only thing sys-cacher users would need to remember to do on dom0 (Where an helper script could call template salt calls too..) making sys-cacher nearly seemless solution.
Idea coming from:
What would still be needed on top of cacher is a wrapper of some sorts to change repo definitions URLs as soon as they are added (probably per cacher deployment) to be apt-cacher-ng compliant prior to the users trying to use those repo definitions not being transformed to be apt-cacher-ng compliant and failing. More thinking is needed to fix that. It could be a directory watching deamon (inotifywait), triggered when the repo directory files are modified and applying a sed to transform all current https URLs to use apt-cacher-ng as part of the cacher (or sys-whonix, or both). @unman, please check inotifywait. If that wrapper was deployed in templates per cacher, the whole problem of users adding repos after cacher deployment would be dealt with and become a non-issue, and would not require qubes updater to be modified either.
@unman
On Debian, deployed inotifywait service could look like (remove sudo if service launched as root):
sudo inotifywait -m -e modify,create -r /etc/apt/sources.list* | while read path action file; do sudo sed -i s^https://^http://HTTPS///^ $path$file; done
PoC Terminal 1:
user@heads-tests:~/heads$ sudo inotifywait -m -e modify,create -r /etc/apt/sources.list* | while read path action file; do sudo sed -i s^https://^http://HTTPS///^ "$path$file"; done
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
sed: can't read /etc/apt/sources.list.d/sedxaic3G: No such file or directory
sed: can't read /etc/apt/sources.list.d/sedxaic3G: No such file or directory
sed: can't read /etc/apt/sources.list.d/sedmCVVfQ: No such file or directory
sed: can't read /etc/apt/sources.list.d/sedmCVVfQ: No such file or directory
PoC Terminal 2:
user@heads-tests:~/heads$ sudo cp /etc/apt/sources.list.d/signal-xenial.list.bak /etc/apt/sources.list.d/test.list
user@heads-tests:~/heads$ diff /etc/apt/sources.list.d/signal-xenial.list.bak /etc/apt/sources.list.d/test.list
1c1
< deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] https://updates.signal.org/desktop/apt xenial main
---
> deb [arch=amd64 signed-by=/usr/share/keyrings/signal-desktop-keyring.gpg] http://HTTPS///updates.signal.org/desktop/apt xenial main
That service being deployed would be no cpu cost at all, and since the watch directory is really small, memory cost being also neglictable to be part of all templates and qubes depending on them, having repositories deployed useable right away
user@heads-tests:~/heads$ time sudo inotifywait -m -e modify,create -r /etc/apt/sources.list* | while read path action file; do sudo sed -i s^https://^http://HTTPS///^ "$path$file"; done
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
sed: can't read /etc/apt/sources.list.d/sedl6Qh9w: No such file or directory
sed: can't read /etc/apt/sources.list.d/sedl6Qh9w: No such file or directory
sed: can't read /etc/apt/sources.list.d/sedIrl5fF: No such file or directory
sed: can't read /etc/apt/sources.list.d/sedIrl5fF: No such file or directory
^C
real 0m18.858s
user 0m0.044s
sys 0m0.045s
As said in past discussions, that would permit also to take a stance, and offer the possibility to have all app-qubes also be able to use cacher/cacher under sys-whonix if
1- policy permits all qubes to use cacher (already defined templates to use apt-cacher-ng anyway, so why block that)
[user@dom0 ~]$ sudo cat /etc/qubes/policy.d/30-user.policy
qubes.UpdatesProxy * @anyvm @default allow target=cacher
#qubes.UpdatesProxy * @tag:whonix-updatevm @default allow target=sys-whonix
#qubes.UpdatesProxy * @type:AppVM @default allow target=cacher
#qubes.UpdatesProxy * @type:TemplateVM @default allow target=cacher
or being more conservative and document instead of deploying the above by default (I see Windows being a problem to push the above by default)
2- Have all qubes use that proxy by enabling update-proxy-setup, which as of now is only enabled for Templates (by default).
Otherwise, qubes cannot download packages through their defined netvm.
Example:
1- without update-proxy-setup service
user@personal:~$ sudo apt update
Err:1 http://HTTPS///deb.debian.org/debian bullseye InRelease
Could not resolve 'HTTPS'
Err:2 http://HTTPS///deb.debian.org/debian-security bullseye-security InRelease
Could not resolve 'HTTPS'
Err:3 http://HTTPS///packages.element.io/debian default InRelease
Could not resolve 'HTTPS'
Err:4 http://HTTPS///deb.qubes-os.org/r4.1/vm bullseye InRelease
Could not resolve 'HTTPS'
Err:5 http://HTTPS///updates.signal.org/desktop/apt xenial InRelease
Could not resolve 'HTTPS'
Err:6 http://HTTPS///wire-app.wire.com/linux/debian stable InRelease
Could not resolve 'HTTPS'
Reading package lists... Done
E: Failed to fetch http://HTTPS///deb.debian.org/debian/dists/bullseye/InRelease Could not resolve 'HTTPS'
E: Failed to fetch http://HTTPS///deb.debian.org/debian-security/dists/bullseye-security/InRelease Could not resolve 'HTTPS'
E: Failed to fetch http://HTTPS///packages.element.io/debian/dists/default/InRelease Could not resolve 'HTTPS'
E: Failed to fetch http://HTTPS///deb.qubes-os.org/r4.1/vm/dists/bullseye/InRelease Could not resolve 'HTTPS'
E: Failed to fetch http://HTTPS///updates.signal.org/desktop/apt/dists/xenial/InRelease Could not resolve 'HTTPS'
E: Failed to fetch http://HTTPS///wire-app.wire.com/linux/debian/dists/stable/InRelease Could not resolve 'HTTPS'
E: Some index files failed to download. They have been ignored, or old ones used instead.
2- with update-proxy-setup service
user@heads-tests:~/heads$ sudo apt update
Hit:1 http://HTTPS///deb.debian.org/debian bullseye InRelease
Hit:2 http://HTTPS///deb.debian.org/debian-security bullseye-security InRelease
Hit:3 http://HTTPS///packages.element.io/debian default InRelease
Hit:4 http://HTTPS///deb.qubes-os.org/r4.1/vm bullseye InRelease
Hit:5 http://HTTPS///updates.signal.org/desktop/apt xenial InRelease
Hit:6 http://HTTPS///wire-app.wire.com/linux/debian stable InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
102 packages can be upgraded. Run 'apt list --upgradable' to see them.
Forgot to mention that inotify-tools are required.
Interesting notes under debian-12:
sudo apt install extrepo extrepo-offline-data inotify-toolssudo vim /etc/extrepo/config.yaml #enable contrib and non-free- run in another terminal (aimed to be deployed as services per this issue):
sudo inotifywait -m -e modify,create -r /etc/apt/sources.list* | while read path action file; do sudo sed -i s^https://^http://HTTPS///^ "$path$file"; done sudo extrepo --offlinedata enable signal && sudo apt update && sudo apt install signal-desktop
Signal installed which repo info magically changed for the user in the background
Unrelated:
Opening issues upstream on extrepo-data project for Element: https://salsa.debian.org/extrepo-team/extrepo-data/-/issues/21
Other software support issues will follow.
in link with qubes-repo-templates packages now needed under dom0's updatevm's template, the following will also need to be modified, whatever distro:
sudo sed -i s^https://^http://HTTPS///^ /etc/qubes/repo-templates/qubes-templates.repo
But might need more thinking. Would not make really any sense having sys-firewall's being dom0 updatevm and having cacher actually download the template rpm. That would mean sys-firewall having a rpm passed by cacher to dom0. Sounds like pool explosion.
Also linked to #14