Dynamically build/load ZFS module on default archiso

Question

Dynamically build/load ZFS module on default archiso

eoli3n opened this issue 4 years ago · 33 comments

Hi,

I'm trying to setup my first Archlinux ZFS.
I want to be able to use default archiso, because i use archlinux netboot as installer, or recovery.

Problem

Today, latest archiso runs on 5.5.13.arch2-1 kernel.
Archzfs repo has only zfs-linux build for latest kernel.

Possible workaround

Compile module with `zfs-dkms`

You need to:

add archzfs repo and pacman -Sy to update lists
script current linux-headers install, then install zfs-dkms
Not as simple, here's a try : https://github.com/eoli3n/arch-config/blob/02540d746e9961016b6574ef55cfd386855ebf1c/scripts/install/01-configure.sh#L50
But zfs-dkms fails build, i'm on it. It seems that updating all lists, breaks some dependencies.

A better way : keep archive packages

Store precompiled zfs-linux and zfs-linux-lts packages for all kernels since latest archiso release. As archiso is rebuilt each month, it should not cost a lot to store only current month versions. Just to get an idea of how many versions you may need to store per month.

For April 2020

❯ curl -s https://archive.archlinux.org/packages/l/linux/ | grep 'Apr-2020' | grep sig | wc -l
10

For Mars 2020

❯ curl -s https://archive.archlinux.org/packages/l/linux/ | grep 'Mar-2020' | grep sig | wc -l
9

For February 2020

❯ curl -s https://archive.archlinux.org/packages/l/linux/ | grep 'Feb-2020' | grep sig | wc -l
10

If you do this, maybe it is possible to add meta-package on archzfs, which match current iso kernel version, then install the right zfs module automatically if possible. I can help on this.

Have a nice day.

Answer 1 · 2020-05-01T15:29:41.000Z

Here's a workaround : https://github.com/eoli3n/archiso-zfs

Answer 2 · 2020-05-01T15:55:19.000Z

There used to be the zfs-archiso package, but it hasn't been updated in a while since I didn't add it to the new build system.
However all packages that have ever been build are stored here: http://archzfs.com/archive_archzfs/.
So in theory you could fetch the right one and use it (unless the right version was never build)

Answer 3 · 2020-05-01T16:41:28.000Z

Thanks for your answer.
Is there a way to find for which kernel package a specific zfs-linux package has been built ? (without extracting it).

Answer 4 · 2020-05-01T16:50:41.000Z

Is there a way to reorganize how you store versions, to match how Arch Linux Archive works ?

No need to rearrange everything. Just making the change from today will make easier for my script to know what it needs to use.

See : https://github.com/eoli3n/archiso-zfs/blob/master/init#L18

EDIT : I'm writing a script which automates it for you

Answer 5 · 2020-05-02T13:00:52.000Z

Here's what i did : https://gist.github.com/eoli3n/a410087b856fce02cefe0ddeccad159b

With this, you can keep your repo as is, it just generates symlinks. For now, it uses stat to find creation date, but we can adapt it...

Answer 6 · 2020-05-02T23:42:36.000Z

https://github.com/seblu/archivetools
It uses hardlinks. I will update the gist.

Answer 7 · 2020-05-03T16:33:56.000Z

It seems that the only good solution, is to store hardlinks by kernel version. To be able to install easily packages by kernel versions as

.
├── linux
│   └── 5.6.7-arch1-1
│       └── zfs-linux-1.1.tar.xz
├── linux-lts
│   └── 5.6.7-arch1-1
│       └── zfs-linux-1.1.tar.xz

Answer 8 · 2020-05-03T16:37:53.000Z

We just discussed this on irc, the only proper solution is to provide a custom repo with a zfs-modules-arch package in sync with archiso.

Answer 9 · 2020-05-03T17:36:40.000Z

Another alternative might be ALEZ, which is the Arch installer + working ZFS. It has some install script, but you can just not use it. This is what I use as my "recovery" ZFS image mostly, but I've installed from it too.

Answer 10 · 2020-05-03T17:37:59.000Z

I want to be able to use default archiso, because i use archlinux netboot as installer, or recovery.

Answer 11 · 2020-05-03T17:40:37.000Z

I guess I don't understand enough about netboot to grok why you couldn't use a different archiso, one that is actively maintained and kept up to date.

Answer 12 · 2020-05-03T17:45:22.000Z

ALEZ can solve this easily by just shipping base-devel and linux-headers (to build zfs-dkms), or zfs-modules-arch in the iso.

Answer 13 · 2020-05-03T17:47:05.000Z

But ALEZ comes w/ a kernel and zfs for that kernel, no need to build zfs-dkms at all?

Answer 14 · 2020-05-03T17:47:53.000Z

Then it's not the official arch kernel, but that works too.

Answer 15 · 2020-05-03T18:06:12.000Z

Do you mean official latest kernel? Or just official kernel? I'm pretty sure it uses the official kernel (from the latest installer) and the archzfs package that goes w/ it. But it isn't kept up to date w/ the latest kernel.

Answer 16 · 2020-05-03T18:08:20.000Z

Ah, sorry. Misread your reply. It seems like it comes with a zfs-modules-arch package then, or at least, that's what it should be called.

Answer 17 · 2020-05-03T18:49:23.000Z

@fryfrog https://eoli3n.github.io/archlinux/2020/04/25/recovery.html
That issue is about using default archiso, not anything else.

Answer 18 · 2020-05-05T09:57:52.000Z

@FFY00 can you simple explain what would be your solution ?

New archiso release triggers github action to build current archiso kernel zfs module
A github action task uploads the built package in a repo
This repo provides a zfs-modules-arch which install that current module on archiso booted system.

Questions :

How to automatically triggers github action to build package, as i don't control archiso release scripts ?
Why build a package, as archzfs already does it ?
How zfs-modules-arch will work if a user doesn't use latest archiso release ?

Answer 19 · 2020-05-05T14:37:29.000Z

1. How to automatically triggers github action to build package, as i don't control archiso release scripts ?

Trigger the build on a cronjob running every hour or so on the 1st of every month. In the job you will check if the iso has been released already and if it has you can update the package.

2. Why build a package, as `archzfs` already does it ?

Because you need the modules for the kernel used to boot archiso.

3. How `zfs-modules-arch` will work if a user doesn't use latest archiso release ?

Nop. But there's no better solution. I mean, you can always roll out your own iso based on archiso, that would be the better way to do it I guess.

Instead of replacing the modules package in the repos, you can have multiple repos and keep the modules for older versions of archiso.

Answer 20 · 2020-05-05T16:53:26.000Z

So, i highly disagree with you:

Trigger a cron will be exactly as effective as just querying archives by dates. In both solutions, you have a gap of some hours.
As effective module exists, i just need to know how to query it.
My solution is working as effective as your with the latest archiso. It also work with all archiso ever.

An archiso is a recovering system, when you host is not booting, you need to be able to boot what you have. If you have an usb stick with archiso 2020/01/01, then with my solution, you can install already compiled zfs module matching the running kernel.

Answer 21 · 2020-05-05T17:06:20.000Z

Trigger a cron will be exactly as effective as just querying archives by dates. In both solutions, you have a gap of some hours.

You can run a cron job every minute for all I care, I am not the one hosting the runners. AFAIK it doesn't go agaisnt github TOS.

As effective module exists, i just need to know how to query it.

It doesn't. Can you elaborate?

My solution is working as effective as your with the latest archiso. It also work with all archiso ever.

It isn't. You are assuming the kernel will not get updated the same day archiso is released, after archiso is released. That assumption is not correct, and things will break when it doesn't happen.

An archiso is a recovering system, when you host is not booting, you need to be able to boot what you have. If you have an usb stick with archiso 2020/01/01, then with my solution, you can install already compiled zfs module matching the running kernel.

Hum? Aren't you installing zfs-dkms? The module is definitely not compiled.

And if you want to compile it you run into problems, linux-headers is not installed in archiso and it's needed to compile the module. Tracking down the correct linux-headers package is relatively easy, but as pointed out in IRC, it depends on GCC. So now you also need to track down the correct GCC package, which is less trivial. Alternatively you can build a linux-headers package yourself with the archiso GCC, but that requires you to download a tarball of the kernel source, which might take some time.

Answer 22 · 2020-05-05T17:41:19.000Z

It doesn't. Can you elaborate?

It actually does. http://archzfs.com/archive_archzfs/

It isn't. You are assuming the kernel will not get updated the same day archiso is released, after archiso is released. That assumption is not correct, and things will break when it doesn't happen.

That's the problem of the gap i said. You're right on this, but it's easy to script a fix:

Search for zfs-kernel package based on date.
Extract it, if PKGBUILD match kernel version, so go with it
Else, try day+1

Hum? Aren't you installing zfs-dkms? The module is definitely not compiled.

Did you read the start of that issue ?
Please read #337 (comment)

You don't get what i'm doing.

For now, my script (please read https://github.com/eoli3n/archiso-zfs/blob/master/init) use Arch Linux Archives to find community archive based on linux package build date.
Then it install a matching version of linux-headers, to be able to build with dkms-zfs.

What i ask to @minextu , is to rearrange http://archzfs.com/archive_archzfs/ with the script i did, to be able to use the same mecanic as Arch Linux Archive does.
That repo hosts every zfs-linux versions ever built, which are zfs precompiled module for a specific kernel version.

If he use my script (please read #337 (comment)) then, i will be able to rewrite my script (https://github.com/eoli3n/archiso-zfs/blob/master/init) to search for the right zfs-linux package instead of the right linux-headers package.

For those both solutions, as i can set the repo by date, for archzfs repo and community repo, then i can install linux-headers package OR zfs-linux package matching the running kernel version.

DKMS version of the script seems better, because, for any archiso/kernel, you will have last zfs module version.

Answer 23 · 2020-05-05T18:01:41.000Z

It actually does. http://archzfs.com/archive_archzfs/

Which package are you refeering to? None of those works here.

That's the problem of the gap i said. You're right on this, but it's easy to script a fix:
* Search for `zfs-kernel` package based on date.

* Extract it, if PKGBUILD match kernel version, so go with it

* Else, try day+1

zfs-kernel provides a kernel, you need to reboot for it to become active. archiso boots to a tmpfs, not on disk, installing zfs-kernel does absolutely nothing as it gets lost on reboots.

Did you read the start of that issue ?
Please read #337 (comment)

You don't get what i'm doing.

For now, my script (please read https://github.com/eoli3n/archiso-zfs/blob/master/init ) use Arch Linux Archives to find community archive based on linux package build date.

Then it install a matching version of linux-headers, to be able to build with dkms-zfs.

You are the one not reading my replies:

linux-headers is not installed in archiso and it's needed to compile the module. Tracking down the correct linux-headers package is relatively easy, but as pointed out in IRC, it depends on GCC. So now you also need to track down the correct GCC package, which is less trivial

So currently when you install zfs-dkms you build the module with the incorrect GCC version. The ABI might, by chance, be compatible but it might break anytime. And when it breaks it might not be obvious -- unstable system, kernel crashes, etc.

What i ask to @minextu , is to rearrange http://archzfs.com/archive_archzfs/ with the script i did, to be able to use the same mecanic as Arch Linux Archive does.
That repo hosts every zfs-linux versions ever built, which are zfs precompiled module for a specific kernel version.

Okay, so zfs-linux is the kernel module. That naming is incorrect, it should be something like zfs-linux-arch or zfs-linux-archzfs.

There is no rearrangement needed, you can just search the index since it lists every package.

If he use my script (please read #337 (comment)) then, i will able to rewrite my script to search for the right zfs-linux package instead of the right linux-headers package.

Answer 24 · 2020-05-05T18:09:45.000Z

As you don't understand what i am doing, you are not helping. Please let me see it with @minextu

Answer 25 · 2020-05-05T20:50:48.000Z

First of all: Thanks @eoli3n for bringing this up and even writing a script to organize the archive!

I personally was never a fan of having just one package for the newest archiso in the repo. When I have to fix my system I mostly just use a usb stick with a random (and most likely older) archiso. So I can't use the zfs-archiso package anyway. Because of that I haven't had enough motivation to integrate it into the build system.

However I do think that having a automated built archiso image with zfs integrated would be a great solution and I actually started work on that some time ago. I just need to find time to fix a few issues with it (probably just needs to be rebased onto the newest archiso profile).
But that doesn't mean that this has to be the only solution. I don't see any reason why organizing the archive would be a bad thing and if it helps people make zfs based installations and recovery easier, even better.

@eoli3n are you ok with it, if I modify your script and put it into this repository? It think it might make sense to run it every time when packages get moved to the archive.

Answer 26 · 2020-05-05T21:13:53.000Z

The symlinker script is WTFPL, you're welcome :)

Actually i need to apologize to @FFY00, we talked again on IRC and he made me notice that your zfs-linux packages actually embeed kernel version in their names...
So i rewrote the script : https://github.com/eoli3n/archiso-zfs/blob/nodkms/init
The bad news is that there isn't any zfs-linux build for the current archiso.
Current archiso embeed kernel 5.6.8.arch1-1, on repos i only get zfs-linux for 5.6.8.arch1.1-1.

❯ curl -s http://archzfs.com/archzfs/x86_64/ http://archzfs.com/archive_archzfs/ | grep -Po 'href="\Kzfs-linux-[0-9].*?_5.6.8.*?(?!.+\.sig)x86_64[^\"]+'
zfs-linux-0.8.3_5.6.8.arch1.1-1-x86_64.pkg.tar.xz

I will continue that script to fallback to DKMS method, but this is still not the best solution.

I will look at @FFY00 solution. Maybe we need to work together @minextu.

I really don't like the fact to use a custom iso, as said in the first issue, i use netboot as recovery, it makes Archlinux on ZFS still a specific management with custom process, not dynamic and flexible.

Answer 27 · 2020-05-05T21:20:14.000Z

The zfs module for 5.6.8.arch1-1 is versioned as 5.6.8.arch1.1-1. The pkgrel of the kernel get's attached to the end with a . instead of a -, so every zfs package can have their own pkgrel

Answer 28 · 2020-05-05T21:22:19.000Z

But you might still need the dkms workaround, for cases where a kernel was released immediately after the iso and the kernel was skipped by the build system.
Remember that older zfs modules might depend on older utils packages. So you would need to fetch the right one in those cases

Answer 29 · 2020-05-05T21:46:10.000Z

The zfs module for 5.6.8.arch1-1 is versioned as 5.6.8.arch1.1-1

Dirty fixed that eoli3n/archiso-zfs@54ac44f
This will work in any cases ?

Remember that older zfs modules might depend on older utils packages. So you would need to fetch the right one in those cases

So i will combine two methods: in any cases i will set community repo to kernel build date.

I can't install the package, signature problem, any idea how to fix it ?

Fixed by adding archzfs repo and sign it eoli3n/archiso-zfs@53ee10a

But now, dependency problem :/

Is there a way to add the complete archzfs archives dir as a repo ? I tried but it didn't work

Answer 30 · 2020-05-05T22:00:00.000Z

Dirty fixed that eoli3n/archiso-zfs@54ac44f
This will work in any cases ?

Yes but there could be multiple packages for the same kernel. You should probably grab the latest one (highest pkgrel)

I can't install the package, signature problem, any idea how to fix it ?

Looks like the signing key hasn't been imported, looks like you fixed it in the latest commit

But now, dependency problem :/

You need zfs-utils which is in the archzfs repo. That's what I meant earlier: You might need to fetch a older version depending on which version the module needs.

Is there a way to add the complete archzfs archives dir as a repo ? I tried but it didn't work

No since it's not an actual repo :|

Answer 31 · 2020-05-05T22:03:29.000Z

Yes but there could be multiple packages for the same kernel. You should probably grab the latest one (highest pkgrel)

The space regex does that.
https://github.com/eoli3n/archiso-zfs/blob/53ee10aeb0a71c870a81ab4023915dde59966aac/init#L40

You need zfs-utils which is in the archzfs repo. That's what I meant earlier: You might need to fetch a older version depending on which version the module needs.

Yes but how ? Do I need to extract PKGBUILD to look for which version of zfs-utils it uses, then install it first ?

Answer 32 · 2020-05-05T22:05:18.000Z

That would be the best way. Alternately you could extract the zfs version (0.8.3) from zfs-linux and find the utils package with the same version

Answer 33 · 2020-05-06T11:03:55.000Z

The space regex does that.

Actually it didn't, that's now fixed.

Done ! No fallback on DKMS yet

EDIT : It works on 2020/04 archiso !

And on netbooted 2020/03 \o/