Proposed Architecture for Expanded Flake Support
nrdxp opened this issue · 15 comments
Meta
This draft concerns, not only NUR's own flake.nix in this repo, but an optional, standardized flake.nix format for the already existing user repos. This way, we can keep NUR as is while providing a nice interface for those using flakes, and a potential upgrade path if/when flakes become standard. If there is a better place for meta issues like this, I apologize.
Motivation
I love NUR and its lineage in the AUR, and Nix allows us to do things not possible with a traditional distribution mechanism. Now that flakes exist, they stand to solve a lot of outstanding problems (not least of which is the manual management of fixed ouput derivations and their associated hashes) in an official (though not yet stable) API for distributing Nix expressions. Thanks to NUR, we have a huge collection of community built derivations. This seems like the perfect time to build the bridge from stable Nix to 3.0 and reap the potential rewards while maintaining a majority of the infrastructure of NUR.
unordered benefits include:
- maximum flexibility for the user, allowing standard flake usage, while optionally adopting NUR.
- a future where autogeneration of sources is managed directly in Nix, not external tooling.
- No radical change means no loss of the existing architecture.
- Much of the existing CI infrastructure for NUR requires only moderate modification
- Flakes need more users! The more adoption occurs while unstable, the better the end result for Nix, which benefits its entire ecosystem. NUR is uniquely positioned to illicit a gentle push toward flake adoption, a totally optional position, while maintaining/using existing NUR code as is.
Schema for user repos
Conceptually, a simple, namespaced copy of what flakes already expect.
{
inputs = { /* ... */ };
outputs = { ... }: {
nur = {
nixosConfigurations = { /* ... */ };
overlay = final: prev: { /* ... */ };
overlays = {
firstOverlay = final: prev: { /* ... */ };
};
packages = { /* ... */ };
legacyPackages = { /* ... */ };
# ...
};
}We could optionally exclude overlays for the same reasons outlined in the README.md.
An additional benefit of this approach is a possible CI sanity check to ensure a flake is actually structured properly for NUR.
Usage
The contents of an above flake's outputs.nur is exported from NUR as "github:nix-community/NUR#packages.<system>.repoNamePackageName" for each output. Not only does this keep with flakes spec, it reduces depth. If we don't care about the standard flake spec, we could be more flexible: "github:nix-community/NUR#<repo>.<system>.packages.packageName".
If we come up with solid and usable alternate spec, we may be able to encourage changes to the upstream flake spec, benefiting everyone.
Changes to NUR
In theory, we could generate a simple list of flake refs with github's API, the above schema and a special tag: nur-flake. This allows adding repos to NUR in a completely decentralized fashion.
Of course, if this is too radical, we can keep the exact same workflow as currently implemented, except instead of modifying repos.json, they simply post a flake ref to the list, possibly repos.nix. This allows an added benefit of hosting repositories outside of GitHub.
If it's decided to autogenerate the list, users can still fallback to a traditional PR for repos outside of GitHub.
Alternatives
The obvious alternative is the adoption of a standard flake.nix in user repos. This may not be ideal — a user may wish to employ standard flake outputs that aren't necessarily meant for NUR, i.e. their personal NixOS/Darwin config. Maintaining an external repo just for NUR and proceeding to wire it in to their own flake config may discourage contribution. This friction may be overestimated and/or people may want to share most/or all of what they use personally. We can still achieve the important benefits outlined above by using a standard flake.
Conclusion
However we decide to do it, we may benefit immensely by wiring up existing NUR expressions through the existing flake outputs, while encouraging users to do the same.
This document is certainly not an exhaustive consideration, but a starting point for the next generation of NUR and Nix 3.0. Please help advance the discussion below.
Would outputs.nur not break nix flake show since its an not standardized output?
Output for a random test flake with one nur attr:
path:/home/johannes/testflake?narHash=sha256-PnWMhWdWqT030brMRE8boaXTpY0s0CP8yqaC311vqy0=
[...]
├───nur: unknown
└───packages
└───x86_64-linux
[...]
Why should I want to ensure that an attribute of a public flake not be visible in NUR, apart from it including secrets/being unbuildable? The latter case can be caught by respecting a meta.broken attribute everywhere.
Is there any other reason to keep NUR outpus in a separate namespace? It would mean duplication (plain flake + nur namespace), only this time in a flake.nix, while currently you'd need to create analogous to the nur namespace a shim file to wire NUR and the flake.
Also, why is your NUR flake not namespaced by system? This way it is not usable directly from a flake as nixpkgs cannot infer the currentSystem in the flake. It then needs an initialized nixpkgs passed (e.g. by NUR).
This is also a concern for NUR+flakes: I think NUR needs to require (e.g. via ci) the flakes to be compatible with the heads of the nix{os,pkgs}-unstable and nixos-<current-release branches. Therefore it must inject its own nixpkgs into the flake. Otherwise you might end up with 20 different nixpkgs in your store just because you're using packages from 5 NURs depending on other NURs.
You could also introduce to the repos.nix a flake repo type, for a smooth upgrade path. When using NUR with a stable Nix, you'd then need flake-compat to use them, if they declare inputs unknown to NUR.
Would outputs.nur not break nix flake show since its an not standardized output?
nix flake show would be broken for the nur output directly from the users repo, but if NUR parses this information and includes it in a standard flake output of it's own, show and search should work properly directly from NUR.
Why should I want to ensure that an attribute of a public flake not be visible in NUR, apart from it including secrets/being unbuildable? The latter case can be caught by respecting a meta.broken attribute everywhere.
Using meta.broken would not be good if your derivation isn't actually broken on your end, but includes a secret that breaks it for the public. You could also use a let binding to ensure that nur and the flake outputs are almost identical like so:
{
outputs = { ... }: let out = { /* ... */ }; in
out // { nur = out; somethingElseNotForNUR = { /* ... */ }; };
} In practice, a call to recursiveUpdate would be better than //, but this is just for illustration.
Also, why is your NUR flake not namespaced by system? This way it is not usable directly from a flake as nixpkgs cannot infer the currentSystem in the flake. It then needs an initialized nixpkgs passed (e.g. by NUR).
Not all flake outputs are "systemed", and since nur would essentially be a replica of the flake, the ones that are will still be namespaced by system.
This is also a concern for NUR+flakes: I think NUR needs to require (e.g. via ci) the flakes to be compatible with the heads of the nix{os,pkgs}-unstable and nixos-<current-release branches. Therefore it must inject its own nixpkgs into the flake. Otherwise you might end up with 20 different nixpkgs in your store just because you're using packages from 5 NURs depending on other NURs.
I have considered this, but hadn't yet nailed down a solution. Perhaps we could take arguments to the NUR output, a benefit a tranditional flake does not yet have, although it's been discussed. So something like:
{
outputs = { ... }: {
nur = { nixpkgs, possiblyOtherInputs, ... }: { /* ... */ };
};
}Then when NUR picks it up, it can pass in it's nixpkgs, avoiding the n nixpkgs problem.
NUR can always pass in its own nixpkgs (if they are called nixpkgs) by declaring
inputs.some-nur-repo.inputs.nixpkgs.follows = "nixpkgs";
Why should I want to ensure that an attribute of a public flake not be visible in NUR, apart from it including secrets/being unbuildable? The latter case can be caught by respecting a meta.broken attribute everywhere.
Using meta.broken would not be good if your derivation isn't actually broken on your end, but includes a secret that breaks it for the public.
Still, the flake is also public and you don't want it to be broken. So, either use another private flake (or privately a flake combining the public flake with your secrets) or you'd better use a solution with flags, e.g. meta.needSecrets/being a function of the secrets/some more intelligent detection of secrets use (secrets are only provided by the Reader Secrets Monad and everything in this Monad is filtered out ;D).
Not all flake outputs are "systemed", and since nur would essentially be a replica of the flake, the ones that are will still be namespaced by system.
You're right, I didn't read your example carefully enough and thought they were missing, but they are only hidden in ...s.
The question is what the core value would be if NUR would provide deeper flake integration. I would say it's discoverability - however in that case one might also just build a search engine around flake no?
The question is what the core value would be if NUR would provide deeper flake integration. I would say it's discoverability - however in that case one might also just build a search engine around flake no?
I agree that discoverability is the key benefit. You could simply nix search "github:nix-community/NUR" SEARCHTERM and find everything NUR has to offer. I guess the benefit of this over an external search engine is that it's built right into Nix. If you know how to use nix search and you know about NUR, you're good to go.
It also helps the community stay cohesive. It seemed, when flakes were first proposed, that the main objection was the possible fragmentation that it could cause. I've thought a lot about this problem, but it took a little push from @jorsn to help me realize that NUR was in a unique position to solve it.
The proposed expansion to NUR could mitigate this, as users no longer have to go tracking down every useful flake themselves, or through yet another external tool, but can pull right what they need, directly from NUR. Since NUR is already included in the default Nix registry, it's at the users disposal right after installing (unstable) Nix. With the ability to declare substituters directly from a flake, the nix-community binary cache would "just work".
Another benefit would be to direct people toward standard Nix features rather than the myriad of tools that have cropped up to adress the issues that flakes can help solve.
I agree that discoverability is the key benefit. You could simply
nix search "github:nix-community/NUR" SEARCHTERMand find everything NUR has to offer. I guess the benefit of this over an external search engine is that it's built right into Nix. If you know how to usenix searchand you know about NUR, you're good to go.
Ok. Is this something your proposed design would solve? Also it would that not require to download all nur repositories, which might be potentially slow or might even fail because someones gitea hosted on a pi is down? The latter one might be solvable by having some sort of fallback: https://github.com/nix-community/nur-combined but I am not quite sure how, maybe git branches?
One could import the flake with all its inputs to the store using nix flake archive and then cache the flakes in cachix. This might work, but I'm not sure. Alternatively, one could use nur-combined directly.
The question is what the core value would be if NUR would provide deeper flake integration.
Another value would be that NUR is ci-checked. You have no guarantees for arbitrary flakes.
Here is an example of how you can flakify your NUR already somehow: https://github.com/jorsn/nur-packages/tree/flake
Somehow, because it tries to access urls (flake inputs) and therefore doesn't pass the CI checks, at least for non-flake nix + flake-compat + restrict-eval = true.
Another benefit would be to direct people toward standard Nix features rather than the myriad of tools that have cropped up to adress the issues that flakes can help solve.
I think this statement can only be underrated. This is the most annyoing part of all in nix. And I would add: that each solution is compatible (thanks to nix-lang) with each other, but there are probably as many ways to plug solutions together as there are solutions. An absolutly magnificent productivity killer, as far as I'm concerned.
(btw that's something that divnix/devos first addresses in a compelling way)
Ok. Is this something your proposed design would solve?
In theory at least, I was playing around with an implementation a few days ago and ran into some strange issues with nix not wanting to fetch the repos. I attempted to workaround with nur-combined and got farther, but nix search wasn't showing everything which kinda defeats the purpose.
I'll keep experimenting in my spare time, and possibly revise the design if need be.
Did you set allow-import-from-derivation = true?
In case namespacing was a problem, did you try putting nur in legacyPackages? This is used for the namespaced package set by nixpkgs, as packages.${system} must be flat, I think. However, in nixpkgs nix search doesn't show the contents of every namespace either. Does nixpkgs.lib.recurseIntoAttrs help you?
ceterum censeo: simple flakes has an elegant solution for how to deal with name spacing locally (flattened nix output) and remotely when sharing overlays:
I'm gonna close this for now since my experiments ended in failure thanks to flakes rigid limitation on inputs (cannot be run through any sort of computation). Working around that would sorta defeat the simplicity of the design. Perhaps, if the restrictions are relaxed before flakes stabalize, I can try again.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/state-of-nur-in-the-world-of-flakes/12543/4