Flakes UX: make it easy to evaluate an arbitrary expression in a context of a flake

Question

Flakes UX: make it easy to evaluate an arbitrary expression in a context of a flake

Opened this issue 3 years ago · 40 comments

This is a design discussion issue, so feel free to give as much feedback and criticism as you like.

Preamble

In ye olde days of nix-shell, it was possible to easily evaluate expressions in the context of nixpkgs. Something like this:

nix-shell -p 'python3.withPackages (ps: with ps; [ numpy scipy ])'

Drops you into a shell in which there's a python3 executable with numpy and scipy available. This is really useful for ad-hoc scripting. You could also do something like

nix-shell -p 'hello.overrideAttrs (_: { src = ./.; })'

It was also possible to add alias p='nix-shell -p' to save some typing.

The flexibility is endless -- and all just a few keystrokes away!

Current situation

It is possible to do something similar with new, flaky interface:

nix shell --impure --expr 'with builtins.getFlake "nixpkgs"; with legacyPackages.${builtins.currentPlatform}; python3.withPackages (ps: with ps; [ numpy scipy ])'

However, this is a chore to do, and I'd much rather just run pip install numpy scipy and deal with all the usual stateful package management issues than type this monstrosity every time I need to try a python package.

Proposal

Add a way to easily evaluate expressions "in the context of" a flake, meaning with the top-level attributes and {packages,legacyPackages}.${currentPlatform} of that flake available in scope.

Possible implementations

I came up with three different ideas, all of which have some advantages and drawbacks.

(my current favourite) Allow flake fragments to be arbitrary expressions

Example: nix shell 'nixpkgs#(python3.withPackages (ps: with ps; [ numpy scipy ]))'

This feels like a natural extension of the already-existing flake-fragment-as-attrpath interpretation. It shouldn't be too difficult to implement, and doesn't break compatibility. It does however introduce some potential complexity to newcomers.

With alias p='nix shell' it is possible to do p 'nixpkgs#python3.withPackages (...)' , and some more keystrokes could be saved with a registry entry n -> nixpkgs: p 'n#python3.withPackages (...)'

Add a new flag, like `--with`

Example: nix shell --with nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

This plays nicely with Nix' with keyword. It implies that some expression will be evaluated with some additional context.

Aliasing p='nix shell --with nixpkgs' can give experience identical to the status quo -- p 'python3.withPackages (...)'

Change the behaviour of `--expr`

Example: nix shell --expr nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

This breaks compatibility, but may be nicer than --with since --expr was not very usable with flakes anyways and it's not nice to have a lot of useless flags around.

K900 commented 2 years ago

Bad bot.

🎉1

Answer 1 · 2021-11-15T16:39:21.000Z

+1 to nix shell nixpkgs#(<expression>) syntax

One thing I wonder about also is being able to pull multiple flakes into scope. (the equivalent of with x; with y;) Maybe with a syntax like [nixpkgs otherFlake]#(<expression>) Maybe once you get to this point it's better to make a flake.nix file and take them both in as inputs.

This may go too far down the road of complexity without enough real use-cases.I remember there was a use-case for this that came up for me a little while ago, but I don't remember what it was. I'll write it down if it ever comes up again.

EDIT: I talked it through with @balsoft and we're in agreement that my addition isn't a great move right now. It changes the semantics of flake paths away from being "urls" of sorts, which is a much bigger decision, even if we want to go there.

Answer 2 · 2021-11-17T09:06:29.000Z

I’m a big fan of the overall idea, that’s something I want to have available reasonably frequently, and it’s always a pain. I’m not sure what the UX should be though:

The nixpkgs#(expression) syntax would make a lot of sense if the Nix language had a x.(expr) syntax (like what OCaml has for modules), but it doesnt, (and for good reasons imho). So it’s a bit confusing. (and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs)
--with <flakeref> sounds nice, but since it only makes sense with --expr, which isn’t very clean
--expr <flakeref> <expr> isn’t really viable, because part of the point of --expr is to make it possible to use the nix command with non-flake things.

A fourth way could be to have a --expr-with <flakeref> <expr> flag. But it’s not utterly pretty either.

Answer 3 · 2021-11-17T09:36:28.000Z

My --with flag suggestion is actually --with <flakeref> <expr>, so like your --expr-with suggestion. But having --with <flakeref> --expr <expr> is also a possible solution.

The nixpkgs#(expression) syntax would make a lot of sense if the Nix language had a x.(expr) syntax

I think the installables syntax is already different from usual Nix syntax. E.g. <flakeref>#<attrpath> will not evaluate to the same thing in actual Nix language as it does in the attributes, so I think some syntactic liberty could be allowed. But I also see your point, because it can be even more confusing for newcomers if there's even more strange syntax to learn and understand.

Answer 4 · 2021-11-18T15:26:49.000Z

and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs

As far as I understand it, this evaluates to the same thing in the existing and proposed cases. It's like the difference between (with foo; bar) and (foo.bar)

Answer 5 · 2021-11-18T16:07:43.000Z

and to be pedantic, it’s reusing some already valid syntax: nixpkgs#(foo) will evaluate to the (foo) field of nixpkgs

As far as I understand it, this evaluates to the same thing in the existing and proposed cases. It's like the difference between (with foo; bar) and (foo.bar)

Not exactly. Current nixpkgs#(foo) is (builtins.getFlake "nixpkgs").[Search path Magic]."(foo)" while with this proposal it would be (builtins.getFlake "nixpkgs").[Search path Magic].foo (so the attr name is different).
But that’s really nitpicking because that’s a really small edge-case (and the syntax is officially unstable anyways)

Answer 6 · 2021-11-18T20:15:17.000Z

For the new UI, we have to careful about adding features that are not discoverable or have a steep learning curve. For instance, in the example

nix shell --expr nixpkgs 'python3.withPackages (ps: with ps; [ numpy scipy ])'

it's not obvious how a user would discover the existence of python3.withPackages or figure out that they need to write (ps: ...).

There's also the issue of cacheability. Currently we can cache flake output attributes like nixpkgs#pythonPackages.foo, but not arbitrary expressions like the above.

Answer 7 · 2021-11-18T20:57:45.000Z

If we can translate CLI arguments to function calls, it's possible to build functions that work for common cases. This assumes some cooperation between nixpkgs and the nix CLI.

Eg: nix apply nixpkgs#python3.with --packages numpy,scipy

Could map to (builtins.getFlake "nixpkgs).legacyPackages.${builtins.currentSystem}.python3.with { packages = "numpy,scipy"; }

It makes me think of httpie since they also try to convert CLI to JSON data structures: https://httpie.io/docs#non-string-json-fields

Answer 8 · 2021-11-25T15:03:41.000Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-22/16251/1

Answer 9 · 2021-12-10T16:49:39.000Z

I recently wanted to do the exact same thing as @balsoft, and I did the exact same workaround. I didn't know whether it was supported natively or not so here's what I tried (What I intuitively thought would work): nix shell nixpkgs/nixos-21.11#python39 --expr "withPackages (p: [p.numpy])" I expected python39's attributes to be in scope for the expression but it wasn't

Answer 10 · 2021-12-13T15:12:29.000Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-shell-for-python-packages/16575/4

Answer 11 · 2022-06-20T02:26:06.000Z

I marked this as stale due to inactivity. → More info

Answer 12 · 2022-07-24T06:57:30.000Z

I recently wanted to do the exact same thing as @balsoft, and I did the exact same workaround. I didn't know whether it was supported natively or not so here's what I tried (What I intuitively thought would work): nix shell nixpkgs/nixos-21.11#python39 --expr "withPackages (p: [p.numpy])" I expected python39's attributes to be in scope for the expression but it wasn't

I wanted this to work too, so I made a patch for it: https://github.com/Minion3665/nix/compare/master...Minion3665:nix:5567-make-installables-expr-context.patch

Right now it only works on nixos-unstable (I believe it needs nix 2.9? I've only properly tested it on 2.8 (failing) and 2.10 (working fine) though)

You can apply it by doing something like this in an overlay:

final: prev:  {
  nixFlakes = prev.nixFlakes.overrideAttrs (old: {
    patches = (old.patches or []) ++ [
      ./nix/5567-make-installables-expr-context.patch
    ];
  });
}

where ./nix/5567-make-installables-expr-context.patch is the path to the patch file.

The patch works by adding with statements to the front of your expression; in the future I hope to improve the patch by doing something similar to how nix repl's :l command works but for now I wanted something that was a little quicker for me to make.

Answer 13 · 2022-09-20T08:45:40.000Z

@Minion3665 please upstream this patch, it's awesome 🤩

Answer 14 · 2022-11-13T00:14:54.000Z

I wanted this as well. I created this shell function until it's resolved:

# Python with packages
pwp () {
    local python=$1;
    shift;
    local pkgs=$@;
    nix build \
        --impure \
        --expr "with builtins; with getFlake \"nixpkgs\"; (getAttr currentSystem legacyPackages).$python.withPackages (ps: with ps; [ $pkgs ])"    
}

Example:

pwp python310 pika boto3 kafka-python
result/bin/python

It's even better than the nix-shell -p stuff IMHO

Answer 15 · 2022-12-02T15:28:08.000Z

If Nix was to allow specifying functions in place of "installables" and passing positional arguments to them, we could use it like: nix shell nixpkgs#python3.withPackages --argPos '(ps: with ps; [ foo bar baz ])'.
That said, Nix doesn't support passing positional arguments at all, and probably for a good reason.

Answer 16 · 2022-12-02T15:39:08.000Z

nix eval has a similar flag, --apply, that maybe could be ported to nix shell/build

$ nix eval --apply "p: (p.withPackages (pp: [pp.requests])).outPath" nixpkgs#python3
"/nix/store/vfyr42phq4h4gz63zg9qizz61a59503r-python3-3.10.8-env"

Answer 17 · 2023-03-03T21:38:57.000Z

I came here by searching "nix shell flake expr" and my use-case was python3.withPackages as well! 😅

My reasoning was that since the following works with the repl:

$ cd ~/projects/nixpkgs
$ nix repl .
nix-repl> python3.withPackages (pkgs: [ pkgs.pjsua2 ])
«derivation /nix/store/kchdd811lan46mlk8bdhqq70i87lv2r6-python3-3.10.9-env.drv»
nix-repl> :b python3.withPackages (pkgs: [ pkgs.pjsua2 ])

this derivation produced the following outputs:
  out -> /nix/store/cyc8nhbf0w4dsydi1alvby4xgcg3zja5-python3-3.10.9-env

Then why doesn't this work for nix shell?:

$ nix shell . --expr 'python3.withPackages (pkgs: [ pkgs.pjsua2 ])'

That doesn't make sense to me.

The patch of @Minion3665 seem to align with this reasoning. I couldn't find a PR of your branch, is there a PR already?

If not, do you mind to create one? Or if it helps, is it ok for me to create one from your patch?

Answer 18 · 2023-03-04T01:56:09.000Z

That doesn't make sense to me.

The patch of @Minion3665 seem to align with this reasoning. I couldn't find a PR of your branch, is there a PR already?

If not, do you mind to create one? Or if it helps, is it ok for me to create one from your patch?

There is no pr yet, I've considered making one however:

I'm pretty sure the patch doesn't work on the latest master, some work would need to be put in to make it work again
The patch tacks on a with statement to the given expression and calls it a day, this is both a little janky and also can cause error messages to be obscured by a blob of 'with' garbage

For these reasons I haven't PRed, however I'm more than willing to have someone else make a PR or to work with someone on improving the patch and PR myself.

I've planned to do these and PR for a while but I'm sure you know how life can be...

Answer 19 · 2023-07-27T16:16:40.000Z

Thanks to posters above for giving me a clue how to access e.g. pkgs for interactively running tests:

$ nix eval --impure \
    --apply 'pkgs: pkgs.callPackage pkgs/development/libraries/science/math/libtorch/test { cudaSupport = false; }' \
    .#legacyPackages.aarch64-darwin

Something like this for nix shell would be amazing (again, with the python + packages example being what has brought me to this thread most frequently).

Answer 20 · 2023-08-02T18:29:08.000Z

Adding on to examples like @viperML from above, I hadn't realized that nix shell accepts a path, so one can combine nix eval --raw --apply with nix shell and get a reasonable (?) replacement for nix-shell:

For example, working on a local checkout of nixpkgs:

$ nix shell "$(
        nix eval --raw --apply '
            py: (py.withPackages (pp: [ pp.torch ]))
        ' .#python310
    )" \
    -c python -c 'import torch; print(torch.__version__)'
2.0.1

Interestingly, I was initially getting segfaults and stack overflows trying to get this to work (maybe this issue?):

$ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
Segmentation fault: 11
$
$ # on another machine:
$ nix eval --apply 'py: (py.withPackages (pp: [ pp.requests ]))' nixpkgs#python310
...
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
trace: warning: Use `stdenv.tests` instead. `passthru` is a `mkDerivation` detail.
error: stack overflow (possible infinite recursion)

But simply adding .outPath made it work (thanks @viperML):

$ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ])).outPath' nixpkgs#python310
"/nix/store/ir5m4j69icbgh7rgi8hpcn75jl16jshq-python3-3.10.12-env"
$ nix eval --apply 'py: (py.withPackages (pp: [ pp.requests ])).outPath' nixpkgs#python310
"/nix/store/89smqkd3q9vqfjbxv12hlaznp5v1j39i-python3-3.10.12-env"

So does --raw:

$ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
Segmentation fault: 11
$ nix eval --raw --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
/nix/store/ir5m4j69icbgh7rgi8hpcn75jl16jshq-python3-3.10.12-env
$ nix eval --apply 'py: (py.withPackages (pp: [ pp.flask ]))' nixpkgs#python310
Segmentation fault: 11

--raw is required to strip the extra quotes from the path anyway, so it seems reasonable to use.

Oddly, when I add a second package it fails:

$ nix build "$(nix eval --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ]))' nixpkgs#python3)"
error: path '/nix/store/h2i9izgfc5fh5c4b4nrg1pjn5y12axzw-python3-3.10.12-env' is required, but there is no substituter that can build it
$ nix-shell -I nixpkgs=~/git/nixpkgs -p 'python310.withPackages (ps: with ps; [ flask requests ])'
[nix-shell:/tmp/test]$ echo $?
0

Answer 21 · 2023-08-02T19:09:36.000Z

@n8henrie, here's a fix for your last cmd:

nix shell "$(nix eval nixpkgs#python3 --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).drvPath')"

In general (and as you mentioned) you can run nix on arbitrary drv expressions that have single flake output as their input like so:

nix <cmd> "$(nix eval nixpkgs#pkgs --raw --apply 'pkgs:
 (<arbitrary expression that yields a derivation>).drvPath
')"

Answer 22 · 2023-08-02T19:17:43.000Z

Thanks! This also seems to work pretty well and doesn't require a separate call to nix shell:

"$(nix eval --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).out' .#python310)"/bin/python -c 'import flask; print(flask.__version__)'

Answer 23 · 2023-08-02T19:53:05.000Z

A bash function similar to what @Uthar uses above:

$ nixShellWithPythonPackages() {
    nix shell "$(nix eval --raw --apply "py: (py.withPackages (pp: with pp; [ $* ])).drvPath" nixpkgs#python310)"
}
$ nixShellWithPythonPackages torch requests
$ type -p python
/nix/store/p306n49i437di59p6lwn33qcksqrjk3z-python3-3.10.12-env/bin/python
$ python -c 'import torch; import requests; print(torch.__version__, requests.__version__)'
2.0.1 2.29.0

Answer 24 · 2023-08-02T20:21:20.000Z

This also seems to work pretty well and doesn't require a separate call to nix shell:

This only works because the drv has already been built previously. You want this:

$(nix build  --no-link --print-out-paths "$(nix eval nixpkgs#python3 --raw --apply 'py: (py.withPackages (pp: with pp; [ flask requests ])).drvPath')")/bin/python -c 'import flask; print(flask.__version__)'

Answer 25 · 2023-08-03T04:13:07.000Z

you probably want to use the nix shell variant to make sure the path isn't garbage collected between the call to nix eval and the execution of the python binary

Answer 26 · 2023-09-25T05:42:13.000Z

I second any sort of higher order implementation.

Infact I think we should match flake.nix output syntax:

$ nix build nixpkgs# github:somerepo#pkg --with '{nixpkgs, pkg}: nixpkgs.pkg2.override {inherit pkg;}'

Answer 27 · 2023-10-26T16:20:14.000Z

I'm interested in implementing this. It looks like a high-demand feature that could bridge the gap between the simple nix shell nixpkgs#asdf commands and writing a full-on flake. Just to make sure I'm grasping the problem and solution space correctly, here are my thoughts and the two potential solutions:

`--apply`, solution with precedent

The least controversial (but also least "nice") solution would be to port the --apply flag from nix eval to nix shell and nix build (or to any command that takes installables in general):

$ nix shell nixpkgs#python3 --apply 'py: (py.withPackages (p: [ p.flask ]))'

Differences to `nix eval`

This solution should not require the .outPath workaround you need for nix eval, it should not require --impure because the function itself is pure, and it should work for multiple installables, but nix eval applies the function passed to --apply to all of its inputs, which will very rarely be useful here. Rather, we want to be able to apply different functions to different installables.

There are two potential solutions for this:

Flake-like outputs functions

This is the option proposed by @YellowOnion:

$ nix shell nixpkgs github:somerepo#pkg --apply '{nixpkgs, pkg}: nixpkgs.pkg2.override {inherit pkg;}'

Unused inputs would just be put into the shell without applying the function:

$ nix shell nixpkgs#python3 nixpkgs#gnugrep --apply '{python3}: (python3.withPackages (p: [ p.flask ]))'

This is cool because it is very similar to how flakes work, making the transition easy and giving users a familiar interface. However, you have to type the name of the input three or more times, which is very annoying.

Split options

Basically the way --apply currently works, but extended to an arbitrary number of functions and installables.

$ nix shell nixpkgs github:somerepo#pkg --apply 'pkgs: p: pkgs.pkg2.override {pkg: p;}'

This is nice because it can be much terser. Again, if some inputs are unused, they will just be used verbatim.

$ nix shell nixpkgs#python3 --apply 'py: (py.withPackages (p: [ p.flask ]))' nixpkgs#gnugrep

Note that this order is not mandatory, but good style as it puts the function close to its input. It's also important to note that now the evaluation of the functions depends on the order of the installables. This is not the case with the flake-like solution.

Closer to `nix repl`: `--expr`

I do like the idea from @bobvanderlinden as well to align more with the interface of nix repl:

$ nix repl .
nix-repl> python3.withPackages (pkgs: [ pkgs.pjsua2 ])
«derivation /nix/store/kchdd811lan46mlk8bdhqq70i87lv2r6-python3-3.10.9-env.drv»
$ nix shell . --expr 'python3.withPackages (pkgs: [ pkgs.pjsua2 ])'
# or
$ nix shell nixpkgs#python3 --expr 'withPackages (pkgs: [ pkgs.pjsua2 ])'

Very clean, perfect for this simple usecase. I can't think of a way to really make this work with multiple installables, though.

Additionally, this doesn't cover all use-cases, it just makes the simple ones even simpler and allows jumping directly from a working repl statement to a nix shell invocation. So if this was added, I think it would have to be done in addition to --apply, not as a replacement.

Collision with current `--expr` installables

This solution completely reverses the current meaning of --expr, which replaces the initial part of the installable:

#           <installable> <                    output                         >
$ nix shell       nixpkgs#python3 --expr 'withPackages (pkgs: [ pkgs.pjsua2 ])'
#           <        installable       >  <output>
$ nix shell --expr 'import <nixpkgs> {}'  hello

Alternative 1: just call it `--eval`

Somewhat arbitrary, just call it --eval instead of --expr. Not a great solution, but not terrible either.

Alternative 2: two argument flag `--with`

A better option might be to implement this as 2-argument flag --with that can mirror the behavior of with inside the nix language:

$ nix shell nixpkgs#gnugrep --with nixpkgs#python3 'withPackages (p: [ p.flask ])'

This makes it very clear which function is applied to which flake, and makes confusing ordering an explicit error.

Interop

With other installables

Apart from flakes outputs, we also have store paths, files, and (as already mentioned) expressions.

For store paths, this feature doesn't seem to be applicable.
For files, the --apply and --eval functions could operate on the output of those files. Only the proposed --with would not be compatible with file installables due to argument ordering.
For expressions, the feature doesn't make a lot of sense, because you can already put everything you need into the expression.

With other commands operating on installables

nix shell and nix build are the obvious cases we care about. nix profile install is a non-obvious case, but should work as well. nix copy should also work. nix edit might work, but I doubt it will ever be used. nix eval makes sense in some cases for debugging and should. nix run seems nonsensical, but maybe there's a good use-case?

Man that got long 😅 let me know what you think!

Answer 28 · 2023-10-26T18:29:58.000Z

Thanks for the writeup @iFreilicht , LGTM . Personally I think --apply would be more meaningful than --expr, which has already an use. And as you said, perhaps having a different flag with the behaviour of the --expr examples, could be useful.

nix profile install is a non-obvious case, but should work as well

I would be concerned about the manifest.json, which follows a schema that might need to be updated. The schema is used to properly update the packages by referencing the same flakeref used to install it, so any --apply should be stored too. Personally I can live without nix profile --apply for the time being :)

$ nix profile install nixpkgs#python3 --profile ./test_profile

$ jq . ./test_profile/manifest.json
{
  "elements": [
    {
      "active": true,
      "attrPath": "legacyPackages.aarch64-linux.python3",
      "originalUrl": "flake:nixpkgs",
      "outputs": null,
      "priority": 5,
      "storePaths": [
        "/nix/store/idpjrasbr9n8kij11s5mphrw770sf13s-python3-3.10.12"
      ],
      "url": "path:/nix/store/xjviahzwa7x51vl51kc3c1k1n1jmhpd5-source?lastModified=1697059129&narHash=sha256-9NJcFF9CEYPvHJ5ckE8kvINvI84SZZ87PvqMbH6pro0%3D&rev=5e4c2ada4fcd54b99d56d7bd62f384511a7e2593"
    }
  ],
  "version": 2
}

Answer 29 · 2023-10-26T20:03:10.000Z

I would be concerned about the manifest.json, which follows a schema that might need to be updated.

The manifest format already supports store paths with barely any additional info. For example, I compiled nix locally and installed it with nix profile install ./result:

   {
      "active": true,
      "priority": 4,
      "storePaths": [
        "/nix/store/vfzyrg15gf2gvzi76ijsgjd004gcrncv-nix-2.19.0pre20231026_79e0c5c"
      ]
    }

So while an element like this can't be upgraded, it can absolutely be installed without any changes to the manifest format. And as all the installables produced by either --apply or --with would just be store paths, they can be treated as such in every context where installables are valid.

I don't want to advocate for "extensive support" for this kind of thing though, I'd much rather people write a flake with everything they want to have in their profile and install that.

Answer 30 · 2023-10-27T05:59:53.000Z

Thanks for the write-up and the offer to implement @iFreilicht !

I would weight in favour of the simplest --apply flavour (although --expr could also make sense, and could actually make sense for multiple installable if they are just all added to the scope). I've added that to the @nixos/nix-team backlog, hopefully it can be discussed today or on Monday, and we can get a decision by then.

Answer 31 · 2023-10-27T10:39:25.000Z

According to the docs of nix eval --apply, it applies the function expression to each installable, except the second installable is error: unexpected argument.

This isn't quite as general as I think it should be. What if each installable creates a positional argument? That way you can really use anything as an input.

nix eval --apply "hello: runCommand: runCommand "hi" { nativeBuildInputs = [ hello ]; } ''hello >$out''" nixpkgs#hello nixpkgs#runCommand

Not sure whether getting the default package is a good idea, when a fragment is missing. I guess nixpkgs#. could represent the whole Nixpkgs flake, but I would find nixpkgs to be simpler, and I'd be happy to write nixpkgs#default for a hypothetical default package.

Answer 32 · 2023-10-27T11:05:14.000Z

Hmm, looking at the nix profile example and it being unable to upgrade... Could we unify this more so that things will work everywhere? For instance, have the expression as part of the url. That way it is easier to store and will work for any cmdline tool. Not sure if it'll lead to the perfect UX, but just to explore that space a bit:

The same has been done for explicitly stating outputs: nix shell nixpkgs#openssl^bin which is basically openssl.bin in nix repl. This already is kind of confusing, as nix shell nixpkgs#openssl.bin is equal (different discussion I guess).

If the suffix of # would be any Nix expression inside the context of the flake, then you would be able to:

nix shell "nixpkgs#openssl.override { withZlib = false; }"

Same for python:

nix shell "nixpkgs#python3.withPackages (ps: [ ps.pandas ])"

It should then also work for packages that do not use . notation:

nix build 'nixpkgs#writeShellScript "fancyapp" "echo hello"'

Because it is part of the URL, it'll work for build, shell, profile. For nix shell it'll support multiple expressions.

All of these will work using nix profile upgrade as well.

Edit: thinking about this some more, it even makes sense to drop the 'multiple output'-syntax for URLs (nixpkgs#openssl^bin,lib) and replace it with an array (nixpkgs#with openssl; [ bin lib ]). It works just like using nix repl after:

:lf nixpkgs
with legacyPackages.${system}; 
```.

Answer 33 · 2023-10-27T12:51:18.000Z

What if each installable creates a positional argument? That way you can really use anything as an input.

@roberth Yes, I discussed this solution under the "Split options" heading.

Not sure whether getting the default package is a good idea, when a fragment is missing.

I would like this to not have as little special behavior as possible. I'll have to look at a few examples to see potential drawbacks.

If the suffix of # would be any Nix expression inside the context of the flake, then you would be able to:
nix shell "nixpkgs#openssl.override { withZlib = false; }"

@bobvanderlinden Very interesting idea as well! I guess in terms of parsing the flake name this isn't an issue, but about the output I'm not sure. It should be doable.

Because it is part of the URL, it'll work for build, shell, profile. For nix shell it'll support multiple expressions.

That's true and makes this a very elegant solution, though it does make parsing a flake URL much more difficult and in some cases potentially impossible. I'm thinking especially about the work in #8678. Also, as I understand, this would not solve the multiple-inputs usecase. So --apply would still be needed when you want to combine nixpkgs with an external overlay.

thinking about this some more, it even makes sense to drop the 'multiple output'-syntax for URLs (nixpkgs#openssl^bin,lib) and replace it with an array (nixpkgs#with openssl; [ bin lib ])

Hmm, I'm not so sure about that. Having the output syntax as syntactic sugar is nice. This shields new and consumer-only users from Nix syntax on the command line, something that the old CLI required for some basic use-cases IIRC. But these should be equivalent, I agree with that.

Answer 34 · 2023-10-27T12:55:13.000Z

Triaged in Nix meeting:

Most solution proposals are iffy
How about @roberth's: #5567 (comment)
Pretty much just function and argument, very general
Why is the language itself not sufficient?
- It is, but not as convenient when on the fly
Applying functions is a one way street from installables
The for each and the currying/positional argument interpretation are both valid and we have to clarify the use cases
Just sugar, but we're not clear on how to structure the underlying thing yet
- There may be some answers when clarifying the role of what's currently callFlake
Risk of getting stuck with something we don't want
Hopefully in the future something like flake-compat defines flakes, then we can have a direct mapping from CLI flags to flake-compat functionality
We discuss a lot of flakes samples, but the same questions apply to any expression in a file
Decision: Postponed until after RFC 136 is implemented

Answer 35 · 2023-10-27T13:09:34.000Z

Yeah that decision is fair. Especially as all proposals we had are non-breaking additions, stabilizing the current state is more important.

Answer 36 · 2023-10-27T15:14:46.000Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-10-27-nix-team-meeting-minutes-98/34695/1

Answer 37 · 2023-10-29T15:55:27.000Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/decision-making-2-0-1-0-1/12851/24

Answer 38 · 2023-10-30T14:41:02.000Z

That's true and makes this a very elegant solution, though it does make parsing a flake URL much more difficult and in some cases potentially impossible

I was under the impression that would be easily parsable. Both NodeJS and Python are able to parse them. However, according to RFC2396 it is indeed not a valid URI. Nix seems to follow that standard more strictly. Allowing more than RFC2396 defines URIs makes it a harder decision.

Decision: Postponed until after RFC 136 is implemented

👍

Answer 39 · 2023-10-30T17:03:09.000Z

I was under the impression that would be easily parsable. Both NodeJS and Python are able to parse them.

I should've been clearer. What I meant was that it makes it impossible to find a human-readable name for flake from the URL by just parsing. You have to actually evaluate the Nix code inside of it to get all the output paths and can then potentially create a compound name.
I guess that's not necessarily an argument against implementing this, just something to be aware of, as it's definitely a solvable problem.

Preamble

Current situation

Proposal

Possible implementations

(my current favourite) Allow flake fragments to be arbitrary expressions

Add a new flag, like --with

Change the behaviour of --expr

--apply, solution with precedent

Differences to nix eval

Flake-like outputs functions

Split options

Closer to nix repl: --expr

Collision with current --expr installables

Alternative 1: just call it --eval

Alternative 2: two argument flag --with

Interop

With other installables

With other commands operating on installables

Add a new flag, like `--with`

Change the behaviour of `--expr`

`--apply`, solution with precedent

Differences to `nix eval`

Closer to `nix repl`: `--expr`

Collision with current `--expr` installables

Alternative 1: just call it `--eval`

Alternative 2: two argument flag `--with`