Redesign of the nix command line
Closed this issue · 74 comments
Initial writeup by @edolstra: https://gist.github.com/edolstra/efcadfd240c2d8906348
I did a little writeup with my thoughts & proposals:
https://gist.github.com/Profpatsch/d5c8e1ccd68baab0e3f0
In that I address the problem of search path clashes @edolstra mentioned in the General section.
(Commenting here as @copumpkin requested)
I strongly agree with @Profpatsch's points. To be fair there, there is a use for imperative package management, as articulated by @lethalman in #684. Since under @Profpatsch's plan the profile is always defined declarative, and since @lethalman's usecases boil down to pinning the exact version of a package, I think we should speak of "pinned" vs "regular" packages.
Start a development shell based on ./default.nix or ./shell.nix:
$ nix shell(Hm, do we want a special case like this?)
I'm reminded of git changing the default push settings. I suggest we disable this by default, and then have a setting to enable this power users can enable. In the name of orthogonality, I also propose that the setting be that the namespace is queried unless default.nix/shell.nix is present, and have it apply to at least nix build and nix use in addition to nix shell. [I don't think nix install should have this, because that command is more global, and it violates our new attribute path agenda].
More broadly, I see a convergence in the way userland nix and nixos are configured and used. We've already talked about using the nixos module system both for user-mode services (e.g. on OSX), and for configuring packages. Likewise there was recently a thread to the mailing list about decoratively pinning nixpkgs for nixos in https://www.mail-archive.com/nix-dev@lists.science.uu.nl/msg17827.html .
Ultimately It would be great if root and normal users alike can define and enable service and packages in the same module system, and the same CLI tools can assist with querying and editing the config files.
Converging all this will take a great deal of work, but I think a few decisions can be made now to help the process go faster. In the thread I linked, @edolstra mentioned we could use fetchTarball, or move the module system in nix itself (presumably into the corepkgs). We have a similar choice for the UI itself---to use the git terminology, we could have just the minimal plumbing commands in nix, and porcelain commands like nix install kept elsewhere for more rapid iteration. [This also provides a way to make nix repl work without bundling the repl with nix].
Is it possible to cut down the proposal to the essentials ? It's good to show the end-goal but for implementation it might be a bit daunting. How would we go on implementing all of this ? Are we going to port one command at the time and update the docs accordingly ?
One thing that might feel inherent but that needs to be specified is that thenix executable wouldn't be doing much, it would behaves like git and just dispatch sub-commands to nix-<command> executables in it's libexec/ and PATH.
First my strongest points:
- Very nice ideas overall.
- There are too many similar top-level commands, as noted already. I believe it's better to group them more, e.g. create more hierarchy or introduce options switching some behavior changes.
- The commands should have short-cuts, even though the long form should be preferred in documentation. Some people like to have short aliases for frequent commands. One way is adding aliases like
nix -i bash,nix -q foo, another way is allowing to use a unique prefix of a command likebtrfs fi defforbtrfs filesystem defragment. Enhanced alternative: allow defining aliases, e.g. likegit. - Commands that take attribute names should accept arbitrary nix expressions instead. This could be extended even to package updates, e.g. if I
nix install 'foo.override { myOption = bar; }', I want to keep the override even afternix upgrade foo. - Installation actions should have an interactive mode, perhaps even by default if connected to terminal: first show essentially
--dry-runand then ask for confirmation whether to proceed. - Implementation: I assume that to simplify #341 we'll only do the new C++ implementation for the new command scheme, and the old commands would require perl and we would remove them anyway after several months of overlap.
Now some replies:
- @itkovian:
nix buildis what adds stuff into the store (and a symlink into the current directory). To most people "installation" is expect to do much more than that, e.g. make$PATHcontain the executables, etc. - @madjar: I've got a set of packages that I use daily, and I want those consistent and often updated. Then I have some I rarely use and sometimes huge, so updating them often would be rather wasteful, e.g. LibreOffice.
- @CMCDragonkai: that sounds like ctags and similar tools. Perhaps such functionality belongs into the REPL, but note that the "location" is often difficult to pinpoint, e.g. the binding is done in all-packages.nix but most of the code is in another file.
- @Profpatsch: the
nixpkgs.pkgs.prefix seems redundant. Anyone can introduce extra namespaces viapackageOverridesalready. (I use it to define personalized easy to update custom package sets.) There's only a singlenixpkgs=...in$NIX_PATHand that should be the default namespace, I believe.
More detailed thoughts of mine, ordered according to the original proposal:
nix search --name foois rather confusing terminology, as we'll likely want the attribute name to be the (unqualified) namenix install--declarative: rather--auto-updateor similar. Calling it "declarative" just seems strange to me (from a fresh point of view).-f: perhaps replace by-Ito handle all$NIX_PATHchanges more consistently?machine:package: also support the other direction?
nix rebuild: wouldn't that better be somewhere innix upgrade? For example,nix upgradeby itself working only on auto-upgraded packages and with-aor--alldoing the "usual" upgrade?nix upgrade: maybe I likeupdatebetter. From some mainstream distro I've fixedupgradeas something big and dangerous... but we could e.g. make them synonymous.nix listandnix status: these two seem to work too similar to warrant extra (top-level) subcommands. Perhaps call itnix queryornix infoinstead. (Also noted by @trishume.)nix uninstall: maybeunlink? (analogy to UNIX* traditions around file deletion)nix rollback: could be alsoundo, butrollbackseems a more popular term in general. Improvement: generations might remember which one came before, as currentlynix-env --rollback; nix-env -i foo; nix-env --rollbackdoes an unintuitive thing.nix use,nix sandbox,nix shell: very similar, again. Maybenix use --sandboxor--pureor something...nix query-options: perhaps this doesn't warrant another subcommand. I'd join this with the above, e.g.nix query --options bashandnix query --plugins firefox.- Note: perhaps among this the various variants should be handled. I'd think that instead of
firefox-esrattribute we should have an option, e.g.variant="esr". Similarly for thebarFullpackages, etc. - Note: for options, especially global ones, there's a question whether to propagate them to dependencies. We might consider bringing nixos and nixpkgs closer to each other.
- Note: perhaps among this the various variants should be handled. I'd think that instead of
nix build: again the-fconsiderations.nix replandnix eval: similar, maybenix eval -ior--interactive, though "repl" might be too well remembered already.nix make-store-derivation: why notnix-build --drvinstead? It should do the same thing, only stop after instantiation.nix fetch-url: maybe havenix fetchinstead, accepting multiple schemes:https?,ftp,git, ...- add
--unpackasfetchzipequivalent.
- add
nix fetch-closure,nix send-closure: maybe not needed when we allownix build --from machine /some/store/pathand--tosimilarly.nix query-closure,nix closure-size: why notnix query --closure...?- TBD answer: it should be able find out closure from binary caches, even without fetching the paths.
nix path-info,nix substitutes: why not undernix query?nix source: perhaps add--rollbackto address some concerns.
The commands should have short-cuts
+1 for unique substrings of command names, -1 for alias definiton (that’s what the shell is for, and the git folks got it all wrong!)
Commands that take attribute names should accept arbitrary nix expressions instead
What should be in scope by default?
Installation actions should have an interactive mode
I find that absolutely crucial with a “destructive” package manager like e.g. pacman, but what use is it with nix?
Anyone can introduce extra namespaces via packageOverrides already
Yet, that is not useful if one uses two versions of e.g. nixpkgs, e.g. 15.09 and master (which is a pretty standard usecase).
@vcunat I notice that wherever you suggested a third level in the hierarchy, you did it as an --option; but couldn't it be turtles all the way down, so that sub-subcommands look and work the same as subcommands, e.g. nix query closure instead of nix query --closure?
-1 for alias definiton (that’s what the shell is for, and the git folks got it all wrong!)
@Profpatsch: how do you simply define aliases for sub-commands in bash?
What should be in scope by default?
I imagine it like this: the default $NIX_PATH points to some channel so I can use nix install bash directly, and if I want to use a different nixpkgs tree (channel), I'll have other ways to do that, among them e.g. modifying $NIX_PATH by -I options.
I find that absolutely crucial with a “destructive” package manager like e.g. pacman, but what use is it with nix?
Personally, I sometimes like to see what would happen beforehand, instead of doing it and the rolling back immediately.
Anyone can introduce extra namespaces via packageOverrides already
Yet, that is not useful if one uses two versions of e.g. nixpkgs, e.g. 15.09 and master (which is a pretty standard usecase).
Even today you should be able to do this:
{
packageOverrides = pkgs: {
pkgs-1509 = import /nix/var/nix/profiles/per-user/root/channels/nixos-15.09 {};
};
}Other ways are e.g. adding nixpkgs-15.09=... into $NIX_PATH instead and importing that. Of course, we could automatize something like this when adding channels, in some way...
(And even today you can relatively simply get rid of the prefixes; the only question is what should be the default. There was a similar discussion on the mailing-list a couple days ago.)
@glaebhoerl: for clarity reasons, I assume it's better to prefix by -- in cases where it's optional to specify a sub-command. For example, if we allowed nix query git bash to display information about multiple packages, it would seems strange to allow nix query closure bash and instead I feel it would be better to have nix query --closure bash.
@vcunat Yea I think the functionality should be in REPL or in the manual. See for example the Haskell documentation, and how if you look up the source for any particular function, it points to the actual implementation. For the REPL if it could show the type signature (even though nix is untyped, it could just be manually specified, like a help page for that exact command), that would be great.
Once some of the initial discussion settles down, I would suggest creating a new repo to host a design document. PRs would then be issued to get feedback on proposals before updating the doc.
I'm really enjoying the level of discourse here, but I think it's important that we have an authoritative document so we're all on the same page, and have a more directed approach to converging on a final, actionable design.
Ideally, we'd have @edolstra and a couple others that we could trust to review and give feedback on each proposal, merging in the changes if they sound good.
A deadline for the design might also be a good idea, to avoid this work languishing.
Thoughts?
Yes, a very good point. Best agree on basics of the UI before implementation (perhaps except for some prototypes).
how do you simply define aliases for sub-commands in bash?
alias myalias='nix subcommand'
It’s nice to see the difference between built-in and not. Otherwise you sit at a different system and wonder why the most basic commands are missing …
Even today you should be able to do this:
packageOverrides is dependent on nixpkgs, though. I don’t like the idea that most functionality of nix only works in conjunction with the default package set. In the same manner I don’t think we should hardcode the string nixpkgs anywhere in nix.
Based on my email:
Just wanted to ask, is this package search enhanced with showing package
parameters planned to be added to the standard command line Nix query
tools? Or if it's already available and I haven't found it.
I would like to suggest that any querying functionality should include the ability to show package parameters.
Another suggestion, an inline way of overriding package parameters instead of using packageOverrides inside ~/.nixpkgs/config.nix. Something like nix-env -i firefox -o '{ blah = true; }'. Where the attribute set could be a recursive attribute set evaluated inside a function that takes the old pkgs, like pkgs: rec { blah = true; }.
Oh this already has been suggested. NVM.
Something like nix-env -i firefox -o '{ blah = true; }'.
That is of course only an instance of the more general case of -E 'with import <nixpkgs> {}; firefox.override { blah = true; };'. We will have to do very close evaluation what special cases we want to introduce, since we are going to have to support them for all eternity.
I’m in strong favor of not depending on anything nixpkgs specific in the new tool.
Derivations and Nix sees them do not reflect any configuration options, and I'd like to keep it that way. I earlier proposed that this be developed in nixpkgs. But really there are 3 layers: nix itself, modules system tools and config idioms (other parts of nixpkgs/lib perhaps), and the actual nixpkgs packages. I agree nothing in the Nix repo should depend on nixpkgs at runtime, but whether we move this and module system into nix, or put tools in nixpkgs, we are combining those 3 layers into 2.
Disclaimer: just an implementation detail.
You may want to consider docopt. There's a cpp port and some examples of how you may implement a huge multi-command app like git.
Greets
Don't forget to look at guix for inspiration. I've read some of its CLI documentation, it looks very good. Annoyingly good actually... can I haz?
Here is one (out of many) really useful looking guix sub-commands:
http://www.gnu.org/software/guix/manual/guix.html#Invoking-guix-refresh
The code now contains nix command with some basic functionality (in src/nix/). I've been using it to see what it feels like. I was missing some flags, e.g. -kKj, so I looked into adding them, but I see they were moved into LegacyArgs which suggests they're not planned to be supported. @edolstra: can you confirm/explain that? I would assume at least some of those would be kept. Another point is that nix build doesn't print results or create those symlinks, but that seems just like not implemented yet.
BTW, working with the code is relatively hard for me as I can see just the code, almost without any explanation of intention etc., which makes it a little problematic to understand, especially as it's WIP. (When trying to understand header definitions, I often had to look into implementation and call sites to see what they're for.)
If we're planning a multi command app like nix build and nix shell... etc. There should be an easy to to create nix aliases like how git allows git aliases.
That was already discussed above in more detail.
When this will be implemented, will it be done in a backwards compatible manner? So that there are scripts around in my PATH which are nix-env -i and map to nix install for example, including printing a big fat "WARNING" or something that the nix UI has changed?
Another idea: Some of you might know my nixos-scripts which are basically scripts around nix-{env, shell, build} to provide some more functionality (like diffing of generations). These things will be covered in the new UI as far as I can see, but maybe you guys can get some more inspiration from my scripts on what to include (for example nix container new --template ~/container-templates/apache-container.template.nix or something like that). Just suggesting!
Edit: Another thing I really like to say that the idea for aliases is really great, but we should take this a small step further: the nix binary should be able to find nix-<foo> binaries in the $PATH and provide them as subcommands nix <foo>. This way, one could for example write a command nix ui to start a ncurses interface or nix gui to start a graphical interface for the nix command or even more tools I cannot think of yet.
nix-env will coexist with new nix command. They both use the same api while exposing a different layer to the user.
I thought nix-env's name-centric approach was mostly getting removed in the new UI?
@copumpkin I hope @domenkozar means the same C++ API internally, and the new commands simply won't use the name-centric parts.
Is there a nixos-setting for using nixUnstable so we can try this out more easily?
I don't see why not add a switch for name-based resolution.
@vcunat: I find it weird to have two different ways to name packages at install time (by attr or name). What is the use case of being able to install by name? Isn't it just slower?
EDIT: typo
I didn't mean to imply I know about some particular use cases. (And there's probably little sense to add the flag unless/until there's a nontrivial demand for it, especially as the old nix-env interface isn't going away soon.)
Is there a checklist somewhere for how much of this is already done and which things still need to be done? I might be able to contribute, but I'm not sure where to start.
I couldn't find anything like a checklist, and the current nix command implementation takes quite a while to understand if you come from outside, and getting reactions to PRs in the nix repo isn't easy either (from some of the few with push access).
boy tbh i like every one
@vcunat, @domenkozar, how far along is this, and how does the community experiment with the current state of implementation?
I haven't been watching it. To me it feels very difficult to get any nix-repo changes reviewed/merged, so I focused on nixpkgs instead. My nix PRs commonly wait for weeks for any reaction from someone with merge rights, which is just difficult on my memory and motivation, etc.
But note that the experimental nix command has been there for a long time, in unstable nix versions at least; you can try it.
@vcunat, @domenkozar As far as I understand, I can install nixUnstable for the new UI and use it and the old nix alongside eachother? Is that correct?
Yes, mostly. I do that myself. (You may run into problems when using it without the daemon, e.g. for operations requiring root privileges.)
@matthiasbeyer the nix daemon of nixUnstable will upgrade the sqlite schema of nix, so be careful.
@garbas said in his talk at NixCon 2015 that we need color and emojis.
For changing the color you can use ANSI escape codes.
http://stackoverflow.com/a/5947802/2611995
Example: printf "Nix: \033[0;31mError \033[0m25" ('Error' is red)
Emoji is also simple.
Example: printf '😀'
(i would really like to have colorful output but not sure if emoji are professional. brew uses only one and you can disable it https://evanhahn.com/disable-homebrew-emoji/)
I opened a separate (CLI-related) issue about (not) paging by default: #1261.
Another thing is that when running nix-build, if the current directory already has a result symlink, that nix store path pointed to by the symlink shouldn't be left to rot, and should be garbage collected IMO. Otherwise repeated invocations of nix-build results in a "store leak".
@CMCDragonkai Not sure I understand. Previous builds will be garbage-collected eventually. Or do you mean that they should be GC'ed immediately?
I think the point is that src should filter out result symlinks.
We do that as part of cleanSourceFilter in https://github.com/NixOS/nixpkgs/blob/master/lib/sources.nix#L24
@edolstra Yes I meant gced immediately. I didn't see any use of the existing store path.
Bump everyone.
At least something?
It is crucial for Nix to evolve CLI before it becomes widely adopted and used in business production.
- Than all going to support for 30 years what was while it became popular.
@davidak - I watched video long ago. That garbas jokes about emojis and you didn't get that. There you here: "color output, ascii art, emojis, more emojis, ponies?, more ponies! - the more - the better!"
What unprofessional means? When boss looks at utility you try to promote to roll on 1000 instances and XXXXXXL AWS databases, - and it uses emojis to explain messages.
My strictest best university teacher even tough us why and how to talk really as technical engineer.
No words 'we', 'here we have', 'you have', passive voice without any link to pearson actions.
And it does appeal. It is the most accepted, portable, informative scientific language. Nix in many places adopts diametrically opposite language.
A couple - possible, better in form of occasional small logo, maybe mascot in color ASCII art.
One smile in the end of successful deployment, when your hart otherwise stops.
And when epic fail occurred - it is better not to send any smiles to engineer, who now not going to sleep for a week troubleshooting some Nix related losses of data, in such cases those smile going to be established as a meme to avoid, with Nix altogether.
.cc .kkk, .xk'
'cc: 'xkx'.dkk;
.cc:. .dkkxkx'
;cccccccc::::;okkk. .,
.....,lll'......:kko. .cc;
.xxx' 'xxl,cc;
,llllllxkd. .c:ccc;;;;.
,lllokkko;. .:c:,,,,,,.
okkc;::. .:::.
lkk, ;::,.''''',ccc,''''.
c. .ccc:lxxkkkkkkkkkkc
.ccccc:. .xkk;
'cc: .ccc. .okk;
.cc .ccc. xk.
:ccc. lkkkkkk, .kkkd
;cccc:. lkkkkkx. .xkkkkl
,cccccc. :kkkkkk, ,kkkkkkc
.cccccc, ,kkkkkkcckkkkkk,
.:ccccc; .xkkkkkkkkkkx.
',,,,,,,:cccccc:,,,,,,,,'dkkkkkkkkd.
;ccccccccccccccccc::::::::'lkkkkkkd .c;
:cccccccccccccccccc:::::::::,ckkkkkx, .ccc:
..........:ccccc:............ ,xkkkkk: ,cccccc
lxxxxxd. .xkxxxxl ;cccccc.
okxxxxd. .dxxxxx,:ccccc:
.dkkkkko oxxd,:ccccc:
oxxxxxxxxxxxxkkkkkc :l,ccccccccccccccc:
okkkkkkkkkkkkkkkkk: ,ccccccccccccccccc;
oxxxxxxxxkkkkkkx;;, ,::ccc::::::::::::;
.dkkkkkd,;::; ;::::::.
.xkkkkko'::::::. .::::::;
'xkkkkkc ,::::::. .::::::,
xkkkkk: .:::::c' ............;;;;;;;...........
okkx' .:ccccc;:xxxxxxkkkkkkkkkkkkkkkkkkkkko
cx. ;cccccc;;xxxxxkkkkkkkkkkkkkkkkkkkkl
:cccccccc:,ccccccccokkkkkkdccccccc;
.:ccccccccccc. ckkkkkx'
'cccccc,'cccccc' :kkkkkk;
,cccccc. .cccccc, 'xkkkkkl
,cccc:. .:ccccc: .xkkkkl
:ccc. .cccccc: .kkkd Color coding - is crusial.
It is the most useful design feature.
This article was created by me: https://wiki.archlinux.org/index.php/Color_output_in_console
Emojis are a useful way to encode ample information which can be seen at a glance, without requiring the reader to read and process text (ala "a picture is worth a thousand words"). I think they should be used way more often in general, even in technical tools, and I fully agree with @garbas.
Occasional emojis is cool with me. Especially when big complex process finished.
Smiles are great to get moral of person up. And to: additionally to technical information give emotional express.
And as I love you, Nix team, I share a secret that only my parents and girlfriend knows:
Ponies - are my favorite cartoon to relax.
I want to say serious stuff. I am a serious cat and this is serious thread:
There is classic system Syslog protocol (RFC 5424).
I wrote about it here: https://wiki.archlinux.org/index.php/Systemd#Priority_level
It very important for such tools as Nix to understand why it is important to correspond to it while logging/printing messages.
On bleeding-edge people forget about industry standards because they didn't know or remember (but I think most developers at least once seen that gradation), but on Enterprise servers, where still System V/SysV init system and Syslog is very strict and beautiful. systemd journal as you see also corresponds to it.
Why I talk about corresponding to Syslog protocol (RFC 5424) model?
Than - your messages is automatically sorted in logging system. So DevOps don't need to look through emojis of bagillions of log messages on ELK and do custom rules for every new error message.
He just says to report him:
journalctl -p 3..0And as you can see - they are already color coded in terminal. journalctl does that, because it corresponds to that RFC message model. Color coded by ELK stack on cluster monitoring the same thing.
Color coded stuff can scroll at really fast speed, but you catch suden color change. Smiles are hard to grasp as efficient as that.
And your night allert on:
journalctl -p 1..0Because it is a data loss happened. Financial dataloss, our worldwide service stopped to work. It is Russian slur here that can't be translated... It's a huge responsibility on shoulders. You probaly know that also.
Emojis not help me then.
As I wrote all that I really think you already know that message level protocol.
I mean.
Sysadmin had 1-3 mainframe 20-30 years ago. When that log message model was created.
It is still true and most usefull thing today. It works across all Linux distributions. And it going to be until developers remember about it and DevOps can be sure that warnings are on 4-th level, errors on 3-d.
Now DevOps is not so respected profession. But somehow we need to manage 1 000 instances, no big deal right?, without seconds of downtime and without errors in infrastructure. With managing/curating processes of development, production releases, scaling, migrations, oh, and security. Did I mentioned products on 10 languages we need to utilize and troubleshoot to make integration smooth. And all products has their perspective of how they want to do things. And you need to figure-out how to tie such products togather to have more stable system with less headache.
I tearfully ask you all, provide as strictly useful incightfull info for user of what is happening, as possible, with details that look like noone can understand them, don't reduse them to smiles. There is going to be a time - it going to play a role.
Color coded stuff can scroll at really fast speed, but you catch suden color change. Smiles are hard to grasp as efficient as that.
This is a good point you bring up. I just felt that seeing emojis as unprofessional seemed a bit short-sighted to me. But I'd agree that colour is probably faster to decipher, and once the reader has recognized the colour, log messages should probably be verbose and not reduced to emojis, as you say.
stdout stderr also logs, they logged as an example - in containers.
I try to not be curmudgeon, but whatever I want to say, - it sounds so. Sorry, probably I am.
Respect for the new CLI. Great.
I described, why all tools of corporations today use --help:
https://gist.github.com/Profpatsch/d5c8e1ccd68baab0e3f0
And I propose hack:
If on receiving command --help is get parsed - discard all options, legitimate they or not (because if person needs help, he could probably entered not full data or supplied wring keys (optinasa).
He wants help - show him help of current command argument.
command argument --option -o --option2 option.argument.is here (wait .. I forgot)
command argument --option -o --option2 option.argument.is here --help (reading `command argument --help`)
(Arrow up)
command argument --option -o --option2 option.argument.is here --help| (ALT + Backspace)
command argument --option -o --option2 option.argument.is here |
command argument --option -o --option2 option.argument.is here I am finishing commandAnd promote to use that hack.
I think some Nix commands inevitably going to be long (but it is great, because so much will be done greatly with it), and this hack can ease expirience for users that learn the tool.
Bear in mind that color coding might not help a visually impaired person all that much, the information needs some semantic information as well (read: journal log levels).
I haven't read all the previous discussion so I'm not sure if this has already been brought up but I had a quick look and it didn't seem like it. With nix-shell we can use it as a shebang interpreter (this is outlined in the man page for it):
#! /usr/bin/env nix-shell
#! nix-shell -i bash -p bash
echo hello world$ ./hello
hello world
$ But with nix shell it doesn't work:
#! /usr/bin/env nix shell
#! nix shell -i bash -p bash
echo hello world$ ./hello
/usr/bin/env: ‘nix shell’: No such file or directory
$This is because the POSIX script invocation mechanism only allows one argument to appear on the #! line after the path to the executable, and so it reads nix shell as one "nix shell" string (I have nixUnstable installed). So I was wondering if this was already considered somewhere?
Guile uses a 'meta switch' as a workaround for this. So if this hasn't been considered maybe something like that could also be implemented for nix, to make it continue to work as a shebang interpreter?
Edit: I also found https://github.com/shlevy/long-shebang.
Maybe it can be just:
nix-shell
---
#!/bin/sh
nix shell
So manually you can use nix shell seamlessly as all other commands.
And nix-shell for scripting.
Having a distinction between the two seems very reasonable. It might even be called nix-script in that case? That way there at least is no confusion between the two (with dash, without dash).
@bobvanderlinden It exists already, nix-script
@Anton-Latukha But that's sort of my point though, we can't use nix-shell if it becomes deprecated.
I'm not sure whether this is still active here, but I'd like to propose a hook interface.
The idea behind that is a use-case (of course): I run custom scripts when rebuilding my system. The relevant ones are the channel-update and switch which both have a single purpose: Build the update (channel or system) and tag my configuration with a special git tag (special as in it identifies the generation by number) so I can see in my configuration repository which generation was build from what git commit.
Having a hook executed for each action in the rebuild process would simplify that a lot.
I think about these hook points:
- Starting to build a new generation
- Rebuiling a generation failed
- Rebuiling a generation succeeded
And parameters should give the script the ability to see what gets rebuild, who did the rebuild, how it gets build (switch, boot, build, test, ...) and possibly more I do not think of right now.
Don't know about those use cases, but it would be good to have a garbage collector hook to allow NixOS to remove boot menu entries for GC'ed versions.
@vcunat @Profpatsch @edolstra Really like this new interface! Quick comment on the subject of commands that take attributes being able to work with expression though.
Given that I can tell our users (Canadian HPC center) to do this to get an environment with python3
nix run nixpkgs.python3it would be very nice if I could tell them to do this to get an environment with python3 and numpy
nix run 'nixpkgs.python3.withPackages (pkgs: [pkgs.numpy])'instead of having to drop this sort of thing on them
nix run '(let nixpkgs = import <nixpkgs> { }; in nixpkgs.python3.withPackages (pkgs: [pkgs.numpy]))'@twhitehead, you can already simplify that to the following, can't you?
nix run '(import <nixpkgs> {}).python3.withPackages (p: [p.numpy])'@Wizek Yes, that is more compact and avoids introducing let ...; in ..., which is nice.
I would still really like to provide colleagues and users with a command that looks like just a small extension of the basic one and avoids exposing any more of the nix language than absolutely required.
I know it may seem silly, but being able to do the basics (e.g., install/use python with a select set of packages) with as uncryptic command as possible is really important for adoption in our organization.
On the subject of being able to do the basics like installing python with a set of packages with as nonthreatening looking command as possible (i.e., gives the impression that I understand what is going on without having to learn a whole bunch), I've been wondering if even this could be improved upon
nix run 'nixpkgs.python3.withPackages (pkgs: [pkgs.numpy pkgs.matplotlib])'For example (not suggesting this is the best way to do this, but more throwing it out as an example to get the conversation started), as withPackages (pkgs: [ ... ]) seems to have become a fairly standard construct, what if there was some syntatic sugar like so
expr1 | expr2 ... --> expr1 (dict: with dict; [ expr2 ... ])
to enable to following sort of commands
nix run 'nixpkgs.python3.withPackages | numpy matplotlib'This could be done by having an option to pass regular arguments like --arg does for attribute arguments. So something like this (not sure what the best option name is though):
nix run nixpkgs.python.withPackages --arg1 'pkgs: [pkgs.numpy pkgs.matplotlib]'
Or, if the attribute selection is to remain special syntax, then it could just be extended to also be able to take an optional space separated list on the end that is translated like so
attr --> attr
attr val1 ... --> attr (dict: with dict; [val1 ...])
allowing the fabulously clean and intuitive looking
nix run 'nixpkgs.python3.withPackages numpy matplotlib'
Regardless of how the final syntax is, it would be great if all the language-specific packages had the same style of invocation.
@hedning @CMCDragonkai with regard to options and the same style of invocation, maybe it would be possible to add command line options that do common invocation things (thus also encourage further packages to implement those invocation styles).
For example, a --with-packages option could be added
nix run nixpkgs.python3 --with-packages 'numpy matplotlib'that, given the above, would check if nixpkgs.python3 has a withPackages attribute and then invoke it like so
nixpkgs.python3.withPackages (p: with p; [ numpy matplotlib ])Likewise, an --override option could be added
nix use nixpkgs.python3 --override x11Support=true'
that would check for an override attribute and then invoke it like so if it exists
nixpkgs.python3.override { x11Support = true; }This has the advantage of being fairly future proof as it provides a layer of abstraction between the user and the continuously evolving nixpkgs interfaces. Options can be added, deprecated, removed, or even just have their internal implementation details changed to continue providing the same functionality against new nixpkgs interfaces.
One pain I could see is that as nixpkgs evolves, nix would start having to check versions to provide the appropriate glue code. This suggests that possible the majority of the functionality should live in nixpkgs itself. For example, nixpkgs could provide a top-level attribute set that nix can use to extend the options it takes. Combined with a standard way of serializing option data, nix could then accept these options and invoke the corresponding nixpkgs expressions and let them manipulate the underlying nix expression that is ultimately used.
That is, nix would start with an expression for the specified attribute. This would be passed to the option function corresponding to the first option along with the serialized option data. This would give a new expression, which would then be passed to the option function corresponding to the second option along with its serialized option data and so on until there are no more options. Then nix uses the final expression to obtain the required derivation instead of the specified initial attribute.
I was thinking some more about this, and had another suggestion along these lines. What if individual packages could be passed command line options, so something like this
nix run nixpkgs.python3 --with numpy matplotlib --foo --bar -- nixpkgs.gcc
would be translated into something like this
nixpkgs.python3.cmdline { with = [ "numpy" "matplotlib" ]; foo = []; bar = []; }
nixpkgs.gcc
That is, if a set of options are specified after an attribute selection, they are collected into an attribute set and passed to a cmdline handling attribute of the selection attribute (or whatever name would be appropriate) if it exists, otherwise an x does not support command line options error is generated.
Thanks! -Tyson
There is an interesting trade off between providing a generic (more power, the ultimate being an arbitrary function transformation) and a specific interface (better error messages and such) for these things. It seems you want to be a specific as possible but no more specific. This allowing everything that needs to be done to be done while still giving the best possible error messages and such.
Most of my suggestions so far have been for generic machinery. Here is one for specific machinery instead. Maybe packages could optionally provide a options (or meta.options) attribute as with NixOS modules. This could specify what options (flags) could be provided, their types (i.e., boolean, string, etc.), and document them. All of which would make for a very good user experience (think help and error messages).
This might ultimately also dovetail nicely with also having a better config system. That is, one where options are declared in a modular manner, documented, and the user's config is verified against them in order to provide a top-level pkgs config attribute, rather than the current case of the top-level config attribute just be sucked in from ~/.config/nixpkgs/config.nix. Deeper operations (i.e., more fundamental modifications to nixpkgs) would then be left to be done as overlays.
In terms of the command line usage, you could imagine the NixOs options/config machinery merging and verifying nixpkgs.config from /etc/nixos/config.nix, ~/.config/nixpkgs/config.nix, and finally a config specification generated from the command line options to provide the top level config attribute to nixpkgs. A high-level package like python could then provide a packages list-of-strings option declaration and access it via the top-level config.python.packages attribute.
nix run nixpkgs.python --packages numpy scipy -- nixpkgs.gcc
Can we close this ticket? It feels to me like it's not terribly useful any more. If anyone has issues with the new UI, then it's probably better to open a new, specific ticket about that.
Did everything mentioned here get addressed? If not, maybe they can be packaged into a feature wish-list wiki?
Since nix is a bit confusing for beginners and this GitHub issue is a main source of information, here are the release notes where this tool was introduced, and one of the few places this tool is documented: https://nixos.org/nix/manual/#ssec-relnotes-2.0
It appears that nix install doesn't exist yet (and I am not personally convinced imperative package management with nix is a good idea). Is there a plan to replace nix-env eventually? Is there a GitHub issue?
@lrworth There's now the experimental nix profile install to do that. See https://nixos.wiki/wiki/Nix_command/profile
(yes, it's been a year, but I'm still seeing this issue in search results, so I thought I'd answer for future visits)
The problem is that the new commands are not stable yet. Using them requires use of unstable nix in addition to adding experimental flags in nix.conf. newcomers will be confused. This all just adds just additional complexity to using nix which is totally unnecessary.
Personally I think this issue is still open as the new command redesign is not released yet.
Plus 1 for reopening, but there is now also a guideline for more information at least:
https://github.com/NixOS/nix/blob/master/doc/manual/src/contributing/cli-guideline.md