About this document

Nix Pills Compact Edition is an on-going attempt to reformulate the excellent series of articles by Luca Bruno, known as Nix Pills, with the following differences in mind:

a more compact, outline-based format
an attempt at a more definition-based approach
..while trying to give a more complete picture

Mostly I have been outright plagiarizing Luca through liberal cut’n’paste, so this is very much a derived work, and so, as usual – all the greatness is due to Luca, all the mistakes were added by me.

The pie-in-the-sky

import the rest of Nix Pills
review the imported material, try to distill further
review the distillation, use the outline to recombine things, breaking inter-/Pill/ boundaries
review, research and add Proposed Extras:
- generic things that didn’t get into original Nix Pills
- Haskell-specific things
- Emacs-specific things

The preferred way to edit this document

..is through the excellent Org Mode: http://orgmode.org/ – a richer, but still light-weight, alternative to Markdown.

Index

Nix Pill #1: why you should give it a try
Nix Pill #2: install on your running system (incomplete)
Nix Pill #3: enter the environment
Nix Pill #4: the basics of the language
Nix Pill #5: functions and imports
Nix Pill #6: our first derivation
Nix Pill #7: a working derivation
Nix Pill #8: generic builders
Nix Pill #9: automatic runtime dependencies
Nix Pill #10: developing with nix-shell
Nix Pill #11: the garbage collector
Nix Pill #12: the inputs design pattern
Nix Pill #13: the callPackage design pattern
Nix Pill #14: the override design pattern
Nix Pill #15: nix search paths
Nix Pill #16: nixpkgs, the parameters
Nix Pill #17: nixpkgs, overriding packages
Nix Pill #18: nix store paths

Proposed extras

Generic:
- callPackageWith – the way callPackage is actually implemented
- callPackageWithScope
- build-support/replace-dependency
- build-support/autonix/
  - KDE only
    - mkPackage
      - resolveDeps
    - renameDeps
    - propagateDeps
    - userEnvDeps
    - extendDerivation
  - mostly KDE, but also: zed, tessel, airfield, nixui
    - nativeDeps
  - Haskell, KDE
    - overrideScope
  - overrideDerivation
    overrideDerivation drv f takes a derivation (i.e., the result of a call to the builtin function derivation) and returns a new derivation in which the attributes of the original are overriden according to the function f. The function f is called with the original derivation attributes.
    
    overrideDerivation allows certain “ad-hoc” customisation scenarios (e.g. in ~/.nixpkgs/config.nix). For instance, if you want to “patch” the derivation returned by a package function in Nixpkgs to build another version than what the function itself provides, you can do something like this:
```
mySed = overrideDerivation pkgs.gnused (oldAttrs: {
  name    = "sed-4.2.2-pre";
  patches = [];
  src     = fetchurl {
    url    = ftp://alpha.gnu.org/gnu/sed/sed-4.2.2-pre.tar.bz2;
    sha256 = "11nq06d131y4wmf3drm0yk502d2xc6n5qy82cg88rb9nqd2lj41k";
  };
});
            
```
    For another application, see build-support/vm, where this function is used to build arbitrary derivations inside a QEMU virtual machine.
- build-support/source-from-head-fun/
Haskell:
- mkScope
- ghcWithPackages wrapper
- overrideCabal

Nix Pill #1: why you should give it a try

Motivation: problems with other package managers

not being purely functional
multiple versions are possible, but take a lot of effort
changing installation paths means breaking implicit assumptions about the file system hierarchy
containers, as a blunt solution, have their serious drawbacks
- nearly useless without: orchestration, shared package cache
per-language sandbox-like solutions: virtualenv, cabal sandbox
- scope that is too narrow – compiler, C libraries are not taken into account
- a bunch of specific solutions with different flaws

Nix way: purely functional, universal solution at the right level of abstraction

no assumption about the global state of the system
system composed of elements (called derivations), that are explicitly derived from each other
global, hash-addressed, immutable /nix/store, storing derivation outputs – results of building packages from source, where, during build:
- all gratuitious variables are set to their mathematical equivalent of zero (or identity, if you will)
  - time/date
  - environment
  - non-essential file system contents – the build is chrooted
- all essential variables (called build inputs) count into the hash of the store elements, and are fixed forever:
  - build toolchain – compiler / linker / etc.
  - other build time dependencies – kernel headers, libc, libraries, make, automake, coreutils
  - runtime dependencies
..which means that nearly identical subsets of packages can peacefully coexist along each other, with their only difference, for example, being the compiler they were built with
- which means painless, entirely separate upgrades – at the cost of disc space, of course
(indirectly) composing PATH from elements stored in /nix/store

Nix way: immutability

Other package managers mutate global state by replacing packages in-place:
- the library namespace granularity is mostly just the soname
- new library gets used by all the current packages, linking to that soname, mostly available at a well-known, standard, shared, global location
In Nix, the namespace granularity for libraries (and other inputs, build or runtime) is its hash:
- a new version of glibc becomes available at a location based on its hash, which will be known only to the packages that will be newly built with that particular derivation as a specific build input – passed as if by a function call
Ergo, the Nix equivalent of the traditional glibc “upgrade” is recompiling everything!
- In practice, build farms provide you with the pre-built derivations, through an opt-out mechanism known as binary cache
- Security updates is a separate story, a story of compromise with the principle of purity
Another problem is that it’s hard to compose at runtime applications that don’t have in mind a pure functional model, and that can’t be adapted to:
- example: Firefox looks for the Flash plugin binary at a global (albeit per-user) path
  - which breaks the Nix model, entirely
  - the solution (known as wrapping) is to produce another Firefox derivation, that will at runtime feed unmodified Firefox the plugin path we desire, allowing us to put the Flash binary into /nix/store
Yet another question is upgrading data:
- typically done in-place by other package managers, in best tradition of mutation
- Nix makes it your own responsibility

Conclusion

Nix (manual) maximises both flexibility and reproducibility of system composition
NixOps (based on Nix, Nixpkgs and NixOS) makes cloud deployment easy, consistent and reliable, effectively deprecating all existing containment and orchestration solutions, such as Puppet, Vagrant, you name it
Weak spots are:
- dynamic composition at runtime – solutions exist, but are one-off
- massive rebuilds due to fundamental component upgrades (kernel, compilers, base libraries) – mostly made irrelevant by build farms
However, reality shows that Nix is an eminently livable environment, and progress in solving the above problems is being steadily made
Nixpkgs (search) is a completely new repository of all existing software
- fresh concept
- growing contribution
- the current state is far beyond the experimental stage

Nix Pill #2: install on your running system

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-2-install-on-your-running.html

Download

Hydra, the Nix-based CI system, hosts the builds of Nix: http://hydra.nixos.org/project/nix#tabs-releases
The Nix manual contains a chapter on installation

Installation

/nix/store and a separate user, to isolate the store and build processes:

adduser nix
mkdir -m 0755 /nix && chown nix /nix

From now on, all the operations we do on the shell are done from this nix user:

su - nix
tar -xf nix-1.9-x86_64-linux.tar.bz2
cd nix-1.9-x86_64-linux
./install

*** INCOMPLETE ***

Nix Pill #3: enter the environment

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-3-enter-environment.html

*** INCOMPLETE ***

Nix Pill #4: the basics of the language

Introduction

Nix-repl

This chapter makes a heavy use of nix-repl. To install it, issue nix-env -i nix-repl.

CAVEAT: the nix-repl syntax is slightly different than nix syntax when it comes to assigning variables.

Value types

Simple types

integer
string
path
boolean
null

Complex types

list
attribute set
function – yes, a first-class value that can be passed to and returned from functions

Operators

Basic arithmetic:

+, -, * and integer division as builtins.div:

nix-repl> 1+3
4
nix-repl> builtins.div 6 3
2

CAVEAT: / is path concatenation instead:
```
nix-repl> 2/3
/home/nix/2/3
    
```
- Nix parsed 2/3 as a relative path to the current directory.
- Paths are parsed as long as there’s a slash.
- Therefore to specify the current directory, use ./.
- In addition, Nix also parses urls.
- Not all urls or paths can be parsed this way. If a syntax error occurs, it’s still possible to fallback to plain strings.
NOTE: builtins.div is not being used in the whole of Nixpkgs repository, hence its second-class syntax status.

Boolean expressions

||, &&, !
!=, ==
less used tests: <, >, >=, <=

Other operators

http://nixos.org/nix/manual/#table-operators

Identifiers

Dash (-) is allowed in identifiers:

nix-repl> a-b
error: undefined variable `a-b' at (string):1:1
nix-repl> a - b
error: undefined variable `a' at (string):1:1

Strings

String literals: ..are enclosed by double-quotes (“), or two single-quotes (”), with \-based escaping:

nix-repl> "''foo''"
"''foo''"
nix-repl> ''"foo"''
"\"foo\""
nix-repl> "\"foo\""
"\"foo\""

String literal syntax provides means for interpolation of expressions within ${...}:

nix-repl> foo = "strval"
nix-repl> "$foo"
"$foo"
nix-repl> "${foo}"
"strval"
nix-repl> "${2+3}"
error: cannot coerce an integer to a string, at (string):1:2

NOTE: ignore the foo = “strval” assignment, it’s nix-repl-specific syntax.

Escaping ${...} within double-quoted string literals is done with the backslash. Within two single quotes, it’s done with =”=:

nix-repl> "\${foo}"
"${foo}"
nix-repl> ''test ''${foo} test''
"test ${foo} test"

Lists

List: an immutable sequence of expressions delimited by space (not comma):

nix-repl> [ 2 "foo" true (2+3) ]
[ 2 "foo" true 5 ]

Adding or removing elements from a list is only possible through production of a new list.

Attribute sets

Attribute set

a set of associations between keys and values, where:

keys can be either/identifiers/ or strings, for the cases when desired key names aren’t valid identifiers
values can be arbitrary Nix expressions

Example value:
```
nix-repl> s = { foo = "bar"; a-b = "baz"; "123" = "num"; }
nix-repl> s
{ 123 = "num"; a-b = "baz"; foo = "bar"; }
    
```
- The output from nix-repl is wrong, you can’t write { 123 = “num”; } because 123 is not an identifier.
- Semicolon (;) is required after every key-value assignment.
- For those reading Nix expressions from Nixpkgs: do not confuse attribute sets (which are values) with argument sets used in function definitions (which are argument specifiers).
Accessing elements:

nix-repl> s.a-b
"baz"
nix-repl> s."123"
"num"

Defining recursive attribute sets:
- Exhibit of the problem:

nix-repl> { a = 3; b = a+4; }
error: undefined variable `a' at (string):1:10

Problem statement – a isn’t in scope for b
Solution: INCOMPLETE: URL

nix-repl> rec { a= 3; b = a+4; }
{ a = 3; b = 7; }

If expression

nix-repl> a = 3
nix-repl> b = 4
nix-repl> if a > b then "yes" else "no"
"no"

Both then and else must be available – so the value of the expression is always defined.

Let expression

Introducing variables into scope:

nix-repl> let a = 3; b = 4; in a + b
7

..with recursion:

nix-repl> let a = 4; b = a + 5; in b
9

Variable scopes compose..:

nix-repl> let a = 3; in let b = 4; in a + b
7

..with shadowing:

nix-repl> let a = 3; in let a = 8; b = 4; in a + b
12

With expression

with allows “opening” attribute sets, binding names of its keys to their corresponding values:

nix-repl> longExpression = { a = 3; b = 4; "123" = 5; }
nix-repl> longExpression.a + longExpression.b
7
nix-repl> with longExpression; a + b
7

CAVEAT: only valid identifiers from the set keys will be included

nix-repl> let a = 10; in with longExpression; a + b + longExpression."123"
19

CAVEAT: if an identifier is bound in the outer scope and is also present in the attribute set of with, it will not be shadowed

nix-repl> let a = 10; in with longExpression; a + b
14
nix-repl> let a = 10; in with longExpression; longExpression.a + b
7

Laziness

Nix evaluates expressions only when needed. This allows easy definition of mutually referencing entities and efficient handling of large package repository definitions.

*** INCOMPLETE ***

Nix Pill #5: functions and imports

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-5-functions-and-imports.html

*** INCOMPLETE ***

Nix Pill #6: our first derivation

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-6-our-first-derivation.html

*** INCOMPLETE ***

Nix Pill #7: a working derivation

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-7-working-derivation.html

*** INCOMPLETE ***

Nix Pill #8: generic builders

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-8-generic-builders.html

*** INCOMPLETE ***

Nix Pill #9: automatic runtime dependencies

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-9-automatic-runtime.html

*** INCOMPLETE ***

Nix Pill #10: developing with nix-shell

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-10-developing-with-nix-shell.html

*** INCOMPLETE ***

Nix Pill #11: the garbage collector

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-11-garbage-collector.html

*** INCOMPLETE ***

Nix Pill #12: the inputs design pattern

Composing package definitions: repositories in Nix

We only packaged a single program so far – but how do we compose package definitions?
As we have already seen, from the point of view of a single package X, Nix is a language for describing:
- names of the externalities that are required to build (and run) package X
- how to use these externalities, given their names
- ..which looks suspiciously like function definition – which it is!
However, function definition at package level isn’t enough for whole-system description:
- ..functions need arguments supplied, which is the Nix way of saying that:
- ..packages need to have their dependencies supplied
- ..which means that particular versions and build configurations of the dependencies need to be decided upon somewhere
- Traditionally, this somewhere is called a package repository
Nix, by itself, doesn’t enforce a package repository structure, as the only inherent requirement of its functional decomposition approach is that all the functions that define packages must be supplied proper arguments.
Nix, however has a particular package repository, with a particular structure – Nixpkgs
- essentially, a single, giant expression in the Nix language:
  - mostly organized across individual, per-package files,
  - the root import-ing the nodes and leaves
  - evaluates to a giant attribute set with name -> package pairs
  - ..which works efficiently, due to the lazy evaluation property of Nix, meaning it only evaluates parts of the expression that are actually needed
- ..which contrasts with, for example Debian and Fedora, which pull package definitions from several repositories (through indexes like /etc/apt/sources.list)
- ..but coincides, for example, with Gentoo
The structure of Nixpkgs has patterns (like the above – pulling everything into a single coherent definition) that aren’t enforced by Nix, but are nonetheless present, codifying a distillation of successful practices of describing the world

Packaging graphviz

Graphviz:
- uses the standard autotools build system
- requires no patching
- dependencies are optional
- source: http://www.graphviz.org/pub/graphviz/stable/SOURCES/graphviz-2.38.0.tar.gz

Expression:

let
  pkgs = import <nixpkgs> {};
  mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
}

reuses autotools.nix from hello.nix

Build, producing runnable binaries under result/bin:

nix-build graphviz.nix

Let’s create a simple png:
```
$ echo 'graph test { a -- b }' | result/bin/dot -Tpng -o test.png
Format: "png" not recognized. Use one of: canon cmap [...]
    
```
- ..meaning that only the output formats graphviz supports natively, without using any extra library, were built.
in autotools.nix there’s a buildInputs variable, which gets concatenated to baseInputs. That would be the perfect place to add a build dependency. We created that variable exactly for this reason to be overridable from package expressions.

Digression about gcc and ld wrappers

build systems for gd, jpeg, fontconfig and bzip2 libraries (dependencies of gd) don’t use pkg-config to specify which flags to pass to the compiler, and so rely, instead, on the traditional, system-global locations, such as /usr/lib and /usr/include to find dependency headers and binaries – which are exactly absent in the Nix model.
gcc and binutils package definitions provided by Nixpkgs include wrappers, that allow passing extra arguments to gcc and ld binaries – bypassing and overriding the project build systems we call into, and effectively providing us with a project-independent way of supplying tool flags and dependencies:

NIX_CFLAGS_COMPILE
extra flags to gcc at compile time

NIX_LDFLAGS
extra flags to ld

These variables can be filled from derivation inputs the same way as was previously done for PATH – here is the relevant snippet of setup.sh:

for p in $baseInputs $buildInputs; do
  if [ -d $p/bin ]; then
    export PATH="$p/bin${PATH:+:}$PATH"
  fi
  if [ -d $p/include ]; then
    export NIX_CFLAGS_COMPILE="-I $p/include${NIX_CFLAGS_COMPILE:+ }$NIX_CFLAGS_COMPILE"
  fi
  if [ -d $p/lib ]; then
    export NIX_LDFLAGS="-rpath $p/lib -L $p/lib${NIX_LDFLAGS:+ }$NIX_LDFLAGS"
  fi
done

The -rpath flag in ld is needed because at runtime, the executable must use exactly that version of the library.
If unneeded paths are specified, the fixup phase will automatically shrink the rpath.

Completing graphviz with gd

Building upon the results above, we now can transparently supply the graphviz build system with more libraries – which it will find without any configure parameters, thanks to the gcc and ld wrappers:

# graphviz.nix
let
  pkgs = import <nixpkgs> {};
  mkDerivation = import ./autotools.nix pkgs;
in mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
  buildInputs = with pkgs; [ gd fontconfig libjpeg bzip2 ];
}

Composing package definitions: the repository expression

It’s nice to be able to abstract out the file-level repository structure, replacing the file paths with names, and that’s what Nixpkgs does – the top level expression imports the file names and provides the results as elements of the attribute set:

# default.nix:
{
  hello    = import ./hello.nix;
  graphviz = import ./graphviz.nix;
}

Trying it:

$ nix-repl
nix-repl> :l default.nix
Added 2 variables.
nix-repl> hello
«derivation /nix/store/dkib02g54fpdqgpskswgp6m7bd7mgx89-hello.drv»
nix-repl> graphviz
«derivation /nix/store/zqv520v9mk13is0w980c91z7q1vkhhil-graphviz.drv»

With nix-build:
```
$ nix-build default.nix -A hello
[...]
$ result/bin/hello
Hello, world!
    
```
- The -A argument is used to access an attribute of the set from the given .nix expression.
- When a directory (by default the current directory) has a default.nix, it will be used by default, so the following will work as well:

nix-build -A hello

Install the package in your user environment:
```
$ nix-env -f . -iA graphviz
[...]
$ dot -V
    
```
- -f is used to specify the expression to use, in this case the current directory, therefore ./default.nix.
- -i stands for installation
- -A is the same as above for nix-build
..which concludes an exhibit of the essence of nixpkgs – a collection of package definitions.

The inputs pattern

Three problems with hello.nix and graphviz.nix definitions, rooting, essentially in their dependence on Nixpkgs structure:
- They import Nixpkgs directly. In autotools.nix instead we pass Nixpkgs as an argument. That’s a much better approach.
- No way to define graphviz without libgd support
- No way to vary libgd version in graphviz definition
So far, the answer was to edit the callee
The essence of the inputs pattern is to actually use the functional abstraction, shifting these high-level decisions where they belong – to the caller

Inputs of an expression

the set of derivations needed to build that expression. In this case:

mkDerivation from autotools. Recall that mkDerivation has an implicit dependency on the toolchain.
libgd and its dependencies.

NOTE: src is also an input but it’s pointless to change the source from the caller. For version bumps, in Nixpkgs we prefer to write another expression (e.g. because patches are needed or different inputs are needed).

Leveraging functional abstraction for package expression independence

for graphviz.nix:

{ mkDerivation, gdSupport ? true, gd, fontconfig, libjpeg, bzip2 }:

mkDerivation {
  name = "graphviz";
  src = ./graphviz-2.38.0.tar.gz;
  buildInputs = if gdSupport then [ gd fontconfig libjpeg bzip2 ] else [];
}

{...}: ... is syntax for defining functions accepting an attribute set as argument. INCOMPLETE: URL
when omitted by the caller, gdSupport defaults to true

for default.nix:
```
let
  pkgs         = (import <nixpkgs>) {};
  mkDerivation = (import ./autotools.nix) pkgs;
in with pkgs; {
  hello        = (import ./hello.nix)    { inherit mkDerivation; };
  graphviz     = (import ./graphviz.nix) { inherit mkDerivation gd fontconfig libjpeg bzip2; };
  graphvizCore = (import ./graphviz.nix) { inherit mkDerivation gd fontconfig libjpeg bzip2;
                                           gdSupport = false; };
}
    
```
- let binds convenience variables
- for pedagogical purposes we cheat, by using a <nixpkgs>, which already contains everything one might want – defining contents of <nixpkgs> from ground up would have obscured the subject matter of this exhibition.
- with pkgs “opens” the Nixpkgs, binding gd, fontconfig, libjpeg and bzip2
- the toolchain is captured in the particular value of mkDerivation
- inherit a b c; is syntactic sugar for a = a; b = b; c = c; – essentially capturing a part of the variable scope in an attribute set
- import reads the .nix files, which contain functions – which are subsequently called with the attribute sets arguments – which is clarified by the added parentheses (which are otherwise unnecessary)

Conclusion

The inputs pattern is another name for leveraging functional abstraction to separate the repository in two conceptual parts:

Package definitions

import-ed leaves, containing flexible function expressions, that are free from policy decision-making, such as:

repository structure
specific versions of build inputs
other choices that the author of a particular package function expression chose to abstract out

High-level structure

the part that:

maintains knowledge of the file structure of repository, at the points of import expressions
provides a global namespace of package names, associating them to attribute sets, which are produced by:
- instantiation of the import-ed package functions
  - note that the same package function can be instantiated several times, resulting in several packages, bound to different package names
- ..while making decisions about their arguments – whose interpretation is determined by the package functions

Nix Pill #13: the callPackage design pattern

Origin: http://lethalman.blogspot.ru/2014/09/nix-pill-13-callpackage-design-pattern.html

The callPackage convenience

Problem: duplicate listing of package function arguments – explicit argument passing:
- In package function:

# package1.nix
{ input1, input2, ... }:
...

In package derivation expressions:

rec {
  lib1     = import package1.nix { inherit input1 input2 ...; };
  program2 = import package2.nix { inherit inputX inputY lib1 ...; };
}

NOTE: this “problem” wouldn’t have arised, had we not strategically chosen the package function parameter names to coincide with the derivation names in the global derivation namespace. To be clear: this is a simplification opportunity, not a problem.
So, what form do we choose for the lucky derivation expressions – those enjoying the parameter/namespace coincidence? How about this:

{
  lib1     = callPackage package1.nix { };
  program2 = callPackage package2.nix { someoverride = overriddenDerivation; };
}

The exact semantics should this implicit argument passing have? This:
- Import the given expression (which still must evaluate to a function).
- Determine the name of its arguments.
- Pass the values bound within global derivation namespace to the names of those arguments, and let us override those arguments.

Implementing callPackage

To implement implicit argument passing, we need a way to introspect at runtime the argument names of a function, to know what names it needs:
```
nix-repl> add = { a ? 3, b }: a+b
nix-repl> builtins.functionArgs add
{ a = true; b = false; }
    
```
- As a bonus, builtins.functionArgs returns not just the names, but also a boolean that determines whether the names have a default value or not.
We need access to the global derivation namespace, to know what is available

Given both of the above, the next step is to extract a subset from the global derivation namespace, determined by the chosen names:

# Mocking the /global derivation namespace/ here:
nix-repl> values = { a = 3; b = 5; c = 10; }
nix-repl> builtins.intersectAttrs values (builtins.functionArgs add)
{ a = true; b = false; }
nix-repl> builtins.intersectAttrs (builtins.functionArgs add) values
{ a = 3; b = 5; }

builtins.intersectAttrs is the function that does the extraction

This allows for a preliminary implementation of callPackage – function calling with implicit argument passing, without overrides:
```
nix-repl> callPackage = set: f: f (builtins.intersectAttrs (builtins.functionArgs f) set)
nix-repl> callPackage values add
8
nix-repl> with values; add { inherit a b; }
8
    
```
- callPackage is a function of two arguments:
  - the attribute set from which to take arguments for..
  - ..the function that is to be called
- the second expression shows the original function call expression, with explicit argument passing that we sought to avoid
What if the attribute set is missing a key/value pair for the required argument of the function being called? Nothing special – that’s an error.
The remaining piece is being able to override the attribute set, for example if the input derivation we want to supply is named with something else than the canonical derivation name – like gcc-5.2, instead of gcc.

This can be done through adding a third argument to callPackage – the override attribute set:

nix-repl> callPackage = set: f: overrides: f ((builtins.intersectAttrs (builtins.functionArgs f) set) // overrides)
nix-repl> callPackage values add { }
8
nix-repl> callPackage values add { b = 12; }
15

the // operator is an attribute set union, with the attribute set on the right taking precedence in case of key conflicts.

Use callPackage to simplify the repository

Given our brand new tool, we can simplify the repository expression (default.nix):

let
  nixpkgs     = import <nixpkgs> {};
  allPkgs     = nixpkgs // pkgs;
  callPackage = path: overrides:
    let f = import path;
    in f ((builtins.intersectAttrs (builtins.functionArgs f) allPkgs)
          // overrides);
  pkgs        = with nixpkgs; {
    mkDerivation = import ./autotools.nix nixpkgs;
    hello        = callPackage ./hello.nix { };
    graphviz     = callPackage ./graphviz.nix { };
    graphvizCore = callPackage ./graphviz.nix { gdSupport = false; };
  };
in pkgs

We renamed the old pkgs of the previous pill to nixpkgs. “Our” package attribute set is now instead named pkgs.
For convenience, in callPackage we first import the file argument, instead of calling it directly. Otherwise each derivation expression would have to perform the import itself.
Since our expressions use packages from Nixpkgs, in callPackage we use allPkgs, which is the union of Nixpkgs and our packages.
We moved mkDerivation into pkgs itself, so that it gets also passed implicitly

Note how easy is to override arguments in the case of graphviz without gd. But most importantly, how easy it was to merge two repositories: nixpkgs and our pkgs!

NOTE: ..how deeply the new scheme depends on the ability of let to define mutually recursive structure – which is only made possible by lazy evaluation semantics of Nix:

allPkgs depends on pkgs
callPackage depends on allPkgs
pkgs depends on callPackage

Conclusion

Implicit argument passing enabled by callPackage, (aka the callPackage pattern) allow us to reduce maintenance burden:
- whenever the set of package function arguments change, we’ll most likely need to change just one place – the package function itself
- whenever the derivation expression needs an override of the default inputs to the package function, it can easily do just this
builtins.functionArgs is undocumented in the Nix Manual, mostly because it’s a function that only makes sense for the sort of plumbing we’re going through.
Most of all, we get to see how Nix, the language, is a generic tool suitable for construction of arbitrary purpose-built abstractions, that can support any policy we choose.

Nix Pill #14: the override design pattern

Prelude: about composability

Functional update pattern: update functions return a modified copy of the original structure.

Common type in Nix: a -> Derivation

Example:

input derivation drv
with debug info

with patches applied

debugVersion (applyPatches [ ./patch1.patch ./patch2.patch ] drv)

The override pattern

Let us recall the repository structure:

A recursive attribute set: Name -> Derivation

..that employs introspection to supply arguments – that are its own elements – to the Derivation expressions

..it has the following form:

graphviz = callPackage ./graphviz.nix { };

If we wanted to produce a derivation of graphviz with a customized gd version, we would have to:
```
mygraphviz = callPackage ./graphviz.nix { gd = customgd; };
    
```
The problem here is a little unobvious – our override is tightly coupled to the original definition – which forces us to:
1. find the original definition
2. derive a modified version
3. maintain the modification against the possible changes of the original
So, we would like to decouple the override from the form of the definition:
```
mygraphviz = graphviz.override { gd = customgd; };
    
```
- NOTE: in this form, .override is just a plain element of an attribute set

The override implementation

As a reminder, the graphviz attribute is the derivation (which is an attribute set) returned by the derivation function import-ed from graphviz.nix.

So, to follow the form we preferred above, we would have to modify the import-ed function to return its original attribute set enriched with an element named override.

First attempt:
```
{
  # makeOverridable :: (ASet Inputs -> Drv) -> ASet Inputs
  makeOverridable = f: origArgs:
    let
      origRes = f origArgs;
    in
      origRes // { override = newArgs: f (origArgs // newArgs); };
}
    
```
- takes a derivation function
- returns a derivation function, that returns a derivation with an enriched attribute set – the added override attribute
- the override attribute too is a derivation function, but one, that calls the original derivation function with original arguments enriched by the attribute set supplied to the override function

How does it work?

$ nix-repl
nix-repl> :l lib.nix
Added 1 variables.
nix-repl> f = { a, b }: { result = a+b; }
nix-repl> f { a = 3; b = 5; }
{ result = 8; }
nix-repl> res = makeOverridable f { a = 3; b = 5; }
nix-repl> res
{ override = «lambda»; result = 8; }

so far, so good..

But, can we chain overrides?
```
nix-repl> res.override { a = 10; }
{ result = 15; }
    
```
- ..the result of an override is missing the override attribute, so no, we can’t.

Try #2:

rec {
  makeOverridable = f: origArgs:
    let
      origRes = f origArgs;
    in
      origRes // { override = newArgs: makeOverridable f (origArgs // newArgs); };
}

the rec keyword allows us to refer to makeOverridable from its own definition
overrides can now be chained

Conclusion

makeOverridable allows us to decouple expressions defining derivations, from the points where we want to override them.
This allows a degree of separation between the package definitions, and the package users.

Nix Pill #15: nix search paths

Origin: http://lethalman.blogspot.ru/2014/09/nix-pill-15-nix-search-paths.html

*** INCOMPLETE ***

Nix Pill #16: nixpkgs, the parameters

Origin: http://lethalman.blogspot.ru/2014/11/nix-pill-16-nixpkgs-parameters.html

*** INCOMPLETE ***

Nix Pill #17: nixpkgs, overriding packages

Origin: http://lethalman.blogspot.ru/2014/11/nix-pill-17-nixpkgs-overriding-packages.html

Overriding a package

In Nix Pill #14, to produce an overriden derivation by producing a modified derivation expression, in order to avoid coupling we have built-in overridability into the derivation function, by putting an override function into the attribute set returned by the derivation function under the name override.

As an example usage, here’s how we override graphviz to build without X support:

$ nix-repl
nix-repl> :l <nixpkgs>
Added 4360 variables.
nix-repl> :b graphviz.override { xlibs = null; }

However, how do we make other packages refer to this overriden graphviz, instead of the original one?
- There is no global, shared state to mutate – our only option is to re-evaluate everything in a modified context.
- Note also, how not only the new derivations need to refer to the new graphviz, but they also must refer to the new versions of each other.
- Hence the option of overriding everything manually would entail a prohibitive amount of work – basically amounting to repeating the whole repository with a new, parallel set of names.

Fixed point and beyond

Enter fix, a fix-point combinator, that:
- takes a function
- call it with its own returned value made available to it as argument
```
fix = f: let result = f result; in result;
    
```
The effective call structure is f(f(f(....
If f always used the entirety of its argument, that would have been an infinite loop.
```
nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: { x = "abc"; y = self.x + "123"; }
nix-repl> fix pkgs
{ x = "abc"; y = "abc123"; }
    
```
Squinting a little, we can see how:
1. the subexpression to the right of self is our repository declaration, where:
  - x and y correspond to our package names..
  - =”abc”=, =self.x + “123”= correspond to our derivation expressions
2. fix pkgs is the repository expression, yielding an attribute set mapping names to derivations

Note, how y is determined in terms of self – which isn’t quite pretty. Can we do without that, so we can squint less? Yes:

nix-repl> fix = f: let result = f result; in result
nix-repl> pkgs = self: with self; { x = "abc"; y = x + "123"; }
nix-repl> fix pkgs
{ x = "abc"; y = "abc123"; }

Notice how we didn’t even start with the overriding part, having merely established a correspondence between our toy example and Nixpkgs structure. So, what about overriding? Let’s note several details:
1. We want to intervene at the level of repository expression, separately from the repository declaration, since pkgs has just the perfect structure, and we don’t want to mess with it.
2. By the time we call fix, it’s too late to intervene – it seals the value.
3. Adding up #1 and #2, we see that we need to produce an function that, given an overriding set, would transform pkgs, but without changing its type – the transformed value must be still something that we can pass to fix.
4. Given our goals, we can postulate the following about this function:
  - it must take an overriding attribute set
  - it must take the object being overriden (a concrete value of pkgs)
  - it must return a function that would behave as pkgs does, which implies:
    - the returned function must take an attribute set – the value of self
    - the returned function must return an attribute set – with the overrides applied
Here’s how this can be done:
```
nix-repl> pkgs = self: with self; { x = "abc"; y = x + "123"; }
nix-repl> withOverride = overrides: f: self: f self // overrides
nix-repl> fix (withOverride { x = "def"; } pkgs)
{ x = "def"; y = "def123"; }
    
```

Now, we would like to use the package environment inside the override expression, wouldn’t we – for example to be able to override y? The only way to do that would be to change the override expression from being an attribute set to a function taking the package environment and returning the attribute set:

nix-repl> pkgs = self: with self; { x = "abc"; y = x + "123"; z = y + "456"; }
nix-repl> withOverride = overrides: f: self: f self // overrides self
nix-repl> fix (withOverride (self: with self; { y = x + "---"; }) pkgs)
{ x = "abc"; y = "abc---"; z = "abc---456"; }

The next step is to package the override method into the attribute set itself, in a way similar to how we did it in Nix Pill #14:

nix-repl> withOverride = overrides: f: self: f self // overrides self
nix-repl> virtual = let virt = f: fix f // { _override = overrides: virt (withOverride overrides f); }; in virt
nix-repl> pkgs = virtual (self: with self; { x = "abc"; y = x + "123"; z = y + "456"; })
nix-repl> pkgs
{ _override = «lambda»; x = "abc"; y = "abc123"; z = "abc123456"; }
nix-repl> pkgs._override (self: with self; { y = x + "---"; })
{ _override = «lambda»; x = "abc"; y = "abc---"; z = "abc---456"; }

Finally, we have produced a way to define an attribute set, with the following properties:

its definition is recursive – its elements can be defined in terms of each other
it ships with a method to perform a deep override of any element and, recursively, all the elements that refer to it

Overriding packages globally, the user-exposed way

Nixpkgs exposes the mechanism we examined above through the packageOverrides attribute in config / ~/.nixpkgs/config.nix:

{
  packageOverrides = pkgs: {
    graphviz = pkgs.graphviz.override { xlibs = null; };
  };
}

Overriding graphviz like this will make the change propagate to all packages that refer to it.

The `~/.nixpkgs/config.nix` file

pkgs = import <nixpkgs> { config = import ./config.nix; }

Nix Pill #18: nix store paths

Origin: http://lethalman.blogspot.ru/2015/01/nix-pill-18-nix-store-paths.html

*** INCOMPLETE ***

deepfire/nix-pills-compact-edition

About this document

The pie-in-the-sky

The preferred way to edit this document

Index

Proposed extras

Nix Pill #1: why you should give it a try

Motivation: problems with other package managers

Nix way: purely functional, universal solution at the right level of abstraction

Nix way: immutability

Conclusion

Nix Pill #2: install on your running system

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-2-install-on-your-running.html

Download

Installation

Nix Pill #3: enter the environment

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-3-enter-environment.html

Nix Pill #4: the basics of the language

Introduction

Nix-repl

Value types

Simple types

Complex types

Operators

Basic arithmetic:

Boolean expressions

Other operators

Identifiers

Strings

Lists

Attribute sets

If expression

Let expression

With expression

Laziness

Nix Pill #5: functions and imports

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-5-functions-and-imports.html

Nix Pill #6: our first derivation

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-6-our-first-derivation.html

Nix Pill #7: a working derivation

Origin: http://lethalman.blogspot.ru/2014/07/nix-pill-7-working-derivation.html

Nix Pill #8: generic builders

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-8-generic-builders.html

Nix Pill #9: automatic runtime dependencies

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-9-automatic-runtime.html

Nix Pill #10: developing with nix-shell

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-10-developing-with-nix-shell.html

Nix Pill #11: the garbage collector

Origin: http://lethalman.blogspot.ru/2014/08/nix-pill-11-garbage-collector.html

Nix Pill #12: the inputs design pattern

Composing package definitions: repositories in Nix

Packaging graphviz

Digression about gcc and ld wrappers

Completing graphviz with gd

Composing package definitions: the repository expression

The inputs pattern

Leveraging functional abstraction for package expression independence

Conclusion

Nix Pill #13: the callPackage design pattern

Origin: http://lethalman.blogspot.ru/2014/09/nix-pill-13-callpackage-design-pattern.html

The callPackage convenience

Implementing callPackage

Use callPackage to simplify the repository

Conclusion

Nix Pill #14: the override design pattern

Prelude: about composability

The override pattern

The override implementation

Conclusion

Nix Pill #15: nix search paths

Origin: http://lethalman.blogspot.ru/2014/09/nix-pill-15-nix-search-paths.html

Nix Pill #16: nixpkgs, the parameters

Origin: http://lethalman.blogspot.ru/2014/11/nix-pill-16-nixpkgs-parameters.html

Nix Pill #17: nixpkgs, overriding packages

Origin: http://lethalman.blogspot.ru/2014/11/nix-pill-17-nixpkgs-overriding-packages.html

Overriding a package

Fixed point and beyond

Overriding packages globally, the user-exposed way

The ~/.nixpkgs/config.nix file

Nix Pill #18: nix store paths

The `~/.nixpkgs/config.nix` file