node2nix
Deploy NPM Package Manager (NPM) packages with the Nix package manager!
Table of Contents
Table of Contents
- node2nix
- Table of Contents
- Installation
- Usage
- Deploying a Node.js development project
- Generating a tarball from a Node.js development project
- Deploying a development environment of a Node.js development project
- Deploying a collection of NPM packages from the NPM registry
- Generating packages for Node.js 8.x
- Generating packages for Node.js 10.x
- Generating packages for Node.js 4.x
- Advanced options
- API documentation
- License
- Acknowledgements
Prerequisites
To be able to convert Git dependencies, the presence of the nix-hash
command-line utility (that is included with the
Nix package manager) is required.
Installation
There are two ways this package can installed.
To install this package through the Nix package manager, obtain a copy of Nixpkgs and run:
$ nix-env -f '<nixpkgs>' -iA nodePackages.node2nix
Alternatively, this package can also be installed through NPM by running:
$ npm install -g node2nix
Usage
node2nix
can be used for a variety of use cases.
Deploying a Node.js development project
The primary use case of node2nix
is to deploy a development project as a NPM
package.
What Node.js developers typically do in a development setting is opening the source code folder and running:
$ npm install
The above command-line instruction deploys all dependencies declared in the
package.json
configuration
file so that the application can be run.
With node2nix
you can use the Nix package manager for exactly the same purpose.
Running the following command generates a collection of Nix expressions from
package.json
:
$ node2nix
The above command generates three files node-packages.nix
containing Nix
expressions for the requested packge, node-env.nix
contains the build logic
and default.nix
is a composition expression allowing users to deploy the
package.
By running the following Nix command with these expressions, the project can be built:
$ nix-build -A package
The above instruction places a result
symlink in the current working dir
pointing to the build result. An executable part of the project can be run as
follows:
$ ./result/bin/node2nix
Generating a tarball from a Node.js development project
The expressions that are generated by node2nix
(shown earlier) can also be
used to generate a tarball from the project:
$ nix-build -A tarball
The above command-line instruction produces a tarball that can is placed in the following location:
$ ls result/tarballs/node2nix-1.0.1.tgz
The above tarball can be distributed to others and installed with NPM by running:
$ npm install node2nix-1.0.1.tgz
Deploying a development environment of a Node.js development project
Besides deploying a development project, it may also be useful to only install the project's dependencies and spawning a shell session in which they can be found.
The following command-line instruction uses the earlier generated expressions to deploy all the dependencies and opens a development environment:
$ nix-shell -A shell
Within this shell session, files can be modified and run without any hassle. For example, the following command should work without any trouble:
$ node bin/node2nix.js --help
Deploying a collection of NPM packages from the NPM registry
The secondary use of node2nix
is deploying existing NPM packages from the NPM
registry.
Deployment of packages from the registry is driven by a JSON specification that looks as follows:
[
"async",
"underscore",
"slasp",
{ "mocha" : "1.21.x" },
{ "mocha" : "1.20.x" },
{ "nijs": "0.0.18" },
{ "node2nix": "git://github.com/svanderburg/node2nix.git" }
]
The above specification is basically an array of objects. For each element that
is a string, the latest
version is obtained from the registry.
To obtain a specific version of a package, an object must defined in which the keys are the name of the packages and the values the versions that must be obtained.
Any version specification that NPM supports can be used, such as version numbers, version ranges, HTTP(S) URLs, Git URLs, and GitHub identifiers.
Nix expressions can be generated from this JSON specification as follows:
$ node2nix -i node-packages.json
And by using the generated Nix expressions, we can install async
through Nix as
follows:
$ nix-env -f default.nix -iA async
For every package for which the latest version has been requested, we can directly refer to the name of the package to deploy it.
For packages for which a specific version has been specified, we must refer to it using an attribute that name that is composed of its name and version specifier.
The following command can be used to deploy the first specific version of mocha
declared in the JSON configuration:
$ nix-env -f default.nix -iA '"mocha-1.21.x"'
node2nix
can be referenced as follows:
$ nix-env -f default.nix -iA '"node2nix-git://github.com/svanderburg/node2nix.git"'
Since every NPM package resolves to a package name and version number we can also deploy any package by using an attribute consisting of its name and resolved version number.
This command deploys NiJS version 0.0.18:
$ nix-env -f default.nix -iA '"nijs-0.0.18"'
Generating packages for Node.js 8.x
Node.js 8.x includes npm 5.x that supports lock files pinpointing the exact versions used of all dependencies and transitive dependencies, and a content addressable cache.
Unfortunately, in a Nix builder environment the cache is empty and NPM does not
seem to trust dependencies that are already stored in the bundled
node_modules/
folder, because they lack the meta data that can be used for
integrity checks.
We can bypass the cache by augmenting package configuration files with these
mandatory meta data fields, by providing the --bypass-cache
parameter.
Additionally, to make the entire generation for Node.js 8.x work, you can
provide the -8
parameter:
$ node2nix -8 -i node-package.json
Some Node.js development projects may include a package-lock.json
file
pinpointing the exact versions of the dependencies and transitive dependencies.
node2nix
can use this file to generate a Nix expression from it so that Nix
uses the exact same packages:
$ node2nix -8 -l package-lock.json
Generating packages for Node.js 10.x
Node.js 10.x includes npm 6.x but its generation process is no different than 8.x.
To tell node2nix
to generate for version 10.x, use the --nodejs-10
parameter:
$ node2nix --nodejs-10 -l package-lock.json
Generating packages for Node.js 4.x
By default, node2nix
generates Nix expressions that should be used in
conjuction with Node.js 6.x, which is currently the oldest supported release.
When it is desired, it is still possible to generate expression for Node.js 4.x that does not use a flattening/deduplication algorithm.
The old non-flattening structure can be simulated by adding the --no-flatten
parameter.
Additionally, to enable all flags to make generation for Node.js 4.x work, add
the -4
parameter. For example, running the following command generates
expressions that can be used with Node.js 4.x:
$ node2nix -4 -i node-package.json
By running the following command, Nix deploys NiJS version 0.0.18 using Node.js 4.x and npm 2.x:
$ nix-env -f default.nix -iA '"nijs-0.0.18"'
Advanced options
node2nix
also has a number of advanced options.
Development mode
By default, NPM packages are deployed in production mode, meaning that the
development dependencies are not installed by default. By adding the
--development
command line option, you can also deploy the development
dependencies:
$ node2nix --development
Specifying paths
If no options are specified, node2nix
makes implicit assumptions on the
filenames of the input JSON specification and the output Nix expressions. These
filenames can be modified with command-line options:
$ node2nix --input package.json --output registry.nix --composition default.nix --node-env node-env.nix
Using alternative NPM registries
You can also use an alternative NPM registry (such as a private one), by adding
the --registry
option:
$ node2nix -i node-packages.json --registry http://private.registry.local
Adding unspecified dependencies
A few exotic NPM packages may have dependencies on native libraries that reside somewhere on the user's host system. Unfortunately, NPM's metadata does not specify them, and as a consequence, it may result in failing Nix builds due to missing dependencies.
As a solution, the generated expressions by node2nix
are made overridable. The
override mechanism can be used to manually inject additional unspecified
dependencies.
The easiest way to do this is to create a wrapper Nix expression that imports
the generated composition expression from node2nix
and injects additional
dependencies.
Consider the following package collection file (named: node-packages.json
)
that installs one NPM package named floomatic
:
[
"floomatic"
]
We can generate Nix expressions from the above specification, by running:
$ node2nix -i node-packages.json
One of floomatic's dependencies is an NPM package named native-diff-match-patch
that requires the Qt 4.x library and pkgconfig, which are native dependencies not
detected by the node2nix
generator.
With the following wrapper expression (named: override.nix
), we can inject
these dependencies ourselves:
{pkgs ? import <nixpkgs> {
inherit system;
}, system ? builtins.currentSystem}:
let
nodePackages = import ./default.nix {
inherit pkgs system;
};
in
nodePackages // {
floomatic = nodePackages.floomatic.override (oldAttrs: {
buildInputs = oldAttrs.buildInputs ++ [ pkgs.pkgconfig pkgs.qt4 ];
});
}
The expression does the following:
- We import the composition expression (
default.nix
) generated bynode2nix
. - We take the old derivation that builds the
floomatic
package, and we add the missing native dependencies as build inputs by defining an override.
With the above wrapper expression, we can correctly deploy floomatic, by running:
$ nix-build override.nix -A floomatic
Adding additional/global NPM packages to a packaging process
Sometimes it may also be required to supplement a packaging process with additional NPM packages. For example, when building certain NPM projects, some dependencies have to be installed globally.
A prominent example of such a workflow is a Grunt project. The grunt CLI is typically installed globally, whereas its plugins are installed as development dependencies.
We can automate such a workflow as follows. Consider the following
package.json
example:
{
"name": "grunt-test",
"version": "0.0.1",
"private": "true",
"devDependencies": {
"grunt": "*",
"grunt-contrib-jshint": "*",
"grunt-contrib-watch": "*"
}
}
The above configuration declares grunt and two grunt plugins (jshint
and
watch
) as development dependencies.
We can create a supplemental package specification that represents additional NPM packages that are supposed to be installed globally:
[
"grunt-cli"
]
The above configuration (supplement.json
) states that we need the grunt-cli
as an additional package, installed globally.
Running the following command-line instruction generates the Nix expressions for the project:
$ node2nix -d -i package.json --supplement-input supplement.json
By overriding the generated expressions, we can instruct the builder to execute
grunt
after the dependencies have been deployed:
{ pkgs ? import <nixpkgs> {}
, system ? builtins.currentSystem
}:
let
nodePackages = import ./default.nix {
inherit pkgs system;
};
in
nodePackages // {
package = nodePackages.package.override {
postInstall = "grunt";
};
}
The above expression (override.nix
) defines a postInstall
hook that executes
grunt after the NPM package has been deployed.
Running the following command executes the packaging process, including the grunt post-processing step:
$ nix-build override.nix -A package
Disabling running NPM install
node2nix
tries to mimic npm's dependency resolver as closely as possible.
However, it may happen that there is a small difference and the deployment fails
a result.
A mismatch is typically caused by versions that can't be reliably resolved (e.g. due to wildcards) or errors in lifting bundled dependencies. In many cases, the package should still work despite the error.
To prevent the deployment from failing, we can disable the npm install
step,
by overriding the package:
{pkgs ? import <nixpkgs> {
inherit system;
}, system ? builtins.currentSystem}:
let
nodePackages = import ./default.nix {
inherit pkgs system;
};
in
nodePackages // {
express = nodePackages.express.override (oldAttrs: {
dontNpmInstall = true;
});
}
By overriding a package and setting the dontNpmInstall
parameter to true
, we
skip the install step (which merely serves as a check). The generated expression
is actually responsible for obtaining and extracting the dependencies.
Using private Git repositories
In some development projects, it may be desired to deploy private Git
repositories as dependencies. The fetchgit {}
function in Nixpkgs, however,
only supports public repositories.
It is also possible to instruct the generator to use the fetchgitPrivate {}
function, that adds support for private repositories that can be reached with
SSH:
$ node2nix --use-fetchgit-private
Before running the node2nix
command shown above, you probably want to set
up ssh-agent
first and use ssh-add
to add a private key to the keychain to
prevent the generator from asking for passphrases.
When deploying a project or package, you need to pass an additional parameter
that provides an SSH configuration file with a reference to an identify file.
The following SSH config file (e.g. ~/ssh_config
) suffices for me:
StrictHostKeyChecking=no
UserKnownHostsFile /dev/null
IdentityFile ~/id_rsa
When deploying a package with Nix, you must propagate the location of the SSH config file as a parameter:
$ nix-build -A package -I ssh-config-file=~/ssh_config
It is also possible to provide the location of the config file by adapting the
NIX_PATH
environment variable, as opposed to using the -I
parameter:
$ export NIX_PATH=ssh-config-file=~/ssh_config:$NIX_PATH
The above approach also makes it possible to deploy a NPM package with private dependencies as part of a NixOS, NixOps or Disnix configuration.
API documentation
This package includes API documentation, which can be generated with JSDoc.
License
The contents of this package is available under the MIT license
Acknowledgements
This package is based on ideas and principles pioneered in npm2nix.