/floco

Using Nix to put NPM and Yarn in a coffin

Primary LanguageNixGNU General Public License v3.0GPL-3.0

floco

floco is a JavaScript package management and build tool powered by Nix.

floco is a bold departure from conventional JavaScript package management tooling focusing on reproducibility, distributed caching, and sandboxing.

Every package is built in a strictly declared sandbox isolated from the runtime system, much like an OCI container. This approach allows packages to be built once and reused on any system that shares the same architecture and platform. Artifacts are cached locally and may be distributed among a cluster of systems allowing development environments to be created at more than twice the speed of npm or yarn.

Underlying installers are implemented in bash and limit themselves to using coreutils, findutils, and jq improving reproducibility and readability compared to tooling implemented using a sprawling maze of JavaScript.

Despite this repository’s seemingly small form, it is the result of nearly a year of exploration, trial, and refinement. A great deal of effort was expended to make this piece of software suck less than the competition, but rest assured that these routines were not implemented naively. Writing a JavaScript package management framework in bash and nix was done not because it is easy - on the contrary it was fucking hard.

Getting Started

There’s a dedicated Getting Started guide that is the best place to dive in.

For projects that depend only on registry packages without install scripts that require native dependencies or cycle breaking, the following process will have you up and running:

set -euo pipefail;
cd ./my-project;
nix flake init -t github:aakropotkin/floco;
nix run github:aakropotkin/floco -- translate;
[[ -r ./package-lock.json~ ]] && mv ./package-lock.json{~,};
nix build -f ./. global;
ls -R ./result;

Documentation

A collection of documentation is hosted on GitHub on the project’s wiki tab, and is also available under <floco>/doc alongside many of the workspace directories used in examples.

Additionally all CLI tooling and scripts support the --help option. The floco help CMD sub-command may also be used to view “help” messages for a given CMD.

Templates

A simple template with the boilerplate needed for use with our updaters is available through nix flake init -t github:aakropotkin/floco.

More templates are on the way.

CLI

The floco CLI interface is currently under active development and is expected to change rapidly in the near future.

Common Behaviors

Across the floco CLI a few behaviors are consistent.

The following config files are included/applied if they exist:

  • /etc/floco/floco-cfg.{nix,json}
  • ${XDG_CONFIG_HOME:-$HOME/.config}/floco/floco-config.{nix,json}
  • “Local” floco-config.{nix,json} searched for between PWD and git project root, or / if PWD is not a git repository checkout.

References to floco will be pulled from the nearest flake.lock, or nix registry list, using github:aakropotkin/floco/main as a fallback.

floco translate

Generate a pdefs.nix file from a package[-lock].json or registry package.

This routine utilized npm internally to resolve packages and form node_modules trees.

Local Project Example

mkdir -p /tmp/foo;
pushd /tmp/foo;
echo '{
  "name": "@floco/phony",
  "version": "4.2.0",
  "dependencies": {
    "lodash": "^4.17.21"
  },
  "scripts": {
    "build": "touch ./built"
  }
}' > ./package.json;
nix shell github:aakropotkin/floco;
floco -- translate -pt;
nix flake init github:aakropotkin/floco -t;
floco build;  # or `nix build -f ./. global;
ls ./result/lib/node_modules/@floco/phony/;

Remote Project Example

mkdir -p /tmp/foo;
pushd /tmp/foo;
nix shell github:aakropotkin/floco;
floco -- translate -pt lodash@4.17.21;
nix flake init github:aakropotkin/floco -t registry;
echo '{ ident = "lodash"; version = "4.17.21"; }' > ./info.nix;
floco build;  # or `nix build -f ./.;'
ls ./result/lib/node_modules/lodash/;

floco list

List all declared projects by “key”, being <IDENT>/<VERSION>, such as @foo/bar/4.2.0 or baz/4.2.0.

nix run github:aakropotkin/floco -- list;
@webassemblyjs/wast-printer/1.9.0
@xtuc/ieee754/1.2.0
@xtuc/long/4.2.2
abab/2.0.6
abbrev/1.1.1
abbrev/2.0.0
abort-controller/3.0.0
accepts/1.3.8
acorn/6.4.2
acorn/7.4.1
acorn/8.8.2
acorn-globals/4.3.4
acorn-jsx/5.3.2

floco show

Print the pdef record for a given package.

nix run github:aakropotkin/floco -- show lodash@4.17.21 --json;
{
  "ident": "lodash",
  "version": "4.17.21",
  "ltype": "file",
  "fetchInfo": {
    "narHash": "sha256-amyN064Yh6psvOfLgcpktd5dRNQStUYHHoIqiI6DMek=",
    "type": "tarball",
    "url": "https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz"
  },
  "treeInfo": {}
}

Core Scripts

floco uses a small collection of bash scripts to perform install tasks and drive builds.

These scripts do not depend on Nix, and are suitable for standalone use as replacements for pacote extract ( install-module.sh ) and (npm|yarn) run ( run-script.sh ).

You can install these as standalone executables using the floco-utils installable in the top-level flake.

For example usage and more details please see Core Scripts.

Updaters/Generators

The top level flake provides an installable floco-updaters as well as app targets ( fromPlock and fromRegistry ) that can be used to generate pdefs.nix and pdefs.json files to be loaded by as configs.

These scripts will allow you to convert existing JavaScript projects to be used with floco, and update/regenerate configs as projects’ dependencies and build requirements change.

fromPlock

This generator is intended for use with local projects. It is essentially a wrapper around npm i --package-lock-only.

For example usage please see the Getting Started guide.

fromRegistry

This generator is intended for use with published registry packages that you’d like to make accessible to floco and nix.

This script behaves almost identically to fromPlock, except that it ignores devDependencies entirely, and accepts package descriptors as an argument ( as npm or yarn would ).

For example usage please see the Native Dependencies guide.

This script most useful for packaging executables and generating treeInfo information for packages that have install scripts ( such as node-gyp compilation ).

Modules

Package metadata collection, also called translation, and project composition is managed using Nixpkgs Modules similar to those used by NixOS, dream2nix, or home-manager.

These modules are organized as sets of interface.nix and implementation.nix files and are designed to be extensible.

Organization

The core of the module system revolves around a record called pdef, short for “package definition”, which organizes translated metadata, and package records which organize the build pipelines.

This separation simplifies the organization of the translation and builder APIs, but the rationale runs further. The split allows us to flatly state: build routines must never perform impure operations, and translation routines must only produce fields that can be serialized to JSON.

Serialization of translated metadata allows Nix’s flake features to drastically improve performance by leveraging eval caching to avoid re-evaluation of recipe generation on successive runs.

pdef Package Definitions

The pdef record closely mirrors the pseudo-standard schema used by most package.json files; but is much stricter about how declarations are written.

If desired, users could ditch package.json files altogether and simply write pdef records for their projects.

Translators

At time of writing only a few translators have been migrated from the alpha iteration, at-node-nix, but in the near future these will be finalized for production use.

package.json

This is our bread and butter, and serves as the default implementation for creating a pdef record.

On its own this translator would require users to explicitly declare the structure of their node_modules/ tree using the treeInfo submodule. For this reason we strongly recommend using the package-lock.json translator for projects with large dependency graphs.

Progress on Ideal Tree

The term ideal tree refers to the mapping of a node_modules/ tree from a dependency graph. This process is by far the most complex and challenging aspect of Node.js package management.

While floco currently relies on npm to generate ideal trees, this is expected to end soon.

The alpha repository at-node-nix contains a large body of routines to perform best effort treeInfo mapping, specifically handling projects which only require a single version of any package ( this property is called The Golden Rule in package management contexts ).

Additionally the semver resolution routines used to fetch closures of packument records effectively solve half of the ideal tree process, leaving only scope and follows management to be completed.

package-lock.json v2/v3

This is by far the most developed translator, and is the recommended option for large projects.

This translator will automatically fill treeInfo submodules, and triggers minimal network fetching.

yarn.lock v5

A rudimentary translator exists to collect information from yarn.lock v5 ( produced by yarn v3 ), but because these lockfiles lack ideal tree information users will need to provide treeInfo themselves.

In the future we intend to produce treeInfo from these locks using the pinned version information they contain; but this routine still needs to be authored.

Experimental Features

treeFor

A CLI frontend for the npm ideal tree routine, arborist, modified such that package-lock.json files can be emitted to STDOUT without modifying the project.

This is expected to be used in later iterations of the updaters allowing them to be run on /nix/store/ paths. The builtins.npmLock example in the section takes advantage of this.

This executable is exposed as an installable and app in the top-level flake.

Nix Plugin

A nix plugin for use with nix --plugin-files ... is available in the top level flake, along with a wrapper executable, floco-nix, which automatically loads this plugin.

In the future this plugin is expected to grow into a full executable that provides a suite of CLI commands; but for now it accepts nix arguments and sub-commands.

This plugin was developed for Nix v2.12.0, but is likely compatible with a wider range of versions.

New Builtins

Our plugin adds a few new builtins to the nix evaluator which are useful for dynamically generating package definitions.

builtins.npmShow

Wraps npm show allowing Nix to query package registries using a users existing npm config and any environment NPM_CONFIG_* variables.

While floco is already able to fetch package registry information without any external tools; this builtin is useful for accessing private package registries and inheriting authorization settings with minimal setup.

nix run github:aakropotkin/floco#floco-nix -- eval --json --expr '
builtins.attrNames ( builtins.npmShow "lodash" )
'|jq;
[
  "_cached",
  "_contentLength",
  "_hasShrinkwrap",
  "_id",
  "_nodeVersion",
  "_npmOperationalInternal",
  "_npmUser",
  "_npmVersion",
  "_rev",
  "author",
  "bugs",
  "contributors",
  "description",
  "directories",
  "dist",
  "dist-tags",
  "gitHead",
  "homepage",
  "icon",
  "keywords",
  "license",
  "main",
  "maintainers",
  "name",
  "readmeFilename",
  "repository",
  "scripts",
  "time",
  "users",
  "version",
  "versions"
]

builtins.npmResolve

Resolves package descriptors such as foo@^1.0.0 or bar@latest using npm, returning a resolved URI.

This has the same environment and configuration properties as npmShow.

NOTE: if you use ranges such as lodash@2.x you will want to use builtins.split to parse the output.

nix run github:aakropotkin/floco#floco-nix -- eval --expr '
builtins.npmResolve "lodash@latest"
';
"https://registry.npmjs.org/lodash/-/lodash-4.17.21.tgz"

builtins.npmLock

Produces a virtual package-lock.json for a given project path without modifying the project or making any writes to the filesystem.

This is an ideal alternative to the fromRegistry updater when used in combination with builtins.fetchTree and builtins.npmResolve.

In practice you can dynamically generate full dependency closures’ treeInfo records using this routine. I currently use it for this purpose out in the field; but have avoided using it in the default modules so that they are usable without plugins.

nix run github:aakropotkin/floco#floco-nix -- eval --impure  \
  --expr 'let
    url   = builtins.npmResolve "pacote@latest";
    src   = builtins.fetchTree { type = "tarball"; inherit url; };
    plock = builtins.npmLock src;
  in builtins.attrNames plock
';
[ "lockfileVersion" "name" "packages" "requires" "version" ]

builtins.semverSat

Runs node-semver to test whether a semantic version satisfies a constraint. In the future node-semver will be replaced using a native C++ port semi.

This largely exists as a stop-gap until the pure nix implementation from the alpha repository is polished and/or semi is completed.

nix run github:aakropotkin/floco#floco-nix -- eval --expr '[
  ( builtins.semverSat "^4.2.0" "4.0.0" )
  ( builtins.semverSat "^4.2.0" "4.2.0" )
  ( builtins.semverSat "^4.2.0" "4.2.1" )
  ( builtins.semverSat "^4.2.0" "4.3.0" )
]
';
[ false true true true ]

Future Extensions

Many of the following extensions have function drafts or well tested prototypes in the alpha release of floco; but are not developed enough for use in production code-bases as pieces of reliable infrastructure.

  • Improved support for package.json workspaces.
    • Currently reliance on npm and special configuration based on in depth knowledge of floco is necessary to accomplish workspace support.
    • Practically a template or example using workspaces is likely sufficient for the immediate future; but the NixOS Module system is expected to resolve issues that previously made workspaces complex to manage.
  • Expanded CLI tooling.
    • Currently users are asked to interact with nix to drive builds, tests, update metadata, etc. Ideally a simple bash script would provide familiar commands such as floco add <PKG>, floco publish, floco update, floco build, etc that npm and yarn users are already familiar with.
  • Nix plugin to read/write caches globally and into flake.lock.
    • This is the real end goal for floco. It should be possible to read/write floco metadata to flake.lock and existing nix caches.
    • There is currently a draft plugin which allows nix to adopt npm URIs to refer to packages as lodash@4.17.21 which could be expanded upon.
    • Project templates and propagation of build recipes could allow nix to abstract away the generation of flake.nix for the vast majority of projects which would be a significant UX breakthrough.
  • Semantic version parsing, and ideal tree formation.
    • Currently floco really relies on npm and its package-lock.json to construct non-trivial node_module/ metadata declarations. This reliance is a major pain point for handling projects which currently use yarn since interoperability between yarn and npm across their associated lockfiles is implemented incredibly poorly, to such a degree that you cannot trust them to behave predictably in the same source tree.
    • Semver parsing and solving SAT is implemented in the alpha repository, and has been testing on large non-trivial inputs quite successfully. Still this effort requires a few weeks of polishing to really approve for use in production.
      • For now we have provided node-semver as an installable in the top-level flake for use in scripts and our floco-nix through builtins.semverSat.
    • Construction of ideal tree from semver SAT is a project in and of itself in order to support things like optionDependencies, peerDependencies, bundledDependencies, and other oddballs which are a prerequisite for use in the general case.

Community

Matrix

Sadly IRC is dead. IRC remains dead. So like most folks these days we use Matrix Chat.

Space: #floco:matrix.org

General Room: https://matrix.to/#/!wMSeevIIjIbAOVbqHh:matrix.org?via=matrix.org ( Recommended )

Support Room: https://matrix.to/#/!tBPFHeGmZfhbuYgvcw:matrix.org?via=matrix.org

Development Room: https://matrix.to/#/!qDFpEnHkbpkhLSenko:matrix.org?via=matrix.org

Supporters

Tulip Interfaces

floco was originally developed for use by Tulip Interfaces. Without their support this project never would have been possible.