Another unique selling point of Nix is that it provides strong reproducibility guarantees. If a build succeeds on one machine, then performing the same build on another should result in a build that is (nearly) bit-identical.
Nix improves build reproducibility by complementing build processes with features, such as:
- Storing all artifacts in isolation in a so-called Nix store: /nix/store (e.g. packages, configuration files), in which every path is unique by prefixing it with an SHA256 hash code derived from all build inputs (e.g. dependencies, build scripts etc.). Isolated paths make it possible for multiple variants and versions of the same packages to safely co-exist.
- Clearing environment variables or setting them to dummy values. In combination with unique and isolated Nix store paths, search environment variables must configured in such a way that the build script can find its dependencies in the Nix store, or it will fail.
Having to specify all search environment variables may sound inconvenient, but prevents undeclared dependencies to accidentally make a build succeed -- deployment of such a package is very likely to fail on machine that misses an unknown dependency. - Running builds as an unprivileged user that does not have any rights to make modifications to the host system -- a build can only write in its designated temp folder or output paths.
- Optionally running builds in a chroot environment, so that a build cannot possibly find any undeclared host system dependencies through hard-coded absolute paths.
- Restricting network access to prevent a build from obtaining unknown dependencies that may influence the build outcome.
For many build tools, the Nixpkgs repository provides abstraction functions that allow you to easily construct a package from source code (e.g. GNU Make, GNU Autotools, Apache Ant, Perl's MakeMaker, SCons etc.).
However, certain tools are difficult to use in combination with Nix -- for example, NPM that is used to deploy Node.js projects.
NPM is both a dependency and build manager and the former aspect conflicts with Nix -- builds in Nix are typically prevented from downloading files from remote network locations, with the exception of so-called fixed-output derivations in which the output hash is known in advance.
If network connections would be allowed in regular builds, then Nix can no longer ensure that a build is reproducible (i.e. that the hash code in the Nix store path reflects the same build output derived from all inputs).
To cope with the conflicting dependency management feature of NPM, various kinds of integrations have been developed. npm2nix was the first, and several years ago I have started node2nix to provide a solution that aims for accuracy.
Basically, the build process of an NPM package in Nix boils down to performing the following steps in a Nix derivation:
# populate the node_modules/ folder npm install --offline
We must first obtain the required dependencies of a project through the Nix package manager and install them in the correct locations in the node_modules/ directory tree.
Finally, we should run NPM in offline mode forcing it not to re-obtain or re-install any dependencies, but still perform build management tasks, such as running build scripts.
From a high-level point of view, this principle may look simple, but in practice it is not:
- With earlier versions of NPM, we were forced to imitate its dependency resolution algorithm. At first sight, it looked simple, but getting it right (such as coping with circular dependencies and dependency de-duplication) is much more difficult than expected.
- NPM 5.x introduced lock files. For NPM development projects, they provide exact version specifiers of all dependencies and transitive dependencies, making it much easier to know which dependencies need to be installed.
Unfortunately, NPM also introduced an offline cache, that prevents us from simply copying packages into the node_modules/ tree. As a result, we need to make additional complex modifications to the package.json configuration files of all dependencies.
Furthermore, end user package installations do not work with lock files, requiring us to still keep our custom implementation of the dependency resolution algorithm. - NPM's behaviour with dependencies on directories on the local file system has changed. In old versions of NPM, such dependencies were copied, but in newer versions, they are symlinked. Furthermore, each directory dependency maintains its own node_modules/ directory for transitive dependencies.
Because we need to take many kinds of installation scenarios into account and work around the directory dependency challenges, the implementation of the build environment: node-env.nix in node2nix has become very complicated.
It has become so complicated that I consider it a major impediment in making any significant changes to the build environment.
In the last few weeks, I have been working on a companion tool named: placebo-npm that should simplify the installation process. Moreover, it should also fix a number of frequently reported issues.
In this blog post, I will explain how the tool works.
Lock-driven deployments
In NPM 5.x, package-lock.json files were introduced. The fact that they capture the exact versions of all dependencies and make all transitive dependencies known, makes certain aspects of an NPM deployment in a Nix build environment easier.
For lock-driven projects, we no longer have to run our own implementation of the dependency resolution algorithm to figure out what the exact versions of all dependencies and transitive dependencies are.
For example, a project with the following package.json:
{
"name": "simpleproject",
"version": "0.0.1",
"dependencies": {
"underscore": "*",
"prom2cb": "github:svanderburg/prom2cb",
"async": "https://mylocalserver/async-3.2.1.tgz"
}
}
may have the following package-lock.json file:
{
"name": "simpleproject",
"version": "0.0.1",
"lockfileVersion": 1,
"requires": true,
"dependencies": {
"async": {
"version": "https://mylocalserver/async-3.2.1.tgz",
"integrity": "sha512-XdD5lRO/87udXCMC9meWdYiR+Nq6ZjUfXidViUZGu2F1MO4T3XwZ1et0hb2++BgLfhyJwy44BGB/yx80ABx8hg=="
},
"prom2cb": {
"version": "github:svanderburg/prom2cb#fab277adce1af3bc685f06fa1e43d889362a0e34",
"from": "github:svanderburg/prom2cb"
},
"underscore": {
"version": "1.13.1",
"resolved": "https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz",
"integrity": "sha512-hzSoAVtJF+3ZtiFX0VgfFPHEDRm7Y/QPjGyNo4TVdnDTdft3tr8hEkD25a1jC+TjTuE7tkHGKkhwCgs9dgBB2g=="
}
}
}
As you may notice, the package.json file declares three dependencies:
- The first dependency is underscore that refers to the latest version in the NPM registry. In the package-lock.json file, the dependency is frozen to version 1.13.1. The resolved property provides the URL where the tarball should be obtained from. Its integrity can be verified with the given SHA512 hash.
- The second dependency: prom2cb refers to the latest revision of the main branch of the prom2cb Git repository on GitHub. In the package-lock.json file, it is pinpointed to the fab277... revision.
- The third dependency: async refers to a tarball that is downloaded from an arbitrary HTTP URL. The package-lock.json records its SHA512 integrity hash to make sure that we can only deploy with the version that we have used previously.
As explained earlier, to ensure purity, in a Nix build environment, we cannot allow NPM to obtain the required dependencies of a project. Instead, we must let Nix obtain all the dependencies.
When all dependencies have been obtained, we should populate the node_modules/ folder of the project. In the above example, it is just simply a matter of unpacking the tarballs or copying the Git clones into the node_modules/ folder of the project. No transitive dependencies need to be deployed.
For projects that do not rely on build scripts (that perform tasks, such as linting, compiling code, such as TypeScript etc.) this typically suffices to make a project work.
However, when we also need build management, we need to run the full installation process:
$ npm install --offline npm ERR! code ENOTCACHED npm ERR! request to https://registry.npmjs.org/async/-/async-3.2.1.tgz failed: cache mode is 'only-if-cached' but no cached response available. npm ERR! A complete log of this run can be found in: npm ERR! /home/sander/.npm/_logs/2021-08-29T12_56_13_978Z-debug.log
Unfortunately, NPM still tries to obtain the dependencies despite the fact that they have already been copied into the right locations into node_modules folder.
Bypassing the offline cache
To cope with the problem that manually obtained dependencies cannot be detected, my initial idea was to use the NPM offline cache in a specific way.
The offline cache claims to be content-addressable, meaning that every item can be looked up by using a hash code that represents its contents, regardless of its origins. Unfortunately, it turns out that this property cannot be fully exploited.
For example, when we obtain the underscore tarball (with the exact same contents) from a different URL:
$ npm cache add http://mylocalcache/underscore-1.13.1.tgz
and run the installation in offline mode:
$ npm install --offline npm ERR! code ENOTCACHED npm ERR! request to https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz failed: cache mode is 'only-if-cached' but no cached response available. npm ERR! A complete log of this run can be found in: npm ERR! /home/sander/.npm/_logs/2021-08-26T13_50_15_137Z-debug.log
The installation still fails, despite the fact that we already have a tarball (with the exact same SHA512 hash) in our cache.
However, downloading underscore from its original location (the NPM registry):
$ npm cache add underscore@1.13.1
makes the installation succeed.
The reason why downloading the same tarball from an arbitrary HTTP URL does not work is because NPM will only compute a SHA1 hash. Obtaining a tarball from the NPM registry causes NPM to compute a SHA512 hash. Because it was downloaded from a different source, it fails to recognize the SHA512 hash in the package-lock.json file.
We also run into similar issues when we obtain an old package from the NPM registry that only has an SHA1 hash. Importing the same file from a local file path causes NPM to compute a SHA512 hash. As a result, npm install tries to re-obtain the same tarball from the remote location, because the hash was not recognized.
To cope with these problems, placebo-npm will completely bypass the cache. After all dependencies have been copied to the node_modules folder, it modifies their package.json configuration files with hidden metadata properties to trick NPM that they came from their original locations.
For example, to make the underscore dependency work (that is normally obtained from the NPM registry), we must add the following properties to the package.json file:
{
...
_from: "underscore@https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz",
_integrity: "sha512-XdD5lRO/87udXCMC9meWdYiR+Nq6ZjUfXidViUZGu2F1MO4T3XwZ1et0hb2++BgLfhyJwy44BGB/yx80ABx8hg==",
_resolved: "https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz"
}
For prom2cb (that is a Git dependency), we should add:
{
...
_from = "github:svanderburg/prom2cb",
_integrity = "",
_resolved = "github:svanderburg/prom2cb#fab277adce1af3bc685f06fa1e43d889362a0e34"
}
and for HTTP/HTTPS dependencies and local files we should do something similar (adding _from and _integrity fields).
With these modifications, NPM will no longer attempt to consult the local cache, making the dependency installation step succeed.
Handling directory dependencies
Another challenge is dependencies on local directories, that are frequently used for local development projects:
{
"name": "simpleproject",
"version": "0.0.1",
"dependencies": {
"underscore": "*",
"prom2cb": "github:svanderburg/prom2cb",
"async": "https://mylocalserver/async-3.2.1.tgz",
"mydep": "../../mydep",
}
}
In the package.json file shown above, a new dependency has been added: mydep that refers to a relative local directory dependency: ../../mydep.
If we run npm install, then NPM creates a symlink to the folder in the project's node_modules/ folder and installs the transitive dependencies in the node_modules/ folder of the target dependency.
If we want to deploy the same project to a different machine, then it is required to put mydep in the exact same relative location, or the deployment will fail.
Deploying such an NPM project with Nix introduces a new problem -- all packages deployed by Nix are stored in the Nix store (typically /nix/store). After deploying the project, the relative path to the project (from the Nix store) will no longer be correct. Moreover, we also want Nix to automatically deploy the directory dependency as part of the deployment of the entire project.
To cope with these inconveniences, we are required to implement a tricky solution -- we must rewrite directory dependencies in such a way that can refer to a folder that is automatically deployed by Nix. Furthermore, the dependency should still end up being symlink to satisfy NPM -- copying directory dependencies in the node_modules/ folder is not accepted by NPM.
Usage
To conveniently install NPM dependencies from a local source (and satisfying npm in such a way that it believes the dependencies came from their original locations), I have created a tool called: placebo-npm.
We can, for example, obtain all required dependencies ourselves and put them in a local cache folder:
$ mkdir /home/sander/mycache $ wget https://mylocalserver/async-3.2.1.tgz $ wget https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz $ git clone https://github.com/svanderburg/prom2cb
The deployment process that placebo-npm executes is driven by a package-placebo.json configuration file that has the following structure:
{
"integrityHashToFile": {
"sha512-hzSoAVtJF+3ZtiFX0VgfFPHEDRm7Y/QPjGyNo4TVdnDTdft3tr8hEkD25a1jC+TjTuE7tkHGKkhwCgs9dgBB2g==": "/home/sander/mycache/underscore-1.13.1.tgz",
"sha512-XdD5lRO/87udXCMC9meWdYiR+Nq6ZjUfXidViUZGu2F1MO4T3XwZ1et0hb2++BgLfhyJwy44BGB/yx80ABx8hg==": "/home/sander/mycache/async-3.2.1.tgz"
},
"versionToFile": {
github:svanderburg/prom2cb#fab277adce1af3bc685f06fa1e43d889362a0e34": "/home/sander/mycache/prom2cb"
},
"versionToDirectoryCopyLink": {
"file:../dep": "/home/sander/alternatedir/dep"
}
}
The placebo config maps dependencies in a package-lock.json file to local file references:
- integrityHashToFile maps dependencies with an integrity hash to local files, which is useful for HTTP/HTTPS dependencies, registry dependencies, and local file dependencies.
- versionToFile: maps dependencies with a version property to local directories. This is useful for Git dependencies.
- versionToDirectoryCopyLink: specifies directories that need to be copied into a shadow directory named: placebo_node_dirs and creates symlinks to the shadow directories in the node_modules/ folder. This is useful for installing directory dependencies from arbitrary locations.
With the following command, we can install all required dependencies from the local cache directory and make all necessary modifications to let NPM accept the dependencies:
$ placebo-npm package-placebo.json
Finally, we can run:
$ npm install --offline
The above command does not attempt to re-obtain or re-install the dependencies, but still performs all required build management tasks.
Integration with Nix
All the functionality that placebo-npm provides has already been implemented in the node-env.nix module, but over the years it has evolved into a very complex beast -- it is implemented as a series of Nix functions that generates shell code.
As a consequence, it suffers from recursion problems and makes it extremely difficult to tweak/adjust build processes, such as modifying environment variables or injecting arbitrary build steps to work around Nix integration problems.
With placebo-npm we can reduce the Nix expression that builds projects (buildNPMProject) to an implementation that roughly has the following structure:
{stdenv, placebo-npm}:
{packagePlacebo}:
stdenv.mkDerivation ({
pname = builtins.replaceStrings [ "@" "/" ] [ "_at_" "_slash_" ] pname; # Escape characters that aren't allowed in a store path
placeboJSON = builtins.toJSON packagePlacebo;
passAsFile = [ "placeboJSON" ];
buildInputs = [ nodejs placebo-npm ] ++ buildInputs;
buildPhase = ''
runHook preBuild
true
runHook postBuild
'';
installPhase = ''
runHook preInstall
mkdir -p $out/lib/node_modules/${pname}
mv * $out/lib/node_modules/${pname}
cd $out/lib/node_modules/${pname}
placebo-npm --placebo $placeboJSONPath
npm install --offline
runHook postInstall
'';
} // extraArgs)
As may be observed, the implementation is much more compact and fits easily on one screen. The function accepts a packagePlacebo attribute set as a parameter (that gets translated into a JSON file by the Nix package manager).
Aside from some simple house keeping work, most of the complex work has been delegated to executing placebo-npm inside the build environment, before we run npm install.
The function above is also tweakable -- it is possible to inject arbitrary environment variables and adjust the build process through build hooks (e.g. preInstall and postInstall).
Another bonus feature of delegating all dependency installation functionality to the placebo-npm tool is that we can also use this tool as a build input for other kinds projects -- we can use it the construction process of systems that are built from monolithic repositories, in which NPM is invoked from the build process of the encapsulating project.
The only requirement is to run placebo-npm before npm install is invoked.
Other use cases
In addition to using placebo-npm as a companion tool for node2nix and setting up a simple local cache, it can also be useful to facilitate offline installations from external media, such as USB flash drives.
Discussion
With placebo-npm we can considerably simplify the implementation of node-env.nix (part of node2nix) making it much easier to maintain. I consider the node-env.nix module the second most complicated aspect of node2nix.
As a side effect, it has also become quite easy to provide tweakable build environments -- this should solve a large number of reported issues. Many reported issues are caused by the fact that it is difficult or sometimes impossible to make changes to a project so that it will cleanly deploy.
Moreover, placebo-npm can also be used as a build input for projects built from monolithic repositories, in which a sub set needs to be deployed by NPM.
The integration of the new node-env.nix implementation into node2nix is not completely done yet. I have reworked it, but the part that generates the package-placebo.json file and lets Nix obtain all required dependencies is still a work-in-progress.
I am experimenting with two implementations: a static approach that generates Nix expressions and dynamic implementation that directly consumes a package-lock.json file in the Nix expression language. Both approaches have pros and cons. As a result, node2nix needs to combine both of them into a hybrid approach.
In a next blog post, I will explain more about them.
Availability
The initial version of placebo-npm can be obtained from my GitHub page.