Deploying NPM packages with the Nix package manager

I have encountered several people saying that the Nix package manager is a nice tool, but they do not want to depend on it to build software. Instead, they say that they want to keep using the build tools they are familiar with.

To clear up some confusion: Nix's purpose is not to replace any build tools, but complementing them by composing isolated environments in which these build tools are executed.

Isolated environments

Isolated environments composed by Nix have the following traits:

• All environment variables are initially cleared or set to dummy values.
• Environment variables are modified in such a way that only the declared dependencies can be found, e.g. by adding the full path of these packages (residing in separate directories) to PATH, PERL5LIB, CLASSPATH etc.
• Processes can only write to a designated temp folder and output folders in the Nix store. Write access to any other folder is restricted.
• After the build has finished, the output files in the Nix store are made read-only and their timestamps are reset to 1 UNIX-time.
• The environment can optionally be composed in a chroot environment in which no undeclared dependencies and non-package related arbitrary files on the filesystem can be accidentally accessed, no network activity is possible and other processes cannot interfere.

In these environments, you can execute many kinds of build tools, such as GNU Autotools, GNU Make, CMake, Apache Ant, SCons, Perl's MakeMaker and Python's setuptools, typically with little problems. In Nixpkgs, a collection of more than 2500 mostly free and open-source packages, we run many kinds of build tools inside isolated environments composed by Nix.

Moreover, besides running build tools, we can also do other stuff in isolated environments, such as running unit tests, or spawning virtual machine instances in which system integration tests are performed.

So what are the benefits of using such an approach as opposed to running build tools directly in an ad-hoc way? The main benefit is that package deployment (and even entire system configurations and networks of services and machines) become much more reliable and reproducible. Moreover, we can also run multiple builds safely in parallel improving the efficiency of deployment processes.

The only requirements that must be met in a software project are some simple rules so that builds do not fail because of the restrictions that these isolated environments impose. A while ago, I have written a blog post on techniques and lessons to improve software deployment that gives some more details on this. Moreover, if you follow these rules you should still be able to build your software project with your favourite build tools outside Nix.

(As a sidenote: Nix can actually also be a used as a build tool, but this application area is still experimental and not frequently used. More info on this can be found in Chapter 10 of Eelco Dolstra's PhD thesis that can be obtained from his publications page).

Dependency management

The fact that many build tools can be complimented by Nix probably sounds good, but there is one particular class of build tools that are problematic to use with Nix -- namely build tools that also do dependency management in addition to build management. For these kinds of tools, the Nix package manager conflicts, because the build tool typically insists taking over Nix's responsibilities as a dependency manager.

Moreover, Nix's facilities typically restrict such tools to consult external resources, but if we would allow them to do their own dependency management tasks (which is actually possible by hacking around Nix's deployment model), then the corresponding hash codes inside the Nix store paths (which are derived from all buildtime dependencies) are no longer guaranteed to accurately represent the same build results limiting reliable and reproducible deployment. The fact that other dependency managers use weaker nominal version specifications mainly contributes to that.

Second, regardless of what package manager is used, you can no longer rely on the package management system's dependency manager to deploy a system, but you also depend on extra tools and additional distribution channels, which is generally considered tedious by software distribution packagers and end-users.

NPM package manager

A prominent example of a tool doing both build and dependency management is the Node.js Package Manager (NPM), which is the primary means within the Node.js community to build and distribute software packages. It can be used for a variety of Node.js related deployment tasks.

The most common deployment task is probably installing the NPM package dependencies of a development project. What developers typically do is entering the project's working directory and running:

$ npm install

To install all its dependencies (which are obtained from the NPM registry, external URLs and Git repositories) in a special purpose folder named node_modules/ in the project workspace so that it can be run.

You can also globally install NPM packages from the NPM registry (such as command-line utilities), by running:

$ npm install -g nijs

The above command installs a NPM package named NiJS globally including all its dependencies. After the installation has been completed you should be able to run the following instruction on the command-line:

$ nijs-build --help

NPM related deployment tasks are driven by a specification called package.json that is included in every NPM package or the root folder of a development project. For example, NiJS' package.json file looks as follows:

{
  "name" : "nijs",
  "version" : "0.0.18",
  "description" : "An internal DSL for the Nix package manager in JavaScript",
  "repository" : {
    "type" : "git",
    "url" : "https://github.com/svanderburg/nijs.git"
  },
  "author" : "Sander van der Burg",
  "license" : "MIT",
  "bin" : {
    "nijs-build" : "./bin/nijs-build.js",
    "nijs-execute" : "./bin/nijs-execute.js"
  },
  "main" : "./lib/nijs",
  "dependencies" : {
    "optparse" : ">= 1.0.3",
    "slasp": "0.0.4"
  }
}

The above package.json file defines a package configuration object having the following properties:

• The name and version attributes define the name of the package and its corresponding version number. These two attributes are mandatory and if they are undefined, NPM deployment fails. Moreover, version numbers are required to follow the semver standard. One of semver's requirements is that the version attribute should consist of three version components.
• The description, repository, author and license attributes are simply just meta information. They are not used during the execution of deployment steps.
• The bin attribute defines which executable files it should deploy and to which CommonJS modules in the package they map.
• The main attribute refers to the module that is primary entry point to the package if it is included through require().
• The dependencies parameter specifies the dependencies that this package has on other NPM packages. This package depends on a library called optparse that must be of version 1.0.3 or higher and a library named slasp which must be exactly of version 0.0.4. More information on how NPM handles dependencies is explained in the next section.

Since the above package is a pure JavaScript package (which most NPM packages are) no build steps are needed. However, if some package do need to perform build steps, e.g. compiling CoffeeScript to JavaScript, or building bindings to native code, then a collection of scripts can be specified, which are run at various times in the lifecycle of a package, e.g. before and after the installation steps. These scripts can (for example) execute the CoffeeScript compiler, or invoke Gyp that compiles bindings to native code.

Replacing NPM's dependency management

So how can we deploy NPM packages in an isolated environment composed by Nix? In other words: how can we "complement" NPM with Nix?

To accomplish this view, we must substitute NPM's dependency manager, that conflicts with the Nix package manager, by something that does the dependency management the "Nix way" while retaining the NPM semantics and keeping its build facilities.

Luckily, we can easily do that by just running NPM inside a Nix expression and "fooling" it not to install any dependencies itself, by providing a copies of these dependencies in the right locations ourselves.

For example, to make deployment of NiJS work, we can just simply extract the tarball's contents, copy the result into the Nix store, entering the output folder, and copying its dependencies into the node_modules directory ourselves:

mkdir -p node_modules
cp -r ${optparse} node_modules
cp -r ${slasp} node_modules

(The above antiquoted expressions, such as ${optparse} refer to the result of Nix expressions that build the corresponding dependencies).

Finally, we should be able to run NPM inside a Nix expression as follows:

$ npm --registry http://www.example.com --nodedir=${nodeSources} install

When running the above command-line instruction after the copy commands, NPM notifies that all the required dependencies of NiJS are already present and simply proceeds without doing anything.

We also provide a couple of additional parameters to npm install:

• The --registry parameter prevents that, if any dependency is appears to be missing, the NPM registry is consulted, which is undesirable. We want deployment of NPM package dependencies to be Nix's responsibility and making it fail when dependency specifications are incomplete is exactly what we need to be sure that we correctly specify all required dependencies.
• The --nodedir parameter specifies where the Node.js source code can be found, which is used to build NPM packages that have bindings to native code. nodeSources is a directory containing the unpacked Node.js source code:
  
  

  nodeSources = runCommand "node-sources" {} ''
    tar --no-same-owner --no-same-permissions -xf ${nodejs.src}
    mv node-* $out
  '';
  

  
  
• When running NPM in a source code directory (as shown earlier), all development dependencies are installed as well, which is often not required. By providing the --production parameter, we can deploy the package in production mode, skipping the development dependencies.
  
  Unfortunately, there is one small problem that could occur with some packages defining a prepublish script -- NPM tries to execute this script while a development dependency might be missing causing the deployment to fail. To remedy this problem, I also provide the --ignore-scripts parameter to npm install and I only run the install scripts afterwards, through:
  
  

  $ npm run install --registry http://www.example.com --nodedir=${nodeSources}
  

Translating NPM's dependencies

The main challenge of deploying NPM packages with Nix is implementing a Nix equivalent for NPM's dependency manager.

Dependency classes

Currently, an NPM package configuration could declare the following kinds of dependencies which we somehow have to fit in Nix's deployment model:

• The dependencies attribute specifies which dependencies must be installed along with the package to run it. As we have seen earlier, simply copying the package of the right version into the node_modules folder in the Nix expression suffices.
• The devDependencies attribute specifies additional dependencies that are installed in development mode. For example, when running: npm install inside the folder of a development project, the development dependencies are installed as well. Also, simply copying them suffices to allow deployment in a Nix expression to work.
• The peerDependencies attribute might suggest another class of dependencies that are installed along with the package, because of the following sentence in the package.json specification:
  
  

  The host package is automatically installed if needed.

  
  After experimenting with a basic package configuration containing only one peer dependency, I discovered that peer dependencies are basically used as a checking mechanism to see whether no incompatible versions are accidentally installed. In a Nix expression, we don't have to do any additional work to support this and we can leave the check up to NPM that we run inside the Nix expression.
  
  UPDATE: It looks like the new NPM bundled with Node.js 0.12.0 does seem to actually install peer dependencies.
  
  
• bundledDependencies affects the publishing process of the package to the NPM registry. The bundled dependencies refer to a subset of the declared dependencies that are statically bundled along with the package when it's published to the NPM registry.
  
  When downloading and unpacking a package from the NPM registry that has bundled dependencies, then a node_modules folder exist that contains these dependencies including all their dependencies.
  
  To support bundled dependencies in Nix, we must first check whether a dependency already exists in the node_modules folder. If this is the case, we should leave as it is, instead of providing the dependency ourselves.
• optionalDependencies are also installed along with a package, but do not cause the deployment to fail if any error occurs. In Nix, optional dependencies can be supported by using the same copying trick as regular dependencies. However, accepting failures (especially non-deterministic ones), is not something the Nix deployment model supports. Therefore, I did not derive any equivalent for it.

Version specifications

There are various ways to refer to a specific version of a dependency. Currently, NPM supports the following kinds of version specifications:

• Exact version numbers (that comply with the semver standard), e.g. 1.0.1
• Version ranges complying with the semver standard, e.g. >= 1.0.3, 5.0.0 - 7.2.3
• Wildcards complying with the semver standard, e.g. any version: * or any 1.0 version: 1.0.x
• The latest keyword referring to the latest stable version and unstable keyword referring to the latest unstable version.
• HTTP/HTTPS URLs referring to a TGZ file being an NPM package, e.g. http://localhost/nijs-0.0.18.tgz.
• Git URLs referring to a Git repositories containing a NPM package, e.g. https://github.com/svanderburg/nijs.git.
• GitHub identifiers, referring to an NPM package hosted at GitHub, e.g. svanderburg/nijs
• Local paths, e.g. /home/sander/nijs

As described earlier, we can't leave fetching the dependencies up to NPM, but Nix has to do this instead. For most version specifications (the only exception being local paths) we can't simply write a function that takes a version specifier as input and fetches it:

• Packages with exact version numbers and version ranges are fetched from the NPM registry. In Nix, we have to translate these into fetchurl {} invocations, which requires an URL and an output hash value as as parameters allowing us to check the result to make builds reliable and reproducible.
  
  Luckily, we can retrieve the URL to the NPM package's TGZ file and its corresponding SHA1 hash by fetching the package's metadata from the NPM registry, by running:
  

  $ npm info nijs@0.0.18
  { name: 'nijs',
    description: 'An internal DSL for the Nix package manager in JavaScript',
    'dist-tags': { latest: '0.0.18' },
    ...
    dist: 
     { shasum: 'bfdf140350d2bb3edae6b094dbc31035d6c7bec8',
       tarball: 'http://registry.npmjs.org/nijs/-/nijs-0.0.18.tgz' },
    ...
  }
  

  
  We can translate the above metadata into the following Nix function invocation:
  
  

  fetchurl {
    name = "nijs-0.0.18.tgz";
    url = http://registry.npmjs.org/nijs/-/nijs-0.0.18.tgz;
    sha1 = "bfdf140350d2bb3edae6b094dbc31035d6c7bec8";
  }
  

  
  
• Version ranges are in principle unsupported in Nix in the sense that you cannot write a function that takes a version range specifier and simply downloads the latest version of the package that conforms to it, since it conflicts with Nix's reproducibility properties.
  
  If we would allow version ranges to be downloaded then the hash code inside a Nix store path does not necessarily refer to the same build result anymore. For example, running the same download tomorrow might give a different result, because the package has been updated.
  
  For example, the following path:
  
  

  /nix/store/j631r0ak98156v1xkx22n4fsl3zbmzi8-node-slasp-0.0.x
  

  
  Might refer to slasp version 0.0.4 today and to version 0.0.5 tomorrow, while the hash code remains identical. This is incompatible with Nix's deployment model.
  
  To still support deployment of packages having dependencies on version ranges of packages, we basically have to "snapshot" a dependency version by running:
  
  

  $ npm info nijs@0.0.x
  

  
  and create a fetchurl {} invocation from the particular version that is returned. The disadvantage of this approach is that, if we want to keep our versions up to date, we have to repeat this step every time a package has been updated.
  
  
• The same thing applies to wildcard version specifiers. However, there is another caveat -- if we encounter a wildcard version specifier, we cannot always assume that the latest conforming version can be taken, because NPM also supports shared dependencies.
  
  If a shared dependency conforms to a wildcard specifier, then the dependency is not downloaded, but the shared dependency is used instead, which may not necessarily be the latest version. Otherwise, the latest conforming version is downloaded. Shared dependencies are explained in the next section.
• Also for 'latest' and 'unstable' we must do a snapshot trick. However, we must also do something else. If NPM encounters version specifiers like these, it will always try to consult the NPM registry to check which version corresponds, which is undesirable. To prevent that we must substitute these version specifiers in the package.json file by '*'.
• For HTTP/HTTPS and Git/GitHub URLs, we must manually compose fetchurl {} and fetchgit {} function invocations, and we must compute their output hashes in advance. The nix-prefetch-url and nix-prefetch-git utilities are particularly useful for this. Moreover, we also have to substitute URLs by '*' in the package.json before we run NPM inside a Nix expression, to prevent it from consulting external resources.

Private, shared and cyclic dependencies

Like the Nix package manager, NPM has the ability to support multiple versions of packages simultaneously -- not only the NPM package we intend to deploy, but also all its dependencies (which are also NPM packages) can have their own node_modules/ folder that contain a package's private dependencies.

Isolation works for CommonJS modules, because when a module inside a package tries to include another package, e.g. through:

var slasp = require('slasp');

then first the node_modules/ folder of the package is consulted and the module is loaded from that folder if it exists. Furthermore, the CommonJS module system uses the absolute resolved full paths to the modules to make a distinction between module variants and not only their names. As a consequence, if a resolved path to a module with a same name is different, it's considered a different module by the module loader and thus does not conflict with others.

If a module cannot be found in the private node_modules/ folder, the module loading system recursively looks for node_modules/ folders in the parent directories, e.g.:

./nijs/node_modules/slasp/node_modules
./nijs/node_modules
./node_modules

This is how package sharing is accomplished in NPM.

NPM's policy regarding dependencies is basically that each package stores all its dependencies privately unless a dependency can be found in any of the parent directories that conforms to the version specification declared in the package. In such cases, the private dependency is omitted and a shared one will be used instead.

Also, because a dependency is installed only once, it's also possible to define cyclic dependencies. Although it's generally known that cyclic dependencies are a bad practice, they are actually used by some NPM packages, such as es6-iterator.

The npm help install manual page says the following about cycles:

To avoid this situation, npm flat-out refuses to install any name@version that is already present anywhere in the tree of package folder ancestors. A more correct, but more complex, solution would be to symlink the existing version into the new location. If this ever affects a real use-case, it will be investigated.

In Nix, private and shared dependencies are handled differently. In Nix, packages can be "private" because they are stored in separate folders in the Nix store which paths are made unique because they contain hash codes derived from all its build-time dependencies.

Sharing is accomplished when a package refers to the same Nix store path with the same hash code. In Nix these mechanisms are more powerful, because they are not restricted to specific component types.

Nix does not support cyclic dependencies and lacks the ability to refer to a parent if a package is a dependency of another package.

To simulate NPM's way of sharing packages (and means of breaking dependency cycles) in Nix, I ended up write our function that deploys NPM packages (named: buildNodePackage {}) roughly as follows:

{stdenv, nodejs, ...}:
{name, version, src, dependencies, ...}:
{providedDependencies}:

let
  requiredDependencies = ...;
  shimmedDependencies = ...;
in
stdenv.mkDerivation {
  name = "node-${name}-${version}";
  inherit src;
  ...
  buildInputs = [ nodejs ... ] ++ requiredDependencies;
  buildCommand = ''
    # Move extracted package into the Nix store
    mkdir -p $out/lib/node_modules/${name}
    mv * $out/lib/node_modules/${name}
    cd $out/lib/node_modules/${name}
    ...
    
    mkdir -p node_modules
    # Copy the required dependencies
    # Generate shims for the provided dependencies
    ...

    # Perform the build by running npm install
    npm --registry http://www.example.com --nodedir=${nodeSources} install
    ...

    # Remove the shims
  '';
}

The above expression defines a nested function with the following structure:

• The first (outermost) function's parameters refer to the build time dependencies used for the deployment of any NPM package, such as the Nix standard environment that contains a basic UNIX toolset (stdenv) and Node.js (nodejs).
• The second function's parameters refer to a specific NPM package's deployment parameters, such as the name of the package, the version, a reference to the source code (e.g. local path, URL or Git repository) and its dependencies.
• The third (innermost) function's parameter (providedDependencies) is used by a package to propagate the identifiers of the already provided shared dependencies to a dependency that's being included, so that they are not deployed again. This is required to simulate NPM's shared dependency mechanism and to escape infinite recursion, because of cyclic dependencies.
• From the dependencies and providedDependencies parameters, we can determine the required dependencies that we actually need to include privately to deploy the package. requiredDependencies are the dependencies minus the providedDependencies. The actual computation is quite tricky:
  
  • The dependencies parameter could be encoded as follows:
    

    {
      optparse = {
        ">= 1.0.3" = {
          version = "1.0.5";
          pkg = registry."optparse-1.0.5";
        };
      };
      ...
    }
    

    
    The dependencies parameter refers to an attribute set in which each attribute name represents a package name. Each member of this attribute set represents a dependency specification. The dependency specification refers to an attribute set that provides the latest snapshot of the corresponding version specification.
    
  • The providedDependences parameter could be encoded as follows:
    

    {
      optparse."1.0.5" = true;
      ...
    }
    

    
    The providedDependencies parameter is an attribute set composed of package names and a versions. If a package is in this attribute set then it means it has been provided by any of the parents and should not be included again.
    
  • We use the semver utility to see whether any of the provided dependencies map to any of the version specifications in dependencies. For example for optparse means that we run:
    
    

    $ semver -r '>= 1.0.3' 1.0.5
    $ echo $?
    0
    

    
    The above command exits with a zero exit status, meaning that there is a shared dependency providing it and we should not deploy optparse privately again. As a result, it's not added to the required dependencies.
    
  • The above procedure is basically encapsulated in a derivation that generates a Nix expression with the list of required dependencies that gets imported again -- a "trick" that I also used in NiJS and the Dynamic Disnix framework.
    
    The reason why we execute this procedure in a separate derivation is that, if we do the same thing in the builder environment of the NPM package, we always refer to all possible dependencies which prevents us escaping any potential infinite recursion.
• The required dependencies are copied into the private node_modules/ as follows:
  
  

  mkdir -p node_modules
  cp -r ${optparse propagatedProvidedDependencies} node_modules
  cp -r ${slasp propagatedProvidedDependencies} node_modules
  

  
  Now the innermost function parameter comes in handy -- to each dependency, we propagate the already provided dependencies, our own dependencies, and the package itself, to properly simulate NPM's way of sharing and breaking any potential cycles.
  
  As a sidenote: to ensure that dependencies are always correctly addressed, we must copy the dependencies. In older implementations, we used to create symlinks, which works fine for private dependencies, but not for shared dependencies.
  
  If a shared dependency is addressed, the module system looks relative to its own full resolved path, not to the symlink. Because the resolved path is completely different, the shared dependency cannot be found.
  
• For the packages that are not considered required dependencies, we must generate shims to allow the deployment to still succeed. While these dependencies are provided by the includers at runtime, they are not visible in the Nix builder environment at buildtime and, as a consequence, deployment will fail.
  
  Generating shims is quite easy. Simply generating a directory with a minimal package.json file only containing the name and version is enough. For example, the following suffices to fool NPM that the shared dependency optparse version 1.0.5 is actually present:
  
  

  mkdir node_modules/optparse
  cat > node_modules/optparse/package.json <<EOF
  {
    "name": "optparse",
    "version": "1.0.5"
  }
  EOF
  

  
  
• Then we run npm install to execute the NPM build steps, which should succeed if all dependencies are correctly specified.
• Finally, we must remove the generated shims, since they do not have any relevant meaning anymore.

Manually writing a Nix expression to deploy NPM packages

The earlier described function: buildNodePackage {} can be used to manually write Nix expressions to deploy NPM packages:

with import <nixpkgs> {};

let
  buildNodePackage = import ./build-node-package.nix {
    inherit (pkgs) stdenv nodejs;
  };

  registry = {
    "optparse-1.0.5" = buildNodePackage {
      ...
    };
    "slasp-0.0.4" = buildNodePackage {
      ...
    };
    "nijs-0.0.18" = buildNodePackage {
      name = "nijs";
      version = "0.0.18";
      src = ./.;
      dependencies = {
        optparse = {
          ">= 1.0.3" = {
            version = "1.0.5";
            pkg = registry."optparse-1.0.5";
          };
        };
        slasp = {
          "0.0.4" = {
            version = "0.0.4";
            pkg = registry."slasp-0.0.4";
          };
        };
      };
    };
  };
in
registry

The partial Nix expression (shown above) can be used to deploy the NiJS NPM package through Nix.

Moreover, it also provides NiJS' dependencies that are also built by the same function abstraction. By using the above expression, and the following command-line instruction:

$ nix-build -A '"nijs-0.0.18"'
/nix/store/y5r0raja6d8xlaav1mhw8jjxvx7bap85-node-nijs-0.0.18

NiJS is deployed by the Nix package manager including its dependencies.

Generating Nix packages from NPM package configurations

The buildNodePackage {} function shown earlier makes it possible to deploy NPM packages with Nix. However, its biggest drawback is that we have to manually write expressions for the package we want to deploy including all its dependencies. Moreover, since version ranges are unsupported, we must manually check for updates and update the corresponding expressions every time, which is labourious and tedious.

To solve this problem, a tool has been developed named: npm2nix that can automatically generate Nix expressions from NPM package.json specifications and collection specifications. It has several kinds of use cases.

Deploying a Node.js development project

Running the following command generates a collection of Nix expressions from a package.json file of a development project:

$ npm2nix

The above command generates three files registry.nix containing Nix expressions for all package dependencies and the packge itself, node-env.nix contains the build logic and default.nix is a composition expression allowing users to deploy the package.

By running the following Nix command with these expressions, the project can be built:

$ nix-build -A build

Generating a tarball from a Node.js development project

The earlier generated expressions can also be used to generate a tarball from the project:

$ nix-build -A tarball

The above command-line instruction (that basically runs npm pack) produces a tarball that can is placed in the following location:

$ ls result/tarballs/npm2nix-6.0.0.tgz

The above tarball can be distributed to others and installed with NPM by running:

$ npm install npm2nix-6.0.0.tgz

Deploying a development environment of a Node.js development project

The following command-line instruction uses the earlier generated expressions to deploy all the dependencies and opens a development environment:

$ nix-shell -A build

Within this shell session, files can be modified and run without any hassle. For example, the following command should work without any trouble:

$ node bin/npm2nix.js --help

Deploying a collection of NPM packages from the NPM registry

You can also deploy existing NPM packages from the NPM registry, which is driven by a JSON specification that looks as follows:

[
  "async",
  "underscore",
  "slasp",
  { "mocha" : "1.21.x" },
  { "mocha" : "1.20.x" },
  { "nijs": "0.0.18" },
  { "npm2nix": "git://github.com/NixOS/npm2nix.git" }
]

The above specification is basically an array of objects. For each element that is a string, the latest version is obtained from the NPM registry. To obtain a specific version of a package, an object must defined in which the keys are the names of the packages and the values are their version specifications. Any version specification that NPM supports can be used.

Nix expressions can be generated from this JSON specification as follows:

$ npm2nix -i node-packages.json

And using the generated Nix expressions, we can install async through Nix as follows:

$ nix-env -f default.nix -iA async

For every package for which the latest version has been requested, we can directly refer to the name of the package to deploy it.

For packages for which a specific version has been specified, we must refer to it using an attribute that name that is composed of its name and version specifier.

The following command can be used to deploy the first version of mocha declared in the JSON configuration:

$ nix-env -f default.nix -iA '"mocha-1.21.x"'

npm2nix can be referenced as follows:

$ nix-env -f default.nix \
    -iA '"npm2nix-git://github.com/NixOS/npm2nix.git"'

Since every NPM package resolves to a package name and version number we can also deploy any package by using an attribute consisting of its name and resolved version number. This command deploys NiJS version 0.0.18:

$ nix-env -f default.nix -iA '"nijs-0.0.18"'

The above command also works with dependencies of any package that are not declared in the JSON configuration file, e.g.:

$ nix-env -f default.nix -iA '"slasp-0.0.4"'

Concluding remarks

In this lengthy blog post (which was quite a project!) I have outlined some differences between NPM and Nix, sketched an approach that can be used to deploy NPM packages with Nix, and described a generator: npm2nix that automates this approach.

The reason why I wrote this stuff down is that the original npm2nix developer has relinquished his maintainership and I became co-maintainer. Since the NixOS sprint in Ljubljana I've been working on reengineering npm2nix and solving the problem with cyclic dependencies and version mismatches with shared dependencies. Because the problem is quite complicated, I think it would be good to have something documented that describes the problems and my thoughts.

As part of the reengineering process, I ported npm2nix from CoffeeScript to JavaScript, used some abstraction facilities to tidy up the pyramid code (caused by nesting of callbacks), and modularized the codebase it a bit further.

I am using NiJS for the generation of Nix expressions, and I modified it to have most Nix language concepts supported (albeit some of them can only be written in an abstract syntax). Moreover, the expressions generated by NiJS are now also pretty printed so that the generated code is still (mostly) readable.

The reengineered npm2nix can be obtained from the reengineering branch of my private GitHub fork and is currently in testing phase. Once it is considered stable enough, it will replace the old implementation.

Acknowledgements

The majority of npm2nix is not my work. Foremost, I'd like to thank Shea Levy, who is the original developer/author of npm2nix. He was maintaining it since 2012 and figured out most of NPM's internals, mappings of NPM concepts to Nix and how to use NPM specific modules (such as the NPM registry client) to obtain metadata from the NPM registry. Most of the stuff in the reengineered version is ported directly from the old implementation done by him.

Also I'd like to thank the other co-maintainers: Rok Garbas and Rob Vermaas for their useful input during the NixOS sprint in Ljubljana.

Finally, although the feedback period is open for only a short time, I've already received some very useful comments on #nixos and the Nix mailing list by various developers that I would like to thank.

Related work

NPM is not the only tool that does build and dependency management. Another famous (or perhaps notorious!) tool I found myself struggling with in the past was Apache Maven, which is quite popular in the Java world.

Furthermore, converters for other kinds of packages to Nix also exists. Other converters I am currently aware of are: cabal2nix, python2nix, go2nix, and bower2nix.