Asynchronous package management with NiJS

Last week, I have implemented some additional features in NiJS: an internal DSL for Nix in JavaScript. One of its new features is an alternative formalism to write package specifications and some use cases.

Synchronous package definitions

Traditionally, a package in NiJS can be specified in JavaScript as follows:

var nijs = require('nijs');

exports.pkg = function(args) {
  return args.stdenv().mkDerivation ({
    name : "file-5.11",
    
    src : args.fetchurl()({
      url : new nijs.NixURL("ftp://ftp.astron.com/pub/file/file-5.11.tar.gz"),
      sha256 : "c70ae29a28c0585f541d5916fc3248c3e91baa481f63d7ccec53d1534cbcc9b7"
    }),
    
    buildInputs : [ args.zlib() ],
    
    meta : {
      description : "A program that shows the type of files",
      homepage : new nijs.NixURL("http://darwinsys.com/file")
    }
  });
};

The above CommonJS module exports a function which specifies a build recipe for a package named file, that uses zlib as a dependency and executes the standard GNU Autotools build procedure (i.e. ./configure; make; make install) to build it.

The above module specifies how to build a package, but not which versions or variants of the dependencies that should be used. The following CommonJS module specifies how to compose packages:

var pkgs = {

  stdenv : function() {
    return require('./pkgs/stdenv.js').pkg;
  },

  fetchurl : function() {
    return require('./pkgs/fetchurl').pkg({
      stdenv : pkgs.stdenv
    });
  },

  zlib : function() {
    return require('./pkgs/zlib.js').pkg({
      stdenv : pkgs.stdenv,
      fetchurl : pkgs.fetchurl
    });
  },
  
  file : function() {
    return require('./pkgs/file.js').pkg({
      stdenv : pkgs.stdenv,
      fetchurl : pkgs.fetchurl,
      zlib : pkgs.zlib
    });
  }
}

export.pkgs = pkgs;

As can be seen, the above module includes the previous package specification and provides all its required parameters (such as a variant of the zlib library that we need). Moreover, all its dependencies are composed in the above module as well.

Asynchronous package definitions

The previous modules are synchronous package definitions, meaning that once they are being evaluated nothing else can be done. In the latest version of NiJS, we can also write asynchronous package definitions:

var nijs = require('nijs');
var slasp = require('slasp');

exports.pkg = function(args, callback) {
  var src;
  
  slasp.sequence([
    function(callback) {
      args.fetchurl()({
        url : new nijs.NixURL("ftp://ftp.astron.com/pub/file/file-5.11.tar.gz"),
        sha256 : "c70ae29a28c0585f541d5916fc3248c3e91baa481f63d7ccec53d1534cbcc9b7"
      }, callback);
    },
    
    function(callback, _src) {
      src = _src;
      args.zlib(callback);
    },
    
    function(callback, zlib) {
      args.stdenv().mkDerivation ({
        name : "file-5.11",
        src : src,
        buildInputs : [ zlib ],
    
        meta : {
          description : "A program that shows the type of files",
          homepage : new nijs.NixURL("http://darwinsys.com/file")
        }
      }, callback);
    }
  ], callback);
};

The above module defines exactly the same package as shown earlier, but defines it asynchronously. For example, it does not return, but uses a callback function to pass the evaluation result back to the caller. I have used the slasp library to flatten its structure to make it better readable and maintainable.

Moreover, because packages implement an asynchronous function interface, we also have to define the composition module in a slightly different way:

var pkgs = {

  stdenv : function(callback) {
    return require('./pkgs-async/stdenv.js').pkg;
  },
   
  fetchurl : function(callback) {
    return require('./pkgs-async/fetchurl').pkg({
      stdenv : pkgs.stdenv
    }, callback);
  },
  
  zlib : function(callback) {
    return require('./pkgs-async/zlib.js').pkg({
      stdenv : pkgs.stdenv,
      fetchurl : pkgs.fetchurl
    }, callback);
  },
  
  file : function(callback) {
    return require('./pkgs-async/file.js').pkg({
      stdenv : pkgs.stdenv,
      fetchurl : pkgs.fetchurl,
      zlib : pkgs.zlib
    }, callback);
  }
}

exports.pkgs = pkgs;

Again, this composition module has the same meaning as the one showed earlier, but each object member implements an asynchronous function interface having a callback.

So why are these asynchronous package specifications useful? In NiJS, there are two use cases for them. The first use case is to compile them to Nix expressions and build them with the Nix package manager (which can also be done with synchronous package definitions):

$ nijs-build pkgs-async.js -A file --async
/nix/store/c7zy6w6ls3mfmr9mvzz3jjaarikrwwrz-file-5.11

The only minor difference is that in order to use asynchronous package definitions, we have to pass the --async parameter to the nijs-build command so that they are properly recognized.

The second (and new!) use case is to execute the functions directly with NiJS. For example, we can also use the same composition module to do the following:

$ nijs-execute pkgs-async.js -A file
/home/sander/.nijs/store/file-5.11

When executing the above command, the Nix package manager is not used at all. Instead, NiJS directly executes the build function implementing the corresponding package and all its dependencies. All resulting artifacts are stored in a so-called NiJS store, which resides in the user's home directory, e.g.: /home/sander/.nijs/store.

The latter command does not depend on Nix at all making it possible for NiJS to act as an independent package manager, yet having the most important features that Nix also has.

Implementation

The implementation of nijs-execute is straight forward. Every package directly or indirectly invokes the same function that actually executes a build operation: args.stdenv().mkDerivation(args, callback).

The original implementation for nijs-build (that compiles a JavaScript composition module to a Nix expression) looks as follows:

var nijs = require('nijs');

exports.pkg = {
  mkDerivation : function(args, callback) {
    callback(null, new nijs.NixExpression("pkgs.stdenv.mkDerivation "
        +nijs.jsToNix(args)));
  }
};

To make nijs-execute work, we can simply replace the above implementation with the following:

var nijs = require('nijs');

exports.pkg = {
  mkDerivation : function(args, callback) {
      nijs.evaluateDerivation(args, callback);
  }
};

We replace the generated Nix expression that invokes Nixpkgs' stdenv.mkDerivation {} by a direct invocation to nijs.evaluateDerivation() that executes a build directly.

The evaluateDerivation() translates the first parameter object (representing build parameters) to environment variables. Each key corresponds to an environment variable and each value is translated as follows:

• A null value is translated to an empty string
• true translates to "1" and false translates to an empty string.
• A string, number, or xml object are translated to strings literally.
• Objects that are instances of the NixFile and NixURL prototypes are also translated to strings literally.
• Objects instances of the NixInlineJS prototype are converted into a separate builder script, which gets executed by the default builder.
• Objects instances of the NixRecursiveAttrSet prototype and arbitrary objects are considered derivations that need to be evaluated separately.

Furthermore, evaluateDerivation() invokes a generic builder script with similar features as the one in Nixpkgs:

• All environment variables are cleared or set to dummy values, such as HOME=/homeless-shelter.
• It supports the execution of phases. By default, it runs the following phases: unpack, patch, configure, build, install and can be extended with custom ones.
• By default, it executes a GNU Autotools build procedure: ./configure; make; make install with configurable settings (that have a default fallback value).
• It can also take custom build commands so that a custom build procedure can be performed
• It supports build hooks so that the appropriate environment variables are set when providing a buildInputs parameter. By default, the builder automatically sets PATH, C_INCLUDE_PATH and LIBRARY_PATH environment variables. Build hooks can be used to support other languages and environments' settings, such as Python (e.g. PYTHONPATH) and Node.js (e.g. NODE_PATH)

Discussion

Now that NiJS has the ability to act as an independent package manager in addition to serving the purpose of an internal DSL, means that we can deprecate Nix and its sub projects soon and use Nix (for the time being) as a fallback for things that are not supported by NiJS yet.

NiJS has the following advantages over Nix and its sub projects:

• I have discovered that the Nix expression language is complicated and difficult to learn. Like Haskell, it has a solid theoretical foundation and powerful features (such as laziness), but it's too hard to learn by developers without an academic background.
  
  Moreover, I had some difficulties accepting JavaScript in the past, but after discovering how to deal with prototypes and asynchronous programming, I started to appreciate it and really love it now.
  
  JavaScript has all the functional programming abilities that we need, so why should we implement our own language to accomplish the same? Furthermore, many people have proven that JavaScript is the future and we can attract more users if we use a language that more people are familiar with.
• NiJS also prevents some confusion with a future Linux distribution that is going to be built around it. For most people, it is too hard to make a distinction between Nix and NixOS.
  
  With NiJS this is not a problem -- NiJS is supposed to be pronounced in Dutch as: "Nice". The future Linux distribution that will be built around it will be called: "NiJSOS", which should be pronounced as "Nice O-S" in Dutch. This is much easier to remember.
• Same thing holds for Disnix -- Nix sounds like "Nothing" in Dutch and Disnix sounds like: "This is nothing!". This strange similarity has prevented me to properly spread the word to the masses. However "DisniJS" sounds like "This is nice!" which (obviously) sounds much better and is much easier to remember.
• NiJS also makes continuous integration more scalable than Nix. We can finally get rid of all the annoying Perl code (and the Template Toolkit) in Hydra and reimplement it in Node.js using all its powerful frameworks. Since in Node.js all I/O operations are non-blocking, we can make Hydra even more faster and more scalable.

Conclusion

In this blog post, I have shown that we can also specify packages asynchronously in NiJS. Asynchronous package specifications can be built directly with NiJS, without requiring them to be compiled to Nix expressions that must be built with Nix.

Since NiJS has become an independent package manager and JavaScript is the future, we can deprecate Nix (and its sub projects) soon, since NiJS has significant advantages over Nix.

NiJS can be downloaded from my GitHub page and from NPM. NiJS can also bootstrap itself :-)

Moreover, soon I will create a website, set up mailing lists, create an IRC channel, and define the other sub projects that can be built on top of it.

Follow up

UPDATE: It seems that this blog post has attracted quite a bit of attention today. For example, there has been some discussion about it on the Nix mailing list as well as the GNU Guix mailing list. Apparently, I also made a few people upset :-)

Moreover, a lot readers probably did not notice the publishing date! So let me make it clear:

IT'S APRIL FOOLS' DAY!!!!!!!!!!!!!!!

The second thing you may probably wonder is: what exactly is this "joke" supposed to mean?

In fact, NiJS is not a fake package -- it actually does exists, can be installed through Nix and NPM, and is really capable of doing the stuff described in this blog post (as well the two previous ones).

However, the intention to make NiJS a replacement for Nix was a joke! As a matter of fact, I am a proponent of external DSLs and Nix already does what I need!

Furthermore, only 1% of NiJS' features are actually used by me. For the rest, the whole package is just simply a toy, which I created to explore the abilities of internal DSLs and to explore some "what if" scenarios, no matter how silly they would look :-)

Although NiJS can build packages without reliance on Nix, its mechanisms are extremely primitive! The new feature described in this blog post was basically a silly experiment to develop a JavaScript specification that can be both compiled (to Nix) and interpreted (executed by NiJS directly)!

Moreover, the last few years I have heard a lot of funny, silly, and stupid things, about all kinds of aspects related to Nix, NixOS, Disnix and Node.js which I kept in mind. I (sort of) integrated these things into a story and used a bit of sarcasm as a glue! What these things exactly are is an open exercise for the reader :-).