Saturday, December 30, 2017

Seventh annual blog reflection

Today, it is my blog's seventh anniversary. As with previous years, it is a nice opportunity to reflect over last year's writings.

Nix expression generators


As usual, the majority of my work is Nix related. This year, a substantial amount of my time was spend on the development of several kinds of Nix expression generators.

In the first quarter of the year, I improved node2nix's accuracy by implementing a more robust version pinning strategy to prevent NPM from consulting external sources. I also gave a talk about node2nix's state of affairs at FOSDEM 2017, attracting quite a large audience.

In addition to node2nix, I revived a number old PHP projects, modernized them a bit and developed composer2nix that can be used to automatically generate Nix expressions from composer (a package manager for PHP projects) configuration files. Similar to node2nix I have developed PNDP: an internal DSL for Nix in PHP to make the code generation process more robust and reliable.

I later unified some of the generation concepts of node2nix and composer2nix and extended both NiJS (an internal DSL for Nix in JavaScript) and PNDP (an internal DSL for Nix in PHP) to support custom object transformations, making the code of both generators much cleaner and better maintainable.

Finally, I had to make a substantial revision to node2nix to support NPM 5.x's content-addressable cache that conflicts with Nix's purity principles.

Disnix


In addition to the Nix expression generators, I have also released a new major version of Disnix in March: version 0.7.

For this new version, I have developed a new abstraction layer implementing multi-process programming patterns to improve the performance of certain deployment activities and to make the code more readable and better maintainable.

Another major feature addition is a deployment configuration reconstructor. Disnix deployment is centralized and when the coordinator machine disappears, a system can no longer be reliably upgraded. With the reconstructor, it is possible to recover from such failures.

Contrary to 2015 and 2016, I did not do as much Disnix development this year. Apart from these two major feature additions, I only did a couple of maintenance releases.

Other Nix features


I have also developed an NPM package providing an API that can be used to remotely control a Hydra server, a Nix-based continuous integration service, and a command-line client to demonstrate its possibilities.

Web technology


Besides Nix-development, I made a brief journey back in time to a technology area that I used to be very interested in (years ago) before I got involved with anything Nix-related and my research: web technology.

I used to develop a custom web framework around that time. I already published some pieces of it in 2014. In the middle of this year, I have isolated, documented and released the remaining parts. I also developed a number of simple example applications to demonstrate how the framework's features can be used.

In the process, some memories of the past resurfaced and I wrote an essay to reflect over them.

One of my not-so-strong points when it comes to web technology is anything layout and style related. I did a small investigation and wrote a checklist of minimalistic layout considerations for myself.

Mobile app development


I also wrote a blog post about the Model-View-Controller (MVC) paradigm and considerations while extending the company's product with chat functionality.

Blog posts


Like every year, I will publish the top 10 of the most frequently read blog posts so far:

  1. On Nix and GNU Guix. This blog post is still the most popular for five years in a row. It seems that there are still plenty of people out there who want to know the differences. However, it appears that its position can soon be overtaken by the number 2.
  2. Managing private Nix packages outside the Nixpkgs tree. This blog post is a tutorial written for Nix-beginners and seems to have grown quite considerably in popularity. It may soon become my most popular blog post.
  3. An evaluation and comparison of Snappy Ubuntu. Still popular, but gradually dropping. I believe this can be attributed to the fact that Snappy has not been in the news for quite a while.
  4. Setting up a multi-user Nix installation on non-NixOS systems. As with previous blog reflections, this post remains popular and still shows that this is an area open for improvement.
  5. Yet another blog post about Object Oriented Programming and JavaScript. Was in last year's top 10 and remains popular. It appears that I did a fairly good job explaining the prototypes concept.
  6. An alternative explanation of the Nix package manager. This blog post has also been in the top 10 for the last five years. It is gradually dropping in popularity. I still believe that it is important to have a good Nix package manager explanation recipe.
  7. On NixOps, Disnix, service deployment and infrastructure deployment. This blog post was also in last year's top 10 and still popular. It is good to observe that people take interest in both NixOps and Disnix.
  8. Asynchronous programming with JavaScript. A JavaScript-related blog post that remains popular.
  9. The NixOS project and deploying systems declaratively. This is the only blog post that was not in last year's top 10. It seems to have quite some impact, in particular the corresponding presentation slides.
  10. Composing FHS-compatible chroot environments with Nix (or deploying Steam in NixOS). Is still popular, but I expect this one to disappear from the top 10 next year.

Some thoughts


I am quite happy with the blog posts I produced this year, yet I have a few observations and ideas for next year to improve upon.

In the middle of this year, I had a significant drop in my blogging productivity (as may be observed by checking the publishing dates on the panel on the right). This drop was caused by a variety of things I will not elaborate about. It took me quite a bit of effort to get back into my usual rhythm and get another story published. This is something I should look after next year.

Another thing I observed by looking at my overall top 10 is that all blog posts except the 10th (about FHS-compatible chroot environments) were written for educational purposes. This year, I have not published any blog posts with education in mind. This is also something I should focus myself a bit more on next year.

Finally, the fact that I did not do so much Disnix development, does not mean that it is finished or that I am out of ideas. I still have a huge list of things that I would like to explore.

Conclusion


I'm still not out of ideas, so stay tuned! The final thing I'd like to say is:


HAPPY NEW YEAR!!!!!

Tuesday, December 19, 2017

Bypassing NPM's content addressable cache in Nix deployments and generating expressions from lock files

Roughly half a year ago, Node.js version 8 was released that also includes a major NPM package manager update (version 5). NPM version 5 has a number of substantial changes over the previous version, such as:

  • It uses package lock files that pinpoint the resolved versions of all dependencies and transitive dependencies. When a project with a bundled package-lock.json file is deployed, NPM will use the pinpointed versions of the packages that are in the lock file making it possible to exactly reproduce a deployment elsewhere. When a project without a lock file is deployed for the first time, NPM will generate a lock file.
  • It has a content-addressable cache that optimizes package retrieval processes and allows fully offline package installations.
  • It uses SHA-512 hashing (as opposed to the significantly weakened SHA-1), for packages published in the NPM registry.

Although these features offer significant benefits over previous versions, e.g. NPM deployments are now much faster, more secure and more reliable, it also comes with a big drawback -- it breaks the integration with the Nix package manager in node2nix. Solving these problems were much harder than I initially anticipated.

In this blog post, I will explain how I have adjusted the generation procedure to cope with NPM's new conflicting features. Moreover, I have extended node2nix with the ability to generate Nix expressions from package-lock.json files.

Lock files


One of the major new features in NPM 5.0 is the lock file (the idea itself is not so new since NPM-inspired solutions such as yarn and the PHP-based composer already support them for quite some time).

A major drawback of NPM's dependency management is that version specifiers are nominal. They can refer to specific versions of packages in the NPM registry, but also to version ranges, or external artifacts such as Git repositories. The latter category of version specifiers affect reproducibility -- for example, the version range specifier >= 1.0.0 may refer to version 1.0.0 today and to version 1.0.1 tomorrow making it extremely hard to reproduce a deployment elsewhere.

In a development project, it is still possible to control the versions of dependencies by using a package.json configuration that only refers to exact versions. However, for transitive dependencies that may still have loose version specifiers there is only very little control.

To solve this reproducibility problem, a package-lock.json file can be used -- a package lock file pinpoints the resolved versions of all dependencies and transitive dependencies making it possible to reproduce the exact same deployment elsewhere.

For example, for the NiJS package with the following package.json configuration:

{
  "name": "nijs",
  "version": "0.0.25",
  "description": "An internal DSL for the Nix package manager in JavaScript",
  "bin": {
    "nijs-build": "./bin/nijs-build.js",
    "nijs-execute": "./bin/nijs-execute.js"
  },
  "main": "./lib/nijs",
  "dependencies": {
    "optparse": ">= 1.0.3",
    "slasp": "0.0.4"
  },
  "devDependencies": {
    "jsdoc": "*"
  }
}

NPM may produce the following partial package-lock.json file:

{
  "name": "nijs",
  "version": "0.0.25",
  "lockfileVersion": 1,
  "requires": true,
  "dependencies": {
    "optparse": {
      "version": "1.0.5",
      "resolved": "https://registry.npmjs.org/optparse/-/optparse-1.0.5.tgz",
      "integrity": "sha1-dedallBmEescZbqJAY/wipgeLBY="
    },
    "requizzle": {
      "version": "0.2.1",
      "resolved": "https://registry.npmjs.org/requizzle/-/requizzle-0.2.1.tgz",
      "integrity": "sha1-aUPDUwxNmn5G8c3dUcFY/GcM294=",
      "dev": true,
      "requires": {
        "underscore": "1.6.0"
      },
      "dependencies": {
        "underscore": {
          "version": "1.6.0",
          "resolved": "https://registry.npmjs.org/underscore/-/underscore-1.6.0.tgz",
          "integrity": "sha1-izixDKze9jM3uLJOT/htRa6lKag=",
          "dev": true
        }
      }
    },
    ...
}

The above lock file pinpoints all dependencies and development dependencies including transitive dependencies to exact versions, including the locations where they can be obtained from and integrity hash codes that can be used to validate them.

The lock file can also be used to derive the entire structure of the node_modules/ folder in which all dependencies are stored. The top level dependencies property captures all packages that reside in the project's node_modules/ folder. The dependencies property of each dependency captures all packages that reside in a dependency's node_modules/ folder.

If NPM 5.0 is used and no package-lock.json is present in a project, it will automatically generate one.

Substituting dependencies


As mentioned in an earlier blog post, the most important technique to make Nix-NPM integration work is by substituting NPM's dependency management activities that conflict with Nix's dependency management -- Nix is much more strict with handling dependencies (e.g. it uses hash codes derived from the build inputs to identify a package as opposed to a name and version number).

Furthermore, in Nix build environments network access is restricted to prevent unknown artifacts to influence the outcome of a build. Only so-called fixed output derivations, whose output hashes should be known in advance (so that Nix can verify its integrity), are allowed to obtain artifacts from external sources.

To substitute NPM's dependency management, populating the node_modules/ folder ourselves with all required dependencies and substituting certain version specifiers, such as Git URLs, used to suffice. Unfortunately, with the newest NPM this substitution process no longer works. When running the following command in a Nix builder environment:

$ npm --offline install ...

The NPM package manager is forced to work in offline mode consulting its content-addressable cache for the retrieval of external artifacts. If NPM needs to consult an external resource, it throws an error.

Despite the fact that all dependencies are present in the node_modules/ folder, deployment fails with the following error message:

npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/optparse failed: cache mode is 'only-if-cached' but no cached response available.

At first sight, the error message suggests that NPM always requires the dependencies to reside in the content-addressable cache to prevent it from downloading it from external sites. However, when we use NPM outside a Nix builder environment, wipe the cache, and perform an offline installation, it does seem to work properly:

$ npm install
$ rm -rf ~/.npm/_cacache
$ npm --offline install

Further experimentation reveals that NPM augments the package.json configuration files of all dependencies with additional metadata that are prefixed by an underscore (_):

{
  "_from": "optparse@>= 1.0.3",
  "_id": "optparse@1.0.5",
  "_inBundle": false,
  "_integrity": "sha1-dedallBmEescZbqJAY/wipgeLBY=",
  "_location": "/optparse",
  "_phantomChildren": {},
  "_requested": {
    "type": "range",
    "registry": true,
    "raw": "optparse@>= 1.0.3",
    "name": "optparse",
    "escapedName": "optparse",
    "rawSpec": ">= 1.0.3",
    "saveSpec": null,
    "fetchSpec": ">= 1.0.3"
  },
  "_requiredBy": [
    "/"
  ],
  "_resolved": "https://registry.npmjs.org/optparse/-/optparse-1.0.5.tgz",
  "_shasum": "75e75a96506611eb1c65ba89018ff08a981e2c16",
  "_spec": "optparse@>= 1.0.3",
  "_where": "/home/sander/teststuff/nijs",
  "name": "optparse",
  "version": "1.0.5",
  ...
}

It turns out that when the _integrity property in a package.json configuration matches the integrity field of the dependency in the lock file, NPM will not attempt to reinstall it.

To summarize, the problem can be solved in Nix builder environments by running a script that augments the package.json configuration files with _integrity fields with the values from the package-lock.json file.

For Git repository dependency specifiers, there seems to be an additional requirement -- it also seems to require the _resolved field to be set to the URL of the repository.

Reconstructing package lock files


The fact that we have discovered how to bypass the cache in a Nix builder environment makes it possible to fix the integration with the latest NPM. However, one of the limitations of this approach is that it only works for projects that have a package-lock.json file included.

Since lock files are still a relatively new concept, many NPM projects (in particular older projects that are not frequently updated) may not have a lock file included. As a result, their deployments will still fail.

Fortunately, we can reconstruct a minimal lock file from the project's package.json configuration and compose dependencies objects by traversing the package.json configurations inside the node_modules/ directory hierarchy.

The only attribute that cannot be immediately derived are the integrity fields containing hashes that are used for validation. It seems that we can bypass the integrity check by providing a dummy hash, such as:

integrity: "sha1-000000000000000000000000000=",

NPM does not seem to object when it encounters these dummy hashes allowing us to deploy projects with a reconstructed package-lock.json file. The solution is a very ugly hack, but it seems to work.

Generating Nix expressions from lock files


As explained earlier, lock files pinpoint the exact versions of all dependencies and transitive dependencies and describe the structure of the entire dependency graph.

Instead of simulating NPM's dependency resolution algorithm, we can also use the data provided by the lock files to generate Nix expressions. Lock files appear to contain most of the data we need -- the URLs/locations of the external artifacts and integrity hashes that we can use for validation.

Using lock files for generation offer the following advantages:

  • We no longer need to simulate NPM's dependency resolution algorithm. Despite my best efforts and fairly good results, it is hard to truly make it 100% identical to NPM's. When using a lock file, the dependency graph is already given, making deployment results much more accurate.
  • We no longer need to consult external resources to resolve versions and compute hashes making the generation process much faster. The only exception seems to be Git repositories -- Nix needs to know the output hash of the clone whereas for NPM the revision hash suffices. When we encounter a Git dependency, we still need to download it and compute the output hash.

Another minor technical challenge are the integrity hashes -- in NPM lock files integrity hashes are in base-64 notation, whereas Nix uses heximal notation or its own custom base-32 notation. We need to convert the NPM integrity hashes to a notation that Nix understands.

Unfortunately, lock files can only be used in development projects. It appears that packages that are installed directly from the NPM registry, e.g. end-user packages that are installed globally through npm install -g, never include a package lock file. (It even seems that the NPM registry blacklist the lock files when publishing a package in the registry).

For this reason, we still need to keep our own implementation of the dependency resolution algorithm.

Usage


By adding a script that augments the dependencies' package.json configuration files with _integrity fields and by optionally reconstructing a package-lock.json file, NPM integration with Nix has been restored.

Using the new NPM 5.x features is straight forward. The following command can be used to generate Nix expressions for a development project with a lock file:

$ node2nix -8 -l package-lock.json

The above command will directly generate Nix expressions from the package lock file, resulting in a much faster generation process.

When a development project does not ship with a lock file, you can use the following command-line instruction:

$ node2nix -8

The generator will use its own implementation of NPM's dependency resolution algorithm. When deploying the package, the builder will reconstruct a dummy lock file to allow the deployment to succeed.

In addition to development projects, it is also possible to install end-user software, by providing a JSON file (e.g. pkgs.json) that defines an array of dependency specifiers:

[
  "nijs"
, { "node2nix": "1.5.0" }
]

A Node.js 8 compatible expression can be generated as follows:

$ node2nix -8 -i pkgs.json

Discussion


The approach described in this blog post is not the first attempt to fix NPM 5.x integration. In my first attempt, I tried populating NPM's content-addressable cache in the Nix builder environment with artifacts that were obtained by the Nix package manager and forcing NPM to work in offline mode.

NPM exposes its download and cache-related functionality as a set of reusable APIs. For downloading packages from the NPM registry, pacote can be used. For downloading external artifacts through the HTTP protocol make-fetch-happen can be used. Both APIs are built on top of the content-addressable cache that can be controlled through the lower-level cacache API.

The real difficulty is that neither the high-level NPM APIs nor the npm cache command-line instruction work with local directories or local files -- they will only add artifacts to the cache if they come from a remote location. I have partially built my own API on top of cacache to populate the NPM cache with locally stored artifacts pretending that they were fetched from a remote location.

Although I had some basic functionality supported, it turned out to be much more complicated and time consuming to get all functionality implemented.

Furthermore, the NPM authors never promised that these APIs are stable, so the implementation may break at some point in time. As a result, I have decided to look for another approach.

Availability


I just released node2nix version 1.5.0 with NPM 5.x support. It can be obtained from the NPM registry, Github, or directly from the Nixpkgs repository.

Friday, December 8, 2017

Controlling a Hydra server from a Node.js application

In a number of earlier blog posts, I have elaborated about the development of custom configuration management solutions that use Nix to carry out a variety (or all) of its deployment tasks.

For example, an interesting consideration is that when a different programming language than Nix's expression language is used, an internal DSL can be used to make integration with the Nix expression language more convenient and safe.

Another problem that might surface when developing a custom solution is the scale at which builds are carried out. When it is desired to carry out builds on a large scale, additional concerns must be addressed beyond the solutions that the Nix package manager provides, such as managing a build queue, timing out builds that appear to be stuck, and sending notifications in case of a success or failure.

In such cases, it may be very tempting to address these concerns in your own custom build solution. However, some of these concerns are quite difficult to implement, such as build queue management.

There is already a Nix project that solves many of these problems, namely: Hydra, the Nix-based continuous integration service. One of Hydra's lesser known features is that it also exposes many of its operations through a REST API. Apart from a very small section in the Hydra manual, the API is not very well documented and -- as a result -- a bit cumbersome to use.

In this blog post, I will describe a recently developed Node.js package that exposes most of Hydra's functionality with an API that is documented and convenient to use. It can be used to conveniently integrate a Node.js application with Hydra's build management services. To demonstrate its usefulness, I have developed a simple command-line utility that can be used to remotely control a Hydra instance.

Finding relevant API calls


The Hydra API is not extensively documented. The manual has a small section titled: "Using the external API" that demonstrates some basic usage scenarios, such as querying data in JSON format.

Querying data is basically a nearly identical operation to opening the Hydra web front-end in a web browser. The only difference is that the GET request should include a header field stating that the output format should be displayed in JSON format.

For example, the following request fetches an overview of projects in JSON format (as opposed to HTML which is the default):

$ curl -H "Accept: application/json" https://hydra.nixos.org

In addition to querying data in JSON format, Hydra supports many additional REST operations, such as creating, updating or deleting projects and jobsets. Unfortunately, these operations are not documented anywhere in the manual.

By analyzing the Hydra source code and running the following script I could (sort of) semi-automatically derive the REST operations that are interesting to invoke:

$ cd hydra/src/lib/Hydra/Controller
$ find . -type f | while read i
do
    echo "# $i"
    grep -E '^sub [a-zA-Z0-9]+[ ]?\:' $i
    echo
done

Running the script shows me the following (partial) output:

# ./Project.pm
sub projectChain :Chained('/') :PathPart('project') :CaptureArgs(1) {
sub project :Chained('projectChain') :PathPart('') :Args(0) :ActionClass('REST::ForBrowsers') { }
sub edit : Chained('projectChain') PathPart Args(0) {
sub create : Path('/create-project') {
...

The web front-end component of Hydra uses Catalyst, a Perl-based MVC framework. One of the conventions that Catalyst uses is that every request invokes an annotated Perl subroutine.

The above script scans the controller modules, and shows for each module annotated subroutines that may be of interest. In particular, the sub routines with a :ActionClass('REST::ForBrowsers') annotation are relevant -- they encapsulate create, read, update and delete operations for various kinds of data.

For example, the project subroutine shown above supports the following REST operations:

sub project_GET {
    ...
}

sub project_PUT {
    ...
}

sub project_DELETE {
    ...
}

The above operations will query the properties a project (GET), create a new or update an existing project (PUT) or delete a project (DELETE).

With the manual, the extracted information and reading the source code a bit (which is unavoidable since many details are missing such as formal function parameters), I was able to develop a client API supporting a substantial amount of Hydra features.

API usage


Using client the API is relatively straight forward. The general idea is to instantiate the HydraConnector prototype to connect to a Hydra server, such as a local test instance:

var HydraConnector = require('hydra-connector').HydraConnector;

var hydraConnector = new HydraConnector("http://localhost");

and then invoke any of its supported operations:

hydraConnector.queryProjects(function(err, projects) {
    if(err) {
        console.log("Some error occurred: "+err);
    } else {
        for(var i = 0; i < projects.length; i++) {
            var project = projects[i];
            console.log("Project: "+project.name);
        }
    }
});

The above code fragment fetches all projects and displays their names.

Write operations require a user to be logged in with the appropriate permissions. By invoking the login() method, we can authenticate ourselves:

hydraConnector.login("admin", "myverysecretpassword", function(err) {
    if(err) {
        console.log("Login succeeded!");
    } else {
        console.log("Some error occurred: "+err);
    }
});

Besides logging in, the client API also implements a logout() operation to relinquish write operation rights.

As explained in an earlier blog post, write operations require user authentication but all data is publicly readable. When it is desired to restrict read access, Hydra can be placed behind a reverse proxy that requires HTTP basic authentication.

The client API also supports HTTP basic authentication to support this usage scenario:

var hydraConnector = new HydraConnector("http://localhost",
    "sander",
    "12345"); // HTTP basic credentials

Using the command-line client


To demonstrate the usefulness of the API, I have created a utility that serves as a command-line equivalent of Hydra's web front-end. The following command shows an overview of projects:

$ hydra-connect --url https://hydra.nixos.org --projects


As may be observed in the screenshot above, the command-line utility provides suggestions for additional command-line instructions that could be relevant, such as querying more detailed information or modifying data.

The following command shows the properties of an individual project:

$ hydra-connect --url https://hydra.nixos.org --project disnix


By default, the command-line utility will show a somewhat readable textual representation of the data. It is also possible to display the "raw" JSON data of any request for debugging purposes:

$ hydra-connect --url https://hydra.nixos.org --project disnix --json


We can also use the command-line utility to create or edit data, such as a project:

$ hydra-connect --url https://hydra.nixos.org --login
$ export HYDRA_SESSION=...
$ hydra-connect --url https://hydra.nixos.org --project disnix --modify


The application will show command prompts asking the user to provide all the relevant project properties.

When changes have been triggered, the active builds in the queue can be inspected by running:

$ hydra-connect --url https://hydra.nixos.org --status


Many other operations are supported. For example, you can inspect the properties of an individual build:

$ hydra-connect --url https://hydra.nixos.org --build 65100054


And request various kinds of build related artifacts, such as the raw build log of a build:

$ hydra-connect --url https://hydra.nixos.org --build 65100054 --raw-log

or download one of its build products:

$ hydra-connect --url https://hydra.nixos.org --build 65100054 \
  --build-product 4 > /home/sander/Download/disnix-0.7.2.tar.gz

Conclusion


In this blog post, I have shown the node-hydra-connector package that can be used to remotely control a Hydra server from a Node.js application. The package can be obtained from the NPM registry and the corresponding GitHub repository.