Wednesday, October 26, 2016

Push and pull deployment of Nix packages

In earlier blog posts, I have covered various general concepts of the Nix package manager. For example, I have written three blog posts explaining Nix from a system administrator, programming language, and a sales perspective.

Furthermore, I have written a number of blog posts demonstrating how Nix can be used, such as managing a set of private packages, packaging binary-only software, and Nix's declarative nature -- the Nix package manager (as well as other tools in the Nix project), are driven by declarative specifications describing the structure of the system to deploy (e.g. the packages and its dependencies). From such a specification, Nix carries out all required deployment activities, including building the packages from its sources, distributing packages from the producer to consumer site, installation and uninstallation.

In addition to the execution of these activities, Nix provides a number of powerful non-functional properties, such as strong guarantees that dependency specifications are complete, that package builds can be reproduced, and that upgrades are non-destructive, atomic, and can always be rolled back.

Although many of these blog posts cover the activities that the Nix package manager typically carries out (e.g. the three explanation recipes cover building and the blog post explaining the user environment concepts discusses installing, uninstalling and garbage collection), I have not elaborated much about the distribution mechanisms.

A while ago, I noticed that people were using one of my earlier blog post as a reference. Despite being able to set up a collection of private Nix expressions, there were still some open questions left, such as fully setting up a private package repository, including the distribution of package builds.

In this blog post, I will explain Nix's distribution concepts and show how they can be applied to private package builds. As with my earlier blog post on managing private packages, these steps should be relatively easy to repeat.

Source and transparent binary deployments


As explained in my earlier blog post on packaging private software, Nix is in principle a source-based package manager. Nix packages are driven by Nix expressions describing how to build packages from source code and all its build-time dependencies, for example:

{ stdenv, fetchurl
, pkgconfig, perl, glib, gpm, slang, zip, unzip, file, gettext
, libX11, libICE, e2fsprogs
}:

stdenv.mkDerivation {
  name = "mc-4.8.12";
  
  src = fetchurl {
    url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
    sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
  };
  
  buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
      libX11 libICE e2fsprogs ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    license = "GPLv2+";
    maintainers = [ stdenv.lib.maintainers.sander ];
  };
}

The above expression (mc.nix) describes how to build Midnight Commander from source code and its dependencies, such as pkgconfig, perl and glib. Because the above expression does not specify any build procedure, the Nix builder environment reverts to the standard GNU Autotools build procedure, that typically consists of the following build steps: ./configure; make; make install.

Besides describing how to build a package, we must also compose a package by providing it the right versions or variants of the dependencies that it requires. Composition is typically done in a second expression referring to the former:

{ nixpkgs ? <nixpkgs>
, system ? builtins.currentSystem
}:

let
  pkgs = import nixpkgs { inherit system; };
  
  callPackage = pkgs.lib.callPackageWith (pkgs // pkgs.xlibs // self);
  
  self = rec {
    mc = callPackage ./mc.nix { };
    
    # Other package imports
  };
in
self

In the above expression (default.nix), the mc attribute imports our former expression and provides the build-time dependencies as function parameters. The dependencies originate from the Nixpkgs collection.

To build the Nix expression shown above, it typically suffices to run:

$ nix-build -A mc
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The result of a Nix build is a Nix store path in which the build result is stored.

As may be noticed, the prefix of the package name (jal99995sk6rixym4gfwcagmdiqrwv9a) is a SHA256 hash code that has been derived from all involved build dependencies, such as the source tarball, build-time dependencies and build scripts. Changing any of these dependencies (such as the version of the Midnight Commander) triggers a rebuild and yields a different hash code. Because hash codes ensure that the Nix store paths to packages will be unique, we can safely store multiple versions and variants of the same packages next to each other.

In addition to executing builds, Nix also takes as many precautions to ensure purity. For example, package builds are carried out in isolated environments in which only the specified dependencies can be found. Moreover, Nix uses all kinds of techniques to make builds more deterministic, such as resetting the timestamps of all files to 1 UNIX time, making build outputs read-only etc.

The combination of unique hash codes and pure builds results in a property called transparent binary deployments -- a package with an identical hash prefix results in a (nearly) bit-identical build regardless on what machine the build has been performed. If we want to deploy a package with a certain hash prefix that already exists on a trustable remote machine, then we can also transfer the package instead of building it again.

Distribution models


The Nix package manager (as well as its sub projects) support two kinds of distribution models -- push and pull deployment.

Push deployment


Push deployment is IMO conceptually the simplest, but at the same time, infrequently used on package management level and not very well-known. The idea of push deployment is that you take an existing package build on your machine (the producer) and transfer it elsewhere, including all its required dependencies.

With Nix this can be easily accomplished with the nix-copy-closure command, for example:

$ nix-copy-closure --to sander@consumermachine.org \
  /nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The above command serializes the Midnight Commander store path including all its dependencies, transfers them to the provided target machine through SSH, and then de-serializes and imports the store paths into the remote Nix store.

An implication of push deployment is that the producer requires authority over the consumer machine. Moreover, nix-copy-closure can transfer store paths from one machine to another, but does not execute any additional deployment steps, such as the "installation" of packages (in Nix packages become available to end-users by composing a Nix user environment that is in the user's PATH).

Pull deployment


With pull deployment the consumer machine is control instead of the producer machine. As a result, the producer does not require any authority over another machine.

As with push deployment, we can also use the nix-copy-closure command for pull deployment:

$ nix-copy-closure --from sander@producermachine.org \
  /nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

The above command invocation is similar to the previous example, but instead copies a closure of the Midnight Commander from the producer machine.

Aside from nix-copy-closure, Nix offers another pull deployment mechanism that is more powerful and more commonly used, namely: channels. This is what people typically use when installing "end-user" packages with Nix.

The idea of channels is that they are remote HTTP/HTTPS URLs where you can subscribe yourself to. They provide a set of Nix expressions and a binary cache of substitutes:

$ nix-channel --add https://nixos.org/channels/nixos-unstable

The above command adds the NixOS unstable channel to the list of channels. Running the following command:

$ nix-channel --update

Fetches or updates the collection of Nix expressions from the channels allowing us to install any package that it provides. By default, the NIX_PATH environment variable is configured in such a way that they refer to the expressions obtained from channels:

$ echo $NIX_PATH
/nix/var/nix/profiles/per-user/root/channels/nixos:...

With a preconfigured channel, we can install any package we want including prebuilt substitutes, by running:

$ nix-env -i mc

The above command installs the Midnight Commander from the set of Nix expressions from the channel and automatically fetches the substitutes of all involved dependencies. After the installation succeeds, we can start it by running:

$ mc

The push deployment mechanisms of nix-copy-closure and pull deployment mechanisms of the channels can also be combined. When running the following command:

$ nix-copy-closure --to sander@consumermachine.org \
  /nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12 \
  --use-substitutes

The consumer machine first attempts to pull the substitutes of the dependency closure from the binary caches first, and finally the producer pushes the missing packages to the consumer machine. This approach is particularly useful if the connection between the producer and the consumer is slow, but the connection between the consumer and the binary cache is fast.

Setting up a private binary cache


With the concepts of push and pull deployment explained, you may probably wonder how these mechanisms can be used to your own private set of Nix packages?

Fortunately, with nix-copy-closure no additional work is required as it works on any store path, regardless of how it is produced. However, when it is desired to set up your own private binary cache, some additional work is required.

As channels/binary caches use HTTP as a transport protocol, we need to set up a web server with a document root folder serving static files. In NixOS, we can easily configure an Apache HTTP server instance by adding the following lines to a NixOS configuration: /etc/nixos/configuration.nix:

services.httpd = {
  enable = true;
  adminAddr = "admin@producer.org";
  hostName = "producer";
  documentRoot = "/var/www";
};

With the nix-push command we can generate a binary cache for our private package set (that includes the Midnight Commander and all its dependencies):

$ nix-push --dest /var/www $(nix-build default.nix)
/nix/store/jal99995sk6rixym4gfwcagmdiqrwv9a-mc-4.8.12

As may be observed by inspecting the document root folder, we now have a set of compressed NAR files representing the serialized forms of all packages involved and narinfo files capturing a package's metadata:

$ ls /var/www
007mwvk5kskrrgr0xgfnwxmwprd1jdvfxi079maq36mhg9dgwqlv.nar.xz
1c5z374p3z8065hfgyy82k2gi82wq147mrapkpivygcd6yjahcs6.nar.xz
a6gkm5vdylpjnsp9wzcb0c7kz11a3pak.narinfo
02jcvjlg2xha2bibl2ivb3nd0c3gh3nq.narinfo
1cgxv7679hqjvcs7jjjyjrk92n58ldj01yx1bsrvv6q2msh8r2m1.nar.xz
cfx2dvzj062qyj7qya9kcpwi1pk1bgin.narinfo
...

In addition to the binary cache, we also need to make the corresponding Nix expressions available. This can be done by simply creating a tarball out of the private Nix expressions and publishing the corresponding file through the web server:

tar cfvz /var/www/custompkgs.tar.gz custompkgs
custompkgs/
custompkgs/mc.nix
custompkgs/default.nix

On the customer machine, we need to configure the binary cache, by adding the following property to /etc/nix.conf:

binary-caches = http://producermachine.org/

In NixOS, this property can be set by adding the following property to /etc/nixos/configuration.nix:

nix.binaryCaches = [ "http://producermachine.org/" ];
nix.requireSignedBinaryCaches = false;

Additionally, we have to configure the NIX_PATH environment variable to refer to our tarball of Nix expressions:

$ export NIX_PATH=custompkgs=http://producermachine.org/custompkgs.tar.gz:$NIX_PATH

Now, when we run the following command-line instruction:

$ nix-env -f '<custompkgs>' -iA mc
downloading ‘http://producermachine.org/custompkgs.tar.gz’... [0/0 KiB, 0.0 KiB/s]
unpacking ‘http://producermachine.org/custompkgs.tar.gz’...
installing ‘mc-4.8.12’

We can install our custom Midnight Commander package by pulling the package from our own custom HTTP server without explicitly obtaining the set of Nix expressions or building it from source code.

Discussion


In this blog post, I have explained Nix's push and pull deployment concepts and shown how we can use them for a set of private packages, including setting up a private binary cache. The basic idea of binary cache distribution is quite simple: create a tarball of your private Nix expressions, construct a binary cache with nix-push and publish the files with an HTTP server.

In real-life production scenarios, there are typically more aspects you need to take into consideration beyond the details mentioned in this blog post.

For example, to make binary caches safe and trustable, it is also recommended to use HTTPS instead of plain HTTP connections. Moreover, you may want to sign the substitutes with a cryptographic key. The manual page of nix-push provides more details on how to set this up.

Some inconvenient aspects of the binary cache generation approach shown in this blog post (in addition to the fact that we need to set up an HTTP server), is that the approach is static -- whenever, we have a new version of a package built, we need to regenerate the binary cache and the package set.

To alleviate these inconveniences, there is also a utility called nix-serve that spawns a standalone web server generating substitutes on the fly.

Moreover, the newest version of Nix also provides a so-called binary cache Nix store. When Nix performs operations on the Nix store, it basically talks to a module with a standardized interface. When using the binary cache store module (instead of the standalone or remote Nix store plugin), Nix automatically generates NAR files for any package that gets imported into the Nix store, for example after the successful completion a build. Besides an ordinary binary cache store plugin, there is also plugin capable of uploading substitutes directly to an Amazon AWS S3 bucket.

Apart from the Nix package manager, also Nix-related projects use Nix's distribution facilities in some extent. Hydra, the Nix-based continuous integration server, also supports pull deployment as it can dynamically generate channels from jobsets. Users can subscribe to these channels to install the bleeding-edge builds of a project.

NixOps, a tool that deploys networks of NixOS configurations and automatically instantiates VMs in the cloud, as well as Disnix, a tool that deploys service-oriented systems (distributed systems that can be decomposed in a "distributable units", a.k.a. services) both use push deployment -- from a coordinator machine (that has authority over a collection of target machines) packages are distributed.

Concluding remarks


After writing this blog post and some thinking, I have become quite curious to see what a pull deployment variant of Disnix (and maybe NixOps) would look like.

Although Disnix and NixOps suit all my needs at the moment, I can imagine that when we apply the same concepts in large organizations with multiple distributed teams, it can no longer considered to be practical to work with a centralized deployment approach that requires authority over all build artefacts and the entire production environment.

Monday, September 26, 2016

Simulating NPM global package installations in Nix builds (or: building Grunt projects with the Nix package manager)

A while ago, I "rebranded" my second re-engineered version of npm2nix into node2nix and officially released it as such. My main two reasons for giving the tool a different name is that node2nix is neither a fork nor a continuation of npm2nix, but a tool that is written from scratch (though it incorporates some of npm2nix's ideas and concepts including most of its dependencies).

Furthermore, it approaches the expression generation problem in a fundamentally different way -- whereas npm2nix generates derivations for each package in a dependency tree and composes symlinks to their Nix store paths to allow a package to find its dependencies, node2nix deploys an entire dependency tree in one derivation so that it can more accurately mimic NPM's behaviour including flat-module installations (at the expense of losing the ability to share dependencies among packages and projects).

Because node2nix is conceptually different, I have decided to rename the project so that it can be used alongside the original npm2nix tool that still implements the old generation concepts.

Besides officially releasing node2nix, I have recently extended its feature set with a new concept for a recurring class of NPM development projects.

Global NPM development dependencies


As described in earlier blog posts, node2nix (as well as npm2nix) generate Nix expressions from a set of third party NPM packages (obtained from external sources, such as the NPM registry) or a development project's package.json file.

Although node2nix works fine for most of my development projects, I have noticed that for a recurring class of projects, the auto generation approach is too limited -- some NPM projects may require the presence of globally installed packages and must run additional build steps in order to be deployed properly. A prominent example of such a category of projects are Grunt projects.

Grunt advertises itself as a "The JavaScript Task Runner" and can be used to run all kinds of things, such as code generation, linting, minification etc. The tasks that Grunt carries out are implemented as plugins that must be deployed as a project's development dependencies with the NPM package manager.

(As a sidenote: it is debatable whether Grunt is a tool that NPM developers should use, as NPM itself can also carry out build steps through its script directive, but that discussion is beyond the scope of this blog post).

A Grunt workflow typically looks as follows. Consider an example project, with the following Gruntfile.js:

module.exports = function(grunt) {

  grunt.initConfig({
    jshint: {
      files: ['Gruntfile.js', 'src/**/*.js'],
      options: {
        globals: {
          jQuery: true
        }
      }
    },
    watch: {
      files: ['<%= jshint.files %>'],
      tasks: ['jshint']
    }
  });

  grunt.loadNpmTasks('grunt-contrib-jshint');
  grunt.loadNpmTasks('grunt-contrib-watch');

  grunt.registerTask('default', ['jshint']);

};

The above Gruntfile defines a configuration that iterates over all JavaScript files (*.js files) in the src/ directory and invokes jshint to check for potential errors and code smells.

To deploy the development project, we first have to globally install the grunt-cli command-line utility:

$ npm install -g grunt-cli
$ which grunt
/usr/local/bin/grunt

To be able to carry out the steps, we must update a project's package.json file to have grunt and all its required Grunt plugins as development dependencies:

{
  "name": "grunt-test",
  "version": "0.0.1",
  "private": "true",
  "devDependencies": {
    "grunt": "*",
    "grunt-contrib-jshint": "*",
    "grunt-contrib-watch": "*"
  }
}

Then we must install the development dependencies with the NPM package manager:

$ npm install

And finally, we can run the grunt command-line utility to execute our tasks:

$ grunt

"Global dependencies" in Nix


Contrary to NPM, Nix does not support "global dependencies". As a matter of fact, it takes all kinds of precautions to prevent global dependencies to influence the builds that it performs, such as storing all packages in isolation in a so-called Nix store (e.g. /nix/store--grunt-1.0.1 as opposed to storing files in global directories, such as /usr/lib), initially clearing all environment variables (e.g. PATH) and setting these to the Nix store paths of the provided dependencies to allow packages to find them, running builds in chroot environments, etc.

These precautions are taken for a very good reason: purity -- each Nix package stored in the Nix store has a hash prefix that is computed from all involved build-time dependencies.

With pure builds, we know that (for example) if we encounter a build performed on one machine with a specific hash code and a build with an identical hash code on another machine, their build results are identical as well (with some caveats, but in general there are no observable side effects). Pure package builds are a crucial ingredient to make deployments of systems reliable and reproducible.

In Nix, we must always be explicit about the dependencies of a build. When a dependency is unspecified (something that commonly happens with global dependencies), a build will typically fail because it cannot be (implicitly) found. Similarly, when a build has dependencies on packages that would normally have to be installed globally (e.g. non-NPM dependencies), we must now explicitly provide them as a build inputs.

The problem with node2nix is that it automatically generates Nix expressions and that global dependencies cannot be detected automatically, because they are not specified anywhere in a package.json configuration file.

To cope with this limitation, the generated Nix expressions are made overridable, so that any missing dependency can be provided manually. For example, we may want to deploy an NPM package named floomatic from the following JSON file (node-packages.json):

[
  "floomatic"
]

We can generate Nix expressions from the above specification, by running:

$ node2nix -i node-packages.json

One of floomatic's dependencies is an NPM package named: native-diff-match-patch that requires the Qt 4.x library and pkgconfig. These two packages are non-NPM package dependencies left undetected by the node2nix generator. In conventional Linux distributions, these packages typically reside in global directories, such as /usr/lib, and can still be implicitly found.

By creating an override expression (named: override.nix), we can inject these missing (global) dependencies ourselves:

{pkgs ? import <nixpkgs> {
    inherit system;
}, system ? builtins.currentSystem}:

let
  nodePackages = import ./default.nix {
    inherit pkgs system;
  };
in
nodePackages // {
  floomatic = nodePackages.floomatic.override (oldAttrs: {
    buildInputs = oldAttrs.buildInputs ++ [
      pkgs.pkgconfig
      pkgs.qt4
    ];
  });
}

With the override expression shown above, we can correctly deploy the floomatic package, by running:

$ nix-build override.nix -A floomatic

Providing supplemental NPM packages to an NPM development project


Similar to non-NPM dependencies, we also need to supply the grunt-cli as an additional dependency to allow a Grunt project build to succeed in a Nix build environment. What makes this process difficult is that grunt-cli is also an NPM package. As a consequence, we need to generate a second set of Nix expressions and propagate their generated package configurations as parameters to the former expression. Although it was already possible to do this, because the Nix language is flexible enough, the process is quite complex, hacky and inconvenient.

In the latest node2nix version, I have automated this workflow -- when generating expressions for a development project, it is now also possible to provide a supplemental package specification. For example, for our trivial Grunt project, we can create the following supplemental JSON file (supplement.json) that provides the grunt-cli:

[
  "grunt-cli"
]

We can generate Nix expressions for the development project and supplemental package set, by running:

$ node2nix -d -i package.json --supplement-input supplement.json

Besides providing the grunt-cli as an additional dependency, we also need to run grunt after obtaining all NPM dependencies. With the following wrapper expression (override.nix), we can run the Grunt task runner after all NPM packages have been successfully deployed:

{ pkgs ? import <nixpkgs> {}
, system ? builtins.currentSystem
}:

let
  nodePackages = import ./default.nix {
    inherit pkgs system;
  };
in
nodePackages // {
  package = nodePackages.package.override {
    postInstall = "grunt";
  };
}

As may be observed in the expression shown above, the postInstall hook is responsible for invoking the grunt command.

With the following command-line instruction, we can use the Nix package manager to deploy our Grunt project:

$ nix-build override.nix -A package

Conclusion


In this blog post, I have explained a recurring limitation of node2nix that makes it difficult to deploy projects having dependencies on NPM packages that (in conventional Linux distributions) are typically installed in global file system locations, such as grunt-cli. Furthermore, I have described a new node2nix feature that provides a solution to this problem.

In addition to Grunt projects, the solution described in this blog post is relevant for other tools as well, such as ESLint.

All features described in this blog post are part of the latest node2nix release (version 1.1.0) that can be obtained from the NPM registry and the Nixpkgs collection.

Besides a new release, node2nix is now also used to generate the expressions for the set of NPM packages included with the development and upcoming 16.09 versions of Nixpkgs.

Monday, August 22, 2016

An extended self-adaptive deployment framework for service-oriented systems


Five years ago, while I was still in academia, I built an extension framework around Disnix (named: Dynamic Disnix) that enables self-adaptive redeployment of service-oriented systems. It was an interesting application as it demonstrated the full potential of service-oriented systems having their deployment processes automated with Disnix.

Moreover, the corresponding research paper was accepted for presentation at the SEAMS 2011 symposium (co-located with ICSE 2011) in Honolulu (Hawaii), which was (obviously!) a nice place to visit. :-)

Disnix's development was progressing at a very low pace for a while after I left academia, but since the end of 2014 I made some significant improvements. In contrast to the basic toolset, I did not improve Dynamic Disnix -- apart from the addition of a port assigner tool, I only kept the implementation in sync with Disnix's API changes to prevent it from breaking.

Recently, I have used Dynamic Disnix to give a couple of demos. As a result, I have improved some of its aspects a bit. For example, some basic documentation has been added. Furthermore, I have extended the framework's architecture to take a couple of new deployment planning aspects into account.

Disnix


For readers unfamiliar with Disnix: the primary purpose of the basic Disnix toolset is executing deployment processes of service-oriented systems. Deployments are driven by three kinds of declarative specifications:

  • The services model captures the services (distributed units of deployments) of which a system consists, their build/configuration properties and their inter-dependencies (dependencies on other services that may have to be reached through a network link).
  • The infrastructure model describes the target machines where services can be deployed to and their characteristics.
  • The distribution model maps services in the services model to machines in the infrastructure model.

By writing instances of the above specifications and running disnix-env:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix executes all activities to get the system deployed, such as building their services from source code, distributing them to the target machines in the network and activating them. Changing any of these models and running disnix-env again causes the system to be upgraded. In case of an upgrade, Disnix will only execute the required activities making the process more efficient than deploying a system from scratch.

"Static" Disnix


So, what makes Disnix's deployment approach static? When looking at software systems from a very abstract point of view, they are supposed to meet a collection of functional and non-functional requirements. A change in a network of machines affects the ability for a service-oriented system to meet them, as the services of which these systems consist are typically distributed.

If a system relies on a critical component that has only one instance deployed and the machine that hosts it crashes, the functional requirements can no longer be met. However, even if we have multiple instances of the same components giving better guarantees that no functional requirements will be broken, important non-functional requirements may be affected, such as the responsiveness of a system.

We may also want to optimize a system's non-functional properties, such as its responsiveness, by adding more machines to the network that offer more system resources, or by changing the configuration of existing machine, e.g. upgrading the amount available RAM.

The basic Disnix toolset is considered static, because all these events events require manual modifications to the Disnix models for redeployment, so that a system can meet its requirements under the changed conditions.

For simple systems, manual reconfiguration is still doable, but with one hundred services, one hundred machines or a high frequency of events (or a combination of the three), it becomes too complex and time consuming.

For example, when a machine has been added or removed, we must rewrite the distribution model in such a way that all services are deployed to at least one machine and that none of them are mapped to machines that are not capable or allowed to host them. Furthermore, with microservices (one of their traits is that they typically embed HTTP servers), we must typically bind them to unique TCP ports that do not conflict with system services or other services deployed by Disnix. None of these configuration aspects are trivial for large service-oriented systems.

Dynamic Disnix


Dynamic Disnix extends Disnix's architecture with additional models and tools to cope with the dynamism of service oriented-systems. In the latest version, I have extended its architecture (which has been based on the old architecture described in the SEAMS 2011 paper and corresponding blog post):


The above diagram shows the structure of the dydisnix-self-adapt tool. The ovals denote command-line utilities, the rectangles denote files and the arrows denote files as inputs or outputs. As with the basic Disnix toolset, dydisnix-self-adapt is composed of command-line utilities each being responsible for executing an individual deployment activity:

  • On the top right, the infrastructure generator is shown that captures the configurations of the machines in the network and generates an infrastructure model from it. Currently, two different kinds of generators can be used: disnix-capture-infra (included with the basic toolset) that uses a bootstrap infrastructure model with connectivity settings, or dydisnix-geninfra-avahi that uses multicast DNS (through Avahi) to retrieve the machines' properties.
  • dydisnix-augment-infra is responsible for augmenting the generated infrastructure model with additional settings, such as passwords. It is typically undesired to automatically publish privacy-sensitive settings over a network using insecure connection protocols.
  • disnix-snapshot can be optionally used to preemptively capture the state of all stateful services (services with property: deployState = true; in the services model) so that the state of these services can be restored if a machine crashes or disappears. This tool is new in the extended architecture.
  • dydisnix-gendist generates a mapping of services to machines based on technical and non-functional properties defined in the services and infrastructure models.
  • dydisnix-port-assign assigns unique TCP port numbers to previously undeployed services and retains assigned TCP ports in a previous deployment for optimization purposes. This tool is new in the extended architecture.
  • disnix-env redeploys the system with the (statically) provided services model and the dynamically generated infrastructure and distribution models.

An example usage scenario


When a system has been configured to be (statically) deployed with Disnix (such as the infamous StaffTracker example cases that come in several variants), we need to add a few additional deployment specifications to make it dynamically deployable.

Auto discovering the infrastructure model


First, we must configure the machines in such a way that they publish their own configurations. The basic toolset comes with a primitive solution called: disnix-capture-infra that does not require any additional configuration -- it consults the Disnix service that is installed on every target machine.

By providing a simple bootstrap infrastructure model (e.g. infrastructure-bootstrap.nix) that only provides connectivity settings:

{
  test1.properties.hostname = "test1";
  test2.properties.hostname = "test2";
}

and running disnix-capture-infra, we can obtain the machines' configuration properties:

$ disnix-capture-infra infrastructure-bootstrap.nix

By setting the following environment variable, we can configure Dynamic Disnix to use the above command to capture the machines' infrastructure properties:

$ export DYDISNIX_GENINFRA="disnix-capture-infra infrastructure-bootstrap.nix"

Alternatively, there is the Dynamic Disnix Avahi publisher that is more powerful, but at the same time much more experimental and unstable than disnix-capture-infra.

When using Avahi, each machine uses multicast DNS (mDNS) to publish their own configuration properties. As a result, no bootstrap infrastructure model is needed. Simply gathering the data published by the machines on the same subnet suffices.

When using NixOS on a target machine, the Avahi publisher can be enabled by cloning the dydisnix-avahi Git repository and adding the following lines to /etc/nixos/configuration.nix:

imports = [ /home/sander/dydisnix/dydisnix-module.nix ];
services.dydisnixAvahiTest.enable = true;

To allow the coordinator machine to capture the configurations that the target machines publish, we must enable the Avahi system service. In NixOS, this can be done by adding the following lines to /etc/nixos/configuration.nix:

services.avahi.enable = true;

When running the following command-line instruction, the machines' configurations can be captured:

$ dydisnix-geninfra-avahi

Likewise, when setting the following environment variable:

$ export DYDISNIX_GENINFRA=dydisnix-geninfra-avahi

Dynamic Disnix uses the Avahi-discovery service to obtain an infrastructure model.

Writing an augmentation model


The Java version of StaffTracker for example uses MySQL to store data. Typically, it is undesired to publish the authentication credentials over the network (in particular with mDNS, which is quite insecure). We can augment these properties to the captured infrastructure model with the following augmentation model (augment.nix):

{infrastructure, lib}:

lib.mapAttrs (targetName: target:
  target // (if target ? containers && target.containers ? mysql-database then {
    containers = target.containers // {
      mysql-database = target.containers.mysql-database //
        { mysqlUsername = "root";
          mysqlPassword = "secret";
        };
    };
  } else {})
) infrastructure

The above model implements a very simple password policy, by iterating over each target machine in the discovered infrastructure model and adding the same mysqlUsername and mysqlPassword property when it encounters a MySQL container service.

Mapping services to machines


In addition to a services model and a dynamically generated (and optionally augmented) infrastructure model, we must map each service to machine in the network using a configured strategy. A strategy can be programmed in a QoS model, such as:

{ services
, infrastructure
, initialDistribution
, previousDistribution
, filters
, lib
}:

let
  distribution1 = filters.mapAttrOnList {
    inherit services infrastructure;
    distribution = initialDistribution;
    serviceProperty = "type";
    targetPropertyList = "supportedTypes";
  };

  distribution2 = filters.divideRoundRobin {
    distribution = distribution1;
  };
in
distribution2

The above QoS model implements the following policy:

  • First, it takes the initialDistribution model that is a cartesian product of all services and machines. It filters the machines on the relationship between the type attribute and the list of supportedTypes. This ensures that services will only be mapped to machines that can host them. For example, a MySQL database should only be deployed to a machine that has a MySQL DBMS installed.
  • Second, it divides the services over the candidate machines using the round robin strategy. That is, it divides services over the candidate target machines in equal proportions and in circular order.

Dynamically deploying a system


With the services model, augmentation model and QoS model, we can dynamically deploy the StaffTracker system (without manually specifying the target machines and their properties, and how to map the services to machines):

$ dydisnix-env -s services.nix -a augment.nix -q qos.nix

The Node.js variant of the StaffTracker example requires unique TCP ports for each web service and web application. By providing the --ports parameter we can include a port assignment specification that is internally managed by dydisnix-port-assign:

$ dydisnix-env -s services.nix -a augment.nix -q qos.nix --ports ports.nix

When providing the --ports parameter, the specification gets automatically updated when ports need to be reassigned.

Making a system self-adaptable from a deployment perspective


With dydisnix-self-adapt we can make a service-oriented system self-adaptable from a deployment perspective -- this tool continuously monitors the network for changes, and runs a redeployment when a change has been detected:

$ dydisnix-self-adapt -s services.nix -a augment.nix -q qos.nix

For example, when shutting down a machine in the network, you will notice that Dynamic Disnix automatically generates a new distribution and redeploys the system to get the missing services back.

Likewise, by adding the ports parameter, you can include port assignments as part of the deployment process:

$ dydisnix-self-adapt -s services.nix -a augment.nix -q qos.nix --ports ports.nix

By adding the --snapshot parameter, we can preemptively capture the state of all stateful services (services annotated with deployState = true; in the services model), such as the databases in which the records are stored. If a machine hosting databases disappears, Disnix can restore the state of the databases elsewhere.

$ dydisnix-self-adapt -s services.nix -a augment.nix -q qos.nix --snapshot

Keep in mind that this feature uses Disnix's snapshotting facilities, which may not be the best solution to manage state, in particular with large databases.

Conclusion


In this blog post, I have described an extended architecture of Dynamic Disnix. In comparison to the previous version, a port assigner has been added that automatically provides unique port numbers to services, and the disnix-snapshot utility that can preemptively capture the state of services, so that they can be restored if a machine disappears from the network.

Despite the fact that Dynamic Disnix has some basic documentation and other improvements from a usability perspective, Dynamic Disnix remains a very experimental prototype that should not be used for any production purposes. In contrast to the basic toolset, I have only used it for testing/demo purposes and I still have no real-life production experience with it. :-)

Moreover, I still have no plans to officially release it yet as many aspects still need to be improved/optimized. For now, you have to obtain the Dynamic Disnix source code from Github and use the included release.nix expression to install it. Furthermore, you probably need to a lot of courage. :-)

Finally, I have extended the Java and Node.js versions of the StaffTracker example as well as the virtual hosts example with simple augmentation and QoS models.

Wednesday, August 3, 2016

Porting node-simple-xmpp from the Node.js ecosystem to Titanium

As may have become obvious by reading some of my previous blog posts, I am frequently using JavaScript for a variety of programming purposes. Although JavaScript was originally conceived as a programming language for use in web browsers (mainly to make web pages more interactive), it is also becoming increasingly more popular in environments outside the browser, such as Node.js, a runtime environment for developing scalable network applications and Appcelerator Titanium, an environment to develop cross-platform mobile applications.

Apart from the fact that these environments share a common programming language -- JavaScript -- and a number of basic APIs that come with the language, they all have their own platform-specific APIs to implement most of an application's basic functionality.

Moreover, they have their own ecosystem of third-party software packages. For example, in Node.js the NPM package manager is the ubiquitous way of publishing and obtaining software. For web browsers, bower can be used, although its adoption is not as widespread as NPM.

Because of these differences, reuse between JavaScript environments is not optimal, in particular for packages that have dependencies on functionality that is not part of JavaScript's core API.

In this blog post, I will describe our experiences with porting the simple-xmpp from the Node.js ecosystem to Titanium. This library has dependencies on Node.js-specific APIs, but with our porting strategy we were able to integrate it in our Titanium app without making any modifications to the original package.

Motivation: Adding chat functionality


Not so long ago, me and my colleagues have been looking into extending our mobile app product-line with chat functionality. As described in an earlier blog post, we use Titanium for developing our mobile apps and one of my responsibilities is automating their builds with Nix (and Hydra, the Nix-based continuous integration server).

Developing chat functionality is quite complex, and requires one to think about many concerns, such as the user-experience, security, reliability and scalability. Instead of developing a chat infrastructure from scratch (which would be much too costly for us), we have decided to adopt the XMPP protocol, for the following reasons:

  • Open standard. Everyone is allowed to make software implementing aspects of the XMPP standard. There are multiple server implementations and many client libraries available, in many programming languages including JavaScript.
  • Decentralized. Users do not have to connect to a single server -- a server can relay messages to users connected to another server. A decentralized approach is good for reliability and scalability.
  • Web-based. The XMPP protocol is built on technologies that empower the web (XML and HTTP). The fact that HTTP is used as a transport protocol, means that we can also support clients that are behind a proxy server.
  • Mature. XMPP has been in use for a quite some time and has some very prominent users. Currently, Sony uses it to enrich the PlayStation platform with chat functionality. In the past, it was also used as the basis for Google and Facebook's chat infrastructure.

Picking a server implementation was not too hard, as ejabberd was something I had experience with in my previous job (and as an intern at Philips Research) -- it supports all the XMPP features we need, and has proven to be very mature.

Unfortunately, for Titanium, there was no module available that implements XMPP client functionality, except for an abandoned project named titanium-xmpp that is no longer supported, and no longer seems to work with any recent versions of Titanium.

Finding a suitable library


As there was no working XMPP client module available for Titanium and we consider developing such a client for Titanium from scratch too costly, we first tried to fix titanium-xmpp, but it turned out that too many things were broken. Moreover, it used all kinds of practices (such as an old fashioned way of module loading through Ti.include()) that have been deprecated a long time ago.

Then we tried porting other JavaScript-based libraries to Titanium. The first candidate was strophe.js which is mainly browser-oriented (and can be used in Node.js through phantomjs, an environment providing a non-interactive web technology stack), but had too many areas that had to be modified and browser-specific APIs that require substitutes.

Finally, I discovered node-xmpp, an XMPP framework for Node.js that has a very modular architecture. For example, the client and server aspects are very-well separated as well as the XML parsing infrastructure. Moreover, we discovered simple-xmpp, a library built on top of it to make a number of common tasks easier to implement. Moreover, the infrastructure has been ported to web browsers using browserify.

Browserify is an interesting porting tool -- its main purpose is to provide a replacement for the CommonJS module system, which is a first-class citizen in Node.js, but unavailable in the browser. Browserify statically analyzes closures of CommonJS modules, and packs them into a single JavaScript file so that the module system is no longer needed.

Furthermore, browserify provides browser-equivalent substitutes for many core Node.js APIs, such as events, stream and path, making it considerably easier to migrate software from Node.js to the browser.

Porting simple-xmpp to Titanium


In addition to browserify, there also exists a similar approach for Titanium: titaniumifier, that has been built on top of the browserify architecture.

Similar to browserify, titaniumifier also packs a collection of CommonJS modules into a single JavaScript file. Moreover, it constructs a Titanium module from it, packs it into a Zip file that can be distributed to any Titanium developer so that it can be used by simply placing it into the root folder of the app project and adding the following module requirement to tiapp.xml:

<module platform="commonjs">ti-simple-xmpp</module>

Furthermore, it provides Titanium-equivalent substitute APIs for Node.js core APIs, but its library is considerably more slim and incomplete than browserify.

We can easily apply titatiumifier to simple-xmpp, by creating a new NPM project (package.json file) that has a dependency on simple-xmpp:

{
  "name": "ti-simple-xmpp",
  "version": "0.0.1",
  "dependencies": {
    "simple-xmpp": "1.3.x"
  }
}

and a proxy CommonJS module (index.js) that simply exposes the Simple XMPP module:

exports.SimpleXMPP = require('simple-xmpp');

After installing the project dependencies (simple-xmpp only) with:

$ npm install

We can attempt to migrate it to Titanium, by running the following command-line instruction:

$ titaniumifier

In my first titaniumify attempt, the tool says that some mandatory Titanium specific properties, such as a unique GUID identifier, are missing that need to be added to package.json:

"titaniumManifest": {
    "guid": "76cb731c-5abf-3b79-6cde-f04202e9ea6d"
},

After adding the missing GUID property, a CommonJS titanium module gets produced that we can integrate in any Titanium project we want:

$ titaniumifier
$ ls
ti-simple-xmpp-commonjs-0.0.1.zip

Fixing API mismatches


With our generated CommonJS package, we can start experimenting by creating a simple app that only connects to a remote XMPP server, by adding the following lines to a Titanium app's entry module (app.js):

var xmpp = require('ti-simple-xmpp').SimpleXMPP;

xmpp.connect({
    websocket: { url: 'ws://myserver.com:5280/websocket/' },
    jid : 'username@myserver.com',
    password : 'password',
    reconnect: true,
    preferred: 'PLAIN',
    skipPresence: false
});

In our first trial run, the app crashed with the following error message:

Object prototype may only be an Object or null

This problem seemed to be caused by the following line that constructs an object with a prototype:

ctor.prototype = Object.create(superCtor.prototype, {

After adding a couple of debugging statements in front of the Object.create() line that print the constructor and the prototype's properties, I noticed that it tries to instantiate a stream object (not a constructor function) without a prototype member. Referring to a prototype that is undefined, is apparently not allowed.

Deeper inspection revealed the following code block:

/*<replacement>*/
var Stream;
(function() {
  try {
    Stream = require('st' + 'ream');
  } catch (_) {} finally {
    if(!Stream) Stream = require('events').EventEmitter;
  }
})();
/*</replacement>*/

The above code block attempts to load the stream module, and provides the event emitter as a fallback if it cannot be loaded. The stream string has been scrambled to prevent browserify to statically bundle the module. It appears that the titaniumifier tool provides a very simple substitute that is an object. As a result, it does not use the event emitter as a fallback.

We can easily fix the stream object's prototype problem, by supplying it with an empty prototype property by creating a module (overrides.js) that modifies it:

try {
    var stream = require('st' + 'ream');
    stream.prototype = {};
} catch(ex) {
    // Just ignore if it didn't work
}

and by importing the overrides module in the index module (index.js) before including simple-xmpp:

exports.overrides = require('./overrides');
exports.SimpleXMPP = require('simple-xmpp');

After fixing the prototype problem, the next trial run crashed the app with the following error message:

undefined is not an object (evaluation process.version.slice)

which seemed to be caused by the following line:

var asyncWrite = !process.browser && [ 'v0.10', 'v0.9.'].indexOf(process.version.slice(0, 5)) > -1 ? setImmediate : processNextTick;

Apparently, titaniumifier does not provide any substitute for process.version and as a result invoking the slice member throws an exception. Luckily, we can circumvent this by making sure that process.browser yields true, by adding the following line to the overrides module (overrides.js):

process.browser = true;

The third trial run crashed the app with the following message:

Can't find variable: XMLHttpRequest at ti-simple.xmpp.js (line 1943)

This error is caused by the fact that there is no XMLHttpRequest object -- an API that a web browser would normally provide. I have found a Titanium-based XHR implementation on GitHub that provides an identical API.

By copying the xhr.js file into our project, wrapping it in a module (XMLHttpRequest.js), we can provide a constructor that is identical to the browser API:

exports.XMLHttpRequest = require('./xhr');

global.XMLHttpRequest = module.exports;

By adding it to our index module:

exports.overrides = require('./overrides');
exports.XMLHttpRequest = require('./XMLHttpRequest');
exports.SimpleXMPP = require('simple-xmpp');

we have provided a substitute for the XMLHttpRequest API that is identical to a browser.

In the fourth run, the app crashed with the following error message:

Can't find variable: window at ti-simple-xmpp.js (line 1789)

which seemed to be caused by the following line:

var WebSocket = require('faye-websocket') && require('faye-websocket').Client ? require('faye-websocket').Client : window.WebSocket

Apparently, there is no window object nor a WebSocket constructor, as they are browser-specific and not substituted by titaniumifier.

Fortunately, there seems to be a Websocket module for Titanium that works both on iOS and Android. The only inconvenience is that its API is similar, but not exactly identical to the browser's WebSocket API. For example, creating a WebSocket in the browser is done as follows:

var ws = new WebSocket("ws://localhost/websocket");

whereas with the TiWS module, it must be done as follows:

var tiws = require("net.iamyellow.tiws");

var ws = tiws.open("ws://localhost/websocket");

These differences make it very tempting to manually fix the converted simple XMPP library, but fortunately we can create an adapter that has an identical interface to the browser's WebSocket API, translating calls to the Titanium WebSockets module:

var tiws = require('net.iamyellow.tiws');

function WebSocket() {
    this.ws = tiws.createWS();
    var url = arguments[0];
    this.ws.open(url);

    var self = this;
    
    this.ws.addEventListener('open', function(ev) {
        self.onopen(ev);
    });
    
    this.ws.addEventListener('close', function() {
        self.onclose();
    });
    
    this.ws.addEventListener('error', function(err) {
        self.onerror(err);
    });
    
    this.ws.addEventListener('message', function(ev) {
        self.onmessage(ev);
    });
}

WebSocket.prototype.send = function(message) {
    return this.ws.send(message);
};

WebSocket.prototype.close = function() {
    return this.ws.close();
};

if(global.window === undefined) {
    global.window = {};
}

global.window.WebSocket = module.exports = WebSocket;

Adding the above module to the index module (index.js):

exports.overrides = require('./overrides');
exports.XMLHttpRequest = require('./XMLHttpRequest');
exports.WebSocket = require('./WebSocket');
exports.SimpleXMPP = require('simple-xmpp');

seems to be the last missing piece in the puzzle. In the fifth attempt, the app seems to properly establish an XMPP connection. Coincidentally, all the other chat functions also seem to work like a charm! Yay! :-)

Conclusion


In this blog post, I have described a process in which I have ported simple-xmpp from the Node.js ecosystem to Titanium. The process was mostly automated, followed by a number of trial, error and fix runs.

The fixes we have applied are substitutes (identical APIs for Titanium), adapters (modules that translate calls to a particular API into a calls to a Titanium-specific API) and overrides (imperative modifications to existing modules). These changes did not require me to modify the original package (the original package is simply a dependency of the ti-simple-xmpp project). As a result, we do not have to maintain a fork and we have only little maintenance on our side.

Limitations


Although the porting approach seems to fit our needs, there are a number of things missing. Currently, only XMPP over WebSocket connections are supported. Ordinary XMPP connections require a Titanium-equivalent replacement for Node.js' net.Socket API, which is completely missing.

Moreover, the Titanium WebSockets library has some minor issues. The first time we tested a secure web socket wss:// connection, the app crashed on iOS. Fortunately, this problem has been fixed now.

References


The ti-simple-xmpp package can be obtained from GitHub. Moreover, I have created a bare bones Alloy/Titanium-based example chat app (XMPPTestApp) exposing most of the library's functionality. The app can be used on both iOS and Android:


Acknowledgements


The work described in this blog post is a team effort -- Yiannis Tsirikoglou first attempted to port strophe.js and manually ported simple-xmpp to Titanium before I managed to complete the automated approach described in this blog post. Carlos Henrique Lustosa Zinato provided us Titanium-related advice and helped us diagnosing the TiWS module problems.

Monday, June 20, 2016

Using Disnix as a remote package deployer

Recently, I was asked whether it is possible to use Disnix as a tool for remote package deployment.

As described in a number of earlier blog posts, Disnix's primary purpose is not remote (or distributed) package management, but deploying systems that can be decomposed into services to networks of machines. To deploy these kinds of systems, Disnix executes all required deployment activities, including building services from source code, distributing them to target machines in the network and activating or deactivating them.

However, a service deployment process is basically a superset of an "ordinary" package deployment process. In this blog post, I will describe how we can do remote package deployment by instructing Disnix to only use a relevant subset of features.

Specifying packages as services


In the Nix packages collection, it is a common habit to write each package specification as a function in which the parameters denote the (local) build and runtime dependencies (something that Disnix's manual refers to as intra-dependencies) that the package needs. The remainder of the function describes how to build the package from source code and its provided dependencies.

Disnix has adopted this habit and extended this convention to services. The main difference between Nix package expressions and Disnix service expressions is that the latter also take inter-dependencies into account that refer to run-time dependencies on services that may have been deployed to other machines in the network. For services that have no inter-dependencies, a Disnix expression is identical to an ordinary package expression.

This means that, for example, an expression for a package such as the Midnight Commander is also a valid Disnix service with no inter-dependencies:

{ stdenv, fetchurl, pkgconfig, glib, gpm, file, e2fsprogs
, libX11, libICE, perl, zip, unzip, gettext, slang
}:

stdenv.mkDerivation {
  name = "mc-4.8.12";
  
  src = fetchurl {
    url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
    sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
  };
  
  buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
      libX11 libICE e2fsprogs ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    license = "GPLv2+";
    maintainers = [ stdenv.lib.maintainers.sander ];
  };
}

Composing packages locally


Package and service expressions are functions that do not specify the versions or variants of the dependencies that should be used. To allow services to be deployed, we must compose them by providing the desired versions or variants of the dependencies as function parameters.

As with ordinary Nix packages, Disnix has also adopted this convention for services. In addition, we have to compose a Disnix service twice -- first its intra-dependencies and later its inter-dependencies.

Intra-dependency composition in Disnix is done in a similar way as in the Nix packages collection:

{pkgs, system}:

let
  callPackage = pkgs.lib.callPackageWith (pkgs // self);

  self = {
    pkgconfig = callPackage ./pkgs/pkgconfig { };
  
    gpm = callPackage ./pkgs/gpm { };
  
    mc = callPackage ./pkgs/mc { };
  };
in
self

The above expression (custom-packages.nix) composes the Midnight Commander package by providing its intra-dependencies as function parameters. The third attribute (mc) invokes a function named: callPackage {} that imports the previous package expression and automatically provides the parameters having the same names as the function parameters.

The callPackage { } function first consults the self attribute set (that composes some of Midnight Commander's dependencies as well, such as gpm and pkgconfig) and then any package from the Nixpkgs repository.

Writing a minimal services model


Previously, we have shown how to build packages from source code and its dependencies, and how to compose packages locally. For the deployment of services, more information is needed. For example, we need to compose their inter-dependencies so that services know how to reach them.

Furthermore, Disnix's end objective is to get a running service-oriented system and carries out extra deployment activities for services to accomplish this, such as activation and deactivation. The latter two steps are executed by a Dysnomia plugin that is determined by annotating a service with a type attribute.

For package deployment, specifying these extra attributes and executing these remaining activities are in principle not required. Nonetheless, we still need to provide a minimal services model so that Disnix knows which units can be deployed.

Exposing the Midnight Commander package as a service, can be done as follows:

{pkgs, system, distribution, invDistribution}:

let
  customPkgs = import ./custom-packages.nix {
    inherit pkgs system;
  };
in
{
  mc = {
    name = "mc";
    pkg = customPkgs.mc;
    type = "package";
  };
}

In the above expression, we import our intra-dependency composition expression (custom-packages.nix), and we use the pkg sub attribute to refer to the intra-dependency composition of the Midnight Commander. We annotate the Midnight Commander service with a package type to instruct Disnix that no additional deployment steps need to be performed beyond the installation of the package, such activation or deactivation.

Since the above pattern is common to all packages, we can also automatically generate services for any package in the composition expression:

{pkgs, system, distribution, invDistribution}:

let
  customPkgs = import ./custom-packages.nix {
    inherit pkgs system;
  };
in
pkgs.lib.mapAttrs (name: pkg: {
  inherit name pkg;
  type = "package";
}) customPkgs

The above services model exposes all packages in our composition expression as a service.

Configuring the remote machine's search paths


With the services models shown in the previous section, we have all ingredients available to deploy packages with Disnix. To allow users on the remote machines to conveniently access their packages, we must add Disnix' Nix profile to the PATH of a user on the remote machines:

$ export PATH=/nix/var/nix/profiles/disnix/default/bin:$PATH

When using NixOS, this variable can be extended by adding the following line to /etc/nixos/configuration.nix:

environment.variables.PATH = [ "/nix/var/nix/profiles/disnix/default/bin" ];

Deploying packages with Disnix


In addition to a services model, Disnix needs an infrastructure and distribution model to deploy packages. For example, we can define an infrastructure model that may look as follows:

{
  test1.properties.hostname = "test1";
  test2 = {
    properties.hostname = "test2";
    system = "x86_64-darwin";
  };
}

The above infrastructure model describes two machines that have hostname test1 and test2. Furthermore, machine test2 has a specific system architecture: x86_64-darwin that corresponds to a 64-bit Intel-based Mac OS X.

We can distribute packages to these two machines with the following distribution model:

{infrastructure}:

{
  gpm = [ infrastructure.test1 ];
  pkgconfig = [ infrastructure.test2 ];
  mc = [ infrastructure.test1 infrastructure.test2 ];
}

In the above distribution model, we distribute package gpm to machine test1, pkgconfig to machine test2 and mc to both machines.

When running the following command-line instruction:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix

Disnix executes all activities to get the packages in the distribution model deployed to the machines, such as building them from source code (including its dependencies), and distributing their dependency closures to the target machines.

Because machine test2 may have a different system architecture as the coordinator machine responsible for carrying out the deployment, Disnix can use Nix's delegation mechanism to forward a build to a machine that is capable of doing it.

Alternatively, packages can also be built on the target machines through Disnix:

$ disnix-env --build-on-targets \
  -s services.nix -i infrastructure.nix -d distribution.nix

After the deployment above command-line instructions have succeeded, we should be able to start the Midnight Commander on any of the target machines, by running:

$ mc

Deploying any package from the Nixpkgs repository


Besides deploying a custom set of packages, it is also possible to use Disnix to remotely deploy any package in the Nixpkgs repository, but doing so is a bit tricky.

The main challenge lies in the fact that the Nix packages set is a nested set of attributes, whereas Disnix expects services to be addressed in one attribute set only. Fortunately, the Nix expression language and Disnix models are flexible enough to implement a solution. For example, we can define a distribution model as follows:

{infrastructure}:

{
  mc = [ infrastructure.test1 ];
  git = [ infrastructure.test1 ];
  wget = [ infrastructure.test1 ];
  "xlibs.libX11" = [ infrastructure.test1 ];
}

Note that we use a dot notation: xlibs.libX11 as an attribute name to refer to libX11 that can only be referenced as a sub attribute in Nixpkgs.

We can write a services model that uses the attribute names in the distribution model to refer to the corresponding package in Nixpkgs:

{pkgs, system, distribution, invDistribution}:

pkgs.lib.mapAttrs (name: targets:
  let
    attrPath = pkgs.lib.splitString "." name;
  in
  { inherit name;
    pkg = pkgs.lib.attrByPath attrPath
      (throw "package: ${name} cannot be referenced in the package set")
      pkgs;
    type = "package";
  }
) distribution

With the above service model we can deploy any Nix package to any remote machine with Disnix.

Multi-user package management


Besides supporting single user installations, Nix also supports multi-user installations in which every user has its own private Nix profile with its own set of packages. With Disnix we can also manage multiple profiles. For example, by adding the --profile parameter, we can deploy another Nix profile that, for example, contains a set of packages for the user: sander:

$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix \
  --profile sander

The user: sander can access its own set of packages by setting the PATH environment variable to:

$ export PATH=/nix/var/nix/profiles/disnix/sander:$PATH

Conclusion


Although Disnix has not been strictly designed for this purpose, I have described in this blog post how Disnix can be used as a remote package deployer by using a relevant subset of Disnix features.

Moreover, I now consider the underlying Disnix primitives to be mature enough. As such, I am announcing the release of Disnix 0.6!

Acknowledgements


I gained the inspiration for writing this blog post from discussions with Matthias Beyer on the #nixos IRC channel.

Saturday, June 11, 2016

Deploying containers with Disnix as primitives for multi-layered service deployments

As explained in an earlier blog post, Disnix is a service deployment tool that can only be used after a collection of machines have been predeployed providing a number of container services, such as a service manager (e.g. systemd), a DBMS (e.g. MySQL) or an application server (e.g. Apache Tomcat).

To deploy these machines, we need an external solution. Some solutions are:

  • Manual installations requiring somebody to obtain a few machines, manually installing operating systems (e.g. a Linux distribution), and finally installing all required software packages, such as Nix, Dysnomia, Disnix and any additional container services. Manually configuring a machine is typically tedious, time consuming and error prone.
  • NixOps. NixOps is capable of automatically instantiating networks of virtual machines in the cloud (such as Amazon EC2) and deploying entire NixOS system configurations to them. These NixOS configurations can be used to automatically deploy Dysnomia, Disnix and any container service that we need. A drawback is that NixOps is NixOS-based and not really useful if you want to deploy services to machines running different kinds of operating systems.
  • disnixos-deploy-network. In a Disnix-context, services are basically undefined units of deployment, and we can also automatically deploy entire NixOS configurations to target machines as services. A major drawback of this approach is that we require predeployed machines running Disnix first.

Although there are several ways to manage the underlying infrastructure of services, they are basically all or nothing solutions with regards to automation -- we either have to manually deploy entire machine configurations ourselves or we are stuck to a NixOS-based solution that completely automates it.

In some scenarios (e.g. when it is desired to deploy services to non-Linux operating systems), the initial deployment phase becomes quite tedious. For example, it took me quite a bit of effort to set up the heterogeneous network deployment demo I have given at NixCon2015.

In this blog post, I will describe an approach that serves as an in-between solution -- since services in a Disnix-context can be (almost) any kind of deployment unit, we can also use Disnix to deploy container configurations as services. These container services can also be deployed to non-NixOS systems, which means that we can alleviate the effort in setting up the initial target system configurations where Disnix can deploy services to.

Deploying containers as services with Disnix


As with services, containers in a Disnix-context could take any form. For example, in addition to MySQL databases (that we can deploy as services with Disnix), we can also deploy the corresponding container: the MySQL DBMS server, as a Disnix service:

{ stdenv, mysql, dysnomia
, name ? "mysql-database"
, mysqlUsername ? "root", mysqlPassword ? "secret"
, user ? "mysql-database", group ? "mysql-database"
}:

stdenv.mkDerivation {
  inherit name;
  
  buildCommand = ''
    mkdir -p $out/bin
      
    # Create wrapper script
    cat > $out/bin/wrapper <<EOF
    #! ${stdenv.shell} -e
      
    case "\$1" in
        activate)
            # Create group, user and the initial database if it does not exists
            # ...

            # Run the MySQL server
            ${mysql}/bin/mysqld_safe --user=${user} --datadir=${dataDir} --basedir=${mysql} --pid-file=${pidDir}/mysqld.pid &
            
            # Change root password
            # ...
            ;;
        deactivate)
            ${mysql}/bin/mysqladmin -u ${mysqlUsername} -p "${mysqlPassword}" -p shutdown
            
            # Delete the user and group
            # ...
            ;;
    esac
    EOF
    
    chmod +x $out/bin/wrapper

    # Add Dysnomia container configuration file for the MySQL DBMS
    mkdir -p $out/etc/dysnomia/containers

    cat > $out/etc/dysnomia/containers/${name} <<EOF
    mysqlUsername="${mysqlUsername}"
    mysqlPassword="${mysqlPassword}"
    EOF
    
    # Copy the Dysnomia module that manages MySQL databases
    mkdir -p $out/etc/dysnomia/modules
    cp ${dysnomia}/libexec/dysnomia/mysql-database $out/etc/dysnomia/modules
  '';
}

The above code fragment is a simplified Disnix expression that can be used to deploy a MySQL server. The above expression produces a wrapper script, which carries out a set of deployment activities invoked by Disnix:

  • On activation, the wrapper script starts the MySQL server by spawning the mysqld_safe daemon process in background mode. Before starting the daemon, it also intitializes some of the server's state, such as creating user accounts under which the daemon runs and setting up the system database if it does not exists (these steps are left out of the example for simplicity reasons).
  • On deactivation it shuts down the MySQL server and removes some of the attached state, such as the user accounts.

Besides composing a wrapper script, we must allow Dysnomia (and Disnix) to deploy databases as Disnix services to the MySQL server that we have just deployed:

  • We generate a Dysnomia container configuration file with the MySQL server settings to allow a database (that gets deployed as a service) to know what credentials it should use to connect to the database.
  • We bundle a Dysnomia plugin module that implements the deployment activities for MySQL databases, such as activation and deactivation. Because Dysnomia offers this plugin as part of its software distribution, we make a copy of it, but we could also compose our own plugin from scratch.

With the earlier shown Disnix expression, we can define the MySQL server as a service in a Disnix services model:

mysql-database = {
  name = "mysql-database";
  pkg = customPkgs.mysql-database;
  dependsOn = {};
  type = "wrapper";
};

and distribute it to a target machine in the network by adding an entry to the distribution model:

mysql-database = [ infrastructure.test2 ];

Configuring Disnix and Dysnomia


Once we have deployed containers as Disnix services, Disnix (and Dysnomia) must know about their availability so that we can deploy services to these recently deployed containers.

Each time Disnix has successfully deployed a configuration, it generates Nix profiles on the target machines in which the contents of all services can be accessed from a single location. This means that we can simply extend Dysnomia's module and container search paths:

export DYSNOMIA_MODULES_PATH=$DYSNOMIA_MODULES_PATH:/nix/var/nix/profiles/disnix/containers/etc/dysnomia/modules
export DYSNOMIA_CONTAINERS_PATH=$DYSNOMIA_CONTAINERS_PATH:/nix/var/nix/profiles/disnix/containers/etc/dysnomia/containers

with the paths to the Disnix profiles that have containers deployed.

A simple example scenario


I have modified the Java variant of the ridiculous Disnix StaffTracker example to support a deployment scenario with containers as Disnix services.

First, we need to start with a collection of machines having a very basic configuration without any additional containers. The StaffTracker package contains a bare network configuration that we can deploy with NixOps, as follows:

$ nixops create ./network-bare.nix ./network-virtualbox.nix -d vbox
$ nixops deploy -d vbox

By configuring the following environment variables, we can connect Disnix to the machines in the network that we have just deployed with NixOps:

$ export NIXOPS_DEPLOYMENT=vbox
$ export DISNIX_CLIENT_INTERFACE=disnix-nixops-client

We can write a very simple bootstrap infrastructure model (infrastructure-bootstrap.nix), to dynamically capture the configuration of the target machines:

{
  test1.properties.hostname = "test1";
  test2.properties.hostname = "test2";
}

Running the following command:

$ disnix-capture-infra infrastructure-bootstrap.nix > infrastructure-bare.nix

yields an infrastructure model (infrastructure-containers.nix) that may have the following structure:

{
  "test1" = {
    properties = {
      "hostname" = "test1";
      "system" = "x86_64-linux";
    };
    containers = {
      process = {
      };
      wrapper = {
      };
    };
    "system" = "x86_64-linux";
  };
  "test2" = {
    properties = {
      "hostname" = "test2";
      "system" = "x86_64-linux";
    };
    containers = {
      process = {
      };
      wrapper = {
      };
    };
    "system" = "x86_64-linux";
  };
}

As may be observed in the captured infrastructure model shown above, we have a very minimal configuration only hosting the process and wrapper containers, that integrate with host system's service manager, such as systemd.

We can deploy a Disnix configuration having Apache Tomcat and the MySQL DBMS as services, by running:

$ disnix-env -s services-containers.nix \
  -i infrastructure-bare.nix \
  -d distribution-containers.nix \
  --profile containers

Note that we have provided an extra parameter to Disnix: --profile to isolate the containers from the default deployment environment. If the above command succeeds, we have a deployment architecture that looks as follows:


Both machines have Apache Tomcat deployed as a service and machine test2 also runs a MySQL server.

When capturing the target machines' configurations again:

$ disnix-capture-infra infrastructure-bare.nix > infrastructure-containers.nix

we will receive an infrastructure model (infrastructure-containers.nix) that may have the following structure:

{
  "test1" = {
    properties = {
      "hostname" = "test1";
      "system" = "x86_64-linux";
    };
    containers = {
      tomcat-webapplication = {
        "tomcatPort" = "8080";
      };
      process = {
      };
      wrapper = {
      };
    };
    "system" = "x86_64-linux";
  };
  "test2" = {
    properties = {
      "hostname" = "test2";
      "system" = "x86_64-linux";
    };
    containers = {
      mysql-database = {
        "mysqlUsername" = "root";
        "mysqlPassword" = "secret";
        "mysqlPort" = "3306";
      };
      tomcat-webapplication = {
        "tomcatPort" = "8080";
      };
      process = {
      };
      wrapper = {
      };
    };
    "system" = "x86_64-linux";
  };
}

As may be observed in the above infrastructure model, both machines provide a tomcat-webapplication container exposing the TCP port number that the Apache Tomcat server has been bound to. Machine test2 exposes the mysql-database container with its connectivity settings.

We can now deploy the StaffTracker system (that consists of multiple MySQL databases and Apache Tomcat web applications) by running:

$ disnix-env -s services.nix \
  -i infrastructure-containers.nix \
  -d distribution.nix \
  --profile services

Note that I use a different --profile parameter, to tell Disnix that the StaffTracker components belong to a different environment than the containers. If I would use --profile containers again, Disnix will undeploy the previously shown containers environment with the MySQL DBMS and Apache Tomcat and deploy the databases and web applications, which will lead to a failure.

If the above command succeeds, we have the following deployment architecture:


The result is that we have all the service components of the StaffTracker example deployed to containers that are also deployed by Disnix.

An advanced example scenario: multi-containers


We could go even one step beyond the example I have shown in the previous section. In the first example, we deploy no more than one instance of each container to a machine in the network -- this is quite common, as it rarely happens that you want to run two MySQL or Apache Tomcat servers on a single machine. Most Linux distributions (including NixOS) do not support deploying multiple instances of system services out of the box.

However, with a few relatively simple modifications to the Disnix expressions of the MySQL DBMS and Apache Tomcat services, it becomes possible to allow multiple instances to co-exist on the same machine. What we basically have to do is identifying the conflicting runtime resources, making them configurable and changing their values in such a way that they no longer conflict.

{ stdenv, mysql, dysnomia
, name ? "mysql-database"
, mysqlUsername ? "root", mysqlPassword ? "secret"
, user ? "mysql-database", group ? "mysql-database"
, dataDir ? "/var/db/mysql", pidDir ? "/run/mysqld"
, port ? 3306
}:

stdenv.mkDerivation {
  inherit name;
  
  buildCommand = ''
    mkdir -p $out/bin
    
    # Create wrapper script
    cat > $out/bin/wrapper <<EOF
    #! ${stdenv.shell} -e
       
    case "\$1" in
        activate)
            # Create group, user and the initial database if it does not exists
            # ...

            # Run the MySQL server
            ${mysql}/bin/mysqld_safe --port=${toString port} --user=${user} --datadir=${dataDir} --basedir=${mysql} --pid-file=${pidDir}/mysqld.pid --socket=${pidDir}/mysqld.sock &
            
            # Change root password
            # ...
            ;;
        deactivate)
            ${mysql}/bin/mysqladmin --socket=${pidDir}/mysqld.sock -u ${mysqlUsername} -p "${mysqlPassword}" -p shutdown
            
            # Delete the user and group
            # ...
            ;;
    esac
    EOF
    
    chmod +x $out/bin/wrapper
  
    # Add Dysnomia container configuration file for the MySQL DBMS
    mkdir -p $out/etc/dysnomia/containers
    
    cat > $out/etc/dysnomia/containers/${name} <<EOF
    mysqlUsername="${mysqlUsername}"
    mysqlPassword="${mysqlPassword}"
    mysqlPort=${toString port}
    mysqlSocket=${pidDir}/mysqld.sock
    EOF
    
    # Copy the Dysnomia module that manages MySQL databases
    mkdir -p $out/etc/dysnomia/modules
    cp ${dysnomia}/libexec/dysnomia/mysql-database $out/etc/dysnomia/modules
  '';
}

For example, I have revised the MySQL server Disnix expression with additional parameters that change the TCP port the service binds to, the UNIX domain socket that is used by the administration utilities and the filesystem location where the databases are stored. Moreover, these additional configuration properties are also exposed by the Dysnomia container configuration file.

These additional parameters make it possible to define multiple variants of container services in the services model:

{distribution, invDistribution, system, pkgs}:

let
  customPkgs = import ../top-level/all-packages.nix {
    inherit system pkgs;
  };
in
rec {
  mysql-production = {
    name = "mysql-production";
    pkg = customPkgs.mysql-production;
    dependsOn = {};
    type = "wrapper";
  };
  
  mysql-test = {
    name = "mysql-test";
    pkg = customPkgs.mysql-test;
    dependsOn = {};
    type = "wrapper";
  };
  
  tomcat-production = {
    name = "tomcat-production";
    pkg = customPkgs.tomcat-production;
    dependsOn = {};
    type = "wrapper";
  };
  
  tomcat-test = {
    name = "tomcat-test";
    pkg = customPkgs.tomcat-test;
    dependsOn = {};
    type = "wrapper";
  };
}

I can, for example, map two MySQL DBMS instances and the two Apache Tomcat servers to the same machines in the distribution model:

{infrastructure}:

{
  mysql-production = [ infrastructure.test1 ];
  mysql-test = [ infrastructure.test1 ];
  tomcat-production = [ infrastructure.test2 ];
  tomcat-test = [ infrastructure.test2 ];
}

Deploying the above configuration:

$ disnix-env -s services-multicontainers.nix \
  -i infrastructure-bare.nix \
  -d distribution-multicontainers.nix \
  --profile containers

yields the following deployment architecture:


As can be observed, we have two instances of the same container hosted on the same machine. When capturing the configuration:

$ disnix-capture-infra infrastructure-bare.nix > infrastructure-multicontainers.nix

we will receive a Nix expression that may look as follows:

{
  "test1" = {
    properties = {
      "hostname" = "test1";
      "system" = "x86_64-linux";
    };
    containers = {
      mysql-production = {
        "mysqlUsername" = "root";
        "mysqlPassword" = "secret";
        "mysqlPort" = "3306";
        "mysqlSocket" = "/run/mysqld-production/mysqld.sock";
      };
      mysql-test = {
        "mysqlUsername" = "root";
        "mysqlPassword" = "secret";
        "mysqlPort" = "3307";
        "mysqlSocket" = "/run/mysqld-test/mysqld.sock";
      };
      process = {
      };
      wrapper = {
      };
    };
    "system" = "x86_64-linux";
  };
  "test2" = {
    properties = {
      "hostname" = "test2";
      "system" = "x86_64-linux";
    };
    containers = {
      tomcat-production = {
        "tomcatPort" = "8080";
        "catalinaBaseDir" = "/var/tomcat-production";
      };
      tomcat-test = {
        "tomcatPort" = "8081";
        "catalinaBaseDir" = "/var/tomcat-test";
      };
      process = {
      };
      wrapper = {
      };
    };
    "system" = "x86_64-linux";
  };
}

In the above expression, there are two instances of MySQL and Apache Tomcat deployed to the same machine. These containers have their resources configured in such a way that they do not conflict. For example, both MySQL instances bind to a different TCP ports (3306 and 3307) and different UNIX domain sockets (/run/mysqld-production/mysqld.sock and /run/mysqld-test/mysqld.sock).

After deploying the containers, we can also deploy the StaffTracker components (databases and web applications) to them. As described in my previous blog post, we can use an alternative (and more verbose) notation in the distribution model to directly map services to containers:

{infrastructure}:

{
  GeolocationService = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-test"; }
    ];
  };
  RoomService = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-production"; }
    ];
  };
  StaffService = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-test"; }
    ];
  };
  StaffTracker = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-production"; }
    ];
  };
  ZipcodeService = {
    targets = [
      { target = infrastructure.test2; container = "tomcat-test"; }
    ];
  };
  rooms = {
    targets = [
      { target = infrastructure.test1; container = "mysql-production"; }
    ];
  };
  staff = {
    targets = [
      { target = infrastructure.test1; container = "mysql-test"; }
    ];
  };
  zipcodes = {
    targets = [
      { target = infrastructure.test1; container = "mysql-production"; }
    ];
  };
}

As may be observed in the distribution model above, we deploy databases and web application to both instances that are hosted on the same machine.

We can deploy the services of which the StaffTracker consists, as follows:

$ disnix-env -s services.nix \
  -i infrastructure-multicontainers.nix \
  -d distribution-advanced.nix \
  --profile services

and the result is the following deployment architecture:


As may be observed in the picture above, we now have a running StaffTracker system that uses two MySQL and two Apache Tomcat servers on one machine. Isn't it awesome? :-)

Conclusion


In this blog post, I have demonstrated an approach in which we deploy containers as services with Disnix. Containers serve as potential deployment targets for other Disnix services.

Previously, we only had NixOS-based solutions to manage the configuration of containers, which makes using Disnix on other platforms than NixOS painful, as the containers had to be deployed manually. The approach described in this blog post serves as an in-between solution.

In theory, the process in which we deploy containers as services first followed by the "actual" services, could be generalized and extended into a layered service deployment model, with a new tool automating the process and declarative specifications capturing the properties of the layers.

However, I have decided not to implement this new model any time soon for practical reasons -- in nearly all of my experiences with service deployment, I have almost never encountered the need to have more than two layers supported. The only exception I can think of is the deployment of Axis2 web services to an Axis2 container -- the Axis2 container is a Java web application that must be deployed to Apache Tomcat first, which in turn requires the presence of the Apache Tomcat server.

Availability


I have integrated the two container deployment examples into the Java variant of the StaffTracker example.

The new concepts described in this blog post are part of the development version of Disnix and will become available in the next release.