Saturday, September 22, 2018

Creating Nix build function abstractions for pluggable SDKs

Two months ago, I decomposed the stdenv.mkDerivation {} function abstraction in the Nix packages collection that is basically the de-facto way in the Nix expression language to build software packages from source.

I identified some of its major concerns and developed my own implementation that is composed of layers in which each layer gradually adds a responsibility until it has most of the features that the upstream version also has.

In addition to providing a better separation of concerns, I also identified a pattern that I repeatedly use to create these abstraction layers:

{stdenv, foo, bar}:
{name, buildInputs ? [], ...}@args:

let
  extraArgs = removeAttrs args [ "name" "buildInputs" ];
in
stdenv.someBuildFunction ({
  name = "mypackage-"+name;
  buildInputs = [ foo bar ] ++ buildInputs;
} // extraArgs)

Build function abstractions that follow this pattern (as outlined in the code fragment shown above) have the following properties:

  • The outer function header (first line) specifies all common build-time dependencies required to build a project. For example, if we want to build a function abstraction for Python projects, then python is such a common build-time dependency.
  • The inner function header specifies all relevant build parameters and accepts an arbitrary number of arguments. Some arguments have a specific purpose for the kind of software project that we want to build (e.g. name and buildInputs) while other arguments can be passed verbatim to the build function abstraction that we use as a basis.
  • In the body, we invoke a function abstraction (quite frequently stdenv.mkDerivation {}) that builds the project. We use the build parameters that have a specific meaning to configure specialized build properties and we pass all remaining build parameters that are not conflicting verbatim to the build function that we use a basis.

    A subset of these arguments have no specific meaning and are simply exposed as environment variables in the builder environment.

    Because some parameters are already being used for a specific purpose and others may be incompatible with the build function that we invoke in the body, we compose a variable named: extraArgs in which we remove the conflicting arguments.

Aside from having a function that is tailored towards the needs of building a specific software project (such as a Python project), using this pattern provides the following additional benefits:

  • A build procedure is extendable/tweakable -- we can adjust the build procedure by adding or changing the build phases, and tweak them by providing build hooks (that execute arbitrary command-line instructions before or after the execution of a phase). This is particularly useful to build additional abstractions around it for more specialized deployment procedures.
  • Because an arbitrary number of arguments can be propagated (that can be exposed as environment variables in the build environment), we have more configuration flexibility.

The original objective of using this pattern is to create an abstraction function for GNU Make/GNU Autotools projects. However, this pattern can also be useful to create custom abstractions for other kinds of software projects, such as Python, Perl, Node.js etc. projects, that also have (mostly) standardized build procedures.

After completing the blog post about layered build function abstractions, I have been improving the Nix packages/projects that I maintain. In the process, I also identified a new kind of packaging scenario that is not yet covered by the pattern shown above.

Deploying SDKs


In the Nix packages collection, most build-time dependencies are fully functional software packages. Notable exceptions are so-called SDKs, such as the Android SDK -- the Android SDK "package" is only a minimal set of utilities (such as a plugin manager, AVD manager and monitor).

In order to build Android projects from source code and manage Android app installations, you need to install a variety of plugins, such as build-tools, platform-tools, platform SDKs and emulators.

Installing all plugins is typically a much too costly operation -- it requires you to download many gigabytes of data. In most cases, you only want to install a very small subset of them.

I have developed a function abstraction that makes it possible to deploy the Android SDK with a desired set of plugins, such as:

with import <nixpkgs> {};

let
  androidComposition = androidenv.composeAndroidPackages {
    toolsVersion = "25.2.5";
    platformToolsVersion = "27.0.1";
    buildToolsVersions = [ "27.0.3" ];
    includeEmulator = true;
    emulatorVersion = "27.2.0";
  };
in
androidComposition.androidsdk

When building the above expression (default.nix) with the following command-line instruction:

$ nix-build
/nix/store/zvailnl4f1261cn87s9n29lhj9i7y7iy-androidsdk

We get an Android SDK installation, with tools plugin version 25.2.5, platform-tools version 27.0.1, one instance of the build-tools (version 27.0.1) and an emulator of version 27.0.2. The Nix package manager will download the required plugins automatically.

Writing build function abstractions for SDKs


If you want to create function abstractions for software projects that depend on an SDK, you not only have to execute a build procedure, but you must also compose the SDK in such a way that all plugins are installed that a project requires. If any of the mandatory plugins are missing, the build will most likely fail.

As a result, the function interface must also provide parameters that allow you to configure the plugins in addition to the build parameters.

A very straight forward approach is to write a function whose interface contains both the plugin and build parameters, and propagates each of the required parameters to the SDK composition function, but manually writing this mapping has a number of drawbacks -- it duplicates functionality of the SDK composition function, it is tedious to write, and makes it very difficult to keep it consistent in case the SDK's functionality changes.

As a solution, I have extended the previously shown pattern with support for SDK deployments:

{composeMySDK, stdenv}:
{foo, bar, ...}@args:

let
  mySDKFormalArgs = builtins.functionArgs composeMySDK;
  mySDKArgs = builtins.intersectAttrs mySDKFormalArgs args;
  mySDK = composeMySDK mySDKArgs;
  extraArgs = removeAttrs args ([ "foo" "bar" ]
    ++ builtins.attrNames mySDKFormalArgs);
in
stdenv.mkDerivation ({
  buildInputs = [ mySDK ];
  buildPhase = ''
    ${mySDK}/bin/build
  '';
} // extraArgs)

In the above code fragment, we have added the following steps:

  • First, we dynamically extract the formal arguments of the function that composes the SDK (mySDKFormalArgs).
  • Then, we compute the intersection of the formal arguments of the composition function and the actual arguments from the build function arguments set (args). The resulting attribute set (mySDKArgs) are the actual arguments we need to propagate to the SDK composition function.
  • The next step is to deploy the SDK with all its plugins by propagating the SDK arguments set as function parameters to the SDK composition function (mySDK).
  • Finally, we remove the arguments that we have passed to the SDK composition function from the extra arguments set (extraArgs), because these parameters have no specific meaning for the build procedure.

With this pattern, the build abstraction function evolves automatically with the SDK composition function without requiring me to make any additional changes.

To build an Android project from source code, I can write an expression such as:

{androidenv}:

androidenv.buildApp {
  # Build parameters
  name = "MyFirstApp";
  src = ../../src/myfirstapp
  antFlags = "-Dtarget=android-16";

  # SDK composition parameters
  platformVersions = [ 16 ];
  toolsVersion = "25.2.5";
  platformToolsVersion = "27.0.1";
  buildToolsVersions = [ "27.0.3" ];
}

The expression shown above has the following properties:

  • The above function invocation propagates three build parameters: name referring to the name of the Nix package, src referring to a filesystem location that contains the source code of an Android project, and antFlags that contains command-line arguments that are passed to the Apache Ant build tool.
  • It propagates four SDK composition parameters: platformVersions referring to the platform SDKs that must be installed, toolsVersion to the version of the tools package, platformToolsVersion to the platform-tools package and buildToolsVersion to the build-tool packages.

By evaluating the above function invocation, the Android SDK with the plugins will be composed, and the corresponding SDK will be passed as a build input to the builder environment.

In the build environment, Apache Ant gets invoked build that builds the project from source code. The android.buildApp implementation will dynamically propagate the SDK composition parameters to the androidenv.composeAndroidPackages function.

Availability


The extended build function abstraction pattern described in this blog post is among the structural improvements I have been implementing in the mobile app building infrastructure in Nixpkgs. Currently, it is used in standalone test versions of the Nix android build environment, iOS build environment and Titanium build environment.

The Titanium SDK build function abstraction (a JavaScript-based cross-platform development framework that can produce Android, iOS, and several other kinds of applications from the same codebase) automatically composes both Xcode wrappers and Android SDKs to make the builds work.

The test repositories can be found on my GitHub page and the changes live in the nextgen branches. At some point, they will be reintegrated into the upstream Nixpkgs repository.

Besides mobile app development SDKs, this pattern is generic enough to be applied to other kinds of projects as well.

Thursday, August 2, 2018

Automating Mendix application deployments with Nix

As explained in a previous blog post, Mendix is a low-code development platform -- the general idea behind low-code application development is that instead of writing (textual) code, you model an application, such as the data structures and the corresponding views. One of the benefits of Mendix is that it makes you more productive as a developer, for certain classes of applications.

Although low-code development is conceptually different from a development perspective compared to more "traditional" development approaches (that require you to write code), there is one particular aspect a Mendix application lifecycle has in common. Eventually, you will have to deploy your app to an environment that makes your application available to end users.

For users of the Mendix cloud portal, deploying an application is quite convenient: with just a few simple mouse clicks your application gets deployed to a test, acceptance or production environment.

However, managing on-premise application deployments or actually managing applications in the cloud is all but a simple job. There all all kinds of challenges you need to cope with, such as:

  • Making sure that all dependencies of an app are present, such as a database for storage.
  • Executing all relevant deployment activities to make an app available for use.
  • Upgrading is risky and difficult -- it may break the application and introduce downtime.

There are a variety of deployment solutions available to manage deployment processes. However, no solution is perfect -- every tool has its strengths and weaknesses and no tool is a perfect fit. As a result, we still have to develop custom solutions that automate missing parts in a deployment process and we have many kinds of additional complexities that we need to cope with.

Recently, I investigated whether it would be possible to deploy Mendix applications, with my favourite class of deployment utilities from the Nix project, and I gave an introduction to the Nix project to the R&D department at Mendix.

Using tools from the Nix project


For readers not familiar with Nix: the tools in the Nix project solve many configuration management problems in their own unique way. The basis of all the tools is the Nix package manager that borrows concepts from purely functional programming languages, such as Haskell.

To summarize Nix in just a few sentences: deploying a package with Nix is the same thing as invoking a pure function that constructs a package from source code and its build-time dependencies (that are provided as function parameters). To accomplish purity, Nix composes so-called "pure build environments", in which various restrictions are imposed on the build script to ensure that the outcome will be (almost) identical if a package is built with the same build inputs.

The purely functional deployment model has all kinds of benefits -- for example, it provides very strong dependency completeness and reproducibility guarantees, and all kinds of optimizations (e.g. a package that has been deployed before does not have to be built again, packages that have no dependency on each other can be built in parallel, builds can be downloaded from a remote location or delegated to another machine).

Another important property that all tools in the Nix project have in common is declarative deployment -- instead of describing the deployment activities that need to be carried out, you describe the structure of your system that want to deploy, e.g. the packages, a system configuration, or a network of machines/services. The deployment tools infer the activities that need to be carried out to get the system deployed.

Automating Mendix application deployments with Nix


As an experiment, I investigated how Mendix application deployments could fit in Nix's vision of declarative deployment -- the objective is to take a Mendix project created by the modeler (essentially the "source code" form of an application), write a declarative deployment specification for it, and use the tools from the Nix project to get a machine running with all required components to make the app run.

To bring a Mendix application in a running state, we require the following ingredients:

  • We must obtain the Mendix runtime that interprets the Mendix models. Packaging the Mendix runtime in Nix is fairly straight forward -- simply unzipping the distribution, and moving the package contents into the Nix store, and adding a wrapper script launches the runtime suffices.
  • We must produce a Mendix Deployment Archive (MDA file) that creates a Zip container with all artifacts required to run a Mendix app by the runtime. An MDA file can be produced from a Mendix project by invoking the MxBuild tool. Since MxBuild is required for this, I had to package it as well. Packaging mxbuild is a bit trickier, as it requires mono and Node.js.

Building an MDA file with Nix


The most interesting part is writing a new function abstraction for building MDA files with Nix -- in a Nix builder environment, (almost) any build tool can be used albeit with restrictions that are imposed on them to make builds more pure.

We can also create a function abstraction that invokes mxbuild in a Nix builder environment:

{stdenv, mxbuild, jdk, nodejs}:
{name, mendixVersion, looseVersionCheck ? false, buildInputs ? [], ...}@args:

let
  mxbuildPkg = mxbuild."${mendixVersion}";
  extraArgs = removeAttrs args [ "buildInputs" ];
in
stdenv.mkDerivation ({
  buildInputs = [ mxbuildPkg nodejs ] ++ buildInputs;
  installPhase = ''
    mkdir -p $out
    mxbuild --target=package \
      --output=$out/${name}.mda \
      --java-home ${jdk} \
      --java-exe-path ${jdk}/bin/java \
      ${stdenv.lib.optionalString looseVersionCheck "--loose-version-check"} \
      "$(echo *.mpr)"
    mkdir -p $out/nix-support
    echo "file binary-dist \"$(echo $out/*.mda)\"" > $out/nix-support/hydra-build-products
  '';
} // extraArgs)

The above expression is a function that composes another function that takes common Mendix parameters -- the application name, the version of MxBuild that we want, and whether we want to use a strict or loose version check (it is possible to compile a project developed for a different version of Mendix, if desired).

In the body, we create an output directory in the Nix store, we invoke mxbuild to compile to MDA app and put it in the Nix store, and we generate a configuration file that makes it possible to expose the MDA file as a build product, when Hydra: the Nix-based continuous integration service is being used.

With the build function shown in the code fragment above, we can write a Nix expression for a Mendix project:

{ pkgs ? import  { inherit system; }                                                                                                                             
, system ? builtins.currentSystem
}:

let
  mendixPkgs = import ./nixpkgs-mendix/top-level/all-packages.nix {
    inherit pkgs system;
  };
in
mendixPkgs.packageMendixApp {
  name = "conferenceschedule";
  src = /home/sander/SharedWindowsFolder/ConferenceSchedule-main;
  mendixVersion = "7.13.1";
}

The above expression (conferenceschedule.nix) can be used to build an MDA file for a project named: conferenceschedule, residing in the /home/sander/SharedWindowsFolder/ConferenceSchedule-main directory using Mendix version 7.13.1.

By running the following command-line instruction, we can use Nix to build our MDA:

$ nix-build conferenceschedule.nix 
/nix/store/nbaa7fnzi0xw9nkf27mixyr9awnbj16i-conferenceschedule
$ ls /nix/store/nbaa7fnzi0xw9nkf27mixyr9awnbj16i-conferenceschedule
conferenceschedule.mda  nix-support

In addition to building an MDA, Nix will also download the dependencies: the Mendix runtime and MxBuild, if they have not been installed yet.

Running a Mendix application


Producing an MDA file is an important ingredient in the deployment lifecycle of a Mendix application, but it is not entirely what we want -- what we really want is a running system. To get a running system, additional steps are required beyond producing an MDA:

  • We must unzip the MDA file into a directory with write permissions.
  • We must create writable state sub directories, e.g. data/tmp, data/files.
  • After starting the runtime, we must configure the admin interface, to send instructions to the runtime to initialize the database and start the app:
    $ export M2EE_ADMIN_PORT=9000
    $ export M2EE_ADMIN_PASS=secret
    
  • Finally, we must communicate over the admin interface to configure, initialize the database and start the app:
    curlCmd="curl -X POST http://localhost:$M2EE_ADMIN_PORT \
    -H 'Content-Type: application/json' \
    -H 'X-M2EE-Authentication: $(echo -n "$M2EE_ADMIN_PASS" | base64)' \
    -H 'Connection: close'"
    $curlCmd -d '{ "action": "update_appcontainer_configuration", "params": { "runtime_port": 8080 } }'
    $curlCmd -d '{ "action": "update_configuration", "params": { "DatabaseType": "HSQLDB", "DatabaseName": "myappdb", "DTAPMode": "D" } }'
    $curlCmd -d '{ "action": "execute_ddl_commands" }'
    $curlCmd -d '{ "action": "start" }'
    

These deployment steps cannot be executed by Nix, because Nix's purpose is to manage packages, but not the state of a running process. To automate these remaining parts, we generate scripts that execute the above listed steps.

NixOS integration


NixOS is a Linux distribution that extends Nix's deployment facilities to complete systems. Aside from using the Nix package manage to deploy all packages including the Linux kernel, NixOS' main objective is to deploy an entire system from a single declarative specification capturing the structure of an entire system.

NixOS uses systemd for managing system services. The systemd configuration files are generated by the Nix package manager. We can integrate our Mendix activation scripts with a generated systemd job to fully automate the deployment of a Mendix application.

{pkgs, ...}:

{
  ...

  systemd.services.mendixappcontainer =
    let
      runScripts = ...
      appContainerConfigJSON = ...
      configJSON = ...
    in {
      enable = true;
      description = "My Mendix App";
      wantedBy = [ "multi-user.target" ];
      environment = {
        M2EE_ADMIN_PASS = "secret";
        M2EE_ADMIN_PORT = "9000";
        MENDIX_STATE_DIR = "/home/mendix";
      };
      serviceConfig = {
        ExecStartPre = "${runScripts}/bin/undeploy-app";
        ExecStart = "${runScripts}/bin/start-appcontainer";
        ExecStartPost = "${runScripts}/bin/configure-appcontainer ${appContainerConfigJSON} ${configJSON}";
      };
    };

The partial NixOS configuration shown above defines a systemd job that runs three scripts (as shown in the last three lines):

  • The undeploy-app script removes all non-state artefacts from the working directory.
  • The start-appcontainer script starts the Mendix runtime.
  • The configure-appcontainer script configures the runtime, such as the embedded Jetty server and the database, and starts the application.

Writing a systemd job (as shown above) is a bit cumbersome. To make it more convenient to use, I captured all Mendix runtime functionality in a NixOS module, with an interface exposing all relevant configuration properties.

By importing the Mendix NixOS module into a NixOS configuration, we can conveniently define a machine configuration that runs our Mendix application:

{pkgs, ...}:

{
  require = [ ../nixpkgs-mendix/nixos/modules/mendixappcontainer.nix ];

  services = {
    openssh.enable = true;

     mendixAppContainer = {
       enable = true;
       adminPassword = "secret";
       databaseType = "HSQLDB";
       databaseName = "myappdb";
       DTAPMode = "D";
       app = import ../../conferenceschedule.nix {
         inherit pkgs;
         inherit (pkgs.stdenv) system;
      };
    };
  };

  networking.firewall.allowedTCPPorts = [ 8080 ];
}

In the above configuration, the mendixAppContainer captures all the properties of the Mendix application that we want to run:

  • The password for communicating over the admin interface.
  • The type of database we want to use (in this particular case an in memory HSQLDB instance) and the name of the database.
  • Whether we want to use the application in development (D), test (T), acceptance (A) or production (P) mode.
  • A reference to the MDA that we want to deploy (deployed by a Nix expression that invokes the Mendix build function abstraction shown earlier).

By writing a NixOS configuration file, storing it in /etc/nixos/configuration.nix and running the following command-line instruction:

$ nixos-rebuild switch

A complete system gets deployed with the Nix package manager that runs our Mendix application.

For production use, HSQLDB and directly exposing the embedded Jetty HTTP is not recommended -- instead a more sophisticated database, such as PostgreSQL should be used. For serving HTTP requests, it is recommended to use nginx as a reverse proxy and use it to serve static data and provide caching.

It is also possible to extend the above configuration with a PostgreSQL and nginx system service. The NixOS module system can be used to retrieve the properties from the Mendix app container to make the configuration process more convenient.

Conclusion


In this blog post, I have investigated how Mendix applications can be deployed by using tools from the Nix project. This resulted in the following deployment functionality:

  • A Nix function that can be used to compile an MDA file from a Mendix project.
  • Generated scripts that configure and launch the runtime and the application.
  • A NixOS module that can be used to deploy a running Mendix app as part of a NixOS machine configuration.

Future work


Currently, only single machine deployments are possible. It may also be desirable to connect a Mendix application to a database that is stored on a remote machine. Furthermore, we may also want to deploy multiple Mendix applications to multiple machines in a network. With Disnix, it is possible to automate such scenarios.

Availability


The Nix function abstractions and NixOS module can be obtained from the Mendix GitHub page and used under the terms and conditions of the Apache Software License version 2.0.

Acknowledgements


The work described in this blog post is the result of the so-called "crafting days", in which Mendix supports its employees to experiment completely freely two full days a month.

Furthermore, I have given a presentation about the functionality described in this blog post and an introduction to the Nix project:


and I have also written an introduction-oriented article about it on the Mendix blog.

Thursday, July 26, 2018

Layered build function abstractions for building Nix packages

I have shown quite a few Nix expression examples on my blog. When it is desired to write a Nix expression for a package, it is a common habit to invoke the stdenv.mkDerivation {} function, or functions that are abstractions built around it.

For example, if we want to build a package, such as the trivial GNU Hello package, we can write the following expression:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "hello-2.10";

  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };

  meta = {
    description = "A program that produces a familiar, friendly greeting";
    longDescription = ''
      GNU Hello is a program that prints "Hello, world!" when you run it.
      It is fully customizable.
    '';
    homepage = http://www.gnu.org/software/hello/manual/;
    license = "GPLv3+";
  };
}

and build it with the Nix package manager as follows:

$ nix-build
/nix/store/188avy0j39h7iiw3y7fazgh7wk43diz1-hello-2.10

The above code fragment does probably not look too complicated and is quite easy to repeat for build other kinds of GNU Autotools/GNU Make-based packages. However, stdenv.mkDerivation {} is a big/complex function abstraction that has many responsibilities.

Its most important responsibility is to compose a so-called pure build environments, in which various restrictions are imposed on the build scripts to provide better guarantees that builds are pure (meaning: that they always produce the same (nearly) bit-identical result if the dependencies are the same), such as:

  • Build scripts can only write to designated output directories and temp directories. They are restricted from writing to any other file system location.
  • All environment variables are cleared and some of them are set to default or dummy values, such as search path environment variables (e.g. PATH).
  • All build results are made immutable by removing the write permission bits and their timestamps are reset to one second after the epoch.
  • Running builds as unprivileged users.
  • Optionally, builds run in a chroot environment and use namespaces to restrict access to the host filesystem and the network as much as possible.

In addition to purity, the stdenv.mkDerivation {} function has many additional responsibilities. For example, it also implements a generic builder that is clever enough to build a GNU Autotools/GNU Make project without specifying any build instructions.

For example, the above Nix expression for GNU Hello does not specify any build instructions. The generic builder automatically unpacks the tarball, opens the resulting directory and invokes ./configure --prefix=$out; make; make install with the appropriate parameters.

Because stdenv.mkDerivation {} has many responsibilities and nearly all packages in Nixpkgs depend on it, its implementation is very complex (e.g. thousands of lines of code) and hard to change.

As a personal exercise, I have developed a function abstraction with similar functionality from scratch. My implementation can be decomposed into layers in which every abstraction layer gradually adds additional responsibilities.

Writing "raw" derivations


stdenv.mkDerivation is a function abstraction, not a feature of the Nix expression language. To compose "pure" build environments, stdenv.mkDerivation invokes a Nix expression language construct -- the derivation {} builtin.

(As a sidenote: derivation is strictly speaking not a builtin, but an abstraction built around the derivationStrict builtin, but this is something internal to the Nix package manager. It does not matter for the scope of this blog post).

Despite the fact that this low level function is not commonly used, it is also possible to directly invoke it and compose low-level "raw" derivations to build packages. For example, we can write the following Nix expression (default.nix):

derivation {
  name = "test";
  builder = ./test.sh;
  system = "x86_64-linux";
  person = "Sander";
}

The above expression invokes the derivation builtin function that composes a "pure" build environment:

  • The name attribute specifies the name of the package, that should appear in the resulting Nix store path.
  • The builder attribute specifies that the test.sh executable should be run inside the pure build environment.
  • The system attribute is used to tell Nix that this build should be carried out for x86-64 Linux systems. When Nix is unable to build the package for the requested system architecture, it can also delegate a build to a remote machine that is capable.
  • All attributes (including the attributes described earlier) are converted to environment variables (e.g. strings, numbers and URLs are converted to strings and the boolean value: 'true' is converted to '1') and can be used by the builder process for a variety of reasons.

We can implement the builder process (the test.sh build script) as follows:

#!/bin/sh -e

echo "Hello $person" > $out

The above script generates a greeting message for the provided person (exposed as an environment variable by Nix) and writes it to the Nix store (the output path is provided by the out environment variable).

We can evaluate the Nix expression (and generate the output file with the Hello greeting) by running:

$ nix-build
/nix/store/7j4y5d8rx1vah5v64bpqd5dskhwx5105-test
$ cat result
Hello Sander

The return value of the derivation {} function is a bit confusing. At first sight, it appears to be a string corresponding to the output path in the Nix store. However, some investigation with the nix repl tool reveals that it is much more than that:

$ nix repl
Welcome to Nix version 2.0.4. Type :? for help.

when importing the derivation:

nix-repl> test = import ./default.nix

and describing the result:

nix-repl> :t test
a set

we will see that the result is actually an attribute set, not a string. By requesting the attribute names, we will see the following attributes:

nix-repl> builtins.attrNames test
[ "all" "builder" "drvAttrs" "drvPath" "name" "out" "outPath" "outputName" "person" "system" "type" ]

It appears that the resulting attribute set has the same attributes as the parameters that we passed to derivation, augmented by the following additional attributes:

  • The type attribute that refers to the string: "derivation".
  • The drvAttrs attribute refers to an attribute set containing the original parameters passed to derivation {}.
  • drvPath and outPath refer to the Nix store paths of the store derivation file and output of the build. A side effect of requesting these members is that the expression gets evaluated or built.
  • The out attribute is a reference to the derivation producing the out result, all is a list of derivations of all outputs produced (Nix derivations can also produce multiple output paths in the Nix store).
  • In case there are multiple outputs, the outputName determines the name of the output path that is the default.

Providing basic dependencies


Although we can use the low-level derivation {} function to produce a very simple output file in the Nix store, it is not very useful on its own.

One important limitation is that we only have a (Bourne-compatible) shell (/bin/sh), but no other packages in the "pure" build environment. Nix prevents unspecified dependencies from being found to make builds more pure.

Since a pure build environment is almost entirely empty (with the exception of the shell), the amount of things we can do in an environment created by derivation {} is very limited -- most of the commands that build scripts run are provided by executables belonging to external packages, e.g. commands such as cat, ls (GNU Coreutils), grep (GNU Grep) or make (GNU Make) and should be added to the PATH search environment variable in the build environment.

We may also want to configure additional environment variables to make builds more pure -- for example, on Linux systems, we want to set the TZ (timezone) environment variable to UTC to prevent error messages, such as: "Local time zone must be set--see zic manual page".

To make the execution of more complex build scripts more convenient, we can create a setup script that we can include in a every build script that adds basic utilities to the PATH search environment variable, configures these additional environment variables, and sets the SHELL environment variable to the bash shell residing in the Nix store. We can create a package named: stdenv that provides a setup script to accomplish this:

{bash, basePackages, system}:

let
  shell = "${bash}/bin/sh";
in
derivation {
  name = "stdenv";
  inherit shell basePackages system;
  builder = shell;
  args = [ "-e" ./builder.sh ];
}

The builder script of the stdenv package can be implemented as follows:
set -e

# Setup PATH for base packages
for i in $basePackages
do
    basePackagesPath="$basePackagesPath${basePackagesPath:+:}$i/bin"
done

export PATH="$basePackagesPath"

# Create setup script
mkdir $out
cat > $out/setup <<EOF
export SHELL=$shell
export PATH="$basePackagesPath"
EOF

# Allow the user to install stdenv using nix-env and get the packages
# in stdenv.
mkdir $out/nix-support
echo "$basePackages" > $out/nix-support/propagated-user-env-packages

The above script adds all base packages (GNU Coreutils, Findutils, Diffutils, sed, grep, gawk and bash) to the PATH of builder and creates a script in $out/setup that exports the PATH environment variable and the location to the bash shell.

We can use the stdenv (providing this setup script) as a dependency for building a package, such as:

{stdenv}:

derivation {
  name = "hello";
  inherit stdenv;
  builder = ./builder.sh;
  system = "x86_64-linux";
}

In the corresponding builder script, we include the setup script in the first line and we, for example, invoke various external commands to generate a shell script that says: "Hello world!":

#!/bin/sh -e
source $stdenv/setup

mkdir -p $out/bin

cat > $out/bin/hello <<EOF
#!$SHELL -e

echo "Hello"
EOF

chmod +x $out/bin/hello

The above script works because the setup script adds GNU Coreutils (that includes cat, mkdir and chmod) to the PATH of the builder.

Writing more simple derivations


Using a setup script makes writing build scripts somewhat practical, but there are still a number inconveniences we have to cope with.

The first inconvenience is the system parameter -- in most cases, we want to build a package for the same architecture as the host system's architecture and preferably we want the same architecture for all other packages that we intend to deploy.

Another issue is the shell. /bin/sh is, in a sandbox-enabled Nix installations, a minimal Bourne-compatible shell provided by Busybox, or a reference to the host system's shell in non-sandboxed installations. The latter case could be considered an impurity, because we do not know what kind of shell (e.g. bash, dash, ash ?) or version of a shell we are using (e.g. 3.2.57, 4.3.30 ?). Ideally, we want to use a shell that is provided as a Nix package in the Nix store, because that version is pure.

(As a sidenote: in Nixpkgs, we use the bash shell to run build commands, but this is not a strict requirement. For example, GNU Guix (a package manager that uses several components of the Nix package manager) uses both Guile as a host and guest language. In theory, we could also launch a different kind of interpreter than bash).

The third issue is the meta parameter -- for every package, it is possible to specify meta-data, such as a description, license and homepage reference as an attribute set. Unfortunately, attribute sets cannot be converted to environment variables. To deal with this problem, the meta attribute needs to be removed before we invoke derivation {} and be readded to the return attribute set. (IMO I believe this ideally should be something the Nix package manager could solve by itself).

We can hide all these inconveniences by creating a simple abstraction function that I will call: stdenv.simpleDerivation that can be implemented as follows:

{stdenv, system, shell}:
{builder, ...}@args:

let
  extraArgs = removeAttrs args [ "builder" "meta" ];

  buildResult = derivation ({
    inherit system stdenv;
    builder = shell; # Make bash the default builder
    args = [ "-e" builder ]; # Pass builder executable as parameter to bash
    setupSimpleDerivation = ./setup.sh;
  } // extraArgs);
in
buildResult //
# Readd the meta attribute to the resulting attribute set
(if args ? meta then { inherit (args) meta; } else {})

The above Nix expression basically removes the meta argument, then invokes the derivation {} function, sets the system parameter, uses bash as builder and passes the builder executable as an argument to bash. After building the package, the meta attribute gets readded to the result.

With this abstraction, we can reduce the complexity of the previously shown Nix expression to something very simple:

{stdenv}:

stdenv.simpleDerivation {
  name = "hello";
  builder = ./builder.sh;
  meta = {
    description = "This is a simple testcase";
  };
}

The function abstraction is also sophisticated enough to build something more complex, such as GNU Hello. We can write the following Nix expression that passes all dependencies that it requires as function parameters:

{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.simpleDerivation {
  name = "hello-2.10";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
  inherit stdenv gnumake gnutar gzip gcc binutils;
  builder = ./builder.sh;
}

We can use the following builder script to build GNU Hello:

source $setupSimpleDerivation

export PATH=$PATH:$gnumake/bin:$gnutar/bin:$gzip/bin:$gcc/bin:$binutils/bin

tar xfv $src
cd hello-2.10
./configure --prefix=$out
make
make install

The above script imports a setup script configuring basic dependencies, then extends the PATH environment variable with additional dependencies, and then executes the commands to build GNU Hello -- unpacking the tarball, running the configure script, building the project, and installing the package.

The run command abstraction


We can still improve a bit upon the function abstraction shown previously -- one particular inconvenience that remains is that you have to write two files to get a package built -- a Nix expression that composes the build environment and a builder script that carries out the build steps.

Another repetitive task is configuring search path environment variables (e.g. PATH, PYTHONPATH, CLASSPATH etc.) to point to the appropriate directories in the Nix store. As may be noticed by looking at the code of the previous builder script, this process is tedious.

To address these inconveniences, I have created another abstraction function called: stdenv.runCommand that extends the previous abstraction function -- when no builder parameter has been provided, this function executes a generic builder that will evaluate the buildCommand environment variable containing a string with shell commands to execute. This feature allows us to rewrite the first example (that generates a shell script) to one file:

{stdenv}:

stdenv.runCommand {
  name = "hello";
  buildCommand = ''
    mkdir -p $out/bin
    cat > $out/bin/hello <<EOF
    #! ${stdenv.shell} -e

    echo "Test"
    EOF
    chmod +x $out/bin/hello
  '';
}

Another feature of the stdenv.runCommand abstraction is to provide a generic mechanism to configure build-time dependencies -- all build-time dependencies that a package needs can be provided as a list of buildInputs. The generic builder carries out all necessary build steps to make them available. For example, when a package provides a bin/ sub folder, then it will be automatically added to the PATH environment variable.

Every package can bundle a setup-hook.sh script that modifies the build environment so that it knows how dependencies for this package can be configured. For example, the following partial expression represents the Perl package that bundles a setup script:

{stdenv, ...}:

stdenv.mkDerivation {
  name = "perl";
  ...
  setupHook = ./setup-hook.sh
}

The setup hook can automatically configure the PERL5LIB search path environment variable for all packages that provide Perl modules:

addPerlLibPath()
{
    addToSearchPath PERL5LIB $1/lib/perl5/site_perl
}

envHooks+=(addPerlLibPath)

When we add perl as a build input to a package, then its setup hook configures the generic builder in such a way that the PERL5LIB environment variable is automatically configured when we provide a Perl module as a build input.

We can also more conveniently build GNU Hello, by using the buildInputs parameter:

{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.runCommand {
  name = "hello-2.10";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
  buildInputs = [ gnumake gnutar gzip gcc binutils ];
  buildCommand = ''
    tar xfv $srcb
    cd hello-2.10
    ./configure --prefix=$out
    make
    make install
  '';
}

Compared to the previous GNU Hello example, this Nix expression is much simpler and more intuitive to write.

The run phases abstraction


We can improve the ease of use for build processes even further. GNU Hello, and many other GNU packages and other system software used for Linux are GNU Autotools/GNU Make based and follow similar conventions including the build commands you need to carry out. Likewise, many other software projects use standardized build tools that follow conventions.

As a result, when you have to maintain a collection of packages, you probably end up writing the same kinds of build instructions over and over again.

To alleviate this problem, I have created another abstraction layer, named: stdenv.runPhases making it possible to define and execute phases in a specific order. Every phase has a pre and post hook (a script that executes before and after each phase) and can be disabled or reenabled with a do* or dont* flag.

With this abstraction function, we can divide builds into phases, such as:

{stdenv}:

stdenv.runPhases {
  name = "hello";
  phases = [ "build" "install" ];
  buildPhase = ''
    cat > hello <<EOF
    #! ${stdenv.shell} -e
    echo "Hello"
    EOF
    chmod +x hello
  '';
  installPhase = ''
    mkdir -p $out/bin
    mv hello $out/bin
  '';
}

The above Nix expression executes a build and install phase. In the build phase, we construct a script that echoes "Hello", and in the install phase we move the script into the Nix store and we make it executable.

In addition to environment variables, it is also possible to define the phases in a setup script as shell functions. For example, we can also use a builder script:

{stdenv}:

stdenv.runPhases {
  name = "hello2";
  builder = ./builder.sh;
}

and define the phases in the builder script:

source $setupRunPhases

phases="build install"

buildPhase()
{
    cat > hello <<EOF
#! $SHELL -e
echo "Hello"
EOF
    chmod +x hello
}

installPhase()
{
    mkdir -p $out/bin
    mv hello $out/bin
}

genericBuild

Another feature of this abstraction is that we can also define exitHook and failureHook parameters that will be executed if the builder succeeds or fails.

In the next sections, I will show abstractions built on top of stdenv.runPhases that can be used to hide implementation details of common build procedures.

The generic build abstraction


For many build procedures, we need to carry out the same build steps, such as: unpacking the source archives, applying patches, and stripping debug symbols from the resulting ELF executables.

I have created another build function abstraction named: stdenv.genericBuild that implements a number of common build phases:

  • The unpack phase generically unpacks the provided sources, makes it content writable and opens the source directory. The unpack command is determined by the unpack hook that each potential unpacker provides -- for example, the GNU tar package includes a setup hook that untars the file if it looks like a tarball or compressed tarball:

    _tryUntar()
    {
        case "$1" in
            *.tar|*.tar.gz|*.tar.bz2|*.tar.lzma|*.tar.xz)
                tar xfv "$1"
                ;;
            *)
                return 1
                ;;
        esac
    }
    
    unpackHooks+=(_tryUntar)
    
  • The patch phase applies any patch that is provided by the patches parameter uncompressing them when necessary. The uncompress file operation also works with setup hooks -- uncompressor packages (such as gzip and bzip2) provide a setup hook that uncompresses the file if it is of the right filetype.
  • The strip phase processes all sub directories containing ELF binaries (e.g. bin/ and lib/) and strips their debugging symbols. This reduces the size of the binaries and removes non-deterministic timestamps.
  • The patchShebangs phase processes all scripts with a shebang line and changes it to correspond to a path in the Nix store.
  • The compressManPages phase compresses all manual pages with gzip.

We can also add GNU patch as as base package for this abstraction function, since it is required to execute the patch phase. As a result, it does not need to be specified as a build dependency for each package.

This function abstraction alone is not very useful, but it captures all common aspects that most build tools use, such as GNU Make, CMake or SCons projects.

I can reduce the size of the previously shown GNU Hello example Nix expression to the following:

{stdenv, fetchurl, gnumake, gnutar, gzip, gcc, binutils}:

stdenv.genericBuild {
  name = "hello-2.10";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
  buildInputs = [ gnumake gnutar gzip gcc binutils ];
  buildCommandPhase = ''
    ./configure --prefix=$out
    make
    make install
  '';
}

In the above expression, I no longer have to specify how to unpack the download GNU Hello source tarball.

GNU Make/GNU Autotools abstraction


We can extend the previous function abstraction even further with phases that automate a complete GNU Make/GNU Autotools based workflow. This abstraction is what we can call stdenv.mkDerivation and is comparable in terms of features with the implementation in Nixpkgs.

We can adjust the phases to include a configure, build, check and install phase. The configure phase checks whether a configure script exists and executes it. The build, check and install phases will execute: make, make check and make install with appropriate parameters.

We can also add common packages that we need to build these projects as base packages so that they no longer have to be provided as a build input: GNU Tar, gzip, bzip2, xz, GNU Make, Binutils and GCC.

With these additional phases and base packages, we can reduce the GNU Hello example to the following expression:

{stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "hello-2.10";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.10.tar.gz;
    sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i";
  };
}

The above Nix expression does not contain any installation instructions -- the generic builder is able to figure out all steps on its own.

Composing custom function abstractions


I have shown several build abstraction layers implementing most features that are in the Nixpkgs version of stdenv.mkDerivation. Aside from clarity, another objective of splitting this function in layers is to make the composition of custom build abstractions more convenient.

For example, we can implement the trivial builder named: writeText whose only responsibility is to write a text file into the Nix store, by extending stdenv.runCommand. This abstraction suffices because writeText does not require any build tools, such as GNU Make and GCC, and it also does not need any generic build procedure executing phases:

{stdenv}:

{ name # the name of the derivation
, text
, executable ? false # run chmod +x ?
, destination ? ""   # relative path appended to $out eg "/bin/foo"
, checkPhase ? ""    # syntax checks, e.g. for scripts
}:

stdenv.runCommand {
  inherit name text executable;
  passAsFile = [ "text" ];

  # Pointless to do this on a remote machine.
  preferLocalBuild = true;
  allowSubstitutes = false;

  buildCommand = ''
    target=$out${destination}
    mkdir -p "$(dirname "$target")"

    if [ -e "$textPath" ]
    then
        mv "$textPath" "$target"
    else
        echo -n "$text" > "$target"
    fi

    [ "$executable" = "1" ] && chmod +x "$target" || true
  '';
}

We can also make a builder for Perl packages, by extending: stdenv.mkDerivation -- Perl packages also use GNU Make as a build system. Its only difference is the configuration step -- it runs Perl's MakeMaker script to generate the Makefile. We can simply replace the configuration phase for GNU Autotools by an implementation that invokes MakeMaker.

When developing custom abstractions, I basically follow this pattern:

{stdenv, foo, bar}:
{name, buildInputs ? [], ...}@args:

let
  extraArgs = removeAttrs args [ "name" "buildInputs" ];
in
stdenv.someBuildFunction ({
  name = "mypackage-"+name;
  buildInputs = [ foo bar ] ++ buildInputs;
} // extraArgs)

  • A build function is a nested function in which the first line is a function header that captures the common build-time dependencies required to build a package. For example, when we want to build Perl packages, then perl is such a common dependency.
  • The second line is the inner function header that captures the parameters that should be passed to the build function. The notation allows an arbitrary number of parameters. The parameters in the { } block (name, buildInputs) are considered to have a specific use in the body of the function. The remainder of parameters are non-essential -- they are used as environment variables in the builder environment or they can be propagated to other functions.
  • We compose an extraArgs variable that contains all non-essential arguments that we can propagate to the build function. Basically, all function arguments that are used in the body need to be removed and function arguments that are attribute sets, because they cannot be converted to strings.
  • In the body of the function, we set up important aspects of the build environment, such as the mandatory build parameters, and we propagate the remaining function arguments to the builder abstraction function.

Following this pattern also ensures that the builder is flexible enough to be extended and modified. For example, by extending a function that is based on stdenv.runPhases the builder can be extended with custom phases and build hooks.

Discussion


In this blog post, I have derived my own reimplementation of Nixpkgs's stdenv.mkDerivation function that consists of the following layers each gradually adding functionality to the "raw" derivation {} builtin:

  1. "Raw" derivations
  2. The setup script ($stdenv/setup)
  3. Simple derivation (stdenv.simpleDerivation)
  4. The run command abstraction (stdenv.runCommand)
  5. The run phases abstraction (stdenv.runPhases)
  6. The generic build abstraction (stdenv.genericBuild)
  7. The GNU Make/GNU Autotools abstraction (stdenv.mkDerivation)

The features that the resulting stdenv.mkDerivation provides are very similar to the Nixpkgs version, but not entirely identical. Most notably, cross compiling support is completely absent.

From the experience, I have a number of improvement suggestions that we may want to implement in Nixpkgs version to improve the quality and clarity of the generic builder infrastructure:

  • We could also split the implementation of stdenv.mkDerivation and the corresponding setup.sh script into layered sub functions. Currently, the setup.sh script is huge (e.g. over 1200 LOC) and has many responsibilities (perhaps too many). By splitting the build abstraction functions and their corresponding setup scripts, we can separate concerns better and reduce the size of the script so that it becomes more readable and better maintainable.
  • In the Nixpkgs implementation, the phases that the generic builder executes are built for GNU Make/GNU Autotools specifically. Furthermore, the invocation of pre and post hooks and do and dont flags are all hand coded for every phase (there is no generic mechanism that deals with them). As a result, when you define a new custom phase, you need to reimplement the same aspects over and over again. In my implementation, you only have to define phases -- the generic builder automatically executes the coresponding pre and post hooks and evaluates the do and dont flags.
  • In the Nixpkgs implementation there is no uncompressHook -- as a result, the decompression of patch files is completely handcoded for every uncompressor, e.g. gzip, bzip2, xz etc. In my implementation, we can delegate this responsibility to any potential uncompressor package.
  • In my implementation, I turned some of the phases of the generic builder into command-line tools that can be invoked outside the build environment (e.g. patch-shebangs, compress-man). This makes it easier to experiment with these tools and to make adjustments.

The biggest benefit of having separated concerns is flexibility when composing custom abstractions -- for example, the writeText function in Nixpkgs is built on top of stdenv.mkDerivation that includes GNU Make and GCC as dependencies, but does not depend on it. As a result, when one of these packages get updated all generated text files need to be updated as well, while there is no real dependency on it. When using a more minimalistic function, such as stdenv.runCommand this problem will go away.

Availability


I have created a new GitHub repository called: nix-lowlevel-experiments. It contains the implementation of all function abstractions described in this blog post, including some test cases that demonstrate how these functions can be used.

In the future, I will probably experiment with other low level Nix concepts and add them to this repository as well.

Tuesday, June 19, 2018

My introduction to Mendix and low-code application development

As explained in my previous blog post, I started a new challenge two months ago and joined Mendix, a company that develops a low-code application development platform. Because I am receiving many questions about the platform and low-code application development in general, I have decided to write a blog post about it.

Low-code application development


With Mendix, the idea is that instead of writing (textual) code, applications are developed by modeling (and configuring) various kinds of application aspects, such as the data model and graphical user interfaces.

Why would that be useful you might ask? It makes you a developer more productive.

As many seasoned software developers may probably already know, developing applications is typically a costly investment and time consuming process. Aside from implementing all desired functionality and making it work correctly, a developer typically also needs to solve many kinds of secondary issues, such as:

  • Memory management.
  • Interacting with a database for storage.
  • Pagination: Displaying a fixed amount of rows on a page for a collection of data
  • Making screens work on various kinds of displays, e.g. monitors, tablets, phones.
  • Deploying the application to a production environment so that it can be used by (possibly) a large number of end users.

For certain classes of systems, such as information systems, these secondary issues are frequently implemented over and over again, wasting precious development time and increasing the likelihood on errors.

With Mendix, most of these secondary concerns have been addressed in a reasonable manner, and primary functionality can be implemented by modeling and writing none to just a tiny amount of code, providing an incredible boost in developer productivity.

In addition to productivity, another objective of the Mendix platform is to reach a broader audience than just developers, such as business consultants.

A simple development scenario


What does the development process of a very tiny Mendix application look like? Typically, an application development process starts by creating a data model that specifies what kind of data needs to be stored (either in a database or in memory).


As shown in the picture above, in Mendix, a data model is created by drawing an ER (Entity-Relationship) diagram. For example, the example above defines a 'Contact' entity representing a contact person (with properties: first name and last name) and a 'Message' entity representing chat messages. A contact can send zero or more messages, as denoted by the 'Message_Contact' relationship.

After defining a data model, you may want to develop screens containing user interface components allowing users to inspect and change the data. In Mendix, these screens can be generated with just a few simple mouse clicks: creating a blank page, dragging and dropping a data grid to the page, and dragging and dropping the 'Contact' entity to the data grid:


In addition to an overview page displaying a collection of data items, we also want a page allowing us to change a record or to create a new record. This screen can be created by right clicking on the 'New' button and picking the 'Generate page...' option. The result is a page allowing us to edit an individual contact record:


Finally, to make the screen available to end users, we must add it to the navigation layout by creating a button that redirects the user to the overview page.

The result


With just a few simple clicks, we have already constructed a very small, but working application. For example, when clicking on the 'Run locally...' button in the IDE, we can start an instance of our test app and run it in a web browser.

The overview page looks as follows:


As may be observed by looking at the image above, the screen offers all functionality needed to work with a collection of records: navigation buttons, a search function, pagination, and edit functionality.

By clicking on an item in the list or clicking on the 'Edit' button, we can navigate to a page allowing us to edit the properties of a record:


After changing any of the attributes and clicking on the 'Save' button, the changes will be committed to the database.

Implementing all this functionality did not require me to write a single line of code.

Programming additional functionality


In addition to defining the domain model and generating views for all data elements, we may also want to program additional functionality, for example, a method that counts the amount of contacts having a first name that starts with an 'S'.

In Mendix, server-side functionality can be programmed by constructing microflows. Microflows use a graphical notation based on the Business Process Model and Notation (BPMN), a standardized graphical notation.


The above microflow retrieves all contacts from the database, sets an initial counter variable to 0 and iterates over all contacts. When a first name that starts with an 'S' has been encountered, the counter will be increased. Finally, it will return the counter value to the caller.

Microflows can be used to program many kinds of aspects in an application -- for example, they can be launched on startup, attached to buttons as event handlers, or attached as pre- or post commit hooks to data entities.

Other features


Besides the features described in this blog post, the Mendix platform has many additional features to offer, such as access control, internationalization, custom page layouts and collaboration tools (with a Subversion-based team server, and sprintr application managing the development workflow).

Furthermore, the deployment process of a Mendix application is completely automated in the Mendix cloud -- with just a few simple mouse clicks, your application becomes publicly available, without having to worry about managing a database or application server.

Finally, the platform is extensible -- you can implement custom actions in Java (server-side) and your own widgets in JavaScript (client-side) and there is an app store allowing you to download all kinds of pre-built third-party extensions.

Availability


You can sign up for a free Mendix account to start experimenting.

Tuesday, May 22, 2018

A new challenge

It has been quiet on my blog for a while, but don't worry, I'm still alive. In my absence, many things have happened. Most importantly, I have decided to embark on a new challenge.

Almost two months ago, I joined Mendix, a Rotterdam-based company (with its headquarters located in Boston) that ships a low-code application development platform. I will be working on their deployment infrastructure and I will be thinking about software modularity.

Some reflection


I vividly remember the days when I completed my PhD and left academia -- I still had to wait a couple of months for the defence ceremony, mainly because of the availability of the committee members.

Although I was quite happy with some of the work I had done during my research, I also wanted to leave my ivory tower and be more connected to the "real world" and my "research audience": developers. I joined a small startup company named Conference Compass (located in the YES!Delft incubator centre) that consisted of fewer than 10 people around the time I joined.

They had been looking into a setting up a product-line for their mobile conference apps, which sounded like an interesting challenge.

In the years that I was employed at Conference Compass, quite a few things happened. Most notably, the size of the company, the product and the service portfolio have grown considerably. Aside from these developments, I am particularly proud that I made quite a number of impacting open source contributions as part of my daily work.

The app building infrastructure


The biggest contribution I made by far is the mobile app building infrastructure. Most of its components have been part of Nixpkgs (the ecosystem of packages that can be deployed with the Nix package manager) for several years, such as:


To carry out all the builds on a large and timely scale, I installed a Hydra cluster: a Nix-based continuous integration service. I also developed an NPM module and command-line tool that we could use to remotely control a Hydra server from our custom built applications.

Chat functionality


Another interesting development area was enriching the app's product line with chat functionality built around the XMPP protocol/ejabberd service. I have ported the Simple XMPP library from the Node.js package ecosystem to Titanium by using a zero-forking strategy and I made a simple test application that somewhat resembles the Pidgin chat application.

I also ran into a number of practical issues while trying to keep the architecture of the test app clean. I did an in-depth study on the MVC paradigm and wrote a blog post about my findings.

Node.js


I also learned quite a lot about Node.js and its underlying concepts, such as asynchronous programming. Before joining Conference Compass, my JavaScript knowledge was limited to browser usage only, and I was not using JavaScript on a very extensive scale.

My learning experiences resulted in the following blog posts elaborating about various kinds of related concepts:


As of today, some of these blog posts are still in my all-time top 10 of most frequently read blog posts.

As a result of having to work with Node.js and being involved with the Nix project, I became the maintainer of node2nix, a tool that can generate Nix expressions from NPM package configurations after the maintainer of npm2nix decided to hand over the project.

Building a service deployment platform


In the first two years of my employment, my chief responsibility was the app building infrastructure. Another thing I am particularly proud of is the deployment infrastructure for the service platform that I built from scratch, that grew from just a single virtual machine hosted in the Amazon EC2 cloud to a platform managing the data and configuration services for 100+ apps per year, with some events attracting tens of thousands of app users.

I used variety of solutions, such as various Amazon web services, e.g. EC2, Route 53, S3. Most importantly, I used NixOps for infrastructure deployment and Disnix (the tool I created as part of my research) for service deployment.

Although Disnix already supported all the features I needed before I actually started using it at Conference Compass, my company experiences helped to substantially improve Disnix from a usability perspective -- in academia, Disnix was mostly used to validate my research objectives. To make it suitable for using it in a company with non-specialized deployment people, you need iron out many additional issues. I added more helpful error messages, assistance in recovering from errors and additional utilities to make diagnosing problems and carrying out maintenance tasks more convenient.

At the end of 2015, after using Disnix for almost one year in production, I gave a talk about Disnix's deployment concepts at NixCon 2015, the first edition of a conference fully centered around Nix and its related technologies.

Conclusion


I am grateful to my previous employer: Conference Compass who gave me the opportunity to do all the things described in this blog post.

At Mendix, there will be many new interesting challenges for me -- I will be working with different kinds of technologies, a new platform and new people. Stay tuned, for more information...

Sunday, February 25, 2018

A more realistic public Disnix example

It has almost been ten years ago when I started developing Disnix -- February 2008 marked the start of my master's thesis internship at Philips Research that resulted in the first prototype version.

Originally, Disnix was specifically developed for one use case only -- a medical service-oriented system called the "Service Development Support System" (SDS2) that can be used for asset tracking and utilisation analysis for medical devices in a hospital environment. More information about this case study can be found in my master's thesis, some of my research papers and my PhD thesis (all of them can be found on my publications page).

Many developments have happened since the realization of the first prototype -- its feature set has been extended considerably, its architecture has been overhauled several times and the code has evolved significantly. Most notably, I have been maintaining a production system for over three years with it.

In all these years, there is always one recurring question that I regularly receive from various kinds of people:

Why should I use Disnix and why would it be useful?

The answer is that Disnix becomes useful when you have a system that can be decomposed into distributable services, such as web services, RESTful services, web applications or processes.

In addition to the fact that Disnix automates its deployment and offers a number of powerful quality properties (e.g. non-destructive upgrades for the static parts of a system), it also helps componentized systems in reaching their full potential -- for example, when services can be built, deployed, and managed individually you can scale a system up and down (e.g. by distributing services to dedicated machines or consolidating all services on a single machine) and you can anticipate more flexibly to events (e.g. by redeploying services when we encounter a crashing machine).

Although the answer may sound simple, service-oriented systems are complicated -- besides facing all kinds of deployment complexities, properly dividing a system into distributable components is also quite challenging. For all the systems I have seen in the last decade, the requirements and their modularization strategies were all quite different from each other. I have also seen a number of systems for which decomposing into services did not work and unnecessary complexities were introduced.

Moreover, it is hard to find representative public examples that people can use as a reference. I was fortunate that I had access to an industrial case study during my research. Nonetheless, I was suffering from many difficulties because of the lack of any meaningful public case studies. As a countermeasure, I developed a collection of example cases in addition to SDS2, but because of their over-simplicity, proving my point often remained hard.

Roughly half a year ago, I have released most parts of my ancient web framework that I used to actively develop before I started doing research in software deployment and I created a couple of example applications for it.


Although my web framework development predates my deployment research, I was already using it to implement information systems that followed some modularity principles that are beneficial when using Disnix as a deployment system.

Recently, I have extended my web framework's example applications repository (providing a homework assistant, CMS, photo gallery and literature survey assistant) to become another public Disnix example case following the same modularity principles I used for the information systems I used to implement at that time.

Creating a componentized web information system


As mentioned earlier in this blog post, I have already implemented a (fairly simple) componentized web information system before I started working on Disnix using my ancient custom made web framework. The "componentization process" (a term that I had neither learned about yet nor something I was consciously implementing at that time) was partially driven by evolution and partially by non-functional requirements.

Originally, the system started out as just one single web application for one specific purpose and consisted of only two components -- a MySQL database responsible for storing the data and web front-end implemented in PHP, which is quite a common separation pattern for PHP applications.

Later, I was asked to implement another PHP application with similar functionality. Initially, I wrote the application from scratch without any reuse in mind, but at some point I made two important decisions:

  • I decided to keep the databases of each applications separate as opposed to integrating all the tables into one single database. My main motivating factor was that I wanted to prevent another developer's wrong decisions from messing up the other application. Moreover, I realized that for the data that was specific to the application domain that other systems did not have to know about it.
  • In addition to domain specific data, I noticed that both databases also stored the same kind of data, namely: user accounts -- both systems had a user account system to allow users to change the data. This also did not motivate me to integrate both databases into one database. Instead, I created a separate user database and authentication system (as a library API) that was shared among both applications.

After completing the two web applications, I had to implement more functionality. I decided to keep all of these new features for these new problem domains in separate applications with separate databases. The only thing they had in common was a shared user authentication system.

At some point I ended up having many sub applications. As a result, I needed a portal application that redirected users to these sub applications. Essentially, what I implemented became a system of systems.

Deployment with Disnix


The "architectural decisions" that I described earlier resulted in a system composed of several kinds of components:

  • Domain-specific web applications exposing functionality that logically belongs together.
  • Domain-specific databases storing tables that are strongly correlated.
  • A shared user database.
  • A portal application redirecting users to the domain-specific web applications.

The above listed components can be distributed over multiple machines in a network, because they connect to each other through network links (e.g. connecting to a MySQL database can be done with a TCP connection and connecting to a domain specific web application can be done through HTTP). As a result, they can also be modeled as services that can be deployed with Disnix.

To replicate the same patterns for demo purposes, I integrated my framework's example applications into a similar system of sub systems. We can deploy the corresponding example system to one single target machine with Disnix, by running:

$ disnixos-env -s services.nix \
  -n network-single.nix \
  -d distribution-single.nix --use-nixops

The entire system gets deployed to a single machine because of the distribution model (distribution.nix) that maps all services to one target machine:

{infrastructure}:

{
  usersdb = [ infrastructure.test1 ];
  cmsdb = [ infrastructure.test1 ];
  cmsgallerydb = [ infrastructure.test1 ];
  homeworkdb = [ infrastructure.test1 ];
  literaturedb = [ infrastructure.test1 ];
  portaldb = [ infrastructure.test1 ];

  cms = [ infrastructure.test1 ];
  cmsgallery = [ infrastructure.test1 ];
  homework = [ infrastructure.test1 ];
  literature = [ infrastructure.test1 ];
  users = [ infrastructure.test1 ];
  portal = [ infrastructure.test1 ];
}

The resulting deployment architecture looks as follows:


The above visualization of the deployment architecture shows the following aspects:

  • The surrounding light grey colored box denotes a target machine. In this particular example, we only have one single target machine where services are deployed to.
  • The dark grey colored boxes correspond to container environments. For our example system, we have two of them: mysql-database corresponding to a MySQL DBMS server and apache-webapplication corresponding to an Apache HTTP server.
  • The ovals denote services corresponding to MySQL databases and web applications.
  • The arrows denote inter-dependency links that correspond to network connections. As explained in my previous blog post, solid arrows are dependencies with a strict ordering requirement while dashed arrows are dependencies without an ordering requirement.

Some people may argue that it is not really beneficial to deploy such a system with Disnix -- with NixOps you can define a machine configuration having a MySQL DBMS server and an Apache HTTP server with the corresponding databases and web application components. With Disnix, you must first ensure that the machines, the MySQL and Apache HTTP servers are configured by other means first (that could for example be done with NixOps), and then you have to deploy the system's components with Disnix.

In a single machine deployment scenario, it may indeed not be that beneficial. However, what you get in addition to automated deployment is also more flexibility. Since Disnix manages the services directly, as opposed to entire machine configurations as a whole, you can anticipate better in case of events by redeploying the system.

For example, when the amount of visitors keeps growing, you may run into the problem that a single server can no longer handle all the traffic. In such cases, you can easily add another machine to the network and adjust the distribution model to move (for example) the databases to another machine:

{infrastructure}:

{
  usersdb = [ infrastructure.test2 ];
  cmsdb = [ infrastructure.test2 ];
  cmsgallerydb = [ infrastructure.test2 ];
  homeworkdb = [ infrastructure.test2 ];
  literaturedb = [ infrastructure.test2 ];
  portaldb = [ infrastructure.test2 ];

  cms = [ infrastructure.test1 ];
  cmsgallery = [ infrastructure.test1 ];
  homework = [ infrastructure.test1 ];
  literature = [ infrastructure.test1 ];
  users = [ infrastructure.test1 ];
  portal = [ infrastructure.test1 ];
}

By redeploying the system, we can take advantage of the additional system resources that the new machine provides:

$ disnixos-env -s services.nix \
  -n network-separate.nix \
  -d distribution-separate.nix --use-nixops

resulting in the following deployment architecture:


Likewise, there are countless of other deployment strategies possible to meet all kinds of non-functional requirements. For example, we can also distribute bundles of domain specific application and database pairs over two machines:

$ disnixos-env -s services.nix \
  -n network-bundles.nix \
  -d distribution-bundles.nix --use-nixops

resulting in the following deployment architecture:


This approach is even more scalable than simply offloading the databases to another server.

In addition to scalability, there are countless of other reasons to pick a certain distribution strategy. You could also, for example, distribute redundant instances of databases and applications as a failover to improve availability or improve security by deploying the databases with privacy sensitive data to a machine with restrictive network access.

State management


When updating the deployment of systems with Disnix (such as moving a database from one machine to another), there may be a recurring limitation that you could run frequently into -- like Nix, Disnix only manages the static parts of the system, but not any state. This means that a service's deployment can be reproduced elsewhere, but data, such as the content of a database is not migrated.

For example, the sub system of example applications stores two kinds of data -- records in the MySQL database and files, such as images uploaded in the photo gallery or PDF files uploaded to the literature application. When moving these applications around the data is not migrated.

As a possible solution, Disnix also provides simple state management facilities. When enabled, Disnix will take snapshots of the databases and filesets on the source machines, transfers the snapshots to the target machines, and finally restores the snapshots when moving a service one machine to another in the distribution model.

State management can be enabled globally by passing the --deploy-state parameter to (disnix-env or annotating the services with deployState = true; in the services model):

$ disnixos-env -s services.nix \
  -n network-bundles.nix \
  -d distribution-bundles.nix --use-nixops --deploy-state

We can also directly use the state management system, e.g. for backup purposes. When running the following command:

$ disnix-snapshot

Disnix takes snapshots of all databases and web application state (e.g. the images in the photo gallery and uploaded PDF files) and transfers them to the coordinator machine. With the dysnomia-snapshots tool we can inspect the snapshot store:

$ dysnomia-snapshots --query-all
apache-webapplication/cms/1f9ed847885d2b3e3c67c51231122d958751eb5e2443c281e02e1d7108a505a3
apache-webapplication/cmsgallery/28d17a6941cb195a92e748aae737ccf524747477c6943436b734891d0f36fd53
apache-webapplication/literature/ed5ec4f8b9b4fcdb8b740ad1fa7ecb40b10dece03548f1d6e09a6a82c804131b
apache-webapplication/portal/5bbea499f8f8a4f708bb873ad683dbf088afa4c553f90ab287a9249a7ef02651
mysql-database/cmsdb/aa75992f780991c39a0969dcac5f69b04685c4fa764937476b816e938d6972ba
mysql-database/cmsgallerydb/31ebdaba658ca376123ff6a91a3e275731b383346a07840b1acaa1e44d921b65
mysql-database/homeworkdb/f0fda91545af0cb300afd84592d4914dcd48257053401e232438e34d83af828d
mysql-database/literaturedb/cb881c2200a5f1562f0b66f1394d0902bbb8e2361068fe096faac3bc31f76b5d
mysql-database/portaldb/5d8a5cb952f40ce76f93eb939d0b37eab33736d7b1e1426038322f8a572034ee
mysql-database/usersdb/64d11fc7f8969da5da318276a666f2e00e0a020ba619a1d82ed9b84a7f1c2ca6

and with some shell scripting, the actual contents of the snapshot store:

$ find $(dysnomia-snapshots --resolve $(dysnomia-snapshots --query-all)) -type f
/home/sander/state/snapshots/apache-webapplication/cms/1f9ed847885d2b3e3c67c51231122d958751eb5e2443c281e02e1d7108a505a3/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/cmsgallery/28d17a6941cb195a92e748aae737ccf524747477c6943436b734891d0f36fd53/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/literature/ed5ec4f8b9b4fcdb8b740ad1fa7ecb40b10dece03548f1d6e09a6a82c804131b/state.tar.xz
/home/sander/state/snapshots/apache-webapplication/portal/5bbea499f8f8a4f708bb873ad683dbf088afa4c553f90ab287a9249a7ef02651/state.tar.xz
/home/sander/state/snapshots/mysql-database/cmsdb/aa75992f780991c39a0969dcac5f69b04685c4fa764937476b816e938d6972ba/dump.sql.xz
/home/sander/state/snapshots/mysql-database/cmsgallerydb/31ebdaba658ca376123ff6a91a3e275731b383346a07840b1acaa1e44d921b65/dump.sql.xz
/home/sander/state/snapshots/mysql-database/homeworkdb/f0fda91545af0cb300afd84592d4914dcd48257053401e232438e34d83af828d/dump.sql.xz
/home/sander/state/snapshots/mysql-database/literaturedb/cb881c2200a5f1562f0b66f1394d0902bbb8e2361068fe096faac3bc31f76b5d/dump.sql.xz
/home/sander/state/snapshots/mysql-database/portaldb/5d8a5cb952f40ce76f93eb939d0b37eab33736d7b1e1426038322f8a572034ee/dump.sql.xz
/home/sander/state/snapshots/mysql-database/usersdb/64d11fc7f8969da5da318276a666f2e00e0a020ba619a1d82ed9b84a7f1c2ca6/dump.sql.xz

The above output shows that for each MySQL database, we store a compressed SQL dump of the database and for each stateful web application, a compressed tarball of state files.

Conclusion


In this blog post, I have described a more realistic public Disnix example that is inspired by my web framework developments a long time ago. Aside from automating a system's deployment, the purpose of this blog post is to describe how a system that can be decomposed into distributable services that can be deployed with Disnix. Implementing such a system is all but trivial and driven by various kinds of design decisions.

Availability


The example web application system can be obtained from my GitHub page. The Disnix deployment expressions can be found in the deployment/ sub folder.

In addition, I have created a Dysnomia module named: fileset that can capture the state files of web applications in a compressed tarball.

After the recent developments the Disnix toolset has reached a new stable point. As a result, I have decided to release Disnix 0.8. Consult the Disnix homepage for more information!