Mendix is a low-code application development platform. Low-code application development offers all kinds of benefits over traditional development approaches involving code, such as a boost in productivity. For some applications, it is possible to develop up to ten times faster compared to traditional coding approaches and frameworks.
However, despite being different from a development perspective, there is one particular activity that all application development approaches have in common – at some point, you need to deploy your application to an environment (e.g. test, acceptance, or production) to make it available to end users.
For users of the Mendix cloud portal, deployment is automated in a convenient way – with just a few simple mouse clicks, you can make your application available to all potential users in the world.
However, managing on-premise deployments or the cloud infrastructure itself is all but a trivial job – for example, there are many complex dependencies that need to be deployed to run a Mendix application, upgrading may introduce unnecessary downtime and break a system, and the infrastructure needs to be scalable so that it can manage thousands of applications.
Fortunately, there are many automated deployment solutions that come to our aid, such as Kubernetes. Although many of them are useful, none of these solutions are perfect -- they all have their strengths and weaknesses. As a result, there are still complexities we need to solve ourselves and incidents that require fixing.
At Mendix R&D, everybody is encouraged to freely experiment two days a month (the so-called “crafting days”). One of my crafting day projects is to experiment with deployment tools from a different and unorthodox solution spectrum: The Nix project. The goal is to fully automate the deployment of a Mendix application from source – the input is a Mendix project created with the modeler and the end-result is a system running the application.
The Nix Project
The Nix project provides a family of tools that solve configuration management problems in a unique way. Some tools that are part of the Nix project are:
- The Nix package manager
- The NixOS Linux distribution
- NixOps: A NixOS-based cloud deployment tool
- Hydra: The Nix-based continuous integration service
- Disnix: A Nix-based service deployment tool
The basis of all tools in the Nix project is the Nix package manager. Nix is quite different from almost any conventional package manager (such as RPM, APT, or Homebrew) because it borrows concepts from purely functional programming languages, such as Haskell.
The Nix Package Manager
The Nix package manager implements a purely functional deployment model. In Nix, deploying a package reliably is the same thing as invoking a pure function, without any side effects. To make this possible, Nix provides a purely functional domain-specific language called the Nix expression language.
{ stdenv, fetchurl, acl }:
stdenv.mkDerivation {
name = "gnutar-1.30";
src = fetchurl {
url = http://ftp.gnu.org/tar/tar-1.30.tar.xz;
sha256 = "1lyjyk8z8hdddsxw0ikchrsfg3i0x3fsh7l63a8jgaz1n7dr5gzi";
};
buildCommand = ''
tar xfv $src
cd tar-1.30
./configure --prefix=$out --with-acl=${acl}
make
make install
'';
}
The above code fragment is an example of a Nix expression that describes how to build GNU tar from source code and its build-time dependencies:
- The entire expression is a function definition. The first line corresponds to a function header in which every argument is a build-time dependency:
- stdenv is an environment providing standard UNIX utilities, such as cat, ls and make.
- fetchurl is a function that is used to download files from an external location.
- acl is a library dependency of GNU tar that provides access control list support.
- In the body of the function, we invoke the mkDerivation {} function that composes so-called “pure build environments” in which arbitrary build commands can be executed.
- As function arguments to mkDerivation, we specify the name of the package (name), how the source can be obtained (src) and the shell commands (buildCommand) that need to be executed to build the package.
The above expression is a function definition describing how to build something from sources, but the expression does not specify which version or variants of the sources that are supposed to be used. Function definitions alone are not useful. Instead, functions must be invoked with all the required function arguments. In Nix, they need to correspond to the versions or variants of the build-time dependencies that we want to use.
Packages are composed in a second Nix expression that has the following structure:
rec {
stdenv = import ...
fetchurl = import ...
acl = import ../pkgs/os-specific/linux/acl {
inherit stdenv fetchurl …;
};
gnutar = import ../pkgs/tools/archivers/gnutar {
inherit stdenv fetchurl acl;
};
...
}
The above partial Nix expression is an attribute set (a language construct conceptually similar to objects in JSON) in which every key represents a package name and every value refers to a function invocation that builds the package from source code. The GNU tar expression (shown in the previous code fragment) is imported in this expression and invoked with function arguments referring to the keys in the same attribute set, such as stdenv, fetchurl, and acl.
In addition to GNU tar, all build-time dependencies are composed in the same Nix expression. These dependencies are also constructed by following the same convention – invoking a function that builds the package from source code and its build-time dependencies.
In a Nix build environment, you can execute (almost) any build tool. In the GNU tar example, we run a standard GNU Autotools build procedure, but it is also possible to run Apache Ant (for Java software), Python setup tools, Perl’s MakeMaker or CMake and many other tools.
The only catch is that Nix imposes restrictions on what the tools are permitted to do to provide better guarantees that builds are pure, such as:
- Every package is stored in an isolated directory, not in global directories, such as /lib, /bin or C:\Windows\System32
- Files are made read-only after build completion
- Timestamps are reset to 1 second after the epoch
- Search environment variables are cleared and configured explicitly, e.g. PATH
- Private temp folders and designated output directories
- Network access is restricted (except when an output hash is given)
- Running builds as unprivileged users
- Chroot environments, namespaces, bind-mounting dependency packages
The most important restriction is the first – in Nix, all packages are stored in a so-called Nix store, in which every package is prefixed by a cryptographic hash code derived from all build inputs, such as: /nix/store/fjh974kzdcab7yp0ibmwwymmgbi6cg59-gnutar-1.30. Because hash prefixes are unique, no package shares the same name and as a result, we can safely store multiple versions and variants of the same package alongside each other in the store.
The result of complementing build tools with these restrictions is that when you build a package with Nix with certain build-time dependencies and you perform the build with the same inputs on another machine, the result will be the exact same (nearly bit-identical) build.
Purity offers many kinds of benefits, such as:
- Strong dependency completeness guarantees
- Strong reproducibility guarantees
- Build only the packages and dependencies that you need
- Packages that don’t depend on each other can be safely built in parallel
- Ability to download substitutes from a remote machine (e.g. build server) if the hash prefix is identical
- Ability to delegate builds to remote machines and be sure that the result is identical if it were built locally
By taking the composition expression (shown earlier) and running nix-build, we can build GNU tar, including all of its build-time dependencies:
$ nix-build all-packages.nix -A gnutar /nix/store/fjh974kzdcab7yp0ibmwwymmgbi6cg59-gnutar-1.30
The result of the Nix-build instruction is a Nix store path that contains a hash code that has been derived from all build inputs.
Building Mendix Deployment Archives (MDAs) with Nix
As explained earlier, in Nix build environments any kind of build tool can be used albeit with purity restrictions.
For Mendix applications, there is also an important artifact that needs to be produced in the deployment lifecycle – the Mendix Deployment Archive (MDA) that captures all relevant files that an application needs to run in production.
MDA files can be produced by running the MxBuild tool. We can also package MxBuild and the Mendix runtime as Nix packages and write our own Nix function abstraction that builds MDA files from Mendix projects:
{stdenv, mxbuild, jdk, nodejs}:
{name, mendixVersion, looseVersionCheck ? false, ...}@args:
let mxbuildPkg = mxbuild."${mendixVersion}";
in
stdenv.mkDerivation ({
buildInputs = [ mxbuildPkg nodejs ];
installPhase = ''
mkdir -p $out
mxbuild --target=package --output=$out/${name}.mda \
--java-home ${jdk} --java-exe-path ${jdk}/bin/java \
${stdenv.lib.optionalString looseVersionCheck "--loose-version-check"} \
"$(echo *.mpr)"
'';
} // args)
The above function returns another function taking Mendix-specific parameters (e.g. the name of the project, Mendix version), invokes MxBuild, and stores the resulting MDA file in the Nix store.
By using the function abstraction and a Mendix project created by the modeler, we can build the Mendix project by writing the following Nix expression:
{packageMendixApp}:
packageMendixApp {
name = "conferenceschedule";
src = /home/sander/ConferenceSchedule-main;
mendixVersion = "7.13.1";
}
The above expression specifies that we want to build a project named: conferenceschedule, we want to use the Mendix project that is stored in the directory: /home/sander/ConferenceSchedule-main and we want to use Mendix version 7.13.1.
Using NixOS: A Nix-Based Linux Distribution
One of the common objectives that all tools in the Nix project have in common is declarative deployment, meaning that you can express the structure of your system, and the tools infer all the activities that need to be carried out to deploy it.
As a Mendix developer, generating an MDA archive is not entirely what we want – what we really want is a system running a Mendix application. To accomplish this, additional deployment activities need to be carried out beyond producing an MDA file.
NixOS is a Linux distribution that extends Nix’s deployment features to complete systems. In addition to the fact that the Nix package manager is being used to deploy all packages (including the Linux kernel) and configuration files, it also deploys entire machine configurations from a single declarative specification:
{pkgs, ...}:
{
boot.loader.grub.device = "/dev/sda";
fileSystems."/".device = "/dev/sda1";
services = {
openssh.enable = true;
xserver = {
enable = true;
displayManager.sddm.enable = true;
desktopManager.plasma5.enable = true;
};
};
environment.systemPackages = [
pkgs.firefox
];
}
The above code fragment is an example of a NixOS configuration file that captures the following properties:
- The GRUB bootloader should be installed on the Master Boot Record of the first harddrive (/dev/sda)
- The first partition of the first harddrive (/dev/sda1) should be mounted as a root partition
- We want to run OpenSSH and the X Window System as system services
- We configure the X Window Server to use SDDM as a login manager and the KDE Plasma Desktop as desktop manager.
- We want to install Mozilla Firefox as an end-user package.
By running a single command-line instruction, we can deploy an entire system configuration with the Nix package manager:
$ nixos-rebuild switch
The result is a running system implementing the configuration described above.
Creating a NixOS Module for Mendix App Containers
To automate the remaining Mendix deployment activities (that need to be carried out after composing an MDA file), we can create a systemd job (the service manager that NixOS uses) that unpacks the MDA file into a writable directory, creates additional state directories for storing temp files, and configure the runtime by communicating over the admin interface to start the embedded Jetty HTTP service, configure the database and start the app.
Composing a systemd job can be done by adding a systemd configuration setting to a NixOS configuration. The following partial Nix expression shows the overall structure of a systemd job for a Mendix app container:
{pkgs, ...}:
{
systemd.services.mendixappcontainer =
let
mendixPkgs = import ../nixpkgs-mendix/top-level/all-packages.nix { inherit pkgs; };
appContainerConfigJSON = pkgs.writeTextFile { ... };
configJSON = pkgs.writeTextFile {
name = "config.json";
text = builtins.toJSON {
DatabaseType = "HSQLDB";
DatabaseName = "myappdb";
DTAPMode = "D";
};
};
runScripts = mendixPkgs.runMendixApp {
app = import ../conferenceschedule.nix { inherit (mendixPkgs) packageMendixApp; };
};
in {
enable = true;
description = "My Mendix App";
wantedBy = [ "multi-user.target" ];
environment = {
M2EE_ADMIN_PASS = "secret";
M2EE_ADMIN_PORT = "9000";
MENDIX_STATE_DIR = "/home/mendix";
};
serviceConfig = {
ExecStartPre = "${runScripts}/bin/undeploy-app";
ExecStart = "${runScripts}/bin/start-appcontainer";
ExecStartPost = "${runScripts}/bin/configure-appcontainer ${appContainerConfigJSON} ${configJSON}";
};
};
}
The above systemd job declaration does the following:
- It generates JSON configuration files with app container and database settings
- It composes an environment with environment variables configuring the admin interface
- It launches scripts: one script before startup that cleans the old state, a start script that starts the app container and a script that runs after startup that configures the app container settings, such as the database
Writing a systemd job as a Nix expression is quite cumbersome and a bit impractical when it is desired to compose NixOS configurations that should run Mendix applications. Fortunately, we can hide all these implementation details behind a more convenient interface by wrapping all Mendix app container properties in a NixOS module.
By importing this NixOS module in a NixOS configuration, we can more concisely express the properties of a system running a Mendix app container:
{pkgs, ...}:
{
require = [ ../nixpkgs-mendix/nixos/modules/mendixappcontainer.nix ];
services = {
openssh.enable = true;
mendixAppContainer = {
enable = true;
adminPassword = "secret";
databaseType = "HSQLDB";
databaseName = "myappdb";
DTAPMode = "D";
app = import ../../conferenceschedule.nix {
inherit pkgs;
inherit (pkgs.stdenv) system;
};
};
};
networking.firewall.allowedTCPPorts = [ 8080 ];
}
The above code fragment is another NixOS configuration that imports the Mendix app container NixOS module. It defines a Mendix app container system service that connects to an in-memory HSQL database, runs the app in development mode, and deploys the MDA file that is the result of building one of our test projects, by invoking the Nix build abstraction function that builds MDAs.
By running a single command-line instruction, we can deploy a machine configuration running our Mendix application:
$ nixos-rebuild switch
After the deployment has succeeded, we should able to open a web browser and test our app.
In production scenarios, only deploying an app container is not enough to make an application reliably available to end users. We must also deploy a more robust database service, such as PostgreSQL, and use a reverse proxy, such as nginx, to more efficiently serve static files and cache common requests to improve the performance of the application.
It is also possible to extend the NixOS configuration with a PostgreSQL and nginx system service and use the NixOS module system to refer to the relevant properties of a Mendix app container.
Conclusion
This blog post covers tools from the Nix project implementing deployment concepts inspired by purely functional programming languages and declarative programming. These tools offer a number of unique advantages over more traditional deployment tools. Furthermore, we have demonstrated that Mendix application deployments could fit into such a deployment model.
Availability
The Nix build abstraction function for Mendix projects and the NixOS module for running app containers can be obtained from the nixpkgs-mendix repository on GitHub. The functionality should be considered experimental – it is not yet recommended for production usage.
The Nix package manager and NixOS Linux distribution can be obtained from the NixOS website.
This blog post originally appeared on: https://www.mendix.com/blog/automating-mendix-application-deployments-with-nix/)
No comments:
Post a Comment