Although Docker is primarily used for managing single root application process containers, multi-process containers can sometimes be useful to deploy systems that consist of multiple, tightly coupled, processes.
The Docker manual has a section that describes how to construct images for multi-process containers, but IMO the configuration process is a bit tedious and cumbersome.
To make this process more convenient, I have built a wrapper function: createMultiProcessImage around the dockerTools.buildImage function (provided by Nixpkgs) that does the following:
- It constructs an image that runs a Linux and Docker compatible process manager as an entry point. Currently, it supports supervisord, sysvinit, disnix and s6-rc.
- The Nix process management framework is used to build a configuration for a system that consists of multiple processes, that will be managed by any of the supported process managers.
Although the framework makes the construction of multi-process images convenient, a big drawback of multi-process Docker containers is upgrading them -- for example, for Debian-based containers you can imperatively upgrade packages by connecting to the container:
$ docker exec -it mycontainer /bin/bash
and upgrade the desired packages, such as file:
$ apt install file
The upgrade instruction above is not reproducible -- apt may install file version 5.38 today, and 5.39 tomorrow.
To cope with these kinds of side-effects, Docker works with images that snapshot the outcomes of all the installation steps. Constructing a container from the same image will always provide the same versions of all dependencies.
As a consequence, to perform a reproducible container upgrade, it is required to construct a new image, discard the container and reconstruct the container from the new image version, causing the system as a whole to be terminated, including the processes that have not changed.
For a while, I have been thinking about this limitation and developed a solution that makes it possible to upgrade multi-process containers without stopping and discarding them. The only exception is the process manager.
To make deployments reproducible, it combines the reproducibility properties of Docker and Nix.
In this blog post, I will describe how this solution works and how it can be used.
Creating a function for building mutable Docker images
As explained in an earlier blog post, that compares the deployment properties of Nix and Docker, both solutions support reproducible deployment, albeit for different application domains.
Moreover, their reproducibility properties are built around different concepts:
- Docker containers are reproducible, because they are constructed from images that consist of immutable layers identified by hash codes derived from their contents.
- Nix package builds are reproducible, because they are stored in isolation in a Nix store and made immutable (the files' permissions are set read-only). In the construction process of the packages, many side effects are mitigated.
As a result, when the hash code prefix of a package (derived from all build inputs) is the same, then the build output is also (nearly) bit-identical, regardless of the machine on which the package was built.
By taking these reproducibilty properties into account, we can create a reproducible deployment process for upgradable containers by using a specific separation of responsibilities.
Deploying the base system
For the deployment of the base system that includes the process manager, we can stick ourselves to the traditional Docker deployment workflow based on images (the only unconventional aspect is that we use Nix to build a Docker image, instead of Dockerfiles).
The process manager that the image provides deploys its configuration from a dynamic configuration directory.
To support supervisord, we can invoke the following command as the container's entry point:
supervisord --nodaemon \ --configuration /etc/supervisor/supervisord.conf \ --logfile /var/log/supervisord.log \ --pidfile /var/run/supervisord.pid
The above command starts the supervisord service (in foreground mode), using the supervisord.conf configuration file stored in /etc/supervisord.
The supervisord.conf configuration file has the following structure:
[supervisord] [include] files=conf.d/*
The above configuration automatically loads all program definitions stored in the conf.d directory. This directory is writable and initially empty. It can be populated with configuration files generated by the Nix process management framework.
For the other process managers that the framework supports (sysvinit, disnix and s6-rc), we follow a similar strategy -- we configure the process manager in such a way that the configuration is loaded from a source that can be dynamically updated.
Deploying process instances
Deployment of the process instances is not done in the construction of the image, but by the Nix process management framework and the Nix package manager running in the container.
To allow a processes model deployment to refer to packages in the Nixpkgs collection and install binary substitutes, we must configure a Nix channel, such as the unstable Nixpkgs channel:
$ nix-channel --add https://nixos.org/channels/nixpkgs-unstable $ nix-channel --update
(As a sidenote: it is also possible to subscribe to a stable Nixpkgs channel or a specific Git revision of Nixpkgs).
The processes model (and relevant sub models, such as ids.nix that contains numeric ID assignments) are copied into the Docker image.
We can deploy the processes model for supervisord as follows:
$ nixproc-supervisord-switch
The above command will deploy the processes model in the NIXPROC_PROCESSES environment variable, which defaults to: /etc/nixproc/processes.nix:
- First, it builds supervisord configuration files from the processes model (this step also includes deploying all required packages and service configuration files)
- It creates symlinks for each configuration file belonging to a process instance in the writable conf.d directory
- It instructs supervisord to reload the configuration so that only obsolete processes get deactivated and new services activated, causing unchanged processes to remain untouched.
(For the other process managers, we have equivalent tools: nixproc-sysvinit-switch, nixproc-disnix-switch and nixproc-s6-rc-switch).
Initial deployment of the system
Because only the process manager is deployed as part of the image (with an initially empty configuration), the system is not yet usable when we start a container.
To solve this problem, we must perform an initial deployment of the system on first startup.
I used my lessons learned from the chainloading techniques in s6 (in the previous blog post) and developed hacky generated bootstrap script (/bin/bootstrap) that serves as the container's entry point:
cat > /bin/bootstrap <<EOF
#! ${pkgs.stdenv.shell} -e
# Configure Nix channels
nix-channel --add ${channelURL}
nix-channel --update
# Deploy the processes model (in a child process)
nixproc-${input.processManager}-switch &
# Overwrite the bootstrap script, so that it simply just
# starts the process manager the next time we start the
# container
cat > /bin/bootstrap <<EOR
#! ${pkgs.stdenv.shell} -e
exec ${cmd}
EOR
# Chain load the actual process manager
exec ${cmd}
EOF
chmod 755 /bin/bootstrap
The generated bootstrap script does the following:
- First, a Nix channel is configured and updated so that we can install packages from the Nixpkgs collection and obtain substitutes.
- The next step is deploying the processes model by running the nixproc-*-switch tool for a supported process manager. This process is started in the background (as a child process) -- we can use this trick to force the managing bash shell to load our desired process supervisor as soon as possible.
Ultimately, we want the process manager to become responsible for supervising any other process running in the container. - After the deployment process is started in the background, the bootstrap script is overridden by a bootstrap script that becomes our real entry point -- the process manager that we want to use, such as supervisord.
Overriding the bootstrap script makes sure that the next time we start the container, it will start instantly without attempting to deploy the system again. - Finally, the bootstrap script "execs" into the real process manager, becoming the new PID 1 process. When the deployment of the system is done (the nixproc-*-switch process that still runs in the background), the process manager becomes responsible for reaping it.
With the above script, the workflow of deploying an upgradable/mutable multi-process container is the same as deploying an ordinary container from a Docker image -- the only (minor) difference is that the first time that we start the container, it may take some time before the services become available, because the multi-process system needs to be deployed by Nix and the Nix process management framework.
A simple usage scenario
Similar to my previous blog posts about the Nix process management framework, I will use the trivial web application system to demonstrate how the functionality of the framework can be used.
The web application system consists of one or more webapp processes (with an embedded HTTP server) that only return static HTML pages displaying their identities.
An Nginx reverse proxy forwards incoming requests to the appropriate webapp instance -- each webapp service can be reached by using its unique virtual host value.
To construct a mutable multi-process Docker image with Nix, we can write the following Nix expression (default.nix):
let
pkgs = import <nixpkgs> {};
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
createMutableMultiProcessImage = import "${nix-processmgmt}/nixproc/create-image-from-steps/create-mutable-multi-process-image-universal.nix" {
inherit pkgs;
};
in
createMutableMultiProcessImage {
name = "multiprocess";
tag = "test";
contents = [ pkgs.mc ];
exprFile = ./processes.nix;
idResourcesFile = ./idresources.nix;
idsFile = ./ids.nix;
processManager = "supervisord"; # sysvinit, disnix, s6-rc are also valid options
}
The above Nix expression invokes the createMutableMultiProcessImage function that constructs a Docker image that provides a base system with a process manager, and a bootstrap script that deploys the multi-process system:
- The name, tag, and contents parameters specify the image name, tag and the packages that need to be included in the image.
- The exprFile parameter refers to a processes model that captures the configurations of the process instances that need to be deployed.
- The idResources parameter refers to an ID resources model that specifies from which resource pools unique IDs need to be selected.
- The idsFile parameter refers to an IDs model that contains the unique ID assignments for each process instance. Unique IDs resemble TCP/UDP port assignments, user IDs (UIDs) and group IDs (GIDs).
- We can use the processManager parameter to select the process manager we want to use. In the above example it is supervisord, but other options are also possible.
We can use the following processes model (processes.nix) to deploy a small version of our example system:
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
ids = if builtins.pathExists ./ids.nix then (import ./ids.nix).ids else {};
sharedConstructors = import "${nix-processmgmt}/examples/services-agnostic/constructors/constructors.nix" {
inherit pkgs stateDir runtimeDir logDir cacheDir tmpDir forceDisableUserChange processManager ids;
};
constructors = import "${nix-processmgmt}/examples/webapps-agnostic/constructors/constructors.nix" {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager ids;
};
in
rec {
webapp = rec {
port = ids.webappPorts.webapp or 0;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
};
requiresUniqueIdsFor = [ "webappPorts" "uids" "gids" ];
};
nginx = rec {
port = ids.nginxPorts.nginx or 0;
pkg = sharedConstructors.nginxReverseProxyHostBased {
webapps = [ webapp ];
inherit port;
} {};
requiresUniqueIdsFor = [ "nginxPorts" "uids" "gids" ];
};
}
The above Nix expression configures two process instances, one webapp process that returns a static HTML page with its identity and an Nginx reverse proxy that forwards connections to it.
A notable difference between the expression shown above and the processes models of the same system shown in my previous blog posts, is that this expression does not contain any references to files on the local filesystem, with the exception of the ID assignments expression (ids.nix).
We obtain all required functionality from the Nix process management framework by invoking builtins.fetchGit. Eliminating local references is required to allow the processes model to be copied into the container and deployed from within the container.
We can build a Docker image as follows:
$ nix-build
load the image into Docker:
$ docker load -i result
and create and start a Docker container:
$ docker run -it --name webapps --network host multiprocess:test unpacking channels... warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring created 1 symlinks in user environment 2021-02-21 15:29:29,878 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2021-02-21 15:29:29,878 WARN No file matches via include "/etc/supervisor/conf.d/*" 2021-02-21 15:29:29,897 INFO RPC interface 'supervisor' initialized 2021-02-21 15:29:29,897 CRIT Server 'inet_http_server' running without any HTTP authentication checking 2021-02-21 15:29:29,898 INFO supervisord started with pid 1 these derivations will be built: /nix/store/011g52sj25k5k04zx9zdszdxfv6wy1dw-credentials.drv /nix/store/1i9g728k7lda0z3mn1d4bfw07v5gzkrv-credentials.drv /nix/store/fs8fwfhalmgxf8y1c47d0zzq4f89fz0g-nginx.conf.drv /nix/store/vxpm2m6444fcy9r2p06dmpw2zxlfw0v4-nginx-foregroundproxy.sh.drv /nix/store/4v3lxnpapf5f8297gdjz6kdra8g7k4sc-nginx.conf.drv /nix/store/mdldv8gwvcd5fkchncp90hmz3p9rcd99-builder.pl.drv /nix/store/r7qjyr8vr3kh1lydrnzx6nwh62spksx5-nginx.drv /nix/store/h69khss5dqvx4svsc39l363wilcf2jjm-webapp.drv /nix/store/kcqbrhkc5gva3r8r0fnqjcfhcw4w5il5-webapp.conf.drv /nix/store/xfc1zbr92pyisf8lw35qybbn0g4f46sc-webapp.drv /nix/store/fjx5kndv24pia1yi2b7b2bznamfm8q0k-supervisord.d.drv these paths will be fetched (78.80 MiB download, 347.06 MiB unpacked): ...
As may be noticed by looking at the output, on first startup the Nix process management framework is invoked to deploy the system with Nix.
After the system has been deployed, we should be able to connect to the webapp process via the Nginx reverse proxy:
$ curl -H 'Host: webapp.local' http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Simple test webapp</title>
</head>
<body>
Simple test webapp listening on port: 5000
</body>
</html>
When it is desired to upgrade the system, we can change the system's configuration by connecting to the container instance:
$ docker exec -it webapps /bin/bash
In the container, we can edit the processes.nix configuration file:
$ mcedit /etc/nixproc/processes.nix
and make changes to the configuration of the system. For example, we can change the processes model to include a second webapp process:
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
ids = if builtins.pathExists ./ids.nix then (import ./ids.nix).ids else {};
sharedConstructors = import "${nix-processmgmt}/examples/services-agnostic/constructors/constructors.nix" {
inherit pkgs stateDir runtimeDir logDir cacheDir tmpDir forceDisableUserChange processManager ids;
};
constructors = import "${nix-processmgmt}/examples/webapps-agnostic/constructors/constructors.nix" {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager ids;
};
in
rec {
webapp = rec {
port = ids.webappPorts.webapp or 0;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
};
requiresUniqueIdsFor = [ "webappPorts" "uids" "gids" ];
};
webapp2 = rec {
port = ids.webappPorts.webapp2 or 0;
dnsName = "webapp2.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "2";
};
requiresUniqueIdsFor = [ "webappPorts" "uids" "gids" ];
};
nginx = rec {
port = ids.nginxPorts.nginx or 0;
pkg = sharedConstructors.nginxReverseProxyHostBased {
webapps = [ webapp webapp2 ];
inherit port;
} {};
requiresUniqueIdsFor = [ "nginxPorts" "uids" "gids" ];
};
}
In the above process model model, a new process instance named: webapp2 was added that listens on a unique port that can be reached with the webapp2.local virtual host value.
By running the following command, the system in the container gets upgraded:
$ nixproc-supervisord-switch
resulting in two webapp process instances running in the container:
$ supervisorctl nginx RUNNING pid 847, uptime 0:00:08 webapp RUNNING pid 459, uptime 0:05:54 webapp2 RUNNING pid 846, uptime 0:00:08 supervisor>
The first instance: webapp was left untouched, because its configuration was not changed.
The second instance: webapp2 can be reached as follows:
$ curl -H 'Host: webapp2.local' http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Simple test webapp</title>
</head>
<body>
Simple test webapp listening on port: 5001
</body>
</html>
After upgrading the system, the new configuration should also get reactivated after a container restart.
A more interesting example: Hydra
As explained earlier, to create upgradable containers we require a fully functional Nix installation in a container. This observation made a think about a more interesting example than the trivial web application system.
A prominent example of a system that requires Nix and is composed out of multiple tightly integrated process is Hydra: the Nix-based continuous integration service.
To make it possible to deploy a minimal Hydra service in a container, I have packaged all its relevant components for the Nix process management framework.
The processes model looks as follows:
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
nix-processmgmt-services = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt-services.git;
ref = "master";
};
constructors = import "${nix-processmgmt-services}/services-agnostic/constructors.nix" {
inherit nix-processmgmt pkgs stateDir runtimeDir logDir tmpDir cacheDir forceDisableUserChange processManager;
};
instanceSuffix = "";
hydraUser = hydraInstanceName;
hydraInstanceName = "hydra${instanceSuffix}";
hydraQueueRunnerUser = "hydra-queue-runner${instanceSuffix}";
hydraServerUser = "hydra-www${instanceSuffix}";
in
rec {
nix-daemon = {
pkg = constructors.nix-daemon;
};
postgresql = rec {
port = 5432;
postgresqlUsername = "postgresql";
postgresqlPassword = "postgresql";
socketFile = "${runtimeDir}/postgresql/.s.PGSQL.${toString port}";
pkg = constructors.simplePostgresql {
inherit port;
authentication = ''
# TYPE DATABASE USER ADDRESS METHOD
local hydra all ident map=hydra-users
'';
identMap = ''
# MAPNAME SYSTEM-USERNAME PG-USERNAME
hydra-users ${hydraUser} ${hydraUser}
hydra-users ${hydraQueueRunnerUser} ${hydraUser}
hydra-users ${hydraServerUser} ${hydraUser}
hydra-users root ${hydraUser}
# The postgres user is used to create the pg_trgm extension for the hydra database
hydra-users postgresql postgresql
'';
};
};
hydra-server = rec {
port = 3000;
hydraDatabase = hydraInstanceName;
hydraGroup = hydraInstanceName;
baseDir = "${stateDir}/lib/${hydraInstanceName}";
inherit hydraUser instanceSuffix;
pkg = constructors.hydra-server {
postgresqlDBMS = postgresql;
user = hydraServerUser;
inherit nix-daemon port instanceSuffix hydraInstanceName hydraDatabase hydraUser hydraGroup baseDir;
};
};
hydra-evaluator = {
pkg = constructors.hydra-evaluator {
inherit nix-daemon hydra-server;
};
};
hydra-queue-runner = {
pkg = constructors.hydra-queue-runner {
inherit nix-daemon hydra-server;
user = hydraQueueRunnerUser;
};
};
apache = {
pkg = constructors.reverseProxyApache {
dependency = hydra-server;
serverAdmin = "admin@localhost";
};
};
}
In the above processes model, each process instance represents a component of a Hydra installation:
- The nix-daemon process is a service that comes with Nix package manager to facilitate multi-user package installations. The nix-daemon carries out builds on behalf of a user.
Hydra requires it to perform builds as an unprivileged Hydra user and uses the Nix protocol to more efficiently orchestrate large builds. - Hydra uses a PostgreSQL database backend to store data about projects and builds.
The postgresql process refers to the PostgreSQL database management system (DBMS) that is configured in such a way that the Hydra components are authorized to manage and modify the Hydra database. - hydra-server is the front-end of the Hydra service that provides a web user interface. The initialization procedure of this service is responsible for initializing the Hydra database.
- The hydra-evaluator regularly updates the repository checkouts and evaluates the Nix expressions to decide which packages need to be built.
- The hydra-queue-runner builds all jobs that were evaluated by the hydra-evaluator.
- The apache server is used as a reverse proxy server forwarding requests to the hydra-server.
With the following commands, we can build the image, load it into Docker, and deploy a container that runs Hydra:
$ nix-build hydra-image.nix $ docker load -i result $ docker run -it --name hydra-test --network host hydra:test
After deploying the system, we can connect to the container:
$ docker exec -it hydra-test /bin/bash
and observe that all processes are running and managed by supervisord:
$ supervisorctl apache RUNNING pid 1192, uptime 0:00:42 hydra-evaluator RUNNING pid 1297, uptime 0:00:38 hydra-queue-runner RUNNING pid 1296, uptime 0:00:38 hydra-server RUNNING pid 1188, uptime 0:00:42 nix-daemon RUNNING pid 1186, uptime 0:00:42 postgresql RUNNING pid 1187, uptime 0:00:42 supervisor>
With the following commands, we can create our initial admin user:
$ su - hydra $ hydra-create-user sander --password secret --role admin creating new user `sander'
We can connect to the Hydra front-end in a web browser by opening http://localhost (this works because the container uses host networking):
and configure a job set to a build a project, such as libprocreact:
Another nice bonus feature of having multiple process managers supported is that if we build Hydra's Nix process management configuration for Disnix, we can also visualize the deployment architecture of the system with disnix-visualize:
The above diagram displays the following properties:
- The outer box indicates that we are deploying to a single machine: localhost
- The inner box indicates that all components are managed as processes
- The ovals correspond to process instances in the processes model and the arrows denote dependency relationships.
For example, the apache reverse proxy has a dependency on hydra-server, meaning that the latter process instance should be deployed first, otherwise the reverse proxy is not able to forward requests to it.
Building a Nix-enabled container image
As explained in the previous section, mutable Docker images require a fully functional Nix package manager in the container.
Since this may also be an interesting sub use case, I have created a convenience function: createNixImage that can be used to build an image whose only purpose is to provide a working Nix installation:
let
pkgs = import <nixpkgs> {};
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
createNixImage = import "${nix-processmgmt}/nixproc/create-image-from-steps/create-nix-image.nix" {
inherit pkgs;
};
in
createNixImage {
name = "foobar";
tag = "test";
contents = [ pkgs.mc ];
}
The above Nix expression builds a Docker image with a working Nix setup and a custom package: the Midnight Commander.
Conclusions
In this blog post, I have described a new function in the Nix process management framework: createMutableMultiProcessImage that creates reproducible mutable multi-process container images, by combining the reproducibility properties of Docker and Nix. With the exception of the process manager, process instances in a container can be upgraded without bringing the entire container down.
With this new functionality, the deployment workflow of a multi-process container configuration has become very similar to how physical and virtual machines are managed with NixOS -- you can edit a declarative specification of a system and run a single command-line instruction to deploy the new configuration.
Moreover, this new functionality allows us to deploy a complex, tightly coupled multi-process system, such as Hydra: the Nix-based continuous integration service. In the Hydra example case, we are using Nix for three deployment aspects: constructing the Docker image, deploying the multi-process system configuration and building the projects that are configured in Hydra.
A big drawback of mutable multi-process images is that there is no sharing possible between multiple multi-process containers. Since the images are not built from common layers, the Nix store is private to each container and all packages are deployed in the writable custom layer, this may lead to substantial disk and RAM overhead per container instance.
Deploying the processes model to a container instance can probably be made more convenient by using Nix flakes -- a new Nix feature that is still experimental. With flakes we can easily deploy an arbitrary number of Nix expressions to a container and pin the deployment to a specific version of Nixpkgs.
Another interesting observation is the word: mutable. I am not completely sure if it is appropriate -- both the layers of a Docker image, as well as the Nix store paths are immutable and never change after they have been built. For both solutions, immutability is an important ingredient in making sure that a deployment is reproducible.
I have decided to still call these deployments mutable, because I am looking at the problem from a Docker perspective -- the writable layer of the container (that is mounted on top of the immutable layers of an image) is modified each time that we upgrade a system.
Future work
Although I am quite happy with the ability to create mutable multi-process containers, there is still quite a bit of work that needs to be done to make the Nix process management framework more usable.
Most importantly, trying to deploy Hydra revealed all kinds of regressions in the framework. To cope with all these breaking changes, a structured testing approach is required. Currently, such an approach is completely absent.
I could also (in theory) automate the still missing parts of Hydra. For example, I have not automated the process that updates the garbage collector roots, which needs to run in a timely manner. To solve this, I need to use a cron service or systemd timer units, which is beyond the scope of my experiment.
Availability
The createMutableMultiProcessImage function is part of the experimental Nix process management framework GitHub repository that is still under heavy development.
Because the amount of services that can be deployed with the framework has grown considerably, I have moved all non-essential services (not required for testing) into a separate repository. The Hydra constructor functions can be found in this repository as well.
