Although I am quite happy with the most recent feature addition, another major concern that the basic Disnix toolset does not solve is coping with the dynamism of the services and the environment in which a system has been deployed.
Static modeling of services and the environment have the following consequences:
- We must write an infrastructure model reflecting all relevant properties of all target machines. Although writing such a configuration file for a new environment is doable, it is quite tedious and error prone to keep it up to date and in sync with their actual configurations -- whenever a machine's property or the network changes, the infrastructure model must be updated accordingly.
(As a sidenote: when using the DisnixOS extension, a NixOS network model is used instead of an infrastructure model from which the machine's configurations can be automatically deployed making the consistency problem obsolete. However, the problem remains to persist if we need to deploy to a network of non-NixOS machines) - We must manually specify the distribution of services to machines. This problem typically becomes complicated if services have specific technical requirements on the host that they need to run (e.g. operating system, CPU architecture, infrastructure components such as an application server).
Moreover, a distribution could also be subject to non-functional requirements. For example, a service providing access to privacy-sensitive data should not be deployed to a machine that is publicly accessible from the internet.
Because requirements may be complicated, it is typically costly to repeat the deployment planning process whenever the network configuration changes, especially if the process is not automated.
To cope with the above listed issues, I have developed a prototype extension called Dynamic Disnix and wrote a paper about it. The extension toolset provides the following:
- A discovery service that captures the properties of the machines in the network from which an infrastructure model is generated.
- A framework allowing someone to automate deployment planning processes using a couple of algorithms described in the literature.
Besides the dynamism of the infrastructure model and distribution models, I also observed that the services model (capturing the components of which a system consists) may be too static in certain kinds of situations.
Microservices
Lately, I have noticed that some kind of new paradigm named Microservice architectures is gaining a lot of popularity. In many ways this new trend reminds me of the service-oriented architectures days -- everybody was talking about it and had success stories, but nobody had a full understanding of it, nor an idea what it exactly was supposed to mean.
However, if I would restrict myself to some of their practical properties, microservices (like "ordinary" services in a SOA-context) are software components and one important trait (according to Clemens Szyperski's Component Software book) is that a software component:
is a unit of independent deployment
Another important property of microservices is that they interact with other by sending messages through the HTTP communication protocol. In practice, many people accomplish this by running processes with an embedded HTTP server (as opposed to using application servers or external web servers).
Deploying Microservices with Disnix
Although Disnix was originally developed to deploy a "traditional" service-oriented system case-study (consisting of "real" web services using SOAP as communication protocol), it has been made flexible enough to deploy all kinds of components. Likewise, Disnix can also deploy components that qualify themselves as microservices.
However, when deploying microservices (running embedded HTTP servers) there is one practical issue -- every microservice must listen on their own unique TCP port on a machine. Currently, meeting this requirement is completely the responsibility of the person composing the Disnix deployment models.
In some cases, this problem is more complicated than expected. For example, manually assigning a unique TCP port to every service for the initial deployment is straight forward, but it may also be desired to move a service from one machine to another. It could happen that a previously assigned TCP port will conflict with another service after moving it, breaking the deployment of the system.
The port assignment problem
So far, I take the following aspects into account when assignment ports:
- Each service must listen on a port that is unique to the machine the service runs on. In some cases, it may also be desirable to assign a port that is unique to the entire network (instead of a single machine) so that it can be uniformly accessed regardless of its location.
- The assigned ports must be within a certain range so that (for example) they do not collide with system services.
- Once a port number has been assigned to a service, it must remain reserved until it gets undeployed.
The alternative would be to reassign all port numbers to all services for each change in the network, but that can be quite costly in case of an upgrade. For example, if we upgrade a network running 100 microservices, all 100 of them may need to be deactivated and activated to make them listen on their newly assigned ports.
Dynamically configuring ports in Disnix models
Since it is quite tedious and error prone to maintain port assignments in Disnix models, I have developed a utility to automate the process. To dynamically assign ports to services, they must be annotated with the portAssign property in the services model (which can be changed to any other property through a command-line parameter):
{distribution, system, pkgs}:
let
portsConfiguration = if builtins.pathExists ./ports.nix
then import ./ports.nix else {};
...
in
rec {
roomservice = rec {
name = "roomservice";
pkg = customPkgs.roomservicewrapper { inherit port; };
dependsOn = {
inherit rooms;
};
type = "process";
portAssign = "private";
port = portsConfiguration.ports.roomservice or 0;
};
...
stafftracker = rec {
name = "stafftracker";
pkg = customPkgs.stafftrackerwrapper { inherit port; };
dependsOn = {
inherit roomservice staffservice zipcodeservice;
};
type = "process";
portAssign = "shared";
port = portsConfiguration.ports.stafftracker or 0;
baseURL = "/";
};
}
In the above example, I have annotated the roomservice component with a private port assignment property meaning that we want to assign a TCP port that is unique to the machine and the stafftracker component with a shared port assignment meaning that we want to assign a TCP port that is unique to the network.
By running the following command we can assign port numbers:
$ dydisnix-port-assign -s services.nix -i infrastructure.nix \
-d distribution.nix > ports.nix
The above command generates a port assignment configuration Nix expression (named: ports.nix) that contains port reservations for each service and port assignment configurations for the network and each individual machine:
{
ports = {
roomservice = 8001;
...
zipcodeservice = 3003;
};
portConfiguration = {
globalConfig = {
lastPort = 3003;
minPort = 3000;
maxPort = 4000;
servicesToPorts = {
stafftracker = 3002;
};
};
targetConfigs = {
test2 = {
lastPort = 8001;
minPort = 8000;
maxPort = 9000;
servicesToPorts = {
roomservice = 8001;
};
};
};
};
}
The above configuration attribute set contains three properties:
- The ports attribute contains the actual port numbers that have been assigned to each service. The services defined in the services model (shown earlier) refer to the port values defined here.
- The portConfiguration attribute contains port configuration settings for the network and each target machine. The globalConfig attribute defines a TCP port range with ports that must be unique to the network. Besides the port range it also stores the last assigned TCP port number and all global port reservations.
- The targetConfigs attribute contains port configuration settings and reservations for each target machine.
We can also run the port assign command-utility again with an existing port assignment configuration as a parameter:
$ dydisnix-port-assign -s services.nix -i infrastructure.nix \
-d distribution.nix -p ports.nix > ports2.nix
The above command-line invocation reassigns TCP ports, taking the previous port reservations into account so that these will be reused where possible (e.g. only new services get a port number assigned). Furthermore, it also clears all port reservations of the services that have been undeployed. The new port assignment configuration is stored in a file called ports2.nix.
Conclusion
In this blog post, I have identified another deployment planning problem that manifests itself when deploying microservices that all have to listen on a unique TCP port. I have developed a utility to automate this process.
Besides assigning port numbers, there are many other kinds of problems that need a solution while deploying microservices. For example, you might also want to restrict their privileges (e.g. by running all of them as separate unprivileged users). It is also possible to take care of that with Dysnomia.
Availability
The dydisnix-port-assign utility is part of the Dynamic Disnix toolset that can be obtained from my GitHub page. Unfortunately, the Dynamic Disnix toolset is still a prototype with no end-user documentation or a release, so you have to be brave to use it.
Moreover, I have created yet another Disnix example package (a Node.js variant of the ridiculous StaffTracker example) to demonstrate how "microservices" can be deployed. This particular variant uses Node.js as implementation platform and exposes the data sets through REST APIs. All components are microservices using Node.js' embedded HTTP server listening on their own unique TCP ports.
I have also modified the TCP proxy example to use port assignment configurations generated by the tool described in this blog post.
