Last weekend, I have visited FOSDEM, the Free and Open-source Software Developers' European Meeting held at the Université Libre de Bruxelles (ULB) for the third time. This is a very big event organized every year for free software and open-source people to meet each other. Also, all the major free and open-source projects that you can think of are well represented there.
It's always quite impressive to see the Janson auditorium completely filled with thousands of free and open-source people during the keynote presentations. I took a picture during the keynote of Eben Moglen of the Software Freedom Law Center (shown above) to give an impression how massive this event is.
This time I have also given a presentation about NixOS in the CrossDistro devroom, because I think it's a good idea to promote our ideas and concepts to a bigger audience next to academic people. In my presentation I have explained the complexity of deployment in various scenarios (single installation, distributed environments, virtual machines), the general idea and concepts of NixOS and also some applications we have developed the last couple of months to deal with the complexity of deployment.
One of our recently developed applications is distributed NixOS deployment. So instead of writing a single NixOS configuration, you can also write a network of NixOS configurations, for instance:
{
storage =
{pkgs, config, ...}:
{
services.portmap.enable = true;
services.nfsKernel.server.enable = true;
services.nfsKernel.server.exports = ''
/repos 192.168.1.0/255.255.255.0(rw,no_root_squash)
'';
services.nfsKernel.server.createMountPoints = true;
};
postgresql =
{config, pkgs, ...}:
{
services.openssh.enable = true;
services.postgresql.enable = true;
services.postgresql.enableTCPIP = true;
services.postgresql.authentication = ''
# Generated file; do not edit!
local all all trust
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
host all all 192.168.1.0/24 trust
'';
};
webserver =
{config, pkgs, ...}:
{
fileSystems = pkgs.lib.mkOverride 50
[ { mountPoint = "/repos";
device = "storage:/repos";
fsType = "nfs";
options = "bootwait"; }
];
services.portmap.enable = true;
services.nfsKernel.client.enable = true;
services.httpd.enable = true;
services.httpd.adminAddr = "root@localhost";
services.httpd.extraSubservices =
[ { serviceType = "trac"; } ];
environment.systemPackages =
[ pkgs.pythonPackages.trac pkgs.subversion ];
};
client =
{config, pkgs, ...}:
{
require = [ ./common/x11.nix ];
services.xserver.desktopManager.kde4.enable = true;
};
};
The network expression shown above represents a network of machines describing a Trac environment, a web-based management tool for software projects. A Trac environment can be (of course) deployed on a single system, but also on multiple systems. For example, the Subversion server storing source code may be deployed on a different machine as the PostgreSQL database storing tickets and bug reports. In the network expression shown above, we have defined 4 machines, representing a Subversion server, PostgreSQL server, web server and a client machine running the KDE Plasma desktop.
A network of NixOS machines can be automatically deployed by writing a network expression and by typing:
$ nixos-deploy-network network.nix
The nixos-deploy-network tool first builds all NixOS configurations for all the machines. Then it efficiently transfers the system configurations and all its dependencies to the right machines in the network. Because of the purely functional properties of Nix, this phase will not harm the existing configurations because all files are stored safely next to each other in the Nix store and no files are overwritten or automatically removed.
After all system configurations and dependencies have been transferred to the target machines, the system configurations are activated. In this phase system services are stopped and started on each machine in the network, which may bring some downtime to the complete system, but this time window is a small as possible. In case of a failure, a rollback is performed which activates the previous configuration again. This can be done easily in NixOS, since older configurations are still available in the Nix store, unless they are garbage collected.
Another recently developed application is virtualization. By running the following command:
$ nixos-build-vms; ./result/bin/nixos-run-vms
A network of virtual machines is generated and automatically launched closely resembling the configurations defined in the network model. This allows users to experiment with a specific configuration, without having to deploy a collection of physical machines. Another notable feature is that virtual networks are cheap to instantiate. We don't have to create disk images, but instead we mount the Nix store of the host machine through SMB/CIFS. We can safely do this because of the purely functional concept of the Nix store. An impression of a virtual network running Trac is shown above.
We have also developed a NixOS test driver. This can be used to perform automatic distributed test cases in a network of virtual machines.
testScript =
''
startAll;
$postgresql->waitForJob("postgresql");
$postgresql->mustSucceed("createdb trac");
$webserver->mustSucceed("mkdir -p /repos/trac");
$webserver->mustSucceed("svnadmin create /repos/trac");
$webserver->waitForFile("/var/trac");
$webserver->mustSucceed("mkdir -p /var/trac/projects/test");
$webserver->mustSucceed("trac-admin /var/trac/projects/test ".
"initenv Test postgres://root\@postgresql/trac svn ".
"/repos/trac");
$client->waitForX;
$client->execute("konqueror http://webserver/projects/test &");
$client->waitForWindow(qr/Test.*Konqueror/);
$client->sleep(30); # loading takes a long time
$client->screenshot("screen");
'';
The code fragment above shows an example of a test suite for the Trac environment. This test suite creates a Trac database on the PostgreSQL server, a Subversion repository on the Subversion repository, then it defines a Trac project using the trac-admin tool and launches a web browser to take a screenshot of the entry page. The test suite is performed by the test driver in a non-interactive manner.
We have applied the distributed deployment and testing techniques to various use cases. We used the nixos-deploy-network tool to deploy our complete Hydra build environment for continuous integration and testing of many software components. We also have implemented various test cases for NixOS, various GNU projects and other free software projects using the NixOS test driver.
I am quite happy to see how well the ideas described in my presentation were received at FOSDEM. It was the first time for me to present there and I didn't really know what to expect. It seems that my talk attracted quite a number of people and I received quite a number of positive reactions and a lot of good questions and suggestions. I have to admit that these questions were far better than the ones I usually receive at academic conferences.
The slides of my FOSDEM presentation (titled: 'Using NixOS for declarative deployment and testing') can be obtained from the talks page of my homepage. The distributed testing techniques are also described in our ISSRE 2010 paper titled: 'Automating System Tests Using Declarative Virtual Machines'. The technical report titled: 'Declarative Testing and Deployment of Distributed Systems' describes an earlier implementation of our declarative deployment and testing techniques. Both papers can be downloaded from my publications page.
The techniques are part of NixOS. The NixOS test driver can also be used on any Linux system running the Nix package manager and KVM, which allows you to still use your favourite Linux distribution if you don't want to switch to NixOS.
No comments:
Post a Comment