Free (and Open-Source) software

Last monday, Richard Stallman, the founder of the Free Software Foundation and the GNU project, gave a guest lecture at Delft University of Technology, which I attended. The talk was titled: 'Copyright versus Community'. The contents of the talk was not really about Free Software specifically, but instead it tries answering the question whether the ideas of Free Software extend to other kinds of works.

One of the interesting things he explained in his talk is how copyright has been developed throughout the ages, since its introduction in England in the 1500s. First, it started as a form of censorship for Protestants. Later, the first form of "real" copyright was only applicable for publishers and lasted only 14 years. People were still free to copy printed work if they wish (although this wasn't very doable in most cases). Throughout the time the power of copyright was extended. Nowadays, copyright lasts for something like 75 years and apart from publishers, also individuals get severely punished for the most silly kinds of copyright infringement.

Another thing he pointed out is that governments aren't very ethical either; they often support stronger copyright laws and implement annoying wishes of publishing companies in law, instead of listening to people's desires and needs. He pointed out that it's in the nature of humans to share things which each other. Restricting and taking control is against humanity. Because software has an increasingly importance our daily lives, the importance of Free Software also increases, which do not take freedom away of people. (Maybe somebody is going to threaten me after reading this! :P )

I liked the talk, but I found the question round afterwards quite annoying. There were some good questions, but also some lame ones. As a software developer and researcher I also talk quite frequently about Free (and Open-Source) Software and I often see some common "misconceptions" about what it is and what not and what the implications are. During the question round also some lame questions came by to which Richard Stallman usually responded by saying: "I don't understand the question". Later, he elaborated on the question a bit more. So in this blog I'd like to talk a bit more about some common misconceptions I frequently encounter and I'll try to clarify them a bit, because I have to stress these things out a lot to people.

Meaning

The first class of misconceptions I usually encounter is about the meaning of some terms which are frequently used. When I use the term: "Free Software" often people think that it's about gratis software (free in price). When I use the term: "Open Source" people think that it's about software for which the source code can be obtained. Both of these common misunderstandings don't represent what they are truly about.

Free Software (and Open Source) is not gratis per se. This is probably misunderstood due to the fact that the word 'free' is ambigious in English. The word 'free' refers to freedom, not price. For example, the translation of free software in Dutch is: 'Vrije software' or in French: 'Libre Software'. Moreover, it's perfectly legal to sell free software for whatever price you want, as described here. The point is that it rarely happens in practice nowadays, because one cannot restrict other people from distributing (and selling) it. Also software gets rapidly and easily redistributed through the internet, which is mostly done for free.

Furthermore, Open-Source (and Free Software) is not just about distributing source code. You should also be allowed to study source code, make modifications and redistribute the work (with or without changes). For example, there is software available that includes source code but you are not allowed to distribute your modified versions. This category of software is neither Free nor Open-Source although the source code may be available.

Intermixing

A lot of people intermix the terms "Free Software" and "Open Source" as if it they were the same thing. Although almost all Free Software is Open Source software and vice-versa; they both are two different definitions and represent different philosophies.

Free Software is not just about distributing source code with some additional rights. It's about 4 freedoms, for which having the source code and the ability to study and make changes is an essential precondition. The definition of Open-Source is more or less a pragmatic view of sharing code in an open manner. You need to have to source code and you should be able to study and make/distribute changes. An author, however, may impose some restrictions for integrity. There is also another definition of free software available described in the Debian Free Software Guidelines (mostly used by Debian developers). The Open-Source definition is derived from this.

This page explains some differences between Free Software and Open Source software from the FSF's point of view: http://www.gnu.org/philosophy/free-software-for-freedom.html. It's also important to point out that Free Software and Open Source people are not enemies of each other, although they don't have exactly the same point of view. They work together quite frequently and because in practice nearly all Free Software is Open Source software and vice versa, usually both parties get what they want.

Anti-commercial

A lot of people think that Free Software (and Open Source) is the opposite of commercial software. Similarly, during the talk of Richard Stallman some people thought that all commercial activities were bad/unethical. This is a big mistake and not what free software and freedom in general is about.

Although most Free and Open-Source software is non-commercial, also commercial Free and Open-Source software exists. Commercial activity is just a way for companies to earn money. Free and Open-Source software don't restrict that nor it is considered a bad thing as long as it does not restrict people's freedom.

When I explain this to people, I often receive the question: "But why do companies release software under Free and Open Source licenses then? That's bad for them right?". Usually such companies do not make money from Free and Open Source software by selling copies or licenses, but in different ways, such as by selling support, offering services based around their technology and dual licensing (distributing the same software under a proprietary and free software license simultaneously).

Some companies are quite successful even though their software is Free and Open-Source and can be used by practically anyone. In some cases, releasing software under Free and Open-source licenses gives companies advantages. They can easily built communities around their software and accept external contributions so that the whole community benefits and their commercial services improve.

The opposite of Free (and Open-Source) software is proprietary software. Proprietary software is available both commercially and non-commercially and impose restrictions to users freedom in some degree.

Dangerous

When I try convincing certain people in certain companies to use a particular Free and Open Source product, they often remind me that I should be very careful, because it may be dangerous. Often these people are referring to the implications of copyleft, which is not always fully understood.

I will first explain a bit what copyleft is about. Copyleft is an instrument in a license which guarantees that the 4 freedoms defined in the Free Software definition are preserved in a work in some degree. The most famous copyleft license is the GNU General Public License (GPL). Basically, this license states that a complete derived work must retain all freedoms. The GPL license is not always applicable for all types of software, such as libraries. Therefore, the GNU Lesser General Public License (LGPL) has been developed with a weaker copyleft. This license basically restricts the boundaries of the copyleft to a library instead of a whole work. Also non-copylefted Free and Open-Source software exists. These licenses allow you to incorporate software into your proprietary products, make modifications and keep them secret.

Even though a license is a copyleft license or not, you are always free to make modifications and to run the software for whatever goal you want. It also does not restrict you selling software for whatever price you want. The only obligation that it gives you, is that you have to give users some degree of freedom. If you're not distributing the software itself, you can keep all you custom modifications secret, copylefted or not. These conditions only become applicable when you're distributing software.

In theory, a modified piece of GPL software can be offered as a service, without giving the changes away. This is because you're not distributing the software itself to end users. To guarantee freedom to users of services the GNU Affero General Public License (AGPL) has been developed, which states that users in a network also have the right to access the source code and the same rights. Although an AGPL license exists, most free software products don't use it.

To be clear about this, using Free Software is never dangerous in the sense that you have to give something away. You only have obligations if you want to (re)distribute derived works or when you give external users access to software.

Giving away

Sometimes I ask to people why they don't publish a piece of software under a Free and Open-source license. Frequently, I get an answer saying: "Well, then I give up all my rights and I give everything I have away!". This is also a misconception I want to clarify.

When releasing software under a license, you're still the copyright holder. As a copyright holder you have the ability to change a license, even when it's released under a Free and Open-source license (except when you have received external contributions and didn't let the contributors transfer their copyright to you). Moreover, copyleft may give you some protection, such as the right to obtain the source code of derived works. Releasing software under a free license doesn't mean that it is in the public domain of course.

Conclusion

Hopefully, I have clarified some misconceptions in this blog post a bit. As a Free and Open Source software developer myself, I think it's important to point this out.

References

• The following page gives a good overview about all software categories I have discussed in this blog post: http://www.gnu.org/philosophy/categories.html
• The contents of the 'Copyright vs Community' talk can be found on the GNU website: http://www.gnu.org/philosophy/copyright-versus-community.html.

Disnix: A toolset for distributed deployment

On February the 14th, I have released Disnix 0.2. It seems that I have picked a nice date for it, just like the release date of the of Disnix 0.1, which was released on April the 1st 2010 (and that's no joke). Since I haven't written any blog post about Disnix yet, I'll give some info here.

Disnix is a distributed deployment extension for the Nix package manager. Whereas Nix manages packages and dependencies residing on the same system in the Nix store (which we call intra-dependencies later on), Disnix manages distributable components (or services). Services have intra-dependencies on components residing on the same system, but also dependencies on other services, which may be located on a different machine in the network (which we call inter-dependencies later on). Disnix extends the Nix approach and offers various features to deploy to service-oriented systems, including the management of inter-dependencies.

The figure above shows how Disnix works in a nutshell. In the center the disnix-env command-line tool is shown, which performs the complete deployment process of a service-oriented system. On the left, various models are shown. The services model captures all the service (distributable components) of which a system consists, their types and inter-dependencies. This model includes a reference to all-packages.nix, a Nix expression capturing intra-dependency compositions. The infrastructure model captures the available machines in the network and their relevant properties/capabilities. The distribution model maps services defined in the services model to machines defined in the infrastructure model. On the right, a network of machines is shown, which have all have the DisnixService installed providing remote access to deployment operations.

By writing instances of the models mentioned earlier and by running:

$ disnix-env -s services.nix -i infrastructure.nix \
  -d distribution.nix

The system is built from source-code including all required intra-dependencies. Then the services and intra-dependencies are efficiently transferred to the target machines in the network. Finally, the services are activated in the right order derived from the inter-dependency graph.

By adapting the models and running disnix-env again, an upgrade is performed instead of a full installation. In this case only components which have changed are rebuilt and transferred to the target machines in the network. Moreover, only obsolete services are deactivated and new services are activated.

Similar to writing ordinary Nix expressions for each package, you also write Disnix expressions for each service describing how it can be built from source and its dependencies.

{stdenv, StaffService}:
{staff}:

let
  jdbcURL = "jdbc:mysql://"+
    staff.target.hostname+":"+
    toString (staff.target.mysqlPort)+"/"+
    staff.name+"?autoReconnect=true";
  contextXML = ''
    <Context>
      <Resource name="jdbc/StaffDB" auth="Container"
            type="javax.sql.DataSource"
            maxActivate="100" maxIdle="30" maxWait="10000"
            username="${staff.target.mysqlUsername}"
            password="${staff.target.mysqlPassword}"
            driverClassName="com.mysql.jdbc.Driver"
            url="${jdbcURL}" />
    </Context>
  '';
in
stdenv.mkDerivation {
  name = "StaffService";
  buildCommand = ''
    ensureDir $out/conf/Catalina
    cat > $out/conf/Catalina/StaffService.xml <<EOF
    ${contextXML}
    EOF
    ln -sf ${StaffService}/webapps $out/webapps
  '';
}

The code fragement above shows a Disnix expression for the StaffTracker example included in the Disnix repository. The main difference between this expression and an ordinary Nix expression is that it has two function headers which takes intra-dependencies and inter-dependencies respectively to configure the component. The inter-dependency arguments are used in this expression to generate a so called context XML file, which Apache Tomcat uses to configure resources such as JDBC connections, containing a URL, port number and authentication credentials for a MySQL database residing on a different machine in the network. For other types of components a different configuration file has to be created.

Moreover, you also need to compose a Disnix expression. A Disnix expression must first be composed locally by calling the function with the right intra-dependency arguments. This is done in a similar way as ordinary Nix expressions. Later, the same function is called with the right inter-dependency arguments as well.

{distribution, system, pkgs}:

let customPkgs = import ../top-level/all-packages.nix {
  inherit system pkgs;
};
in
rec {
### Databases
  staff = {
    name = "staff";
    pkg = customPkgs.staff;
    dependsOn = {};
    type = "mysql-database";
  };
  ...

### Web services
  StaffService = {
    name = "StaffService";
    pkg = customPkgs.StaffServiceWrapper;
    dependsOn = {
      inherit staff;
    };
    type = "tomcat-webapplication";
  };
  ...

### Web applications

  StaffTracker = {
    name = "StaffTracker";
    pkg = customPkgs.StaffTracker;
    dependsOn = {
      inherit GeolocationService RoomService;
      inherit StaffService ZipcodeService;
    };
    type = "tomcat-webapplication";
  };
  ...
}

The above expression shows the services model, used to capture of which distributable components a system consists. Basically, this model is a function taking three arguments: the distribution model (shown later), a collection of Nixpkgs and a system identifier indicating the architecture of a target host. The function returns an attribute set in which each attribute represents a service. For each service various properties are defined, such as a name, a pkg attribute referring to a function which creates an intra-dependency composition of a service (defined in an external file not shown here), dependsOn composing the inter-dependencies and a type, which is used for activation and deactivation of the service.

{
  test1 = {
    hostname = "test1.example.org";
    tomcatPort = 8080;
    system = "i686-linux";
  };
  
  test2 = {
    hostname = "test2.example.org";
    tomcatPort = 8080;
    mysqlPort = 3306;
    mysqlUsername = "root";
    mysqlPassword = "secret";
    system = "i686-linux";
  }; 
}

The expression shown above is an infrastructure model, which captures machines in the network and their relevant properties/capabilities. This expression is an attribute set in which each attribute represents a machine in the network. Some properties are mandatory, such as the hostname indicating how the Disnix service can be reached. The system property denotes the system architecture so that a service is built for that particular platform. Other properties can be freely chosen and are used for activation/deactivation of a component.

{infrastructure}:

{
  GeolocationService = [ infrastructure.test1 ];
  RoomService = [ infrastructure.test2 ];
  StaffService = [ infrastructure.test1 ];
  StaffTracker = [ infrastructure.test1 infrastructure.test2 ];
  ZipcodeService = [ infrastructure.test1 ];
  rooms = [ infrastructure.test2 ];
  staff = [ infrastructure.test2 ];
  zipcodes = [ infrastructure.test2 ];
}

The final expression shown above is the distribution model, mapping services to machines in the network. This expression is a function taking the infrastructure model as parameter. The body is an attribute set in which every attribute representing a service refers to a list of machines in the network. It also allows you to map a service to multiple machines for e.g. load balancing.

The models shown earlier are used by Disnix to perform the complete deployment process of a service-oriented system, i.e. building services, transferring services and the activation of services. Because Disnix uses the purely functional properties of Nix, this process is reliable and efficient. If a system is upgraded, no components are removed and overwritten, since everything is stored in isolation in the Nix store. So while upgrading, we can still keep the current system intact. Only during the transition phase in which services are deactivated and activated the system is inconsistent, but Disnix keeps this time window as small as possible. Moreover, a proxy can be used during this phase to queue connections, which makes the upgrade process truly atomic.

Although Disnix supports the deployment of a service-oriented system, some additional extensions have been developed to make deployment more convenient:

• DisnixWebService. By default Disnix uses a SSH connection to connect to remote machines in the network. This extension provides a SOAP interface and disnix-soap-client to perform deployment through the SOAP protocol.
• DisnixOS. Disnix manages the services of which a system is composed, but not the system configurations of the underlying infrastructure. This extension provides additional infrastructure management features to Disnix based on the techniques described in the blog post titled: Using NixOS for declarative deployment and testing. By using this extension you can automatically deploy a network of NixOS configurations next to the services through Disnix. Moreover, you can also use this extension to generate a network of virtual machines and automatically deploy the system in the virtual network. A screenshot is shown above, which runs the StaffTracker example in a network of three virtual machines.
• Dynamic Disnix. Disnix requires developers or system administrators to manually write an infrastructure model and a distribution model. In a network in which events occur, such as a machine which crashes or a new machine with new system resource is added, this introduces a large degree of inflexibility. The Dynamic Disnix toolset offers a discovery service, which dynamically discovers the machines in the network and their relevant properties/capabilities. Moreover, it also includes a distribution model generator, which uses a custom defined policy and a collection of distribution algorithms to dynamically distribute services to machines, based on non-functional properties defined in the services and infrastructure models.
  The Dynamic Disnix extension is still under heavy development and not released as part of Disnix 0.2. It will become part of the next Disnix release.
  

Disnix, the extensions and some examples can be obtained from the Disnix web page: http://nixos.org/disnix. Disnix is also described in several academic papers. The paper: 'Disnix: A toolset for distributed deployment' describes the architecture of the Disnix toolset. This paper is however somewhat outdated, as there are some minor changes in the current implementation. The paper: 'Automated Deployment of a Heterogeneous Service-Oriented System' describes the 0.1 implementation, which we have used for a case study at Philips Research. The publications and presentation slides can be obtained from the publications and talks sections of my homepage. Moreover, there are some earlier publications about Disnix available as well. In a next blog post, I will explain more about the development process and development choices of Disnix.

Using NixOS for declarative deployment and testing

Last weekend, I have visited FOSDEM, the Free and Open-source Software Developers' European Meeting held at the Université Libre de Bruxelles (ULB) for the third time. This is a very big event organized every year for free software and open-source people to meet each other. Also, all the major free and open-source projects that you can think of are well represented there.

It's always quite impressive to see the Janson auditorium completely filled with thousands of free and open-source people during the keynote presentations. I took a picture during the keynote of Eben Moglen of the Software Freedom Law Center (shown above) to give an impression how massive this event is.

This time I have also given a presentation about NixOS in the CrossDistro devroom, because I think it's a good idea to promote our ideas and concepts to a bigger audience next to academic people. In my presentation I have explained the complexity of deployment in various scenarios (single installation, distributed environments, virtual machines), the general idea and concepts of NixOS and also some applications we have developed the last couple of months to deal with the complexity of deployment.

One of our recently developed applications is distributed NixOS deployment. So instead of writing a single NixOS configuration, you can also write a network of NixOS configurations, for instance:

{
  storage = 
    {pkgs, config, ...}:
    {
      services.portmap.enable = true;
      services.nfsKernel.server.enable = true;
      services.nfsKernel.server.exports = ''
        /repos 192.168.1.0/255.255.255.0(rw,no_root_squash)
      '';
      services.nfsKernel.server.createMountPoints = true;
    };

  postgresql =
    {config, pkgs, ...}:
    {
      services.openssh.enable = true;
      services.postgresql.enable = true;
      services.postgresql.enableTCPIP = true;
      services.postgresql.authentication = ''
        # Generated file; do not edit!
        local all all                trust
        host  all all 127.0.0.1/32   trust
        host  all all ::1/128        trust
        host  all all 192.168.1.0/24 trust
      '';
    };

  webserver = 
    {config, pkgs, ...}:
    {
      fileSystems = pkgs.lib.mkOverride 50  
        [ { mountPoint = "/repos";
            device = "storage:/repos";
            fsType = "nfs";
            options = "bootwait"; } 
        ];
      
      services.portmap.enable = true;
      services.nfsKernel.client.enable = true;
      services.httpd.enable = true;
      services.httpd.adminAddr = "root@localhost";
      services.httpd.extraSubservices =
        [ { serviceType = "trac"; } ];
      environment.systemPackages =
        [ pkgs.pythonPackages.trac pkgs.subversion ];
    };
      
  client = 
    {config, pkgs, ...}:
    {
      require = [ ./common/x11.nix ];
      services.xserver.desktopManager.kde4.enable = true;
    };
};

The network expression shown above represents a network of machines describing a Trac environment, a web-based management tool for software projects. A Trac environment can be (of course) deployed on a single system, but also on multiple systems. For example, the Subversion server storing source code may be deployed on a different machine as the PostgreSQL database storing tickets and bug reports. In the network expression shown above, we have defined 4 machines, representing a Subversion server, PostgreSQL server, web server and a client machine running the KDE Plasma desktop.

A network of NixOS machines can be automatically deployed by writing a network expression and by typing:

$ nixos-deploy-network network.nix

The nixos-deploy-network tool first builds all NixOS configurations for all the machines. Then it efficiently transfers the system configurations and all its dependencies to the right machines in the network. Because of the purely functional properties of Nix, this phase will not harm the existing configurations because all files are stored safely next to each other in the Nix store and no files are overwritten or automatically removed.

After all system configurations and dependencies have been transferred to the target machines, the system configurations are activated. In this phase system services are stopped and started on each machine in the network, which may bring some downtime to the complete system, but this time window is a small as possible. In case of a failure, a rollback is performed which activates the previous configuration again. This can be done easily in NixOS, since older configurations are still available in the Nix store, unless they are garbage collected.

Another recently developed application is virtualization. By running the following command:

$ nixos-build-vms; ./result/bin/nixos-run-vms

A network of virtual machines is generated and automatically launched closely resembling the configurations defined in the network model. This allows users to experiment with a specific configuration, without having to deploy a collection of physical machines. Another notable feature is that virtual networks are cheap to instantiate. We don't have to create disk images, but instead we mount the Nix store of the host machine through SMB/CIFS. We can safely do this because of the purely functional concept of the Nix store. An impression of a virtual network running Trac is shown above.

We have also developed a NixOS test driver. This can be used to perform automatic distributed test cases in a network of virtual machines.

testScript =
''
  startAll;
      
  $postgresql->waitForJob("postgresql");
  $postgresql->mustSucceed("createdb trac");
      
  $webserver->mustSucceed("mkdir -p /repos/trac");
  $webserver->mustSucceed("svnadmin create /repos/trac");
      
  $webserver->waitForFile("/var/trac");      
  $webserver->mustSucceed("mkdir -p /var/trac/projects/test");
  $webserver->mustSucceed("trac-admin /var/trac/projects/test ".
    "initenv Test postgres://root\@postgresql/trac svn ".
    "/repos/trac");
      
  $client->waitForX;
  $client->execute("konqueror http://webserver/projects/test &");
  $client->waitForWindow(qr/Test.*Konqueror/);
  $client->sleep(30); # loading takes a long time
      
  $client->screenshot("screen");
'';

The code fragment above shows an example of a test suite for the Trac environment. This test suite creates a Trac database on the PostgreSQL server, a Subversion repository on the Subversion repository, then it defines a Trac project using the trac-admin tool and launches a web browser to take a screenshot of the entry page. The test suite is performed by the test driver in a non-interactive manner.

We have applied the distributed deployment and testing techniques to various use cases. We used the nixos-deploy-network tool to deploy our complete Hydra build environment for continuous integration and testing of many software components. We also have implemented various test cases for NixOS, various GNU projects and other free software projects using the NixOS test driver.

I am quite happy to see how well the ideas described in my presentation were received at FOSDEM. It was the first time for me to present there and I didn't really know what to expect. It seems that my talk attracted quite a number of people and I received quite a number of positive reactions and a lot of good questions and suggestions. I have to admit that these questions were far better than the ones I usually receive at academic conferences.

The slides of my FOSDEM presentation (titled: 'Using NixOS for declarative deployment and testing') can be obtained from the talks page of my homepage. The distributed testing techniques are also described in our ISSRE 2010 paper titled: 'Automating System Tests Using Declarative Virtual Machines'. The technical report titled: 'Declarative Testing and Deployment of Distributed Systems' describes an earlier implementation of our declarative deployment and testing techniques. Both papers can be downloaded from my publications page.

The techniques are part of NixOS. The NixOS test driver can also be used on any Linux system running the Nix package manager and KVM, which allows you to still use your favourite Linux distribution if you don't want to switch to NixOS.

NixOS: A purely functional Linux distribution

My last blog post was about the Nix package manager, a package manager with some distinct features, such as multiple versions/variants packages which are stored in isolation from each other in a Nix store, the Nix expression language and atomic upgrades and rollbacks. I also briefly mentioned NixOS, a Linux distribution using the Nix package manager as a basis. In this blog post I will discuss the concepts of NixOS more thoroughly.

It's probably already very obvious that NixOS uses the Nix package manager for installing, upgrading and removing software components. In fact, except for end-user software such as Mozilla Firefox and OpenOffice.org, development tools such as GCC, Binutils, also all operating system parts such as the Linux kernel, system configuration files and kernel modules are all managed by the Nix package manager and stored in isolation in the Nix store.

Because of the fact that Nix stores all packages in a special directory: /nix/store, the NixOS distribution does not have standard FHS directories such as: /usr and /lib and only a minimal /etc and /bin. The /bin directory only contains a symlink: /bin/sh pointing to the bash shell. This is required in order to make glibc's system() function working properly. I have uploaded a screenshot (seen above) in which a terminal is shown, listing the directory contents of the root filesystem, /bin folder and some other files. As you may notice, certain directories commonly found on other distributions don't exist, and the /bin/sh points to the bash shell which is also stored somewhere in the Nix store.

The fact that Nix is used instead of a conventional package manager such as RPM or dpkg, does probably not sound so exciting to most users. Actually, NixOS is more then just a Linux distribution managed by Nix. The main thing that sets NixOS apart from conventional Linux distributions is that system configurations are also configured with declarative specifications just Nix like packages. Instead of manually installing packages and manually editing configurations files, such as an Apache web server configuration, everything is configured from a single declarative specification, invoking functions which generate configuration files and install required packages.

{pkgs, ...}:

{                                                                                                                                                            
  boot.loader.grub.device = "/dev/sda";
                                                
  fileSystems = [
    { mountPoint = "/";
      device = "/dev/sda2";
    }
  ];

  swapDevices = [ { device = "/dev/sda1"; } ];
  
  services = {
    openssh.enable = true;

    httpd = {
      enable = true;
      documentRoot = "/var/www";
      adminAddr = "admin@localhost";
    };

    xserver = {
      enable = true;
      
      desktopManager = {
        default = "kde4";
        kde4 = { enable = true; };
      };
      
      displayManager = {
        kdm = { enable = true; };
        slim = { enable = false; };
      };
    };
  };
  
  environment.systemPackages = [
    pkgs.mc
    pkgs.subversion
    pkgs.firefox
  ];
}

The code fragment above shows an example of a NixOS configuration. As you may notice, this configuration file contains all kinds of settings for a particular system, such as the MBR partition for the GRUB bootloader, filesystems (such as the root and swap partition), system services (OpenSSH, Apache Webserver and X.Org configured to use the KDE desktop and the KDM login manager). Also, system packages are defined here such as Mozilla Firefox and Midnight Commander, which are automatically in the PATH of all users and also appear automatically in the KDE menu if desktop files are included.

By storing a configuration file in /etc/nixos/configuration.nix and by running: nixos-rebuild switch a complete system configuration is built by the Nix package manager and activated. Because of the fact that the Nix package manager is used, we can safely upgrade a complete system; this is because the complete system configuration and all its dependencies are safely stored in the Nix store and no files are overwritten or removed. Because of this we can also rollback to any previous configuration, which has not been garbage collected yet.

The figure above shows the GRUB boot loader used for NixOS. As you may notice, all the previous configurations of the system are shown, which are not garbage collected yet. The GRUB boot loader of NixOS allows you to boot into any previous configuration (You can also switch back to a previous configuration while running NixOS).

The last missing piece in the puzzle is how configuration files are handled in NixOS. In NixOS you don't manually edit configuration files. Instead, configuration files are generated by Nix expressions and stored in the Nix store (just like ordinary packages). Because they are generated by Nix expressions and they also may contain paths to Nix expressions containing binaries, this also has a nice implication. If a configuration file contains a Nix store path, Nix handles them as dependencies, and automatically includes them if a configuration is deployed somewhere.

Maybe you have become interested in trying out NixOS after reading this. ISOs can be obtained from the NixOS website: http://nixos.org/nixos. In NixOS quite a number of packages are supported. We also have end-user software available and the KDE desktop. Keep in mind that NixOS is an experimental Linux distribution, which may not offer everything you need. We only have a small community of ~25 developers maintaining it.

If you find some concepts of NixOS useful but you don't want to switch, you can also use the virtualization features of NixOS to automatically generate virtual machines running a particular NixOS configuration. You can even do this on regular Linux distributions supporting KVM having the Nix package manager installed.

One point of criticism I always receive is about the fact that we don't edit configuration files manually. I have to admit that this is a bit inconvenient, if you want to try out a particular feature of a service. But manually editing stuff becomes very inconvenient in a distributed setting. Probably everyone agrees that it is unfeasible in case of a configuration change, to manually update the configurations of every affected machine. One of the things we investigate in the PDS research project is expanding the Nix deployment vision to distributed systems.

There are some publications about NixOS available. The first prototype of NixOS was implemented in 2006 by Armijn Hemel for his masters thesis. Nicolas Pierron implemented the NixOS module system. Various publications about NixOS can be obtained from Eelco Dolstra's publications page.

The Nix package manager

In my previous blog post, which was about Pull Deployment of Services, I have explained what the project I'm working on as a PhD student is about. An import foundation is the Nix package manager, which serves as an important basis of my research.

There are dozens of papers and websites available describing Nix, so in this blog post I will not give another extensive description. Instead, I'm going to explain some of the details and the vision that we have about software deployment which sets the Nix approach apart from conventional approaches.

Nix is a package manager developed by Eelco Dolstra as part of his PhD research, which has similar purposes as tools such as RPM and dpkg commonly found in many Linux distributions, such as Red Hat enterprise Linux, Fedora, Debian and Ubuntu.

A package manager is basically a collection of software tools to automate the process of installing, upgrading, configuring, and removing software packages for an operating system in a consistent manner. In many Linux distributions, the package manager is one of the major factors that distinguish them from each other. Similarly, an experimental Linux distribution has been built around Nix, called NixOS using the Nix package manager as its basis.

One of the major differences of Nix compared to conventional deployment tooling, is the way Nix addresses packages (or components). Most tools use nominal dependency specifications, consisting of a name and version number, such as:

openssl-1.0.0c

Although it is a good thing to have these attributes to distinguish packages (which for instance isn't properly done for native Windows DLLs), this dependency specification mechanism has some limitations. A problem may occur when OpenSSL is compiled with a different version of GCC as an executable linking to it. A library compiled with an older version of GCC cannot always be linked to an executable compiled with a newer version of GCC, due to a different ABI. Moreover, OpenSSL also has dependencies on libraries, such as glibc. If OpenSSL is compiled against an incompatible older version of glibc as an executable, also problems may occur.

In Nix, we have different way of addressing components. We store components in a special directory called the Nix store in which every component is stored under an unique filename, such as:

/nix/store/xq2bfcqdsbrmfr8h5ibv7n1qb8xs5s79-openssl-1.0.0c

The former part of the component name: xq2bfcqdsbrmfr8h5ibv7n1qb8xs5s79 is a SHA256 hash code derived from all build time dependencies to build the component from source code. For example, if the same component is built with a different version of GCC, a different hash code is generated.

Using hash codes to address components offers us some nice advantages. Since every component has an unique filename, we can safely store multiple versions and variants next to each other, without overwriting each other. This also offers us the possibility to upgrade a system atomically. In case of an upgrade a new component is safely stored next to an existing one, and after the upgrade only a symlink pointing to the configuration by a user is changed.

The hash codes in the component names are also used to detect runtime dependencies of a component. This is done by scanning for occurrences of a hash inside a component. If a hash code is found (such as a path containing a Nix store component in an ELF header, or a shell script containing a Nix store path) Nix identifies these components as a runtime dependency. Although this technique sounds risky/scary, we have used this for many packages and it turns out to work quite well.

Another key feature is the Nix expression language, which we use to build packages, as shown in the code fragment below.

{stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "hello-2.6";
  
  src = fetchurl {
    url = ftp://ftp.gnu.org/gnu/hello/hello-2.6.tar.gz;
    sha256 = "1h6fjkkwr7kxv0rl5l61ya0b49imzfaspy7jk9jas1fil31sjykl";
  };

  meta = {
    homepage = http://www.gnu.org/software/hello/manual/;
    license = "GPLv3+";
  };
}

Basically each package description is a function which takes some dependencies as arguments (in this case stdenv and fetchurl). In the body of the function we call the mkDerivation function describing a build action, which takes some arguments, such as a reference to the source code and other relevant build properties. In this expression we did not specify how to build this component. By default, Nix assumes that a packages is autotools based (i.e. ./configure; make; make install) if build instructions are omitted.

The expression described earlier cannot be used to build a package directly. The function must be called with the right arguments, i.e. a component must be composed. This is done in another expression, shown below:

rec {
  stdenv = ...;

  fetchurl = import ../build-support/fetchurl {
    inherit stdenv curl;
  };

  hello = import ../applications/misc/hello {
    inherit stdenv fetchurl;
  }

  ...
}

In this expression we call the function building the GNU Hello component (shown in the previous expression) with its required arguments. As you may notice, all the dependencies of the GNU Hello package are composed in this expression as well.

Now you may think that this dependency scheme of using hash codes to address components is a bit unhandy for end users. To solve this problem, we use Nix profiles. A Nix profile is basically an environment in the PATH of a user, containing symlinks indirectly referring to components in the Nix store. For example, by typing:

$ nix-env -i hello

The GNU Hello component is installed into the Nix profile of the current user. By typing the following instruction on the command-line:

$ hello

The GNU Hello component installed earlier is executed.

The last distinguishable feature I'd like to mention is that in Nix packages are not directly removed, but removed by a garbage collector. The Nix garbage collector will safely remove components that are no longer in use. In most conventional tooling you may accidentally remove a package which may be still in use. Packages which are still installed in a profile of a user or which are a dependency of another package, will not be removed by Nix. Moreover, running processes and open files are also garbage collector roots, which makes it safe to run the garbage collector at any time.

After reading this blog post, you may have become interested in trying the Nix package manager. The simplest way to try out Nix is by downloading and installing NixOS. Although NixOS is a Linux distribution built around Nix, you can also use Nix on conventional Linux distributions, if you find it more convenient to keep using the operating system you are used to. Moreover, the Nix package manager is quite portable and can also be used on different operating systems, such as FreeBSD, OpenSolaris, Mac OS X and Windows (through Cygwin).

Except for plain package management and the management of a Linux distribution, Nix is also used as foundation for some other tooling. Hydra is a continuous build and integration server built on top of Nix, which allows you to continuously checkout sources from repositories and to build components in several variations (e.g. using various compilers, libraries and platforms). I have developed an extension called Disnix, which extends the Nix package manager for the deployment of distributable components (or services) into a network of machines.

Another thing I want to clarify is that I don't want to claim that other package management solutions (and deployment tooling) are bad solutions. In fact, I think other package management solutions have done a great job in dealing with certain installation issues. Moreover, I find package management tools one the strong points of Linux distributions, although we support some distinct features which we care about, which other package managers don't.

We are not the only group doing research in package management. There is also a project, called Mancoosi, in which package and system management issues are investigated. They have a different view about certain principles, such as keeping maintainer scripts which imperatively modify the system configuration (whereas we generate everything as packages in the Nix store). Moreover, their solutions are based on package management tools widely used in Linux distributions, such as RPM and dpkg.

For more information about Nix, have a look at the Nix website: http://nixos.org/nix or have a look at Eelco's excellent PhD thesis titled: 'The Purely Functional Software Deployment Model', which can be found on his publications page.

Pull Deployment of Services

It's not very useful to have a blog which is almost empty, so here is my second blog post. As I have explained in the previous post, I'm a PhD student working on the PDS project. So let's give a general description of PDS here, since for a lot of people I see and talk to this is still a big mystery.

PDS is a research project funded by Jacquard, a Dutch funding programme to stimulate software and service research and collaboration in Dutch academia and industry. The participants in my project are Delft University of Technology and Philips Healthcare. So now I'll come to the question: what is PDS actually? The abbreviation stands for 'Pull Deployment of Services' but this still does not makes thing really clear yet.

The project goal is inspired by the medical domain. In hospitals various services are used to assist medical staff in performing their tasks. Examples of these services are viewing X-Ray images or retrieving patient records.

Hospital environments are mostly device-oriented, in which services are bound to dedicated devices. For example, to view an image a dedicated workstation assigned for this purpose (as shown in the picture on the left).

Device-orientation has several undesired implications. Firstly, it introduces overcapacity and suboptimal usage of devices. Each device has an amount of system resources available which are only used when a service is actually used. Most of the times these system resources aren't used at all. Secondly, device-orientation gives a large degree of inflexibility. If a machine breaks, a particular service may become completely unavailable, which is absolutely undesirable in hospital environments. Finally, deploying a service is a very complex process, because this process is often performed semi-manually and in an ad-hoc fashion.

The goal of the PDS project is to move from device-orientation to service-orientation. Services can be used on any device, any place and on-demand. This is what we mean with Pull Deployment of Services. To achieve this goal services must be built in such a way that they are distributable. This means that a service can be offered in multiple variants. For example, to run an image viewer on a workstation all the components can be run locally because most workstations have a powerful GPU. To run the same service on a mobile device, such as a PDA, some components have to deployed elsewhere, since a PDA is not powerful enough to perform all the image rendering itself.

To support pull deployment of services, we must craft an architecture composed of various tools integrated with domain specific modules. We have several tools in mind which we use as a foundation. Nix is used as foundation for deployment, because it has distinct deployment features such as a declarative language to specify build actions, atomic upgrades and rollbacks. Moreover, it serves as the basis of the NixOS GNU/Linux distribution in which complete machine configurations can be derived from a single declarative specification. NixOS can also be atomically upgraded.

Although the PDS project is inspired by the medical domain, the tooling and techniques we are developing are applicable to any service-oriented system. Moreover, tooling is released as part of Nix project and can be used under free and open-source licenses.

The PDS project has started in September 2008 and we have achieved some interesting results. Moreover, we have implemented some tooling. In future blog posts I'll describe some more details about this. More details about the goal of the project can be found in our ICSE Cloud 2009 paper titled: 'Software Deployment in a Dynamic Cloud: From Device to Service Orientation in a Hospital Environment', which is available from the publications section of my homepage. Moreover, I have given two presentations about the general idea of the project. The slides are also on my homepage and titled 'Pull Deployment of Services'.

First blog post

Ok great! This is my first blog post. I've decided to start a blog to let people know what I'm working on and I also think it's a good instrument to get some early feedback.

About me: I'm a PhD student at the Software Engineering Research Group, Delft University of Technology working on the PDS project. The PDS project has started in September 2008 and is mainly about developing distributed services and automatically deploying them on-demand in a complex environment of machines.

Since I'm still in my (small) Christmas holiday, I have decided not to spent too much time on my work (I sometimes need to rest :-) ) and therefore I will post more details about PDS as soon as my Christmas break is over. So please, stay tuned.