First computer

Not so long ago, I bought a new PC. In my previous PC, the video card died and I didn't find it useful to invest any money in a system which is almost 4 years old. In this blog post, I don't want to talk about my new computer, but about my first one which still gives me good memories.

My first computer experience started on a Commodore 64 owned by my parents. Actually, it was a more advanced Commodore 128 (shown in a picture from Wikipedia above), which was automatically booted in C64 mode due to a helper cartridge. The newer Commodore 128 model was still compatible with the older Commodore 64.

The amount of Commodore 64 system resources were very slim compared to systems these days. It included a MOS Technology 6510 processor running at ~1 MHz and 64 KiB of RAM (half of the amount could only be used by BASIC programs). In Commodore 128 mode, you had double the amount of RAM (128 KiB) available and the more advanced 8502 processor running on ~2 MHz.

Usage

The operating system and shell of both the Commodore 64 and 128 were also very simple. It was just a simple BASIC interpreter, which you could use for various tasks such as retrieving the contents of a disk and to load a program. Retrieving the contents of a disk was done by loading a special program named: '$' into the BASIC program space in memory:

LOAD "$",8

And by listing the BASIC program, you could see the disk contents:

LIST

Then you basically had to move your cursor to the line containing the program that you wanted to run then you had to type: LOAD and RUN to load and run the program. Maybe this process sounds scary, since we have fancy GUIs nowadays, but it wasn't so hard back then. I was able to do this when I was 6 years old.

Programming

My first experience with programming also started on this computer. For some reason, knowing that it was possible to create your own stuff (next to running somebody else's stuff) fascinated me. A cousin of mine (who already had some programming experience on the Commodore) showed me the basics. Moreover, I also owned several C64 programming books (given to me by some relatives) which I used as a reference, although I was not always able to understand all these concepts as a kid.

The first Commodore 64 BASIC program I ever wrote looked basically like this:

10 INPUT "WHAT IS YOUR NAME";A$
20 PRINT "HELLO ";A$;"!"

It was just a very simple program which asked the user to type his name and responded by sending a friendly greeting to the user. Of course, these two lines were a little boring, so usually I added two lines in the beginning which cleared the screen, changed the color of the text and I used some POKE'ing to change to colors of the main screen and screen border to make the program look a little prettier. Since the Commodore 64 uses special characters for clearing the screen and changing the color of the text (which I can't list here), I have included a screenshot showing the output of the program and a listing of the source code.

I wrote many more Commodore 64 BASIC programs, such as a variant of the first program to insult persons that I disliked, some simple games (guessing words and numbers or games using a dice) and a lot of useless stuff in which I tried to do things with graphics and to create games from it (but I lacked the skills/knowledge back then to do something really useful).

The challenging part of programming games, was that you had to understand the hardware architecture of the Commodore 64 quite well. Although an internal BASIC interpreter was available, there were no high level instructions for things such as creating sprites and sounds. Most of these things were achieved by using POKEs and PEEKs in BASIC to read and write values into the appropriate memory positions (the Commodore 128 BASIC interpreter was more advanced and did have some high level instructions, but I never used them). Moreover, BASIC was also much too slow for most games and therefore you had to write most things in assembly. But since writing stuff in assembly is not much more difficult then using POKEs and PEEKs this was not the biggest challenge.

Games

Although the Commodore 64 had only 16 predefined colors (which can't be changed), some games had good graphics and were exceptionally creative. Basically, the tiles of side scroller games consisted of programmable characters in multi-color mode. Each multi-color character could use 4 colors out of the predefined 16. Each character pixel could be 1 custom color (for the whole character), the background color or 2 predefined colors (shared with all other characters). Assigning a color value to each individual pixel was too expensive. In order to make something look nice, you really had to think ahead and use some creative tricks. I still have to admit that I'm impressed to see how those games have been developed.

One of the most advanced games I owned was Turrican, a side-scroller shooter game combining several elements of various games such as Metroid. I have uploaded some screenshots shown on the left. Moreover, I also remember this game very well because I had a hard time beating one of the bosses, which gave me nightmares when I was still a kid. The boss that I feared so much is shown in the last screenshot :-) . This game is often praised for it's high technical achievements, showing things people did not believe to be possible on a Commodore 64. The sequel: Turrican 2 contained even more advanced features, although I did not own a copy of it.

Experience

Except for playing games, writing many useless Commodore 64 and 128 BASIC programs, and some "experiments" in Commodore 64 assembly, I also wrote several programs that actually were a bit interesting (in some degree). The "coolest program" I wrote was a demo in which I rapidly abused POKEs and some delays to change the colors of the main screen and screen border, which created cool screen effects.

The most useful program I have created is probably the homework assistant, to help me learning my English/French vocabularies. It could be used to fill in Dutch words and their translations and to test your skills. It also had the ability to save your vocabularies on disk. Although creating the homework assistant was fun, I rarely used it during my studies :-) (probably because learning words wasn't that difficult for me).

Nowadays, computers are many times more powerful than my good old Commodore and have much more abilities. Moreover, throughout the years many more (high-level) programming languages, frameworks, libraries, paradigms and other techniques have been developed to make things more convenient. We have a lot of cool stuff, such as high resolution 3D graphics, but for some reason I don't get the impression that everything has become so many more times "better". Funny...

However, I'm happy to say that the Commodore 128 of my parents is still in my possession and it still seems to work, although I rarely turn it on nowadays.

References

For the screenshots included in this blog post, I used VICE, a collection of Free/Open-Source emulators for various CBM models, including the C64 and C128.

Self-adaptive deployment with Disnix

In an earlier blog post I have described Disnix, a Nix-based distributed service deployment system. Disnix uses declarative specifications of the services, infrastructure and distribution to automatically, reliably and efficiently deploy a service-oriented system using the purely functional deployment properties of Nix.

Although Disnix offers useful deployment features, networks in which service-oriented systems are deployed are often dynamic. Various events may occur, such as a crashing machine which disappears from the network, an addition of a new machine with new capabilities or a change of a capability, e.g. an increase of memory. Such events may partially or completely break a system or render a deployment scenario suboptimal.

For these types of events a redeployment may be required to fix the system or change the deployment to have all the desired non-functional constraints supported, which takes some effort. The infrastructure model must be updated reflecting the new configuration of machines. Moreover, a new distribution model must be written mapping services to the right machines in the network. Manually updating these models every time an event occurs is often a complex and time consuming process.

We have developed a self-adaptive framework on top of Disnix to deal with such events, shown in the figure above, to automatically redeploy a system in such a way that desired non-functional constraints are supported in case of an event.

At the top, an infrastructure generator is shown. This is a tool generating a Disnix infrastructure model using a discovery service, capturing the present machines in the network and their capabilities and properties.

The generated infrastructure model, however, may not contain all required properties for deployment, such as authentication credentials and other privacy-sensitive information. The infrastructure augmenter is used to augment those additional properties into the discovered infrastructure model.

Then the distribution generator is invoked with the services model and generated infrastructure model. This tool generates a Disnix distribution model and uses a policy described in a QoS model, to dynamically map services to machines using non-functional properties defined in the services and infrastructure models. Various algorithms can be invoked from the QoS model to generate a suitable mapping.

Finally, the generated infrastructure and distribution models along with the services model are passed to disnix-env, which performs the actual (re)deployment of the system as reliable and efficient as possible.

The current implementation uses Avahi for capturing machines in the network and their properties, but is not limited to a particular protocol. One can easily replace this tool by a different lookup service using SSDP or a custom protocol.

The distribution generator is a very generic and extensible approach. In a QoS model, a developer can pick from a range of distribution filters and combine them together to achieve a desired result. New algorithms can be easily integrated in this architecture.

{ services, infrastructure
, initialDistribution, previousDistribution
, filters
}:

filters.minsetcover {
  inherit services infrastructure;
  targetProperty = "cost";
  
  distribution = filters.mapAttrOnList {
    inherit services infrastructure;
    serviceProperty = "type";
    targetPropertyList = "supportedTypes";
  };
}

The code fragment above shows an example of a QoS model. This model is a function taking several arguments: the services model (provided by the user), the generated infrastructure model, an initial distribution model (which maps all services to all targets), a previous distribution model (can be used to reflect about upgrade effort) and a set of filter functions.

In the body of the expression various filter functions are used. Each filter function takes a candidate distribution and return a filtered distribution. These functions are composed together to achieve a desirable mapping. In the example above, we first map all services having a specific type attribute to machines which support that particular type (i.e. the supportedTypes attribute in the infrastructure model containing a list of possibilities). By using this filter function we can e.g. map Apache Tomcat web applications to servers running Apache Tomcat and MySQL databases to services running a MySQL DBMS. This is a very crucial feature, as we don't want to deploy a service to a machine which is not capable of running it.

In the outer function, we use a minimum set cover approximation method over the result of the previous function. This function can be used to find the minimum cost deployment given that every machine has a fixed cost price, no matter how many services it will host.

We have applied the dynamic Disnix framework to a number of use cases, such as several toy systems, ViewVC and an industrial case study from Philips research.

The dynamic Disnix framework is still under heavy development and is going to be part of Disnix 0.3. In the dynamic framework, a number of algorithms and division strategies are supported. For more information have a look at the 'A Self-Adaptive Deployment Framework for Service-Oriented Systems' paper, available from my publications page. I will present this paper at SEAMS 2011 (colocated with ICSE 2011) on Waikiki, Honolulu, Hawaii. You can find the slides on my talks page, once I have prepared them. See you in Hawaii!

Free (and Open-Source) software

Last monday, Richard Stallman, the founder of the Free Software Foundation and the GNU project, gave a guest lecture at Delft University of Technology, which I attended. The talk was titled: 'Copyright versus Community'. The contents of the talk was not really about Free Software specifically, but instead it tries answering the question whether the ideas of Free Software extend to other kinds of works.

One of the interesting things he explained in his talk is how copyright has been developed throughout the ages, since its introduction in England in the 1500s. First, it started as a form of censorship for Protestants. Later, the first form of "real" copyright was only applicable for publishers and lasted only 14 years. People were still free to copy printed work if they wish (although this wasn't very doable in most cases). Throughout the time the power of copyright was extended. Nowadays, copyright lasts for something like 75 years and apart from publishers, also individuals get severely punished for the most silly kinds of copyright infringement.

Another thing he pointed out is that governments aren't very ethical either; they often support stronger copyright laws and implement annoying wishes of publishing companies in law, instead of listening to people's desires and needs. He pointed out that it's in the nature of humans to share things which each other. Restricting and taking control is against humanity. Because software has an increasingly importance our daily lives, the importance of Free Software also increases, which do not take freedom away of people. (Maybe somebody is going to threaten me after reading this! :P )

I liked the talk, but I found the question round afterwards quite annoying. There were some good questions, but also some lame ones. As a software developer and researcher I also talk quite frequently about Free (and Open-Source) Software and I often see some common "misconceptions" about what it is and what not and what the implications are. During the question round also some lame questions came by to which Richard Stallman usually responded by saying: "I don't understand the question". Later, he elaborated on the question a bit more. So in this blog I'd like to talk a bit more about some common misconceptions I frequently encounter and I'll try to clarify them a bit, because I have to stress these things out a lot to people.

Meaning

The first class of misconceptions I usually encounter is about the meaning of some terms which are frequently used. When I use the term: "Free Software" often people think that it's about gratis software (free in price). When I use the term: "Open Source" people think that it's about software for which the source code can be obtained. Both of these common misunderstandings don't represent what they are truly about.

Free Software (and Open Source) is not gratis per se. This is probably misunderstood due to the fact that the word 'free' is ambigious in English. The word 'free' refers to freedom, not price. For example, the translation of free software in Dutch is: 'Vrije software' or in French: 'Libre Software'. Moreover, it's perfectly legal to sell free software for whatever price you want, as described here. The point is that it rarely happens in practice nowadays, because one cannot restrict other people from distributing (and selling) it. Also software gets rapidly and easily redistributed through the internet, which is mostly done for free.

Furthermore, Open-Source (and Free Software) is not just about distributing source code. You should also be allowed to study source code, make modifications and redistribute the work (with or without changes). For example, there is software available that includes source code but you are not allowed to distribute your modified versions. This category of software is neither Free nor Open-Source although the source code may be available.

Intermixing

A lot of people intermix the terms "Free Software" and "Open Source" as if it they were the same thing. Although almost all Free Software is Open Source software and vice-versa; they both are two different definitions and represent different philosophies.

Free Software is not just about distributing source code with some additional rights. It's about 4 freedoms, for which having the source code and the ability to study and make changes is an essential precondition. The definition of Open-Source is more or less a pragmatic view of sharing code in an open manner. You need to have to source code and you should be able to study and make/distribute changes. An author, however, may impose some restrictions for integrity. There is also another definition of free software available described in the Debian Free Software Guidelines (mostly used by Debian developers). The Open-Source definition is derived from this.

This page explains some differences between Free Software and Open Source software from the FSF's point of view: http://www.gnu.org/philosophy/free-software-for-freedom.html. It's also important to point out that Free Software and Open Source people are not enemies of each other, although they don't have exactly the same point of view. They work together quite frequently and because in practice nearly all Free Software is Open Source software and vice versa, usually both parties get what they want.

Anti-commercial

A lot of people think that Free Software (and Open Source) is the opposite of commercial software. Similarly, during the talk of Richard Stallman some people thought that all commercial activities were bad/unethical. This is a big mistake and not what free software and freedom in general is about.

Although most Free and Open-Source software is non-commercial, also commercial Free and Open-Source software exists. Commercial activity is just a way for companies to earn money. Free and Open-Source software don't restrict that nor it is considered a bad thing as long as it does not restrict people's freedom.

When I explain this to people, I often receive the question: "But why do companies release software under Free and Open Source licenses then? That's bad for them right?". Usually such companies do not make money from Free and Open Source software by selling copies or licenses, but in different ways, such as by selling support, offering services based around their technology and dual licensing (distributing the same software under a proprietary and free software license simultaneously).

Some companies are quite successful even though their software is Free and Open-Source and can be used by practically anyone. In some cases, releasing software under Free and Open-source licenses gives companies advantages. They can easily built communities around their software and accept external contributions so that the whole community benefits and their commercial services improve.

The opposite of Free (and Open-Source) software is proprietary software. Proprietary software is available both commercially and non-commercially and impose restrictions to users freedom in some degree.

Dangerous

When I try convincing certain people in certain companies to use a particular Free and Open Source product, they often remind me that I should be very careful, because it may be dangerous. Often these people are referring to the implications of copyleft, which is not always fully understood.

I will first explain a bit what copyleft is about. Copyleft is an instrument in a license which guarantees that the 4 freedoms defined in the Free Software definition are preserved in a work in some degree. The most famous copyleft license is the GNU General Public License (GPL). Basically, this license states that a complete derived work must retain all freedoms. The GPL license is not always applicable for all types of software, such as libraries. Therefore, the GNU Lesser General Public License (LGPL) has been developed with a weaker copyleft. This license basically restricts the boundaries of the copyleft to a library instead of a whole work. Also non-copylefted Free and Open-Source software exists. These licenses allow you to incorporate software into your proprietary products, make modifications and keep them secret.

Even though a license is a copyleft license or not, you are always free to make modifications and to run the software for whatever goal you want. It also does not restrict you selling software for whatever price you want. The only obligation that it gives you, is that you have to give users some degree of freedom. If you're not distributing the software itself, you can keep all you custom modifications secret, copylefted or not. These conditions only become applicable when you're distributing software.

In theory, a modified piece of GPL software can be offered as a service, without giving the changes away. This is because you're not distributing the software itself to end users. To guarantee freedom to users of services the GNU Affero General Public License (AGPL) has been developed, which states that users in a network also have the right to access the source code and the same rights. Although an AGPL license exists, most free software products don't use it.

To be clear about this, using Free Software is never dangerous in the sense that you have to give something away. You only have obligations if you want to (re)distribute derived works or when you give external users access to software.

Giving away

Sometimes I ask to people why they don't publish a piece of software under a Free and Open-source license. Frequently, I get an answer saying: "Well, then I give up all my rights and I give everything I have away!". This is also a misconception I want to clarify.

When releasing software under a license, you're still the copyright holder. As a copyright holder you have the ability to change a license, even when it's released under a Free and Open-source license (except when you have received external contributions and didn't let the contributors transfer their copyright to you). Moreover, copyleft may give you some protection, such as the right to obtain the source code of derived works. Releasing software under a free license doesn't mean that it is in the public domain of course.

Conclusion

Hopefully, I have clarified some misconceptions in this blog post a bit. As a Free and Open Source software developer myself, I think it's important to point this out.

References

• The following page gives a good overview about all software categories I have discussed in this blog post: http://www.gnu.org/philosophy/categories.html
• The contents of the 'Copyright vs Community' talk can be found on the GNU website: http://www.gnu.org/philosophy/copyright-versus-community.html.

Disnix: A toolset for distributed deployment

On February the 14th, I have released Disnix 0.2. It seems that I have picked a nice date for it, just like the release date of the of Disnix 0.1, which was released on April the 1st 2010 (and that's no joke). Since I haven't written any blog post about Disnix yet, I'll give some info here.

Disnix is a distributed deployment extension for the Nix package manager. Whereas Nix manages packages and dependencies residing on the same system in the Nix store (which we call intra-dependencies later on), Disnix manages distributable components (or services). Services have intra-dependencies on components residing on the same system, but also dependencies on other services, which may be located on a different machine in the network (which we call inter-dependencies later on). Disnix extends the Nix approach and offers various features to deploy to service-oriented systems, including the management of inter-dependencies.

The figure above shows how Disnix works in a nutshell. In the center the disnix-env command-line tool is shown, which performs the complete deployment process of a service-oriented system. On the left, various models are shown. The services model captures all the service (distributable components) of which a system consists, their types and inter-dependencies. This model includes a reference to all-packages.nix, a Nix expression capturing intra-dependency compositions. The infrastructure model captures the available machines in the network and their relevant properties/capabilities. The distribution model maps services defined in the services model to machines defined in the infrastructure model. On the right, a network of machines is shown, which have all have the DisnixService installed providing remote access to deployment operations.

By writing instances of the models mentioned earlier and by running:

$ disnix-env -s services.nix -i infrastructure.nix \
  -d distribution.nix

The system is built from source-code including all required intra-dependencies. Then the services and intra-dependencies are efficiently transferred to the target machines in the network. Finally, the services are activated in the right order derived from the inter-dependency graph.

By adapting the models and running disnix-env again, an upgrade is performed instead of a full installation. In this case only components which have changed are rebuilt and transferred to the target machines in the network. Moreover, only obsolete services are deactivated and new services are activated.

Similar to writing ordinary Nix expressions for each package, you also write Disnix expressions for each service describing how it can be built from source and its dependencies.

{stdenv, StaffService}:
{staff}:

let
  jdbcURL = "jdbc:mysql://"+
    staff.target.hostname+":"+
    toString (staff.target.mysqlPort)+"/"+
    staff.name+"?autoReconnect=true";
  contextXML = ''
    <Context>
      <Resource name="jdbc/StaffDB" auth="Container"
            type="javax.sql.DataSource"
            maxActivate="100" maxIdle="30" maxWait="10000"
            username="${staff.target.mysqlUsername}"
            password="${staff.target.mysqlPassword}"
            driverClassName="com.mysql.jdbc.Driver"
            url="${jdbcURL}" />
    </Context>
  '';
in
stdenv.mkDerivation {
  name = "StaffService";
  buildCommand = ''
    ensureDir $out/conf/Catalina
    cat > $out/conf/Catalina/StaffService.xml <<EOF
    ${contextXML}
    EOF
    ln -sf ${StaffService}/webapps $out/webapps
  '';
}

The code fragement above shows a Disnix expression for the StaffTracker example included in the Disnix repository. The main difference between this expression and an ordinary Nix expression is that it has two function headers which takes intra-dependencies and inter-dependencies respectively to configure the component. The inter-dependency arguments are used in this expression to generate a so called context XML file, which Apache Tomcat uses to configure resources such as JDBC connections, containing a URL, port number and authentication credentials for a MySQL database residing on a different machine in the network. For other types of components a different configuration file has to be created.

Moreover, you also need to compose a Disnix expression. A Disnix expression must first be composed locally by calling the function with the right intra-dependency arguments. This is done in a similar way as ordinary Nix expressions. Later, the same function is called with the right inter-dependency arguments as well.

{distribution, system, pkgs}:

let customPkgs = import ../top-level/all-packages.nix {
  inherit system pkgs;
};
in
rec {
### Databases
  staff = {
    name = "staff";
    pkg = customPkgs.staff;
    dependsOn = {};
    type = "mysql-database";
  };
  ...

### Web services
  StaffService = {
    name = "StaffService";
    pkg = customPkgs.StaffServiceWrapper;
    dependsOn = {
      inherit staff;
    };
    type = "tomcat-webapplication";
  };
  ...

### Web applications

  StaffTracker = {
    name = "StaffTracker";
    pkg = customPkgs.StaffTracker;
    dependsOn = {
      inherit GeolocationService RoomService;
      inherit StaffService ZipcodeService;
    };
    type = "tomcat-webapplication";
  };
  ...
}

The above expression shows the services model, used to capture of which distributable components a system consists. Basically, this model is a function taking three arguments: the distribution model (shown later), a collection of Nixpkgs and a system identifier indicating the architecture of a target host. The function returns an attribute set in which each attribute represents a service. For each service various properties are defined, such as a name, a pkg attribute referring to a function which creates an intra-dependency composition of a service (defined in an external file not shown here), dependsOn composing the inter-dependencies and a type, which is used for activation and deactivation of the service.

{
  test1 = {
    hostname = "test1.example.org";
    tomcatPort = 8080;
    system = "i686-linux";
  };
  
  test2 = {
    hostname = "test2.example.org";
    tomcatPort = 8080;
    mysqlPort = 3306;
    mysqlUsername = "root";
    mysqlPassword = "secret";
    system = "i686-linux";
  }; 
}

The expression shown above is an infrastructure model, which captures machines in the network and their relevant properties/capabilities. This expression is an attribute set in which each attribute represents a machine in the network. Some properties are mandatory, such as the hostname indicating how the Disnix service can be reached. The system property denotes the system architecture so that a service is built for that particular platform. Other properties can be freely chosen and are used for activation/deactivation of a component.

{infrastructure}:

{
  GeolocationService = [ infrastructure.test1 ];
  RoomService = [ infrastructure.test2 ];
  StaffService = [ infrastructure.test1 ];
  StaffTracker = [ infrastructure.test1 infrastructure.test2 ];
  ZipcodeService = [ infrastructure.test1 ];
  rooms = [ infrastructure.test2 ];
  staff = [ infrastructure.test2 ];
  zipcodes = [ infrastructure.test2 ];
}

The final expression shown above is the distribution model, mapping services to machines in the network. This expression is a function taking the infrastructure model as parameter. The body is an attribute set in which every attribute representing a service refers to a list of machines in the network. It also allows you to map a service to multiple machines for e.g. load balancing.

The models shown earlier are used by Disnix to perform the complete deployment process of a service-oriented system, i.e. building services, transferring services and the activation of services. Because Disnix uses the purely functional properties of Nix, this process is reliable and efficient. If a system is upgraded, no components are removed and overwritten, since everything is stored in isolation in the Nix store. So while upgrading, we can still keep the current system intact. Only during the transition phase in which services are deactivated and activated the system is inconsistent, but Disnix keeps this time window as small as possible. Moreover, a proxy can be used during this phase to queue connections, which makes the upgrade process truly atomic.

Although Disnix supports the deployment of a service-oriented system, some additional extensions have been developed to make deployment more convenient:

• DisnixWebService. By default Disnix uses a SSH connection to connect to remote machines in the network. This extension provides a SOAP interface and disnix-soap-client to perform deployment through the SOAP protocol.
• DisnixOS. Disnix manages the services of which a system is composed, but not the system configurations of the underlying infrastructure. This extension provides additional infrastructure management features to Disnix based on the techniques described in the blog post titled: Using NixOS for declarative deployment and testing. By using this extension you can automatically deploy a network of NixOS configurations next to the services through Disnix. Moreover, you can also use this extension to generate a network of virtual machines and automatically deploy the system in the virtual network. A screenshot is shown above, which runs the StaffTracker example in a network of three virtual machines.
• Dynamic Disnix. Disnix requires developers or system administrators to manually write an infrastructure model and a distribution model. In a network in which events occur, such as a machine which crashes or a new machine with new system resource is added, this introduces a large degree of inflexibility. The Dynamic Disnix toolset offers a discovery service, which dynamically discovers the machines in the network and their relevant properties/capabilities. Moreover, it also includes a distribution model generator, which uses a custom defined policy and a collection of distribution algorithms to dynamically distribute services to machines, based on non-functional properties defined in the services and infrastructure models.
  The Dynamic Disnix extension is still under heavy development and not released as part of Disnix 0.2. It will become part of the next Disnix release.
  

Disnix, the extensions and some examples can be obtained from the Disnix web page: http://nixos.org/disnix. Disnix is also described in several academic papers. The paper: 'Disnix: A toolset for distributed deployment' describes the architecture of the Disnix toolset. This paper is however somewhat outdated, as there are some minor changes in the current implementation. The paper: 'Automated Deployment of a Heterogeneous Service-Oriented System' describes the 0.1 implementation, which we have used for a case study at Philips Research. The publications and presentation slides can be obtained from the publications and talks sections of my homepage. Moreover, there are some earlier publications about Disnix available as well. In a next blog post, I will explain more about the development process and development choices of Disnix.

Using NixOS for declarative deployment and testing

Last weekend, I have visited FOSDEM, the Free and Open-source Software Developers' European Meeting held at the Université Libre de Bruxelles (ULB) for the third time. This is a very big event organized every year for free software and open-source people to meet each other. Also, all the major free and open-source projects that you can think of are well represented there.

It's always quite impressive to see the Janson auditorium completely filled with thousands of free and open-source people during the keynote presentations. I took a picture during the keynote of Eben Moglen of the Software Freedom Law Center (shown above) to give an impression how massive this event is.

This time I have also given a presentation about NixOS in the CrossDistro devroom, because I think it's a good idea to promote our ideas and concepts to a bigger audience next to academic people. In my presentation I have explained the complexity of deployment in various scenarios (single installation, distributed environments, virtual machines), the general idea and concepts of NixOS and also some applications we have developed the last couple of months to deal with the complexity of deployment.

One of our recently developed applications is distributed NixOS deployment. So instead of writing a single NixOS configuration, you can also write a network of NixOS configurations, for instance:

{
  storage = 
    {pkgs, config, ...}:
    {
      services.portmap.enable = true;
      services.nfsKernel.server.enable = true;
      services.nfsKernel.server.exports = ''
        /repos 192.168.1.0/255.255.255.0(rw,no_root_squash)
      '';
      services.nfsKernel.server.createMountPoints = true;
    };

  postgresql =
    {config, pkgs, ...}:
    {
      services.openssh.enable = true;
      services.postgresql.enable = true;
      services.postgresql.enableTCPIP = true;
      services.postgresql.authentication = ''
        # Generated file; do not edit!
        local all all                trust
        host  all all 127.0.0.1/32   trust
        host  all all ::1/128        trust
        host  all all 192.168.1.0/24 trust
      '';
    };

  webserver = 
    {config, pkgs, ...}:
    {
      fileSystems = pkgs.lib.mkOverride 50  
        [ { mountPoint = "/repos";
            device = "storage:/repos";
            fsType = "nfs";
            options = "bootwait"; } 
        ];
      
      services.portmap.enable = true;
      services.nfsKernel.client.enable = true;
      services.httpd.enable = true;
      services.httpd.adminAddr = "root@localhost";
      services.httpd.extraSubservices =
        [ { serviceType = "trac"; } ];
      environment.systemPackages =
        [ pkgs.pythonPackages.trac pkgs.subversion ];
    };
      
  client = 
    {config, pkgs, ...}:
    {
      require = [ ./common/x11.nix ];
      services.xserver.desktopManager.kde4.enable = true;
    };
};

The network expression shown above represents a network of machines describing a Trac environment, a web-based management tool for software projects. A Trac environment can be (of course) deployed on a single system, but also on multiple systems. For example, the Subversion server storing source code may be deployed on a different machine as the PostgreSQL database storing tickets and bug reports. In the network expression shown above, we have defined 4 machines, representing a Subversion server, PostgreSQL server, web server and a client machine running the KDE Plasma desktop.

A network of NixOS machines can be automatically deployed by writing a network expression and by typing:

$ nixos-deploy-network network.nix

The nixos-deploy-network tool first builds all NixOS configurations for all the machines. Then it efficiently transfers the system configurations and all its dependencies to the right machines in the network. Because of the purely functional properties of Nix, this phase will not harm the existing configurations because all files are stored safely next to each other in the Nix store and no files are overwritten or automatically removed.

After all system configurations and dependencies have been transferred to the target machines, the system configurations are activated. In this phase system services are stopped and started on each machine in the network, which may bring some downtime to the complete system, but this time window is a small as possible. In case of a failure, a rollback is performed which activates the previous configuration again. This can be done easily in NixOS, since older configurations are still available in the Nix store, unless they are garbage collected.

Another recently developed application is virtualization. By running the following command:

$ nixos-build-vms; ./result/bin/nixos-run-vms

A network of virtual machines is generated and automatically launched closely resembling the configurations defined in the network model. This allows users to experiment with a specific configuration, without having to deploy a collection of physical machines. Another notable feature is that virtual networks are cheap to instantiate. We don't have to create disk images, but instead we mount the Nix store of the host machine through SMB/CIFS. We can safely do this because of the purely functional concept of the Nix store. An impression of a virtual network running Trac is shown above.

We have also developed a NixOS test driver. This can be used to perform automatic distributed test cases in a network of virtual machines.

testScript =
''
  startAll;
      
  $postgresql->waitForJob("postgresql");
  $postgresql->mustSucceed("createdb trac");
      
  $webserver->mustSucceed("mkdir -p /repos/trac");
  $webserver->mustSucceed("svnadmin create /repos/trac");
      
  $webserver->waitForFile("/var/trac");      
  $webserver->mustSucceed("mkdir -p /var/trac/projects/test");
  $webserver->mustSucceed("trac-admin /var/trac/projects/test ".
    "initenv Test postgres://root\@postgresql/trac svn ".
    "/repos/trac");
      
  $client->waitForX;
  $client->execute("konqueror http://webserver/projects/test &");
  $client->waitForWindow(qr/Test.*Konqueror/);
  $client->sleep(30); # loading takes a long time
      
  $client->screenshot("screen");
'';

The code fragment above shows an example of a test suite for the Trac environment. This test suite creates a Trac database on the PostgreSQL server, a Subversion repository on the Subversion repository, then it defines a Trac project using the trac-admin tool and launches a web browser to take a screenshot of the entry page. The test suite is performed by the test driver in a non-interactive manner.

We have applied the distributed deployment and testing techniques to various use cases. We used the nixos-deploy-network tool to deploy our complete Hydra build environment for continuous integration and testing of many software components. We also have implemented various test cases for NixOS, various GNU projects and other free software projects using the NixOS test driver.

I am quite happy to see how well the ideas described in my presentation were received at FOSDEM. It was the first time for me to present there and I didn't really know what to expect. It seems that my talk attracted quite a number of people and I received quite a number of positive reactions and a lot of good questions and suggestions. I have to admit that these questions were far better than the ones I usually receive at academic conferences.

The slides of my FOSDEM presentation (titled: 'Using NixOS for declarative deployment and testing') can be obtained from the talks page of my homepage. The distributed testing techniques are also described in our ISSRE 2010 paper titled: 'Automating System Tests Using Declarative Virtual Machines'. The technical report titled: 'Declarative Testing and Deployment of Distributed Systems' describes an earlier implementation of our declarative deployment and testing techniques. Both papers can be downloaded from my publications page.

The techniques are part of NixOS. The NixOS test driver can also be used on any Linux system running the Nix package manager and KVM, which allows you to still use your favourite Linux distribution if you don't want to switch to NixOS.

NixOS: A purely functional Linux distribution

My last blog post was about the Nix package manager, a package manager with some distinct features, such as multiple versions/variants packages which are stored in isolation from each other in a Nix store, the Nix expression language and atomic upgrades and rollbacks. I also briefly mentioned NixOS, a Linux distribution using the Nix package manager as a basis. In this blog post I will discuss the concepts of NixOS more thoroughly.

It's probably already very obvious that NixOS uses the Nix package manager for installing, upgrading and removing software components. In fact, except for end-user software such as Mozilla Firefox and OpenOffice.org, development tools such as GCC, Binutils, also all operating system parts such as the Linux kernel, system configuration files and kernel modules are all managed by the Nix package manager and stored in isolation in the Nix store.

Because of the fact that Nix stores all packages in a special directory: /nix/store, the NixOS distribution does not have standard FHS directories such as: /usr and /lib and only a minimal /etc and /bin. The /bin directory only contains a symlink: /bin/sh pointing to the bash shell. This is required in order to make glibc's system() function working properly. I have uploaded a screenshot (seen above) in which a terminal is shown, listing the directory contents of the root filesystem, /bin folder and some other files. As you may notice, certain directories commonly found on other distributions don't exist, and the /bin/sh points to the bash shell which is also stored somewhere in the Nix store.

The fact that Nix is used instead of a conventional package manager such as RPM or dpkg, does probably not sound so exciting to most users. Actually, NixOS is more then just a Linux distribution managed by Nix. The main thing that sets NixOS apart from conventional Linux distributions is that system configurations are also configured with declarative specifications just Nix like packages. Instead of manually installing packages and manually editing configurations files, such as an Apache web server configuration, everything is configured from a single declarative specification, invoking functions which generate configuration files and install required packages.

{pkgs, ...}:

{                                                                                                                                                            
  boot.loader.grub.device = "/dev/sda";
                                                
  fileSystems = [
    { mountPoint = "/";
      device = "/dev/sda2";
    }
  ];

  swapDevices = [ { device = "/dev/sda1"; } ];
  
  services = {
    openssh.enable = true;

    httpd = {
      enable = true;
      documentRoot = "/var/www";
      adminAddr = "admin@localhost";
    };

    xserver = {
      enable = true;
      
      desktopManager = {
        default = "kde4";
        kde4 = { enable = true; };
      };
      
      displayManager = {
        kdm = { enable = true; };
        slim = { enable = false; };
      };
    };
  };
  
  environment.systemPackages = [
    pkgs.mc
    pkgs.subversion
    pkgs.firefox
  ];
}

The code fragment above shows an example of a NixOS configuration. As you may notice, this configuration file contains all kinds of settings for a particular system, such as the MBR partition for the GRUB bootloader, filesystems (such as the root and swap partition), system services (OpenSSH, Apache Webserver and X.Org configured to use the KDE desktop and the KDM login manager). Also, system packages are defined here such as Mozilla Firefox and Midnight Commander, which are automatically in the PATH of all users and also appear automatically in the KDE menu if desktop files are included.

By storing a configuration file in /etc/nixos/configuration.nix and by running: nixos-rebuild switch a complete system configuration is built by the Nix package manager and activated. Because of the fact that the Nix package manager is used, we can safely upgrade a complete system; this is because the complete system configuration and all its dependencies are safely stored in the Nix store and no files are overwritten or removed. Because of this we can also rollback to any previous configuration, which has not been garbage collected yet.

The figure above shows the GRUB boot loader used for NixOS. As you may notice, all the previous configurations of the system are shown, which are not garbage collected yet. The GRUB boot loader of NixOS allows you to boot into any previous configuration (You can also switch back to a previous configuration while running NixOS).

The last missing piece in the puzzle is how configuration files are handled in NixOS. In NixOS you don't manually edit configuration files. Instead, configuration files are generated by Nix expressions and stored in the Nix store (just like ordinary packages). Because they are generated by Nix expressions and they also may contain paths to Nix expressions containing binaries, this also has a nice implication. If a configuration file contains a Nix store path, Nix handles them as dependencies, and automatically includes them if a configuration is deployed somewhere.

Maybe you have become interested in trying out NixOS after reading this. ISOs can be obtained from the NixOS website: http://nixos.org/nixos. In NixOS quite a number of packages are supported. We also have end-user software available and the KDE desktop. Keep in mind that NixOS is an experimental Linux distribution, which may not offer everything you need. We only have a small community of ~25 developers maintaining it.

If you find some concepts of NixOS useful but you don't want to switch, you can also use the virtualization features of NixOS to automatically generate virtual machines running a particular NixOS configuration. You can even do this on regular Linux distributions supporting KVM having the Nix package manager installed.

One point of criticism I always receive is about the fact that we don't edit configuration files manually. I have to admit that this is a bit inconvenient, if you want to try out a particular feature of a service. But manually editing stuff becomes very inconvenient in a distributed setting. Probably everyone agrees that it is unfeasible in case of a configuration change, to manually update the configurations of every affected machine. One of the things we investigate in the PDS research project is expanding the Nix deployment vision to distributed systems.

There are some publications about NixOS available. The first prototype of NixOS was implemented in 2006 by Armijn Hemel for his masters thesis. Nicolas Pierron implemented the NixOS module system. Various publications about NixOS can be obtained from Eelco Dolstra's publications page.

The Nix package manager

In my previous blog post, which was about Pull Deployment of Services, I have explained what the project I'm working on as a PhD student is about. An import foundation is the Nix package manager, which serves as an important basis of my research.

There are dozens of papers and websites available describing Nix, so in this blog post I will not give another extensive description. Instead, I'm going to explain some of the details and the vision that we have about software deployment which sets the Nix approach apart from conventional approaches.

Nix is a package manager developed by Eelco Dolstra as part of his PhD research, which has similar purposes as tools such as RPM and dpkg commonly found in many Linux distributions, such as Red Hat enterprise Linux, Fedora, Debian and Ubuntu.

A package manager is basically a collection of software tools to automate the process of installing, upgrading, configuring, and removing software packages for an operating system in a consistent manner. In many Linux distributions, the package manager is one of the major factors that distinguish them from each other. Similarly, an experimental Linux distribution has been built around Nix, called NixOS using the Nix package manager as its basis.

One of the major differences of Nix compared to conventional deployment tooling, is the way Nix addresses packages (or components). Most tools use nominal dependency specifications, consisting of a name and version number, such as:

openssl-1.0.0c

Although it is a good thing to have these attributes to distinguish packages (which for instance isn't properly done for native Windows DLLs), this dependency specification mechanism has some limitations. A problem may occur when OpenSSL is compiled with a different version of GCC as an executable linking to it. A library compiled with an older version of GCC cannot always be linked to an executable compiled with a newer version of GCC, due to a different ABI. Moreover, OpenSSL also has dependencies on libraries, such as glibc. If OpenSSL is compiled against an incompatible older version of glibc as an executable, also problems may occur.

In Nix, we have different way of addressing components. We store components in a special directory called the Nix store in which every component is stored under an unique filename, such as:

/nix/store/xq2bfcqdsbrmfr8h5ibv7n1qb8xs5s79-openssl-1.0.0c

The former part of the component name: xq2bfcqdsbrmfr8h5ibv7n1qb8xs5s79 is a SHA256 hash code derived from all build time dependencies to build the component from source code. For example, if the same component is built with a different version of GCC, a different hash code is generated.

Using hash codes to address components offers us some nice advantages. Since every component has an unique filename, we can safely store multiple versions and variants next to each other, without overwriting each other. This also offers us the possibility to upgrade a system atomically. In case of an upgrade a new component is safely stored next to an existing one, and after the upgrade only a symlink pointing to the configuration by a user is changed.

The hash codes in the component names are also used to detect runtime dependencies of a component. This is done by scanning for occurrences of a hash inside a component. If a hash code is found (such as a path containing a Nix store component in an ELF header, or a shell script containing a Nix store path) Nix identifies these components as a runtime dependency. Although this technique sounds risky/scary, we have used this for many packages and it turns out to work quite well.

Another key feature is the Nix expression language, which we use to build packages, as shown in the code fragment below.

{stdenv, fetchurl}:

stdenv.mkDerivation {
  name = "hello-2.6";
  
  src = fetchurl {
    url = ftp://ftp.gnu.org/gnu/hello/hello-2.6.tar.gz;
    sha256 = "1h6fjkkwr7kxv0rl5l61ya0b49imzfaspy7jk9jas1fil31sjykl";
  };

  meta = {
    homepage = http://www.gnu.org/software/hello/manual/;
    license = "GPLv3+";
  };
}

Basically each package description is a function which takes some dependencies as arguments (in this case stdenv and fetchurl). In the body of the function we call the mkDerivation function describing a build action, which takes some arguments, such as a reference to the source code and other relevant build properties. In this expression we did not specify how to build this component. By default, Nix assumes that a packages is autotools based (i.e. ./configure; make; make install) if build instructions are omitted.

The expression described earlier cannot be used to build a package directly. The function must be called with the right arguments, i.e. a component must be composed. This is done in another expression, shown below:

rec {
  stdenv = ...;

  fetchurl = import ../build-support/fetchurl {
    inherit stdenv curl;
  };

  hello = import ../applications/misc/hello {
    inherit stdenv fetchurl;
  }

  ...
}

In this expression we call the function building the GNU Hello component (shown in the previous expression) with its required arguments. As you may notice, all the dependencies of the GNU Hello package are composed in this expression as well.

Now you may think that this dependency scheme of using hash codes to address components is a bit unhandy for end users. To solve this problem, we use Nix profiles. A Nix profile is basically an environment in the PATH of a user, containing symlinks indirectly referring to components in the Nix store. For example, by typing:

$ nix-env -i hello

The GNU Hello component is installed into the Nix profile of the current user. By typing the following instruction on the command-line:

$ hello

The GNU Hello component installed earlier is executed.

The last distinguishable feature I'd like to mention is that in Nix packages are not directly removed, but removed by a garbage collector. The Nix garbage collector will safely remove components that are no longer in use. In most conventional tooling you may accidentally remove a package which may be still in use. Packages which are still installed in a profile of a user or which are a dependency of another package, will not be removed by Nix. Moreover, running processes and open files are also garbage collector roots, which makes it safe to run the garbage collector at any time.

After reading this blog post, you may have become interested in trying the Nix package manager. The simplest way to try out Nix is by downloading and installing NixOS. Although NixOS is a Linux distribution built around Nix, you can also use Nix on conventional Linux distributions, if you find it more convenient to keep using the operating system you are used to. Moreover, the Nix package manager is quite portable and can also be used on different operating systems, such as FreeBSD, OpenSolaris, Mac OS X and Windows (through Cygwin).

Except for plain package management and the management of a Linux distribution, Nix is also used as foundation for some other tooling. Hydra is a continuous build and integration server built on top of Nix, which allows you to continuously checkout sources from repositories and to build components in several variations (e.g. using various compilers, libraries and platforms). I have developed an extension called Disnix, which extends the Nix package manager for the deployment of distributable components (or services) into a network of machines.

Another thing I want to clarify is that I don't want to claim that other package management solutions (and deployment tooling) are bad solutions. In fact, I think other package management solutions have done a great job in dealing with certain installation issues. Moreover, I find package management tools one the strong points of Linux distributions, although we support some distinct features which we care about, which other package managers don't.

We are not the only group doing research in package management. There is also a project, called Mancoosi, in which package and system management issues are investigated. They have a different view about certain principles, such as keeping maintainer scripts which imperatively modify the system configuration (whereas we generate everything as packages in the Nix store). Moreover, their solutions are based on package management tools widely used in Linux distributions, such as RPM and dpkg.

For more information about Nix, have a look at the Nix website: http://nixos.org/nix or have a look at Eelco's excellent PhD thesis titled: 'The Purely Functional Software Deployment Model', which can be found on his publications page.