On Nix, NixOS and the Filesystem Hierarchy Standard (FHS)

Our work on NixOS has been topic of discussion within various free software/open source projects and websites. For example, we have been discussed on the Debian mailing list and on Linux Weekly News.

One of the criticisms I often receive is that we don't comply with the Filesystem Hierarchy Standard (FHS). In this blog post, I'd like to give a response on the FHS and why NixOS deviates from it.

What is the Filesystem Hierarchy Standard?

The purpose of the Filesystem Hierarchy Standard (FHS) is to define the main directories and their contents in Linux operating systems. More specifically:

• It defines directory names and their purposes. For example, the bin directory is for storing executable binaries, sbin is for storing executable binaries only accessible by the super-user, lib is for storing libraries.
• It defines several hierarchies within the filesystem, which have separate purposes:
  
  The primary / hierarchy contains essential components for the boot process and system recovery. The secondary hierarchy: /usr contains components not relevant for booting and recovery. Moreover, files in this directory should be shareable across multiple machines, e.g. through a network filesystem. The tertiary hierarchy: /usr/local is for the system administrator to install software locally.
  
  Hierarchies and special purpose directories are usually combined. For example, the /bin directory contains executable binaries relevant for booting and recovery which can be used by anyone. The /usr/bin directory contains executable binaries not relevant for booting or recovery, such as a web browser, which can be shared across multiple machines. The /usr/local/bin directory contains local executables installed by the system administrator.
  
  The FHS also defines the /opt directory for installing add-on application software packages. This directory can also be considered a separate hierarchy, although it is not defined as such in the standard. This directory also contains the same special purpose directories, such as bin and lib like the primary, secondary and tertiary hierarchies. Furthermore, also the /opt/<package name> convention may be used, which stores files specific to a particular application in a single folder.
  
• It also makes a distinction between static and variable parts of a system. For example, the contents of the secondary hierarchy /usr are static and could be stored on a read-only mounted partition. However, many programs need to modify their state at runtime and store files on disks, such as cache and log files. These files are stored in variable storage directories, such as /var and /tmp.
  
• It defines what the contents of some folders should look like. For example, a list of binaries which should reside in /bin, such as a Bourne compatible shell.
  
• The standard has some practical problems. Some aspects of the filesystem are undefined and need clarification, such as a convention to store cross compiling libraries. Furthermore, the standard is quite old and newer Linux features such as the /sys directory are not defined in the standard. To cope with these issues, many distributions have additional clarifications and policies, such as the Debian FHS policy.

Linux Standard Base

The Filesystem Hierarchy Standard (FHS) is part of another standard: The Linux Standard Base (LSB), which incorporates and extends several other standards, such as POSIX and the Single UNIX specification.

The LSB standard has been developed, because there are many Linux based systems out there. What they all have in common is that they run the Linux kernel, and quite often a set of GNU utilities (that's why the free software foundation advocates GNU/Linux as the official name for these kind of systems).

Apart from some similarities, there are many ways in which these systems differ from each other, such as the package manager which is being used, the way the file system is organized, the software which is supported etc.

Because there are many Linux systems available, the Linux Standards Base is designed to increase compatibility among these Linux distributions, so that it becomes easier for software vendors to ship and run Linux software, even in binary form. Many of the common Linux distributions such as Debian, SuSE and Fedora try to implement this standard, although it has several issues and criticisms.

NixOS and the FHS

In NixOS, we have an informal policy to follow the FHS as closely as possible, but to deviate from it where necessary. An important aspect is that we can't follow the structure of the primary /, secondary /usr and tertiary hierarchies: /usr/local.

The main reason to refrain from using these hierarchies is that they don't provide isolation. For example the /usr/lib directory contains a wide collection of shared libraries belonging to many packages. To determine to which package a particular library belongs, you need to have a look in the database of the package manager. Because most libraries can be found in the same location, e.g. /usr/lib, it is very tempting to forget specifying a dependency, because they can still be implicitly found.

The purpose of the Nix package manager is to achieve purity, which means that the build result of a package should exclusively depends on the source code and input parameters. Purity ensures that the build of a package can be reproduced anywhere and that a package can be transferred to any machine we want, with the guarantee that the dependencies are present and correct.

Nix achieves purity by using the filesystem as a database and to store packages in isolation in a special directory called the Nix store. For example:

/nix/store/r8vvq9kq18pz08v249h8my6r9vs7s0n3-firefox-8.0.1

The path above refers to the Mozilla Firefox web browser. The first part of the directory name: r8vvq9kq18pz08v249h8my6r9vs7s0n3 is a hash-code derived from all build time dependencies. By using this naming convention, components can be stored safely in isolation from each other, because no component shares the same name. If Firefox is compiled with a different version of GCC or linked to a different version of the GTK+ library, the hash code will differ and thus not interfere with another variant.

Because of the naming scheme of the Nix store and the fact that we don't use global directories to store binaries and libraries, the build of a Nix package will typically fail if a required build time dependency is omitted. Furthermore, it also restricts undeclared dependencies which may allow a build to accidentally succeed (and therefore also prevents incomplete dependency specifications).

Another deviation of the FHS, is not following the list of required binaries and libraries in the /bin, /lib etc. directories. For example, the FHS requires a Bourne compatible shell in /bin/sh, some mkfs.* utilities in /sbin and for historic reasons, a sendmail executable in /usr/lib.

In NixOS, apart from the /bin/sh executable (which is a symlink to the bash shell in the Nix store), we don't store any binaries in these mandatory locations because it is not necessary and it also makes builds impure.

Discussion

Now that I have explained why we deviate from the FHS on some points, you probably may wonder, how we achieve certain properties defined in the FHS in NixOS:

• How to determine files for the boot process and recovery? Because we don't use hierarchies, files relevant for the boot process can't be found in /bin and /lib. In NixOS, we basically generate a small base system as a Nix component serving this purpose, which is stored as a single component in the Nix store.
  
• How to share components? Because we have no secondary hierarchy, you can't share components by storing the /usr directory on a network file system. In Nix, however, the entire Nix store is static and shareable. Furthermore, it's even more powerful, because the hash codes inside the component names prevent different variants of NixOS to use the incorrect versions of a component. For example, you can safely share the same Nix store across 32-bit and 64-bit machines, because this parameter is reflected in the hash.
  
  In fact, we already use sharing extensively in our build farm, to generate virtual machines to perform testcases. Apart from a Linux kernel image and an initial RAM disk, the components of the entire virtual network are shared through the host system's Nix store, which is mounted as a network file system.
• How to find dependencies? Because we store all software packages in the Nix store, it is harder to address components in e.g. scripts and configuration files, because all these components are stored in separate directories, and aren't necessarily in the user's PATH.
  
  I have to admit that this is inconvenient in some cases, however in NixOS we usually don't manually edit configuration files and write scripts.
  NixOS is a distribution that is unconventional in this sense, because its goal is to make the entire deployment process model-driven. Instead of manually installing packages and adapting configuration files in e.g. /etc. In NixOS, we generate all the static parts of a system from Nix expressions, including all configuration files and scripts. The Nix expression language provides the paths to the components.

Improvements

Although we have to deviate from the FHS for a number of reasons, there are also a few improvements we could make in the organization of the filesystem:

• The directory names and purposes within the hierarchies can be more consistently used within Nix packages. Essentially, you could say that every component in the Nix store is a separate hierarchy.
  
  For example, I think it's a good thing to use /nix/store/<package>/bin for storing binaries, and /nix/store/<package>/lib for libraries. Sometimes installation scripts of packages do not completely adhere to the FHS and need some fixing. For example, some packages install libraries in the libexec directory, which is not defined by the FHS.
  
  We could make an additional check in the generic builder of Nixpkgs, checking the structure of a package and to report inconsistencies.
  
• For the variable parts of the system, we can adhere better to the FHS. For example, the current version of the FHS defines the /srv directory used for serving files to end-users, through a service (e.g. a web server). In NixOS, this directory is not used.
• Because the FHS standard has some gaps, we could also define some additional clarification, like other Linux distributions do.

Conclusion

In this blog post I've explained the idea behind the Filesystem Hierarchy Standard (FHS) and I have explained why we deviate from it in NixOS. The main reason is that the organization of specific parts of the file system, conflict with important aspects of the Nix package manager, such as the ability to store components isolation and to guarantee correct and complete dependencies.

Furthermore, apart from the FHS, NixOS cannot completely implement other parts of the LSB either. For example, in the LSB a subset of the RPM package manager is defined as default package manager, which imperatively modifies the state of a system.

If we would implement this in NixOS, we can no longer ensure that the deployment of a system is pure and reproducible. Therefore, we sometimes have to break from the traditional organization and management of Linux systems. However, sometimes I get the impression that the FHS is considered a holy book and by breaking from it, we are considered heretics.

Related work

• GoboLinux is also a Linux distribution not obeying the FHS, with a custom filesystem organization. They also have the vision that the filesystem can act as a database for organizing packages.
• The Mancoosi project is also a research project investigating package management related problems, like we do. Their research differs from us, because they don't break away from the traditional filesystem organization and management of Linux systems. For example, a lot of their research deals with maintainer scripts of packages, which goal is to "glue" files from a package to files on the system, e.g. by imperatively modifying configuration files.
  
  By keeping the traditional management intact, this introduces a whole bunch of challenges to make deployment efficient, reliable and reproducible. For example, they have investigated modeling techniques for maintainer scripts and system configurations, to simulate whether an upgrade succeed and to determine inverse operations to perform rollbacks.
  
  In our research, we achieve more reliable deployment by breaking from the traditional model.
  

Using NixOS for declarative deployment and testing

Last weekend, I have visited FOSDEM, the Free and Open-source Software Developers' European Meeting held at the Université Libre de Bruxelles (ULB) for the third time. This is a very big event organized every year for free software and open-source people to meet each other. Also, all the major free and open-source projects that you can think of are well represented there.

It's always quite impressive to see the Janson auditorium completely filled with thousands of free and open-source people during the keynote presentations. I took a picture during the keynote of Eben Moglen of the Software Freedom Law Center (shown above) to give an impression how massive this event is.

This time I have also given a presentation about NixOS in the CrossDistro devroom, because I think it's a good idea to promote our ideas and concepts to a bigger audience next to academic people. In my presentation I have explained the complexity of deployment in various scenarios (single installation, distributed environments, virtual machines), the general idea and concepts of NixOS and also some applications we have developed the last couple of months to deal with the complexity of deployment.

One of our recently developed applications is distributed NixOS deployment. So instead of writing a single NixOS configuration, you can also write a network of NixOS configurations, for instance:

{
  storage = 
    {pkgs, config, ...}:
    {
      services.portmap.enable = true;
      services.nfsKernel.server.enable = true;
      services.nfsKernel.server.exports = ''
        /repos 192.168.1.0/255.255.255.0(rw,no_root_squash)
      '';
      services.nfsKernel.server.createMountPoints = true;
    };

  postgresql =
    {config, pkgs, ...}:
    {
      services.openssh.enable = true;
      services.postgresql.enable = true;
      services.postgresql.enableTCPIP = true;
      services.postgresql.authentication = ''
        # Generated file; do not edit!
        local all all                trust
        host  all all 127.0.0.1/32   trust
        host  all all ::1/128        trust
        host  all all 192.168.1.0/24 trust
      '';
    };

  webserver = 
    {config, pkgs, ...}:
    {
      fileSystems = pkgs.lib.mkOverride 50  
        [ { mountPoint = "/repos";
            device = "storage:/repos";
            fsType = "nfs";
            options = "bootwait"; } 
        ];
      
      services.portmap.enable = true;
      services.nfsKernel.client.enable = true;
      services.httpd.enable = true;
      services.httpd.adminAddr = "root@localhost";
      services.httpd.extraSubservices =
        [ { serviceType = "trac"; } ];
      environment.systemPackages =
        [ pkgs.pythonPackages.trac pkgs.subversion ];
    };
      
  client = 
    {config, pkgs, ...}:
    {
      require = [ ./common/x11.nix ];
      services.xserver.desktopManager.kde4.enable = true;
    };
};

The network expression shown above represents a network of machines describing a Trac environment, a web-based management tool for software projects. A Trac environment can be (of course) deployed on a single system, but also on multiple systems. For example, the Subversion server storing source code may be deployed on a different machine as the PostgreSQL database storing tickets and bug reports. In the network expression shown above, we have defined 4 machines, representing a Subversion server, PostgreSQL server, web server and a client machine running the KDE Plasma desktop.

A network of NixOS machines can be automatically deployed by writing a network expression and by typing:

$ nixos-deploy-network network.nix

The nixos-deploy-network tool first builds all NixOS configurations for all the machines. Then it efficiently transfers the system configurations and all its dependencies to the right machines in the network. Because of the purely functional properties of Nix, this phase will not harm the existing configurations because all files are stored safely next to each other in the Nix store and no files are overwritten or automatically removed.

After all system configurations and dependencies have been transferred to the target machines, the system configurations are activated. In this phase system services are stopped and started on each machine in the network, which may bring some downtime to the complete system, but this time window is a small as possible. In case of a failure, a rollback is performed which activates the previous configuration again. This can be done easily in NixOS, since older configurations are still available in the Nix store, unless they are garbage collected.

Another recently developed application is virtualization. By running the following command:

$ nixos-build-vms; ./result/bin/nixos-run-vms

A network of virtual machines is generated and automatically launched closely resembling the configurations defined in the network model. This allows users to experiment with a specific configuration, without having to deploy a collection of physical machines. Another notable feature is that virtual networks are cheap to instantiate. We don't have to create disk images, but instead we mount the Nix store of the host machine through SMB/CIFS. We can safely do this because of the purely functional concept of the Nix store. An impression of a virtual network running Trac is shown above.

We have also developed a NixOS test driver. This can be used to perform automatic distributed test cases in a network of virtual machines.

testScript =
''
  startAll;
      
  $postgresql->waitForJob("postgresql");
  $postgresql->mustSucceed("createdb trac");
      
  $webserver->mustSucceed("mkdir -p /repos/trac");
  $webserver->mustSucceed("svnadmin create /repos/trac");
      
  $webserver->waitForFile("/var/trac");      
  $webserver->mustSucceed("mkdir -p /var/trac/projects/test");
  $webserver->mustSucceed("trac-admin /var/trac/projects/test ".
    "initenv Test postgres://root\@postgresql/trac svn ".
    "/repos/trac");
      
  $client->waitForX;
  $client->execute("konqueror http://webserver/projects/test &");
  $client->waitForWindow(qr/Test.*Konqueror/);
  $client->sleep(30); # loading takes a long time
      
  $client->screenshot("screen");
'';

The code fragment above shows an example of a test suite for the Trac environment. This test suite creates a Trac database on the PostgreSQL server, a Subversion repository on the Subversion repository, then it defines a Trac project using the trac-admin tool and launches a web browser to take a screenshot of the entry page. The test suite is performed by the test driver in a non-interactive manner.

We have applied the distributed deployment and testing techniques to various use cases. We used the nixos-deploy-network tool to deploy our complete Hydra build environment for continuous integration and testing of many software components. We also have implemented various test cases for NixOS, various GNU projects and other free software projects using the NixOS test driver.

I am quite happy to see how well the ideas described in my presentation were received at FOSDEM. It was the first time for me to present there and I didn't really know what to expect. It seems that my talk attracted quite a number of people and I received quite a number of positive reactions and a lot of good questions and suggestions. I have to admit that these questions were far better than the ones I usually receive at academic conferences.

The slides of my FOSDEM presentation (titled: 'Using NixOS for declarative deployment and testing') can be obtained from the talks page of my homepage. The distributed testing techniques are also described in our ISSRE 2010 paper titled: 'Automating System Tests Using Declarative Virtual Machines'. The technical report titled: 'Declarative Testing and Deployment of Distributed Systems' describes an earlier implementation of our declarative deployment and testing techniques. Both papers can be downloaded from my publications page.

The techniques are part of NixOS. The NixOS test driver can also be used on any Linux system running the Nix package manager and KVM, which allows you to still use your favourite Linux distribution if you don't want to switch to NixOS.

NixOS: A purely functional Linux distribution

My last blog post was about the Nix package manager, a package manager with some distinct features, such as multiple versions/variants packages which are stored in isolation from each other in a Nix store, the Nix expression language and atomic upgrades and rollbacks. I also briefly mentioned NixOS, a Linux distribution using the Nix package manager as a basis. In this blog post I will discuss the concepts of NixOS more thoroughly.

It's probably already very obvious that NixOS uses the Nix package manager for installing, upgrading and removing software components. In fact, except for end-user software such as Mozilla Firefox and OpenOffice.org, development tools such as GCC, Binutils, also all operating system parts such as the Linux kernel, system configuration files and kernel modules are all managed by the Nix package manager and stored in isolation in the Nix store.

Because of the fact that Nix stores all packages in a special directory: /nix/store, the NixOS distribution does not have standard FHS directories such as: /usr and /lib and only a minimal /etc and /bin. The /bin directory only contains a symlink: /bin/sh pointing to the bash shell. This is required in order to make glibc's system() function working properly. I have uploaded a screenshot (seen above) in which a terminal is shown, listing the directory contents of the root filesystem, /bin folder and some other files. As you may notice, certain directories commonly found on other distributions don't exist, and the /bin/sh points to the bash shell which is also stored somewhere in the Nix store.

The fact that Nix is used instead of a conventional package manager such as RPM or dpkg, does probably not sound so exciting to most users. Actually, NixOS is more then just a Linux distribution managed by Nix. The main thing that sets NixOS apart from conventional Linux distributions is that system configurations are also configured with declarative specifications just Nix like packages. Instead of manually installing packages and manually editing configurations files, such as an Apache web server configuration, everything is configured from a single declarative specification, invoking functions which generate configuration files and install required packages.

{pkgs, ...}:

{                                                                                                                                                            
  boot.loader.grub.device = "/dev/sda";
                                                
  fileSystems = [
    { mountPoint = "/";
      device = "/dev/sda2";
    }
  ];

  swapDevices = [ { device = "/dev/sda1"; } ];
  
  services = {
    openssh.enable = true;

    httpd = {
      enable = true;
      documentRoot = "/var/www";
      adminAddr = "admin@localhost";
    };

    xserver = {
      enable = true;
      
      desktopManager = {
        default = "kde4";
        kde4 = { enable = true; };
      };
      
      displayManager = {
        kdm = { enable = true; };
        slim = { enable = false; };
      };
    };
  };
  
  environment.systemPackages = [
    pkgs.mc
    pkgs.subversion
    pkgs.firefox
  ];
}

The code fragment above shows an example of a NixOS configuration. As you may notice, this configuration file contains all kinds of settings for a particular system, such as the MBR partition for the GRUB bootloader, filesystems (such as the root and swap partition), system services (OpenSSH, Apache Webserver and X.Org configured to use the KDE desktop and the KDM login manager). Also, system packages are defined here such as Mozilla Firefox and Midnight Commander, which are automatically in the PATH of all users and also appear automatically in the KDE menu if desktop files are included.

By storing a configuration file in /etc/nixos/configuration.nix and by running: nixos-rebuild switch a complete system configuration is built by the Nix package manager and activated. Because of the fact that the Nix package manager is used, we can safely upgrade a complete system; this is because the complete system configuration and all its dependencies are safely stored in the Nix store and no files are overwritten or removed. Because of this we can also rollback to any previous configuration, which has not been garbage collected yet.

The figure above shows the GRUB boot loader used for NixOS. As you may notice, all the previous configurations of the system are shown, which are not garbage collected yet. The GRUB boot loader of NixOS allows you to boot into any previous configuration (You can also switch back to a previous configuration while running NixOS).

The last missing piece in the puzzle is how configuration files are handled in NixOS. In NixOS you don't manually edit configuration files. Instead, configuration files are generated by Nix expressions and stored in the Nix store (just like ordinary packages). Because they are generated by Nix expressions and they also may contain paths to Nix expressions containing binaries, this also has a nice implication. If a configuration file contains a Nix store path, Nix handles them as dependencies, and automatically includes them if a configuration is deployed somewhere.

Maybe you have become interested in trying out NixOS after reading this. ISOs can be obtained from the NixOS website: http://nixos.org/nixos. In NixOS quite a number of packages are supported. We also have end-user software available and the KDE desktop. Keep in mind that NixOS is an experimental Linux distribution, which may not offer everything you need. We only have a small community of ~25 developers maintaining it.

If you find some concepts of NixOS useful but you don't want to switch, you can also use the virtualization features of NixOS to automatically generate virtual machines running a particular NixOS configuration. You can even do this on regular Linux distributions supporting KVM having the Nix package manager installed.

One point of criticism I always receive is about the fact that we don't edit configuration files manually. I have to admit that this is a bit inconvenient, if you want to try out a particular feature of a service. But manually editing stuff becomes very inconvenient in a distributed setting. Probably everyone agrees that it is unfeasible in case of a configuration change, to manually update the configurations of every affected machine. One of the things we investigate in the PDS research project is expanding the Nix deployment vision to distributed systems.

There are some publications about NixOS available. The first prototype of NixOS was implemented in 2006 by Armijn Hemel for his masters thesis. Nicolas Pierron implemented the NixOS module system. Various publications about NixOS can be obtained from Eelco Dolstra's publications page.