Friday, September 13, 2013

Composing FHS-compatible chroot environments with Nix (or deploying Steam in NixOS)

As described in a blog post from a while ago, NixOS deviates on certain aspects from the Filesystem Hierarchy Standard (FHS) for very good reasons. For example, NixOS lacks standard directories such as /usr/bin and /usr/lib that conventional Linux distributions have. The most important reason that Nix/NixOS deviates from the FHS is to provide reliable and reproducible deployment of packages.

The filesystem organization that Nix uses, in which every package is stored in a separate folder in a so called "Nix store", is typically not a problem to get software packaged and deployed in source form.

Packages in Nix are deployed from build recipes called Nix expressions that typically invoke functions implementing abstractions ensuring that the specified dependencies can be found in a convenient manner.

For example, when providing Java or Perl dependencies, the generic builder may automatically set some required environment variables, such CLASSPATH or PERL5LIB containing Java or Perl dependencies. Moreover, a number of build utilities have been patched (such as ld) to automatically add shared libraries to an ELF executable's RPATH allowing it to find the libraries that it needs (you could see this as as static linking of shared libraries to an executable).

However, deployment of third party binary packages is a bit complicated and inconvenient in NixOS for the following reasons:

  • Nearly all software must be deployed by the Nix package manager, since nearly all of the system's components reside in the Nix store.
  • Third party binaries frequently require shared libraries and other kinds dependencies to run. However, in most cases these dependencies cannot be found, because executables assume that they reside in standard FHS folders such as /usr/lib, which do not exist in NixOS.
  • Executables using dynamic libraries cannot find the dynamic linker (e.g. /lib/, because the corresponding file does not exist in NixOS.

As a solution, we often use PatchELF to change the path to the dynamic linker and modify the RPATH of a binary allowing it to find the libraries that it needs. We may also have to wrap executables in shell scripts to set certain environment variables, such as LD_LIBRARY_PATH.

Some applications that are packaged in Nixpkgs this way are Adobe Reader, Belastingaangifte (Dutch Tax administration) and the Android SDK, as I have described in two earlier blog posts.

However, I have recently developed something that can solve this problem in a different way.

Composing FHS-compatible chroot environments

In my previous blog post, I have described how Nix user environments work, which are symlink trees that blend the contents of a collection of packages into a single symlink tree. User environments are primarily used to allow end-users to conveniently access packages installed by Nix from a single location.

We can also use the user environment mechanism in combination with setting up a chroot environment to simulate an FHS-compatible filesystem. This allows us to run an unpatched binary that works out of the box, if all required dependencies of the package are (correctly) specified.

I have created a Nix function called buildFHSChrootEnv {} which makes it possible to compose such environments. This function roughly does the following:

  • It invokes buildEnv {} to compose a so-called "/usr user environment". In this user environment, we synthesize the contents of all our specified packages into a single Nix profile. This profile contains all static parts of all packages and is supposed to simulate both the /usr directory as well as the static folders in the root folder of the chroot environment, such as /bin and /lib.
  • Besides the dependencies of the binary that we want to run, we also need a collection of base packages, such as the bash shell, GNU Coreutils, glibc, and the shadow password suite, since we also have to be able to run a basic Linux system and do basic shell activities inside the chroot environment. These packages are always part of the /usr profile.
  • It generates a global bash profile setting the PS1 environment variable to configure the shell prompt and other miscellaneous settings that a program may need in order to run.
  • It generates a collection of scripts taking care of setting up (or destroying) the dynamic parts of the chroot environment, since Nix packages and user environment are static and can never change after they have been built.

Setting up the dynamic parts of the chroot environment

Merely composing a /usr profile is not enough to get a working chroot environment because they only contain the static parts of a system, such as executables, libraries, and data files. We also have to set up a few dynamic parts. A collection of configuration scripts bundled with the generated /usr profile can be used for this purpose:

  • init-*-chrootenv: Initializes most dynamic parts of the chroot environment inside a directory with write permissions (i.e. /run/chrootenv/<name>).

    First, it creates symlinks to the static folders of the /usr profile and the /usr symlink itself. Then the script ensures that the Name Service Switch (NSS) facilities work by symlinking /etc/{passwd,group,shadow,hosts,resolv.conf,nsswitch.conf) of the host system into the chroot environment. Then some other files from the host are symlinked, such as PAM configuration files and font configuration stuff. Finally, a few empty state folders are created, such as: /var, /run, /tmp.

    The initialization script has to be executed only once.
  • mount-*-chrootenv: Bind mounts a number of directories of the host system into the chroot folder, so that files from the host system can be used while we have entered the chroot environment.

    Currently, we bind mount the Nix store (/nix/store), directories used by the kernel: /dev, /dev/pts, /dev/shm, /proc, /sys, users' home directories: /home, a few state directories: /var, /run, and the host's /etc directory containing shared configuration files, such as the ones related to NSS and PAM.

    This script has to be executed every time we have booted into NixOS.
  • load-*-chrootenv: Enters the chroot environment and launches a shell. The shell inherits a few environment variables from the host environment, such as DISPLAY allowing someone to run GUI applications and TERM to make certain fancy console applications to work properly.
  • umount-*-chrootenv: Unmounts the bind mounts.
  • destroy-*-chrootenv: Removes the writable chrootenv directory.

tl;dr - Getting Steam to work under NixOS

If you're still awake and still interested in reading this blog post, I will reveal why I have done all this work, which is probably much more exciting to read. In fact, this function's primary use case is to get Steam working in NixOS. Steam is a very special piece of software compared to other binary-only packages, because:

  • Steam is also a deployment tool (like Nix) capable of automatically downloading, installing and upgrading software packages. In Steam's case, these software packages are mostly games.
  • Software packages installed by Steam have the same problems as the deployment of "ordinary" binary packages in NixOS -- they don't work out of the box, because they cannot find the dynamic linker and their dependencies, such as shared libraries.
  • Of course, we can patch the executables installed by Steam, but Steam wants control over them and does not like patched versions of software. Furthermore, patching the packages makes it impossible for Steam to update them.
  • It is also a bit impractical to patch the executables every time they are updated, because I have seen that the games I have are updated quite frequently.

By using the buildFHSChrootEnv {} Nix function getting Steam (including the games installed by it) to run in NixOS is a piece of cake. First I had to package Steam itself, which was quite easy:

{stdenv, fetchurl, dpkg}:

stdenv.mkDerivation {
  name = "steam-";
  src = fetchurl {
    url =;
    sha256 = "1jyvk0h1z78sdpvl4hs1kdvr6z2kwamf09vjgjx1f6j04kgqrfbw";
  buildInputs = [ dpkg ];
  unpackPhase = "true";
  installPhase = ''
    mkdir -p $out
    dpkg -x $src $out
    cp -av $out/usr/* $out
    rm -Rf $out/usr

I just had to unpack the Steam Debian package and move its contents into the Nix store. I intentionally did not patch or wrap anything.

Then I had to create a Nix expression that composes a FHS-compatible chroot environment with Steam's dependencies and runtime settings (such as OpenGL driver settings):

{ buildFHSChrootEnv, steam
, xterm, libX11, zenity, python, mesa, xdg_utils, dbus_tools, alsaLib

buildFHSChrootEnv {
  name = "steam";
  pkgs = [ steam xterm libX11 zenity python mesa xdg_utils dbus_tools alsaLib ];
  profile = ''
    export LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib:/lib
    export FONTCONFIG_FILE=/etc/fonts/fonts.conf

After becoming the super user, I can install the above Nix expression providing the /usr environment profile and configuration scripts, by running:

$ su
# nix-env -f '<nixpkgs>' -iA steamChrootEnv

Now the chroot environment configuration scripts have been added to the system-wide Nix profile. I can initialize the chroot environment folder by running:

# init-steam-chrootenv

I can do the bind mounts by running:

# mount-steam-chrootenv

And then enter the chroot environment by running:

# load-steam-chrootenv

Now I have entered the chroot environment in which I can run the unpatched Steam executable. However, I should not run Steam as a root user. Therefore, I have to become "myself" again:

# su -s /bin/bash sander

And now I can finally run Steam, install my favourite games and play them:

$ steam

As you may see in the screenshot above, it works quite well on my machine. Moreover, to give you some proof that I'm not lying, I have also opened a terminal in the background showing you NixOS' root folder that does not contain most of the standard FHS directories.

So far, I have tried Half Life (and extensions), Half Life 2 (and extensions), Portal and Counter Strike and they all seem to work without any trouble. Isn't that AWESOME??? :P

Finally, if you need to discard the chroot environment (which should not be necessary IMHO :P), you could run:

# umount-steam-chrootenv
# destroy-steam-chrootenv


In this blog post, I have described the buildFHSChrootEnv {} Nix function that composes FHS-compatible chroot environments in NixOS, allowing me to run Steam and other third party binary packages without patching them.

Moreover, we still have control over the dependencies in the chroot environment, because apart from a number of common base packages, these chroot environments are composed from Nix expressions that provide exactly the dependencies that are specified. Undeclared dependencies cannot influence a binary package running in the chroot environment.

However, this approach also has a big drawback. In order to chroot, we need super-user privileges making it hard to properly support unprivileged user installations of such packages. Most "ordinary" Nix packages can be deployed by unprivileged users without any trouble. Therefore, I would not recommend this approach over patching executables with PatchELF, unless there is no other way, such as in Steam's case.


This is not the first attempt to get Steam packaged in Nixpkgs. In my attempt, I have tried to use the chroot approach which turns out to work. I'd like to thank Evgeny Egorochkin, Aszlig Neusepoff and Carles Pag├Ęs for their earlier efforts.


The buildFHSChrootEnv {} function as well as the Steam components are part of Nixpkgs.

Monday, September 9, 2013

Managing user environments with Nix

For some reason, I'm having quite some discussions about Nix's user environment mechanism lately. For example, in my last two talks I have spent a considerable amount of time answering questions that were related to it.

The user environment mechanism is an important key feature of Nix, which primary purpose is to allow someone to conveniently launch programs installed by the Nix package manager, but it has a few additional use cases as well, such as determining which packages are in use (and which are not) in order to safely garbage collect obsolete packages.

I have only briefly described them in some of my blog posts, but I have never covered in detail how they work on my blog. Because of the amount of questions I have received lately, I think it would be a good idea to write about it.

The Nix store

As described in my exhaustive explanation blog post on Nix (and the original explanation recipe), every Nix package is stored in an isolated directory in a so called "Nix store". In the Nix store every package has a special filename that may look like this:


The first part of the basename (spy13ngvs1fyj82jw2w3nwczmdgcp3ck) is a cryptographic hash derived from all build inputs used to build Firefox. For example, if we would build Firefox with a different compiler or for a different architecture, the generated hash code will be different, which makes it possible to safely store multiple variants of packages next to each other, because none of them will have the same name.

Although this storing convention looks a bit odd, it provides a number of interesting benefits. Since Nix executes builds inside environments in which as many side effects are removed, we can use the hash codes to uniquely identify package variants regardless on what machine they have been built.

They also provide better reproducibility, since these paths to Nix packages including their hash codes must always be explicitly specified, in order to allow a package build to find its dependencies. The advantage of this is that unspecified dependencies cannot accidentally allow a build to succeed, limiting reproducibility guarantees.

More details on the benefits can be found in my two explanation recipes. However, storing packages this way also causes a major inconvenience for end-users. To launch a particular program, the user must always provide the full path to the executable that resides inside the Nix store, which is quite impractical due to the fact that every Nix package has a hash code in it.

Nix profiles

As a means for users to conveniently launch software, the Nix package manager provides the concept of Nix profiles, which are user environments exposing a generation of installed packages to a user. Software packages can be automatically installed in Nix profiles by using the nix-env command-line utility.

To install software, such as Info ZIP's zip and unzip utilities, we can run:

$ nix-env -i zip unzip
installing `zip-3.0'
installing `unzip-6.0'
building path(s) `/nix/store/1zhgva9cmfz0wam9m7ibgaih2m3cic6l-user-environment'
created 33 symlinks in user environment

Allowing us to run any of the installed tools without specifying any paths with hash codes, e.g.:

$ zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at; see README for details.

One of the things the above nix-env command-line invocation does is automatically building the requested packages (or downloading its substitutes) and composing a user environment, which is a symlink tree that synthesizes the contents of the currently installed packages (zip and unzip) into one single component. A user environment is also a package residing in the Nix store, as shown in the following figure:

Besides creating a user environment that exposes all installed packages from a single location, a few additional symlink indirections are used. Each time nix-env performs a deployment activity, a generation symlink is generated (called profile-1 in the figure above). There is also a default symlink (profile) pointing to the generation symlink that is currently active. That symlink is also referenced from the user's home directory through ~/.nix-profile. By adding the latter symlink to the user's PATH, e.g.:

export PATH=~/.nix-profile/bin:$PATH

Packages installed through nix-env can be found automatically through a few layers of symlink indirections.

When we perform another deployment activity with nix-env, such as removing zip:

$ nix-env -e zip
uninstalling `zip-3.0'

Then a new symlink tree is composed containing the then currently installed packages (in this case only unzip), a new generation symlink is generated (profile-2), and the default symlink is updated to point to the new generation. Flipping symlinks is actually an atomic operation on UNIX. As a consequence, we can perform upgrades atomically, because there is never a situation in which we refer to packages belonging to the old and new user environment at the same time.

As may be observed in the picture above, the old generation user environment is still present allowing us to do rollbacks as well. For example, if we regret uninstalling zip, we can switch back to the previous generation by running:

$ nix-env --rollback

Then the default symlink's reference is updated to point to the previous generation symlink that provides the zip utility. Again, this is an atomic operation.

Apart from exposing packages installed by users, the directory containing profile generations also serves a second goal; it is used to determine which packages are in use and which are not.

The generations of the Nix profiles of all users of the system are treated as so called garbage collector roots -- any package that is installed in a profile generation and any of its dependencies are considered packages that are live (a.k.a. in use) and should not be garbage collected. Furthermore, Nix also treats open files and running processes as garbage collector roots. Therefore, it is safe to run the Nix garbage collector at any time.

We can also remove older Nix profile generations if we don't need them anymore:

$ nix-env --delete-generations old

The above command only removes the older generation symlinks, but not the packages that have become obsolete. To actually remove packages from the system that are no longer in use, we must explicitly run the garbage collector:

$ nix-collect-garbage

So far, I have shown examples that work on a per-user profile. In multi-user Nix installations, such as in NixOS, every user has a personal Nix profile (residing in /nix/var/nix/profiles/per-user) and there is also a system-wide default Nix profile (residing in /nix/var/nix/profiles/default) containing packages that are exposed to all users of a system.

NixOS system profiles

Apart from individual packages, NixOS -- the Linux distribution built around the Nix package manager, also uses Nix profiles to manage generations of system configurations.

As described in my earlier blog post about NixOS, a complete system configuration is deployed from a single declarative specification, including all system services and their configurations, the desktop, and user packages. Each time we deploy a system configuration, e.g. by running:

$ nixos-rebuild switch

a Nix profile is built which consists of (mostly) a symlink tree containing all packages and configuration files that constitutes all static parts of a system, including the Linux kernel and the kernel modules. This profile is called a system profile and offers the same benefits as ordinary package profiles, such as the fact that we can store multiple generations next to each other and perform atomic upgrades and rollbacks.

There is a subtle difference with NixOS deployment compared to ordinary package deployment. Merely flipping symlinks is not enough to get a new NixOS configuration deployed. We also have to perform a collection of imperative deployment steps, such as setting up dynamic directories, such as /var, starting and stopping system services, and updating the GRUB bootloader configuration. This is done by an activation script (/run/current-system/bin/switch-to-configuration) part of the NixOS system profile, that is executed each time a new version of a NixOS configuration is deployed.

As can be seen in the screenshot above, once a new NixOS system configuration has been deployed, the GRUB bootloader can be used to boot into any previous NixOS configuration that has not been garbage collected yet.

Generating user environments from Nix expressions

In addition to using nix-env, user environments can also be generated from Nix expression by using the buildEnv {} function. The following Nix expression produces the same user environment (shown earlier) containing zip and unzip:

with import <nixpkgs> {};

buildEnv {
  name = "zipprofile";
  paths = [ zip unzip ];

This expression can be built by running:

$ nix-build zipprofile.nix


In this blog post, I have explained how the Nix user environment mechanism works. Essentially user environments are symlink trees that blend all currently installed packages into a single component so that they can be referenced from a single location. Another symlink layer manages generations of user environments. We can atomically switch between these generations by flipping symlinks. By adding the latter symlink to a user's PATH, packages can be automatically found without providing any absolute paths with hash codes in them that are difficult to remember.

Besides exposing packages in a user's PATH, Nix profiles are also used in NixOS to automatically find other resources, such as manual pages (through MANPATH) and KDE desktop menu items (through XDG_DATA_DIRS).

More information on Nix profiles can be found in Eelco Dolstra's PhD thesis which can be obtained from his publications page. Additionally, my PhD thesis also covers Nix profiles in its background chapter. Furthermore, there is a chapter that extends their concepts to distributed systems. This chapter is loosely based on our HotSWUp 2008 paper titled: "Atomic Upgrading of Distributed Systems" which can be obtained from my publications page.