Setting up a multi-user Nix installation on non-NixOS systems

I have written quite some Nix-related articles on this blog. Nix is typically advertised as the core component of the NixOS Linux distribution. However, it can also be used separately on conventional Linux distributions and other UNIX-like systems, such as FreeBSD and Mac OS X.

Using Nix on conventional systems makes it possible to use the interesting features of Nix and its applications, such as Hydra and Disnix, while still being able to use your favorite distribution.

Single-user Nix installations

I have noticed that on non-NixOS systems, the Nix package manager is often installed for only one single user, as performing single-user installations is relatively simple. For example, in my blog post describing how to build apps for iOS with Nix, I perform a Nix installation that can only be used by my personal user account.

For most of the simple use cases, single user installations are sufficient. However, they have a number of issues besides the fact that only one user of a machine (in addition to the super-user) is able to use it:

• Although Nix creates build environments that remove several important side-effects, e.g. by clearing environment variables and storing all packages in isolation in the Nix store, builds can still refer to executables and other files in global directories by having hardcoded references, such as /usr/bin or /var, which may influence the result of the build.
  
  On NixOS these references are typically not an issue since these directories do not exist, but on conventional distributions these do exist, which may cause (purity) problems.
• Many packages have hard-coded references to the default Bourne-compatible shell in: /bin/sh. Some of these packages assume that this shell is bash and use bash-specific features.
  
  However, some Linux distributions, such as Debian and Ubuntu, use dash as default /bin/sh causing some builds to break. Moreover, BSDs such as FreeBSD also use a simpler Bourne shell implementation by default.
• Some build processes may unknowingly try to download stuff from the Internet causing impurities.
• Package builds have the same privileges as the calling user, allowing external processes run by the user to interfere with the build process. As a consequence, impurities may sneak in when executing multiple package builds in parallel.

Multi-user Nix installations

As a remedy for the issues just described, Nix is also capable of executing each build with separate user privileges in a nearly clean chroot environment in which we bind mount the Nix store. However, in order to use these features, a multi-user Nix installation is required, as these operations require super-user privileges. In NixOS, a multi-user Nix installation comes for free.

In multi-user Nix installations, builds are not executed directly by each individual user, since they cannot be trusted. Instead we run a server, called the nix-daemon that builds packages on behalf of a user. This daemon also takes care of running processes as a unique unprivileged user and setting up chroot environments.

Although the Nix manual provides some pointers to set up a multi-user installation, it turned out to be a bit trickier than I thought. Moreover, I have noticed that a few practical bits were missing in the manual and the Nix distribution.

In this blog post, I have investigated these issues and implemented a few improvements that provide a solution for these missing parts. I have performed these steps on a Ubuntu 12.04 LTS machine.

Installing Nix from source

As a first step, I installed Nix from source by running the following commands:

$ ./configure --prefix=/usr --sysconfdir=/etc
$ make
$ sudo make install

Adding the Nix build group and users

Since every concurrent build must be run as a separate user, we have to define a common group of which these users should be a member:

$ sudo groupadd -g 20000 nixbld

Then we need to add user accounts for each build that gets executed simultaneously:

$ for i in `seq 1 10`
do
    sudo useradd -u `expr 20000 + $i` -g nixbld \
      -c "Nix build user $i" -d /var/empty -s /noshell
done

In the code fragment above, we assume that 10 users are sufficient, but if you want/need to utilise more processors/cores this number needs to be raised.

Finally, we have to specify the build users group in the Nix configuration:

$ sudo echo "build-users-group = nixbld" >> /etc/nix/nix.conf

Changing permissions of the Nix store

In ordinary installations the user is made owner of /nix/store. In multi-user installations, it must be owned by root and group owned by the nixbld user. The following shell commands grant the Nix store the right permissions:

$ sudo chgrp nixbld /nix/store
$ sudo chmod 1775 /nix/store

Creating per-user profile and garbage collection root folders

In single user installations, we only have one system-wide default profile (/nix/var/nix/profiles/default) owned by the user. In multi-user installations, each user should be capable of creating their own profiles and garbage collector roots. The following shell commands ensure that it can be done:

$ sudo mkdir -p -m 1777 /nix/var/nix/profiles/per-user
$ sudo mkdir -p -m 1777 /nix/var/nix/gcroots/per-user

Setting up the Nix daemon

We must also run the nix-daemon that executes builds on behalf of a user. To be able to start and stop it, I have created an init.d script for the Nix daemon. The interesting part of this script is the start operation:

DAEMON=/usr/bin/nix-daemon
NAME=nix-daemon

if test -f /etc/default/nix-daemon; then
    . /etc/default/nix-daemon
fi

...

case "$1" in

start)
    if test "$NIX_DISTRIBUTED_BUILDS" = "1"; then
        NIX_BUILD_HOOK=$(dirname $DAEMON)/../libexec/nix/build-remote.pl
                
        if test "$NIX_REMOTE_SYSTEMS" = "" ; then
            NIX_REMOTE_SYSTEMS=/etc/nix/remote-systems.conf
        fi
                
        # Set the current load facilities
        NIX_CURRENT_LOAD=/var/run/nix/current-load
                
        if test ! -d $NIX_CURRENT_LOAD; then
            mkdir -p $NIX_CURRENT_LOAD
        fi
    fi
                
    start-stop-daemon -b --start --quiet \
        --exec /usr/bin/env \
        NIX_REMOTE_SYSTEMS=$NIX_REMOTE_SYSTEMS \
        NIX_BUILD_HOOK=$NIX_BUILD_HOOK \
        NIX_CURRENT_LOAD=$NIX_CURRENT_LOAD \
        $DAEMON -- $DAEMON_OPTS
    echo "$NAME."
    ;;

...

esac

For the start operation, we have to spawn the nix-daemon in background mode. Moreover, to allow Nix to perform distributed builds, we must set a number of environment variables that provide the locations of the build hook script, the configuration file containing the properties of the external machines and a directory containing files keeping track of the load of the machines. Moreover, some directories may also have to be created if they don't exist.

To ensure that it's automatically launched on startup, we must add the following symlinks for the relevant runlevels:

$ sudo -i
# cd /etc/rc2.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc3.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc4.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc5.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# exit

I think the above init.d script can be trivially ported to other distributions.

Setting up user profiles

To allow users to install software through Nix and allow them to refer to their installed programs from a simple command-line invocation, we need to add some stuff to the user's shell profile, such as setting the PATH environment variable pointing to certain Nix profiles.

However, the nix.sh profile.d script in the Nix distribution only performs the necessary steps for single user installations. For example, it only adds to system-wide Nix profile and assumes that the user has all the rights to configure a channel.

I have ported all the relevant features from NixOS to create /etc/profile.d/nix-multiuser.sh, supporting all required features to set up a shell profile for multi-user installations:

First, we have to set up a user's profile directory in the per-user profile directory, if it doesn't exist:

export NIX_USER_PROFILE_DIR=/nix/var/nix/profiles/per-user/$USER

mkdir -m 0755 -p $NIX_USER_PROFILE_DIR
if test "$(stat --printf '%u' $NIX_USER_PROFILE_DIR)" != "$(id -u)"; then
    echo "WARNING: bad ownership on $NIX_USER_PROFILE_DIR" >&2
fi

In single user installations, a ~/.nix-profile symlink is created pointing to the system-wide default Nix profile. For multi-user installations, we must create a ~/.nix-profile symlink pointing to the per-user profile. For the root user, we can still use the system wide Nix profile providing software for all users of the system:

if ! test -L $HOME/.nix-profile; then
    echo "creating $HOME/.nix-profile" >&2
    if test "$USER" != root; then
        ln -s $NIX_USER_PROFILE_DIR/profile $HOME/.nix-profile
    else
        # Root installs in the system-wide profile by default.
        ln -s /nix/var/nix/profiles/default $HOME/.nix-profile
    fi
fi

In single user installations, we add the bin directory of the system-wide Nix profile to PATH. In multi-user installations, we have to do this both for the system-wide and the user profile:

export NIX_PROFILES="/nix/var/nix/profiles/default $HOME/.nix-profile"

for i in $NIX_PROFILES; do
    export PATH=$i/bin:$PATH
done

In single user installations, the user can subscribe itself to the Nixpkgs unstable channel providing pre-built substitutes for packages. In multi-user installations only the super-user can do this (as ordinary users cannot be trusted). Although root can only subscribe to a channel, ordinary users can still install from the subscribed channels:

if [ "$USER" = root -a ! -e $HOME/.nix-channels ]; then
    echo "http://nixos.org/channels/nixpkgs-unstable nixpkgs" \
      > $HOME/.nix-channels
fi

We have to create a garbage collector root folder for the user, if it does not exists:

NIX_USER_GCROOTS_DIR=/nix/var/nix/gcroots/per-user/$USER
mkdir -m 0755 -p $NIX_USER_GCROOTS_DIR
if test "$(stat --printf '%u' $NIX_USER_GCROOTS_DIR)" != "$(id -u)"; then
    echo "WARNING: bad ownership on $NIX_USER_GCROOTS_DIR" >&2
fi

We must also set the default Nix expression, so that we can conveniently install packages from Nix channels:

if [ ! -e $HOME/.nix-defexpr -o -L $HOME/.nix-defexpr ]; then
    echo "creating $HOME/.nix-defexpr" >&2
    rm -f $HOME/.nix-defexpr
    mkdir $HOME/.nix-defexpr
    if [ "$USER" != root ]; then
        ln -s /nix/var/nix/profiles/per-user/root/channels \
          $HOME/.nix-defexpr/channels_root
    fi
fi

Unprivileged users do not have the rights to build package directly, since they cannot be trusted. Instead the daemon must do that on behalf of the user. The following shell code fragment ensures that:

if test "$USER" != root; then
    export NIX_REMOTE=daemon
else
    export NIX_REMOTE=
fi

Using multi-user Nix

After having performed the previous steps, we can start the Nix daemon by running the init.d script as root user:

$ sudo /etc/init.d/nix-daemon start

Then if we login as root, we can update the Nix channels and install packages that are supposed to be available system-wide:

$ nix-channel --update
$ nix-env -i hello
$ hello
Hello, world!

We should also be able to log in as an unprivileged user and capable of installing software:

$ nix-env -i wget
$ wget # Only available to the user that installed it
$ hello # Also works because it's in the system-wide profile

Enabling parallel builds

With a multi-user installation, we should also be able to safely run multiple builds concurrently. The following change can be made to allow 4 builds to be run in parallel:

$ sudo echo "build-max-jobs = 4" >> /etc/nix/nix.conf

Enabling distributed builds

To enable distributed builds (for example to delegate a build to a system with a different architecture) we can run the following:

$ sudo echo "NIX_DISTRIBUTED_BUILDS=1" > /etc/defaults/nix-daemon

$ sudo cat > /etc/nix/remote-systems.conf << "EOF"
sander@macosx.local x86_64-darwin /root/.ssh/id_buildfarm 2
EOF

The above allows us to delegate builds for Mac OS X to a Mac OS X machine.

Enabling chroot builds

On Linux, we can also enable chroot builds allowing us to remove many undesired side-effects that single-user Nix installations have. Chroot environments require some directories of the host system to be bind mounted, such as /dev, /dev/pts, /proc and /sys.

Moreover, we need a default Bourne shell in /bin/sh that must be bash, as other more primitive Bourne compatible shells may give us trouble. Unfortunately, we cannot bind mount the host's system /bin folder, as it's filled with all kinds of executables causing impurities. Moreover, these executables have requirements on shared libraries residing in /lib, which we do not want to expose in the chroot environment.

I know two ways to have bash as /bin/sh in our chroot environment:

• We can install bash through Nix and expose that in the chroot environment:
  
  

  $ nix-env -i bash
  $ sudo mkdir -p /nix-bin
  $ sudo ln -s $(readlink -f $(which bash)) /nix-bin/sh
  

  The approach should work, as bash's dependencies all reside in the Nix store which is available in the chroot environment.
  
  
• We could also create a static bash that does not depend on anything. The following can be run to compile a static bash manually:
  
  

  $ ./configure --prefix=~/bash-static --enable-static-link \
      --without-bash-malloc --disable-nls
  $ make
  $ make install
  $ sudo mkdir -p /nix-bin
  $ sudo cp bash /nix-bin/sh
  

  
  Or by using the following Nix expression:
  
  

  with import <nixpkgs> {};
  
  stdenv.mkDerivation {
    name = "bash-static-4.2";
    src = fetchurl {
      url = mirror://gnu/bash/bash-4.2.tar.gz;
      sha256 = "1n5kbblp5ykbz5q8aq88lsif2z0gnvddg9babk33024wxiwi2ym2";
    };
    patches = [ ./bash-4.2-fixes-11.patch ];
    buildInputs = [ bison ];
    configureFlags = [
      "--enable-static-link"
      "--without-bash-malloc"
      "--disable-nls"
    ];
  }
  

Finally, we have to add a few properties to Nix's configuration to enable chroot builds:

$ sudo echo "build-use-chroot = true" >> /etc/nix/nix.conf
$ sudo echo "build-chroot-dirs = /dev /dev/pts /bin=/nix-bin $(nix-store -qR /nix-bin/sh | tr '\n' ' ')" \
    >> /etc/nix/nix.conf

The last line exposes /dev, /dev/pts and /nix-bin (mounted on /bin, containing only sh) of the host system to the chroot environment allowing us to build stuff purely. If we have installed bash through Nix, then we also have to add the Nix closure of bash, which we can obtain through the nix-store -qR command. For a static bash this invocation has to be omitted.

Conclusion

In this blog post, I have described everything I did to set up a multi-user Nix installation supporting distributed and chroot builds on a conventional Linux distribution (Ubuntu) which was a bit tricky.

I'm planning to push some of things I did upstream, so that others can benefit from it. This is a good thing, because I have the feeling that most non-NixOS Nix users will lose their interest if they have figure out the same stuff I just did.