Monday, June 17, 2013

Setting up a multi-user Nix installation on non-NixOS systems

I have written quite some Nix-related articles on this blog. Nix is typically advertised as the core component of the NixOS Linux distribution. However, it can also be used separately on conventional Linux distributions and other UNIX-like systems, such as FreeBSD and Mac OS X.

Using Nix on conventional systems makes it possible to use the interesting features of Nix and its applications, such as Hydra and Disnix, while still being able to use your favorite distribution.

Single-user Nix installations


I have noticed that on non-NixOS systems, the Nix package manager is often installed for only one single user, as performing single-user installations is relatively simple. For example, in my blog post describing how to build apps for iOS with Nix, I perform a Nix installation that can only be used by my personal user account.

For most of the simple use cases, single user installations are sufficient. However, they have a number of issues besides the fact that only one user of a machine (in addition to the super-user) is able to use it:

  • Although Nix creates build environments that remove several important side-effects, e.g. by clearing environment variables and storing all packages in isolation in the Nix store, builds can still refer to executables and other files in global directories by having hardcoded references, such as /usr/bin or /var, which may influence the result of the build.

    On NixOS these references are typically not an issue since these directories do not exist, but on conventional distributions these do exist, which may cause (purity) problems.
  • Many packages have hard-coded references to the default Bourne-compatible shell in: /bin/sh. Some of these packages assume that this shell is bash and use bash-specific features.

    However, some Linux distributions, such as Debian and Ubuntu, use dash as default /bin/sh causing some builds to break. Moreover, BSDs such as FreeBSD also use a simpler Bourne shell implementation by default.
  • Some build processes may unknowingly try to download stuff from the Internet causing impurities.
  • Package builds have the same privileges as the calling user, allowing external processes run by the user to interfere with the build process. As a consequence, impurities may sneak in when executing multiple package builds in parallel.

Multi-user Nix installations


As a remedy for the issues just described, Nix is also capable of executing each build with separate user privileges in a nearly clean chroot environment in which we bind mount the Nix store. However, in order to use these features, a multi-user Nix installation is required, as these operations require super-user privileges. In NixOS, a multi-user Nix installation comes for free.

In multi-user Nix installations, builds are not executed directly by each individual user, since they cannot be trusted. Instead we run a server, called the nix-daemon that builds packages on behalf of a user. This daemon also takes care of running processes as a unique unprivileged user and setting up chroot environments.

Although the Nix manual provides some pointers to set up a multi-user installation, it turned out to be a bit trickier than I thought. Moreover, I have noticed that a few practical bits were missing in the manual and the Nix distribution.

In this blog post, I have investigated these issues and implemented a few improvements that provide a solution for these missing parts. I have performed these steps on a Ubuntu 12.04 LTS machine.

Installing Nix from source


As a first step, I installed Nix from source by running the following commands:

$ ./configure --prefix=/usr --sysconfdir=/etc
$ make
$ sudo make install

Adding the Nix build group and users


Since every concurrent build must be run as a separate user, we have to define a common group of which these users should be a member:
$ sudo groupadd -g 20000 nixbld
Then we need to add user accounts for each build that gets executed simultaneously:

$ for i in `seq 1 10`
do
    sudo useradd -u `expr 20000 + $i` -g nixbld \
      -c "Nix build user $i" -d /var/empty -s /noshell
done

In the code fragment above, we assume that 10 users are sufficient, but if you want/need to utilise more processors/cores this number needs to be raised.

Finally, we have to specify the build users group in the Nix configuration:

$ sudo echo "build-users-group = nixbld" >> /etc/nix/nix.conf

Changing permissions of the Nix store


In ordinary installations the user is made owner of /nix/store. In multi-user installations, it must be owned by root and group owned by the nixbld user. The following shell commands grant the Nix store the right permissions:

$ sudo chgrp nixbld /nix/store
$ sudo chmod 1775 /nix/store

Creating per-user profile and garbage collection root folders


In single user installations, we only have one system-wide default profile (/nix/var/nix/profiles/default) owned by the user. In multi-user installations, each user should be capable of creating their own profiles and garbage collector roots. The following shell commands ensure that it can be done:

$ sudo mkdir -p -m 1777 /nix/var/nix/profiles/per-user
$ sudo mkdir -p -m 1777 /nix/var/nix/gcroots/per-user

Setting up the Nix daemon


We must also run the nix-daemon that executes builds on behalf of a user. To be able to start and stop it, I have created an init.d script for the Nix daemon. The interesting part of this script is the start operation:

DAEMON=/usr/bin/nix-daemon
NAME=nix-daemon

if test -f /etc/default/nix-daemon; then
    . /etc/default/nix-daemon
fi

...

case "$1" in

start)
    if test "$NIX_DISTRIBUTED_BUILDS" = "1"; then
        NIX_BUILD_HOOK=$(dirname $DAEMON)/../libexec/nix/build-remote.pl
                
        if test "$NIX_REMOTE_SYSTEMS" = "" ; then
            NIX_REMOTE_SYSTEMS=/etc/nix/remote-systems.conf
        fi
                
        # Set the current load facilities
        NIX_CURRENT_LOAD=/var/run/nix/current-load
                
        if test ! -d $NIX_CURRENT_LOAD; then
            mkdir -p $NIX_CURRENT_LOAD
        fi
    fi
                
    start-stop-daemon -b --start --quiet \
        --exec /usr/bin/env \
        NIX_REMOTE_SYSTEMS=$NIX_REMOTE_SYSTEMS \
        NIX_BUILD_HOOK=$NIX_BUILD_HOOK \
        NIX_CURRENT_LOAD=$NIX_CURRENT_LOAD \
        $DAEMON -- $DAEMON_OPTS
    echo "$NAME."
    ;;

...

esac

For the start operation, we have to spawn the nix-daemon in background mode. Moreover, to allow Nix to perform distributed builds, we must set a number of environment variables that provide the locations of the build hook script, the configuration file containing the properties of the external machines and a directory containing files keeping track of the load of the machines. Moreover, some directories may also have to be created if they don't exist.

To ensure that it's automatically launched on startup, we must add the following symlinks for the relevant runlevels:

$ sudo -i
# cd /etc/rc2.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc3.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc4.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc5.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# exit

I think the above init.d script can be trivially ported to other distributions.

Setting up user profiles


To allow users to install software through Nix and allow them to refer to their installed programs from a simple command-line invocation, we need to add some stuff to the user's shell profile, such as setting the PATH environment variable pointing to certain Nix profiles.

However, the nix.sh profile.d script in the Nix distribution only performs the necessary steps for single user installations. For example, it only adds to system-wide Nix profile and assumes that the user has all the rights to configure a channel.

I have ported all the relevant features from NixOS to create /etc/profile.d/nix-multiuser.sh, supporting all required features to set up a shell profile for multi-user installations:

First, we have to set up a user's profile directory in the per-user profile directory, if it doesn't exist:

export NIX_USER_PROFILE_DIR=/nix/var/nix/profiles/per-user/$USER

mkdir -m 0755 -p $NIX_USER_PROFILE_DIR
if test "$(stat --printf '%u' $NIX_USER_PROFILE_DIR)" != "$(id -u)"; then
    echo "WARNING: bad ownership on $NIX_USER_PROFILE_DIR" >&2
fi

In single user installations, a ~/.nix-profile symlink is created pointing to the system-wide default Nix profile. For multi-user installations, we must create a ~/.nix-profile symlink pointing to the per-user profile. For the root user, we can still use the system wide Nix profile providing software for all users of the system:

if ! test -L $HOME/.nix-profile; then
    echo "creating $HOME/.nix-profile" >&2
    if test "$USER" != root; then
        ln -s $NIX_USER_PROFILE_DIR/profile $HOME/.nix-profile
    else
        # Root installs in the system-wide profile by default.
        ln -s /nix/var/nix/profiles/default $HOME/.nix-profile
    fi
fi

In single user installations, we add the bin directory of the system-wide Nix profile to PATH. In multi-user installations, we have to do this both for the system-wide and the user profile:

export NIX_PROFILES="/nix/var/nix/profiles/default $HOME/.nix-profile"

for i in $NIX_PROFILES; do
    export PATH=$i/bin:$PATH
done

In single user installations, the user can subscribe itself to the Nixpkgs unstable channel providing pre-built substitutes for packages. In multi-user installations only the super-user can do this (as ordinary users cannot be trusted). Although root can only subscribe to a channel, ordinary users can still install from the subscribed channels:

if [ "$USER" = root -a ! -e $HOME/.nix-channels ]; then
    echo "http://nixos.org/channels/nixpkgs-unstable nixpkgs" \
      > $HOME/.nix-channels
fi

We have to create a garbage collector root folder for the user, if it does not exists:

NIX_USER_GCROOTS_DIR=/nix/var/nix/gcroots/per-user/$USER
mkdir -m 0755 -p $NIX_USER_GCROOTS_DIR
if test "$(stat --printf '%u' $NIX_USER_GCROOTS_DIR)" != "$(id -u)"; then
    echo "WARNING: bad ownership on $NIX_USER_GCROOTS_DIR" >&2
fi

We must also set the default Nix expression, so that we can conveniently install packages from Nix channels:

if [ ! -e $HOME/.nix-defexpr -o -L $HOME/.nix-defexpr ]; then
    echo "creating $HOME/.nix-defexpr" >&2
    rm -f $HOME/.nix-defexpr
    mkdir $HOME/.nix-defexpr
    if [ "$USER" != root ]; then
        ln -s /nix/var/nix/profiles/per-user/root/channels \
          $HOME/.nix-defexpr/channels_root
    fi
fi

Unprivileged users do not have the rights to build package directly, since they cannot be trusted. Instead the daemon must do that on behalf of the user. The following shell code fragment ensures that:

if test "$USER" != root; then
    export NIX_REMOTE=daemon
else
    export NIX_REMOTE=
fi

Using multi-user Nix


After having performed the previous steps, we can start the Nix daemon by running the init.d script as root user:

$ sudo /etc/init.d/nix-daemon start
Then if we login as root, we can update the Nix channels and install packages that are supposed to be available system-wide:

$ nix-channel --update
$ nix-env -i hello
$ hello
Hello, world!

We should also be able to log in as an unprivileged user and capable of installing software:

$ nix-env -i wget
$ wget # Only available to the user that installed it
$ hello # Also works because it's in the system-wide profile

Enabling parallel builds


With a multi-user installation, we should also be able to safely run multiple builds concurrently. The following change can be made to allow 4 builds to be run in parallel:

$ sudo echo "build-max-jobs = 4" >> /etc/nix/nix.conf

Enabling distributed builds


To enable distributed builds (for example to delegate a build to a system with a different architecture) we can run the following:

$ sudo echo "NIX_DISTRIBUTED_BUILDS=1" > /etc/defaults/nix-daemon

$ sudo cat > /etc/nix/remote-systems.conf << "EOF"
sander@macosx.local x86_64-darwin /root/.ssh/id_buildfarm 2
EOF

The above allows us to delegate builds for Mac OS X to a Mac OS X machine.

Enabling chroot builds


On Linux, we can also enable chroot builds allowing us to remove many undesired side-effects that single-user Nix installations have. Chroot environments require some directories of the host system to be bind mounted, such as /dev, /dev/pts, /proc and /sys.

Moreover, we need a default Bourne shell in /bin/sh that must be bash, as other more primitive Bourne compatible shells may give us trouble. Unfortunately, we cannot bind mount the host's system /bin folder, as it's filled with all kinds of executables causing impurities. Moreover, these executables have requirements on shared libraries residing in /lib, which we do not want to expose in the chroot environment.

I know two ways to have bash as /bin/sh in our chroot environment:

  • We can install bash through Nix and expose that in the chroot environment:

    $ nix-env -i bash
    $ sudo mkdir -p /nix-bin
    $ sudo ln -s $(readlink -f $(which bash)) /nix-bin/sh
    
    The approach should work, as bash's dependencies all reside in the Nix store which is available in the chroot environment.

  • We could also create a static bash that does not depend on anything. The following can be run to compile a static bash manually:

    $ ./configure --prefix=~/bash-static --enable-static-link \
        --without-bash-malloc --disable-nls
    $ make
    $ make install
    $ sudo mkdir -p /nix-bin
    $ sudo cp bash /nix-bin/sh
    

    Or by using the following Nix expression:

    with import <nixpkgs> {};
    
    stdenv.mkDerivation {
      name = "bash-static-4.2";
      src = fetchurl {
        url = mirror://gnu/bash/bash-4.2.tar.gz;
        sha256 = "1n5kbblp5ykbz5q8aq88lsif2z0gnvddg9babk33024wxiwi2ym2";
      };
      patches = [ ./bash-4.2-fixes-11.patch ];
      buildInputs = [ bison ];
      configureFlags = [
        "--enable-static-link"
        "--without-bash-malloc"
        "--disable-nls"
      ];
    }
    

Finally, we have to add a few properties to Nix's configuration to enable chroot builds:

$ sudo echo "build-use-chroot = true" >> /etc/nix/nix.conf
$ sudo echo "build-chroot-dirs = /dev /dev/pts /bin=/nix-bin $(nix-store -qR /nix-bin/sh | tr '\n' ' ')" \
    >> /etc/nix/nix.conf
The last line exposes /dev, /dev/pts and /nix-bin (mounted on /bin, containing only sh) of the host system to the chroot environment allowing us to build stuff purely. If we have installed bash through Nix, then we also have to add the Nix closure of bash, which we can obtain through the nix-store -qR command. For a static bash this invocation has to be omitted.

Conclusion


In this blog post, I have described everything I did to set up a multi-user Nix installation supporting distributed and chroot builds on a conventional Linux distribution (Ubuntu) which was a bit tricky.

I'm planning to push some of things I did upstream, so that others can benefit from it. This is a good thing, because I have the feeling that most non-NixOS Nix users will lose their interest if they have figure out the same stuff I just did.

6 comments:

  1. Yo, I tried to set up Nix in multi-user mode on Xubuntu. I followed the manual, but I had no success. Looks like you made progress to fix this, but you didn't push it upstream. What's the status on that?
    https://github.com/svanderburg/nix/commit/7ab1054e6cae7a8c48fd907d98a0ed8642026311

    ReplyDelete
    Replies
    1. It seems that my suggested changes are still not integrated yet. I don't know why.

      Anyway, this blog post should contain all the additional information that you need to do it properly.

      Delete
    2. To merge your changes into Nix, you need to create a Pull Request. Basically, click the "Fork" button on NixPkgs, edit the code and ensure the new code is in your fork of NixPkgs on GitHub, then push the "Pull Request" button from your fork's page.

      Your work to fix this and post some documentation is appreciated! Now if we can only ensure it works on the current Nix code base and can be merged in. ;-) This isn't a priority for me right now, but if you have free time to work on this, I might be able to help you out. (I'm sure you'd rather leave this work in the past, which I understand. I'm sure somebody, sometime in the future, will pick up from here.)

      Delete
  2. Hi Alex,

    I appreciate your help, but I have already created a pull request quite some time ago. It can be found here: https://github.com/NixOS/nix/pull/135

    What I meant is that despite the fact that I have created this commit and pull request, I haven't received any comments of the Nix maintainer nor has he integrated my changes.

    Best thing is to do now is probably to notify him through the IRC or mailing list...

    ReplyDelete
    Replies
    1. Ah, I see it now. Thanks for the reply.

      Hey, you have some *really great* posts on your blog. I'd like to add some of them to the NixOS wiki. Can I get your permission to do that? If so, I will edit them to make them easier for noobs, but the pages will link back to your blog.

      Delete
    2. Of course you can add them. Everything Nix-related that I post on this blog also appears automatically on: http://planet.nixos.org

      Unfortunately, I'm a bit too lazy to update the wiki as well, but feel free to add pointers to any you'd like to share.

      Delete