Thursday, October 29, 2015

Deploying prebuilt binary software with the Nix package manager

As described in a number of older blog posts, Nix is primarily a source based package manager -- it constructs packages from source code by executing their build procedures in isolated environments in which only specified dependencies can be found.

As an optimization, it provides transparent binary deployment -- if a package that has been built from the same inputs exists elsewhere, it can be downloaded from that location instead of being built from source improving the efficiency of deployment processes.

Because Nix is a source based package manager, the documentation mainly describes how to build packages from source code. Moreover, the Nix expressions are written in such a way that they can be included in the Nixpkgs collection, a repository containing build recipes for more than 2500 packages.

Although the manual contains some basic packaging instructions, I noticed that there a few practical bits were missing. For example, how to package software privately outside the Nixpkgs tree is not clearly described, which makes experimentation a bit less convenient, in particular for newbies.

Despite being a source package manager, Nix can also be used to deploy binary software packages (i.e. software for which no source code and build scripts have been provided). Unfortunately, getting prebuilt binaries to run properly is quite tricky. Furthermore, apart from some references, there are no examples in the manual describing how to do this either.

Since I am receiving too many questions about this lately, I have decided to write a blog post about it covering two examples that should be relatively simple to repeat.

Why prebuilt binaries will typically not work


Prebuilt binaries deployed by Nix typically do not work out of the box. For example, if we want to deploy a simple binary package such as pngout (only containing a set of ELF execuables) we may initially think that copying the executable into the Nix store suffices:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "pngout-20130221";

  src = fetchurl {
    url = http://static.jonof.id.au/dl/kenutils/pngout-20130221-linux.tar.gz;
    sha256 = "1qdzmgx7si9zr7wjdj8fgf5dqmmqw4zg19ypg0pdz7521ns5xbvi";
  };

  installPhase = ''
    mkdir -p $out/bin
    cp x86_64/pngout $out/bin
  '';
}

However, when we build the above package:

$ nix-build pngout.nix

and attempt to run the executable, we stumble upon the following error:

$ ./result/bin/pngout
bash: ./result/bin/pngout: No such file or directory

The above error is quite strange -- the corresponding file resides in exactly the specified location yet it appears that it cannot be found!

The actual problem is not that the executable is missing, but one of its dependencies. Every ELF executable that uses shared libraries consults the dynamic linker/loader (that typically resides in /lib/ld-linux.so.2 (on x86 Linux platforms) and /lib/ld-linux-x86-64.so.2 on (x86-64 Linux platforms)) to provide the shared libraries it needs. This path is hardwired into the ELF executable, as can be observed by running:

$ readelf -l ./result/bin/pngout 

Elf file type is EXEC (Executable file)
Entry point 0x401160
There are 8 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001c0 0x00000000000001c0  R E    8
  INTERP         0x0000000000000200 0x0000000000400200 0x0000000000400200
                 0x000000000000001c 0x000000000000001c  R      1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000001593c 0x000000000001593c  R E    200000
  LOAD           0x0000000000015940 0x0000000000615940 0x0000000000615940
                 0x00000000000005b4 0x00000000014f9018  RW     200000
  DYNAMIC        0x0000000000015968 0x0000000000615968 0x0000000000615968
                 0x00000000000001b0 0x00000000000001b0  RW     8
  NOTE           0x000000000000021c 0x000000000040021c 0x000000000040021c
                 0x0000000000000044 0x0000000000000044  R      4
  GNU_EH_FRAME   0x0000000000014e5c 0x0000000000414e5c 0x0000000000414e5c
                 0x00000000000001fc 0x00000000000001fc  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     8

In NixOS, most parts of the system are stored in a special purpose directory called the Nix store (i.e. /nix/store) including the dynamic linker. As a consequence, the dynamic linker cannot be found because it resides elsewhere.

Another reason why most binaries will not work is because they must know where to find its required shared libraries. In most conventional Linux distributions these reside in global directories (e.g. /lib and /usr/lib). In NixOS, these folders do not exist. Instead, every package is stored in isolation in separate folders in the Nix store.

Why compilation from source works


In contrast to prebuilt ELF binaries, binaries produced by a source build in a Nix build environment work out of the box typically without problems (i.e. they often do not require any special modifications in the build procedure). So why is that?

The "secret" is that the linker (that gets invoked by the compiler) has been wrapped in the Nix build environment -- if we invoke ld, then we actually end up using a wrapper: ld-wrapper that does a number of additional things besides the tasks the linker normally carries out.

Whenever we supply a library to link to, the wrapper appends an -rpath parameter providing its location. Furthermore, it appends the path to the dynamic linker/loader (-dynamic-linker) so that the resulting executable can load the shared libraries on startup.

For example, when producing an executable, the compiler may invoke the following command that links a library to a piece of object code:

$ ld test.o -lz -o test

in reality, ld has been wrapped and executes something like this:

$ ld test.o -lz \
  -rpath /nix/store/31w31mc8i...-zlib-1.2.8/lib \
  -dynamic-linker \
    /nix/store/hd6km3hscb...-glibc-2.21/lib/ld-linux-x86-64.so.2 \
  ...
  -o test

As may be observed, the wrapper transparently appends the path to zlib as an RPATH parameter and provides the path to the dynamic linker.

The RPATH attribute is basically a colon separated string of paths in which the dynamic linker looks for its shared dependencies. The RPATH is hardwired into an ELF binary.

Consider the following simple C program (test.c) that displays the version of the zlib library that it links against:

#include <stdio.h>
#include <zlib.h>

int main()
{
    printf("zlib version is: %s\n", ZLIB_VERSION);
    return 0;
}

With the following Nix expression we can compile an executable from it and link it against the zlib library:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "test";
  buildInputs = [ zlib ];
  buildCommand = ''
    gcc ${./test.c} -lz -o test
    mkdir -p $out/bin
    cp test $out/bin
  '';
}

When we build the above package:

nix-build test.nix

and inspect the program headers of the ELF binary, we can observe that the dynamic linker (program interpreter) corresponds to an instance residing in the Nix store:

$ readelf -l ./result/bin/test 

Elf file type is EXEC (Executable file)
Entry point 0x400680
There are 9 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000400040 0x0000000000400040
                 0x00000000000001f8 0x00000000000001f8  R E    8
  INTERP         0x0000000000000238 0x0000000000400238 0x0000000000400238
                 0x0000000000000050 0x0000000000000050  R      1
      [Requesting program interpreter: /nix/store/hd6km3hs...-glibc-2.21/lib/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                 0x000000000000096c 0x000000000000096c  R E    200000
  LOAD           0x0000000000000970 0x0000000000600970 0x0000000000600970
                 0x0000000000000260 0x0000000000000268  RW     200000
  DYNAMIC        0x0000000000000988 0x0000000000600988 0x0000000000600988
                 0x0000000000000200 0x0000000000000200  RW     8
  NOTE           0x0000000000000288 0x0000000000400288 0x0000000000400288
                 0x0000000000000020 0x0000000000000020  R      4
  GNU_EH_FRAME   0x0000000000000840 0x0000000000400840 0x0000000000400840
                 0x0000000000000034 0x0000000000000034  R      4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     8
  PAX_FLAGS      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000         8

Furthermore, if we inspect the dynamic section of the binary, we will see that an RPATH attribute has been hardwired into it providing a collection of library paths (including the path to zlib):

$ readelf -d ./result/bin/test 

Dynamic section at offset 0x988 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000f (RPATH)              Library rpath: [
/nix/store/8w39iz6sp...-test/lib64:
/nix/store/8w39iz6sp...-test/lib:
/nix/store/i9nn1fkcy...-gcc-4.9.3/libexec/gcc/x86_64-unknown-linux-gnu/4.9.3:
/nix/store/31w31mc8i...-zlib-1.2.8/lib:
/nix/store/hd6km3hsc...-glibc-2.21/lib:
/nix/store/i9nn1fkcy...-gcc-4.9.3/lib]
 0x000000000000001d (RUNPATH)            Library runpath: [
/nix/store/8w39iz6sp...-test/lib64:
/nix/store/8w39iz6sp...-test/lib:
/nix/store/i9nn1fkcy...-gcc-4.9.3/libexec/gcc/x86_64-unknown-linux-gnu/4.9.3:
/nix/store/31w31mc8i...-zlib-1.2.8/lib:
/nix/store/hd6km3hsc...-glibc-2.21/lib:
/nix/store/i9nn1fkcy...-gcc-4.9.3/lib]
 0x000000000000000c (INIT)               0x400620
 0x000000000000000d (FINI)               0x400814
 0x0000000000000019 (INIT_ARRAY)         0x600970
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x600978
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x0000000000000004 (HASH)               0x4002a8
 0x0000000000000005 (STRTAB)             0x400380
 0x0000000000000006 (SYMTAB)             0x4002d8
 0x000000000000000a (STRSZ)              528 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x600b90
 0x0000000000000002 (PLTRELSZ)           72 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x4005d8
 0x0000000000000007 (RELA)               0x4005c0
 0x0000000000000008 (RELASZ)             24 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x4005a0
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x400590
 0x0000000000000000 (NULL)               0x0

As a result, the program works as expected:

$ ./result/bin/test 
zlib version is: 1.2.8

Patching existing ELF binaries


To summarize, the reason why ELF binaries produced in a Nix build environment work is because they refer to the correct path of the dynamic linker and have an RPATH value that refers to the paths of the shared libraries that it needs.

Fortunately, we can accomplish the same thing with prebuilt binaries by using the PatchELF tool. With PatchELF we can patch existing ELF binaries to have a different dynamic linker and RPATH.

Running the following instruction in a Nix expression allows us to change the dynamic linker of the pngout executable shown earlier:

$ patchelf --set-interpreter \
    ${stdenv.glibc}/lib/ld-linux-x86-64.so.2 $out/bin/pngout

By inspecting the dynamic section of a binary, we can find out what shared libraries it requires:

$ readelf -d ./result/bin/pngout

Dynamic section at offset 0x15968 contains 22 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x400ea8
 0x000000000000000d (FINI)               0x413a78
 0x0000000000000004 (HASH)               0x400260
 0x000000006ffffef5 (GNU_HASH)           0x4003b8
 0x0000000000000005 (STRTAB)             0x400850
 0x0000000000000006 (SYMTAB)             0x4003e8
 0x000000000000000a (STRSZ)              379 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x615b20
 0x0000000000000002 (PLTRELSZ)           984 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x400ad0
 0x0000000000000007 (RELA)               0x400a70
 0x0000000000000008 (RELASZ)             96 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x400a30
 0x000000006fffffff (VERNEEDNUM)         2
 0x000000006ffffff0 (VERSYM)             0x4009cc
 0x0000000000000000 (NULL)               0x0

According to the information listed above, two libraries are required (libm.so.6 and libc.so.6) which can be provided by the glibc package. We can change the executable's RPATH in the Nix expression as follows:

$ patchelf --set-rpath ${stdenv.glibc}/lib $out/bin/pngout

We can write a revised Nix expression for pngout (taking patching into account) that looks as follows:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "pngout-20130221";

  src = fetchurl {
    url = http://static.jonof.id.au/dl/kenutils/pngout-20130221-linux.tar.gz;
    sha256 = "1qdzmgx7si9zr7wjdj8fgf5dqmmqw4zg19ypg0pdz7521ns5xbvi";
  };

  installPhase = ''
    mkdir -p $out/bin
    cp x86_64/pngout $out/bin
    patchelf --set-interpreter \
        ${stdenv.glibc}/lib/ld-linux-x86-64.so.2 $out/bin/pngout
    patchelf --set-rpath ${stdenv.glibc}/lib $out/bin/pngout
  '';
}

When we build the expression:

$ nix-build pngout.nix

and try to run the executable:

$ ./result/bin/pngout 
PNGOUT [In:{PNG,JPG,GIF,TGA,PCX,BMP}] (Out:PNG) (options...)
by Ken Silverman (http://advsys.net/ken)
Linux port by Jonathon Fowler (http://www.jonof.id.au/pngout)

We will see that the executable works as expected!

A more complex example: Quake 4 demo


The pngout example shown earlier is quite simple as it is only a tarball with only one executable that must be installed and patched. Now that we are familiar with some basic concepts -- how should we a approach a more complex prebuilt package, such as a computer game like the Quake 4 demo?

When we download the Quake 4 demo installer for Linux, we actually get a Loki setup tools based installer that is a self-extracting shell script executing an installer program.

Unfortunately, we cannot use this installer program in NixOS for two reasons. First, the installer executes (prebuilt) executables that will not work. Second, to use the full potential of NixOS, it is better to deploy packages with Nix in isolation in the Nix store.

Fortunately, running the installer with the --help parameter reveals that it is also possible to extract its contents without running the installer:

$ bash ./quake4-linux-1.0-demo.x86.run --noexec --keep

After executing the above command-line instruction, we can find the extracted files in the ./quake4-linux-1.0-demo in the current working directory.

The next step is figuring out where the game files reside and which binaries need to be patched. A rough inspection of the extracted folder:

$ cd quake4-linux-1.0-demo
$ ls
bin
Docs
License.txt
openurl.sh
q4base
q4icon.bmp
README
setup.data
setup.sh
version.info

reveals to me that we have both files of installer (./setup.data) and the game intermixed with each other. Some files seem to be required to run the game, but the some others, such as the setup files (e.g. the ones residing in setup.data/) are unnecessary.

Running the following command helps me to figure out which ELF binaries we may have to patch:

$ file $(find . -type f)         
./Docs/QUAKE4_demo_readme.txt:     Little-endian UTF-16 Unicode text, with CRLF line terminators
./bin/Linux/x86/libstdc++.so.5:    ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
./bin/Linux/x86/quake4-demo:       POSIX shell script, ASCII text executable
./bin/Linux/x86/quake4.x86:        ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.0.30, stripped
./bin/Linux/x86/quake4-demoded:    POSIX shell script, ASCII text executable
./bin/Linux/x86/libgcc_s.so.1:     ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, not stripped
./bin/Linux/x86/q4ded.x86:         ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.0.30, stripped
./README:                          ASCII text
./version.info:                    ASCII text
./q4base/game100.pk4:              Zip archive data, at least v2.0 to extract
./q4base/mapcycle.scriptcfg:       ASCII text, with CRLF line terminators
./q4base/game000.pk4:              Zip archive data, at least v1.0 to extract
./License.txt:                     ISO-8859 text, with very long lines
./openurl.sh:                      POSIX shell script, ASCII text executable
./q4icon.bmp:                      PC bitmap, Windows 3.x format, 48 x 48 x 24
...

As we can see in the output, the ./bin/Linux/x86 sub folder contains a number of ELF executables and shared libraries that most likely require patching.

As with the previous example (pngout), we can use readelf to inspect what libraries the ELF executables require. The first executable q4ded.x86 has the following dynamic section:

$ cd ./bin/Linux/x86
$ readelf -d q4ded.x86 

Dynamic section at offset 0x366220 contains 25 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libdl.so.2]
 0x00000001 (NEEDED)                     Shared library: [libstdc++.so.5]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
...

According to the above information, the executable requires a couple of libraries that seem to be stored in the same package (in the same folder to be precise): libstdc++.so.5 and libgcc_s.so.1.

Furthermore, it also requires a number of libraries that are not in the same folder. These missing libraries must be provided by external packages. I know from experience that the remaining libraries: libpthread.so.0, libdl.so.2, libm.so.6, libc.so.6, are provided by the glibc package.

The other ELF executable has the following library references:

$ readelf -d ./quake4.x86 

Dynamic section at offset 0x3779ec contains 29 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libSDL-1.2.so.0]
 0x00000001 (NEEDED)                     Shared library: [libpthread.so.0]
 0x00000001 (NEEDED)                     Shared library: [libdl.so.2]
 0x00000001 (NEEDED)                     Shared library: [libstdc++.so.5]
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
 0x00000001 (NEEDED)                     Shared library: [libX11.so.6]
 0x00000001 (NEEDED)                     Shared library: [libXext.so.6]
...

This executable has a number dependencies that are identical to the previous executable. Additionally, it requires: libSDL-1.2.so.0 that can be provided by SDL, libX11.so.6 by libX11 and libXext.so.6 by libXext

Besides the executables, the shared libraries bundled with the package may also have dependencies on shared libraries. We need to inspect and fix these as well.

Inspecting the dynamic section of libgcc_s.so.1 reveals the following:

$ readelf -d ./libgcc_s.so.1 

Dynamic section at offset 0x7190 contains 23 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
...

The above library has a dependency on libc.so.6 which can be provided by glibc

The remaining library (libstdc++.so.5) has the following dependencies:

$ readelf -d ./libstdc++.so.5 

Dynamic section at offset 0xadd8c contains 25 entries:
  Tag        Type                         Name/Value
 0x00000001 (NEEDED)                     Shared library: [libm.so.6]
 0x00000001 (NEEDED)                     Shared library: [libgcc_s.so.1]
 0x00000001 (NEEDED)                     Shared library: [libc.so.6]
...

It seems to depend on libgcc_s.so.1 residing in the same folder. Similar to the previous binaries, libm.so.6, libc.so.6 provided can be provided by glibc.

With the gathered information so far, we can write the following Nix expression that we can use as a first attempt to run the game:

with import <nixpkgs> { system = "i686-linux"; };

stdenv.mkDerivation {
  name = "quake4-demo-1.0";
  src = fetchurl {
    url = ftp://ftp.idsoftware.com/idstuff/quake4/demo/quake4-linux-1.0-demo.x86.run;
    sha256 = "0wxw2iw84x92qxjbl2kp5rn52p6k8kr67p4qrimlkl9dna69xrk9";
  };
  buildCommand = ''
    # Extract files from the installer
    cp $src quake4-linux-1.0-demo.x86.run
    bash ./quake4-linux-1.0-demo.x86.run --noexec --keep
    
    # Move extracted files into the Nix store
    mkdir -p $out/libexec
    mv quake4-linux-1.0-demo $out/libexec
    cd $out/libexec/quake4-linux-1.0-demo
    
    # Remove obsolete setup files
    rm -rf setup.data
    
    # Patch ELF binaries
    cd bin/Linux/x86
    patchelf --set-interpreter ${stdenv.cc.libc}/lib/ld-linux.so.2 ./quake4.x86
    patchelf --set-rpath $(pwd):${stdenv.cc.libc}/lib:${SDL}/lib:${xlibs.libX11}/lib:${xlibs.libXext}/lib ./quake4.x86
    chmod +x ./quake4.x86
    
    patchelf --set-interpreter ${stdenv.cc.libc}/lib/ld-linux.so.2 ./q4ded.x86
    patchelf --set-rpath $(pwd):${stdenv.cc.libc}/lib ./q4ded.x86
    chmod +x ./q4ded.x86
    
    patchelf --set-rpath ${stdenv.cc.libc}/lib ./libgcc_s.so.1
    patchelf --set-rpath $(pwd):${stdenv.cc.libc}/lib ./libstdc++.so.5
  '';
}

In the above Nix expression, we do the following:

  • We import the Nixpkgs collection so that we can provide the external dependencies that the package needs. Because the executables are 32-bit x86 binaries, we need to refer to packages built for the i686-linux architecture.
  • We download the Quake 4 demo installer from Id software's FTP server.
  • We automate the steps we have done earlier -- we extract the files from the installer, move them into Nix store, prune the obsolete setup files, and finally we patch the ELF executables and libraries with the paths to the dependencies that we have discovered in our investigation.

We should now be able to build the package:

$ nix-build quake4demo.nix

and investigate whether the executables can be started:

./result/libexec/quake4-linux-1.0-demo/bin/Linux/x86/quake4.x86

Unfortunately, it does not seem to work:

...
no 'q4base' directory in executable path /nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86, skipping
no 'q4base' directory in current durectory /home/sander/quake4, skipping

According to the output, it cannot find the q4base/ folder. Running the same command with strace reveals why:

$ strace -f ./result/libexec/quake4-linux-1.0-demo/bin/Linux/x86/quake4.x86
...
stat64("/nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86/q4base", 0xffd7b230) = -1 ENOENT (No such file or directory)
write(1, "no 'q4base' directory in executa"..., 155no 'q4base' directory in executable path /nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86, skipping
) = 155
...

It seems that the program searches relative to the current working directory. The missing q4base/ folder apparently resides in the base directory of the extracted folder.

By changing the current working directory and invoking the executable again, the q4base/ directory can be found:

$ cd result/libexec/quake4-linux-1.0-demo
$ ./bin/Linux/x86/quake4.x86
...
--------------- R_InitOpenGL ----------------
Initializing SDL subsystem
Loading GL driver 'libGL.so.1' through SDL
libGL error: unable to load driver: i965_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: i965
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  33
  Current serial number in output stream:  34

Despite fixing the problem, we have run into another one! Apparently the OpenGL driver cannot be loaded. Running the same command again with the following environment variable (source):

$ export LIBGL_DEBUG=verbose

shows us what is causing it:

--------------- R_InitOpenGL ----------------
Initializing SDL subsystem
Loading GL driver 'libGL.so.1' through SDL
libGL: OpenDriver: trying /run/opengl-driver-32/lib/dri/tls/i965_dri.so
libGL: OpenDriver: trying /run/opengl-driver-32/lib/dri/i965_dri.so
libGL: dlopen /run/opengl-driver-32/lib/dri/i965_dri.so failed (/nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86/libgcc_s.so.1: version `GCC_3.4' not found (required by /run/opengl-driver-32/lib/dri/i965_dri.so))
libGL error: unable to load driver: i965_dri.so
libGL error: driver pointer missing
libGL error: failed to load driver: i965
libGL: OpenDriver: trying /run/opengl-driver-32/lib/dri/tls/swrast_dri.so
libGL: OpenDriver: trying /run/opengl-driver-32/lib/dri/swrast_dri.so
libGL: dlopen /run/opengl-driver-32/lib/dri/swrast_dri.so failed (/nix/store/0kfgsjryycsk5kfv97phj8ypv66n6caz-quake4-demo-1.0/libexec/quake4-linux-1.0-demo/bin/Linux/x86/libgcc_s.so.1: version `GCC_3.4' not found (required by /run/opengl-driver-32/lib/dri/swrast_dri.so))
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  154 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  33
  Current serial number in output stream:  34

Apparently, the libgcc_so.1 library bundled with the game is conflicting with Mesa3D. According to this GitHub issue, replacing the conflicting version with the host system's GCC's version fixes it.

In our situation, we can accomplish this by appending the path to the host system's GCC library folder to the RPATH of the binaries referring to it and by removing the conflicting library from the package.

Moreover, we can address the annoying issue with the missing q4base/ folder by creating wrapper scripts that change the current working folder and invoke the executable.

The revised expression taking these aspects into account will be as follows:

with import <nixpkgs> { system = "i686-linux"; };

stdenv.mkDerivation {
  name = "quake4-demo-1.0";
  src = fetchurl {
    url = ftp://ftp.idsoftware.com/idstuff/quake4/demo/quake4-linux-1.0-demo.x86.run;
    sha256 = "0wxw2iw84x92qxjbl2kp5rn52p6k8kr67p4qrimlkl9dna69xrk9";
  };
  buildCommand = ''
    # Extract files from the installer
    cp $src quake4-linux-1.0-demo.x86.run
    bash ./quake4-linux-1.0-demo.x86.run --noexec --keep
    
    # Move extracted files into the Nix store
    mkdir -p $out/libexec
    mv quake4-linux-1.0-demo $out/libexec
    cd $out/libexec/quake4-linux-1.0-demo
    
    # Remove obsolete setup files
    rm -rf setup.data
    
    # Patch ELF binaries
    cd bin/Linux/x86
    patchelf --set-interpreter ${stdenv.cc.libc}/lib/ld-linux.so.2 ./quake4.x86
    patchelf --set-rpath $(pwd):${stdenv.cc.cc}/lib:${stdenv.cc.libc}/lib:${SDL}/lib:${xlibs.libX11}/lib:${xlibs.libXext}/lib ./quake4.x86
    chmod +x ./quake4.x86
    
    patchelf --set-interpreter ${stdenv.cc.libc}/lib/ld-linux.so.2 ./q4ded.x86
    patchelf --set-rpath $(pwd):${stdenv.cc.cc}/lib:${stdenv.cc.libc}/lib ./q4ded.x86
    chmod +x ./q4ded.x86
    
    patchelf --set-rpath $(pwd):${stdenv.cc.libc}/lib ./libstdc++.so.5
    
    # Remove libgcc_s.so.1 that conflicts with Mesa3D's libGL.so
    rm ./libgcc_s.so.1
    
    # Create wrappers for the executables
    mkdir -p $out/bin
    cat > $out/bin/q4ded <<EOF
    #! ${stdenv.shell} -e
    cd $out/libexec/quake4-linux-1.0-demo
    ./bin/Linux/x86/q4ded.x86 "\$@"
    EOF
    chmod +x $out/bin/q4ded
    
    cat > $out/bin/quake4 <<EOF
    #! ${stdenv.shell} -e
    cd $out/libexec/quake4-linux-1.0-demo
    ./bin/Linux/x86/quake4.x86 "\$@"
    EOF
    chmod +x $out/bin/quake4
  '';
}

We can install the revised package in our Nix profile as follows:

$ nix-env -f quake4demo.nix -i quake4-demo

and conveniently run it from the command-line:

$ quake4


Happy playing!

(As a sidenote: besides creating a wrapper script, it is also possible to create a Freedesktop compliant .desktop entry file, so that it can be launched from the KDE/GNOME applications menu, but I leave this an open exercise to the reader!)

Conclusion


In this blog post, I have explained that prebuilt binaries do not work out of the box in NixOS. The main reason is that they cannot find their dependencies in their "usual locations", because these do not exist in NixOS. As a solution, it is possible to patch binaries with a tool called PatchELF to provide them the correct location to the dynamic linker and the paths to the libraries they need.

Moreover, I have shown two example packaging approaches (a simple and complex one) that should be relatively easy to repeat as an exercise.

Although source deployments typically work out of the box with few or no modifications, getting prebuilt binaries to work is often a journey that requires patching, wrapping, and experimentation. In this blog post I have described a few tricks that can be applied to make prebuilt packages work.

The approach described in this blog post is not the only solution to get prebuilt binaries to work in NixOS. An alternative approach is composing FHS-compatible chroot environments from Nix packages. This solution simulates an environment in which dependencies can be found in their common FHS locations. As a result, we do not require any modifications to a binary.

Although FHS chroot environments are conceptually nice, I would still prefer the patching approach described in this blog post unless there is no other way to make a package work properly -- it has less overhead, does not require any special privileges (e.g. super user rights), we can use the distribution mechanisms of Nix in its full extent, and we can also install a package as an unprivileged user.

Steam is a notable exception for using FHS compatible choot environments, because it is a deployment tool that conflicts with Nix's deployment properties.

As a final practical note: if you want to repeat the Quake 4 demo packing process, please check the following:

  • To enable hardware accelerated OpenGL for 32-bit applications in a 64-bit NixOS, add the following property to /etc/nixos/configuration.nix:

    hardware.opengl.driSupport32Bit = true;
    
  • Id sofware's FTP server seems to be quite slow to download from. You can also obtain the demo from a different download site (e.g. Fileplanet) and run the following command to get it imported into the Nix store:

    $ nix-prefetch-url file:///home/sander/quake4-linux-1.0-demo.x86.run
    

Tuesday, October 13, 2015

Setting up a basic software configuration management process in a small organization

I have been responsible for many things in my past and current career. Besides research and development, I have also been responsible for software and systems configuration management in small/medium sized companies, such as my current employer.

I have observed that in organizations like these, configuration management (CM) is typically nobody's (full) responsibility. Preferably, people want to stick themselves to their primary responsibilities and typically carry out change activities in an ad-hoc and unstructured way.

Not properly implementing changes have a number of serious implications. For example, some problems I have encountered are:

  • Delays. There are many factors that will unnecessarily increase the time it will take to implement a change. Many of my big delays were caused by the fact that I always have to search for all the relevant installation artifacts, such as documentation, installation discs, and so on. I have also encountered many times that artifacts were missing requiring me to obtain copies elsewhere.
  • Errors. Any change could potentially lead to errors for many kinds of reasons. For example, implementing a set of changes in the wrong order could break a system. Also, components of which a system consist may have complex dependencies on other components that have to be met. Quite often, it is not fully clear what the dependencies of a component or system are, especially when documentation is incomplete or lacking.

    Moreover, after having solved an error, you need to remember many arbitrary things, such as workarounds, that tend to become forgotten knowledge over time.
  • Disruptions. When implementing changes, a system may be partially or fully unavailable until all changes have been implemented. Preferably this time window should be as short as possible. Unfortunately, the inconsistency time window most likely becomes quite big when the configuration management process is not optimal or subject to errors.

It does not matter if an organization is small or big, but these problems cost valuable time and money. To alleviate these problems, it is IMO unavoidable to have a structured way of carrying out changes so that a system maintains its integrity.

Big organizations typically have information systems, people and management procedures to support structured configuration management, because failures are typically too costly for them. There are also standards available (such as the IEEE 828-2012 Standard for Configuration Management in Systems and Software Engineering) that they may use as a reference for implementing a configuration management process.

However, in small organizations people typically refrain from thinking about a process at all while they keep suffering from the consequences, because they think they are too small for it. Furthermore, they find it too costly to invest in people or an information system supporting configuration management procedures. Consulting a standard is something that is generally considered a leap too far.

In this blog post, I will describe a very basic software configuration management process I have implemented at my current employer.

The IEEE Standard for Configuration Management


As crazy as this may sound, I have used the IEEE 828-2012 standard as a reference for my implementation. The reason why I consulted this standard besides the fact that using an existing and reasonably well-established reference is good, is that I was already familiar with it, because of my previous background as a PhD researcher in software deployment.

The IEEE standard defines a framework of so-called "lower-level processes" from which a configuration management process can be derived. The most important lower-level processes are:

  • CM identification, which concerns identifying, naming, describing, versioning, storing and retrieving configuration items (CIs) and their baselines.
  • CM change control is about planning, requesting, approval and verification of change activities.
  • CM status accounting is about identifying the status of CIs and change requests.
  • CM auditing concerns identifying, tracing and reporting discrepancies with regards to the CIs and their baselines.
  • CM release management is about planning, defining a format for distribution, delivery, and archival of configuration items.

All the other lower-level processes have references to the lower-level processes listed above. CM planning is basically about defining a plan how to carry out the above activities. CM management is about actually installing the tools involved, executing the process, monitoring its progress, and status and revising/optimizing the plan if any problem occurs.

The remaining two lower-level processes concern outside involvement -- Supplier configuration item control concerns CIs that are provided by external parties. Interface control concerns the implications of configuration changes that concern external parties. I did not take the traits of these lower-level processes into account in the implementation.

Implementing a configuration management process


Implementing a configuration management process (according to the IEEE standard) starts by developing configuration management plan. The standard mentions many planning aspects, such as identifying the information needs, reporting needs, the reporting frequency, and the information needed to manage CM activities. However, from my perspective, in a small organization many of these aspects are difficult to answer in advance, in particular the information needs.

As a rule of thumb, I think that when you do not exactly know what is needed, consider that the worst thing could happen -- the office burns down and everything gets lost. What does it take to reproduce the entire configuration from scratch?

This is how I have implemented the main lower-level processes:

CM identification


A configuration item is any structural unit that is distinguishable and configurable. In our situation, the most important kind of configuration item is a machine configuration (e.g. a physical machine or a virtual machine hosted in an IaaS environment, such as Amazon EC2), or a container configuration (such as an Apache Tomcat container or PaaS service, such as Elastic Beanstalk).

Machines/containers belong to an environment. Some examples of environments that we currently maintain are: the production environment containing the configurations of the production machines of our service, test contains the configurations of the test environment, and internal contains the configurations of our internal IT infrastructure, such as our internal Hydra build cluster, and other peripherals, such as routers and printers.

Machines run a number of applications that may have complex installation procedures. Moreover, identical/similar application configurations may have to be deployed to multiple machines.

For storing the configurations of the CIs, I have set up a Git repository that follows a specific directory structure of three levels:

<environment>/<machine | container>/<application>

Each directory contains all artifacts (e.g. keys, data files, configuration files, scripts, documents etc.) required to reproduce a CI's configuration from scratch. Moreover, each directory has a README.md markdown file:

  • The top-level README.md describes which environments are available and what their purposes are.
  • The environment-level README.md describes which machines/containers are part of it, a brief description of their purpose, and a picture showing how they are connected. I typically use Dia to draw them, because the tool is simple and free.
  • The machine-level README.md describes the purpose of the machine and the activities that must be carried out to reproduce its configuration from scratch.
  • The application-level README.md captures the steps that must be executed to reproduce an application's configuration.

When storing artifacts and writing README.md files, I try to avoid duplication as much as possible, because it makes it harder to keep the repository consistent and maintainable:

  • README.md files are not supposed to be tool manuals. I mention the steps that must be executed and what their purposes are, but I avoid explaining how a tool works. That is the purpose of the tool's manual.
  • When there are common files used among machines, applications or environments, I do not duplicate them. Instead, I use a _common/ folder that I put one directory level higher. For example, the _common/ folder in the internal/ directory contains shared artifacts that are supposed to reused among all machines belonging to our internal IT infrastructure.
  • I also capture common configuration steps in a separate README.md and refer to it, instead of duplicating the same steps in multiple README.md files.

Because I use Git, versioning, storage and retrieval of configurations is implicit. I do not have to invent something myself or think too much about it. For example, I do not have to manually assign version numbers to CIs, because Git already computes them for each change. Moreover, because I use textual representations of most of the artifacts I can also easily compare versions of the configurations.

Furthermore, besides capturing and storing all the prerequisites to reproduce a configuration, I also try to automate this process as much as possible. For most of the automation aspects, I use tools from the Nix project, such as the Nix package manager for individual packages, NixOS for system configurations, Disnix for distributed services, and NixOps for networks of machines.

Tools in the Nix project are driven by declarative specifications -- a specification captures the structure of a system, e.g. their components and their dependencies. From this specification the entire deployment process will be derived, such as building the components from source code, distributing them to the right machines in the network, and activating them in the right order.

Using a declarative deployment approach prevents me writing down the activities to carry out, because they are implicit. Also, there is no need describing the structure of the system because it is already captured in the deployment specification.

Unfortunately, not all machine's deployment processes can be fully automated with Nix deployment tools (e.g. non-Linux machines and special purpose peripherals, such as routers) still requiring me to carry out some configuration activities manually.

CM change control


Implementing changes may cause disruptions costing time and money. That is why the right people must be informed and approval is needed. Big organizations typically have sophisticated management procedures including request and approval forms, but in a small organization it typically suffices to notify people informally before implementing a change.

Besides notifying people, I also take the following things into account while implementing changes:

  • Some configuration changes require validation including review and testing before they can be actually implemented in production. I typically keep the master Git branch in a state releasable state, meaning that it is ready to be deployed into production. Any changes that require explicit validation go into a different branch first.

    Moreover, when using tools from the Nix project it is relatively easy to reliably test changes first by deploying a system into a test environment, or by spawning virtual NixOS machines in which integration tests can be executed.
  • Sometimes you need to make an analysis of the impact and costs that a change would bring. Up-to-date and consistent documentation of the CIs including their dependencies makes this process more reliable.

    Furthermore, with tools from the Nix project you can also make better estimations by executing a dry-run deployment process -- the dry run shows what activities will be carried out without actually executing them or bringing the system down.
  • After a change has been deployed, we also need to validate whether the new configuration is correct. Typically, this requires testing.

    Tools from the Nix project support congruent deployment, meaning that if the deployment has succeeded, the actual configuration is guaranteed to match the deployment specification for the static parts of a system, giving better guarantees about its validity.
  • Also you have to pick the right moment to implement potentially disrupting changes. For example, it is typically a bad idea to do this while your service is under peak load.

CM status accounting


It is also highly desirable to know what the status of the CIs and the change activities are. The IEEE standard goes quite far in this. For example, the overall state of the system may converge into a certain direction (e.g. in terms of features complete, error ratios etc.), which you continuously want to measure and report about. I think that in a small organization these kinds of status indicators are typically too complex to define and measure, in particular in the early stages of a development process.

However, I think the most important status indicator that you never want to lose track of is the following: does everything (still) work?

There are two facilities that help me out a lot in keeping a system in working state:

  • Automating deployment with tools from the Nix project ensure that the static parts of a deployed system are congruent with the deployment configuration specification and atomic -- either a deployment is in the old configuration or the new configuration but never in an inconsistent mix of the two. As a result, we have fewer broken configurations as a result of (re)deploying a system.
  • We must also observe a system's runtime behavior and take action if things will grow out of hand. For example, when a machine runs out of system resources.

    Using a monitoring service, such as Zabbix or Datadog, helps me a lot in accomplishing this. They can also be used to configure alarms that warn you when things become critical.

CM auditing


Another important aspect is the integrity of the configurations repository. How can we be sure that what is stored inside the repository matches the actual configurations and that the configuration procedures still work?

Fortunately, because we use tools from the Nix project, there is relatively little audit work we need to do. With Nix-related tools the deployment process is fully automated. As a consequence, we need to adapt the deployment specification when we intend to make changes. Moreover, since the deployment specifications of Nix-related tools are congruent, we know that the static parts of a system are guaranteed to match the actual configurations if the (re)deployment process succeeded.

However, for non-NixOS machines and other peripherals, we must still manually check once in a while whether the indented configuration matches. I made it a habit to go through them once a month and to adjust the documentation if any discrepancies were found.

CM release management


When updating a configuration file, we must also release the new corresponding configuration items. The IEEE standard describes many concerns, such as approval procedures, requirements on the distribution mechanism and so on.

For me, most of these concerns are unimportant, especially in a small organization. The only thing that matters to me is that a release process is fully automated, reliable, reproducible. Fortunately, the deployment tools from the Nix project support these properties quite well.

Discussion


In this blog post, I have described a basic configuration management process that I have implemented in a small organization.

Some people will probably argue that defining a CM process in a small organization looks crazy. Some people think they do not need a process and that it is too much of an investment. Following an IEEE standard is generally considered a leap too far.

In my opinion, however, the barrier of implementing a CM process is not actually not that high. From my experience, the biggest investment is setting up a configuration management repository. Although big organizations typically have sophisticated information systems, I have also shown that using a simple filesystem layout and collection of free and open source tools (e.g. Git, Dia, Nix) a simple variant of such a repository can be set up with relatively little effort.

I also observed that automating CM tasks helps a lot, in particular using a declarative and congruent deployment approach, such as Nix. With a declarative approach, configuration activities are implicit (they are a consequence of applying a change in the deployment specification) and do not have to be documented. Furthermore, because Nix's deployment models are congruent, the static aspects of a configuration are guaranteed to match the deployment specifications. Moreover, the deployment model serves as the documentation, because it captures the structure of a system.

So how beneficial is setting up a CM process in a small organization? I observed many benefits. For example, a major benefit is that I can carry out many CM tasks much faster. I no longer have to waste much of my time looking for configuration artifacts and documentation. Also, because the steps to carry out are documented or automated, there are fewer things I need to (re)discover or solve while implementing a change.

Another benefit is that I can more reliably estimate the impact of implementing changes, because the CIs and their relationships are known. More knowledge, also causes fewer errors.

Although a simple CM approach provides benefits and many aspects can be automated, it always requires discipline from all people involved. For example, when errors are discovered and configurations must be modified in a stressful situation, it is very tempting to bypass updating the documentation.

Moreover, communication is also an important aspect. For example, when notifying people of a potentially disrupting change, clear communication is required. Typically, also non-technical stakeholders must be informed. Eventually, you have to start developing formalized procedures to properly handle decision processes.

Finally, the CM approach described in this blog post is obviously too limited if a company grows. If an organization gets bigger, a more sophisticated and more formalized CM approach will be required.