A while ago, I have written a blog post about my IFF file format experiments in which I have developed a collection of libraries and tools capable of parsing and displaying two IFF file format applications. Some time before that, I have developed a Nix function capable of building software for AmigaOS allowing me to easily backport these experimental software packages to AmigaOS.
The development of these experimental packages were dormant for quite some time, but recently I was able to find some time to do some improvements. This time I have updated the Amiga video emulation library to support a few new features and to more accurately emulate an Amiga display.
Differences between Amiga and PC displays
So why is emulation of Amiga video modes necessary? Emulation of Amiga graphics hardware is required because pixels are encoded differently than "modern" PC hardware, resolutions have different semantics, and Amiga hardware has a few special screen modes to "squeeze" more colors out of the amount of available color registers.
Pixels
Encoding pixels on modern hardware is not so complicated these days. Modern hardware allows us to address each individual pixel's color value using 4 bytes per pixel in which one byte represents the red color intensity, one byte the blue color intensity and one byte the green color intensity. The fourth byte is often unused, or serves as the alpha component setting the transparency of a pixel in multi-layered images.
However, older PC hardware, such as those used during the DOS era, were often not powerful enough to address every pixel's color value individually, mainly due to memory constraints. Many classic DOS games used a 320x200 video resolution with a color palette consisting of 256 color values. Each pixel was encoded as a byte referring to an index value of a palette. This encoding is also known as chunky graphics.
System resources in the Amiga era were even more scarce, especially during its launch in 1985. The original graphics chipset was only capable of storing 32 color values in a palette, although there were a few special screen modes capable of squeezing more colors out of the 32 configurable ones. It also used a different way of encoding pixels, probably to make storage of graphics surfaces as memory efficient as possible.
In Amiga chipsets pixels are encoded as bitplanes rather than bytes. When using bitplane encoding, an image is stored multiple times in memory. In every image occurence, a bit represents a pixel. In the first occurence, a bit is the least significant bit of an index a value. In the last occurence a bit is the most significant bit of an index value. By adding all the bits together we can
determine the index value of the palette to determine a pixel's color value. For example, if we would encode an image using 16 colors, then we need 4 bitplane surfaces.
To be able to display an Amiga image on a PC we have to convert the bitplane format to either chunky graphics or RGB graphics. Likewise, to be able to display a PC image on an Amiga, we have to convert the pixel surface to bitplane format.
Palette
As explained earlier, due to memory constraints, Amiga graphics use a palette of 32 configurable color values (or 256 colors when using the newer AGA chipset is used). Each pixel refers to an index of a palette instead of a color value of its own.
The original chipset color values are stored in 16-bit color registers in which every color component consists of 4 bits (4 bits are unused). The newer AGA chipset as well as VGA graphics use 8 bits per color component.
Furthermore, the Amiga video chips have special screen modes to squeeze more colors out of the amount of color registers available. The Extra Half Brite (EHB) screen mode is capable of displaying 64 colors out of a predefined 32, in which the last 32 colors values have half of the intensity of the first 32 colors. The above screenshot is an example picture included with Deluxe Paint V using the EHB screenmode. A closer look at the floor in the picture look may reveal to you that an EHB palette is used.
The Hold-and-Modify (HAM) screen mode is used to modify a color component of the previous adjacent pixel or to pick a new color from the given palette. This screen mode makes it possible to use all possible color values (4096) in one screen with some loss of image quality.
The above screenshot is an example image included with Deluxe Paint V using HAM screenmode to utilise many more colors than the amount available color registers. A closer look at the image very may reveal that it has some quality loss because of the limitations of HAM compression.
To be able to properly display an image on a PC we need to convert 12-bit color values to 32-bit color values. Moreover, we also have to calculate each pixel's color value when HAM screen modes are used and convert the image to true color graphics, since a picture using HAM mode may use more than 256 colors.
Resolutions
Another difference between Amiga displays and "modern" displays are the display resolutions. On PCs, resolutions refer to the amount of pixels per scanline and the amount of scanlines per screen.
On the Amiga, resolutions only refer to the amount of pixels per scanline and this amount is often fixed. For example most displays support 320 lowres pixels per scanline, although this value can be slightly increased by utilising the overscan region. A high resolution screen has twice the amount of pixels per scanline compared to a low resolution screen. A super hires screen has double the amount of pixels per scanline compared to a high resolution screen. Moreover, a low resolution pixel is twice a wide as a high resolution pixel and so on.
Vertically, there are only two options. In principle, there are a fixed amount of scanlines on a display. Typically, NTSC displays support 200 scanlines and PAL displays support 256 scanlines. This amount can be slightly increased by utilising the overscan region of a display.
The amount of scanlines can be doubled, using a so-called interlace screen mode. However, interlace screen modes have a big drawback -- they draw the odd scanlines in one frame and the even ones in another. On displays with a short after glow, flickering may be observed.
Because of these differences, we may observe odd results in some cases when we convert an Amiga pixel surface to a chunky/RGB pixel surface. For example, a non-interlaced high resolution image looks twice as wide on a PC display than on an Amiga display, as can be seen in the left picture above, which contains a pixel-by-pixel conversion of a the Workbench screenshot.
To give it the same look as an Amiga display, we must correct its aspect ratio by doubling the amount of scanlines on the PC display, which is done in the right screenshot. People who have used the Amiga Workbench will immediately notice that this image is much closer to what can be seen on a real Amiga.
A library for performing conversions of Amiga display surfaces
While writing my previous blog post, I have developed a library (libamivideo) capable of automatically converting Amiga bitplane surfaces to chunky/RGB pixel surface and vice versa. In the latest release, I have added the following features and improvements:
Support for super hires video modes
Correct aspect ratio support
A better defined API
The library can be obtained from my GitHub page. I have also updated the SDL-based ILBM viewer (SDL_ILBM) as well as the native AmigaOS ILBM viewer (amiilbm) to use the new library implementation.
The filesystem organization that Nix uses, in which every package is stored in a separate folder in a so called "Nix store", is typically not a problem to get software packaged and deployed in source form.
Packages in Nix are deployed from build recipes called Nix expressions that typically invoke functions implementing abstractions ensuring that the specified dependencies can be found in a convenient manner.
For example, when providing Java or Perl dependencies, the generic builder may automatically set some required environment variables, such CLASSPATH or PERL5LIB containing Java or Perl dependencies. Moreover, a number of build utilities have been patched (such as ld) to automatically add shared libraries to an ELF executable's RPATH allowing it to find the libraries that it needs (you could see this as as static linking of shared libraries to an executable).
However, deployment of third party binary packages is a bit complicated and inconvenient in NixOS for the following reasons:
Nearly all software must be deployed by the Nix package manager, since nearly all of the system's components reside in the Nix store.
Third party binaries frequently require shared libraries and other kinds dependencies to run. However, in most cases these dependencies cannot be found, because executables assume that they reside in standard FHS folders such as /usr/lib, which do not exist in NixOS.
Executables using dynamic libraries cannot find the dynamic linker (e.g. /lib/ld-linux.so.2), because the corresponding file does not exist in NixOS.
As a solution, we often use PatchELF to change the path to the dynamic linker and modify the RPATH of a binary allowing it to find the libraries that it needs. We may also have to wrap executables in shell scripts to set certain environment variables, such as LD_LIBRARY_PATH.
Some applications that are packaged in Nixpkgs this way are Adobe Reader, Belastingaangifte (Dutch Tax administration) and the Android SDK, as I have described in twoearlier blog posts.
However, I have recently developed something that can solve this problem in a different way.
Composing FHS-compatible chroot environments
In my previous blog post, I have described how Nix user environments work, which are symlink trees that blend the contents of a collection of packages into a single symlink tree. User environments are primarily used to allow end-users to conveniently access packages installed by Nix from a single location.
We can also use the user environment mechanism in combination with setting up a chroot environment to simulate an FHS-compatible filesystem. This allows us to run an unpatched binary that works out of the box, if all required dependencies of the package are (correctly) specified.
I have created a Nix function called buildFHSChrootEnv {} which makes it possible to compose such environments. This function roughly does the following:
It invokes buildEnv {} to compose a so-called "/usr user environment". In this user environment, we synthesize the contents of all our specified packages into a single Nix profile. This profile contains all static parts of all packages and is supposed to simulate both the /usr directory as well as the static folders in the root folder of the chroot environment, such as /bin and /lib.
Besides the dependencies of the binary that we want to run, we also need a collection of base packages, such as the bash shell, GNU Coreutils, glibc, and the shadow password suite, since we also have to be able to run a basic Linux system and do basic shell activities inside the chroot environment. These packages are always part of the /usr profile.
It generates a global bash profile setting the PS1 environment variable to configure the shell prompt and other miscellaneous settings that a program may need in order to run.
It generates a collection of scripts taking care of setting up (or destroying) the dynamic parts of the chroot environment, since Nix packages and user environment are static and can never change after they have been built.
Setting up the dynamic parts of the chroot environment
Merely composing a /usr profile is not enough to get a working chroot environment because they only contain the static parts of a system, such as executables, libraries, and data files. We also have to set up a few dynamic parts. A collection of configuration scripts bundled with the generated /usr profile can be used for this purpose:
init-*-chrootenv: Initializes most dynamic parts of the chroot environment inside a directory with write permissions (i.e. /run/chrootenv/<name>).
First, it creates symlinks to the static folders of the /usr profile and the /usr symlink itself. Then the script ensures that the Name Service Switch (NSS) facilities work by symlinking /etc/{passwd,group,shadow,hosts,resolv.conf,nsswitch.conf) of the host system into the chroot environment. Then some other files from the host are symlinked, such as PAM configuration files and font configuration stuff. Finally, a few empty state folders are created, such as: /var, /run, /tmp.
The initialization script has to be executed only once.
mount-*-chrootenv: Bind mounts a number of directories of the host system into the chroot folder, so that files from the host system can be used while we have entered the chroot environment.
Currently, we bind mount the Nix store (/nix/store), directories used by the kernel: /dev, /dev/pts, /dev/shm, /proc, /sys, users' home directories: /home, a few state directories: /var, /run, and the host's /etc directory containing shared configuration files, such as the ones related to NSS and PAM.
This script has to be executed every time we have booted into NixOS.
load-*-chrootenv: Enters the chroot environment and launches a shell. The shell inherits a few environment variables from the host environment, such as DISPLAY allowing someone to run GUI applications and TERM to make certain fancy console applications to work properly.
umount-*-chrootenv: Unmounts the bind mounts.
destroy-*-chrootenv: Removes the writable chrootenv directory.
tl;dr - Getting Steam to work under NixOS
If you're still awake and still interested in reading this blog post, I will reveal why I have done all this work, which is probably much more exciting to read. In fact, this function's primary use case is to get Steam working in NixOS. Steam is a very special piece of software compared to other binary-only packages, because:
Steam is also a deployment tool (like Nix) capable of automatically downloading, installing and upgrading software packages. In Steam's case, these software packages are mostly games.
Software packages installed by Steam have the same problems as the deployment of "ordinary" binary packages in NixOS -- they don't work out of the box, because they cannot find the dynamic linker and their dependencies, such as shared libraries.
Of course, we can patch the executables installed by Steam, but Steam wants control over them and does not like patched versions of software. Furthermore, patching the packages makes it impossible for Steam to update them.
It is also a bit impractical to patch the executables every time they are updated, because I have seen that the games I have are updated quite frequently.
By using the buildFHSChrootEnv {} Nix function getting Steam (including the games installed by it) to run in NixOS is a piece of cake. First I had to package Steam itself, which was quite easy:
I just had to unpack the Steam Debian package and move its contents into the Nix store. I intentionally did not patch or wrap anything.
Then I had to create a Nix expression that composes a FHS-compatible chroot environment with Steam's dependencies and runtime settings (such as OpenGL driver settings):
After becoming the super user, I can install the above Nix expression providing the /usr environment profile and configuration scripts, by running:
$ su
# nix-env -f '<nixpkgs>' -iA steamChrootEnv
Now the chroot environment configuration scripts have been added to the system-wide Nix profile. I can initialize the chroot environment folder by running:
# init-steam-chrootenv
I can do the bind mounts by running:
# mount-steam-chrootenv
And then enter the chroot environment by running:
# load-steam-chrootenv
Now I have entered the chroot environment in which I can run the unpatched Steam executable. However, I should not run Steam as a root user. Therefore, I have to become "myself" again:
# su -s /bin/bash sander
And now I can finally run Steam, install my favourite games and play them:
$ steam
As you may see in the screenshot above, it works quite well on my machine. Moreover, to give you some proof that I'm not lying, I have also opened a terminal in the background showing you NixOS' root folder that does not contain most of the standard FHS directories.
So far, I have tried Half Life (and extensions), Half Life 2 (and extensions), Portal and Counter Strike and they all seem to work without any trouble. Isn't that AWESOME??? :P
Finally, if you need to discard the chroot environment (which should not be necessary IMHO :P), you could run:
In this blog post, I have described the buildFHSChrootEnv {} Nix function that composes FHS-compatible chroot environments in NixOS, allowing me to run Steam and other third party binary packages without patching them.
Moreover, we still have control over the dependencies in the chroot environment, because apart from a number of common base packages, these chroot environments are composed from Nix expressions that provide exactly the dependencies that are specified. Undeclared dependencies cannot influence a binary package running in the chroot environment.
However, this approach also has a big drawback. In order to chroot, we need super-user privileges making it hard to properly support unprivileged user installations of such packages. Most "ordinary" Nix packages can be deployed by unprivileged users without any trouble. Therefore, I would not recommend this approach over patching executables with PatchELF, unless there is no other way, such as in Steam's case.
Acknowledgements
This is not the first attempt to get Steam packaged in Nixpkgs. In my attempt, I have tried to use the chroot approach which turns out to work. I'd like to thank Evgeny Egorochkin, Aszlig Neusepoff and Carles Pagès for their earlier efforts.
Availability
The buildFHSChrootEnv {} function as well as the Steam components are part of Nixpkgs.
For some reason, I'm having quite some discussions about Nix's user environment mechanism lately. For example, in my last two talks I have spent a considerable amount of time answering questions that were related to it.
The user environment mechanism is an important key feature of Nix, which primary purpose is to allow someone to conveniently launch programs installed by the Nix package manager, but it has a few additional use cases as well, such as determining which packages are in use (and which are not) in order to safely garbage collect obsolete packages.
I have only briefly described them in some of my blog posts, but I have never covered in detail how they work on my blog. Because of the amount of questions I have received lately, I think it would be a good idea to write about it.
The first part of the basename (spy13ngvs1fyj82jw2w3nwczmdgcp3ck) is a cryptographic hash derived from all build inputs used to build Firefox. For example, if we would build Firefox with a different compiler or for a different architecture, the generated hash code will be different, which makes it possible to safely store multiple variants of packages next to each other, because none of them will have the same name.
Although this storing convention looks a bit odd, it provides a number of interesting benefits. Since Nix executes builds inside environments in which as many side effects are removed, we can use the hash codes to uniquely identify package variants regardless on what machine they have been built.
They also provide better reproducibility, since these paths to Nix packages including their hash codes must always be explicitly specified, in order to allow a package build to find its dependencies. The advantage of this is that unspecified dependencies cannot accidentally allow a build to succeed, limiting reproducibility guarantees.
More details on the benefits can be found in my two explanation recipes. However, storing packages this way also causes a major inconvenience for end-users. To launch a particular program, the user must always provide the full path to the executable that resides inside the Nix store, which is quite impractical due to the fact that every Nix package has a hash code in it.
Nix profiles
As a means for users to conveniently launch software, the Nix package manager provides the concept of Nix profiles, which are user environments exposing a generation of installed packages to a user. Software packages can be automatically installed in Nix profiles by using the nix-env command-line utility.
To install software, such as Info ZIP's zip and unzip utilities, we can run:
$ nix-env -i zip unzip
installing `zip-3.0'
installing `unzip-6.0'
building path(s) `/nix/store/1zhgva9cmfz0wam9m7ibgaih2m3cic6l-user-environment'
created 33 symlinks in user environment
Allowing us to run any of the installed tools without specifying any paths with hash codes, e.g.:
$ zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon. Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.
One of the things the above nix-env command-line invocation does is automatically building the requested packages (or downloading its substitutes) and composing a user environment, which is a symlink tree that synthesizes the contents of the currently installed packages (zip and unzip) into one single component. A user environment is also a package residing in the Nix store, as shown in the following figure:
Besides creating a user environment that exposes all installed packages from a single location, a few additional symlink indirections are used. Each time nix-env performs a deployment activity, a generation symlink is generated (called profile-1 in the figure above). There is also a default symlink (profile) pointing to the generation symlink that is currently active. That symlink is also referenced from the user's home directory through ~/.nix-profile. By adding the latter symlink to the user's PATH, e.g.:
export PATH=~/.nix-profile/bin:$PATH
Packages installed through nix-env can be found automatically through a few layers of symlink indirections.
When we perform another deployment activity with nix-env, such as removing zip:
$ nix-env -e zip
uninstalling `zip-3.0'
Then a new symlink tree is composed containing the then currently installed packages (in this case only unzip), a new generation symlink is generated (profile-2), and the default symlink is updated to point to the new generation. Flipping symlinks is actually an atomic operation on UNIX. As a consequence, we can perform upgrades atomically, because there is never a situation in which we refer to packages belonging to the old and new user environment at the same time.
As may be observed in the picture above, the old generation user environment is still present allowing us to do rollbacks as well. For example, if we regret uninstalling zip, we can switch back to the previous generation by running:
$ nix-env --rollback
Then the default symlink's reference is updated to point to the previous generation symlink that provides the zip utility. Again, this is an atomic operation.
Apart from exposing packages installed by users, the directory containing profile generations also serves a second goal; it is used to determine which packages are in use and which are not.
The generations of the Nix profiles of all users of the system are treated as so called garbage collector roots -- any package that is installed in a profile generation and any of its dependencies are considered packages that are live (a.k.a. in use) and should not be garbage collected. Furthermore, Nix also treats open files and running processes as garbage collector roots. Therefore, it is safe to run the Nix garbage collector at any time.
We can also remove older Nix profile generations if we don't need them anymore:
$ nix-env --delete-generations old
The above command only removes the older generation symlinks, but not the packages that have become obsolete. To actually remove packages from the system that are no longer in use, we must explicitly run the garbage collector:
$ nix-collect-garbage
So far, I have shown examples that work on a per-user profile. In multi-user Nix installations, such as in NixOS, every user has a personal Nix profile (residing in /nix/var/nix/profiles/per-user) and there is also a system-wide default Nix profile (residing in /nix/var/nix/profiles/default) containing packages that are exposed to all users of a system.
As described in my earlier blog post about NixOS, a complete system configuration is deployed from a single declarative specification, including all system services and their configurations, the desktop, and user packages. Each time we deploy a system configuration, e.g. by running:
$ nixos-rebuild switch
a Nix profile is built which consists of (mostly) a symlink tree containing all packages and configuration files that constitutes all static parts of a system, including the Linux kernel and the kernel modules. This profile is called a system profile and offers the same benefits as ordinary package profiles, such as the fact that we can store multiple generations next to each other and perform atomic upgrades and rollbacks.
There is a subtle difference with NixOS deployment compared to ordinary package deployment. Merely flipping symlinks is not enough to get a new NixOS configuration deployed. We also have to perform a collection of imperative deployment steps, such as setting up dynamic directories, such as /var, starting and stopping system services, and updating the GRUB bootloader configuration. This is done by an activation script (/run/current-system/bin/switch-to-configuration) part of the NixOS system profile, that is executed each time a new version of a NixOS configuration is deployed.
As can be seen in the screenshot above, once a new NixOS system configuration has been deployed, the GRUB bootloader can be used to boot into any previous NixOS configuration that has not been garbage collected yet.
Generating user environments from Nix expressions
In addition to using nix-env, user environments can also be generated from Nix expression by using the buildEnv {} function. The following Nix expression produces the same user environment (shown earlier) containing zip and unzip:
with import <nixpkgs> {};
buildEnv {
name = "zipprofile";
paths = [ zip unzip ];
}
This expression can be built by running:
$ nix-build zipprofile.nix
Conclusion
In this blog post, I have explained how the Nix user environment mechanism works. Essentially user environments are symlink trees that blend all currently installed packages into a single component so that they can be referenced from a single location. Another symlink layer manages generations of user environments. We can atomically switch between these generations by flipping symlinks. By adding the latter symlink to a user's PATH, packages can be automatically found without providing any absolute paths with hash codes in them that are difficult to remember.
Besides exposing packages in a user's PATH, Nix profiles are also used in NixOS to automatically find other resources, such as manual pages (through MANPATH) and KDE desktop menu items (through XDG_DATA_DIRS).
More information on Nix profiles can be found in Eelco Dolstra's PhD thesis which can be obtained from his publications page. Additionally, my PhD thesis also covers Nix profiles in its background chapter. Furthermore, there is a chapter that extends their concepts to distributed systems. This chapter is loosely based on our HotSWUp 2008 paper titled: "Atomic Upgrading of Distributed Systems" which can be obtained from my publications page.
In the previous blog post about the Nix iOS build function, I have described how to install various iPhone simulator SDK versions by picking the preferences option from the 'Xcode' menu bar and selecting the 'Downloads' tab in the window, as shown in the screenshot below:
However, I did not implement any facilities in the simulator function that take these SDK versions into account yet.
Picking a specific SDK version can be done quite easily, by providing the -currentSDKRoot parameter to the iPhone simulator, such as:
In the above case, we have configured the simulator to use the iPhone simulator 6.1 SDK. By changing this parameter, we can use a different version, such as version 5.1.
I have encapsulated the above command-line invocation in the xccodeenv.simulateApp {} Nix function and the paths to the iPhone simulator SDKs in the xcodewrapper. The SDK version can be configured by providing the sdkVersion parameter (which defaults to 6.1):
This parameter makes it possible to spawn a simulator instance running a specific iOS version.
The described facilities earlier are only for simulating apps. To build an app for a specific iOS revision, the iOS target version must be changed inside the Xcode project settings.
Building signed Apps for release
Apart from the simulator, we may also want to deploy apps to real devices, such as an iPhone, iPad or iPod, either directly or through the App store. In order to do that, we require permission from Apple by signing the apps that we have just built.
As described in the previous blog post, we require a certificate that contains a developer's or company's identity and a mobile provisioning file describing which (groups of) apps of a certain company/developer can be run of which (groups of) devices. These files must be obtained through Apple's Dev Center.
Because of these restrictions, it's hard to test the trivial "Hello World" example case I have developed in the previous blog post on a real device, since it contains a dummy app and company name.
To alleviate these problems, I have created a script that "renames" the example app into a different app, so that an existing developer or company certificate and a mobile provisioning profile of a different app can be used.
By setting the rename parameter to true when calling the composition expression of the example case, we can automatically generate jobs building IPAs and xcarchives for the "renamed" app:
In the above expression, we changed the name of the app into: "MyVeryCoolApp" and the company name into: "My Cool Company". Moreover, we provide a certificate and mobile provisioning file for the IPA job (which can be deployed to a device directly) and the xcarchive job (which can be used to deploy to the App store).
By running the following command-line instruction:
$ nix-build -A renamedPkgs.testapp_ipa
We can build an IPA allowing us to deploy the test app on a real device. Of course, to do this yourself, the company name and app names should correspond to the ones defined in your certificate and mobile provisioning profile.
Moreover, as with the Android environment in the previous blog post, we can also use Hydra to provide parameters to the composition expression and to retrieve all the resulting artifacts:
Conclusion
In this blog post, I have described some improvements to the Nix iOS facilities. We are now also capable of configuring the SDK versions for the iPhone simulator, and we can build the trivial testcase to run on a real device in addition to the simulator. This is useful for testing the Nix iOS build function.
Some time ago, I have packaged the Android SDK and some of its plugins in Nix, which was quite a challenge. Moreover, I have developed two Nix functions that can be used to automatically build Android apps and spawn emulator instances.
Meanwhile, I have implemented some major updates since the last blog post. Besides upgrading to a newer Android SDK version, we now also support a bunch of new interesting features.
Android SDK packaging updates
Currently, I have packaged the 22.05 version of the Android SDK (which is the latest version at the time writing this blog post). To package it, I have followed the same recipe as described in the previous blog post, with some additions:
Newer versions of the Android SDK include x86-64 versions of the emulator executables. I had to patch/wrap these executables to allow them to find the x86-64 libraries they depend on.
The newer Android SDK has a new sub package called build-tools. Apparently, this sub package contains some tools that were formerly part of platform-tools, such as debugging libraries.
The GUI front-end of the android utility did not work in the last blog post, which I have fixed by wrapping it, so that it can find its dependencies, such as GTK+.
There are also x86 and MIPS application binary interface (ABI) system images, but they reside in external package repositories. I have adapted the Nix expression generator to include these as well.
I have adapted the build and emulation Nix functions to take both API-level and ABI versions into account
I have implemented some facilities to improve the performance of the emulator instances
Packaging build-tools
The build-tools package contains various tools and libraries that used to belong to platform-tools. It turns out that some of these files, such as the debugging libraries, are API-level specific. The purpose of the build-tools package is to provide API-level specific build tools. Currently, the Android SDK supports two versions: 17 and 18.01 that can be installed next to each other.
I have packaged the 18.01 version in a straight forward manner. The only thing I needed to do is inspecting the dependencies of the executables to the paths to these libraries to the executables' RPATH.
Every version of the build-tools package is symlinked into the Android SDK basedir as follows: build-tools/android-<version>
Fixing the Android GUI front-end
If the android tool is called without any parameters, it should show a GUI that allows somebody to download plugins or to configure AVDs. However, the GUI did not work so far. Invoking 'android' gave me the following error message:
$ android
Exception in thread "main" java.lang.UnsatisfiedLinkError:
no swt-pi-gtk-3550 or swt-pi-gtk in swt.library.path,
java.library.path or the jar file
at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
at org.eclipse.swt.internal.gtk.OS.(Unknown Source)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
at org.eclipse.swt.widgets.Display.(Unknown Source)
at com.android.sdkmanager.Main.showSdkManagerWindow(Main.java:346)
at com.android.sdkmanager.Main.doAction(Main.java:334)
at com.android.sdkmanager.Main.run(Main.java:120)
at com.android.sdkmanager.Main.main(Main.java:103)
First, I thought that some file belonging to SWT was missing, but after manually digging through all Android files I couldn't find it and I gave up. However, after using strace, I discovered that it's actually trying to load the GTK+ library and some other libraries:
Besides the libraries, I also had to wrap android to use the right JDK version. Apparently, loading these libraries only works with OpenJDK in Nix.
After wrapping the android utility and running it without parameters, it shows me the following:
As you may see in the above image, the GUI works. It also shows us all the plugins that are installed through Nix. Although the GUI seems to work fine, we have to keep in mind that we cannot use it to install additional Android SDK plugins. All plugins must be packaged and installed with Nix instead.
Supporting external system images
The Android distribution provides a collection of system images that can be used with an emulator to test an app during development. However, the default repository only contains ARM-based system images, which are quite slow to emulate.
There also seem to be x86-based system images available (provided by Intel) and MIPS-based images (provided by MIPS Technologies). These system images reside in external repositories. I was able to discover the locations of these repositories, by running the following command:
The latter two URLs referring to a file called sys-img.xml provide x86 and MIPS images. The structure of these XML files are similar to the main repository. Both files contain XML elements that look like this:
The generated expression shown above is nearly identical to the one shown in the previous blog post. The only difference is that we also use the ABI identifier in the attribute names. For example: sysimg_x86_16 is used to refer to a x86-based system image for Android API-level 16. Likewise, we can refer to the ARM-based variant with: sysimg_armeabi-v7a_16.
Apart from giving each attribute a proper name, the resulting system image package must be symlinked into the Android SDK basedir taking the ABI identifier into account. We symlink every architecture-dependent system image package into: system-images/android-<api-level>/<abi>.
I have adapted the androidenv.emulateApp {} Nix function to also take ABI versions into account. By default, the emulator function uses an ARM-based system image (armeabi-v7a), since this is the most common hardware architecture used by Android OSes. By setting the abiVersion parameter to x86, we can use an x86-based Android system-image:
An interesting benefit of using an x86 system image with the emulator (which is based on QEMU) is that KVM can be used, allowing programs inside the emulator to run at nearly native speed with only little emulation.
And of course, setting the abiVersion parameter to mips makes it possible to use MIPS-based system images.
Enabling GPU acceleration
The x86 system images in the emulator may give testing a significant boost. Emulation performance can even be improved a bit further by enabling GPU acceleration.
To make GPU acceleration possible, I had to adapt the emulator's wrapper script to include Mesa in the linker path. Moreover, in the AVD configuration I have to add the following boolean value to $ANDROID_SDK_HOME/.android/avd/device.avd/config.ini:
hw.gpu.enabled = yes
The androidenv.emulateApp {} automatically enables GPU acceleration from API-levels 15 and onwards. It can be disabled by setting the enableGPU parameter to false.
Supporting Google APIs in non-ARM-based system images
Although supporting x86 system images have various benefits, they also have a major drawback -- the Google APIs are not supported preventing some applications from working. One of the solutions I have seen (described in this Stack Overflow post) is to create a custom system image by starting an ARM-based emulator instance with Google APIs, fetch the libraries, start an x86-based emulator instance without Google APIs and manually install the fetched libraries.
I was able to automate the steps in the Stackflow article in a Nix expression:
The above Nix function creates an ARM emulator instance with Google APIs and an x86 emulator instance without Google APIs. Then it starts the ARM emulator instance, fetches the Google API files from it and then stops it. Then the x86 emulator instance is started and the Google APIs are pushed to it. Finally we generate a system image from the system folder that is stored in the Nix store.
By using the Nix function shown above and setting the extraAVDFiles parameter of the emulateApp {} function, we can use our custom system image in a script that automatically spawns the emulator instance:
The above expression invokes the function shown previously and starts an emulator instance with the system image containing Google APIs. Of course, we can also use the same trick for MIPS-based system images, which also don't include Google APIs.
Building Android apps for arbitrary API-level revisions
I have also made a small change to the androidenv.buildApp {} Nix function to take API-levels into account. By default, it will use the version that is inside the Ant configuration. However, by setting the target property through the command-line, we can override this:
$ ant -Dtarget=android-17
The above command-line instruction ensures that we build the app against the API-level 17 platform. I have added an antFlags parameter to the Nix function to make it provide arbitrary flags to Ant:
In the previous blog post, I have implemented a testcase based on the Android introduction tutorial (MyFirstApp) that I have used throughout this article as well. To test all these new features, I have extended its composition expression to take three parameters: buildPlatformVersions (defaults to 16), emulatePlatformVersions (defaults to 16), and abiVersions (defaults to armeabi-v7a). The expression automatically generates all possible combinations with the values inside these three lists:
For example, to build a debug version of the test app APK for API-level 16, we can run:
$ nix-build -A myfirstapp_debug.build_16
We can also build a release version of the same APK that is signed with a key:
$ nix-build -A myfirstapp_release.build_16
Furthermore, we can automatically generate a script starting an emulator instance running the app. The following instruction builds an App for Android API-level 16, generates a script launching an emulator with a Android API-level 16 system-image that uses the armeabi-v7a ABI:
$ nix-build -A emulate_myfirstapp_debug.build_16.emulate_16.armeabi-v7a
$ ./result/run-test-emulator
To support more API-levels and ABIs, the parameters of the composition function must be altered. For example, by running the following command-line instruction:
The cartesian product of build and emulator instances is created taking the three dimensions into account. This allows us to (for example) build the App against the API-level 16 platform API, emulate on an API-level 17 system image using the x86 ABI, which emulates much more efficiently on x86 machines, because hardware visualization features (such as KVM) can be used.:
Apart from specifying these dimensions through the command line, the composition expression can also be used together with Hydra, the Nix-based continuous integration server, allowing it to pass these parameters through its web interface. Hydra will build all the combinations generated by the composition expression automatically:
As can be observed from the above screenshot, quite a few jobs are being generated.
Conclusion
In this blog post, I have described some new changes and features that I have implemented in the Nix Android SDK package since the last blog post, such as the fact that x86 and MIPS system-images can be used. All these changes are part of the Nixpkgs project.
Besides the issues previously described, there is even more pain/frustration to cope with, which I'd like to discuss in this blog post. The main motivation to write this down is because I'm a control freak that wants to know what happens, it took me some effort discovering these details, and I have found myself repeating the same stuff to others.
Executing tasks concurrently
In addition to organizing data in a program, the ability to execute tasks concurrently is also important for many types of applications, such as:
Applications with graphical user interfaces (GUIs). For example, if a user clicks on a button executing a task, the GUI must remain responsive while the task is being processed.
Server applications. A server application has to serve multiple connected clients simultaneously, while each task remains responsive. For example, if a task is being processed of some client, it should not be necessary for a new client to wait until the requests of the others are done.
Applications communicating with hardware peripherals. Many peripherals, such as (hard/DVD/Blu-Ray) disk drives or printers are significantly slower than the CPU of the computer. For example, it's quite inconvenient that execution of a program temporarily blocks the entire system when a file is being read from the hard drive. Instead, it's (obviously) better to keep the programming running (and respond on user events) while a file is being read, and to notify the user when the task is done.
To allow tasks to be executed concurrently, many operating systems/execution environments have facilities, such as interrupts, processes and/or threads allowing a developer to do that in a (slightly) convenient way. The execution environment ensures that these processes or threads are executed simultaneously (well not really, in fact it divides CPU execution time over processes and threads, but in order to keep this blog post short, I won't go into detail on that).
Executing tasks concurrently in JavaScript
Although most operating systems, execution environments, and programming languages have facilities for threads and/or processes, JavaScript -- a programming language originally used inside a web browser with increasing popularity outside the browser -- does not have such facilities. In JavaScript, there is only one single execution thread.
Consider the following example HTML page:
<!DOCTYPE html>
<html>
<head><title>Stretching button</title></head>
<body>
<h1>Stretching button!</h1>
<button id="button" onclick="stretchButton();">Stretch</button>
<script type="text/javascript">
enabled = false;
function updateButtonSizes() {
var grow = true;
var width = 100;
var height = 20;
var button = document.getElementById('button');
while(enabled) {
/* Update the button sizes */
button.style.width = width + "px";
button.style.height = height + "px";
/* Adjust the button sizes */
if(grow) { width++; height++; }
else { width--; height--; }
/* If the limits are reached, stretch in
* the opposite direction
*/
if(width >= 150) grow = false; else if(width <= 100) grow = true;
}
}
function stretchButton() {
if(enabled) {
enabled = false;
} else {
enabled = true;
updateButtonSizes();
}
}
</script>
</body>
</html>
The purpose of the HTML code shown above is to display a button that animates by increasing and decreasing its width and height once the user has clicked on it. The animation keeps repeating itself and stops if the user clicks on the same button again.
By just looking at the the code, it may give you the impression that it should work fine. However, once we have clicked on the button, the browser blocks completely and prevents us clicking on the button again. Furthermore, because we cannot click on the button again, we are also incapable of stopping the animation sequence. As a consequence, we have run into an infinite loop.
The reason that we run into trouble, is because if JavaScript code is being executed -- that runs in a single execution thread -- the browser cannot do anything else, such as responding to user events or updating the screen. After a minute or so, the browser will notice that some script is blocking it and asks you if it should terminate the script for you.
So you may probably wonder how to deal with this blocking issue? To allow the browser to respond to user events and update the screen, we must stop the execution of the program (by reaching the end of its control flow). Then at a later stage, we must resume execution by generating an event. Besides generating events from page elements, such as buttons as can be seen in the code fragment above, we can also generate timeout events:
In the snippet above, the setTimeout() function invocation tells the browser to execute the function displaying "Hello World" after 1000 milliseconds. Like events generated by page elements, timed events can also only be processed if nothing is being executed at that moment. So the program must reach the end of its execution flow first before the timeout event can be processed.
An adapted version of the earlier example with the button that correctly handles events and screen updates, may look as follows:
<!DOCTYPE html>
<html>
<head>
<title>Stretching button</title>
</head>
<body>
<h1>Stretching button!</h1>
<button id="button" onclick="stretchButton();">Stretch</button>
<script type="text/javascript">
enabled = false;
button = document.getElementById('button');
function updateButtonSizes() {
/* Update the button sizes */
button.style.width = width + "px";
button.style.height = height + "px";
/* Adjust the button sizes */
if(grow) { width++; height++; }
else { width--; height--; }
/* If the limits are reached then stretch in
* the opposite direction
*/
if(width >= 150) grow = false; else if(width <= 100) grow = true;
/* Keep repeating this until the user disables it */
if(enabled) setTimeout(updateButtonSizes, 0);
}
function stretchButton() {
grow = true;
width = 100;
height = 20;
if(enabled) {
enabled = false;
} else {
enabled = true;
updateButtonSizes();
}
}
</script>
</body>
</html>
The above example code is quite similar to the previous one, with the following differences:
The updateButtonSizes() function does not contain a while loop. Instead, it implements a single iteration step and calls setTimeout() to generate a timeout event that calls itself.
After setting the timeout event, the program reaches the end of the execution flow.
The browser updates the screen, handles user input and resumes execution by processing the timeout event that invokes updateButtonSizes() again.
I also made several variables global so that their values are remembered after each execution of updateButtonSizes()
To prove that the above example works, I have included an embedded version in this blog post:
Stretching button!
Cooperative multi-tasking
In short, to execute tasks in a browser while keeping the user interface responsive, we must do the following:
Generate timeout events when it's desired to give other tasks the opportunity to do something.
Reach the end of the control flow so that events can be processed by the execution environment. The generated timeout event calling a function ensures that execution is resumed at a later stage.
This approach looks very similar to cooperative multi-tasking using by classic Mac OS 9 and earlier and Windows operating systems prior to 3.x.
The fact that a function needs to be called every time you want to allow the browser/runtime environment to process events, reminds of the way I was developing programs many years ago on the AmigaOS. I still remember quite vividly that when using AMOS BASIC, if I wanted Workbench applications running in the background to do something while my program was running, I had to add the 'Multi Wait' instruction to the main loop of the program:
While busy
' Perform some tasks
Multi Wait
Wend
By using the 'Multi Wait' instruction, the operating system gets notified, interrupts the program's execution and the scheduler decides which other process is allowed to use the CPU. Then at some point, the OS scheduler decides that my program is allowed to continue and the program's execution continues as if nothing happended.
However, compared the JavaScript approach, the AmigaOS approach is a bit more powerful since it also takes care of interrupting and resuming a program at the right location within the right context.
Cooperative multitasking works as long as tasks properly cooperate, putting great responsibility by developers of a program. It's not uncommon however, that people make mistakes and the system as a whole blocks forever.
Asynchronous programming
Apart from the browser, many other environments embedding a JavaScript runtime require developers to use cooperative multitasking techniques.
Node.js, a server-side software system designed for writing scalable Internet applications, uses almost the same approach as the browser just described earlier. One minor difference is that instead of using setTimeout(), a more efficient alternative is used:
The process.nextTick() function executes the given callback function once the end of the execution flow of the program has been reached. Because it does not use the timer, it's more efficient.
To save users the burden of calling process.nextTick() and ending the execution flow each time something may block the system, most library functions in Node.js are asynchronous by default (with some having a synchronous alternative). Asynchronous functions have the following properties:
They return (almost) immediately.
A callback function (provided as function parameter) gets invoked when the task is done, which can be used to retrieve its result or status.
They automatically generate tick events and reach the end of their control flow to prevent the system being blocked.
For example, to read a file and store its contents in a string without blocking the system, one could do:
The above example opens /etc/hosts asynchronously. The fs.readFile() function returns immediately, then the string "Hello" is printed and the program reaches the end of its control flow. Once the file has been read, the callback function is invoked displaying either an error message (if the file cannot be opened) or the contents of the file.
Although asynchronous functions prevent the system being blocked and allow multitasking, one of its big drawbacks is that the callback mechanism makes it much more difficult to structure a program properly. For example, if I want to create a folder called "out", then a subfolder called "test" in the previous folder, inside the "test" subfolder a file called "hello.txt" containing "Hello world!", and finally verify whether the written text file contains "Hello world!", we may end up writing:
var fs = require('fs');
var path = require('path');
fs.mkdir("out", 0755, function(err) {
if(err) throw err;
fs.mkdir(path.join("out, "test"), 0755, function(err) {
if (err) throw err;
var filename = path.join("out", "test", "hello.txt");
fs.writeFile(filename, "Hello world!", function(err) {
if(err) throw err;
fs.readFile(filename, function(err, data) {
if(err) throw err;
if(data == "Hello world!")
process.stderr.write("File is correct!\n");
else
process.stderr.write("File is incorrect!\n");
});
});
});
});
For each step in the above sequential program structure, we end up one indentation level deeper. For the above example it's probably not so bad, but what if we have to execute 10 asynchronous functions sequentially?
Moreover, it is also non-trivial to modify the above piece of code. What, for example, if we want to write another text file, before the function that creates the test folder? It requires us to weave another function callback in the middle of the program and to refactor the entire indentation structure.
Fortunately, there are also solutions to cope with that. For example, the async library contains a collection of functions supporting control flow patterns and functions supporting data collections that can be used to deal with complexities of asynchronous functions.
The waterfall pattern, for example, can be used to execute a collection of asynchronous function sequentially and stops if any of the callbacks returns an error. This can be used to flatten the structure of our previous code example and also makes it better maintainable:
Besides the waterfall pattern, the async library supports many other control flow and collection patterns, such as executing functions in parallel or asynchronous loops over items in an array.
Mutable state
There is another important aspect that we have to keep in mind while executing stuff concurrently. Shared mutable state may influence the result of an execution of a task. For example, in our last example using async.waterfall(), I have declared the filename variable as a global variable, because I (naively) thought that it was a good way to share its value among the callback functions.
However, since it's possible to run multiple tasks concurrently, another task may change this global variable to something else. As a consequence, when we have reached the callback using it, it may has been changed to a different value and thus our expected result may differ. Moreover, it's also extremely hard to debug these kind of problems.
To protect ourselves from these kind of problems, it's better to scope a variable, which can be done with the var keyword. However, although most programming languages, such as Java (which influenced JavaScript) use block-level scoping, in JavaScript variables are function-level scoped.
For example, the following Java example shows the effect of block-level scoping:
public class Test {
public static void main(String[] args) {
int i = 1;
int j = 0;
if(j == 0) {
int i = 2;
}
return i; // 1;
}
}
In the example above we have two i variables. The first one is declared inside the main() body, the second inside the if-statement block. Because the variables are scoped within the nearest block boundary, the value of i inside the if-block is discarded after it terminates.
However, a direct port of the above code to JavaScript yields a different result, because i is scoped to the nearest function boundary:
function test() {
var i = 1;
var j = 0;
if(j == 0) {
var i = 2;
}
return i; // 2;
}
So in our JavaScript code fragment, the second var i; declaration inside the if-block is also bound to the scope of the test() function, therefore yielding a different result. Fortunately, block level scoping can be easily simulated by declaring a function without parameters and calling it without parameters (function() { var x; ... } ();):
function test() {
var i = 1;
var j = 0;
if(j == 0) {
function() {
var i = 2; /* i is scoped within the surrounding function */
}();
}
return i; // 1;
}
Similarly this trick also allows us to scope the filename variable shown in our earlier example with async.waterfall():
function() {
var filename; /* filename is scoped within the surrounding function */
async.waterfall(...);
}();
In the above example, the filename variable is not global anymore. The callback functions inside async.waterfall() will not refer to the global variable named filename any longer, but to the scoped variable inside the function block. As a result, other tasks running concurrently adapting global variables cannot influence this task any longer.
Conclusion
In this blog post, I have described some of the pains I have experienced with multitasking inside a JavaScript environment. From this, I have learned the following lessons:
If a script is being executed, then nothing else can be done, such as screen updates, processing user events, handling client connections etc.
To allow multitasking (and prevent the system being blocked entirely) we have to cooperate by generating timeout or tick events and ending the control flow of the program.
Asynchronous functions use callbacks to retrieve their results and/or statuses making it extremely hard to structure your program properly. In other words, it easily becomes a mess. To cope with this, you need to think about patterns to combine them. Libraries, such as async, may help you with this.
Shared state is evil, error prone and makes a program hard/impossible to debug. Therefore, only use global variables that are immutable or refer to singleton instances (such as CommonJS modules). Otherwise, scope them.
JavaScript does not support block level scoping, but it can be simulated by encapsulating a block inside a function that gets called without parameters. Fortunately, the ECMAScript 6 standard defines the let keyword supporting block-level scoping natively, but it's not supported in most implementations, such as in Node.js.
Although I'm a little happy about the fact that I know how to handle these problems, I'm very sad at the same time because of the following reasons:
JavaScript was originally advertised as "a lightweight and simpler variant of Java". The fact that every JavaScript developer must cope with all these concurrency issues, which Java mostly solves with a threading API, definitely makes JavaScript NOT simpler than Java in this respect IMHO.
Cooperative multitasking is a technique that was very common/useful in the early 1980s (and probably invented years before it), but has been replaced by preemptive multitasking in modern operating systems. One of the reasons to take responsibility away from the processes (besides the fact that it makes the lives of programmers easier) is because they cannot be trusted. However, we still seem to "trust" JavaScript code embedded in malicious web pages.
We're using a programming language for all kinds of purposes for which it has not been designed. Moreover, we require solutions for these issues that are not part of the language, API or runtime, such as (ab)using the timer and relying on external libraries implementing patterns to combine asynchronous functions.
Sometimes these new use cases of JavaScript are called innovation, but the fact that we use techniques that are over 30 years old (and intentionally not using better modern alternatives), does not really look like innovation to me IMHO.
So are solutions using JavaScript that bad? I see that there is practical use and value in it (e.g. the V8 runtime compiles to efficient native code and there is a big eco-system of JavaScript libraries), but dealing with concurrency like this is definitely something I consider a big drawback and a huge step backwards.
I have written quite some Nix-related articles on this blog. Nix is typically advertised as the core component of the NixOS Linux distribution. However, it can also be used separately on conventional Linux distributions and other UNIX-like systems, such as FreeBSD and Mac OS X.
Using Nix on conventional systems makes it possible to use the interesting features of Nix and its applications, such as Hydra and Disnix, while still being able to use your favorite distribution.
Single-user Nix installations
I have noticed that on non-NixOS systems, the Nix package manager is often installed for only one single user, as performing single-user installations is relatively simple. For example, in my blog post describing how to build apps for iOS with Nix, I perform a Nix installation that can only be used by my personal user account.
For most of the simple use cases, single user installations are sufficient. However, they have a number of issues besides the fact that only one user of a machine (in addition to the super-user) is able to use it:
Although Nix creates build environments that remove several important side-effects, e.g. by clearing environment variables and storing all packages in isolation in the Nix store, builds can still refer to executables and other files in global directories by having hardcoded references, such as /usr/bin or /var, which may influence the result of the build.
On NixOS these references are typically not an issue since these directories do not exist, but on conventional distributions these do exist, which may cause (purity) problems.
Many packages have hard-coded references to the default Bourne-compatible shell in: /bin/sh. Some of these packages assume that this shell is bash and use bash-specific features.
However, some Linux distributions, such as Debian and Ubuntu, use dash as default /bin/sh causing some builds to break. Moreover, BSDs such as FreeBSD also use a simpler Bourne shell implementation by default.
Some build processes may unknowingly try to download stuff from the Internet causing impurities.
Package builds have the same privileges as the calling user, allowing external processes run by the user to interfere with the build process. As a consequence, impurities may sneak in when executing multiple package builds in parallel.
Multi-user Nix installations
As a remedy for the issues just described, Nix is also capable of executing each build with separate user privileges in a nearly clean chroot environment in which we bind mount the Nix store. However, in order to use these features, a multi-user Nix installation is required, as these operations require super-user privileges. In NixOS, a multi-user Nix installation comes for free.
In multi-user Nix installations, builds are not executed directly by each individual user, since they cannot be trusted. Instead we run a server, called the nix-daemon that builds packages on behalf of a user. This daemon also takes care of running processes as a unique unprivileged user and setting up chroot environments.
Although the Nix manual provides some pointers to set up a multi-user installation, it turned out to be a bit trickier than I thought. Moreover, I have noticed that a few practical bits were missing in the manual and the Nix distribution.
In this blog post, I have investigated these issues and implemented a few improvements that provide a solution for these missing parts. I have performed these steps on a Ubuntu 12.04 LTS machine.
Installing Nix from source
As a first step, I installed Nix from source by running the following commands:
$ ./configure --prefix=/usr --sysconfdir=/etc
$ make
$ sudo make install
Adding the Nix build group and users
Since every concurrent build must be run as a separate user, we have to define a common group of which these users should be a member:
$ sudo groupadd -g 20000 nixbld
Then we need to add user accounts for each build that gets executed simultaneously:
$ for i in `seq 1 10`
do
sudo useradd -u `expr 20000 + $i` -g nixbld \
-c "Nix build user $i" -d /var/empty -s /noshell
done
In the code fragment above, we assume that 10 users are sufficient, but if you want/need to utilise more processors/cores this number needs to be raised.
Finally, we have to specify the build users group in the Nix configuration:
In ordinary installations the user is made owner of /nix/store. In multi-user installations, it must be owned by root and group owned by the nixbld user. The following shell commands grant the Nix store the right permissions:
Creating per-user profile and garbage collection root folders
In single user installations, we only have one system-wide default profile (/nix/var/nix/profiles/default) owned by the user. In multi-user installations, each user should be capable of creating their own profiles and garbage collector roots. The following shell commands ensure that it can be done:
We must also run the nix-daemon that executes builds on behalf of a user. To be able to start and stop it, I have created an init.d script for the Nix daemon. The interesting part of this script is the start operation:
DAEMON=/usr/bin/nix-daemon
NAME=nix-daemon
if test -f /etc/default/nix-daemon; then
. /etc/default/nix-daemon
fi
...
case "$1" in
start)
if test "$NIX_DISTRIBUTED_BUILDS" = "1"; then
NIX_BUILD_HOOK=$(dirname $DAEMON)/../libexec/nix/build-remote.pl
if test "$NIX_REMOTE_SYSTEMS" = "" ; then
NIX_REMOTE_SYSTEMS=/etc/nix/remote-systems.conf
fi
# Set the current load facilities
NIX_CURRENT_LOAD=/var/run/nix/current-load
if test ! -d $NIX_CURRENT_LOAD; then
mkdir -p $NIX_CURRENT_LOAD
fi
fi
start-stop-daemon -b --start --quiet \
--exec /usr/bin/env \
NIX_REMOTE_SYSTEMS=$NIX_REMOTE_SYSTEMS \
NIX_BUILD_HOOK=$NIX_BUILD_HOOK \
NIX_CURRENT_LOAD=$NIX_CURRENT_LOAD \
$DAEMON -- $DAEMON_OPTS
echo "$NAME."
;;
...
esac
For the start operation, we have to spawn the nix-daemon in background mode. Moreover, to allow Nix to perform distributed builds, we must set a number of environment variables that provide the locations of the build hook script, the configuration file containing the properties of the external machines and a directory containing files keeping track of the load of the machines. Moreover, some directories may also have to be created if they don't exist.
To ensure that it's automatically launched on startup, we must add the following symlinks for the relevant runlevels:
$ sudo -i
# cd /etc/rc2.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc3.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc4.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc5.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# exit
I think the above init.d script can be trivially ported to other distributions.
Setting up user profiles
To allow users to install software through Nix and allow them to refer to their installed programs from a simple command-line invocation, we need to add some stuff to the user's shell profile, such as setting the PATH environment variable pointing to certain Nix profiles.
However, the nix.shprofile.d script in the Nix distribution only performs the necessary steps for single user installations. For example, it only adds to system-wide Nix profile and assumes that the user has all the rights to configure a channel.
I have ported all the relevant features from NixOS to create /etc/profile.d/nix-multiuser.sh, supporting all required features to set up a shell profile for multi-user installations:
First, we have to set up a user's profile directory in the per-user profile directory, if it doesn't exist:
export NIX_USER_PROFILE_DIR=/nix/var/nix/profiles/per-user/$USER
mkdir -m 0755 -p $NIX_USER_PROFILE_DIR
if test "$(stat --printf '%u' $NIX_USER_PROFILE_DIR)" != "$(id -u)"; then
echo "WARNING: bad ownership on $NIX_USER_PROFILE_DIR" >&2
fi
In single user installations, a ~/.nix-profile symlink is created pointing to the system-wide default Nix profile. For multi-user installations, we must create a ~/.nix-profile symlink pointing to the per-user profile. For the root user, we can still use the system wide Nix profile providing software for all users of the system:
if ! test -L $HOME/.nix-profile; then
echo "creating $HOME/.nix-profile" >&2
if test "$USER" != root; then
ln -s $NIX_USER_PROFILE_DIR/profile $HOME/.nix-profile
else
# Root installs in the system-wide profile by default.
ln -s /nix/var/nix/profiles/default $HOME/.nix-profile
fi
fi
In single user installations, we add the bin directory of the system-wide Nix profile to PATH. In multi-user installations, we have to do this both for the system-wide and the user profile:
export NIX_PROFILES="/nix/var/nix/profiles/default $HOME/.nix-profile"
for i in $NIX_PROFILES; do
export PATH=$i/bin:$PATH
done
In single user installations, the user can subscribe itself to the Nixpkgs unstable channel providing pre-built substitutes for packages. In multi-user installations only the super-user can do this (as ordinary users cannot be trusted). Although root can only subscribe to a channel, ordinary users can still install from the subscribed channels:
if [ "$USER" = root -a ! -e $HOME/.nix-channels ]; then
echo "http://nixos.org/channels/nixpkgs-unstable nixpkgs" \
> $HOME/.nix-channels
fi
We have to create a garbage collector root folder for the user, if it does not exists:
NIX_USER_GCROOTS_DIR=/nix/var/nix/gcroots/per-user/$USER
mkdir -m 0755 -p $NIX_USER_GCROOTS_DIR
if test "$(stat --printf '%u' $NIX_USER_GCROOTS_DIR)" != "$(id -u)"; then
echo "WARNING: bad ownership on $NIX_USER_GCROOTS_DIR" >&2
fi
We must also set the default Nix expression, so that we can conveniently install packages from Nix channels:
if [ ! -e $HOME/.nix-defexpr -o -L $HOME/.nix-defexpr ]; then
echo "creating $HOME/.nix-defexpr" >&2
rm -f $HOME/.nix-defexpr
mkdir $HOME/.nix-defexpr
if [ "$USER" != root ]; then
ln -s /nix/var/nix/profiles/per-user/root/channels \
$HOME/.nix-defexpr/channels_root
fi
fi
Unprivileged users do not have the rights to build package directly, since they cannot be trusted. Instead the daemon must do that on behalf of the user. The following shell code fragment ensures that:
if test "$USER" != root; then
export NIX_REMOTE=daemon
else
export NIX_REMOTE=
fi
Using multi-user Nix
After having performed the previous steps, we can start the Nix daemon by running the init.d script as root user:
$ sudo /etc/init.d/nix-daemon start
Then if we login as root, we can update the Nix channels and install packages that are supposed to be available system-wide:
We should also be able to log in as an unprivileged user and capable of installing software:
$ nix-env -i wget
$ wget # Only available to the user that installed it
$ hello # Also works because it's in the system-wide profile
Enabling parallel builds
With a multi-user installation, we should also be able to safely run multiple builds concurrently. The following change can be made to allow 4 builds to be run in parallel:
The above allows us to delegate builds for Mac OS X to a Mac OS X machine.
Enabling chroot builds
On Linux, we can also enable chroot builds allowing us to remove many undesired side-effects that single-user Nix installations have. Chroot environments require some directories of the host system to be bind mounted, such as /dev, /dev/pts, /proc and /sys.
Moreover, we need a default Bourne shell in /bin/sh that must be bash, as other more primitive Bourne compatible shells may give us trouble. Unfortunately, we cannot bind mount the host's system /bin folder, as it's filled with all kinds of executables causing impurities. Moreover, these executables have requirements on shared libraries residing in /lib, which we do not want to expose in the chroot environment.
I know two ways to have bash as /bin/sh in our chroot environment:
We can install bash through Nix and expose that in the chroot environment:
The last line exposes /dev, /dev/pts and /nix-bin (mounted on /bin, containing only sh) of the host system to the chroot environment allowing us to build stuff purely. If we have installed bash through Nix, then we also have to add the Nix closure of bash, which we can obtain through the nix-store -qR command. For a static bash this invocation has to be omitted.
Conclusion
In this blog post, I have described everything I did to set up a multi-user Nix installation supporting distributed and chroot builds on a conventional Linux distribution (Ubuntu) which was a bit tricky.
I'm planning to push some of things I did upstream, so that others can benefit from it. This is a good thing, because I have the feeling that most non-NixOS Nix users will lose their interest if they have figure out the same stuff I just did.
