An evaluation and comparison of GoboLinux

In this blog post, I'm going to evaluate deployment properties of GoboLinux and compare it with NixOS. Both Linux distributions are unconventional because they deviate from the Filesystem Hierarchy Standard (FHS). Apart from this, GoboLinux shares the same idea that the filesystem can be used to organize the management of packages and other system components, instead of relying on databases.

The purpose of this blog post is not to argue which distribution is better, but to see how this distribution achieves certain deployment properties, what some differences are compared to NixOS and what we can learn from it.

Filesystem organization

Unlike NixOS, which only deviates from the FHS where necessary, GoboLinux has a filesystem tree which completely deviates from the FHS, as shown below:

GoboLinux stores static parts of programs in isolation in the /Programs folder. The /System folder is used to compose the system, to store system configuration settings and variable data. /Files is used for storing data not belonging to any program. /Mount is used for accessing external devices such as DVD-ROM drives. The /Users folder contains home directories. /Depot is a directory without a predefined structure, which can be organized by the user.

GoboLinux also provides compatibility with the FHS. For example, the /usr and /bin directories are actually also present, but not visible. These directories are in fact symlinks trees referring to files in the /System directory (as you may see in the picture above). They are made invisible to end-users by a special kernel module called GoboHide.

Package organization

Like NixOS, GoboLinux stores each package in a seperate directories, which do not change after they have been built. However, GoboLinux uses a different naming convention compared to NixOS. In GoboLinux every package uses the /Program/<Name>/<Version> naming convention, such as: /Programs/Bzip2/1.0.4 or /Programs/LibOGG/1.1.3. Furthermore, each program directory contains a Current symlink which points to the version of the component, that is actually in use. Some program directories also have a Settings/ directory containing system configuration files for the given package.

The use of isolated directories offers a number of benefits like NixOS:

• Because every version is stored in its own directory, we can safely store multiple versions next to each other, without having to worry that a file of another version gets overwritten.
• By using a symlink, pointing to the current version, we can atomically switch between versions by flipping the symlink (also used by Nix to switch Nix profiles), which ensures that there is no situation in which a package contains both files of the old version and new version.

In contrast to NixOS, GoboLinux uses a nominal naming convention and dependency specifications, which are based on the name of the package and version number. With a nominal naming convention, it cannot make a distinction between several variants of components, for example:

• It does not reflect which optional features are enabled in a package. For example, many libraries and programs have optional dependencies on other libraries. Some programs may require a particular variant of a library with a specific option enabled.
• It does not take the build-time dependencies, such as build tools or library versions into account, such as the version of GCC or glibc. Older versions may have ABI incompatibilities with newer versions.
  
• It does not take the architecture of the binaries into account. Using this naming scheme, it is harder to store 32-bit and 64-bit binaries safely next to each other.
  

In NixOS, however, every package name is an exact specification, because the component name contains an hash-code derived from all build-time dependencies to build the package, such as the compiler version, libraries and build scripts.

For example, in NixOS bzip2 may be stored under the following path: /nix/store/6iyhp953ay3c0f9mmvw2xwvrxzr0kap5-bzip2-1.0.5. If bzip2 is compiled with a different version of GCC or linked to a different version of glibc, it gets a different hash-code and thus another filename. Because no component shares the same name, it can be safely stored next to other variants.

Another advantage of the Nix naming convention is that unprivileged users can also build and install software without interfering with other users. If for example, a user injects a trojan horse in a particular package, the resulting component is stored in a different Nix store path and will not affect other users.

However, because of these exact dependency specifications, upgrades in NixOS may be more expensive. In order to link a particular program to a new version of a library, it must be rebuild, while in GoboLinux the library dependency can be replaced without rebuilding (although it may not guarantee that the upgraded version will work).

System composition

Of course, storing packages in separate directories does not immediately result in a working system. In GoboLinux, the system configuration and structure is composed in the /System directory.

The /System directory contains the following directories:

• Kernel/, contains everything related to the Linux kernel, such as Boot/ storing kernel images, Modules/ storing kernel modules and Objects/ storing device files.
  
• Links/, is a big symlink tree composing the contents of all packages that are currently in use. This symlink tree makes it possible to refer to executables in Executables/, libraries in Libraries/ and other files from a single location. The Environment/ directory is used to set essential environment variables for the installed programs.
• Settings/ contains configuration files, which in other distributions are commonly found in /etc. Almost all files are symlinks to the Settings/ folder included in each program directory.
• Variable/ contains all non-static (variable) data, such as cache and log files. In conventional distributions these files are stored in /var.

Like NixOS, GoboLinux uses symlink trees to compose a system and to make the contents of packages accessible to end users.

Recipes

Like NixOS, GoboLinux uses declarative specifications to build packages. GoboLinux calls these specifications recipes. Each recipe directory contains a file called Recipe which describes how to build a package from source code and a folder Resources/ defining various other package properties, such as a description and a specification of build-time and run-time dependencies.

compile_version=1.8.2
url="$httpSourceforge/lesstif/lesstif-0.95.2.tar.bz2"
file_size=2481073
file_md5=754187dbac09fcf5d18296437e72a32f
recipe_type=configure
make_variables=(
   "ACLOCALDIR=$target/Shared/aclocal"
   "aclocaldir=$target/Shared/aclocal"
)

An example of a recipe, describing how to build Lesstif, is shown above. Basically, for autotools based projects, you only need to specify the location where the source code tarball can be obtained, and some optional parameters. From this recipe, the tarball is download from the sourceforge web server and the standard autotools build procedure is performed (i.e. ./configure; make; make install) with the given parameters.

In the Resources/Dependencies run-time dependencies can be specified and in Resource/BuildDependencies build-time dependencies can be specified. The dependencies are specified by giving a name and an optional version number or version number range.

GoboLinux recipes and Nix expressions both offer abstractions to declaratively specify build actions. A big difference between those specifications, is the way dependencies are specified. In GoboLinux only nominal dependency specifications are used, which may not be complete enough, as explained earlier. In Nix expressions, you refer to build functions that build these dependencies from source-code and their build-time dependencies, which are stored in isolation in the Nix store using hash codes.

Furthermore, in Nix expressions run-time and build-time dependencies are not separately specified. In Nix, every run-time dependency is specified as build-time dependency. After a build has been performed, Nix conservatively scans for hash occurrences of build-time dependencies inside a realized component to identify them as run-time dependencies. Although this sounds risky, it works extremely well, because chances are very slim that an exact occurrence of a hash code represents something else.

Building packages

In GoboLinux, the Compile tool can be used to build a package from a recipe. Builds performed by this tool are not entirely pure, because it looks for dependencies in the /System/Links directory. Because this directory contains all the installed packages on the system, it may be possible that the build of a recipe may accidentally succeed when a dependency is not specified, because it can be implicitly found.

In order to make builds pure, GoboLinux provides the ChrootCompile extension, which performs builds in a chroot environment. ChrootCompile tool bind mounts the /Program directory in the chroot environment and creates a small /System only containing symlinks to the dependencies that are specified in the recipe. Because only the specified dependencies can be found in /System/Links directory, a build cannot accidentally succeed if a dependency has been omitted.

Both Nix and ChrootCompile have the ability to prevent undeclared dependencies to accidentally succeed a build, which improves reproducibility. In Nix, this goal is achieved differently. In Nix, the environment variables in which a build is performed are completely cleared (well not completely, but almost :-) ), and dependencies which are specified are added to the PATH and other environment variables, which allow build tools to find the dependencies.

Nix builds can be optionally performed in a chroot environment, but this is not mandatory. In NixOS, the traditional FHS directories, such as /usr don't exist and cannot make a build impure. Furthermore, common utilities such as GCC have been patched so that they ignore standard directories, such as /usr/include, removing many impurities.

Furthermore, Nix also binds dependency relationships statically to the executables (e.g. by modifying the RPATH header in an ELF binary), instead of allowing binaries to look in global directories like: /System/Links, which GoboLinux executables do. Although, GoboLinux builds are pure inside a chroot environment, their run-time behaviour may be different when a user decides to upgrade a version of its dependency.

Conclusion

In this blog post I did an evaluation of GoboLinux and I compared some features with NixOS. In my opinion, evaluating GoboLinux shows a number of interesting lessons:

• There are many deployment related aspects (and perhaps other aspects as well) that can be solved and improved by just using the filesystem. I often see that people write additional utilities, abstraction layers and databases to perform similar things, while you can also use symlink trees and bind mounts to provide abstractions and compositions. Additionally, this also shows that the filesystem, which is an essential key component for UNIX-like systems, is still important and very powerful.
• Storing packages in separate directories is a simple way to manage their contents, store multiple versions safely next to each other and to make upgrades more reliable.

GoboLinux and NixOS have a number of similar deployment properties, because of their unorthodox filesystem organization, but also some differences and limitations. The following table summarizes the differences covered in this blog post:

	GoboLinux	NixOS
FHS compatibility	Yes (through hidden symlink trees)	No (deviates on some FHS aspects)
Component naming	Nominal (name-version)	Exact (hash-name-version)
Component binding	Dynamic (dependency can be replaced without rebuild)	Static (rebuild required if a dependency changes)
Granularity of component isolation	Only between versions	Between all build-time dependencies (including version)
Unprivileged user installations	No	Yes
Build specifications	Stand-alone recipes	Nix expressions which need to refer to all dependency expressions
Build-time dependency addressing	/System/Links symlink tree in chroot environment	Setting environment variables + modified tools + optional chroot environment
Run-time dependencies	Specified manually	Extracted by scanning for hash occurrences
Run-time dependency resolving	Dynamic, by looking in /System/Links	Static (e.g. encoded in RPATH)

The general conclusion of this blog post is that both distributions achieve better deployment properties compared to conventional Linux distributions, by their unorthodox filesystem organisation. Because of the purely functional deployment model of the Nix package manager, NixOS is more powerful (and strict) than GoboLinux when it comes to reliability, although this comes at the expense of extra rebuild times and additional disk space.

The only bad thing I can say about GoboLinux is that the latest 014.01 release is outdated (2008) and it looks like 015 is in progress for almost three years... I'm not sure if there will be a new release soon, which is a pity.

And of course, apart from these deployment properties, there are many other differences I haven't covered here, but that's up to the reader to make a decision.

I'm planning to use these lessons for a future blog post, which elaborates more on techniques for making software deployment processes more reliable.