Unfortunately, he did not read my blog post (or possibly one of the three Nix and NixOS explanation recipes) in detail.
Moreover, I was hoping that somebody else involved with Snappy Ubuntu would do a more in depth comparison and write a response, but this still has not happened yet. As a matter of fact, there is still no answer as of today.
Because of these reasons, I have decided to take a look at Snappy Ubuntu Core and do an evaluation myself.
What is Snappy Ubuntu?
Snappy is Ubuntu's new mechanism for delivering applications and system upgrades. It is used as the basis of their upcoming cloud and mobile distributions and supposed to be offered alongside the Debian package manager that Ubuntu currently uses for installing and upgrading software in their next generation desktop distribution.
Besides the ability to deploy packages, Snappy also has a number of interesting non-functional properties. For example, the website says the following:
The snappy approach is faster, more reliable, and lets us provide stronger security guarantees for apps and users -- that's why we call them "snappy" applications.
Snappy apps and Ubuntu Core itself can be upgraded atomically and rolled back if needed -- a bulletproof approach that is perfect for deployments where predictability and reliability are paramount. It's called "transactional" or "image-based" systems management, and we’re delighted to make it available on every Ubuntu certified cloud.
The text listed above contains a number of interesting quality aspects that have a significant overlap with Nix -- reliability, atomic upgrades and rollbacks, predictability, and being "transactional" are features that Nix also implements.
The Snappy Ubuntu Core distribution uses mostly a FHS compliant filesystem layout. One notable deviation is the folder in which applications are installed.
For application deployment the /app folder is used in which files belonging to a specific application version reside in separate folders. Application folders use the following naming convention:
Each application is identified by its name, version identifier and optionally a developer identifier, as shown below:
$ ls -l /apps drwxr-xr-x 2 root ubuntu 4096 Apr 25 20:38 bin drwxr-xr-x 3 root root 4096 Apr 25 15:56 docker drwxr-xr-x 3 root root 4096 Apr 25 20:34 go-example-webserver.canonical drwxr-xr-x 3 root root 4096 Apr 25 20:31 hello-world.canonical drwxr-xr-x 3 root root 4096 Apr 25 20:38 webcam-demo.canonical drwxr-xr-x 3 root ubuntu 4096 Apr 23 05:24 webdm.sideload
For example, the /app/webcam-demo.canonical/1.0.1 refers to a package named: webcam version 1.0.1 that is delivered by Canonical.
There are almost no requirements on the contents of an application folder. I have observed that the example packages seem to follow some conventions though. For example:
$ cd /apps/webcam-demo.canonical/1.0.1 $ find . -type f ! -iname ".*" ./bin/x86_64-linux-gnu/golang-static-http ./bin/x86_64-linux-gnu/fswebcam ./bin/golang-static-http ./bin/runner ./bin/arm-linux-gnueabihf/golang-static-http ./bin/arm-linux-gnueabihf/fswebcam ./bin/fswebcam ./bin/webcam-webui ./meta/readme.md ./meta/package.yaml ./lib/x86_64-linux-gnu/libz.so.1 ./lib/x86_64-linux-gnu/libc.so.6 ./lib/x86_64-linux-gnu/libX11.so.6 ./lib/x86_64-linux-gnu/libpng12.so.0 ./lib/x86_64-linux-gnu/libvpx.so.1 ... ./lib/arm-linux-gnueabihf/libz.so.1 ./lib/arm-linux-gnueabihf/libc.so.6 ./lib/arm-linux-gnueabihf/libX11.so.6 ./lib/arm-linux-gnueabihf/libpng12.so.0 ./lib/arm-linux-gnueabihf/libvpx.so.1 ...
Binaries are typically stored inside the bin/ sub folder, while libraries are stored inside the lib/ sub folder. Moreover, the above example also ships binaries for two kinds of system architectures (x86_64 and ARM) that reside inside bin/x86_64-linux, bin/arm-linux-gnueabihf, and lib/x86_64-linux, arm-linux-gnueabihf sub folders.
The only sub folder that has a specific purpose is meta/ that is supposed to contain at least two files -- the readme.md file contains documentation in which the first heading and the first paragraph have a specific meaning, and the package.yaml file contains various meta attributes related to the deployment of the package.
Snappy's package storing convention also makes it possible to store multiple versions of a package next to each other, as shown below:
$ ls -l /apps/webcam-demo.canonical drwxr-xr-x 7 clickpkg clickpkg 4096 Apr 24 19:38 1.0.0 drwxr-xr-x 7 clickpkg clickpkg 4096 Apr 25 20:38 1.0.1 lrwxrwxrwx 1 root root 5 Apr 25 20:38 current -> 1.0.1
Moreover, every application folder contains a symlink named: current/ that refers to the version that is currently in use. This approach makes it possible to do atomic upgrades and rollbacks by merely flipping the target of the current/ symlink. As a result, the system always refers to an old or new version of the package, but never to an inconsistent mix of the two.
Apart from storing packages in isolation, they also must be made accessible to end users. For each binary that is declared in the package.yaml file, e.g.:
# for debugging convience we also make the binary available as a command binaries: - name: bin/webcam-webui
a wrapper script is placed inside /apps/bin that is globally accessible by the users of a system through the PATH environment variable.
Each wrapper script contains the app name. For example, the webcam-webui binary (shown earlier) must be started as follows:
Besides binaries, package configurations can also declare services from which systemd jobs are composed. The corresponding configuration files are put into the /etc/systemd/system folder and also use a naming convention containing the package name.
Unprivileged users can also install their own packages. The corresponding application files are placed inside $HOME/app and are organized in the same way as the global /app folder.
Snappy's package organization has many similarities with Nix's package organization -- Nix also stores files files belonging to a package in isolated folders in a special purpose directory called the Nix store.
However, Nix uses a more powerful way of identifying packages. Whereas Snappy only identifies packages with their names, version numbers and vendor identifiers, Nix package names are prefixed with unique hash codes (such as /nix/store/wan65mpbvx2a04s2s5cv60ci600mw6ng-firefox-with-plugins-27.0.1) that are derived from all build time dependencies involved to build the package, such as compilers, libraries and the build scripts themselves.
The purpose of using hash codes is to make a distinction between any variant of the same package. For example, when a package is compiled with a different version of GCC, linked against a different library dependency, when debugging symbols are enabled or disabled or certain optional features enabled or disabled, or the build procedure has been modified, a package with a different hash is built that is safely stored next to existing variants.
Moreover, Nix also uses symlinking to refer to specific versions of packages, but the corresponding mechanism is more powerful. Nix generates so-called Nix profiles which synthesize the contents of a collection of installed packages in the Nix store into a symlink tree so that their files (such as executables) can be referenced from a single location. A second symlink indirection refers to Nix profile containing the desired versions of the packages.
Nix profiles also allow unprivileged users to manage their own set of private packages that do not conflict with other user's private packages or the system wide installed packages. However, partly because of Nix's package naming convention, also the packages of unprivileged users can be safely stored in the global Nix store, so that common dependencies can be shared among users in a secure way.
Software packages are rarely self contained -- they typically have dependencies on other packages, such as shared libraries. If a dependency is missing or incorrect, a package may not work properly or not at all.
I observed that in the Snappy example packages, all dependencies are bundled statically. For example, in the webcam-demo package, the lib/ sub folder contains the following files:
./lib/x86_64-linux-gnu/libz.so.1 ./lib/x86_64-linux-gnu/libc.so.6 ./lib/x86_64-linux-gnu/libX11.so.6 ./lib/x86_64-linux-gnu/libpng12.so.0 ./lib/x86_64-linux-gnu/libvpx.so.1 ... ./lib/arm-linux-gnueabihf/libz.so.1 ./lib/arm-linux-gnueabihf/libc.so.6 ./lib/arm-linux-gnueabihf/libX11.so.6 ./lib/arm-linux-gnueabihf/libpng12.so.0 ./lib/arm-linux-gnueabihf/libvpx.so.1 ...
As can be seen in the above output, all the library dependencies, including the libraries' dependencies (even libc) are bundled into the package. When running an executable or starting a systemd job, a container (essentially an isolated/restricted environment) is composed in which the process runs (with some restrictions) where it can find its dependencies in the "common FHS locations", such as /lib.
Besides static bundling, there seems to be a primitive mechanism that provides some form of sharing. According to the packaging format specification, it also possible to declare dependencies on frameworks in the package.yaml file:
frameworks: docker, foo, bar # list of frameworks required
Frameworks are managed like ordinary packages in /app, but they specify additional required system privileges and require approval from Canonical to allow them to be redistributed.
Although it is not fully clear to me from the documentation how these dependencies are addressed, I suspect that the contents of the frameworks is made available to packages inside the containers in which they run.
Moreover, I noticed that dependencies are only addressed by their names and that they refer to the current versions of the corresponding frameworks. In the documentation, there seems to be no way (yet) to refer to other versions or variants of frameworks.
The Nix-way of managing dependencies is quite different -- Nix packages are constructed from source and the corresponding build procedures are executed in isolated environments in which only the specified build-time dependencies can be found.
Moreover, when constructing Nix packages, runtime dependencies are bound statically to executables, for example by modifying the RPATH of an ELF executable or wrapping executables in scripts that set environment variables allowing it to find its dependencies (such as CLASSPATH or PERL5LIB). A subset of the buildtime dependencies are identified by Nix as runtime dependencies by scanning for hash occurrences in the build result.
Because dependencies are statically bound to executables, there is no need to compose containers to allow executables to find them. Furthermore, they can also refer to different versions or variants of library dependencies of a package without conflicting with other package's dependencies. Sharing is also supported, because two packages can refer to the same dependency with the same hash prefix in the Nix store.
As a sidenote: with Nix you can also use a containerized approach by composing isolated environments (e.g. a chroot environment or container) in which packages can find their dependencies from common locations. A prominent Nix package that uses this approach is Steam, because it is basically a deployment tool conflicting with Nix's deployment properties. Although such an approach is also possible, it is only used in very exceptional cases.
Besides applications and frameworks, the base system of the Snappy Ubuntu Core distribution can also be upgraded and downgraded. However, a different mechanism is used to accomplish this.
According to the filesystem layout & updates guide, the Snappy Ubuntu Core distribution follows a specific partition layout:
- boot partition. This is a very tiny partition used for booting and should be big enough to contain a few kernels.
- system-a partition. This partition contains a minimal working base system. This partition is mounted read-only.
- system-b partition. An alternative partition containing a minimal working base system. This partition is mounted read-only as well.
- writable partition. A writable partition that stores everything else including the applications and frameworks.
Snappy uses an "A/B system partitions mechanism" to allow a base system to be updated as a single unit by applying a new system image. It is also used to roll back to the "other" base system in case of problems with the most recently-installed system by making the bootloader switch root filesystems.
NixOS (the Linux distribution built around the Nix package manager) approaches system-level upgrades in a different way and is much more powerful. In NixOS, a complete system configuration is composed from packages residing in isolation in the Nix store (like ordinary packages) and these are safely stored next to existing versions. As a result, it is possible to roll back to any previous system configuration that has not been garbage collected yet.
According to the packaging guide, creating Snap files is very simple. It is just creating a directory, putting some files in there, creating a meta/ sub folder with a readme.md and package.yaml file, and running:
$ snappy build .
The above file generates a Snap file, which is basically just a tarball file containing the contents of the folder.
In my opinion, creating Snap packages is not that easy -- the above process demonstrates that delivering files from one machine to another is straight forward, but getting a package right is another thing.
Many packages on Linux systems are constructed from source code. To properly do that, you need to have the required development tools and libraries deployed first, a process that is typically easier said than done.
Snappy does not provide facilities to make that process manageable. With Snappy, it is the packager's own responsibility.
In contrast, Nix is a source package manager and provides a DSL that somebody can use construct isolated environments in which builds are executed and automatically deploys all buildtime dependencies that are required to build a package.
The build facilities of Nix are quite accessible. For example, you can easily construct your own private set of Nix packages or a shell session containing all development dependencies.
Moreover, Nix also implements transparent binary deployment -- if a particular Nix package with an identical hash exists elsewhere, we can download it from a remote location instead of building it from source ourselves.
Another thing the Snappy Ubuntu Core distribution does with containers (besides using them to let a package find its dependencies) is restricting the things programs are allowed to do, such as the TCP/UDP ports they are allowed to bind to.
In Nix and NixOS, it is not a common practice to restrict the runtime behaviour of programs by default. However, it is still possible to impose restrictions on running programs, by composing a systemd job for a program yourself in a system's NixOS configuration.
The following table summarizes the conceptual differences between the Snappy Ubuntu Core and Nix/NixOS covered in this blog post:
|Snappy Ubuntu Core||Nix/NixOS|
|Concept||Binary package manager||Source package manager (with transparent binary deployment)|
|Dependency addressing||By name||Exact (using hash codes)|
|Dependency binding||Container composition||Static binding (e.g. by modifying RPATH or wrapping executables)|
|Systems composition management||"A/B" partitions||Package compositions|
|Construction from source||Unmanaged||Managed|
|Unprivileged user installations||Supported without sharing||Supported with sharing|
|Runtime isolation||Part of package configuration||Supported optionally, by manually composing a systemd job|
Snappy shares some interesting traits with Nix that provide a number of huge deployment benefits -- by deviating from the FHS and storing packages in isolated folders, it becomes easier to store multiple versions of packages next to each other and to perform atomic upgrades and rollbacks.
However, something that I consider a huge drawback of Snappy is the way dependencies are managed. In the Snappy examples, all library dependencies are bundled statically consuming more disk space (and RAM at runtime) than needed.
Moreover, packaging shared dependencies as frameworks is not very convenient and require approval from Canonical if they must be distributed. As a consequence, I think it will not be very encouraging to modularize systems, which is generally considered a good practice.
According to the framework guide the purpose of frameworks is to extend the base system, but not to be a sharing mechanism. Also the guide says:
Frameworks exist primarily to provide mediation of shared resources (eg, device files, sensors, cameras, etc)
So it appears that sharing in general is discouraged. In many common Linux distributions (including Debian and derivatives such as Ubuntu), it is common that the reuse-degree is raised to almost a maximum. For example, each library is packaged individually and sometimes libraries are even split into binary, development and documentation sub packages. I am not sure how Snappy is going to cope with such a fine granularity of reuse. Is Snappy going to be improved to support reuse as well, or is it now considered a good thing to package huge monolithic blobs?
Also, Snappy does only binary deployment and is not really helpful to alleviate the problem of constructing packages from source which is also quite a challenge in my opinion. I see lots of room for improvement in this area as well.
Another funny observation is the fact that Snappy Ubuntu Core relies on advanced concepts such as containers to make programs work, while there are also simpler solutions available, such as static linking.
Finally, something that Nix/NixOS could learn from the Snappy approach is the runtime isolation of programs out-of-the-box. Currently, doing this with Nix/NixOS this is not as convenient as with Snappy.
This is not the only comparison I have done between Nix/NixOS and another deployment approach. A few years ago while I was still a PhD student, I also did a comparison between the deployment properties of GoboLinux and NixOS.
Interestingly enough, GoboLinux addresses packages in a similar way as Snappy, supports sharing, does not provide runtime isolation of programs, but does have a very powerful source construction mechanism that Snappy lacks.
There is an ongoing discussion about this blog post on reddit:ReplyDelete
But how could snappy take advantage of sharing if it's meant to use containers?. Also, is Nix meant for distributed, redundant systems?ReplyDelete
There are various ways to accomplish sharing while having a containerized deployment approach. One of the things I have seen people mentioning in the reddit threads is to use the deduplication feature of the file system, such as btrfs.ReplyDelete
However, there is also another way that I consider a better approach. For example, with Nix we can also compose containers in which dependencies can be found in their "common FHS locations", which we use to make Steam work.
Moreover, we support sharing by bind mounting the /nix/store folder (roughly an equivalent to Snappy's /app folder) in the container and by composing a symlink trees for the /bin, /lib etc. folders to make the packages' contents accessible from a single (and common) location.
There is a small security risk involved while bind mounting the entire Nix store (for example, a process in the container can execute any installed program), but this can be solved by only bind mounting the package directories of the dependencies only.
As far as I can see, it is also possible to implement such an approach with Snappy.
About Nix's purpose: Nix has a variety of applications built on top of it -- supporting distributed deployment is one them (Disnix and NixOps), but it can also be used for continuous integration (Hydra) and managing a Linux distribution (NixOS).
More ongoing discussions:ReplyDelete
ok, I'll keep reading on the matterReplyDelete
thanks for sharing your insights!ReplyDelete
Nice post, little unclear on shared library hell though. Would snappy solve that issue too. The biggest problem that Linux has is that one is stuck to stale software versions in the repo because of shared library conflicts.ReplyDelete
What I was trying to make clear is that there is no "shared dependency hell" in neither Snappy nor Nix.ReplyDelete
However, the Snappy approach solves it by setting up isolated containers that contain everything it needs to run the executable. The container isolation property prevents processes from referencing any undeclared dependencies.
The latter aspect solves the "shared library hell" but comes at a great price in terms of disk space and RAM usage. As a matter of fact, the problem is solved by eliminating sharing.
With Nix the ability to share is retained -- it links shared libraries statically to an executable (by modifying an ELF executable's RPATH). To clear up some confusion: this is not the same thing as statically linking static libraries (e.g. *.a files)!!
This approach also solves the "shared dependency hell" because executables only look inside the library folders residing in the RPATH which correspond to isolated folders in the Nix store (and not to global directories, such as /lib). As a result, only the libraries to which the executable refers are used and no other versions (such as undeclared dependencies).
Moreover, because multiple executables can still refer to the same library package, sharing is retained, both on disk (and in RAM because they happen to reference the same file).
NixOS is very promising but need more people spreading it. I tried some weeks ago, but got stuck with some problems. Documentation :-(ReplyDelete
(as a Gentoo user I'm used to just search in the docs/forums and solve the problem, NixOS I needed to use IRC)
PS: Congrats for another excellent post!
Yeah, so this blog post's purpose is to compare the deployment properties of the underlying package managers of both distributions.Delete
However, in many other areas (e.g. usability) there are definitely some things that could be improved in NixOS. This is what Ubuntu obviously does better than us.
We're working on improving these others aspects as well, but we don't have a community that is as big as Ubuntu's. Something that could be worth investigating is how usable the NixOS distribution is from an end-user perspective.
I had a similar experience with Nix. I installed Python and pandas (a python library) and I never managed to run python with the pandas library at the same time.Delete
My feelings about Nix is that it is very promising and that offers great flexibility, but it requires to expend a large amount of time to get things working.
My suggestion is to provide easy to use common use cases and let users dig into the details if they need something more customized.
For some tasks, I feel that I would have to invest more time in Nix (to learn Nix DSL and Nix internals) than in Snappy to accomplish the same things (certainly without the same advantages).
Yes, so usability is definitely an area we have to look into.Delete
From my perspective, I think many things are not too hard to learn. The biggest challenge is that some concepts are quite unconventional and require people to look at problems and solutions in a different way. Also, most of Nix's unique advantages stem from these unconventional concepts.
I think that once people are past this barrier, the learning curve is not that steep. However, we have to find the right means to accomplish this in an accessible way. I have some ideas about this, but I haven't really put much effort in it. Maybe I should give it a higher priority...
The other "issue" is (obviously) that the tools and documentation can be a bit more polished here and there.
We have created an issue on GitHub to gather some UX-related user feedback: