Using Nix while doing development

I have noticed that while doing development work, many outsiders experience the way I work as quite odd and consider it to be inefficient.

The first reason is (probably) because I like the command-line for many tasks and I frequently use a unconventional text editor, which some people don't understand. Second, I also often use Nix (and related Nix utilities) during development.

In this blog post, I'd like to elaborate a bit about the second aspect and discuss why it is actually quite useful to do this.

Obtaining and installing the dependencies of a software project

In all software projects I have been involved with so far, the first thing I typically had to do is installing all its required dependencies. Examples of such dependencies are a compiler/interpreter + runtime environment for a specific programming language (such as GCC, the Java Development Kit, Perl, Python, Node.js etc.), a development IDE (such as Eclipse), and many library packages.

I have experienced that this step is often quite time consuming, since many dependencies have to be downloaded and installed. Moreover, I have frequently encountered situations in which a bunch of special settings have to be configured to make everything work right, which can be quite cumbersome and tedious.

Many developers just simply download and install everything on their machine in an ad-hoc manner and manually perform all the steps they have to do.

Automating the deployment of dependencies of a software project

What I often do instead is to immediately use Nix to automate the process of installing all software project dependencies and configuring all their settings.

To fully automate deployment with Nix, all dependencies have to be provided as Nix packages. First, I investigate if everything I need is in the Nix packages collection. Fortunately, the Nix package collection provides packages for many kinds of programming languages and associated libraries. If anything is missing, I package it myself. Although some things are very obscure to package (such as the Android SDK), most things can be packaged in a straight forward manner.

After packaging all the missing dependencies, I must install them and generate an environment in which all the dependencies can be found. For example, in order to let Java projects find their library dependencies, we must set the CLASSPATH environment variable to refer to directories or JAR files containing our required compiled classes, PERL5LIB to let Perl find its Perl modules, NODE_PATH to let Node.js find its CommonJS modules etc.

The Nix packages collection has a nice utility function called myEnvFun that can be used to automatically do this. This function can be used to compose development environments that automatically set nearly all of its required development settings. The following example Nix expression (disnixenv.nix) can be used to deploy a development environment for Disnix:

with import <nixpkgs> {};

myEnvFun {
  name = "disnix";
  buildInputs = [
    pkgconfig dbus dbus_glib libxml2 libxslt getopt
    autoconf automake libtool dysnomia
  ];
}

The development environment can be deployed by installing it with the Nix package manager:

$ nix-env -f disnixenv.nix -i env-disnix

The above command-line invocation deploys all Disnix's dependencies and generates a shell script (named: load-env-disnix) that launches a shell session in an environment in which all build settings are configured. We can enter this development environment by running:

$ load-env-disnix
env-disnix loaded

disnix:[sander@nixos:~]$

As can be seen in the code fragment above, we have entered an environment in which all its development dependencies are present and configured. For example, the following command-line instruction should work, since libxml2 is declared as a development dependency:

$ xmllint --version
xmllint: using libxml version 20901

We can also unpack the Disnix source tarball and run the following build instructions, which should work without any trouble:

$ ./bootstrap
$ ./configure
$ make

Besides deploying a development environment, we can also discard it if we don't need it anymore by running:

$ nix-env -e env-disnix

Automating common deployment tasks

After deploying a development environment, I'm usually doing all kinds of development tasks until the project reaches a more stable state. Once this is the case, I immediately automate most of its deployment operations in a Nix expression (that is often called: release.nix).

This Nix expression is typically an attribute set following Hydra's (the Nix-based continuous integration server) convention in which each attribute represents a deployment job. Jobs that I typically implement are:

• Source package. This is a job that helps me to easily redistribute the source code of a project to others. For GNU Autotools based projects this typically involves the: make dist command-line instruction. Besides GNU Autotools, I also generate source packages for other kinds of projects. For example, in one of my Node.js projects (such a NiJS) I produce a TGZ file that can be deployed with the NPM package manager.
• Binary package jobs that actually build the software project for every target system that I want to support, such as i686-linux, x86_64-linux, and x86_64-darwin.
• A job that automatically compiles the program manual from e.g. Docbook, if this applicable.
• A program documentation catalog job. Many programming languages allow developers to write code-level documentation in comments from which a documentation catalog can be generated, e.g. through javadoc, doxygen or JSDuck. I also create a job that takes care of doing that.
• Unit tests. If my project has a unit test suite, I also create a job executing those, since it is required to deploy most of its development dependencies to run the test suite as well.
• System integration tests. If I have any system integration tests that can run be on Linux, I try to implement a job using the NixOS test driver to run those. The NixOS test driver also automatically deploys all the required environmental dependencies in a VM, such as a DBMS, web server etc. in such a way that you can run them as a unprivileged user without affecting the host system's configuration.

After writing a release Nix expression, I can use the Nix package manager to perform all the deployment tasks for my software project. For example, I can run the following job that comes with Disnix to build a source tarball:

$ nix-build release.nix -A tarball

Another advantage is that I can use the same expression in combination with Hydra to so that I can continuously build and test it.

Spawning development environments from Nix jobs

Apart from being able to deploy a software project and its dependencies, another nice advantage of having its deployment automated through Nix is that I can use the same Nix jobs to reproduce its build environment. By replacing nix-build with nix-shell all its dependencies are deployed, but instead of building the package itself, a shell session is started in which all its dependencies are configured:

$ nix-shell release.nix -A tarball

The above command-line invocation produces a similar build environment as shown earlier with myEnvFun by just using a Nix expression for an existing job.

Discussion

So far I have explained what I typically do with the Nix package manager during development. So why is this actually useful?

• Besides deploying systems into production environments, setting up a development environment is a similar problem with similar complexities. For me it is logical that I solve these kind of problems the same way as I do in production environments.
• Packaging all software project dependencies in Nix ensures reliable and reproducible deployment in production environments as well, since Nix ensures dependency completeness and removes as many side effects as possible. In Nix, we have strong guarantees that a system deployed through Nix in production behaves exactly the same in development and vice-versa.
• We can also share development environments, which is useful for a variety of reasons. For example, we can reproduce the same development environment on a second machine, or give it to another developer to prevent him spending a lot of time in setting up his development environment.
• Supporting continuous integration and testing with Hydra comes (almost) for free.
• With relatively little effort you can also enable distributed deployment of your software project with Disnix and/or NixOps.

Although I have encountered people telling me that it's consuming too much time, you also get a lot for it in return. Moreover, in most common scenarios I don't think the effort of automating the deployment of a development environment is that big if you have experience with it.

As a concluding remark, I know that things like Agile software development and continuous delivery of new versions is something that (nearly) everybody wants these days. To implement continuous delivery/deployment, the first step is to automate the deployment of a software project from the very beginning. That is exactly what I'm doing by executing the steps described in this blog post.

People may also want to read an old blog post that contains recommendations to improve software deployment processes.