Managing private Nix packages outside the Nixpkgs tree

In a couple of older blog posts, I have explained the basic concepts of the Nix package manager as well as how to write package "build recipes" (better known as Nix expressions) for it.

Although Nix expressions may look unconventional, the basic idea behind specifying packages in the Nix world is simple: you define a function that describes how to build a package from source code and its dependencies, and you invoke the function with the desired variants of the dependencies as parameters to build it. In Nixpkgs, a collection of more than 2500 (mostly free and open source) packages that can be deployed with Nix, all packages are basically specified like this.

However, there might still be some practical issues. In some cases, you may just want to experiment with Nix or package private software not meant for distribution. In such cases, you typically want to store them outside the Nixpkgs tree.

Although the Nix manual describes how things are packaged in Nixpkgs, it does not (clearly) describe how to define and compose packages while keeping them separate from Nixpkgs.

Since it is not officially documented anywhere and I'm getting (too) many questions about this from beginners, I have decided to write something about it.

Specifying a single private package

In situations in which I want to quickly try or test one simple package, I typically write a Nix expression that looks as follows:

with import <nixpkgs> {};

stdenv.mkDerivation {
  name = "mc-4.8.12";
  
  src = fetchurl {
    url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
    sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
  };
  
  buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
      xlibs.libX11 xlibs.libICE e2fsprogs ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    license = "GPLv2+";
    maintainers = [ stdenv.lib.maintainers.sander ];
  };
}

The above expression is a Nix expression that builds Midnight Commander, one of my favorite UNIX utilities (in particular the editor that comes with it :-) ).

In the above Nix expression, there is no distinction between a function definition and invocation. Instead, I directly invoke stdenv.mkDerivation {} to build Midnight Commander from source and its dependencies. I obtain the dependencies from Nixpkgs by importing the composition attribute set into the lexical scope of the expression through with import <nixpkgs> {};.

I can put the above file (named: mc.nix) in a folder outside the Nixpkgs tree, such as my home directory, and build it as follows:

$ nix-build mc.nix
/nix/store/svm98wmbf01dswlfcvvxfqqzckbhp5n5-mc-4.8.12

Or install it in my profile by running:

$ nix-env -f mc.nix -i mc

The dependencies (that are provided by Nixpkgs) can be found thanks to the NIX_PATH environment variable that contains a setting for nixpkgs. On NixOS, this environment variable has already been set. On other Linux distributions or non-NixOS installations, this variable must be manually configured to contain the location of Nixpkgs. An example could be:

$ export NIX_PATH=nixpkgs=/home/sander/nixpkgs

The above setting specifies that a copy of Nixpkgs resides in my home directory.

Maintaining a collection private packages

It may also happen that you want to package a few of the dependencies of a private package while keeping them out of Nixpkgs or just simply maintaining a collection of private packages. In such cases, I basically define every a package as a function, which is no different than the way it is done in Nixpkgs and described in the Nix manual:

{ stdenv, fetchurl, pkgconfig, glib, gpm, file, e2fsprogs
, libX11, libICE, perl, zip, unzip, gettext, slang
}:

stdenv.mkDerivation rec {
  name = "mc-4.8.12";
  
  src = fetchurl {
    url = http://www.midnight-commander.org/downloads/mc-4.8.12.tar.bz2;
    sha256 = "15lkwcis0labshq9k8c2fqdwv8az2c87qpdqwp5p31s8gb1gqm0h";
  };
  
  buildInputs = [ pkgconfig perl glib gpm slang zip unzip file gettext
      libX11 libICE e2fsprogs ];

  meta = {
    description = "File Manager and User Shell for the GNU Project";
    homepage = http://www.midnight-commander.org;
    license = "GPLv2+";
    maintainers = [ stdenv.lib.maintainers.sander ];
  };
}

However, to compose the package (i.e. calling the function with the arguments that are used as dependencies), I have to create a private composition expression instead of adapting pkgs/top-level/all-packages.nix in Nixpkgs.

A private composition expression could be defined as follows:

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
in
rec {
  pkgconfig = import ./pkgs/pkgconfig {
    inherit (pkgs) stdenv fetchurl automake;
  };
  
  gpm = import ./pkgs/gpm {
    inherit (pkgs) stdenv fetchurl flex bison ncurses;
  };
  
  mc = import ./pkgs/mc {
    # Use custom pkgconfig and gpm packages as dependencies
    inherit pkgconfig gpm;
    # The remaining dependencies come from Nixpkgs
    inherit (pkgs) stdenv fetchurl glib file perl;
    inherit (pkgs) zip unzip gettext slang e2fsprogs;
    inherit (pkgs.xlibs) libX11 libICE;
  };
}

The above file (named: custom-packages.nix) invokes the earlier Midnight Commander expression (defining a function) with its required parameters.

Two of its dependencies are also composed in the same expression, namely: pkgconfig and gpm that are also stored outside the Nixpkgs tree. The remaining dependencies of Midnight Commander are provided by Nixpkgs.

To make the above example complete, the directory structure of the set of Nix expressions is supposed to look as follows:

pkgs/
  pkgconfig/
    default.nix
    requires-private.patch 
    setup-hook.sh
  gpm/
    default.nix
  mc/
    default.nix
custom-packages.nix

The expressions for gpm and pkgconfig can be copied from Nixpkgs, by running ($nixpkgs should be replaced by the path to Nixpkgs on your system):

cp -a $nixpkgs/pkgs/pkgs/development/tools/misc/pkgconfig pkgs
cp -a $nixpkgs/pkgs/servers/gpm pkgs

Using the above Nix composition expression file (custom-packages.nix), the other Nix expressions it refers to, and by running the following command-line instruction:

$ nix-build custom-packages.nix -A mc
/nix/store/svm98wmbf01dswlfcvvxfqqzckbhp5n5-mc-4.8.12

I can build our package using our private composition of packages. Furthermore, I can also install it into my Nix profile by running:

$ nix-env -f custom-packages.nix -iA mc

Because the composition expression is also a function taking system as a parameter (which defaults to the same system architecture as the host system), I can also build Midnight Commander for a different system architecture, such as a 32-bit Intel Linux system:

$ nix-build custom-packages.nix -A mc --argstr system i686-linux

Simplifying the private composition expression

The private composition expression shown earlier passes all required function arguments to each package definition, which basically requires anyone to write function arguments twice. First to define them and later to provide them.

In 95% of the cases, the function parameters are typically packages defined in the same composition attribute set having the same attribute names as the function parameters.

In Nixpkgs, there is a utility function named callPackage {} that simplifies things considerably -- it automatically passes all requirements to the function by taking the attributes with the same name from the composition expression. So there is no need to write: inherit gpm ...; anymore.

We can also define our own private callPackage {} function that does this for our private composition expression:

{ system ? builtins.currentSystem }:

let
  pkgs = import <nixpkgs> { inherit system; };
  
  callPackage = pkgs.lib.callPackageWith (pkgs // pkgs.xlibs // self);
  
  self = {
    pkgconfig = callPackage ./pkgs/pkgconfig { };
  
    gpm = callPackage ./pkgs/gpm { };
  
    mc = callPackage ./pkgs/mc { };
  };
in
self

The above expression is a simplified version of our earlier composition expression (named: custom-packages.nix) that uses callPackage {} to automatically pass all required dependencies to the functions that build a package.

callPackage itself is composed from the pkgs.lib.callPackageWith function. The first parameter (pkgs // pkgs.xlibs // self) defines the auto-arguments. In this particular case, I have specified that the automatic function arguments come from self (our private composition) first, then from the xlibs sub attribute set from Nixpkgs, and then from the main composition attribute set of Nixpkgs.

With the above expression, we accomplish exactly the same thing as in the previous expression, but with fewer lines of code. We can also build the Midnight Commander exactly the same way as we did earlier:

$ nix-build custom-packages.nix -A mc
/nix/store/svm98wmbf01dswlfcvvxfqqzckbhp5n5-mc-4.8.12

Conclusion

In this blog post, I have described how I typically maintain a single package or a collection packages outside the Nixpkgs tree. More information on how to package things in Nix can be found in the Nix manual and the Nixpkgs manual.