Monday, December 30, 2013

Third annual blog reflection

Today, it's three years ago that I started this blog, so I thought this is a good opportunity to reflect about last year's writings.

Obtaining my PhD degree


I think the most memorable thing that happened for me this year is the fact that I finally obtained my PhD degree. From that moment on, I can finally call myself Dr. Sander, or actually (if I take my other degree into account): Dr. Ir. Sander (or in fact: Dr. Ir. Ing. Sander if I take all of them into account, but I believe the last degree has been been superseded by the middle one, but I'm not sure :-) ).

Anyway, after obtaining my PhD degree I don't feel that much different, apart from the fact that I feel relieved that it's done. It took me quite some effort to get my PhD dissertation and all the preparations done for the defense. Besides my thesis, I also had to defend my propositions. Most of them were not supposed to be directly related to my research subject.

Programming in JavaScript


From the moment that I have switched jobs, I have also been involved in a lot of JavaScript programming these days. Every programming language and runtime environment have their weird/obscure problems/challenges, but in my opinion JavaScript is a very special case.

As a former teaching assistant for the concepts of programming languages course, I remain interested in discovering important lessons allowing me to prevent turning code into a mess and dealing with challenges that a particular programming gives. So far, I have investigated object oriented programming through prototypes, and two perspectives of dealing with asynchronous programming problems that come with most JavaScript environments.

Besides programming challenges I also have to perform deployment tasks from JavaScript programs. People who happen to know me know that I prefer Nix and Nix-related solutions. I have developed NiJS: an internal DSL for Nix to make my life a bit easier to do that.

Continuous integration and testing


Another technical aspect I have been working on is setting up a continuous integration facility by using Hydra: the Nix-based continuous integration server. I wrote a couple of blog posts describing its features, how to set it up and how to secure it.

I also made a couple improvements to the Android and iOS Nix build functions, so that I can use Hydra to continuously build mobile apps.

Nix/NixOS development


Besides Hydra, I have been involved with various other parts of the Nix project as well. One of the more interesting things I did is developing a Nix function that can be used to compose FHS-compatible chroot environments. This function is particularly useful to run binary-only software in NixOS that cannot be patched, such as Steam.

I also wrote two blog posts to explain the user environment and development environment concepts.

Fun programming


Besides my PhD defense and all other activities, there was a bit of room to do some fun programming as well. I have improved my Amiga video emulation library (part of my IFF file format experiments project) a bit by implementing support for super hires resolutions and redefining its architecture.

Moreover, I have updated all the packages to use the new version of this library.

Research


After obtaining my PhD degree, I'm basically relieved from my research/publication duties. However, there is one research related thing that caught my attention two months ago.

The journal paper titled: 'Disnix: A toolset for distributed deployment' that got accepted in April last year is finally going to get published in Volume 79 of 'Science of Computer Programming', unbelievable!

Although it sounds like good news that another paper of mine gets published, the thing that disturbs me is that the publication process took an insanely long time! I wrote the first version of this paper for the WASDeTT-3 workshop that was held in October 2010. So that technically means that I started doing the work for the paper several months prior to that.

In February 2011, I have adapted/extended the workshop paper and submitted its first journal draft. Now, in January 2014 it finally gets published, which means that it took almost 3 years to get published (if you take the workshop into account as well, then it's actually closer to 3,5 years!).

In some academic assessment standards, journal papers have more value than conference papers. Although this journal paper should increase my value as a researcher, it's actually crazy if you think too much about it. The first reason is that I wrote the first version of this paper before I started this blog. Meanwhile, I have already written 54 blog articles, two tech reports, published two papers at conferences, and I finished my PhD dissertation.

The other reason is that peer reviewing and publishing should help the authors and the research discipline in general. To me this does not look like any help. Meanwhile, in the current development version of Disnix some aspects of its architecture have evolved considerably compared to what has been described in the paper, so it is no use for anyone else in the research community anymore.

The only value the paper still provides are the general ideas and the way Disnix manifests itself externally.

Although the paper is not completely valueless, and I'm happy it gets published, it also feels weird that I don't depend on it anymore.

Blog posts


As with my previous annual reflections, I will also publish the top 10 of my most frequently read blog posts:

  1. On Nix and GNU Guix. This is a critical blog post that also ended up first in last year's top 10. I think this blog posts will remain at the top position for the time being, since it attracted an insane amount of visitors.
  2. An alternative explanation of the Nix package manager. My alternative explanation of Nix, which I wrote to clarify things. It was also second in last year's top 10.
  3. Setting up a Hydra build cluster for continuous integration and testing (part 1). Apparently, Hydra and some general principles about continuous integration have attracted quite some vistors. However, the follow up blog posts I wrote about Hydra don't seem to be that interesting to outsiders.
  4. Using Nix while doing development. I wrote this blog post 2 days ago, and it attracted quite some visitors. I have noticed that setting up development environments is an attractive feature for Nix users.
  5. Second computer. This is an old blog post about my good ol' Amiga. It was also in all previous top 10s and I think it will remain like that for the time being. The Amiga rocks!
  6. An evaluation and comparison of GoboLinux. Another blog article that remains popular from the beginning. It's still a pity that GoboLinux has not been updated and sticks to their 014.01 release, which dates from 2008.
  7. Composing FHS-compatible chroot environments with Nix (or deploying Steam in NixOS). This is something I have developed to be able to run Steam in NixOS. It seems to have attracted quite some users, which does not come as a surprise. NixOS users want to play Half-Life!
  8. Software deployment complexity. An old blog post about software deployment complexity in general. Still remains popular.
  9. Deploying iOS applications with the Nix package manager. A blog post that I wrote last year describing how we can use the Nix package manager to build apps for the iPhone/iPad. For a long time the Android variant of this blog post was more popular, but recently this blog article surpassed it. I have no clue why.
  10. Porting software to AmigaOS (unconventional style). People still seem to like one of my craziest experiments.

Conclusion


I already have three more blog posts in draft/planning stages and more ideas that I like to explore, so expect more to come. The remaining thing I'd like to say is:

HAPPY NEW YEAR!!!

Saturday, December 28, 2013

Using Nix while doing development

I have noticed that while doing development work, many outsiders experience the way I work as quite odd and consider it to be inefficient.

The first reason is (probably) because I like the command-line for many tasks and I frequently use a unconventional text editor, which some people don't understand. Second, I also often use Nix (and related Nix utilities) during development.

In this blog post, I'd like to elaborate a bit about the second aspect and discuss why it is actually quite useful to do this.

Obtaining and installing the dependencies of a software project


In all software projects I have been involved with so far, the first thing I typically had to do is installing all its required dependencies. Examples of such dependencies are a compiler/interpreter + runtime environment for a specific programming language (such as GCC, the Java Development Kit, Perl, Python, Node.js etc.), a development IDE (such as Eclipse), and many library packages.

I have experienced that this step is often quite time consuming, since many dependencies have to be downloaded and installed. Moreover, I have frequently encountered situations in which a bunch of special settings have to be configured to make everything work right, which can be quite cumbersome and tedious.

Many developers just simply download and install everything on their machine in an ad-hoc manner and manually perform all the steps they have to do.

Automating the deployment of dependencies of a software project


What I often do instead is to immediately use Nix to automate the process of installing all software project dependencies and configuring all their settings.

To fully automate deployment with Nix, all dependencies have to be provided as Nix packages. First, I investigate if everything I need is in the Nix packages collection. Fortunately, the Nix package collection provides packages for many kinds of programming languages and associated libraries. If anything is missing, I package it myself. Although some things are very obscure to package (such as the Android SDK), most things can be packaged in a straight forward manner.

After packaging all the missing dependencies, I must install them and generate an environment in which all the dependencies can be found. For example, in order to let Java projects find their library dependencies, we must set the CLASSPATH environment variable to refer to directories or JAR files containing our required compiled classes, PERL5LIB to let Perl find its Perl modules, NODE_PATH to let Node.js find its CommonJS modules etc.

The Nix packages collection has a nice utility function called myEnvFun that can be used to automatically do this. This function can be used to compose development environments that automatically set nearly all of its required development settings. The following example Nix expression (disnixenv.nix) can be used to deploy a development environment for Disnix:

with import <nixpkgs> {};

myEnvFun {
  name = "disnix";
  buildInputs = [
    pkgconfig dbus dbus_glib libxml2 libxslt getopt
    autoconf automake libtool dysnomia
  ];
}

The development environment can be deployed by installing it with the Nix package manager:

$ nix-env -f disnixenv.nix -i env-disnix

The above command-line invocation deploys all Disnix's dependencies and generates a shell script (named: load-env-disnix) that launches a shell session in an environment in which all build settings are configured. We can enter this development environment by running:

$ load-env-disnix
env-disnix loaded

disnix:[sander@nixos:~]$

As can be seen in the code fragment above, we have entered an environment in which all its development dependencies are present and configured. For example, the following command-line instruction should work, since libxml2 is declared as a development dependency:

$ xmllint --version
xmllint: using libxml version 20901

We can also unpack the Disnix source tarball and run the following build instructions, which should work without any trouble:

$ ./bootstrap
$ ./configure
$ make

Besides deploying a development environment, we can also discard it if we don't need it anymore by running:

$ nix-env -e env-disnix

Automating common deployment tasks


After deploying a development environment, I'm usually doing all kinds of development tasks until the project reaches a more stable state. Once this is the case, I immediately automate most of its deployment operations in a Nix expression (that is often called: release.nix).

This Nix expression is typically an attribute set following Hydra's (the Nix-based continuous integration server) convention in which each attribute represents a deployment job. Jobs that I typically implement are:

  • Source package. This is a job that helps me to easily redistribute the source code of a project to others. For GNU Autotools based projects this typically involves the: make dist command-line instruction. Besides GNU Autotools, I also generate source packages for other kinds of projects. For example, in one of my Node.js projects (such a NiJS) I produce a TGZ file that can be deployed with the NPM package manager.
  • Binary package jobs that actually build the software project for every target system that I want to support, such as i686-linux, x86_64-linux, and x86_64-darwin.
  • A job that automatically compiles the program manual from e.g. Docbook, if this applicable.
  • A program documentation catalog job. Many programming languages allow developers to write code-level documentation in comments from which a documentation catalog can be generated, e.g. through javadoc, doxygen or JSDuck. I also create a job that takes care of doing that.
  • Unit tests. If my project has a unit test suite, I also create a job executing those, since it is required to deploy most of its development dependencies to run the test suite as well.
  • System integration tests. If I have any system integration tests that can run be on Linux, I try to implement a job using the NixOS test driver to run those. The NixOS test driver also automatically deploys all the required environmental dependencies in a VM, such as a DBMS, web server etc. in such a way that you can run them as a unprivileged user without affecting the host system's configuration.

After writing a release Nix expression, I can use the Nix package manager to perform all the deployment tasks for my software project. For example, I can run the following job that comes with Disnix to build a source tarball:

$ nix-build release.nix -A tarball

Another advantage is that I can use the same expression in combination with Hydra to so that I can continuously build and test it.

Spawning development environments from Nix jobs


Apart from being able to deploy a software project and its dependencies, another nice advantage of having its deployment automated through Nix is that I can use the same Nix jobs to reproduce its build environment. By replacing nix-build with nix-shell all its dependencies are deployed, but instead of building the package itself, a shell session is started in which all its dependencies are configured:

$ nix-shell release.nix -A tarball

The above command-line invocation produces a similar build environment as shown earlier with myEnvFun by just using a Nix expression for an existing job.

Discussion


So far I have explained what I typically do with the Nix package manager during development. So why is this actually useful?

  • Besides deploying systems into production environments, setting up a development environment is a similar problem with similar complexities. For me it is logical that I solve these kind of problems the same way as I do in production environments.
  • Packaging all software project dependencies in Nix ensures reliable and reproducible deployment in production environments as well, since Nix ensures dependency completeness and removes as many side effects as possible. In Nix, we have strong guarantees that a system deployed through Nix in production behaves exactly the same in development and vice-versa.
  • We can also share development environments, which is useful for a variety of reasons. For example, we can reproduce the same development environment on a second machine, or give it to another developer to prevent him spending a lot of time in setting up his development environment.
  • Supporting continuous integration and testing with Hydra comes (almost) for free.
  • With relatively little effort you can also enable distributed deployment of your software project with Disnix and/or NixOps.

Although I have encountered people telling me that it's consuming too much time, you also get a lot for it in return. Moreover, in most common scenarios I don't think the effort of automating the deployment of a development environment is that big if you have experience with it.

As a concluding remark, I know that things like Agile software development and continuous delivery of new versions is something that (nearly) everybody wants these days. To implement continuous delivery/deployment, the first step is to automate the deployment of a software project from the very beginning. That is exactly what I'm doing by executing the steps described in this blog post.

People may also want to read an old blog post that contains recommendations to improve software deployment processes.

Thursday, December 12, 2013

Asynchronous programming with JavaScript (part 2)

A while ago, I wrote a blog post about my asynchronous programming experiences/frustrations with JavaScript. The main message of this blog post is that if we want to run stuff concurrently, we have to implement cooperative multitasking by generating events (such as timeouts or ticks) and then stop the execution so that the events get processed that invoke callbacks to resume execution.

As a consequence of using callbacks, it has become much harder to structure code properly. Without any proper software abstractions, we may end up writing pyramid code that is hard to write, read, adapt and maintain. One of the solutions to alleviate such problems is by using software abstractions implemented by libraries such as async.

Recently, I've been experimenting with several JavaScript libraries/frameworks (such as AngularJS, Selenium JavaScript webdriver, and jQuery) and I went to a Titanium SDK (a cross-platform JavaScript framework for developing apps) developer meetup. I have noticed that yet another software abstraction has become quite popular in these worlds.

Promises


The software abstraction I am referring to is called "promises". To get some understanding on what they are supposed to mean, I will cite the Promises/A proposal, that is generally regarded as its official specification:

A promise represents the eventual value returned from the single completion of an operation.

To me this description looks quite abstract. Fortunately, the proposal also outlines a few practical bits. It turns out that promises are supposed to be encoded as objects having the following characteristics:

  • A promise can be in one of the following three states: unfulfilled, fulfilled, and failed. A promise in an unfulfilled state can either transition to a fullfilled state or failed state. The opposite transitions are not possible -- once a promise is fulfilled or failed it should remain in that state.
  • A promise object has a then member referring to a function with the following signature:
    .then(onFullfilled, onError, onProgress);
    
  • The then function parameters are all optional and refer to functions with the following purposes:

    • onFullfilled: A function that gets invoked once the promise reaches its fulfilled state.
    • onError: A function that gets invoked once the promise reaches its failed state.
    • onProgress: A function that may get invoked while the promise is still unfulfilled. Promises are actually not required to support this function at all.

    If any of these parameters is not a function, then they are ignored.
  • Each then function is supposed to return another promise (which also have a then property), allowing users to "chain" multiple then function invocations together. This is how promises allow someone to structure code more properly and prevent people from writing pyramid code, because chaining can be done at the same indentation level.

Semantic issues


Although some of the promise characteristics listed above may look useful, it has turned out that the Promises/A proposal has a number of weaknesses as well.

First of all, it is just a proposal containing some random notes and not an official specification. Furthermore, it has been written in an informal way and it is incomplete and underspecified. For example, it leaves a few semantic issues open, such as what should happen if a promise enters a failed state, when a particular then function invocation in a chain does not implement an onError callback? As a consequence, this has lead to several implementations having different and incompatible behaviour.

To address some of these issues, a second specification has been developed called the Promises/A+ specification. This specification is not intended to completely replace the Promises/A proposal, but instead it is concentrated mostly on the semantics of the then function (not including the onProgress parameter). Moreover, this specification is also written in a much more formal and concise way in my opinion.

For example, the Promises/A+ specification adds the following details:

  • It specifies the ability to invoke a promise's then function multiple times.
  • Propagation of results and errors in a then function invocation chain, if any of their callback function parameters are not set.

When using Promises/A+ compliant promise objects, error propagation works in a similar way compared to a synchronous try { } catch { } block allowing one to execute a group of instructions and use a separate block at the end to catch any exceptions that may be thrown while executing these.

An example: Using the Selenium webdriver


To demonstrate how promises can be used, I have developed an example case with the JavaScript Selenium webdriver, a web browser automation framework used frequently for testing web applications. This package uses promises to encode query operations.

Our testcase works on a webpage that contains a form allowing someone to enter its first and last name. By default, they contain my first and last name:


The corresponding HTML code looks as follows:

<!DOCTYPE html>

<html>
    <head><title>Selenium test</title></head>
    
    <body>
        <h1>Selenium test</h1>
        
        <p>Please enter your names:</p>
        
        <form action="test.html" method="post">
            <p>
                <label>First name</label>
                <input type="text" name="firstname" value="Sander">
            </p>
            <p>
                <label>Last name</label>
                <input type="text" name="lastname" value="van der Burg">
            </p>
            <button type="submit" id="submit">Submit</button>
        </form>
    </body>
</html>

To automatically retrieve the values of both input fields and to click on the submit button, I have written the following JavaScript code:

var webdriver = require('selenium-webdriver');
var remote = require('selenium-webdriver/remote');
var webdrvr = require('webdrvr');

/* Create Selenium server */
var server = new remote.SeleniumServer(webdrvr.selenium.path, {
    args: webdrvr.args
});

/* Start the server */
server.start();

/* Set up a Chrome driver */
var driver = new webdriver.Builder()
    .usingServer(server.address())
    .withCapabilities(webdriver.Capabilities.chrome())
    .build();

/* Execute some tests */

driver.get('file://'+process.cwd()+'/test.html')
.then(function() {
    return driver.findElement(webdriver.By.name('firstname'));
})
.then(function(firstNameInput) {
    return firstNameInput.getAttribute('value');
})
.then(function(firstName) {
    console.log("First name: "+firstName);
    return driver.findElement(webdriver.By.name('lastname'));
})
.then(function(lastNameInput) {
    return lastNameInput.getAttribute('value');
})
.then(function(lastName) {
    console.log("Last name: "+lastName);
    return driver.findElement(webdriver.By.id('submit'));
})
.then(function(submitButton) {
    return submitButton.click();
}, function(err) {
    console.log("Some error occured: "+err);
});

/* Stop the server */
server.stop();

As may be observed from the code fragment above, besides setting up a test driver that uses Google Chrome, all the test operations are encoded as a chain of then function invocations each returning a promise:
  • First, we instruct the test driver to get our example HTML page containing the form by creating a promise.
  • After the page has been opened, we create and return a promise that queries the HTML input element that is used for filling in a first name.
  • Then we retrieve the value of the first name input field, by creating and returning a promise that fetches the value attribute of the HTML input element.
  • We repeat the previous two steps in a similar way to fetch the last name as well.
  • Finally, we create and return promises that queries the submit button and clicks on it.
  • In the last then function invocation, we define a onError callback that catches any errors that might occur when executing any of the previous steps.

As can be observed, we do not have to write any nested callback blocks that result in horrible pyramid code.

Implementing promises


The Selenium example shows us how asynchronous test code can be better structured using promises. The next thing I was thinking about is how can I use promises to fix structuring issues in my own code?

It turns out that there are many ways to do this. In fact, the Promise/A and Promise/A+ specifications are just merely interface specifications and leave it up to developers to properly implement them. For example, it can be done completely manually, but there are also several libraries available providing abstractions for that, such as Q, RSVP, and when.

As an experiment, I have restructured the following pyramid example code fragment from my previous blog post (that I originally used to restructure with async) into a better structured code fragment with promises:

var fs = require('fs');
var path = require('path');

fs.mkdir("out", 0755, function(err) {
    if(err) throw err;
    
    fs.mkdir(path.join("out, "test"), 0755, function(err) {
        if (err) throw err;        
        var filename = path.join("out", "test", "hello.txt");

        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) throw err;
                    
            fs.readFile(filename, function(err, data) {
                if(err) throw err;
                
                if(data == "Hello world!")
                    process.stderr.write("File is correct!\n");
                else
                    process.stderr.write("File is incorrect!\n");
            });
        });
    });
});

I have picked the RSVP library to implement promises, since it claims to be lightweight and fully Promises/A+ compliant. The restructured code looks as follows:

var fs = require('fs');
var path = require('path');
var Promise = require('rsvp').Promise;

/* Promise object definitions */

var mkdir = function(dirname) {
    return new Promise(function(resolve, reject) {
        fs.mkdir(dirname, 0755, function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var writeHelloTxt = function(filename) {
    return new Promise(function(resolve, reject) {
        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var readHelloTxt = function(filename) {
    return new Promise(function(resolve, reject) {
        fs.readFile(filename, function(err, data) {
            if(err) reject(err);
            else resolve(data);
        });
    });
};

/* Promise execution chain */

var filename = path.join("out", "test", "hello.txt");

mkdir(path.join("out"))
.then(function() {
    return mkdir(path.join("out", "test"));
})
.then(function() {
    return writeHelloTxt(filename);
})
.then(function() {
    return readHelloTxt(filename);
})
.then(function(data) {
    if(data == "Hello world!")
        process.stderr.write("File is correct!\n");
    else
        process.stderr.write("File is incorrect!\n");
}, function(err) {
    console.log("An error occured: "+err);
});

The revised code fragment above is structured as follows:

  • The overall structure consists of two parts: First we define a number of promises and then we execute them by composing a then function invocation chain.
  • I have encapsulated most of the operations from the previous code fragment into reusable promises: one makes a directory, one writes a text file, one reads the text file.
  • A promise is created by instantiating the rsvp.Promise prototype. The constructor function always takes the same function as a parameter taking resolve and reject as parameters. resolve is a callback that can be used to notify a caller that it has reached its fulfilled state. The reject parameter notifies a caller that it has reached its failed state. Both of these functions can take a value as parameter that can be passed to a then function invocation.
  • To be able to parametrize these promises, I have encapsulated them in functions
  • In the remainder of the code, I instantiate the promises and execute them in a then function invocation chain. As can be observed, we have no pyramid code.
  • However, I observed that our refactored code has a big drawback as well -- it has grown significantly in size which can be considered a deterioration compared to the pyramid code fragment.

Discussion


So far, I have given an explanation of what promises are supposed to mean, how they can be used and how they can be created. We have seen that code structuring issues are "solved" by the fact that then function invocations can be chained.

In my previous blog post, I have explored the async library as a means to solve the same kind of issues, which make me wonder which approach is best? To me, the approach of chaining then function invocations looks quite similar to async's waterfall pattern. However, I find the promise approach slightly better, because it uses return values instead of callbacks preventing us to accidentally reach limbo state if we forget to call them.

However, I think async's waterfall pattern has a lower usage barrier because we do not have to capture all the steps we want into promise objects resulting in less overhead of code.

I think that using promises to structure code works best for things that can be encoded as reusable objects, such as those used to implement a system having a Model-View-Controller (MVC) pattern, that e.g. pass reusable objects from the model to the controller and view. In situations in which we just have to execute a collection of ad-hoc operations (such as those in my example) it simply produces a lot of extra overhead.

Finally, async's waterfall pattern as well as chaining then function invocations work well for things that consist of a fixed number of steps that have to be executed sequentially. However, it may also be desirable to execute steps in parallel, or to use asynchronous for and while loops. The promises specifications do not provide any solutions to cope with these kind of code structuring issues.

Some other thoughts


After doing these experiments and writing this blog post, I still have some remaining thoughts:

  • I have noticed that promises are considered the "the next great paradigm" by some people. This actually makes me wonder what the "previous great paradigm" was. Moreover, I think promises are only useful in certain cases, but definitely not for all JavaScript's asynchronous programming problems.
  • Another interesting observation I did is not related to JavaScript or asynchronous programming in general. I have observed that writing specifications, which purpose is to specify what kind of requirements a solution should fulfill, is a skill that is not present with quite a few people doing software engineering. It's not really surprising to me that the Promise/A proposal leads to so much issues. Fortunately, the Promises/A+ authors demonstrate how to properly write a specification (plus that they are aware of the fact that specifications always have limitations and issues).
  • I'm still surprised how many solutions are being developed for a problem that is non-existent in other programming environments. Actually, I consider these software abstractions dealing with asynchronous programming "workarounds", not solutions. Because none of them is ideal there are many of them (the joyent Node wiki currently lists 79 libraries and I know there are more of them), I expect many more to come.
  • Personally, I consider the best solution a runtime environment and/or programming language that has facilities to cope with concurrency. It is really surprising to me that nobody is working on getting that done. UPDATE: There is a bit of light at the end of the tunnel according to Zef's blog titled: "Callback-Free Harmonious Node.js" post that I just read.
UPDATE: I have written a follow up blog post in which I take a look at asynchronous programming in a more fundamental way. I also derived my own abstraction library from my observations.

Thursday, November 7, 2013

Emulating Amiga display modes

A while ago, I have written a blog post about my IFF file format experiments in which I have developed a collection of libraries and tools capable of parsing and displaying two IFF file format applications. Some time before that, I have developed a Nix function capable of building software for AmigaOS allowing me to easily backport these experimental software packages to AmigaOS.

The development of these experimental packages were dormant for quite some time, but recently I was able to find some time to do some improvements. This time I have updated the Amiga video emulation library to support a few new features and to more accurately emulate an Amiga display.

Differences between Amiga and PC displays


So why is emulation of Amiga video modes necessary? Emulation of Amiga graphics hardware is required because pixels are encoded differently than "modern" PC hardware, resolutions have different semantics, and Amiga hardware has a few special screen modes to "squeeze" more colors out of the amount of available color registers.

Pixels


Encoding pixels on modern hardware is not so complicated these days. Modern hardware allows us to address each individual pixel's color value using 4 bytes per pixel in which one byte represents the red color intensity, one byte the blue color intensity and one byte the green color intensity. The fourth byte is often unused, or serves as the alpha component setting the transparency of a pixel in multi-layered images.

However, older PC hardware, such as those used during the DOS era, were often not powerful enough to address every pixel's color value individually, mainly due to memory constraints. Many classic DOS games used a 320x200 video resolution with a color palette consisting of 256 color values. Each pixel was encoded as a byte referring to an index value of a palette. This encoding is also known as chunky graphics.

System resources in the Amiga era were even more scarce, especially during its launch in 1985. The original graphics chipset was only capable of storing 32 color values in a palette, although there were a few special screen modes capable of squeezing more colors out of the 32 configurable ones. It also used a different way of encoding pixels, probably to make storage of graphics surfaces as memory efficient as possible.

In Amiga chipsets pixels are encoded as bitplanes rather than bytes. When using bitplane encoding, an image is stored multiple times in memory. In every image occurence, a bit represents a pixel. In the first occurence, a bit is the least significant bit of an index a value. In the last occurence a bit is the most significant bit of an index value. By adding all the bits together we can determine the index value of the palette to determine a pixel's color value. For example, if we would encode an image using 16 colors, then we need 4 bitplane surfaces.

To be able to display an Amiga image on a PC we have to convert the bitplane format to either chunky graphics or RGB graphics. Likewise, to be able to display a PC image on an Amiga, we have to convert the pixel surface to bitplane format.

Palette


As explained earlier, due to memory constraints, Amiga graphics use a palette of 32 configurable color values (or 256 colors when using the newer AGA chipset is used). Each pixel refers to an index of a palette instead of a color value of its own.

The original chipset color values are stored in 16-bit color registers in which every color component consists of 4 bits (4 bits are unused). The newer AGA chipset as well as VGA graphics use 8 bits per color component.


Furthermore, the Amiga video chips have special screen modes to squeeze more colors out of the amount of color registers available. The Extra Half Brite (EHB) screen mode is capable of displaying 64 colors out of a predefined 32, in which the last 32 colors values have half of the intensity of the first 32 colors. The above screenshot is an example picture included with Deluxe Paint V using the EHB screenmode. A closer look at the floor in the picture look may reveal to you that an EHB palette is used.

The Hold-and-Modify (HAM) screen mode is used to modify a color component of the previous adjacent pixel or to pick a new color from the given palette. This screen mode makes it possible to use all possible color values (4096) in one screen with some loss of image quality.


The above screenshot is an example image included with Deluxe Paint V using HAM screenmode to utilise many more colors than the amount available color registers. A closer look at the image very may reveal that it has some quality loss because of the limitations of HAM compression.

To be able to properly display an image on a PC we need to convert 12-bit color values to 32-bit color values. Moreover, we also have to calculate each pixel's color value when HAM screen modes are used and convert the image to true color graphics, since a picture using HAM mode may use more than 256 colors.

Resolutions


Another difference between Amiga displays and "modern" displays are the display resolutions. On PCs, resolutions refer to the amount of pixels per scanline and the amount of scanlines per screen.

On the Amiga, resolutions only refer to the amount of pixels per scanline and this amount is often fixed. For example most displays support 320 lowres pixels per scanline, although this value can be slightly increased by utilising the overscan region. A high resolution screen has twice the amount of pixels per scanline compared to a low resolution screen. A super hires screen has double the amount of pixels per scanline compared to a high resolution screen. Moreover, a low resolution pixel is twice a wide as a high resolution pixel and so on.

Vertically, there are only two options. In principle, there are a fixed amount of scanlines on a display. Typically, NTSC displays support 200 scanlines and PAL displays support 256 scanlines. This amount can be slightly increased by utilising the overscan region of a display.

The amount of scanlines can be doubled, using a so-called interlace screen mode. However, interlace screen modes have a big drawback -- they draw the odd scanlines in one frame and the even ones in another. On displays with a short after glow, flickering may be observed.

Because of these differences, we may observe odd results in some cases when we convert an Amiga pixel surface to a chunky/RGB pixel surface. For example, a non-interlaced high resolution image looks twice as wide on a PC display than on an Amiga display, as can be seen in the left picture above, which contains a pixel-by-pixel conversion of a the Workbench screenshot.

To give it the same look as an Amiga display, we must correct its aspect ratio by doubling the amount of scanlines on the PC display, which is done in the right screenshot. People who have used the Amiga Workbench will immediately notice that this image is much closer to what can be seen on a real Amiga.

A library for performing conversions of Amiga display surfaces


While writing my previous blog post, I have developed a library (libamivideo) capable of automatically converting Amiga bitplane surfaces to chunky/RGB pixel surface and vice versa. In the latest release, I have added the following features and improvements:

  • Support for super hires video modes
  • Correct aspect ratio support
  • A better defined API

The library can be obtained from my GitHub page. I have also updated the SDL-based ILBM viewer (SDL_ILBM) as well as the native AmigaOS ILBM viewer (amiilbm) to use the new library implementation.

Friday, September 13, 2013

Composing FHS-compatible chroot environments with Nix (or deploying Steam in NixOS)

As described in a blog post from a while ago, NixOS deviates on certain aspects from the Filesystem Hierarchy Standard (FHS) for very good reasons. For example, NixOS lacks standard directories such as /usr/bin and /usr/lib that conventional Linux distributions have. The most important reason that Nix/NixOS deviates from the FHS is to provide reliable and reproducible deployment of packages.

The filesystem organization that Nix uses, in which every package is stored in a separate folder in a so called "Nix store", is typically not a problem to get software packaged and deployed in source form.

Packages in Nix are deployed from build recipes called Nix expressions that typically invoke functions implementing abstractions ensuring that the specified dependencies can be found in a convenient manner.

For example, when providing Java or Perl dependencies, the generic builder may automatically set some required environment variables, such CLASSPATH or PERL5LIB containing Java or Perl dependencies. Moreover, a number of build utilities have been patched (such as ld) to automatically add shared libraries to an ELF executable's RPATH allowing it to find the libraries that it needs (you could see this as as static linking of shared libraries to an executable).

However, deployment of third party binary packages is a bit complicated and inconvenient in NixOS for the following reasons:

  • Nearly all software must be deployed by the Nix package manager, since nearly all of the system's components reside in the Nix store.
  • Third party binaries frequently require shared libraries and other kinds dependencies to run. However, in most cases these dependencies cannot be found, because executables assume that they reside in standard FHS folders such as /usr/lib, which do not exist in NixOS.
  • Executables using dynamic libraries cannot find the dynamic linker (e.g. /lib/ld-linux.so.2), because the corresponding file does not exist in NixOS.

As a solution, we often use PatchELF to change the path to the dynamic linker and modify the RPATH of a binary allowing it to find the libraries that it needs. We may also have to wrap executables in shell scripts to set certain environment variables, such as LD_LIBRARY_PATH.

Some applications that are packaged in Nixpkgs this way are Adobe Reader, Belastingaangifte (Dutch Tax administration) and the Android SDK, as I have described in two earlier blog posts.

However, I have recently developed something that can solve this problem in a different way.

Composing FHS-compatible chroot environments


In my previous blog post, I have described how Nix user environments work, which are symlink trees that blend the contents of a collection of packages into a single symlink tree. User environments are primarily used to allow end-users to conveniently access packages installed by Nix from a single location.

We can also use the user environment mechanism in combination with setting up a chroot environment to simulate an FHS-compatible filesystem. This allows us to run an unpatched binary that works out of the box, if all required dependencies of the package are (correctly) specified.

I have created a Nix function called buildFHSChrootEnv {} which makes it possible to compose such environments. This function roughly does the following:

  • It invokes buildEnv {} to compose a so-called "/usr user environment". In this user environment, we synthesize the contents of all our specified packages into a single Nix profile. This profile contains all static parts of all packages and is supposed to simulate both the /usr directory as well as the static folders in the root folder of the chroot environment, such as /bin and /lib.
  • Besides the dependencies of the binary that we want to run, we also need a collection of base packages, such as the bash shell, GNU Coreutils, glibc, and the shadow password suite, since we also have to be able to run a basic Linux system and do basic shell activities inside the chroot environment. These packages are always part of the /usr profile.
  • It generates a global bash profile setting the PS1 environment variable to configure the shell prompt and other miscellaneous settings that a program may need in order to run.
  • It generates a collection of scripts taking care of setting up (or destroying) the dynamic parts of the chroot environment, since Nix packages and user environment are static and can never change after they have been built.

Setting up the dynamic parts of the chroot environment


Merely composing a /usr profile is not enough to get a working chroot environment because they only contain the static parts of a system, such as executables, libraries, and data files. We also have to set up a few dynamic parts. A collection of configuration scripts bundled with the generated /usr profile can be used for this purpose:

  • init-*-chrootenv: Initializes most dynamic parts of the chroot environment inside a directory with write permissions (i.e. /run/chrootenv/<name>).

    First, it creates symlinks to the static folders of the /usr profile and the /usr symlink itself. Then the script ensures that the Name Service Switch (NSS) facilities work by symlinking /etc/{passwd,group,shadow,hosts,resolv.conf,nsswitch.conf) of the host system into the chroot environment. Then some other files from the host are symlinked, such as PAM configuration files and font configuration stuff. Finally, a few empty state folders are created, such as: /var, /run, /tmp.

    The initialization script has to be executed only once.
  • mount-*-chrootenv: Bind mounts a number of directories of the host system into the chroot folder, so that files from the host system can be used while we have entered the chroot environment.

    Currently, we bind mount the Nix store (/nix/store), directories used by the kernel: /dev, /dev/pts, /dev/shm, /proc, /sys, users' home directories: /home, a few state directories: /var, /run, and the host's /etc directory containing shared configuration files, such as the ones related to NSS and PAM.

    This script has to be executed every time we have booted into NixOS.
  • load-*-chrootenv: Enters the chroot environment and launches a shell. The shell inherits a few environment variables from the host environment, such as DISPLAY allowing someone to run GUI applications and TERM to make certain fancy console applications to work properly.
  • umount-*-chrootenv: Unmounts the bind mounts.
  • destroy-*-chrootenv: Removes the writable chrootenv directory.

tl;dr - Getting Steam to work under NixOS


If you're still awake and still interested in reading this blog post, I will reveal why I have done all this work, which is probably much more exciting to read. In fact, this function's primary use case is to get Steam working in NixOS. Steam is a very special piece of software compared to other binary-only packages, because:

  • Steam is also a deployment tool (like Nix) capable of automatically downloading, installing and upgrading software packages. In Steam's case, these software packages are mostly games.
  • Software packages installed by Steam have the same problems as the deployment of "ordinary" binary packages in NixOS -- they don't work out of the box, because they cannot find the dynamic linker and their dependencies, such as shared libraries.
  • Of course, we can patch the executables installed by Steam, but Steam wants control over them and does not like patched versions of software. Furthermore, patching the packages makes it impossible for Steam to update them.
  • It is also a bit impractical to patch the executables every time they are updated, because I have seen that the games I have are updated quite frequently.

By using the buildFHSChrootEnv {} Nix function getting Steam (including the games installed by it) to run in NixOS is a piece of cake. First I had to package Steam itself, which was quite easy:

{stdenv, fetchurl, dpkg}:

stdenv.mkDerivation {
  name = "steam-1.0.0.42";
  src = fetchurl {
    url = http://repo.steampowered.com/steam/archive/precise/steam-launcher_1.0.0.42_all.deb;
    sha256 = "1jyvk0h1z78sdpvl4hs1kdvr6z2kwamf09vjgjx1f6j04kgqrfbw";
  };
  buildInputs = [ dpkg ];
  unpackPhase = "true";
  installPhase = ''
    mkdir -p $out
    dpkg -x $src $out
    cp -av $out/usr/* $out
    rm -Rf $out/usr
  '';
}

I just had to unpack the Steam Debian package and move its contents into the Nix store. I intentionally did not patch or wrap anything.

Then I had to create a Nix expression that composes a FHS-compatible chroot environment with Steam's dependencies and runtime settings (such as OpenGL driver settings):

{ buildFHSChrootEnv, steam
, xterm, libX11, zenity, python, mesa, xdg_utils, dbus_tools, alsaLib
}:

buildFHSChrootEnv {
  name = "steam";
  pkgs = [ steam xterm libX11 zenity python mesa xdg_utils dbus_tools alsaLib ];
  profile = ''
    export LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib:/lib
    export FONTCONFIG_FILE=/etc/fonts/fonts.conf
  '';
}

After becoming the super user, I can install the above Nix expression providing the /usr environment profile and configuration scripts, by running:

$ su
# nix-env -f '<nixpkgs>' -iA steamChrootEnv

Now the chroot environment configuration scripts have been added to the system-wide Nix profile. I can initialize the chroot environment folder by running:

# init-steam-chrootenv

I can do the bind mounts by running:

# mount-steam-chrootenv

And then enter the chroot environment by running:

# load-steam-chrootenv

Now I have entered the chroot environment in which I can run the unpatched Steam executable. However, I should not run Steam as a root user. Therefore, I have to become "myself" again:

# su -s /bin/bash sander

And now I can finally run Steam, install my favourite games and play them:

$ steam


As you may see in the screenshot above, it works quite well on my machine. Moreover, to give you some proof that I'm not lying, I have also opened a terminal in the background showing you NixOS' root folder that does not contain most of the standard FHS directories.

So far, I have tried Half Life (and extensions), Half Life 2 (and extensions), Portal and Counter Strike and they all seem to work without any trouble. Isn't that AWESOME??? :P

Finally, if you need to discard the chroot environment (which should not be necessary IMHO :P), you could run:

# umount-steam-chrootenv
# destroy-steam-chrootenv

Discussion


In this blog post, I have described the buildFHSChrootEnv {} Nix function that composes FHS-compatible chroot environments in NixOS, allowing me to run Steam and other third party binary packages without patching them.

Moreover, we still have control over the dependencies in the chroot environment, because apart from a number of common base packages, these chroot environments are composed from Nix expressions that provide exactly the dependencies that are specified. Undeclared dependencies cannot influence a binary package running in the chroot environment.

However, this approach also has a big drawback. In order to chroot, we need super-user privileges making it hard to properly support unprivileged user installations of such packages. Most "ordinary" Nix packages can be deployed by unprivileged users without any trouble. Therefore, I would not recommend this approach over patching executables with PatchELF, unless there is no other way, such as in Steam's case.

Acknowledgements


This is not the first attempt to get Steam packaged in Nixpkgs. In my attempt, I have tried to use the chroot approach which turns out to work. I'd like to thank Evgeny Egorochkin, Aszlig Neusepoff and Carles Pagès for their earlier efforts.

Availability


The buildFHSChrootEnv {} function as well as the Steam components are part of Nixpkgs.

Monday, September 9, 2013

Managing user environments with Nix

For some reason, I'm having quite some discussions about Nix's user environment mechanism lately. For example, in my last two talks I have spent a considerable amount of time answering questions that were related to it.

The user environment mechanism is an important key feature of Nix, which primary purpose is to allow someone to conveniently launch programs installed by the Nix package manager, but it has a few additional use cases as well, such as determining which packages are in use (and which are not) in order to safely garbage collect obsolete packages.

I have only briefly described them in some of my blog posts, but I have never covered in detail how they work on my blog. Because of the amount of questions I have received lately, I think it would be a good idea to write about it.

The Nix store


As described in my exhaustive explanation blog post on Nix (and the original explanation recipe), every Nix package is stored in an isolated directory in a so called "Nix store". In the Nix store every package has a special filename that may look like this:

/nix/store/spy13ngvs1fyj82jw2w3nwczmdgcp3ck-firefox-23.0.1

The first part of the basename (spy13ngvs1fyj82jw2w3nwczmdgcp3ck) is a cryptographic hash derived from all build inputs used to build Firefox. For example, if we would build Firefox with a different compiler or for a different architecture, the generated hash code will be different, which makes it possible to safely store multiple variants of packages next to each other, because none of them will have the same name.

Although this storing convention looks a bit odd, it provides a number of interesting benefits. Since Nix executes builds inside environments in which as many side effects are removed, we can use the hash codes to uniquely identify package variants regardless on what machine they have been built.

They also provide better reproducibility, since these paths to Nix packages including their hash codes must always be explicitly specified, in order to allow a package build to find its dependencies. The advantage of this is that unspecified dependencies cannot accidentally allow a build to succeed, limiting reproducibility guarantees.

More details on the benefits can be found in my two explanation recipes. However, storing packages this way also causes a major inconvenience for end-users. To launch a particular program, the user must always provide the full path to the executable that resides inside the Nix store, which is quite impractical due to the fact that every Nix package has a hash code in it.

Nix profiles


As a means for users to conveniently launch software, the Nix package manager provides the concept of Nix profiles, which are user environments exposing a generation of installed packages to a user. Software packages can be automatically installed in Nix profiles by using the nix-env command-line utility.

To install software, such as Info ZIP's zip and unzip utilities, we can run:

$ nix-env -i zip unzip
installing `zip-3.0'
installing `unzip-6.0'
building path(s) `/nix/store/1zhgva9cmfz0wam9m7ibgaih2m3cic6l-user-environment'
created 33 symlinks in user environment

Allowing us to run any of the installed tools without specifying any paths with hash codes, e.g.:

$ zip --version
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

One of the things the above nix-env command-line invocation does is automatically building the requested packages (or downloading its substitutes) and composing a user environment, which is a symlink tree that synthesizes the contents of the currently installed packages (zip and unzip) into one single component. A user environment is also a package residing in the Nix store, as shown in the following figure:


Besides creating a user environment that exposes all installed packages from a single location, a few additional symlink indirections are used. Each time nix-env performs a deployment activity, a generation symlink is generated (called profile-1 in the figure above). There is also a default symlink (profile) pointing to the generation symlink that is currently active. That symlink is also referenced from the user's home directory through ~/.nix-profile. By adding the latter symlink to the user's PATH, e.g.:

export PATH=~/.nix-profile/bin:$PATH

Packages installed through nix-env can be found automatically through a few layers of symlink indirections.

When we perform another deployment activity with nix-env, such as removing zip:

$ nix-env -e zip
uninstalling `zip-3.0'

Then a new symlink tree is composed containing the then currently installed packages (in this case only unzip), a new generation symlink is generated (profile-2), and the default symlink is updated to point to the new generation. Flipping symlinks is actually an atomic operation on UNIX. As a consequence, we can perform upgrades atomically, because there is never a situation in which we refer to packages belonging to the old and new user environment at the same time.


As may be observed in the picture above, the old generation user environment is still present allowing us to do rollbacks as well. For example, if we regret uninstalling zip, we can switch back to the previous generation by running:

$ nix-env --rollback

Then the default symlink's reference is updated to point to the previous generation symlink that provides the zip utility. Again, this is an atomic operation.

Apart from exposing packages installed by users, the directory containing profile generations also serves a second goal; it is used to determine which packages are in use and which are not.

The generations of the Nix profiles of all users of the system are treated as so called garbage collector roots -- any package that is installed in a profile generation and any of its dependencies are considered packages that are live (a.k.a. in use) and should not be garbage collected. Furthermore, Nix also treats open files and running processes as garbage collector roots. Therefore, it is safe to run the Nix garbage collector at any time.

We can also remove older Nix profile generations if we don't need them anymore:

$ nix-env --delete-generations old

The above command only removes the older generation symlinks, but not the packages that have become obsolete. To actually remove packages from the system that are no longer in use, we must explicitly run the garbage collector:

$ nix-collect-garbage

So far, I have shown examples that work on a per-user profile. In multi-user Nix installations, such as in NixOS, every user has a personal Nix profile (residing in /nix/var/nix/profiles/per-user) and there is also a system-wide default Nix profile (residing in /nix/var/nix/profiles/default) containing packages that are exposed to all users of a system.

NixOS system profiles


Apart from individual packages, NixOS -- the Linux distribution built around the Nix package manager, also uses Nix profiles to manage generations of system configurations.

As described in my earlier blog post about NixOS, a complete system configuration is deployed from a single declarative specification, including all system services and their configurations, the desktop, and user packages. Each time we deploy a system configuration, e.g. by running:

$ nixos-rebuild switch

a Nix profile is built which consists of (mostly) a symlink tree containing all packages and configuration files that constitutes all static parts of a system, including the Linux kernel and the kernel modules. This profile is called a system profile and offers the same benefits as ordinary package profiles, such as the fact that we can store multiple generations next to each other and perform atomic upgrades and rollbacks.

There is a subtle difference with NixOS deployment compared to ordinary package deployment. Merely flipping symlinks is not enough to get a new NixOS configuration deployed. We also have to perform a collection of imperative deployment steps, such as setting up dynamic directories, such as /var, starting and stopping system services, and updating the GRUB bootloader configuration. This is done by an activation script (/run/current-system/bin/switch-to-configuration) part of the NixOS system profile, that is executed each time a new version of a NixOS configuration is deployed.


As can be seen in the screenshot above, once a new NixOS system configuration has been deployed, the GRUB bootloader can be used to boot into any previous NixOS configuration that has not been garbage collected yet.

Generating user environments from Nix expressions


In addition to using nix-env, user environments can also be generated from Nix expression by using the buildEnv {} function. The following Nix expression produces the same user environment (shown earlier) containing zip and unzip:

with import <nixpkgs> {};

buildEnv {
  name = "zipprofile";
  paths = [ zip unzip ];
}

This expression can be built by running:

$ nix-build zipprofile.nix

Conclusion


In this blog post, I have explained how the Nix user environment mechanism works. Essentially user environments are symlink trees that blend all currently installed packages into a single component so that they can be referenced from a single location. Another symlink layer manages generations of user environments. We can atomically switch between these generations by flipping symlinks. By adding the latter symlink to a user's PATH, packages can be automatically found without providing any absolute paths with hash codes in them that are difficult to remember.

Besides exposing packages in a user's PATH, Nix profiles are also used in NixOS to automatically find other resources, such as manual pages (through MANPATH) and KDE desktop menu items (through XDG_DATA_DIRS).

More information on Nix profiles can be found in Eelco Dolstra's PhD thesis which can be obtained from his publications page. Additionally, my PhD thesis also covers Nix profiles in its background chapter. Furthermore, there is a chapter that extends their concepts to distributed systems. This chapter is loosely based on our HotSWUp 2008 paper titled: "Atomic Upgrading of Distributed Systems" which can be obtained from my publications page.

Saturday, August 17, 2013

Improving the testability of the Nix iOS build facilities

In the previous blog post, I have described some new features of the Nix Android build environment. Apart from Android, the Nix iOS build environment has a few improvements as well, which are mostly related to testing.

Simulating multiple iOS versions


In the previous blog post about the Nix iOS build function, I have described how to install various iPhone simulator SDK versions by picking the preferences option from the 'Xcode' menu bar and selecting the 'Downloads' tab in the window, as shown in the screenshot below:


However, I did not implement any facilities in the simulator function that take these SDK versions into account yet.

Picking a specific SDK version can be done quite easily, by providing the -currentSDKRoot parameter to the iPhone simulator, such as:

$ "iPhone Simulator" -SimulateApplication build/HelloWorld \
  -SimulateDevice 'iPhone' -currentSDKRoot \
  "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform\
/Developer/SDKs/iPhoneSimulator6.1.sdk"

In the above case, we have configured the simulator to use the iPhone simulator 6.1 SDK. By changing this parameter, we can use a different version, such as version 5.1.

I have encapsulated the above command-line invocation in the xccodeenv.simulateApp {} Nix function and the paths to the iPhone simulator SDKs in the xcodewrapper. The SDK version can be configured by providing the sdkVersion parameter (which defaults to 6.1):

{xcodeenv, helloworld, device}:

xcodeenv.simulateApp {
  name = "HelloWorld";
  app = helloworld;
  inherit device;
  sdkVersion = "6.1";
}

This parameter makes it possible to spawn a simulator instance running a specific iOS version.

The described facilities earlier are only for simulating apps. To build an app for a specific iOS revision, the iOS target version must be changed inside the Xcode project settings.

Building signed Apps for release


Apart from the simulator, we may also want to deploy apps to real devices, such as an iPhone, iPad or iPod, either directly or through the App store. In order to do that, we require permission from Apple by signing the apps that we have just built.

As described in the previous blog post, we require a certificate that contains a developer's or company's identity and a mobile provisioning file describing which (groups of) apps of a certain company/developer can be run of which (groups of) devices. These files must be obtained through Apple's Dev Center.

Because of these restrictions, it's hard to test the trivial "Hello World" example case I have developed in the previous blog post on a real device, since it contains a dummy app and company name.

To alleviate these problems, I have created a script that "renames" the example app into a different app, so that an existing developer or company certificate and a mobile provisioning profile of a different app can be used.

By setting the rename parameter to true when calling the composition expression of the example case, we can automatically generate jobs building IPAs and xcarchives for the "renamed" app:

import ./nix-xcodeenvtests/deployment {
  rename = true;
  newName = "MyVeryCoolApp";
  newId = "coolapp";
  newCompanyName = "My Cool Company";
  newDomain = "com.coolcompany";
  ipaCertificateFile = ./adhoc.p12;
  ipaCodeSignIdentity = "iPhone Distribution: My Cool Company";
  ipaCertificatePassword = "";
  ipaProvisioningProfile = ./adhoc.mobileprovision;
  xcArchiveCertificateFile = ./distribution.p12;
  xcArchiveCodeSignIdentity = "iPhone Distribution: My Cool Company";
  xcArchiveCertificatePassword = "";
  xcArchiveProvisioningProfile = ./distribution.mobileprovision;
}

In the above expression, we changed the name of the app into: "MyVeryCoolApp" and the company name into: "My Cool Company". Moreover, we provide a certificate and mobile provisioning file for the IPA job (which can be deployed to a device directly) and the xcarchive job (which can be used to deploy to the App store).

By running the following command-line instruction:

$ nix-build -A renamedPkgs.testapp_ipa

We can build an IPA allowing us to deploy the test app on a real device. Of course, to do this yourself, the company name and app names should correspond to the ones defined in your certificate and mobile provisioning profile.

Moreover, as with the Android environment in the previous blog post, we can also use Hydra to provide parameters to the composition expression and to retrieve all the resulting artifacts:


Conclusion


In this blog post, I have described some improvements to the Nix iOS facilities. We are now also capable of configuring the SDK versions for the iPhone simulator, and we can build the trivial testcase to run on a real device in addition to the simulator. This is useful for testing the Nix iOS build function.

Thursday, August 8, 2013

Some improvements to the Nix Android build environment

Some time ago, I have packaged the Android SDK and some of its plugins in Nix, which was quite a challenge. Moreover, I have developed two Nix functions that can be used to automatically build Android apps and spawn emulator instances.

Meanwhile, I have implemented some major updates since the last blog post. Besides upgrading to a newer Android SDK version, we now also support a bunch of new interesting features.

Android SDK packaging updates


Currently, I have packaged the 22.05 version of the Android SDK (which is the latest version at the time writing this blog post). To package it, I have followed the same recipe as described in the previous blog post, with some additions:

  • Newer versions of the Android SDK include x86-64 versions of the emulator executables. I had to patch/wrap these executables to allow them to find the x86-64 libraries they depend on.
  • The newer Android SDK has a new sub package called build-tools. Apparently, this sub package contains some tools that were formerly part of platform-tools, such as debugging libraries.
  • The GUI front-end of the android utility did not work in the last blog post, which I have fixed by wrapping it, so that it can find its dependencies, such as GTK+.
  • There are also x86 and MIPS application binary interface (ABI) system images, but they reside in external package repositories. I have adapted the Nix expression generator to include these as well.
  • I have adapted the build and emulation Nix functions to take both API-level and ABI versions into account
  • I have implemented some facilities to improve the performance of the emulator instances

Packaging build-tools


The build-tools package contains various tools and libraries that used to belong to platform-tools. It turns out that some of these files, such as the debugging libraries, are API-level specific. The purpose of the build-tools package is to provide API-level specific build tools. Currently, the Android SDK supports two versions: 17 and 18.01 that can be installed next to each other.

I have packaged the 18.01 version in a straight forward manner. The only thing I needed to do is inspecting the dependencies of the executables to the paths to these libraries to the executables' RPATH.

For example, I did this to patch aapt:

patchelf --set-rpath ${stdenv.gcc.libc}/lib:${stdenv.gcc.gcc}/lib:${stdenv.zlib}/lib aapt

Every version of the build-tools package is symlinked into the Android SDK basedir as follows: build-tools/android-<version>

Fixing the Android GUI front-end


If the android tool is called without any parameters, it should show a GUI that allows somebody to download plugins or to configure AVDs. However, the GUI did not work so far. Invoking 'android' gave me the following error message:

$ android
Exception in thread "main" java.lang.UnsatisfiedLinkError:
no swt-pi-gtk-3550 or swt-pi-gtk in swt.library.path,
java.library.path or the jar file
        at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
        at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
        at org.eclipse.swt.internal.gtk.OS.(Unknown Source)
        at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
        at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
        at org.eclipse.swt.widgets.Display.(Unknown Source)
        at com.android.sdkmanager.Main.showSdkManagerWindow(Main.java:346)
        at com.android.sdkmanager.Main.doAction(Main.java:334)
        at com.android.sdkmanager.Main.run(Main.java:120)
        at com.android.sdkmanager.Main.main(Main.java:103)

First, I thought that some file belonging to SWT was missing, but after manually digging through all Android files I couldn't find it and I gave up. However, after using strace, I discovered that it's actually trying to load the GTK+ library and some other libraries:

$ strace -f android
...
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk.so", 0x7fe4e0cffb20) = -1 ENOENT (No such file or directory)
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk-3550.so", {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] stat("/tmp/swtlib-64/libswt-pi-gtk-3550.so", {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] open("/tmp/swtlib-64/libswt-pi-gtk-3550.so", O_RDONLY|O_CLOEXEC) = 26
[pid  8169] read(26, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\234\3\0\0\0\0\0"..., 832) = 832
[pid  8169] fstat(26, {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid  8169] mmap(NULL, 1503944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 26, 0) = 0x7fe4b5d7e000
[pid  8169] mprotect(0x7fe4b5dea000, 1044480, PROT_NONE) = 0
[pid  8169] mmap(0x7fe4b5ee9000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 26, 0x6b000) = 0x7fe4b5ee9000
[pid  8169] mmap(0x7fe4b5eec000, 4808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe4b5eec000
[pid  8169] close(26)                   = 0
[pid  8169] open("/run/opengl-driver/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  8169] open("/run/opengl-driver-32/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  8169] open("/nix/store/zm4bhsm8lprkzvrjgqr0klfkvr21als4-glibc-2.17/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...

To allow GTK+ and the other missing libraries to be found, I have wrapped the android utility:

wrapProgram `pwd`/android \
   --prefix PATH : ${jdk}/bin \
   --prefix LD_LIBRARY_PATH : ${glib}/lib:${gtk}/lib:${libXtst}/lib

Besides the libraries, I also had to wrap android to use the right JDK version. Apparently, loading these libraries only works with OpenJDK in Nix.

After wrapping the android utility and running it without parameters, it shows me the following:


As you may see in the above image, the GUI works. It also shows us all the plugins that are installed through Nix. Although the GUI seems to work fine, we have to keep in mind that we cannot use it to install additional Android SDK plugins. All plugins must be packaged and installed with Nix instead.

Supporting external system images


The Android distribution provides a collection of system images that can be used with an emulator to test an app during development. However, the default repository only contains ARM-based system images, which are quite slow to emulate.

There also seem to be x86-based system images available (provided by Intel) and MIPS-based images (provided by MIPS Technologies). These system images reside in external repositories. I was able to discover the locations of these repositories, by running the following command:

$ android list sdk
Refresh Sources:
  Fetching https://dl-ssl.google.com/android/repository/addons_list-2.xml
  Validate XML
  Parse XML
  Fetched Add-ons List successfully
  Refresh Sources
  Fetching URL: https://dl-ssl.google.com/android/repository/repository-8.xml
  Validate XML: https://dl-ssl.google.com/android/repository/repository-8.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/repository-8.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/addon.xml
  Validate XML: https://dl-ssl.google.com/android/repository/addon.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/addon.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Validate XML: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
  Fetching URL: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
  Validate XML: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
  Parse XML:    https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml

The latter two URLs referring to a file called sys-img.xml provide x86 and MIPS images. The structure of these XML files are similar to the main repository. Both files contain XML elements that look like this:

<sdk:system-image>
    <sdk:description>Android SDK Platform 4.1.1</sdk:description>
    <sdk:revision>1</sdk:revision>
    <sdk:api-level>16</sdk:api-level>
    <sdk:abi>x86</sdk:abi>
    <sdk:uses-license ref="intel-android-sysimage-license">
    <sdk:archives>
      <sdk:archive arch="any" os="any">
        <sdk:size>131840348</sdk:size>
        <sdk:checksum type="sha1">9d35bcaa4f9b40443941f32b8a50337f413c021a</sdk:checksum>
        <sdk:url>sysimg_x86-16_r01.zip</sdk:url>
      </sdk:archive>
    </sdk:archives>
</sdk:uses-license></sdk:system-image>

These elements can be converted to a Nix expression through XSL in a straight forward manner:

{stdenv, fetchurl, unzip}:

let
  buildSystemImage = args:
    stdenv.mkDerivation (args // {
      buildInputs = [ unzip ];
      buildCommand = ''
        mkdir -p $out
        cd $out
        unzip $src
    '';
  });
in
{
  sysimg_x86_16 = buildSystemImage {
    name = "sysimg-x86-16";
    src = fetchurl {
      url = https://dl-ssl.google.com/android/repository/sys-img/x86/sysimg_x86-16_r01.zip;
      sha1 = "9d35bcaa4f9b40443941f32b8a50337f413c021a";
    };
  };

  ...
}

The generated expression shown above is nearly identical to the one shown in the previous blog post. The only difference is that we also use the ABI identifier in the attribute names. For example: sysimg_x86_16 is used to refer to a x86-based system image for Android API-level 16. Likewise, we can refer to the ARM-based variant with: sysimg_armeabi-v7a_16.

Apart from giving each attribute a proper name, the resulting system image package must be symlinked into the Android SDK basedir taking the ABI identifier into account. We symlink every architecture-dependent system image package into: system-images/android-<api-level>/<abi>.

I have adapted the androidenv.emulateApp {} Nix function to also take ABI versions into account. By default, the emulator function uses an ARM-based system image (armeabi-v7a), since this is the most common hardware architecture used by Android OSes. By setting the abiVersion parameter to x86, we can use an x86-based Android system-image:

{androidenv, myfirstapp}:

androidenv.emulateApp {
  name = "MyFirstApp";
  app = myfirstapp;
  platformVersion = "16";
  abiVersion = "x86";
  package = "com.example.my.first.app";
  activity = "MainActivity";
}

An interesting benefit of using an x86 system image with the emulator (which is based on QEMU) is that KVM can be used, allowing programs inside the emulator to run at nearly native speed with only little emulation.

And of course, setting the abiVersion parameter to mips makes it possible to use MIPS-based system images.

Enabling GPU acceleration


The x86 system images in the emulator may give testing a significant boost. Emulation performance can even be improved a bit further by enabling GPU acceleration.

To make GPU acceleration possible, I had to adapt the emulator's wrapper script to include Mesa in the linker path. Moreover, in the AVD configuration I have to add the following boolean value to $ANDROID_SDK_HOME/.android/avd/device.avd/config.ini:

hw.gpu.enabled = yes

The androidenv.emulateApp {} automatically enables GPU acceleration from API-levels 15 and onwards. It can be disabled by setting the enableGPU parameter to false.

Supporting Google APIs in non-ARM-based system images


Although supporting x86 system images have various benefits, they also have a major drawback -- the Google APIs are not supported preventing some applications from working. One of the solutions I have seen (described in this Stack Overflow post) is to create a custom system image by starting an ARM-based emulator instance with Google APIs, fetch the libraries, start an x86-based emulator instance without Google APIs and manually install the fetched libraries.

I was able to automate the steps in the Stackflow article in a Nix expression:

{ platformVersion ? "16"
, abiVersion ? "x86"
, androidenv
, jdk
}:

let
  androidsdk = androidenv.androidsdk {
    platformVersions = [];
    abiVersions = [];
    useGoogleAPIs = false;
  };
  
  emulateAndroidARMWithGoogleAPIs = androidenv.emulateApp {
    name = "emulate-arm";
    inherit platformVersion;
    abiVersion = "armeabi-v7a";
    useGoogleAPIs = true;
  };
  
  emulateAndroidX86WithoutGoogleAPIs = androidenv.emulateApp {
    name = "emulate-x86";
    inherit platformVersion abiVersion;
    useGoogleAPIs = false;
  };
  
  mkfsYaffs2X86 = fetchurl {
    url = http://android-group-korea.googlecode.com/files/mkfs.yaffs2.x86;
    sha1 = "9362064c10a87ca68de688f09b162b171c55c66f";
  };
in
stdenv.mkDerivation {
  name = "sysimg_x86_${platformVersion}-with-google-apis";
  buildInputs = [ jdk androidsdk ];
  NIX_ANDROID_EMULATOR_FLAGS = "-no-window";
  buildCommand = ''
    source ${emulateAndroidARMWithGoogleAPIs}/bin/run-test-emulator
    portARM=$port
    adb -s emulator-$portARM pull /system/etc/permissions/com.google.android.maps.xml
    adb -s emulator-$portARM pull /system/framework/com.google.android.maps.jar
    adb -s emulator-$portARM emu kill
    
    export NIX_ANDROID_EMULATOR_FLAGS="$NIX_ANDROID_EMULATOR_FLAGS -partition-size 1024 -no-snapshot-save"
    source ${emulateAndroidX86WithoutGoogleAPIs}/bin/run-test-emulator
    portX86=$port
    adb -s emulator-$portX86 remount rw
    adb -s emulator-$portX86 push com.google.android.maps.jar /system/framework
    adb -s emulator-$portX86 push com.google.android.maps.xml /system/etc/permissions
    
    cp ${mkfsYaffs2X86} mkfs.yaffs2.x86
    adb -s emulator-$portX86 push mkfs.yaffs2.x86 /data
    adb -s emulator-$portX86 shell chmod 777 /data/mkfs.yaffs2.x86
    adb -s emulator-$portX86 shell /data/mkfs.yaffs2.x86 /system /data/system.img
    adb -s emulator-$portX86 pull /data/system.img
    adb -s emulator-$portX86 emu kill
    
    mkdir -p $out
    cp system.img $out
  '';
}

The above Nix function creates an ARM emulator instance with Google APIs and an x86 emulator instance without Google APIs. Then it starts the ARM emulator instance, fetches the Google API files from it and then stops it. Then the x86 emulator instance is started and the Google APIs are pushed to it. Finally we generate a system image from the system folder that is stored in the Nix store.

By using the Nix function shown above and setting the extraAVDFiles parameter of the emulateApp {} function, we can use our custom system image in a script that automatically spawns the emulator instance:

{androidenv, jdk}:

let
  platformVersion = "16";
  systemImg = import ./generateGoogleAPISystemImage.nix {
    inherit androidenv jdk platformVersion;
    abiVersion = "x86";
  };
in
androidenv.emulateApp {
  name = "MyFirstApp";
  extraAVDFiles = [ "${systemImg}/system.img" ];
  inherit platformVersion;
  abiVersion = "x86";
}

The above expression invokes the function shown previously and starts an emulator instance with the system image containing Google APIs. Of course, we can also use the same trick for MIPS-based system images, which also don't include Google APIs.

Building Android apps for arbitrary API-level revisions


I have also made a small change to the androidenv.buildApp {} Nix function to take API-levels into account. By default, it will use the version that is inside the Ant configuration. However, by setting the target property through the command-line, we can override this:

$ ant -Dtarget=android-17

The above command-line instruction ensures that we build the app against the API-level 17 platform. I have added an antFlags parameter to the Nix function to make it provide arbitrary flags to Ant:

{androidenv}:

androidenv.buildApp {
  name = "MyFirstApp";
  src = ../../src/myfirstapp;
  platformVersions = [ "17" ];
  useGoogleAPIs = true;
  antFlags = "-Dtarget=android-17";
}

Improving the test suite


In the previous blog post, I have implemented a testcase based on the Android introduction tutorial (MyFirstApp) that I have used throughout this article as well. To test all these new features, I have extended its composition expression to take three parameters: buildPlatformVersions (defaults to 16), emulatePlatformVersions (defaults to 16), and abiVersions (defaults to armeabi-v7a). The expression automatically generates all possible combinations with the values inside these three lists:

For example, to build a debug version of the test app APK for API-level 16, we can run:

$ nix-build -A myfirstapp_debug.build_16

We can also build a release version of the same APK that is signed with a key:

$ nix-build -A myfirstapp_release.build_16

Furthermore, we can automatically generate a script starting an emulator instance running the app. The following instruction builds an App for Android API-level 16, generates a script launching an emulator with a Android API-level 16 system-image that uses the armeabi-v7a ABI:

$ nix-build -A emulate_myfirstapp_debug.build_16.emulate_16.armeabi-v7a
$ ./result/run-test-emulator

To support more API-levels and ABIs, the parameters of the composition function must be altered. For example, by running the following command-line instruction:

$ nix-build --arg buildPlatformVersions '[ "16" "17" ]' \
  --arg emulatePlatformVersions '[ "15" "16 " 17" ]' \
  --arg abiVersions '[ "armeabi-v7a" "x86" ]' \
  -A emulate_myfirstapp_release.build_16.emulate_17.x86

The cartesian product of build and emulator instances is created taking the three dimensions into account. This allows us to (for example) build the App against the API-level 16 platform API, emulate on an API-level 17 system image using the x86 ABI, which emulates much more efficiently on x86 machines, because hardware visualization features (such as KVM) can be used.:

Apart from specifying these dimensions through the command line, the composition expression can also be used together with Hydra, the Nix-based continuous integration server, allowing it to pass these parameters through its web interface. Hydra will build all the combinations generated by the composition expression automatically:



As can be observed from the above screenshot, quite a few jobs are being generated.

Conclusion


In this blog post, I have described some new changes and features that I have implemented in the Nix Android SDK package since the last blog post, such as the fact that x86 and MIPS system-images can be used. All these changes are part of the Nixpkgs project.

Moreover, the Nix Android example testcase can be obtained from my GitHub page.