Sunday, March 16, 2014

Implementing consistent layouts for websites

Recently, I have wiped the dust off an old dormant project and I have decided to put it on GitHub, since I have found some use for it again. It is a personal project I started a long time ago.

Background


I got the inspiration for this project while working on my bachelor thesis project internship at IBM in 2005. I was developing an application usage analyzer system which included a web front-end implementing their intranet layout. I observed that it was a bit tedious to get it implemented properly. Moreover, I noticed that I had to repeat the same patterns over and over again for each page.

I saw some "tricks" that other people did to cope with these issues, but I considered all of them workarounds -- they were basically a bunch of includes in combination with a bit of iteration to make it work, but looked overly complicated and had all kinds of issues.

Some time before my internship, I learned about the Model-view-controller architectural pattern and I was looking into applying this pattern to the web front-end I was developing.

After some searching on the web using the MVC and Java Enterprise Edition (which was the underlying technology used to implement the system) keywords, I stumbled upon the following JavaWorld article titled: 'Understanding JavaServer Pages Model 2 architecture'. Although the article was specifically about the Model 2 architecture, I considered the Model 1 variant -- also described in the same article -- good enough for what I needed.

I observed that every page of an intranet application looks quite similar to others. For example, they had the same kinds of sections, same style, same colors etc. The only major differences were the selected menu item in the menu section and the contents (such as text) that is being displayed.

I created a model of the intranet layout that basically encodes the structure of the menu section that is being displayed on all pages of the web application. Each item in the menu redirects the user to the same page which -- based on the selected menu option -- displays different contents and a different "active" link. To cite my bachelor's thesis (which was written in Dutch):

De menu instantie bevat dus de structuur van het menu en de JSP zorgt ervoor dat het menu in de juiste opmaak wordt weergegeven. Deze aanpak is gebaseerd is op het Model 1 [model1] architectuur:

which I could translate into something like:

Hence, the menu instance contains the structure of the menu and the JSP is responsible for properly displaying the menu structure. This approach is based on the Model 1 [model1] architecture.

(As a sidenote: The website I am referring to calls "JSP Model 1" an architecture, which I blindly adopted in my thesis. These days, MVC is not something I would call an architecture, but rather an architectural pattern!)

I was quite satisfied with my implementation of the web front-end and some of my coworkers liked the fact that I was capable of implementing the intranet layout completely on my own and to be able to create and modify pages so easily.

Creating a library


After my internship, I was not too satisfied with the web development work I did prior to it. I had developed several websites and web applications that I still maintained, but all of them were implemented in an ad-hoc way -- one web application had a specific aspect implemented in a better way than others. Moreover, I kept reimplementing similar patterns over and over again including layout elements. I also did not reuse code effectively apart from a bit of copying and pasting.

From that moment on, I wanted everything that I had to develop to have the same (and the best possible) quality and to reuse as much code as possible so that every project would benefit from it.

I started a new library project from scratch. In fact, it were two library projects for two different programming languages. Initially I started implementing a Java Servlet/JSP version, since I became familiar with it during my internships at IBM and I considered it to be good and interesting technology to use.

However, all my past projects were implemented in PHP and also most of the web applications I maintained were hosted at shared webhosting providers only supporting PHP. As a result, I also developed a PHP version which became the version that I actually used for most of the time.

I could not use any code from my internship. Apart from the fact that it was IBM's property, it was also too specific for IBM intranet layouts. Moreover, I needed something that was even more general and more flexible so that I could encode all the layouts that I had implemented myself in the past. However, I kept the idea of the Model-1 and Model-2 architectural patterns that I discovered in mind.

Moreover, I also studied some usability heuristics (provided by the Nielsen-Norman Group) which I tried to implement in the library:

  • Visibility of system status. I tried supporting this aspect, by ensuring that the selected links in the menu section were explicitly marked as such so that users always know where they are in the navigation structure.
  • The "Consistency and standards" aspect was supported by the fact that every page has the same kinds of sections with the same purposes. For example, the menu sections have the same behavior as well as the result of clicking on a link.
  • I tried support "Error prevention" by automatically hiding menu links that were not accessible.

I kept evolving and improving the libraries until early 2009. The last thing I did with it was implementing my own personal homepage, which is still up and running today.

Usage


So how can these libraries be used? First, a model has to be created which captures common layout properties and the sub pages of which the application consists. In PHP, a simple application model could be defined as follows:

<?php
$application = new Application(
    /* Title */
    "Simple test website",

    /* CSS stylesheets */
    array("default.css"),

    /* Sections */
    array(
        "header" => new StaticSection("header.inc.php"),
        "menu" => new MenuSection(0),
        "submenu" => new MenuSection(1),
        "contents" => new ContentsSection(true)
    ),

    /* Pages */
    new StaticContentPage("Home", new Contents("home.inc.php"), array(
        "page1" => new StaticContentPage("Page 1", new Contents("page1.inc.php"), array(
            "page11" => new StaticContentPage("Subpage 1.1",
                new Contents("page1/subpage11.inc.php")),
            "page12" => new StaticContentPage("Subpage 1.2",
                new Contents("page1/subpage12.inc.php")),
            "page13" => new StaticContentPage("Subpage 1.3",
                new Contents("page1/subpage13.inc.php")))),
            ...
    )))
);

The above code fragment specifies the following:

  • The title of the entire web application is: "Simple test website", which will be visible in the title bar of the browser window for every sub page.
  • Every sub page of the application uses a common stylesheet: default.css
  • Every sub page has the same kinds of sections:
    • The header section always displays the same (static) content which code resides in a separate PHP include (header.inc.php)
    • The menu section displays a menu navigation section displaying links reachable from the entry page.
    • The submenu section displays a menu navigation section displaying links reachable from the pages in the previous menu section.
    • The contents section displays the actual dynamic contents (usually text) that makes the page unique based on the link that has been selected in one of the menu sections.
  • The remainder of the code defines the sub pages of which the web application consists. Sub pages are organised in a tree-like structure. The first object is entry page, the entry page has zero or more sub pages. Each sub page may have sub pages of their own, and so on.

    Every sub page provides their own contents to be displayed in contents section that has been defined earlier. Moreover, the menu sections automatically display links to the reachable sub pages from the current page that is being displayed.

By calling the following view function, with the application model as parameter we can display any of its sub pages:
displayRequestedPage($application);
?>

The above function generates a basic HTML page. The title of the page is composed of the application's title and the selected page title. Moreover, the sections are translated to div elements having an id attribute set to their corresponding array key. Each of these divs contains the contents of the include operations. The sub page selection is done by taking the last few path components of the URL that come after the script component.

If I create a "fancy" stylesheet, a bit of basic artwork and some actual contents for each include, something like this could appear on your screen:


Although the generated HTML by displayRequestedPage() is usually sufficient, I could also implement a custom one if I want to do more advanced stuff. I decomposed most if its aspects in sub functions that can be easily invoked from a custom function that does something different.

I have also created a Java version of the same concepts, which predates the PHP version. In the Java version, the model would look like this:

package test;

import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;
import io.github.svanderburg.layout.model.*;
import io.github.svanderburg.layout.model.page.*;
import io.github.svanderburg.layout.model.page.content.*;
import io.github.svanderburg.layout.model.section.*;

public class IndexServlet extends io.github.svanderburg.layout.view.IndexServlet
{
    private static final long serialVersionUID = 6641153504105482668L;

    private static final Application application = new Application(
        /* Title */
        "Test website",

        /* CSS stylesheets */
        new String[] { "default.css" },

        /* Pages */
        new StaticContentPage("Home", new Contents("home.jsp"))
            .addSubPage("page1", new StaticContentPage("Page 1", new Contents("page1.jsp"))
                .addSubPage("subpage11", new StaticContentPage("Subpage 1.1",
                    new Contents("page1/subpage11.jsp")))
                .addSubPage("subpage12", new StaticContentPage("Subpage 1.2",
                    new Contents("page1/subpage12.jsp")))
                .addSubPage("subpage13", new StaticContentPage("Subpage 1.3",
                    new Contents("page1/subpage13.jsp"))))
        ...
    )
    /* Sections */
    .addSection("header", new StaticSection("header.jsp"))
    .addSection("menu", new MenuSection(0))
    .addSection("submenu", new MenuSection(1))
    .addSection("contents", new ContentsSection(true));

    protected void doGet(HttpServletRequest req, HttpServletResponse resp)
        throws ServletException, IOException
    {
        dispatchLayoutView(application, req, resp);
    }

    protected void doPost(HttpServletRequest req, HttpServletResponse resp)
        throws ServletException, IOException
    {
        dispatchLayoutView(application, req, resp);
    }
}

As may be observed, since Java is statically typed language, more code is needed to express the same thing. Furthermore, Java has no associative arrays in its language, so I decided to use fluent interfaces instead.

Moreover, the model is also embedded in a Java Servlet, that dispatches the requests to a JSP page (WEB-INF/index.jsp) that represents the view. This JSP page could be implemented as follows:

<%@ page language="java" contentType="text/html; charset=UTF-8"
    pageEncoding="UTF-8" import="io.github.svanderburg.layout.model.*"
    import="io.github.svanderburg.layout.model.page.*,test.*"%>
<%
Application app = (Application)request.getAttribute("app");
Page currentPage = (Page)request.getAttribute("currentPage");
%>
<%@ taglib uri="http://svanderburg.github.io" prefix="layout" %>
<layout:index app="<%= app %>" currentPage="<%= currentPage %>" />

The above page takes the application model and the current page (determined by the URL to call it) as requests parameters. It invokes the index taglib (instead of a function in PHP) to compose an HTML page from it. Moreover, I have also encoded sub parts of the index page as reusable taglibs.

Other features


Besides the simple usage scenario shows earlier, the libraries support a collection of other interesting features, such as:

  • Multiple content section support
  • Per-page style and script includes
  • Error pages
  • Security handling
  • Controller sections to handle GET or POST parameters. In Java, you can invoke Java Servlets to do this, making the new library technically compliant with the JSP Model-2 architectural pattern.
  • Using path components as parameters
  • Internationalised sub pages

Conclusion


In this blog post, I have described an old dormant project that I revived and released. I always had the intention to release it as free/open-source software in the past, but never actually did it until now.

These days, some people do not really consider me a "web guy". I was very active in this domain a long time ago, but I (sort of) put that interest into the background, although I am still very much involved with web application development today (in addition to software deployment techniques and several other interests).

This interesting oatmeal comic clearly illustrates one of the major reasons why I have put my web technology interests into the background. This talk about web technology from Zed Shaw has an overlap with my other major reason.

Today, I am not so interested anymore in making web sites for people or to make this library a killer feature, but I don't mind sharing code. The only thing I care about at this moment is to use it to please myself.

Availability


The Java (java-sblayout) as well as the PHP (php-sblayout) versions of the libraries can be obtained from my GitHub page and used under the terms and conditions of the Apache Software License version 2.0.

Friday, February 28, 2014

Reproducing Android app deployments (or playing Angry Birds on NixOS)

Some time ago, I did a couple of fun experiments with my Android phone and the Android SDK. Moreover, I have developed a function that can be used to automate Android builds with Nix.

Not so long ago, somebody asked me if it would be possible to run arbitrary Android apps in NixOS. I realised that this was exactly the goal of my fun experiments. Therefore, I think it would be interesting to report about it.

Obtaining Android apps from a device


Besides development versions of apps that can be built with the Android SDK and deployed to a device or emulator through a USB connection, the major source of acquiring Android apps is the Google Playstore.

Although most devices (such as my phone and tablet) bundle the Google Playstore app as part of their software distributions, the system images that come with the Android SDK do not seem to have the Google Playstore app included.

Despite the fact that emulator system images do not have the Google Playstore app installed, we can still get most of the apps we want deployed to an emulator instance. What I typically do is installing an app on my phone with the Google Playstore, then downloading it from my phone and installing it in an emulator instance.

If I attach my phone to the computer and enable USB debugging on my device, I can run the following command to open a shell session:

$ adb -d shell

While navigating through the filesystem, I discovered that my phone stores apps in two locations. The system apps are stored in /system/app. All other apps reside in /data/app. One of the annoying things about the latter folder is that root access to my phone is restricted and I'm not allowed to read the contents of it:

$ cd /data/app
$ ls
opendir failed, Permission denied

Later I discovered that Android distributions use a tool called pm to deploy Android packages. Running the following command-line instruction gives me an overview of all the installed packages and the locations where they reside on the filesystem:

$ pm list packages -f
package:/system/app/GoogleSearchWidget.apk=android.googleSearch.googleSearchWidget
package:/data/app/com.example.my.first.app-1.apk=com.example.my.first.app
package:/system/app/KeyChain.apk=com.android.keychain
package:/data/app/com.appcelerator.kitchensink-1.apk=com.appcelerator.kitchensink
package:/system/app/Shell.apk=com.android.shell
package:/data/app/com.capcom.smurfsandroid-1.apk=com.capcom.smurfsandroid
package:/data/app/com.rovio.angrybirds-2.apk=com.rovio.angrybirds
package:/data/app/com.rovio.BadPiggies-1.apk=com.rovio.BadPiggies
package:/data/app/com.android.chrome-2.apk=com.android.chrome
...

As can be seen, the package manager shows me the location of all installed apps, including those that reside in the folder that I could not inspect. Moreover, downloading the actual APKs files through the Android debugger does not seem to be restricted either. For example, I can run the following Android debugger instruction to obtain Angry Birds that I have installed on my phone:

$ adb -d pull /data/app/com.rovio.angrybirds-2.apk
5688 KB/s (45874906 bytes in 7.875s)

Running arbitrary APKs in the emulator


In my earlier blog posts on automating Android builds with Nix, I have described how I implemented a Nix function (called androidenv.emulateApp { }) that generates scripts spawning emulator instances in which a development app is automatically deployed and started.

I have adapted this function to make it more convenient to deploy existing APKs and to make it more suitable for running apps for other purposes than development:

  • The original script stores the state files of the emulator instance in a temp folder, which gets discarded afterwards. For test automation this is quite useful in most cases. However, we don't want to lose our savegames while playing games. Therefore, I added a parameter called avdHomeDir allowing someone to store the state files in a non-volatile location on the filesystem, such as the user's home directory. If this parameter is not provided, the script remains to use a temp directory.
  • Since we want to keep the state of the emulator instance around, there is also no need to create it every time we launch the emulator. I have adapted the script in such a way that it only creates the AVD if it does not exists. Running the following instruction seems to be sufficient to check whether the AVD exists:

    $ android list avd | grep "Name: device"
    

  • The same thing applies to the app that gets deployed to the emulator instance. It's only supposed to be deployed if it is not installed yet. Running the following command-line instruction did the trick for me:

    $ adb -s emulator-5554 pm list packages | \
        grep package:com.rovio.angrybirds
    package:com.rovio.angrybirds
    

    It shows me the name of the package if it is installed already.

Automatically starting apps in the emulator


As described in my earlier blog post, the script that launches the emulator can also automatically start the app. To do this, we need the Java package identifier of the app and the name of the start activity. While developing apps, these properties can be found in the manifest file that is part of the development repository. However, it's a bit trickier to obtain these attributes if you only have a binary APK.

I have discovered that the aapt tool (that comes with the Android SDK) is quite useful to find what I need. While running the following command-line instruction with the Angry Birds APK, I discovered the following:

$ aapt l -a com.rovio.angrybirds-2.apk
...
A: package="com.rovio.angrybirds" (Raw: "com.rovio.angrybirds")

E: application (line=47)
      A: android:label(0x01010001)="Angry Birds" (Raw: "Angry Birds")
      A: android:icon(0x01010002)=@0x7f020001
      A: android:debuggable(0x0101000f)=(type 0x12)0x0
      A: android:hardwareAccelerated(0x010102d3)=(type 0x12)0x0
      E: activity (line=48)
        A: android:theme(0x01010000)=@0x1030007
        A: android:name(0x01010003)="com.rovio.fusion.App" (Raw: "com.rovio.fusion.App")
        A: android:launchMode(0x0101001d)=(type 0x10)0x2
        A: android:screenOrientation(0x0101001e)=(type 0x10)0x0
        A: android:configChanges(0x0101001f)=(type 0x11)0x4a0
        E: intent-filter (line=49)
          E: action (line=50)
            A: android:name(0x01010003)="android.intent.action.MAIN" (Raw: "android.intent.action.MAIN")
          E: category (line=51)
            A: android:name(0x01010003)="android.intent.category.LAUNCHER" (Raw: "android.intent.category.LAUNCHER")

Somewhere at the end of the output, the package name is shown (com.rovio.angrybirds) and the app's activities. The activity that supports android.intent.action.MAIN intent is actually the one we are looking for. According to the information that we have collected, the start activity that we have to call is named com.rovio.fusion.App.

Writing a Nix expression


Now that we have retrieved the Angry Birds APK and discovered the attributes to automatically start it, we can automate the process that sets up an emulator instance. I wrote the following Nix expression to do this:

with import <nixpkgs> {};

androidenv.emulateApp {
  name = "angrybirds";
  app = ./com.rovio.angrybirds-2.apk;
  platformVersion = "18";
  useGoogleAPIs = false;
  enableGPU = true;
  abiVersion = "x86";
  
  package = "com.rovio.angrybirds";
  activity = "com.rovio.fusion.App";

  avdHomeDir = "$HOME/.angrybirds";
}

The above Nix expression sets the following parameters:
  • The name parameter is simply used to make the Nix store path better readable.
  • The app parameter points to the Angry Birds APK that I just downloaded from my phone. It gets automatically installed in the spawned emulator instance.
  • platformVersion refers to the API-level of the system image that the emulator runs. API-level 18 corresponds to Android version 4.3
  • If we need Google specific functionality (such as Google Maps) we need a Google API-enabled system image. Angry Birds does not seem to require it.
  • To allow games to run smoothly, it's better to enable hardware GPU emulator/acceleration through the enableGPU parameter
  • The abiVersion sets the CPU architecture of the emulator. Most apps are actually developed for armeabi-v7a and this is usually the safest or the only option that works (unless the app is not using any native code or supports other desired architectures). Angry Birds also supports x86 which can be much faster emulated.
  • The package and activity parameters are used to automatically start the app
  • We use the avdHomeDir parameter to persistently store the state of the emulator in the .angrybirds folder of my home directory, so that the progress is retained.

I can build the earlier Nix expression with the following command:

$ nix-build angrybirds.nix

And then play Angry Birds, by running:

./result/bin/run-test-emulator

The above script starts the emulator, installs Angry Birds, and starts it. This is the result (to rotate the screen I used the 7 and 9 keys on the numpad):


Isn't it awesome? ;)

Transferring state


I also discovered how to transfer the state of apps (such as settings and savegames) from a device to an emulator instance and vice-versa. For some games, you can obtain these through Android's backup functionality. The following instruction makes a backup from my phone of the state of a particular app:

$ adb -d backup com.rovio.angrybirds -f state
Now unlock your device and confirm the backup operation.

When running the above instruction, you'll be asked for confirmation for making the backup and some details to optionally encrypt it.

With the following instruction, I can restore the captured state in the emulator:

$ adb -s emulator-5554 restore com.rovio.angrybirds -f state
Now unlock your device and confirm the backup operation.

While running the latter operation, you'll also be asked for confirmation.

Conclusion


In this blog post, I have described how we can automatically deploy existing Android APKs in an emulator instance using the Nix package manager. I have used it to play Angry Birds and a couple of other Android games in NixOS.

There are few caveats that you have to keep in mind:

  • I have observed that quite a few apps, especially games, have native dependencies. Most of these games only seem to work on ARM-based systems. Although x86 images are much faster to emulate, you will not benefit from the speed boost they may give you if this CPU architecture is not supported.
  • Some apps use Google API specific functionality. Unfortunately, the Android SDK does not provide non-ARM based system images that support them. In a previous blog post, I have developed a Nix expression that can be used to create x86 Google API enabled system images from the ARM-based images, although it may be a bit tricky to set them up.
  • Some apps may install additional files besides the APK when they are installed through the Google Playstore. For me running adb logcat and inspecting the error messages in the logs helped me out a few times.

Availability


The androidenv.emulateApp { } function is part of Nixpkgs.

It's also important to point out that the Nixpkgs repository does NOT contain any prepackaged Android games or apps. You have to obtain and deploy these apps yourself!

Wednesday, January 29, 2014

Building Appcelerator Titanium apps with Nix

Last month, I have been working on quite a lot of things. One of the things I did was improving the Nix function that builds Titanium SDK applications. In fact, it was in Nixpkgs for quite a while already, but I have never written about it on my blog, apart from a brief reference in an earlier blog post about Hydra.

The reason that I have decided to write about this function is because the process of getting Titanium applications deployable with Nix is quite painful (although I have managed to do it) and I want to report about my experiences so that these issues can be hopefully resolved in the future.

Although I have a strong opinion on certain aspects of Titanium, this blog post is not to discuss about the development aspects of the Titanium framework. Instead, the focus is on getting the builds of Titanium apps automated.

What is Titanium SDK?


Titanium is an application framework developed by Appcelerator, which purpose is to enable rapid development of mobile apps for multiple platforms. Currently, Titanium supports iOS, Android, Tizen, Blackberry and mobile web applications.

With Titanium, developers use JavaScript as an implementation language. The JavaScript code is packaged along with the produced app bundles, deployed to an emulator or device and interpreted there. For example, on Android Google's V8 JavaScript runtime is used, and on iOS Apple's JavaScriptCore is used.

Besides using JavaScript code, Titanium also provides an API supporting database access and (fairly) cross platform GUI widgets that have a (sort of) native look on each platform.

Titanium is not a write once run anywhere approach when it comes to cross platform support, but claims that 60-90% of the app code can be reused among platforms.

Finally, the Titanium Studio software distribution is proprietary software, but most of its underlying components (including the Titanium SDK) are free and open-source software available under the Apache Software License. As far as I can see, the Nix function that I wrote does not depend on any proprietary components, besides the Java Development Kit.

Packaging the Titanium CLI


The first thing that needs to be done to automate Titanium builds is being able to build stuff from the command-line. Appcelerator provides a command-line utility (CLI) that is specifically designed for this purpose and is provided as a Node.js package that can be installed through the NPM package manager.

Packaging NPM stuff in Nix is actually quite straight forward and probably the easiest part of getting the builds of Titanium apps automated. Simply adding titanium to the list of node packages (pkgs/top-level/node-packages.json) in Nixpkgs and running npm2nix, a utility developed by Shea Levy that automatically generates Nix expressions for any node package and all their dependencies, did the job for me.

Packaging the Titanium SDK


The next step is packaging the Titanium SDK that contains API libraries, templates and build script plugins for each target platform. The CLI supports multiple SDK versions at the same time and requires at least one version of an SDK installed.

I've obtained an SDK version from Appcelerator's continuous builds page. Since the SDK distributions are ZIP files containing binaries, I have to use the patching/wrapping tricks I have described in a few earlier blog posts again.

The Nix expression I wrote for the SDK basically unzips the 3.2.1 distribution, copies the contents into the Nix store and makes the following changes:

  • The SDK distribution contains a collection of Python scripts that execute build and debugging tasks. However, to be able to run them in NixOS, the shebangs must be changed so that the Python interpreter can be found:

    find . -name \*.py | while read i
    do
        sed -i -e "s|#!/usr/bin/env python|#!${python}/bin/python|" $i
    done
    
  • The SDK contains a subdirectory (mobilesdk/3.2.1.v20140206170116) with a version number and timestamp in it. However, the timestamp is a bit inconvenient, because the Titanium CLI explicitly checks for SDK folders that correspond to a Titanium SDK version number in a Titanium project file (tiapp.xml). Therefore, I strip it out of the directory name to make my life easier:

    $ cd mobilesdk/*
    $ mv 3.2.1.v20140206170116 3.2.1.GA
    
  • The Android builder script (mobilesdk/*/android/builder.py) packages certain files into an APK bundle (which is technically a ZIP file).

    However, the script throws an exception if it encounters files with timestamps below January 1, 1980, which are not supported by the ZIP file format. This is a problem, because Nix automatically resets timestamps of deployed packages to one second after January 1, 1970 (a.k.a. UNIX-time: 1) to make builds more deterministic. To remedy the issue, I had to modify several pieces of the builder script.

    What I basically did to fix this is searching for invocations to ZipFile.write() that adds a file from the filesystem to a zip archive, such as:

    apk_zip.write(os.path.join(lib_source_dir, 'libtiverify.so'), lib_dest_dir + 'libtiverify.so')
    

    I refactored such invocations into a code fragment using a file stream:

    info = zipfile.ZipInfo(lib_dest_dir + 'libtiverify.so')
    info.compress_type = zipfile.ZIP_DEFLATED
    info.create_system = 3
    tf = open(os.path.join(lib_source_dir, 'libtiverify.so'))
    apk_zip.writestr(info, f.read())
    tf.close()
    

    The above code fragment ignores the timestamp of the files to be packaged and uses the current time instead, thus fixing the issue with files that reside in the Nix store.
  • There were two ELF executables (titanium_prep.{linux32,linux64}) in the distribution. To be able to run them under NixOS, I had to patch them so that the dynamic linker can be found:

    $ patchelf --set-interpreter ${stdenv.gcc.libc}/lib/ld-linux-x86-64.so.2 \
        titanium_prep.linux64
    
  • The Android builder script (mobilesdk/*/android/builder.py) requires the sqlite3 python module and the Java Development Kit. Since dependencies do not reside in standard locations in Nix, I had to wrap the builder script to allow it to find them:

    mv builder.py .builder.py
    cat > builder.py <<EOF
    #!${python}/bin/python
        
    import os, sys
        
    os.environ['PYTHONPATH'] = '$(echo ${python.modules.sqlite3}/lib/python*/site-packages)'
    os.environ['JAVA_HOME'] = '${jdk}/lib/openjdk'
        
    os.execv('$(pwd)/.builder.py', sys.argv)
    EOF
    

    Although the Nixpkgs collection has a standard function (wrapProgram) to easily wrap executables, I could not use it, because this function turns any executable into a shell script. The Titanium CLI expects that this builder script is a Python script and will fail if there is a shell code around it.
  • The iOS builder script (mobilesdk/osx/*/iphone/builder.py) invokes ditto to do a recursive copy of a directory hierarchy. However, this executable cannot be found in a Nix builder environment, since the PATH environment variable is set to only the dependencies that are specified. The following command fixes it:

    $ sed -i -e "s|ditto|/usr/bin/ditto|g" \
        $out/mobilesdk/osx/*/iphone/builder.py
    
  • When building IPA files for iOS devices, the Titanium CLI invokes xcodebuild, that in turn invokes the Titanium CLI again. However, it does not seem to propagate all parameters properly, such as the path to the CLI's configuration file. The following modification allows me to set an environment variable called: NIX_TITANIUM_WORKAROUND providing additional parameters to work around it:

    $ sed -i -e "s|--xcode|--xcode '+process.env['NIX_TITANIUM_WORKAROUND']+'|" \
        $out/mobilesdk/osx/*/iphone/cli/commands/_build.js
    

Building Titanium Apps


Besides getting the Titanium CLI and SDK packaged in Nix, we must also be able to build Titanium apps. Apps can be built for various target platforms and come in several variants.

For some unknown reason, the Titanium CLI (in contrast to the old Python build script) forces people to login with their Appcelerator account, before any build task can be executed. However, I discovered that after logging in a file is written into the ~/.titanium folder indicating that the system has logged in. I can simulate logins by creating this file myself:

export HOME=$TMPDIR
    
mkdir -p $HOME/.titanium
cat > $HOME/.titanium/auth_session.json <<EOF
{ "loggedIn": true }
EOF

We also have to tell the Titanium CLI where the Titanium SDK can be found. The following command-line instruction updates the config to provide the path to the SDK that we have just packaged:

$ echo "{}" > $TMPDIR/config.json
$ titanium --config-file $TMPDIR/config.json --no-colors \
    config sdk.defaultInstallLocation ${titaniumsdk}

The Titanium SDK also contains a collection of prebuilt modules, such as one to connect to Facebook. To allow them to be found, I run the following command line instruction to adapt the module search path:

$ titanium --config-file $TMPDIR/config.json --no-colors \
    config paths.modules ${titaniumsdk}

I have also noticed that if the SDK version specified in a Titanium project file (tiapp.xml) does not match the version of the installed SDK, the Titanium CLI halts with an exception. Of course, the version number in a project file can be adapted, but it in my opinion, it's more flexible to just be able to take any version. The following instruction replaces the version inside tiapp.xml into something else:

$ sed -i -e "s|<sdk-version>[0-9a-zA-Z\.]*</sdk-version>|<sdk-version>${tiVersion}</sdk-version>|" tiapp.xml

Building Android apps from Titanium projects


For Android builds, we must tell the Titanium CLI where to find the Android SDK. The following command-line instruction adds its location to the config file:

$ titanium config --config-file $TMPDIR/config.json --no-colors \
    android.sdkPath ${androidsdkComposition}/libexec/android-sdk-*

The variable: androidsdkComposition refers to an Android SDK plugin composition provided by the Android SDK Nix expressions I have developed earlier.

After performing the previous operation, the following command-line instruction can be used to build a debug version of an Android app:

$ titanium build --config-file $TMPDIR/config.json --no-colors --force \
    --platform android --target emulator --build-only --output $out

If the above command succeeds, an APK bundle called app.apk is placed in the Nix store output folder. This bundle contains all the project's JavaScript code and is signed with a developer key.

The following command produces a release version of the APK (meant for submission to the Play Store) in the Nix store output folder, with a given key store, key alias and key store password:

$ titanium build --config-file $TMPDIR/config.json --no-colors --force \
    --platform android --target dist-playstore --keystore ${androidKeyStore} \
    --alias ${androidKeyAlias} --password ${androidKeyStorePassword} \
    --output-dir $out

Before the JavaScript files are packaged along with the APK file, they are first passed through Google's Closure Compiler, which performs some static checking, removes dead code, and minifies all the source files.

Building iOS apps from Titanium projects


Apart from Android, we can also build iOS apps from Titanium projects.

I have discovered that while building for iOS, the Titanium CLI invokes xcodebuild which in turn invokes the Titanium CLI again. However, it does not propagate the --config-file parameter, causing it to fail. The earlier hack that I made in the SDK expression with the environment variable can be used to circumvent this:

export NIX_TITANIUM_WORKAROUND="--config-file $TMPDIR/config.json"

After applying the workaround, building an app for the iPhone simulator is straight forward:

$ cp -av * $out
$ cd $out
            
$ titanium build --config-file $TMPDIR/config.json --force --no-colors \
    --platform ios --target simulator --build-only \
    --device-family universal --output-dir $out

After running the above command, the simulator executable is placed into the output Nix store folder. It turns out that the JavaScript files of the project folder are symlinked into the folder of the executable. However, after the build has completed these symlink references will become invalid, because the temp folder has been deleted. To allow the app to find these JavaScript files, I simply copy them along with the executable into the Nix store.

Finally, the most complicated task is producing IPA bundles to deploy an app to a device for testing or to the App Store for distribution.

Like native iOS apps, they must be signed with a certificate and mobile provisioning profile. I used the same trick described in an earlier blog post on building iOS apps with Nix to generate a temporary keychain in the user's home directory for this:

export HOME=/Users/$(whoami)
export keychainName=$(basename $out)
            
security create-keychain -p "" $keychainName
security default-keychain -s $keychainName
security unlock-keychain -p "" $keychainName
security import ${iosCertificate} -k $keychainName -P "${iosCertificatePassword}" -A

provisioningId=$(grep UUID -A1 -a ${iosMobileProvisioningProfile} | grep -o "[-A-Z0-9]\{36\}")
        
if [ ! -f "$HOME/Library/MobileDevice/Provisioning Profiles/$provisioningId.mobileprovision" ]
then
    mkdir -p "$HOME/Library/MobileDevice/Provisioning Profiles"
    cp ${iosMobileProvisioningProfile} \
        "$HOME/Library/MobileDevice/Provisioning Profiles/$provisioningId.mobileprovision"
fi

I also discovered that builds fail, because some file (the facebook module) from the SDK cannot be read (Nix makes deployed package read-only). I circumvented this issue by making a copy of the SDK in my temp folder, fixing the file permissions, and configure the Titanium CLI to use the copied SDK instance:

cp -av ${titaniumsdk} $TMPDIR/titaniumsdk
            
find $TMPDIR/titaniumsdk | while read i
do
    chmod 755 "$i"
done

titanium --config-file $TMPDIR/config.json --no-colors \
    config sdk.defaultInstallLocation $TMPDIR/titaniumsdk

Because I cannot use the temp folder as a home directory, I also have to simulate a login again:

$ mkdir -p $HOME/.titanium
$ cat > $HOME/.titanium/auth_session.json <<EOF
{ "loggedIn": true }
EOF

Finally, I can build an IPA by running:

$ titanium build --config-file $TMPDIR/config.json --force --no-colors \
    --platform ios --target dist-adhoc --pp-uuid $provisioningId \
    --distribution-name "${iosCertificateName}" \
    --keychain $HOME/Library/Keychains/$keychainName \
    --device-family universal --output-dir $out

The above command-line invocation minifies the JavaScript code, builds an IPA file with a given certificate, mobile provisioning profile and authentication credentials, and puts the result in the Nix store.

Example: KitchenSink


I have encapsulated all the builds commands shown in the previous section into a Nix function called: titaniumenv.buildApp {}. To test the usefulness of this function, I took KitchenSink, an example app provided by Appcelerator, to show Titanium's abilities. The App can be deployed to all target platforms that the SDK supports.

To package KitchenSink, I wrote the following expression:

{ titaniumenv, fetchgit
, target, androidPlatformVersions ? [ "11" ], release ? false
}:

titaniumenv.buildApp {
  name = "KitchenSink-${target}-${if release then "release" else "debug"}";
  src = fetchgit {
    url = https://github.com/appcelerator/KitchenSink.git;
    rev = "d9f39950c0137a1dd67c925ef9e8046a9f0644ff";
    sha256 = "0aj42ac262hw9n9blzhfibg61kkbp3wky69rp2yhd11vwjlcq1qc";
  };
  tiVersion = "3.2.1.GA";
  
  inherit target androidPlatformVersions release;
  
  androidKeyStore = ./keystore;
  androidKeyAlias = "myfirstapp";
  androidKeyStorePassword = "mykeystore";
}

The above function fetches the KitchenSink example from GitHub and builds it for a given target, such as iphone or android, and supports building a debug version for an emulator/simulator, or a release version for a device or for the Play store/App store.

By invoking the above function as follows, a debug version of the app for Android is produced:

import ./kitchensink {
  inherit (pkgs) fetchgit titaniumenv;
  target = "android";
  release = false;
}
The following function invocation produces an iOS executable that can be run in the iPhone simulator:

import ./kitchensink {
  inherit (pkgs) fetchgit titaniumenv;
  target = "iphone";
  release = false;
}

As may be observed, building KitchenSink through Nix is a straight forward process for most targets. However, the target producing an IPA version of KitchenSink that we can deploy to a real device is a bit complicated to use, because of some restrictions made by Apple.

Since all apps that are deployed to a real device have to be signed and the mobile provisioning profile should match the app's app id, this is sort of problem. Luckily, I can also do a comparable renaming trick as I have described earlier with in a blog post about improving the testability of iOS apps. Simply executing the following commands in the KitchenSink folder were sufficient:

sed -i -e "s|com.appcelerator.kitchensink|${newBundleId}|" tiapp.xml
sed -i -e "s|com.appcelerator.kitchensink|${newBundleId}|" manifest

The above commands change the com.appcelerator.kitchensink app id into any other specified string. If this app id is changed to the corresponding id in a mobile provisioning profile, then you should be able to deploy KitchenSink to a real device.

I have added the above renaming procedure to the KitchenSink expression. The following example invocation to the earlier Nix function, shows how we can rename the app's id to: com.example.kitchensink and how to use a certificate and mobile provisioning profile for an exisiting app:

import ./kitchensink {
  inherit (pkgs) stdenv fetchgit titaniumenv;
  target = "iphone";
  release = true;
  rename = true;
  newBundleId = "com.example.kitchensink";
  iosMobileProvisioningProfile = ./profile.mobileprovision;
  iosCertificate = ./certificate.p12;
  iosCertificateName = "Cool Company";
  iosCertificatePassword = "secret";
}


By using the above expressions KitchenSink can be built for both Android and iOS. The left picture above shows what it looks like on iOS, the right picture shows what it looks like on Android.

Discussion


With the Titanium build function described in this blog post, I can automatically build Titanium apps for both iOS and Android using the Nix package manager, although it was quite painful to get it done and tedious to maintain.

What bothers me the most about this process is the fact that Appcelerator has crafted their own custom build tool with lots of complexity (in terms of code size), flaws (e.g. not propagating the CLI's argument properly from xcodebuild) and weird issues (e.g. an odd way of detecting the presence of the JDK, and invoking the highly complicated legacy python scripts), while there are already many more mature build solutions available that can do the same job.

A quick inspection of Titanium CLI's git repository shows me that it consists of 8174 lines of code. However, not all of their build stuff is there. Some common stuff, such as the JDK and Android detection stuff, resides in the node-appc project. Moreover, the build steps are performed by plugin scripts that are distributed with the SDK.

A minor annoyance is that the new Node.js based Titanium CLI requires Oracle's Java Development Kit to make Android builds work, while the old Python based build script worked fine with OpenJDK. I have no idea yet how to fix this. Since we cannot provide a Nix expression that automatically downloads Oracle's JDK that automatically (due to license restrictions), Nix users are forced to manually download and import it into the Nix store first, before any of the Titanium stuff can be built.

So how did I manage to figure all this mess out?

Besides knowing that I have to patch executables, fix shebangs and wrap certain executables, the strace command on Linux helps me out a lot (since it shows me things like files that can not be opened) as well as the fact that Python and Node.js show me error traces with line numbers when something goes wrong so that I can debug easily what's going on.

However, since I also have to do builds on Mac OS X for iOS devices, I observed that there is no strace making ease my pain on that particular operating system. However, I discovered that there is a similar tool called: dtruss, that provides me similar data regarding system calls.

There is one minor annoyance with dtruss -- it requires super-user privileges to work. Fortunately, thanks to this MacWorld article, I can fix this by setting the setuid bit on the dtrace executable:

$ sudo chmod u+s /usr/sbin/dtrace

Now I can conveniently use dtruss in unprivileged build environments on Mac OS X to investigate what's going on.

Availability


The Titanium build environment as well as the KitchenSink example are part of Nixpkgs.

The top-level expression for KitchenSink example as well as the build operations described earlier is located in pkgs/development/mobile/titaniumenv/examples/default.nix. To build a debug version of KitchenSink for Android, you can run:

$ nix-build -A kitchensink_android_debug

The release version can be built by running:

$ nix-build -A kitchensink_android_release

The iPhone simulator version can be built by running:

$ nix-build -A kitchensink_ios_development

Building an IPA is slightly more complicated. You have to provide a certificate and mobile provisioning profile, and some renaming trick settings as parameters to make it work (which should of course match to what's inside the mobile provisioning profile that is actually used):

$ nix-build --arg rename true \
    --argstr newBundleId com.example.kitchensink \
    --arg iosMobileProvisioningProfile ./profile.mobileprovision \
    --arg iosCertificate ./certificate.p12 \
    --argstr iosCertificateName "Cool Company" \
    --argstr iosCertificatePassword secret \
    -A kitchensink_ipa

There are also a couple of emulator jobs to easily spawn an Android emulator or iPhone simulator instance.

Currently, iOS and Android are the only target platforms supported. I did not investigate Blackberry, Tizen or Mobile web applications.

Presentation


UPDATE: On July 14, 2016 I have given a presentation about this subject at the Nix meetup in Amsterdam. For convenience, I have embedded the sildes into this blog post:

Monday, December 30, 2013

Third annual blog reflection

Today, it's three years ago that I started this blog, so I thought this is a good opportunity to reflect about last year's writings.

Obtaining my PhD degree


I think the most memorable thing that happened for me this year is the fact that I finally obtained my PhD degree. From that moment on, I can finally call myself Dr. Sander, or actually (if I take my other degree into account): Dr. Ir. Sander (or in fact: Dr. Ir. Ing. Sander if I take all of them into account, but I believe the last degree has been been superseded by the middle one, but I'm not sure :-) ).

Anyway, after obtaining my PhD degree I don't feel that much different, apart from the fact that I feel relieved that it's done. It took me quite some effort to get my PhD dissertation and all the preparations done for the defense. Besides my thesis, I also had to defend my propositions. Most of them were not supposed to be directly related to my research subject.

Programming in JavaScript


From the moment that I have switched jobs, I have also been involved in a lot of JavaScript programming these days. Every programming language and runtime environment have their weird/obscure problems/challenges, but in my opinion JavaScript is a very special case.

As a former teaching assistant for the concepts of programming languages course, I remain interested in discovering important lessons allowing me to prevent turning code into a mess and dealing with challenges that a particular programming gives. So far, I have investigated object oriented programming through prototypes, and two perspectives of dealing with asynchronous programming problems that come with most JavaScript environments.

Besides programming challenges I also have to perform deployment tasks from JavaScript programs. People who happen to know me know that I prefer Nix and Nix-related solutions. I have developed NiJS: an internal DSL for Nix to make my life a bit easier to do that.

Continuous integration and testing


Another technical aspect I have been working on is setting up a continuous integration facility by using Hydra: the Nix-based continuous integration server. I wrote a couple of blog posts describing its features, how to set it up and how to secure it.

I also made a couple improvements to the Android and iOS Nix build functions, so that I can use Hydra to continuously build mobile apps.

Nix/NixOS development


Besides Hydra, I have been involved with various other parts of the Nix project as well. One of the more interesting things I did is developing a Nix function that can be used to compose FHS-compatible chroot environments. This function is particularly useful to run binary-only software in NixOS that cannot be patched, such as Steam.

I also wrote two blog posts to explain the user environment and development environment concepts.

Fun programming


Besides my PhD defense and all other activities, there was a bit of room to do some fun programming as well. I have improved my Amiga video emulation library (part of my IFF file format experiments project) a bit by implementing support for super hires resolutions and redefining its architecture.

Moreover, I have updated all the packages to use the new version of this library.

Research


After obtaining my PhD degree, I'm basically relieved from my research/publication duties. However, there is one research related thing that caught my attention two months ago.

The journal paper titled: 'Disnix: A toolset for distributed deployment' that got accepted in April last year is finally going to get published in Volume 79 of 'Science of Computer Programming', unbelievable!

Although it sounds like good news that another paper of mine gets published, the thing that disturbs me is that the publication process took an insanely long time! I wrote the first version of this paper for the WASDeTT-3 workshop that was held in October 2010. So that technically means that I started doing the work for the paper several months prior to that.

In February 2011, I have adapted/extended the workshop paper and submitted its first journal draft. Now, in January 2014 it finally gets published, which means that it took almost 3 years to get published (if you take the workshop into account as well, then it's actually closer to 3,5 years!).

In some academic assessment standards, journal papers have more value than conference papers. Although this journal paper should increase my value as a researcher, it's actually crazy if you think too much about it. The first reason is that I wrote the first version of this paper before I started this blog. Meanwhile, I have already written 54 blog articles, two tech reports, published two papers at conferences, and I finished my PhD dissertation.

The other reason is that peer reviewing and publishing should help the authors and the research discipline in general. To me this does not look like any help. Meanwhile, in the current development version of Disnix some aspects of its architecture have evolved considerably compared to what has been described in the paper, so it is no use for anyone else in the research community anymore.

The only value the paper still provides are the general ideas and the way Disnix manifests itself externally.

Although the paper is not completely valueless, and I'm happy it gets published, it also feels weird that I don't depend on it anymore.

Blog posts


As with my previous annual reflections, I will also publish the top 10 of my most frequently read blog posts:

  1. On Nix and GNU Guix. This is a critical blog post that also ended up first in last year's top 10. I think this blog posts will remain at the top position for the time being, since it attracted an insane amount of visitors.
  2. An alternative explanation of the Nix package manager. My alternative explanation of Nix, which I wrote to clarify things. It was also second in last year's top 10.
  3. Setting up a Hydra build cluster for continuous integration and testing (part 1). Apparently, Hydra and some general principles about continuous integration have attracted quite some vistors. However, the follow up blog posts I wrote about Hydra don't seem to be that interesting to outsiders.
  4. Using Nix while doing development. I wrote this blog post 2 days ago, and it attracted quite some visitors. I have noticed that setting up development environments is an attractive feature for Nix users.
  5. Second computer. This is an old blog post about my good ol' Amiga. It was also in all previous top 10s and I think it will remain like that for the time being. The Amiga rocks!
  6. An evaluation and comparison of GoboLinux. Another blog article that remains popular from the beginning. It's still a pity that GoboLinux has not been updated and sticks to their 014.01 release, which dates from 2008.
  7. Composing FHS-compatible chroot environments with Nix (or deploying Steam in NixOS). This is something I have developed to be able to run Steam in NixOS. It seems to have attracted quite some users, which does not come as a surprise. NixOS users want to play Half-Life!
  8. Software deployment complexity. An old blog post about software deployment complexity in general. Still remains popular.
  9. Deploying iOS applications with the Nix package manager. A blog post that I wrote last year describing how we can use the Nix package manager to build apps for the iPhone/iPad. For a long time the Android variant of this blog post was more popular, but recently this blog article surpassed it. I have no clue why.
  10. Porting software to AmigaOS (unconventional style). People still seem to like one of my craziest experiments.

Conclusion


I already have three more blog posts in draft/planning stages and more ideas that I like to explore, so expect more to come. The remaining thing I'd like to say is:

HAPPY NEW YEAR!!!

Saturday, December 28, 2013

Using Nix while doing development

I have noticed that while doing development work, many outsiders experience the way I work as quite odd and consider it to be inefficient.

The first reason is (probably) because I like the command-line for many tasks and I frequently use a unconventional text editor, which some people don't understand. Second, I also often use Nix (and related Nix utilities) during development.

In this blog post, I'd like to elaborate a bit about the second aspect and discuss why it is actually quite useful to do this.

Obtaining and installing the dependencies of a software project


In all software projects I have been involved with so far, the first thing I typically had to do is installing all its required dependencies. Examples of such dependencies are a compiler/interpreter + runtime environment for a specific programming language (such as GCC, the Java Development Kit, Perl, Python, Node.js etc.), a development IDE (such as Eclipse), and many library packages.

I have experienced that this step is often quite time consuming, since many dependencies have to be downloaded and installed. Moreover, I have frequently encountered situations in which a bunch of special settings have to be configured to make everything work right, which can be quite cumbersome and tedious.

Many developers just simply download and install everything on their machine in an ad-hoc manner and manually perform all the steps they have to do.

Automating the deployment of dependencies of a software project


What I often do instead is to immediately use Nix to automate the process of installing all software project dependencies and configuring all their settings.

To fully automate deployment with Nix, all dependencies have to be provided as Nix packages. First, I investigate if everything I need is in the Nix packages collection. Fortunately, the Nix package collection provides packages for many kinds of programming languages and associated libraries. If anything is missing, I package it myself. Although some things are very obscure to package (such as the Android SDK), most things can be packaged in a straight forward manner.

After packaging all the missing dependencies, I must install them and generate an environment in which all the dependencies can be found. For example, in order to let Java projects find their library dependencies, we must set the CLASSPATH environment variable to refer to directories or JAR files containing our required compiled classes, PERL5LIB to let Perl find its Perl modules, NODE_PATH to let Node.js find its CommonJS modules etc.

The Nix packages collection has a nice utility function called myEnvFun that can be used to automatically do this. This function can be used to compose development environments that automatically set nearly all of its required development settings. The following example Nix expression (disnixenv.nix) can be used to deploy a development environment for Disnix:

with import <nixpkgs> {};

myEnvFun {
  name = "disnix";
  buildInputs = [
    pkgconfig dbus dbus_glib libxml2 libxslt getopt
    autoconf automake libtool dysnomia
  ];
}

The development environment can be deployed by installing it with the Nix package manager:

$ nix-env -f disnixenv.nix -i env-disnix

The above command-line invocation deploys all Disnix's dependencies and generates a shell script (named: load-env-disnix) that launches a shell session in an environment in which all build settings are configured. We can enter this development environment by running:

$ load-env-disnix
env-disnix loaded

disnix:[sander@nixos:~]$

As can be seen in the code fragment above, we have entered an environment in which all its development dependencies are present and configured. For example, the following command-line instruction should work, since libxml2 is declared as a development dependency:

$ xmllint --version
xmllint: using libxml version 20901

We can also unpack the Disnix source tarball and run the following build instructions, which should work without any trouble:

$ ./bootstrap
$ ./configure
$ make

Besides deploying a development environment, we can also discard it if we don't need it anymore by running:

$ nix-env -e env-disnix

Automating common deployment tasks


After deploying a development environment, I'm usually doing all kinds of development tasks until the project reaches a more stable state. Once this is the case, I immediately automate most of its deployment operations in a Nix expression (that is often called: release.nix).

This Nix expression is typically an attribute set following Hydra's (the Nix-based continuous integration server) convention in which each attribute represents a deployment job. Jobs that I typically implement are:

  • Source package. This is a job that helps me to easily redistribute the source code of a project to others. For GNU Autotools based projects this typically involves the: make dist command-line instruction. Besides GNU Autotools, I also generate source packages for other kinds of projects. For example, in one of my Node.js projects (such a NiJS) I produce a TGZ file that can be deployed with the NPM package manager.
  • Binary package jobs that actually build the software project for every target system that I want to support, such as i686-linux, x86_64-linux, and x86_64-darwin.
  • A job that automatically compiles the program manual from e.g. Docbook, if this applicable.
  • A program documentation catalog job. Many programming languages allow developers to write code-level documentation in comments from which a documentation catalog can be generated, e.g. through javadoc, doxygen or JSDuck. I also create a job that takes care of doing that.
  • Unit tests. If my project has a unit test suite, I also create a job executing those, since it is required to deploy most of its development dependencies to run the test suite as well.
  • System integration tests. If I have any system integration tests that can run be on Linux, I try to implement a job using the NixOS test driver to run those. The NixOS test driver also automatically deploys all the required environmental dependencies in a VM, such as a DBMS, web server etc. in such a way that you can run them as a unprivileged user without affecting the host system's configuration.

After writing a release Nix expression, I can use the Nix package manager to perform all the deployment tasks for my software project. For example, I can run the following job that comes with Disnix to build a source tarball:

$ nix-build release.nix -A tarball

Another advantage is that I can use the same expression in combination with Hydra to so that I can continuously build and test it.

Spawning development environments from Nix jobs


Apart from being able to deploy a software project and its dependencies, another nice advantage of having its deployment automated through Nix is that I can use the same Nix jobs to reproduce its build environment. By replacing nix-build with nix-shell all its dependencies are deployed, but instead of building the package itself, a shell session is started in which all its dependencies are configured:

$ nix-shell release.nix -A tarball

The above command-line invocation produces a similar build environment as shown earlier with myEnvFun by just using a Nix expression for an existing job.

Discussion


So far I have explained what I typically do with the Nix package manager during development. So why is this actually useful?

  • Besides deploying systems into production environments, setting up a development environment is a similar problem with similar complexities. For me it is logical that I solve these kind of problems the same way as I do in production environments.
  • Packaging all software project dependencies in Nix ensures reliable and reproducible deployment in production environments as well, since Nix ensures dependency completeness and removes as many side effects as possible. In Nix, we have strong guarantees that a system deployed through Nix in production behaves exactly the same in development and vice-versa.
  • We can also share development environments, which is useful for a variety of reasons. For example, we can reproduce the same development environment on a second machine, or give it to another developer to prevent him spending a lot of time in setting up his development environment.
  • Supporting continuous integration and testing with Hydra comes (almost) for free.
  • With relatively little effort you can also enable distributed deployment of your software project with Disnix and/or NixOps.

Although I have encountered people telling me that it's consuming too much time, you also get a lot for it in return. Moreover, in most common scenarios I don't think the effort of automating the deployment of a development environment is that big if you have experience with it.

As a concluding remark, I know that things like Agile software development and continuous delivery of new versions is something that (nearly) everybody wants these days. To implement continuous delivery/deployment, the first step is to automate the deployment of a software project from the very beginning. That is exactly what I'm doing by executing the steps described in this blog post.

People may also want to read an old blog post that contains recommendations to improve software deployment processes.

Thursday, December 12, 2013

Asynchronous programming with JavaScript (part 2)

A while ago, I wrote a blog post about my asynchronous programming experiences/frustrations with JavaScript. The main message of this blog post is that if we want to run stuff concurrently, we have to implement cooperative multitasking by generating events (such as timeouts or ticks) and then stop the execution so that the events get processed that invoke callbacks to resume execution.

As a consequence of using callbacks, it has become much harder to structure code properly. Without any proper software abstractions, we may end up writing pyramid code that is hard to write, read, adapt and maintain. One of the solutions to alleviate such problems is by using software abstractions implemented by libraries such as async.

Recently, I've been experimenting with several JavaScript libraries/frameworks (such as AngularJS, Selenium JavaScript webdriver, and jQuery) and I went to a Titanium SDK (a cross-platform JavaScript framework for developing apps) developer meetup. I have noticed that yet another software abstraction has become quite popular in these worlds.

Promises


The software abstraction I am referring to is called "promises". To get some understanding on what they are supposed to mean, I will cite the Promises/A proposal, that is generally regarded as its official specification:

A promise represents the eventual value returned from the single completion of an operation.

To me this description looks quite abstract. Fortunately, the proposal also outlines a few practical bits. It turns out that promises are supposed to be encoded as objects having the following characteristics:

  • A promise can be in one of the following three states: unfulfilled, fulfilled, and failed. A promise in an unfulfilled state can either transition to a fullfilled state or failed state. The opposite transitions are not possible -- once a promise is fulfilled or failed it should remain in that state.
  • A promise object has a then member referring to a function with the following signature:
    .then(onFullfilled, onError, onProgress);
    
  • The then function parameters are all optional and refer to functions with the following purposes:

    • onFullfilled: A function that gets invoked once the promise reaches its fulfilled state.
    • onError: A function that gets invoked once the promise reaches its failed state.
    • onProgress: A function that may get invoked while the promise is still unfulfilled. Promises are actually not required to support this function at all.

    If any of these parameters is not a function, then they are ignored.
  • Each then function is supposed to return another promise (which also have a then property), allowing users to "chain" multiple then function invocations together. This is how promises allow someone to structure code more properly and prevent people from writing pyramid code, because chaining can be done at the same indentation level.

Semantic issues


Although some of the promise characteristics listed above may look useful, it has turned out that the Promises/A proposal has a number of weaknesses as well.

First of all, it is just a proposal containing some random notes and not an official specification. Furthermore, it has been written in an informal way and it is incomplete and underspecified. For example, it leaves a few semantic issues open, such as what should happen if a promise enters a failed state, when a particular then function invocation in a chain does not implement an onError callback? As a consequence, this has lead to several implementations having different and incompatible behaviour.

To address some of these issues, a second specification has been developed called the Promises/A+ specification. This specification is not intended to completely replace the Promises/A proposal, but instead it is concentrated mostly on the semantics of the then function (not including the onProgress parameter). Moreover, this specification is also written in a much more formal and concise way in my opinion.

For example, the Promises/A+ specification adds the following details:

  • It specifies the ability to invoke a promise's then function multiple times.
  • Propagation of results and errors in a then function invocation chain, if any of their callback function parameters are not set.

When using Promises/A+ compliant promise objects, error propagation works in a similar way compared to a synchronous try { } catch { } block allowing one to execute a group of instructions and use a separate block at the end to catch any exceptions that may be thrown while executing these.

An example: Using the Selenium webdriver


To demonstrate how promises can be used, I have developed an example case with the JavaScript Selenium webdriver, a web browser automation framework used frequently for testing web applications. This package uses promises to encode query operations.

Our testcase works on a webpage that contains a form allowing someone to enter its first and last name. By default, they contain my first and last name:


The corresponding HTML code looks as follows:

<!DOCTYPE html>

<html>
    <head><title>Selenium test</title></head>
    
    <body>
        <h1>Selenium test</h1>
        
        <p>Please enter your names:</p>
        
        <form action="test.html" method="post">
            <p>
                <label>First name</label>
                <input type="text" name="firstname" value="Sander">
            </p>
            <p>
                <label>Last name</label>
                <input type="text" name="lastname" value="van der Burg">
            </p>
            <button type="submit" id="submit">Submit</button>
        </form>
    </body>
</html>

To automatically retrieve the values of both input fields and to click on the submit button, I have written the following JavaScript code:

var webdriver = require('selenium-webdriver');
var remote = require('selenium-webdriver/remote');
var webdrvr = require('webdrvr');

/* Create Selenium server */
var server = new remote.SeleniumServer(webdrvr.selenium.path, {
    args: webdrvr.args
});

/* Start the server */
server.start();

/* Set up a Chrome driver */
var driver = new webdriver.Builder()
    .usingServer(server.address())
    .withCapabilities(webdriver.Capabilities.chrome())
    .build();

/* Execute some tests */

driver.get('file://'+process.cwd()+'/test.html')
.then(function() {
    return driver.findElement(webdriver.By.name('firstname'));
})
.then(function(firstNameInput) {
    return firstNameInput.getAttribute('value');
})
.then(function(firstName) {
    console.log("First name: "+firstName);
    return driver.findElement(webdriver.By.name('lastname'));
})
.then(function(lastNameInput) {
    return lastNameInput.getAttribute('value');
})
.then(function(lastName) {
    console.log("Last name: "+lastName);
    return driver.findElement(webdriver.By.id('submit'));
})
.then(function(submitButton) {
    return submitButton.click();
}, function(err) {
    console.log("Some error occured: "+err);
});

/* Stop the server */
server.stop();

As may be observed from the code fragment above, besides setting up a test driver that uses Google Chrome, all the test operations are encoded as a chain of then function invocations each returning a promise:
  • First, we instruct the test driver to get our example HTML page containing the form by creating a promise.
  • After the page has been opened, we create and return a promise that queries the HTML input element that is used for filling in a first name.
  • Then we retrieve the value of the first name input field, by creating and returning a promise that fetches the value attribute of the HTML input element.
  • We repeat the previous two steps in a similar way to fetch the last name as well.
  • Finally, we create and return promises that queries the submit button and clicks on it.
  • In the last then function invocation, we define a onError callback that catches any errors that might occur when executing any of the previous steps.

As can be observed, we do not have to write any nested callback blocks that result in horrible pyramid code.

Implementing promises


The Selenium example shows us how asynchronous test code can be better structured using promises. The next thing I was thinking about is how can I use promises to fix structuring issues in my own code?

It turns out that there are many ways to do this. In fact, the Promise/A and Promise/A+ specifications are just merely interface specifications and leave it up to developers to properly implement them. For example, it can be done completely manually, but there are also several libraries available providing abstractions for that, such as Q, RSVP, and when.

As an experiment, I have restructured the following pyramid example code fragment from my previous blog post (that I originally used to restructure with async) into a better structured code fragment with promises:

var fs = require('fs');
var path = require('path');

fs.mkdir("out", 0755, function(err) {
    if(err) throw err;
    
    fs.mkdir(path.join("out, "test"), 0755, function(err) {
        if (err) throw err;        
        var filename = path.join("out", "test", "hello.txt");

        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) throw err;
                    
            fs.readFile(filename, function(err, data) {
                if(err) throw err;
                
                if(data == "Hello world!")
                    process.stderr.write("File is correct!\n");
                else
                    process.stderr.write("File is incorrect!\n");
            });
        });
    });
});

I have picked the RSVP library to implement promises, since it claims to be lightweight and fully Promises/A+ compliant. The restructured code looks as follows:

var fs = require('fs');
var path = require('path');
var Promise = require('rsvp').Promise;

/* Promise object definitions */

var mkdir = function(dirname) {
    return new Promise(function(resolve, reject) {
        fs.mkdir(dirname, 0755, function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var writeHelloTxt = function(filename) {
    return new Promise(function(resolve, reject) {
        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var readHelloTxt = function(filename) {
    return new Promise(function(resolve, reject) {
        fs.readFile(filename, function(err, data) {
            if(err) reject(err);
            else resolve(data);
        });
    });
};

/* Promise execution chain */

var filename = path.join("out", "test", "hello.txt");

mkdir(path.join("out"))
.then(function() {
    return mkdir(path.join("out", "test"));
})
.then(function() {
    return writeHelloTxt(filename);
})
.then(function() {
    return readHelloTxt(filename);
})
.then(function(data) {
    if(data == "Hello world!")
        process.stderr.write("File is correct!\n");
    else
        process.stderr.write("File is incorrect!\n");
}, function(err) {
    console.log("An error occured: "+err);
});

The revised code fragment above is structured as follows:

  • The overall structure consists of two parts: First we define a number of promises and then we execute them by composing a then function invocation chain.
  • I have encapsulated most of the operations from the previous code fragment into reusable promises: one makes a directory, one writes a text file, one reads the text file.
  • A promise is created by instantiating the rsvp.Promise prototype. The constructor function always takes the same function as a parameter taking resolve and reject as parameters. resolve is a callback that can be used to notify a caller that it has reached its fulfilled state. The reject parameter notifies a caller that it has reached its failed state. Both of these functions can take a value as parameter that can be passed to a then function invocation.
  • To be able to parametrize these promises, I have encapsulated them in functions
  • In the remainder of the code, I instantiate the promises and execute them in a then function invocation chain. As can be observed, we have no pyramid code.
  • However, I observed that our refactored code has a big drawback as well -- it has grown significantly in size which can be considered a deterioration compared to the pyramid code fragment.

Discussion


So far, I have given an explanation of what promises are supposed to mean, how they can be used and how they can be created. We have seen that code structuring issues are "solved" by the fact that then function invocations can be chained.

In my previous blog post, I have explored the async library as a means to solve the same kind of issues, which make me wonder which approach is best? To me, the approach of chaining then function invocations looks quite similar to async's waterfall pattern. However, I find the promise approach slightly better, because it uses return values instead of callbacks preventing us to accidentally reach limbo state if we forget to call them.

However, I think async's waterfall pattern has a lower usage barrier because we do not have to capture all the steps we want into promise objects resulting in less overhead of code.

I think that using promises to structure code works best for things that can be encoded as reusable objects, such as those used to implement a system having a Model-View-Controller (MVC) pattern, that e.g. pass reusable objects from the model to the controller and view. In situations in which we just have to execute a collection of ad-hoc operations (such as those in my example) it simply produces a lot of extra overhead.

Finally, async's waterfall pattern as well as chaining then function invocations work well for things that consist of a fixed number of steps that have to be executed sequentially. However, it may also be desirable to execute steps in parallel, or to use asynchronous for and while loops. The promises specifications do not provide any solutions to cope with these kind of code structuring issues.

Some other thoughts


After doing these experiments and writing this blog post, I still have some remaining thoughts:

  • I have noticed that promises are considered the "the next great paradigm" by some people. This actually makes me wonder what the "previous great paradigm" was. Moreover, I think promises are only useful in certain cases, but definitely not for all JavaScript's asynchronous programming problems.
  • Another interesting observation I did is not related to JavaScript or asynchronous programming in general. I have observed that writing specifications, which purpose is to specify what kind of requirements a solution should fulfill, is a skill that is not present with quite a few people doing software engineering. It's not really surprising to me that the Promise/A proposal leads to so much issues. Fortunately, the Promises/A+ authors demonstrate how to properly write a specification (plus that they are aware of the fact that specifications always have limitations and issues).
  • I'm still surprised how many solutions are being developed for a problem that is non-existent in other programming environments. Actually, I consider these software abstractions dealing with asynchronous programming "workarounds", not solutions. Because none of them is ideal there are many of them (the joyent Node wiki currently lists 79 libraries and I know there are more of them), I expect many more to come.
  • Personally, I consider the best solution a runtime environment and/or programming language that has facilities to cope with concurrency. It is really surprising to me that nobody is working on getting that done. UPDATE: There is a bit of light at the end of the tunnel according to Zef's blog titled: "Callback-Free Harmonious Node.js" post that I just read.
UPDATE: I have written a follow up blog post in which I take a look at asynchronous programming in a more fundamental way. I also derived my own abstraction library from my observations.

Thursday, November 7, 2013

Emulating Amiga display modes

A while ago, I have written a blog post about my IFF file format experiments in which I have developed a collection of libraries and tools capable of parsing and displaying two IFF file format applications. Some time before that, I have developed a Nix function capable of building software for AmigaOS allowing me to easily backport these experimental software packages to AmigaOS.

The development of these experimental packages were dormant for quite some time, but recently I was able to find some time to do some improvements. This time I have updated the Amiga video emulation library to support a few new features and to more accurately emulate an Amiga display.

Differences between Amiga and PC displays


So why is emulation of Amiga video modes necessary? Emulation of Amiga graphics hardware is required because pixels are encoded differently than "modern" PC hardware, resolutions have different semantics, and Amiga hardware has a few special screen modes to "squeeze" more colors out of the amount of available color registers.

Pixels


Encoding pixels on modern hardware is not so complicated these days. Modern hardware allows us to address each individual pixel's color value using 4 bytes per pixel in which one byte represents the red color intensity, one byte the blue color intensity and one byte the green color intensity. The fourth byte is often unused, or serves as the alpha component setting the transparency of a pixel in multi-layered images.

However, older PC hardware, such as those used during the DOS era, were often not powerful enough to address every pixel's color value individually, mainly due to memory constraints. Many classic DOS games used a 320x200 video resolution with a color palette consisting of 256 color values. Each pixel was encoded as a byte referring to an index value of a palette. This encoding is also known as chunky graphics.

System resources in the Amiga era were even more scarce, especially during its launch in 1985. The original graphics chipset was only capable of storing 32 color values in a palette, although there were a few special screen modes capable of squeezing more colors out of the 32 configurable ones. It also used a different way of encoding pixels, probably to make storage of graphics surfaces as memory efficient as possible.

In Amiga chipsets pixels are encoded as bitplanes rather than bytes. When using bitplane encoding, an image is stored multiple times in memory. In every image occurence, a bit represents a pixel. In the first occurence, a bit is the least significant bit of an index a value. In the last occurence a bit is the most significant bit of an index value. By adding all the bits together we can determine the index value of the palette to determine a pixel's color value. For example, if we would encode an image using 16 colors, then we need 4 bitplane surfaces.

To be able to display an Amiga image on a PC we have to convert the bitplane format to either chunky graphics or RGB graphics. Likewise, to be able to display a PC image on an Amiga, we have to convert the pixel surface to bitplane format.

Palette


As explained earlier, due to memory constraints, Amiga graphics use a palette of 32 configurable color values (or 256 colors when using the newer AGA chipset is used). Each pixel refers to an index of a palette instead of a color value of its own.

The original chipset color values are stored in 16-bit color registers in which every color component consists of 4 bits (4 bits are unused). The newer AGA chipset as well as VGA graphics use 8 bits per color component.


Furthermore, the Amiga video chips have special screen modes to squeeze more colors out of the amount of color registers available. The Extra Half Brite (EHB) screen mode is capable of displaying 64 colors out of a predefined 32, in which the last 32 colors values have half of the intensity of the first 32 colors. The above screenshot is an example picture included with Deluxe Paint V using the EHB screenmode. A closer look at the floor in the picture look may reveal to you that an EHB palette is used.

The Hold-and-Modify (HAM) screen mode is used to modify a color component of the previous adjacent pixel or to pick a new color from the given palette. This screen mode makes it possible to use all possible color values (4096) in one screen with some loss of image quality.


The above screenshot is an example image included with Deluxe Paint V using HAM screenmode to utilise many more colors than the amount available color registers. A closer look at the image very may reveal that it has some quality loss because of the limitations of HAM compression.

To be able to properly display an image on a PC we need to convert 12-bit color values to 32-bit color values. Moreover, we also have to calculate each pixel's color value when HAM screen modes are used and convert the image to true color graphics, since a picture using HAM mode may use more than 256 colors.

Resolutions


Another difference between Amiga displays and "modern" displays are the display resolutions. On PCs, resolutions refer to the amount of pixels per scanline and the amount of scanlines per screen.

On the Amiga, resolutions only refer to the amount of pixels per scanline and this amount is often fixed. For example most displays support 320 lowres pixels per scanline, although this value can be slightly increased by utilising the overscan region. A high resolution screen has twice the amount of pixels per scanline compared to a low resolution screen. A super hires screen has double the amount of pixels per scanline compared to a high resolution screen. Moreover, a low resolution pixel is twice a wide as a high resolution pixel and so on.

Vertically, there are only two options. In principle, there are a fixed amount of scanlines on a display. Typically, NTSC displays support 200 scanlines and PAL displays support 256 scanlines. This amount can be slightly increased by utilising the overscan region of a display.

The amount of scanlines can be doubled, using a so-called interlace screen mode. However, interlace screen modes have a big drawback -- they draw the odd scanlines in one frame and the even ones in another. On displays with a short after glow, flickering may be observed.

Because of these differences, we may observe odd results in some cases when we convert an Amiga pixel surface to a chunky/RGB pixel surface. For example, a non-interlaced high resolution image looks twice as wide on a PC display than on an Amiga display, as can be seen in the left picture above, which contains a pixel-by-pixel conversion of a the Workbench screenshot.

To give it the same look as an Amiga display, we must correct its aspect ratio by doubling the amount of scanlines on the PC display, which is done in the right screenshot. People who have used the Amiga Workbench will immediately notice that this image is much closer to what can be seen on a real Amiga.

A library for performing conversions of Amiga display surfaces


While writing my previous blog post, I have developed a library (libamivideo) capable of automatically converting Amiga bitplane surfaces to chunky/RGB pixel surface and vice versa. In the latest release, I have added the following features and improvements:

  • Support for super hires video modes
  • Correct aspect ratio support
  • A better defined API

The library can be obtained from my GitHub page. I have also updated the SDL-based ILBM viewer (SDL_ILBM) as well as the native AmigaOS ILBM viewer (amiilbm) to use the new library implementation.