Some time ago, I have used the
Nix package manager to
build and test software packages for AmigaOS, as a fun project. Furthermore, I have announced that
I have switched jobs and that I was exploring the mobile device space. This blog post, is a report on the first step in which I show how to build and emulate
Android Apps through the Nix package manager. The approach is comparable to what I have done with the AmigaOS emulator. I think it may be good to hear that I'm actively turning research into practice!
Packaging the Android SDK
The first step in automating a build process of Android Apps, is to package the
Android SDK as a Nix package, which contains all required utilities for building, packaging and emulating. We must package it (as opposed to referring to an already installed instance), because all build-time dependencies must be handled through Nix in order to achieve reliability and reproducibility.
Unfortunately, the Android SDK is not very trivial to package:
- The Android SDK from the website is not a source package. Google does not seem to provide any proper source releases, except for obtaining the sources from Git yourself. The downloadable distribution is a zip archive with Java JAR files and a hybrid of native i686 and x86_64 executables. Native executables do not work with Nix out of the box, as they try to lookup their run-time dependencies from global locations, which are not present on NixOS. Therefore, they must be patched using PatchELF.
- The Android SDK is not self-contained. It requires developers to install a number of add-ons, such as platform tools, platform SDKs, system images, and support libraries, by running:
$ android update
- In the normal workflow, these additions are downloaded by the android utility and stored in the same base directory as the SDK, which is an imperative action. This conflicts with the Nix deployment model, as components are made immutable after they have been built. Moreover, these additions must be installed non-interactively.
Android SDK base package
I have packaged the Android SDK base package in Nix (which is obtained from the
Android SDK page) by unzipping the zip distribution and by moving the resulting directory into the Nix store. Then I have patched a number of executables, scripts and libraries to allow them to work from the Nix store.
As explained earlier, we cannot run ELF executables out of the box on NixOS, as Nix has no global directories, such as
/usr/lib, in which executables often look for their dependencies. Moreover, the dynamic linker also resides in a different location.
First, we have to patch executables to provide the correct path to the dynamic linker (which is an impurity and does not reside in
/lib). For example, by running
ldd on the ELF executables, we can see that all of them require
libstdc++ (32-bit):
$ ldd ./emulator-x86
linux-gate.so.1 => (0xf76e4000)
libdl.so.2 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libdl.so.2 (0xf76de000)
libpthread.so.0 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libpthread.so.0 (0xf76c4000)
librt.so.1 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/librt.so.1 (0xf76af000)
libstdc++.so.6 => not found
libm.so.6 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libm.so.6 (0xf7689000)
libutil.so.1 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libutil.so.1 (0xf7684000)
libgcc_s.so.1 => not found
libc.so.6 => /nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libc.so.6 (0xf7521000)
/nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/ld-linux.so.2 (0xf76e5000)
In order to allow these executables to find a particular library, we have to add its full path (provided by evaluating a derivation) to the
RPATH header of the ELF executable. The following build commands will patch most of the utilities:
cd tools
for i in dmtracedump emulator emulator-arm emulator-x86 hprof-conv \
mksdcard sqlite3
do
patchelf --set-interpreter ${stdenv.gcc.libc}/lib/ld-linux.so.2 $i
patchelf --set-rpath ${stdenv.gcc.gcc}/lib $i
done
Two other tools apparently do zip compression/decompression and require
zlib in addition to
libstdc++:
for i in etc1tool zipalign
do
patchelf --set-interpreter ${stdenv.gcc.libc}/lib/ld-linux.so.2 $i
patchelf --set-rpath ${stdenv.gcc.gcc}/lib:${zlib}/lib $i
done
A shared library used by the monitor (
lib/monitor-x86/libcairo-swt.so) requires many more libraries, which are mostly related to the
GTK+ framework.
In addition to ELF binaries, we also have a number of shell scripts that start Java programs. They have a
shebang refering to the bash shell residing at
/bin/bash, which does not exist on NixOS. By running the
shebangfix tool this line gets replaced to refer to the right Nix store path of
bash:
for i in ddms draw9patch monkeyrunner monitor lint traceview
do
shebangfix $i
done
After performing these patching steps, there are still a bunch of utilities not properly functioning, such as the emulator, showing:
SDL init failure, reason is: No available video device
I have used
strace to check what's going on:
$ strace -f ./emulator
...
rt_sigaction(SIGINT, {SIG_DFL, [INT], SA_RESTART}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [QUIT], SA_RESTART}, {SIG_DFL, [], 0}, 8) = 0
futex(0xfffffffff7705064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
open("./lib/tls/i686/sse2/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/tls/i686/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/tls/sse2/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/tls/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/i686/sse2/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/i686/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/sse2/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("./lib/libX11.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
...
open("/nix/store/cy8rl8h4yp2j3h8987vkklg328q3wmjz-gcc-4.6.3/lib/libXext.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libXext.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
...
open("/nix/store/cy8rl8h4yp2j3h8987vkklg328q3wmjz-gcc-4.6.3/lib/libXrandr.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/nix/store/7dvylm5crlc0sfafcc0n46mb5ch67q0j-glibc-2.13/lib/libXrandr.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "SDL init failure, reason is: No "..., 55SDL init failure, reason is: No available video device
) = 55
unlink("/home/sander/.android/avd/foo.avd/hardware-qemu.ini.lock") = 0
exit_group(1)
Apparently these utilities also open a number of libraries dynamically, such as the ones belonging to the X Window System, which are not in the executable's RPATH. I have fixed this by wrapping the paths to these additional libraries in a shell script that sets the
LD_LIBRARY_PATH environment variable and then executes the real executable, so that these can be found:
for i in emulator emulator-arm emulator-x86
do
wrapProgram `pwd`/$i \
--prefix LD_LIBRARY_PATH : `pwd`/lib:${libX11}/lib:\
${libxcb}/lib:${libXau}/lib:${libXdmcp}/lib:\
${libXext}/lib
done
Supporting plugins and optional packages
As explained earlier, the Android SDK is not self contained and provides many additions and optional packages, depending on what classes of devices a developer wants to support and what features he wants to provide. Apparently, there is no web page to easily download these additions from. Moreover, we do not need all of them. Downloading all possible additions require developers to download many gigabytes of data.
However, running:
$ android list sdk
reveals some interesting information:
Fetching https://dl-ssl.google.com/android/repository/addons_list-2.xml
Validate XML
Parse XML
Fetched Add-ons List successfully
Refresh Sources
Fetching URL: https://dl-ssl.google.com/android/repository/repository-7.xml
Validate XML: https://dl-ssl.google.com/android/repository/repository-7.xml
Parse XML: https://dl-ssl.google.com/android/repository/repository-7.xml
Fetching URL: https://dl-ssl.google.com/android/repository/addon.xml
Validate XML: https://dl-ssl.google.com/android/repository/addon.xml
Parse XML: https://dl-ssl.google.com/android/repository/addon.xml
The output shows that the Android SDK fetches a collection of XML files from URLs providing package information. I have used these XML files to package all the additions I care about in separate Nix expressions.
Platform tools
One important addition that's not in the base package are the platform tools, which contains the
Android debugger and a number of related utilities. The platform-tools' zip distribution is defined in the
repository-7.xml file.
Packaging the platform tools in very straight forward. It must be unzipped and a number of native ELF executables need to be patched, such as
adb. Fortunately, none of them uses dynamically loaded libraries. There is one shell script:
dx that requires a shebang fix.
Finally, the platform tools must be accessible from the
platform-tools directory from the Android SDK basedir. We can easily solve this by creating a symlink from the Android SDK base package to the platform tools package.
Platform SDKs and system images
Apart from the basic tools and platform tools, we have to be able to actually develop Android Apps. Android Apps are developed for a wide range of devices and operating system versions, ranging from the classic Android 1.5 OS to the recent Android 4.2 OS.
In order to be able to build an Android app for a particular device (or a class of devices), we require the appropriate Android SDK version for that particular Android version. Besides building, we also want to use the emulator for testing. The emulator requires the right system image for a particular Android OS version.
It would be very costly to have all Android versions supported by default, which requires developers to download a lot of data, while they often only need a small subset of it. Therefore, we want to package every SDK and system image separately.
Fortunately, the
repository-7.xml XML file contains all the information that we need to do that. For example, each platform SDK is defined in an XML element, such as:
<sdk:sdk-repository ...>
<sdk:platform>
<sdk:version>2.2</sdk:version>
<sdk:api-level>8</sdk:api-level>
<sdk:codename/>
<sdk:revision>03</sdk:revision>
<sdk:min-tools-rev>
<sdk:major>8</sdk:major>
</sdk:min-tools-rev>
<sdk:description>Android SDK Platform 2.2_r3</sdk:description>
<sdk:desc-url>http://developer.android.com/sdk/</sdk:desc-url>
<sdk:archives>
<sdk:archive arch="any" os="any">
<sdk:size>74652366</sdk:size>
<sdk:checksum type="sha1">231262c63eefdff8...</sdk:checksum>
<sdk:url>android-2.2_r03-linux.zip</sdk:url>
</sdk:archive>
</sdk:archives>
<sdk:layoutlib>
<sdk:api>4</sdk:api>
</sdk:layoutlib>
</sdk:platform>
...
</sdk:sdk-repository>
The given XML elements can be transformed into a Nix expression, using
XSL in a straight forward manner:
let buildPlatform = ...
in
{
...
platform_8 = buildPlatform {
name = "android-platform-2.2";
src = fetchurl {
url =
https://dl-ssl.google.com/android/repository/android-2.2_r03-linux.zip;
sha1 = "231262c63eefdff8fd0386e9ccfefeb27a8f9202";
};
meta = {
description = "Android SDK Platform 2.2_r3";
url = http://developer.android.com/sdk/;
};
};
...
}
The resulting Nix expression is an attribute set, in which every attribute refers to a package containing a platform SDK.
The
buildPlatform function simply unzips the zip file and moves the contents into the Nix store. The
<sdk:api-level> is an important element -- it's a unique version number that the Android SDK uses to make a distinction between various Android operating systems and is also used to make the attribute names in the above attribute set unique. As we will see later, we can use the API level number and this naming convention to relate optional components to a particular Android OS version.
To make a specific platform SDK available to developers, we must symlink it into the
platforms/android-<api-level> directory of the Android base package.
For the system images, a similar approach is used that generates an attribute set in which each attribute refers to a system image package. Here, also a
<sdk:api-level> element is defined, that we can use to relate the system image to a particular Android OS version. A system image can be made available by creating a symlink in the
system-images/android-<api-level>.
Other additions
In addition to the platform SDKs and system images, there are many more optional additions, which are defined in the
addon.xml file. For example, to allow Android Apps to use APIs, such as
Google Maps, we need to make these package available as well. The Google API packages are defined in a similar manner in the XML file as the platform SDKs, with an api-level identifier and must be symlinked in into the
addons/addon-google_apis-<api-level> directory of the Android SDK package.
There is also the
support library that exposes certain newer functionality to older Android OSes and some utility APIs. The support library can be made available by symlinking it into
support/ of the Android SDK base package.
Building Android applications
So far, I have described how we can package the Android SDK and its (optional) additions in Nix. How can we use this to automatically build Android Apps through the Nix package manager?
The first important aspect is that the
Android command-line utility must be used to create an Android project, as opposed to using the Eclipse IDE. In addition to a basic project layout, the command-line utility produces an
Apache Ant build file, that can be used to automatically build the project from the command line. An example of this is:
android create project --target android-8 --name MyFirstApp \
--path /home/sander/MyFirstApp --activity MainActivity \
--package com.example.myfirstapp
The above command-line instruction creates a new project targetting the Android API-level 8 (which corresponds to the Android 2.2 platform SDK, as shown earlier), with the name
MyFirstApp, having a
MainActivity and stores the code in the
com.example.myfirstapp Java package.
By running the following command line instruction, an Android application can be built, which produces an APK archive (a zip archive containing all the files belonging to an App) signed with the debugger key:
$ ant debug
To create releases for production use, we also need to
sign an APK with a custom key. A key can be created by running
keytool, part of the Java SDK:
$ keytool --genkeypair --alias sander
If I add the following lines to the
ant.properties file in the project directory, we can automatically sign the APK with a custom key:
key.store=/home/sander/.keystore
key.alias=sander
key.store.password=foobar
key.alias.password=foobar
By running the following command-line instruction:
$ ant release
A signed APK for release is produced.
I have encapsulated all the previous aspects into a Nix function, named:
androidenv.buildApp, which can be used to conveniently build Android apps from source code and a number of specified options. The following code fragment shows an example invocation, building
the trivial Android example application, that I have implemented to test this:
{androidenv}:
androidenv.buildApp {
name = "MyFirstApp";
src = ../../src/myfirstapp;
platformVersions = [ "8" ];
useGoogleAPIs = true;
release = true;
keyStore = /home/sander/keystore;
keyAlias = "sander";
keyStorePassword = "foobar";
keyAliasPassword = "foobar";
}
The expression above looks similar to an ordinary expression -- it defines a function that requires
androidenv containing all Android related properties. In the remainder of the function, we make a function call to
androidenv.buildApp, which can be used to build an App. As function arguments, we provide a name that ends up in the Nix store, a reference to the source code (which resides on the local filesystem), the API-level which we want to target (as we have seen earlier, API-level 8 corresponds to Android OS 2.2) and whether we want to use the Google APIs.
In this example, we have also enabled key signing. If the
release parameter is omitted (it is
false by default), then the remaining arguments are not required and the resulting APK is signed with the debug key. In our example, we provide the location, alias and keystore passwords that we have created with
keytool, so that signing can be done automatically.
As with ordinary expressions, we also have to
compose an Android package:
rec {
androidenv = import ./androidenv { ... };
myfirstapp = import ./myfirstapp {
inherit androidenv;
};
...
}
The above fragment contains a reference to the Android build infrastructure and invokes the earlier build expression with the given
androidenv argument. The App can be built by calling (
pkgs.nix corresponds to the above code fragement):
$ nix-build pkgs.nix -A myfirstapp
/nix/store/11fz1yxx33k9f9ail53cc1n65r1hhzlg-MyFirstApp
$ ls result/
MyFirstApp-release.apk
By running the above command-line instruction the
complete build process is performed. The Android SDK is downloaded and installed, all the required platform SDKs and system images are installed, the App itself is built and signed, and a Nix component is produced containing the signed APK that is ready to be released.
Our build function composes the Android SDK with its optional features using the function parameters, so that only the additions that we need are downloaded and installed, ensuring reliability, reproducibility and efficiency. It would be a waste of time and disk space to download all possible additions, of course. :-)
Emulating Android apps
Besides building Android apps, it is also desirable to
test them using the Android emulator. To run the emulator, we must first create an AVD (Android Virtual Device). On the command-line this can be done by:
$ android create avd -n device -t android-8
The above instruction generates an AVD named
device targeting the Android API-level 8 (Android 2.2 OS). If we want to use the Google APIs, then we have to pick a different target, which is named: "
Google Inc.:Google APIs:8", if which the integer represents the API-level.
Then we have to start the emulator representing the generated AVD:
$ emulator -avd device -no-boot-anim -port 5554
The above command-line instruction starts the emulator running our AVD, without displaying a boot animation. The debugger interface uses TCP port 5554. (As a sidenote, TCP ports are an impurity inside Nix expressions and it seems that the emulator cannot use Unix domain sockets. In order to cope with this, I wrote a procedure that scans for a free TCP port in the even number range between 5554-5584, by grepping the output of:
adb devices).
When we start the emulator, we have to wait until its booted so that we can install our generated APK. I have discovered that the
Android debugger can wait until a device has reached it's
device state, so that the Android debugger is ready to talk to the emulator. This can be done by the following command-line instruction (the
-s parameter provides the serial for our recently spawned emulator instance, which is composed of the string 'emulator' and the assigned port number shown earlier):
$ adb -s emulator-5554 wait-for-device
Although the device state has been reached, the device is not guaranteed to be booted. By running
getprop command-line tool remotely on the device, we can query various device properties. When the device has been fully booted, the
dev.bootcomplete should be
1, e.g.:
$ adb -s emulator-5554 shell getprop dev.bootcomplete
1
Then we should be able to
install our APK through the debugger, and should we be able to pick it from the application menu on the device:
$ adb -s emulator-5554 install result/MyFirstApp-release.apk
Finally, we must launch the application, which is done by launching the start activity of an App. We can do this automatically, by remotely calling
am that creates an intent to launch the start activity (
MainActivity in the example that we have used) from our App package (
com.example.my.first.app):
$ adb -s emulator-5554 shell am start -a android.intent.action.MAIN \
-n com.example.my.first.app/.MainActivity
Because we always have to compose a SDK having all our desired additions and due to the fact that we have to execute a lot of steps, I have decided to conveniently automate this procedure. I have developed a function called:
androidenv.emulateApp encapsulating these. The following Nix expression shows how it can be invoked:
{androidenv, myfirstapp}:
androidenv.emulateApp {
name = "MyFirstApp";
app = myfirstapp;
platformVersion = "16";
useGoogleAPIs = false;
package = "com.example.my.first.app";
activity = "MainActivity";
}
The above expression is a function that takes two parameters:
androidenv is the Android build infrastructure,
myfirstapp refers to the build function of the example application, I have shown earlier.
In the remainder of the expression, we invoke the
androidenv.emulateApp function that generates a script that automatically instantiates and launches the emulator and finally deploys our APK in it automatically. Here, we also have to specify which app to use, what API-level we want to target (in this example we target API-level 16, which corresponds to Android 4.1) and whether we want to use the Google APIs). The API-level used for emulation may differ from the level we used for building (i.e. it makes sense to test older Apps on newer devices). Finally, we specify the package name and the name of the main activity, so that we can automatically start the App.
By evaluating this expression and executing the resulting script, an emulator is launched with our example app deployed in it, and it's started automatically:
$ nix-build pkgs.nix -A emulate_myfirstapp
./result/bin/run-test-emulator
The above screenshot shows that it works :-)
Deploying Android apps on real devices
I guess the remaining question is how to deploy Android apps on real devices. After building the App through Nix, the following command-line instruction suffices for me, if I attach my phone to the USB port and I enable debugging on my phone:
$ adb -d install result/MyFirstApp-release.apk
This is the result:
It probably does not look that exciting, but it works!
Conclusion
In this (lengthy, I'm sorry :P) blog post, I have packaged the Android SDK in Nix and a large collection of its additions. Furthermore, I have implemented two Nix functions, that may come in handy for Android App development:
- androidenv.buildApp builds an Android App for a particular class of Android devices.
- androidenv.emulateApp generates a script that launches a particular emulator instance and automatically starts an App in it.
These functions take care of almost the entire deployment process of Android Apps hiding most of its complexity, including all its dependencies and (optional) additions. Due to the unique advantages of Nix, we can safely use multiple variants of SDKs and their libraries next to each other, all dependencies are always guaranteed to be included (if they are specified), we can use laziness and function composition to ensure that only the required dependencies are used (which improves efficiency), and we can easily parallelise builds thanks to the purely functional nature of Nix. Furthermore, this function can also be used in conjunction with
Hydra -- the Nix-based continuous build and integration server to continuously assess the state of Android App code.
The only nasty detail is that the emulator and debugger use TCP ports to communicate with each other, which is an impurity. I have implemented some sort of a work around, but it's not very elegant and has various drawbacks. As far as I know, there is no way to use Unix domain sockets.
I'd like to thank my new employer:
Conference Compass, for giving me the space to develop this and taking interest in the deployment technology I was involved in as a researcher. (hmm 'was'? I'm still involved, and this still is research in some way :-) )
Availability
The Android build infrastructure is part of
Nixpkgs, available under the
MIT license. The
androidenv component can be used by including the Nixpkgs top-level expression. The trivial example case and its composition expression (containing the
myfirstapp and
emulate_myfirstapp attributes) can be obtained from my
Nix Android tests GitHub page.
Presentation
UPDATE: On July 14, 2016 I have given a presentation about this subject at the Nix meetup in Amsterdam. For convenience, I have embedded the sildes into this blog post:
