Some time ago, I have packaged
the Android SDK and some of its plugins in Nix, which was quite a challenge. Moreover, I have developed two Nix functions that can be used to automatically build Android apps and spawn emulator instances.
Meanwhile, I have implemented some major updates since the last blog post. Besides upgrading to a newer Android SDK version, we now also support a bunch of new interesting features.
Android SDK packaging updates
Currently, I have packaged the 22.05 version of the Android SDK (which is the latest version at the time writing this blog post). To package it, I have followed the same recipe as described in the previous blog post, with some additions:
- Newer versions of the Android SDK include x86-64 versions of the emulator executables. I had to patch/wrap these executables to allow them to find the x86-64 libraries they depend on.
- The newer Android SDK has a new sub package called build-tools. Apparently, this sub package contains some tools that were formerly part of platform-tools, such as debugging libraries.
- The GUI front-end of the android utility did not work in the last blog post, which I have fixed by wrapping it, so that it can find its dependencies, such as GTK+.
- There are also x86 and MIPS application binary interface (ABI) system images, but they reside in external package repositories. I have adapted the Nix expression generator to include these as well.
- I have adapted the build and emulation Nix functions to take both API-level and ABI versions into account
- I have implemented some facilities to improve the performance of the emulator instances
Packaging build-tools
The build-tools package contains various tools and libraries that used to belong to platform-tools. It turns out that some of these files, such as the debugging libraries, are API-level specific. The purpose of the build-tools package is to provide API-level specific build tools. Currently, the Android SDK supports two versions: 17 and 18.01 that can be installed next to each other.
I have packaged the 18.01 version in a straight forward manner. The only thing I needed to do is inspecting the dependencies of the executables to the paths to these libraries to the executables' RPATH.
For example, I did this to patch
aapt:
patchelf --set-rpath ${stdenv.gcc.libc}/lib:${stdenv.gcc.gcc}/lib:${stdenv.zlib}/lib aapt
Every version of the build-tools package is symlinked into the Android SDK basedir as follows:
build-tools/android-<version>
Fixing the Android GUI front-end
If the
android tool is called without any parameters, it should show a GUI that allows somebody to download plugins or to configure AVDs. However, the GUI did not work so far. Invoking 'android' gave me the following error message:
$ android
Exception in thread "main" java.lang.UnsatisfiedLinkError:
no swt-pi-gtk-3550 or swt-pi-gtk in swt.library.path,
java.library.path or the jar file
at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
at org.eclipse.swt.internal.gtk.OS.(Unknown Source)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
at org.eclipse.swt.widgets.Display.(Unknown Source)
at com.android.sdkmanager.Main.showSdkManagerWindow(Main.java:346)
at com.android.sdkmanager.Main.doAction(Main.java:334)
at com.android.sdkmanager.Main.run(Main.java:120)
at com.android.sdkmanager.Main.main(Main.java:103)
First, I thought that some file belonging to
SWT was missing, but after manually digging through all Android files I couldn't find it and I gave up. However, after using
strace, I discovered that it's actually trying to load the GTK+ library and some other libraries:
$ strace -f android
...
[pid 8169] stat("/tmp/swtlib-64/libswt-pi-gtk.so", 0x7fe4e0cffb20) = -1 ENOENT (No such file or directory)
[pid 8169] stat("/tmp/swtlib-64/libswt-pi-gtk-3550.so", {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid 8169] stat("/tmp/swtlib-64/libswt-pi-gtk-3550.so", {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid 8169] open("/tmp/swtlib-64/libswt-pi-gtk-3550.so", O_RDONLY|O_CLOEXEC) = 26
[pid 8169] read(26, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\234\3\0\0\0\0\0"..., 832) = 832
[pid 8169] fstat(26, {st_mode=S_IFREG|0755, st_size=452752, ...}) = 0
[pid 8169] mmap(NULL, 1503944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 26, 0) = 0x7fe4b5d7e000
[pid 8169] mprotect(0x7fe4b5dea000, 1044480, PROT_NONE) = 0
[pid 8169] mmap(0x7fe4b5ee9000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 26, 0x6b000) = 0x7fe4b5ee9000
[pid 8169] mmap(0x7fe4b5eec000, 4808, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe4b5eec000
[pid 8169] close(26) = 0
[pid 8169] open("/run/opengl-driver/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8169] open("/run/opengl-driver-32/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 8169] open("/nix/store/zm4bhsm8lprkzvrjgqr0klfkvr21als4-glibc-2.17/lib/libgtk-x11-2.0.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
...
To allow GTK+ and the other missing libraries to be found, I have wrapped the
android utility:
wrapProgram `pwd`/android \
--prefix PATH : ${jdk}/bin \
--prefix LD_LIBRARY_PATH : ${glib}/lib:${gtk}/lib:${libXtst}/lib
Besides the libraries, I also had to wrap
android to use the right JDK version. Apparently, loading these libraries only works with
OpenJDK in Nix.
After wrapping the
android utility and running it without parameters, it shows me the following:
As you may see in the above image, the GUI works. It also shows us all the plugins that are installed through Nix. Although the GUI seems to work fine, we have to keep in mind that we cannot use it to install additional Android SDK plugins. All plugins must be packaged and installed with Nix instead.
Supporting external system images
The Android distribution provides a collection of system images that can be used with an emulator to test an app during development. However, the default repository only contains ARM-based system images, which are quite slow to emulate.
There also seem to be x86-based system images available (provided by
Intel) and MIPS-based images (provided by
MIPS Technologies). These system images reside in external repositories. I was able to discover the locations of these repositories, by running the following command:
$ android list sdk
Refresh Sources:
Fetching https://dl-ssl.google.com/android/repository/addons_list-2.xml
Validate XML
Parse XML
Fetched Add-ons List successfully
Refresh Sources
Fetching URL: https://dl-ssl.google.com/android/repository/repository-8.xml
Validate XML: https://dl-ssl.google.com/android/repository/repository-8.xml
Parse XML: https://dl-ssl.google.com/android/repository/repository-8.xml
Fetching URL: https://dl-ssl.google.com/android/repository/addon.xml
Validate XML: https://dl-ssl.google.com/android/repository/addon.xml
Parse XML: https://dl-ssl.google.com/android/repository/addon.xml
Fetching URL: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
Validate XML: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
Parse XML: https://dl-ssl.google.com/android/repository/extras/intel/addon.xml
Fetching URL: https://dl-ssl.google.com/android/repository/sys-img.xml
Validate XML: https://dl-ssl.google.com/android/repository/sys-img.xml
Parse XML: https://dl-ssl.google.com/android/repository/sys-img.xml
Fetching URL: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
Validate XML: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
Parse XML: https://dl-ssl.google.com/android/repository/sys-img/mips/sys-img.xml
Fetching URL: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
Validate XML: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
Parse XML: https://dl-ssl.google.com/android/repository/sys-img/x86/sys-img.xml
The latter two URLs referring to a file called
sys-img.xml provide x86 and MIPS images. The structure of these XML files are similar to the main repository. Both files contain XML elements that look like this:
<sdk:system-image>
<sdk:description>Android SDK Platform 4.1.1</sdk:description>
<sdk:revision>1</sdk:revision>
<sdk:api-level>16</sdk:api-level>
<sdk:abi>x86</sdk:abi>
<sdk:uses-license ref="intel-android-sysimage-license">
<sdk:archives>
<sdk:archive arch="any" os="any">
<sdk:size>131840348</sdk:size>
<sdk:checksum type="sha1">9d35bcaa4f9b40443941f32b8a50337f413c021a</sdk:checksum>
<sdk:url>sysimg_x86-16_r01.zip</sdk:url>
</sdk:archive>
</sdk:archives>
</sdk:uses-license></sdk:system-image>
These elements can be converted to a Nix expression through
XSL in a straight forward manner:
{stdenv, fetchurl, unzip}:
let
buildSystemImage = args:
stdenv.mkDerivation (args // {
buildInputs = [ unzip ];
buildCommand = ''
mkdir -p $out
cd $out
unzip $src
'';
});
in
{
sysimg_x86_16 = buildSystemImage {
name = "sysimg-x86-16";
src = fetchurl {
url = https://dl-ssl.google.com/android/repository/sys-img/x86/sysimg_x86-16_r01.zip;
sha1 = "9d35bcaa4f9b40443941f32b8a50337f413c021a";
};
};
...
}
The generated expression shown above is nearly identical to the one shown in the previous blog post. The only difference is that we also use the ABI identifier in the attribute names. For example:
sysimg_x86_16 is used to refer to a x86-based system image for Android API-level 16. Likewise, we can refer to the ARM-based variant with:
sysimg_armeabi-v7a_16.
Apart from giving each attribute a proper name, the resulting system image package must be symlinked into the Android SDK basedir taking the ABI identifier into account. We symlink every architecture-dependent system image package into:
system-images/android-<api-level>/<abi>.
I have adapted the
androidenv.emulateApp {} Nix function to also take ABI versions into account. By default, the emulator function uses an ARM-based system image (
armeabi-v7a), since this is the most common hardware architecture used by Android OSes. By setting the
abiVersion parameter to
x86, we can use an x86-based Android system-image:
{androidenv, myfirstapp}:
androidenv.emulateApp {
name = "MyFirstApp";
app = myfirstapp;
platformVersion = "16";
abiVersion = "x86";
package = "com.example.my.first.app";
activity = "MainActivity";
}
An interesting benefit of using an
x86 system image with the emulator (which is based on
QEMU) is that
KVM can be used, allowing programs inside the emulator to run at nearly native speed with only little emulation.
And of course, setting the
abiVersion parameter to
mips makes it possible to use MIPS-based system images.
Enabling GPU acceleration
The
x86 system images in the emulator may give testing a significant boost. Emulation performance can even be improved a bit further by enabling GPU acceleration.
To make GPU acceleration possible, I had to adapt the emulator's wrapper script to include
Mesa in the linker path. Moreover, in the AVD configuration I have to add the following boolean value to
$ANDROID_SDK_HOME/.android/avd/device.avd/config.ini:
hw.gpu.enabled = yes
The
androidenv.emulateApp {} automatically enables GPU acceleration from API-levels 15 and onwards. It can be disabled by setting the
enableGPU parameter to false.
Supporting Google APIs in non-ARM-based system images
Although supporting
x86 system images have various benefits, they also have a major drawback -- the Google APIs are not supported preventing some applications from working. One of the solutions I have seen (
described in this Stack Overflow post) is to create a custom system image by starting an ARM-based emulator instance with Google APIs, fetch the libraries, start an x86-based emulator instance without Google APIs and manually install the fetched libraries.
I was able to automate the steps in the Stackflow article in a Nix expression:
{ platformVersion ? "16"
, abiVersion ? "x86"
, androidenv
, jdk
}:
let
androidsdk = androidenv.androidsdk {
platformVersions = [];
abiVersions = [];
useGoogleAPIs = false;
};
emulateAndroidARMWithGoogleAPIs = androidenv.emulateApp {
name = "emulate-arm";
inherit platformVersion;
abiVersion = "armeabi-v7a";
useGoogleAPIs = true;
};
emulateAndroidX86WithoutGoogleAPIs = androidenv.emulateApp {
name = "emulate-x86";
inherit platformVersion abiVersion;
useGoogleAPIs = false;
};
mkfsYaffs2X86 = fetchurl {
url = http://android-group-korea.googlecode.com/files/mkfs.yaffs2.x86;
sha1 = "9362064c10a87ca68de688f09b162b171c55c66f";
};
in
stdenv.mkDerivation {
name = "sysimg_x86_${platformVersion}-with-google-apis";
buildInputs = [ jdk androidsdk ];
NIX_ANDROID_EMULATOR_FLAGS = "-no-window";
buildCommand = ''
source ${emulateAndroidARMWithGoogleAPIs}/bin/run-test-emulator
portARM=$port
adb -s emulator-$portARM pull /system/etc/permissions/com.google.android.maps.xml
adb -s emulator-$portARM pull /system/framework/com.google.android.maps.jar
adb -s emulator-$portARM emu kill
export NIX_ANDROID_EMULATOR_FLAGS="$NIX_ANDROID_EMULATOR_FLAGS -partition-size 1024 -no-snapshot-save"
source ${emulateAndroidX86WithoutGoogleAPIs}/bin/run-test-emulator
portX86=$port
adb -s emulator-$portX86 remount rw
adb -s emulator-$portX86 push com.google.android.maps.jar /system/framework
adb -s emulator-$portX86 push com.google.android.maps.xml /system/etc/permissions
cp ${mkfsYaffs2X86} mkfs.yaffs2.x86
adb -s emulator-$portX86 push mkfs.yaffs2.x86 /data
adb -s emulator-$portX86 shell chmod 777 /data/mkfs.yaffs2.x86
adb -s emulator-$portX86 shell /data/mkfs.yaffs2.x86 /system /data/system.img
adb -s emulator-$portX86 pull /data/system.img
adb -s emulator-$portX86 emu kill
mkdir -p $out
cp system.img $out
'';
}
The above Nix function creates an ARM emulator instance with Google APIs and an x86 emulator instance without Google APIs. Then it starts the ARM emulator instance, fetches the Google API files from it and then stops it. Then the x86 emulator instance is started and the Google APIs are pushed to it. Finally we generate a system image from the system folder that is stored in the Nix store.
By using the Nix function shown above and setting the
extraAVDFiles parameter of the
emulateApp {} function, we can use our custom system image in a script that automatically spawns the emulator instance:
{androidenv, jdk}:
let
platformVersion = "16";
systemImg = import ./generateGoogleAPISystemImage.nix {
inherit androidenv jdk platformVersion;
abiVersion = "x86";
};
in
androidenv.emulateApp {
name = "MyFirstApp";
extraAVDFiles = [ "${systemImg}/system.img" ];
inherit platformVersion;
abiVersion = "x86";
}
The above expression invokes the function shown previously and starts an emulator instance with the system image containing Google APIs. Of course, we can also use the same trick for MIPS-based system images, which also don't include Google APIs.
Building Android apps for arbitrary API-level revisions
I have also made a small change to the
androidenv.buildApp {} Nix function to take API-levels into account. By default, it will use the version that is inside the
Ant configuration. However, by setting the
target property through the command-line, we can override this:
$ ant -Dtarget=android-17
The above command-line instruction ensures that we build the app against the API-level 17 platform. I have added an
antFlags parameter to the Nix function to make it provide arbitrary flags to Ant:
{androidenv}:
androidenv.buildApp {
name = "MyFirstApp";
src = ../../src/myfirstapp;
platformVersions = [ "17" ];
useGoogleAPIs = true;
antFlags = "-Dtarget=android-17";
}
Improving the test suite
In the previous blog post, I have implemented a testcase based on the Android introduction tutorial (
MyFirstApp) that I have used throughout this article as well. To test all these new features, I have extended its composition expression to take three parameters:
buildPlatformVersions (defaults to 16),
emulatePlatformVersions (defaults to 16), and
abiVersions (defaults to
armeabi-v7a). The expression automatically generates all possible combinations with the values inside these three lists:
For example, to build a debug version of the test app APK for API-level 16, we can run:
$ nix-build -A myfirstapp_debug.build_16
We can also build a release version of the same APK that is signed with a key:
$ nix-build -A myfirstapp_release.build_16
Furthermore, we can automatically generate a script starting an emulator instance running the app. The following instruction builds an App for Android API-level 16, generates a script launching an emulator with a Android API-level 16 system-image that uses the
armeabi-v7a ABI:
$ nix-build -A emulate_myfirstapp_debug.build_16.emulate_16.armeabi-v7a
$ ./result/run-test-emulator
To support more API-levels and ABIs, the parameters of the composition function must be altered. For example, by running the following command-line instruction:
$ nix-build --arg buildPlatformVersions '[ "16" "17" ]' \
--arg emulatePlatformVersions '[ "15" "16 " 17" ]' \
--arg abiVersions '[ "armeabi-v7a" "x86" ]' \
-A emulate_myfirstapp_release.build_16.emulate_17.x86
The cartesian product of build and emulator instances is created taking the three dimensions into account. This allows us to (for example) build the App against the API-level 16 platform API, emulate on an API-level 17 system image using the x86 ABI, which emulates much more efficiently on x86 machines, because hardware visualization features (such as KVM) can be used.:
Apart from specifying these dimensions through the command line, the composition expression can also be used together with
Hydra, the Nix-based continuous integration server, allowing it to pass these parameters through its web interface. Hydra will build all the combinations generated by the composition expression automatically:
As can be observed from the above screenshot, quite a few jobs are being generated.
Conclusion
In this blog post, I have described some new changes and features that I have implemented in the Nix Android SDK package since the last blog post, such as the fact that x86 and MIPS system-images can be used. All these changes are part of the
Nixpkgs project.
Moreover, the
Nix Android example testcase can be obtained from my GitHub page.
