Last month, I have been working on quite a lot of things. One of the things I did was improving the
Nix function that builds Titanium SDK applications. In fact, it was in Nixpkgs for quite a while already, but I have never written about it on my blog, apart from a brief reference in
an earlier blog post about Hydra.
The reason that I have decided to write about this function is because the process of getting Titanium applications deployable with Nix is quite painful (although I have managed to do it) and I want to report about my experiences so that these issues can be hopefully resolved in the future.
Although I have a strong opinion on certain aspects of Titanium, this blog post is not to discuss about the development aspects of the Titanium framework. Instead, the focus is on getting the builds of Titanium apps automated.
What is Titanium SDK?
Titanium is an application framework developed by
Appcelerator, which purpose is to enable rapid development of mobile apps for multiple platforms. Currently, Titanium supports
iOS,
Android,
Tizen,
Blackberry and mobile web applications.
With Titanium, developers use JavaScript as an implementation language. The JavaScript code is packaged along with the produced app bundles, deployed to an emulator or device and interpreted there. For example, on Android Google's
V8 JavaScript runtime is used, and on iOS Apple's
JavaScriptCore is used.
Besides using JavaScript code, Titanium also provides an API supporting database access and (fairly) cross platform GUI widgets that have a (sort of) native look on each platform.
Titanium is not a write once run anywhere approach when it comes to cross platform support, but claims that 60-90% of the app code can be reused among platforms.
Finally, the Titanium Studio software distribution is proprietary software, but most of its underlying components (including the Titanium SDK) are free and open-source software available under the
Apache Software License. As far as I can see, the Nix function that I wrote does not depend on any proprietary components, besides the
Java Development Kit.
Packaging the Titanium CLI
The first thing that needs to be done to automate Titanium builds is being able to build stuff from the command-line. Appcelerator provides a command-line utility (CLI) that is specifically designed for this purpose and is provided as a
Node.js package that can be installed through the NPM package manager.
Packaging NPM stuff in Nix is actually quite straight forward and probably the easiest part of getting the builds of Titanium apps automated. Simply adding
titanium to the list of node packages (
pkgs/top-level/node-packages.json) in Nixpkgs and running
npm2nix, a utility developed by Shea Levy that automatically generates Nix expressions for any node package and all their dependencies, did the job for me.
Packaging the Titanium SDK
The next step is packaging the Titanium SDK that contains API libraries, templates and build script plugins for each target platform. The CLI supports multiple SDK versions at the same time and requires at least one version of an SDK installed.
I've obtained an SDK version from Appcelerator's
continuous builds page. Since the SDK distributions are ZIP files containing binaries, I have to use the patching/wrapping tricks I have described in a few earlier blog posts again.
The Nix expression I wrote for the SDK basically unzips the 3.2.1 distribution, copies the contents into the Nix store and makes the following changes:
-
The SDK distribution contains a collection of Python scripts that execute build and debugging tasks. However, to be able to run them in NixOS, the shebangs must be changed so that the Python interpreter can be found:
find . -name \*.py | while read i
do
sed -i -e "s|#!/usr/bin/env python|#!${python}/bin/python|" $i
done
-
The SDK contains a subdirectory (mobilesdk/3.2.1.v20140206170116) with a version number and timestamp in it. However, the timestamp is a bit inconvenient, because the Titanium CLI explicitly checks for SDK folders that correspond to a Titanium SDK version number in a Titanium project file (tiapp.xml). Therefore, I strip it out of the directory name to make my life easier:
$ cd mobilesdk/*
$ mv 3.2.1.v20140206170116 3.2.1.GA
-
The Android builder script (mobilesdk/*/android/builder.py) packages certain files into an APK bundle (which is technically a ZIP file).
However, the script throws an exception if it encounters files with timestamps below January 1, 1980, which are not supported by the ZIP file format. This is a problem, because Nix automatically resets timestamps of deployed packages to one second after January 1, 1970 (a.k.a. UNIX-time: 1) to make builds more deterministic. To remedy the issue, I had to modify several pieces of the builder script.
What I basically did to fix this is searching for invocations to ZipFile.write() that adds a file from the filesystem to a zip archive, such as:
apk_zip.write(os.path.join(lib_source_dir, 'libtiverify.so'), lib_dest_dir + 'libtiverify.so')
I refactored such invocations into a code fragment using a file stream:
info = zipfile.ZipInfo(lib_dest_dir + 'libtiverify.so')
info.compress_type = zipfile.ZIP_DEFLATED
info.create_system = 3
tf = open(os.path.join(lib_source_dir, 'libtiverify.so'))
apk_zip.writestr(info, f.read())
tf.close()
The above code fragment ignores the timestamp of the files to be packaged and uses the current time instead, thus fixing the issue with files that reside in the Nix store.
-
There were two ELF executables (titanium_prep.{linux32,linux64}) in the distribution. To be able to run them under NixOS, I had to patch them so that the dynamic linker can be found:
$ patchelf --set-interpreter ${stdenv.gcc.libc}/lib/ld-linux-x86-64.so.2 \
titanium_prep.linux64
-
The Android builder script (mobilesdk/*/android/builder.py) requires the sqlite3 python module and the Java Development Kit. Since dependencies do not reside in standard locations in Nix, I had to wrap the builder script to allow it to find them:
mv builder.py .builder.py
cat > builder.py <<EOF
#!${python}/bin/python
import os, sys
os.environ['PYTHONPATH'] = '$(echo ${python.modules.sqlite3}/lib/python*/site-packages)'
os.environ['JAVA_HOME'] = '${jdk}/lib/openjdk'
os.execv('$(pwd)/.builder.py', sys.argv)
EOF
Although the Nixpkgs collection has a standard function (wrapProgram) to easily wrap executables, I could not use it, because this function turns any executable into a shell script. The Titanium CLI expects that this builder script is a Python script and will fail if there is a shell code around it.
-
The iOS builder script (mobilesdk/osx/*/iphone/builder.py) invokes ditto to do a recursive copy of a directory hierarchy. However, this executable cannot be found in a Nix builder environment, since the PATH environment variable is set to only the dependencies that are specified. The following command fixes it:
$ sed -i -e "s|ditto|/usr/bin/ditto|g" \
$out/mobilesdk/osx/*/iphone/builder.py
-
When building IPA files for iOS devices, the Titanium CLI invokes xcodebuild, that in turn invokes the Titanium CLI again. However, it does not seem to propagate all parameters properly, such as the path to the CLI's configuration file. The following modification allows me to set an environment variable called: NIX_TITANIUM_WORKAROUND providing additional parameters to work around it:
$ sed -i -e "s|--xcode|--xcode '+process.env['NIX_TITANIUM_WORKAROUND']+'|" \
$out/mobilesdk/osx/*/iphone/cli/commands/_build.js
Building Titanium Apps
Besides getting the Titanium CLI and SDK packaged in Nix, we must also be able to build Titanium apps. Apps can be built for various target platforms and come in several variants.
For some unknown reason, the Titanium CLI (in contrast to the old Python build script) forces people to login with their Appcelerator account, before any build task can be executed. However, I discovered that after logging in a file is written into the
~/.titanium folder indicating that the system has logged in. I can simulate logins by creating this file myself:
export HOME=$TMPDIR
mkdir -p $HOME/.titanium
cat > $HOME/.titanium/auth_session.json <<EOF
{ "loggedIn": true }
EOF
We also have to tell the Titanium CLI where the Titanium SDK can be found. The following command-line instruction updates the config to provide the path to the SDK that we have just packaged:
$ echo "{}" > $TMPDIR/config.json
$ titanium --config-file $TMPDIR/config.json --no-colors \
config sdk.defaultInstallLocation ${titaniumsdk}
The Titanium SDK also contains a collection of prebuilt modules, such as one to connect to Facebook. To allow them to be found, I run the following command line instruction to adapt the module search path:
$ titanium --config-file $TMPDIR/config.json --no-colors \
config paths.modules ${titaniumsdk}
I have also noticed that if the SDK version specified in a Titanium project file (
tiapp.xml) does not match the version of the installed SDK, the Titanium CLI halts with an exception. Of course, the version number in a project file can be adapted, but it in my opinion, it's more flexible to just be able to take any version. The following instruction replaces the version inside
tiapp.xml into something else:
$ sed -i -e "s|<sdk-version>[0-9a-zA-Z\.]*</sdk-version>|<sdk-version>${tiVersion}</sdk-version>|" tiapp.xml
Building Android apps from Titanium projects
For Android builds, we must tell the Titanium CLI where to find the Android SDK. The following command-line instruction adds its location to the config file:
$ titanium config --config-file $TMPDIR/config.json --no-colors \
android.sdkPath ${androidsdkComposition}/libexec/android-sdk-*
The variable:
androidsdkComposition refers to an Android SDK plugin composition provided by the
Android SDK Nix expressions I have developed earlier.
After performing the previous operation, the following command-line instruction can be used to build a debug version of an Android app:
$ titanium build --config-file $TMPDIR/config.json --no-colors --force \
--platform android --target emulator --build-only --output $out
If the above command succeeds, an APK bundle called
app.apk is placed in the Nix store output folder. This bundle contains all the project's JavaScript code and is signed with a developer key.
The following command produces a release version of the APK (meant for submission to the
Play Store) in the Nix store output folder, with a given key store, key alias and key store password:
$ titanium build --config-file $TMPDIR/config.json --no-colors --force \
--platform android --target dist-playstore --keystore ${androidKeyStore} \
--alias ${androidKeyAlias} --password ${androidKeyStorePassword} \
--output-dir $out
Before the JavaScript files are packaged along with the APK file, they are first passed through
Google's Closure Compiler, which performs some static checking, removes dead code, and
minifies all the source files.
Building iOS apps from Titanium projects
Apart from Android, we can also build iOS apps from Titanium projects.
I have discovered that while building for iOS, the Titanium CLI invokes
xcodebuild which in turn invokes the Titanium CLI again. However, it does not propagate the
--config-file parameter, causing it to fail. The earlier hack that I made in the SDK expression with the environment variable can be used to circumvent this:
export NIX_TITANIUM_WORKAROUND="--config-file $TMPDIR/config.json"
After applying the workaround, building an app for the iPhone simulator is straight forward:
$ cp -av * $out
$ cd $out
$ titanium build --config-file $TMPDIR/config.json --force --no-colors \
--platform ios --target simulator --build-only \
--device-family universal --output-dir $out
After running the above command, the simulator executable is placed into the output Nix store folder. It turns out that the JavaScript files of the project folder are symlinked into the folder of the executable. However, after the build has completed these symlink references will become invalid, because the temp folder has been deleted. To allow the app to find these JavaScript files, I simply copy them along with the executable into the Nix store.
Finally, the most complicated task is producing IPA bundles to deploy an app to a device for testing or to the
App Store for distribution.
Like native iOS apps, they must be signed with a certificate and mobile provisioning profile. I used the same trick described in an earlier blog post on
building iOS apps with Nix to generate a temporary keychain in the user's home directory for this:
export HOME=/Users/$(whoami)
export keychainName=$(basename $out)
security create-keychain -p "" $keychainName
security default-keychain -s $keychainName
security unlock-keychain -p "" $keychainName
security import ${iosCertificate} -k $keychainName -P "${iosCertificatePassword}" -A
provisioningId=$(grep UUID -A1 -a ${iosMobileProvisioningProfile} | grep -o "[-A-Z0-9]\{36\}")
if [ ! -f "$HOME/Library/MobileDevice/Provisioning Profiles/$provisioningId.mobileprovision" ]
then
mkdir -p "$HOME/Library/MobileDevice/Provisioning Profiles"
cp ${iosMobileProvisioningProfile} \
"$HOME/Library/MobileDevice/Provisioning Profiles/$provisioningId.mobileprovision"
fi
I also discovered that builds fail, because some file (the facebook module) from the SDK cannot be read (Nix makes deployed package read-only). I circumvented this issue by making a copy of the SDK in my temp folder, fixing the file permissions, and configure the Titanium CLI to use the copied SDK instance:
cp -av ${titaniumsdk} $TMPDIR/titaniumsdk
find $TMPDIR/titaniumsdk | while read i
do
chmod 755 "$i"
done
titanium --config-file $TMPDIR/config.json --no-colors \
config sdk.defaultInstallLocation $TMPDIR/titaniumsdk
Because I cannot use the temp folder as a home directory, I also have to simulate a login again:
$ mkdir -p $HOME/.titanium
$ cat > $HOME/.titanium/auth_session.json <<EOF
{ "loggedIn": true }
EOF
Finally, I can build an IPA by running:
$ titanium build --config-file $TMPDIR/config.json --force --no-colors \
--platform ios --target dist-adhoc --pp-uuid $provisioningId \
--distribution-name "${iosCertificateName}" \
--keychain $HOME/Library/Keychains/$keychainName \
--device-family universal --output-dir $out
The above command-line invocation minifies the JavaScript code, builds an IPA file with a given certificate, mobile provisioning profile and authentication credentials, and puts the result in the Nix store.
Example: KitchenSink
I have encapsulated all the builds commands shown in the previous section into a Nix function called:
titaniumenv.buildApp {}. To test the usefulness of this function, I took
KitchenSink, an example app provided by Appcelerator, to show Titanium's abilities. The App can be deployed to all target platforms that the SDK supports.
To package KitchenSink, I wrote the following expression:
{ titaniumenv, fetchgit
, target, androidPlatformVersions ? [ "11" ], release ? false
}:
titaniumenv.buildApp {
name = "KitchenSink-${target}-${if release then "release" else "debug"}";
src = fetchgit {
url = https://github.com/appcelerator/KitchenSink.git;
rev = "d9f39950c0137a1dd67c925ef9e8046a9f0644ff";
sha256 = "0aj42ac262hw9n9blzhfibg61kkbp3wky69rp2yhd11vwjlcq1qc";
};
tiVersion = "3.2.1.GA";
inherit target androidPlatformVersions release;
androidKeyStore = ./keystore;
androidKeyAlias = "myfirstapp";
androidKeyStorePassword = "mykeystore";
}
The above function fetches the KitchenSink example from GitHub and builds it for a given target, such as
iphone or
android, and supports building a debug version for an emulator/simulator, or a release version for a device or for the Play store/App store.
By invoking the above function as follows, a debug version of the app for Android is produced:
import ./kitchensink {
inherit (pkgs) fetchgit titaniumenv;
target = "android";
release = false;
}
The following function invocation produces an iOS executable that can be run in the iPhone simulator:
import ./kitchensink {
inherit (pkgs) fetchgit titaniumenv;
target = "iphone";
release = false;
}
As may be observed, building KitchenSink through Nix is a straight forward process for most targets. However, the target producing an IPA version of KitchenSink that we can deploy to a real device is a bit complicated to use, because of some restrictions made by Apple.
Since all apps that are deployed to a real device have to be signed and the mobile provisioning profile should match the app's app id, this is sort of problem. Luckily, I can also do a comparable renaming trick
as I have described earlier with in a blog post about improving the testability of iOS apps. Simply executing the following commands in the KitchenSink folder were sufficient:
sed -i -e "s|com.appcelerator.kitchensink|${newBundleId}|" tiapp.xml
sed -i -e "s|com.appcelerator.kitchensink|${newBundleId}|" manifest
The above commands change the
com.appcelerator.kitchensink app id into any other specified string. If this app id is changed to the corresponding id in a mobile provisioning profile, then you should be able to deploy KitchenSink to a real device.
I have added the above renaming procedure to the KitchenSink expression. The following example invocation to the earlier Nix function, shows how we can rename the app's id to:
com.example.kitchensink and how to use a certificate and mobile provisioning profile for an exisiting app:
import ./kitchensink {
inherit (pkgs) stdenv fetchgit titaniumenv;
target = "iphone";
release = true;
rename = true;
newBundleId = "com.example.kitchensink";
iosMobileProvisioningProfile = ./profile.mobileprovision;
iosCertificate = ./certificate.p12;
iosCertificateName = "Cool Company";
iosCertificatePassword = "secret";
}
By using the above expressions KitchenSink can be built for both Android and iOS. The left picture above shows what it looks like on iOS, the right picture shows what it looks like on Android.
Discussion
With the Titanium build function described in this blog post, I can automatically build Titanium apps for both iOS and Android using the Nix package manager, although it was quite painful to get it done and tedious to maintain.
What bothers me the most about this process is the fact that Appcelerator has crafted their own custom build tool with lots of complexity (in terms of code size), flaws (e.g. not propagating the CLI's argument properly from
xcodebuild) and weird issues (e.g. an odd way of detecting the presence of the JDK, and invoking the highly complicated legacy python scripts), while there are already many more mature build solutions available that can do the same job.
A quick inspection of
Titanium CLI's git repository shows me that it consists of 8174 lines of code. However, not all of their build stuff is there. Some common stuff, such as the JDK and Android detection stuff, resides in the
node-appc project. Moreover, the build steps are performed by plugin scripts that are distributed with the SDK.
A minor annoyance is that the new Node.js based Titanium CLI requires Oracle's Java Development Kit to make Android builds work, while the old Python based build script worked fine with OpenJDK. I have no idea yet how to fix this. Since we cannot provide a Nix expression that automatically downloads Oracle's JDK that automatically (due to license restrictions), Nix users are forced to manually download and import it into the Nix store first, before any of the Titanium stuff can be built.
So how did I manage to figure all this mess out?
Besides knowing that I have to patch executables, fix shebangs and wrap certain executables, the
strace command on Linux helps me out a lot (since it shows me things like files that can not be opened) as well as the fact that Python and Node.js show me error traces with line numbers when something goes wrong so that I can debug easily what's going on.
However, since I also have to do builds on Mac OS X for iOS devices, I observed that there is no strace making ease my pain on that particular operating system. However, I discovered that there is a similar tool called:
dtruss, that provides me similar data regarding system calls.
There is one minor annoyance with
dtruss -- it requires super-user privileges to work. Fortunately, thanks to
this MacWorld article, I can fix this by setting the setuid bit on the
dtrace executable:
$ sudo chmod u+s /usr/sbin/dtrace
Now I can conveniently use
dtruss in unprivileged build environments on Mac OS X to investigate what's going on.
Availability
The Titanium build environment as well as the KitchenSink example are part of
Nixpkgs.
The top-level expression for KitchenSink example as well as the build operations described earlier is located in
pkgs/development/mobile/titaniumenv/examples/default.nix. To build a debug version of KitchenSink for Android, you can run:
$ nix-build -A kitchensink_android_debug
The release version can be built by running:
$ nix-build -A kitchensink_android_release
The iPhone simulator version can be built by running:
$ nix-build -A kitchensink_ios_development
Building an IPA is slightly more complicated. You have to provide a certificate and mobile provisioning profile, and some renaming trick settings as parameters to make it work (which should of course match to what's inside the mobile provisioning profile that is actually used):
$ nix-build --arg rename true \
--argstr newBundleId com.example.kitchensink \
--arg iosMobileProvisioningProfile ./profile.mobileprovision \
--arg iosCertificate ./certificate.p12 \
--argstr iosCertificateName "Cool Company" \
--argstr iosCertificatePassword secret \
-A kitchensink_ipa
There are also a couple of emulator jobs to easily spawn an Android emulator or iPhone simulator instance.
Currently, iOS and Android are the only target platforms supported. I did not investigate Blackberry, Tizen or Mobile web applications.
Presentation
UPDATE: On July 14, 2016 I have given a presentation about this subject at the Nix meetup in Amsterdam. For convenience, I have embedded the sildes into this blog post:
