In the previous blog post about the Nix iOS build function, I have described how to install various iPhone simulator SDK versions by picking the preferences option from the 'Xcode' menu bar and selecting the 'Downloads' tab in the window, as shown in the screenshot below:
However, I did not implement any facilities in the simulator function that take these SDK versions into account yet.
Picking a specific SDK version can be done quite easily, by providing the -currentSDKRoot parameter to the iPhone simulator, such as:
In the above case, we have configured the simulator to use the iPhone simulator 6.1 SDK. By changing this parameter, we can use a different version, such as version 5.1.
I have encapsulated the above command-line invocation in the xccodeenv.simulateApp {} Nix function and the paths to the iPhone simulator SDKs in the xcodewrapper. The SDK version can be configured by providing the sdkVersion parameter (which defaults to 6.1):
This parameter makes it possible to spawn a simulator instance running a specific iOS version.
The described facilities earlier are only for simulating apps. To build an app for a specific iOS revision, the iOS target version must be changed inside the Xcode project settings.
Building signed Apps for release
Apart from the simulator, we may also want to deploy apps to real devices, such as an iPhone, iPad or iPod, either directly or through the App store. In order to do that, we require permission from Apple by signing the apps that we have just built.
As described in the previous blog post, we require a certificate that contains a developer's or company's identity and a mobile provisioning file describing which (groups of) apps of a certain company/developer can be run of which (groups of) devices. These files must be obtained through Apple's Dev Center.
Because of these restrictions, it's hard to test the trivial "Hello World" example case I have developed in the previous blog post on a real device, since it contains a dummy app and company name.
To alleviate these problems, I have created a script that "renames" the example app into a different app, so that an existing developer or company certificate and a mobile provisioning profile of a different app can be used.
By setting the rename parameter to true when calling the composition expression of the example case, we can automatically generate jobs building IPAs and xcarchives for the "renamed" app:
In the above expression, we changed the name of the app into: "MyVeryCoolApp" and the company name into: "My Cool Company". Moreover, we provide a certificate and mobile provisioning file for the IPA job (which can be deployed to a device directly) and the xcarchive job (which can be used to deploy to the App store).
By running the following command-line instruction:
$ nix-build -A renamedPkgs.testapp_ipa
We can build an IPA allowing us to deploy the test app on a real device. Of course, to do this yourself, the company name and app names should correspond to the ones defined in your certificate and mobile provisioning profile.
Moreover, as with the Android environment in the previous blog post, we can also use Hydra to provide parameters to the composition expression and to retrieve all the resulting artifacts:
Conclusion
In this blog post, I have described some improvements to the Nix iOS facilities. We are now also capable of configuring the SDK versions for the iPhone simulator, and we can build the trivial testcase to run on a real device in addition to the simulator. This is useful for testing the Nix iOS build function.
Some time ago, I have packaged the Android SDK and some of its plugins in Nix, which was quite a challenge. Moreover, I have developed two Nix functions that can be used to automatically build Android apps and spawn emulator instances.
Meanwhile, I have implemented some major updates since the last blog post. Besides upgrading to a newer Android SDK version, we now also support a bunch of new interesting features.
Android SDK packaging updates
Currently, I have packaged the 22.05 version of the Android SDK (which is the latest version at the time writing this blog post). To package it, I have followed the same recipe as described in the previous blog post, with some additions:
Newer versions of the Android SDK include x86-64 versions of the emulator executables. I had to patch/wrap these executables to allow them to find the x86-64 libraries they depend on.
The newer Android SDK has a new sub package called build-tools. Apparently, this sub package contains some tools that were formerly part of platform-tools, such as debugging libraries.
The GUI front-end of the android utility did not work in the last blog post, which I have fixed by wrapping it, so that it can find its dependencies, such as GTK+.
There are also x86 and MIPS application binary interface (ABI) system images, but they reside in external package repositories. I have adapted the Nix expression generator to include these as well.
I have adapted the build and emulation Nix functions to take both API-level and ABI versions into account
I have implemented some facilities to improve the performance of the emulator instances
Packaging build-tools
The build-tools package contains various tools and libraries that used to belong to platform-tools. It turns out that some of these files, such as the debugging libraries, are API-level specific. The purpose of the build-tools package is to provide API-level specific build tools. Currently, the Android SDK supports two versions: 17 and 18.01 that can be installed next to each other.
I have packaged the 18.01 version in a straight forward manner. The only thing I needed to do is inspecting the dependencies of the executables to the paths to these libraries to the executables' RPATH.
Every version of the build-tools package is symlinked into the Android SDK basedir as follows: build-tools/android-<version>
Fixing the Android GUI front-end
If the android tool is called without any parameters, it should show a GUI that allows somebody to download plugins or to configure AVDs. However, the GUI did not work so far. Invoking 'android' gave me the following error message:
$ android
Exception in thread "main" java.lang.UnsatisfiedLinkError:
no swt-pi-gtk-3550 or swt-pi-gtk in swt.library.path,
java.library.path or the jar file
at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
at org.eclipse.swt.internal.Library.loadLibrary(Unknown Source)
at org.eclipse.swt.internal.gtk.OS.(Unknown Source)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
at org.eclipse.swt.internal.Converter.wcsToMbcs(Unknown Source)
at org.eclipse.swt.widgets.Display.(Unknown Source)
at com.android.sdkmanager.Main.showSdkManagerWindow(Main.java:346)
at com.android.sdkmanager.Main.doAction(Main.java:334)
at com.android.sdkmanager.Main.run(Main.java:120)
at com.android.sdkmanager.Main.main(Main.java:103)
First, I thought that some file belonging to SWT was missing, but after manually digging through all Android files I couldn't find it and I gave up. However, after using strace, I discovered that it's actually trying to load the GTK+ library and some other libraries:
Besides the libraries, I also had to wrap android to use the right JDK version. Apparently, loading these libraries only works with OpenJDK in Nix.
After wrapping the android utility and running it without parameters, it shows me the following:
As you may see in the above image, the GUI works. It also shows us all the plugins that are installed through Nix. Although the GUI seems to work fine, we have to keep in mind that we cannot use it to install additional Android SDK plugins. All plugins must be packaged and installed with Nix instead.
Supporting external system images
The Android distribution provides a collection of system images that can be used with an emulator to test an app during development. However, the default repository only contains ARM-based system images, which are quite slow to emulate.
There also seem to be x86-based system images available (provided by Intel) and MIPS-based images (provided by MIPS Technologies). These system images reside in external repositories. I was able to discover the locations of these repositories, by running the following command:
The latter two URLs referring to a file called sys-img.xml provide x86 and MIPS images. The structure of these XML files are similar to the main repository. Both files contain XML elements that look like this:
The generated expression shown above is nearly identical to the one shown in the previous blog post. The only difference is that we also use the ABI identifier in the attribute names. For example: sysimg_x86_16 is used to refer to a x86-based system image for Android API-level 16. Likewise, we can refer to the ARM-based variant with: sysimg_armeabi-v7a_16.
Apart from giving each attribute a proper name, the resulting system image package must be symlinked into the Android SDK basedir taking the ABI identifier into account. We symlink every architecture-dependent system image package into: system-images/android-<api-level>/<abi>.
I have adapted the androidenv.emulateApp {} Nix function to also take ABI versions into account. By default, the emulator function uses an ARM-based system image (armeabi-v7a), since this is the most common hardware architecture used by Android OSes. By setting the abiVersion parameter to x86, we can use an x86-based Android system-image:
An interesting benefit of using an x86 system image with the emulator (which is based on QEMU) is that KVM can be used, allowing programs inside the emulator to run at nearly native speed with only little emulation.
And of course, setting the abiVersion parameter to mips makes it possible to use MIPS-based system images.
Enabling GPU acceleration
The x86 system images in the emulator may give testing a significant boost. Emulation performance can even be improved a bit further by enabling GPU acceleration.
To make GPU acceleration possible, I had to adapt the emulator's wrapper script to include Mesa in the linker path. Moreover, in the AVD configuration I have to add the following boolean value to $ANDROID_SDK_HOME/.android/avd/device.avd/config.ini:
hw.gpu.enabled = yes
The androidenv.emulateApp {} automatically enables GPU acceleration from API-levels 15 and onwards. It can be disabled by setting the enableGPU parameter to false.
Supporting Google APIs in non-ARM-based system images
Although supporting x86 system images have various benefits, they also have a major drawback -- the Google APIs are not supported preventing some applications from working. One of the solutions I have seen (described in this Stack Overflow post) is to create a custom system image by starting an ARM-based emulator instance with Google APIs, fetch the libraries, start an x86-based emulator instance without Google APIs and manually install the fetched libraries.
I was able to automate the steps in the Stackflow article in a Nix expression:
The above Nix function creates an ARM emulator instance with Google APIs and an x86 emulator instance without Google APIs. Then it starts the ARM emulator instance, fetches the Google API files from it and then stops it. Then the x86 emulator instance is started and the Google APIs are pushed to it. Finally we generate a system image from the system folder that is stored in the Nix store.
By using the Nix function shown above and setting the extraAVDFiles parameter of the emulateApp {} function, we can use our custom system image in a script that automatically spawns the emulator instance:
The above expression invokes the function shown previously and starts an emulator instance with the system image containing Google APIs. Of course, we can also use the same trick for MIPS-based system images, which also don't include Google APIs.
Building Android apps for arbitrary API-level revisions
I have also made a small change to the androidenv.buildApp {} Nix function to take API-levels into account. By default, it will use the version that is inside the Ant configuration. However, by setting the target property through the command-line, we can override this:
$ ant -Dtarget=android-17
The above command-line instruction ensures that we build the app against the API-level 17 platform. I have added an antFlags parameter to the Nix function to make it provide arbitrary flags to Ant:
In the previous blog post, I have implemented a testcase based on the Android introduction tutorial (MyFirstApp) that I have used throughout this article as well. To test all these new features, I have extended its composition expression to take three parameters: buildPlatformVersions (defaults to 16), emulatePlatformVersions (defaults to 16), and abiVersions (defaults to armeabi-v7a). The expression automatically generates all possible combinations with the values inside these three lists:
For example, to build a debug version of the test app APK for API-level 16, we can run:
$ nix-build -A myfirstapp_debug.build_16
We can also build a release version of the same APK that is signed with a key:
$ nix-build -A myfirstapp_release.build_16
Furthermore, we can automatically generate a script starting an emulator instance running the app. The following instruction builds an App for Android API-level 16, generates a script launching an emulator with a Android API-level 16 system-image that uses the armeabi-v7a ABI:
$ nix-build -A emulate_myfirstapp_debug.build_16.emulate_16.armeabi-v7a
$ ./result/run-test-emulator
To support more API-levels and ABIs, the parameters of the composition function must be altered. For example, by running the following command-line instruction:
The cartesian product of build and emulator instances is created taking the three dimensions into account. This allows us to (for example) build the App against the API-level 16 platform API, emulate on an API-level 17 system image using the x86 ABI, which emulates much more efficiently on x86 machines, because hardware visualization features (such as KVM) can be used.:
Apart from specifying these dimensions through the command line, the composition expression can also be used together with Hydra, the Nix-based continuous integration server, allowing it to pass these parameters through its web interface. Hydra will build all the combinations generated by the composition expression automatically:
As can be observed from the above screenshot, quite a few jobs are being generated.
Conclusion
In this blog post, I have described some new changes and features that I have implemented in the Nix Android SDK package since the last blog post, such as the fact that x86 and MIPS system-images can be used. All these changes are part of the Nixpkgs project.
Besides the issues previously described, there is even more pain/frustration to cope with, which I'd like to discuss in this blog post. The main motivation to write this down is because I'm a control freak that wants to know what happens, it took me some effort discovering these details, and I have found myself repeating the same stuff to others.
Executing tasks concurrently
In addition to organizing data in a program, the ability to execute tasks concurrently is also important for many types of applications, such as:
Applications with graphical user interfaces (GUIs). For example, if a user clicks on a button executing a task, the GUI must remain responsive while the task is being processed.
Server applications. A server application has to serve multiple connected clients simultaneously, while each task remains responsive. For example, if a task is being processed of some client, it should not be necessary for a new client to wait until the requests of the others are done.
Applications communicating with hardware peripherals. Many peripherals, such as (hard/DVD/Blu-Ray) disk drives or printers are significantly slower than the CPU of the computer. For example, it's quite inconvenient that execution of a program temporarily blocks the entire system when a file is being read from the hard drive. Instead, it's (obviously) better to keep the programming running (and respond on user events) while a file is being read, and to notify the user when the task is done.
To allow tasks to be executed concurrently, many operating systems/execution environments have facilities, such as interrupts, processes and/or threads allowing a developer to do that in a (slightly) convenient way. The execution environment ensures that these processes or threads are executed simultaneously (well not really, in fact it divides CPU execution time over processes and threads, but in order to keep this blog post short, I won't go into detail on that).
Executing tasks concurrently in JavaScript
Although most operating systems, execution environments, and programming languages have facilities for threads and/or processes, JavaScript -- a programming language originally used inside a web browser with increasing popularity outside the browser -- does not have such facilities. In JavaScript, there is only one single execution thread.
Consider the following example HTML page:
<!DOCTYPE html>
<html>
<head><title>Stretching button</title></head>
<body>
<h1>Stretching button!</h1>
<button id="button" onclick="stretchButton();">Stretch</button>
<script type="text/javascript">
enabled = false;
function updateButtonSizes() {
var grow = true;
var width = 100;
var height = 20;
var button = document.getElementById('button');
while(enabled) {
/* Update the button sizes */
button.style.width = width + "px";
button.style.height = height + "px";
/* Adjust the button sizes */
if(grow) { width++; height++; }
else { width--; height--; }
/* If the limits are reached, stretch in
* the opposite direction
*/
if(width >= 150) grow = false; else if(width <= 100) grow = true;
}
}
function stretchButton() {
if(enabled) {
enabled = false;
} else {
enabled = true;
updateButtonSizes();
}
}
</script>
</body>
</html>
The purpose of the HTML code shown above is to display a button that animates by increasing and decreasing its width and height once the user has clicked on it. The animation keeps repeating itself and stops if the user clicks on the same button again.
By just looking at the the code, it may give you the impression that it should work fine. However, once we have clicked on the button, the browser blocks completely and prevents us clicking on the button again. Furthermore, because we cannot click on the button again, we are also incapable of stopping the animation sequence. As a consequence, we have run into an infinite loop.
The reason that we run into trouble, is because if JavaScript code is being executed -- that runs in a single execution thread -- the browser cannot do anything else, such as responding to user events or updating the screen. After a minute or so, the browser will notice that some script is blocking it and asks you if it should terminate the script for you.
So you may probably wonder how to deal with this blocking issue? To allow the browser to respond to user events and update the screen, we must stop the execution of the program (by reaching the end of its control flow). Then at a later stage, we must resume execution by generating an event. Besides generating events from page elements, such as buttons as can be seen in the code fragment above, we can also generate timeout events:
In the snippet above, the setTimeout() function invocation tells the browser to execute the function displaying "Hello World" after 1000 milliseconds. Like events generated by page elements, timed events can also only be processed if nothing is being executed at that moment. So the program must reach the end of its execution flow first before the timeout event can be processed.
An adapted version of the earlier example with the button that correctly handles events and screen updates, may look as follows:
<!DOCTYPE html>
<html>
<head>
<title>Stretching button</title>
</head>
<body>
<h1>Stretching button!</h1>
<button id="button" onclick="stretchButton();">Stretch</button>
<script type="text/javascript">
enabled = false;
button = document.getElementById('button');
function updateButtonSizes() {
/* Update the button sizes */
button.style.width = width + "px";
button.style.height = height + "px";
/* Adjust the button sizes */
if(grow) { width++; height++; }
else { width--; height--; }
/* If the limits are reached then stretch in
* the opposite direction
*/
if(width >= 150) grow = false; else if(width <= 100) grow = true;
/* Keep repeating this until the user disables it */
if(enabled) setTimeout(updateButtonSizes, 0);
}
function stretchButton() {
grow = true;
width = 100;
height = 20;
if(enabled) {
enabled = false;
} else {
enabled = true;
updateButtonSizes();
}
}
</script>
</body>
</html>
The above example code is quite similar to the previous one, with the following differences:
The updateButtonSizes() function does not contain a while loop. Instead, it implements a single iteration step and calls setTimeout() to generate a timeout event that calls itself.
After setting the timeout event, the program reaches the end of the execution flow.
The browser updates the screen, handles user input and resumes execution by processing the timeout event that invokes updateButtonSizes() again.
I also made several variables global so that their values are remembered after each execution of updateButtonSizes()
To prove that the above example works, I have included an embedded version in this blog post:
Stretching button!
Cooperative multi-tasking
In short, to execute tasks in a browser while keeping the user interface responsive, we must do the following:
Generate timeout events when it's desired to give other tasks the opportunity to do something.
Reach the end of the control flow so that events can be processed by the execution environment. The generated timeout event calling a function ensures that execution is resumed at a later stage.
This approach looks very similar to cooperative multi-tasking using by classic Mac OS 9 and earlier and Windows operating systems prior to 3.x.
The fact that a function needs to be called every time you want to allow the browser/runtime environment to process events, reminds of the way I was developing programs many years ago on the AmigaOS. I still remember quite vividly that when using AMOS BASIC, if I wanted Workbench applications running in the background to do something while my program was running, I had to add the 'Multi Wait' instruction to the main loop of the program:
While busy
' Perform some tasks
Multi Wait
Wend
By using the 'Multi Wait' instruction, the operating system gets notified, interrupts the program's execution and the scheduler decides which other process is allowed to use the CPU. Then at some point, the OS scheduler decides that my program is allowed to continue and the program's execution continues as if nothing happended.
However, compared the JavaScript approach, the AmigaOS approach is a bit more powerful since it also takes care of interrupting and resuming a program at the right location within the right context.
Cooperative multitasking works as long as tasks properly cooperate, putting great responsibility by developers of a program. It's not uncommon however, that people make mistakes and the system as a whole blocks forever.
Asynchronous programming
Apart from the browser, many other environments embedding a JavaScript runtime require developers to use cooperative multitasking techniques.
Node.js, a server-side software system designed for writing scalable Internet applications, uses almost the same approach as the browser just described earlier. One minor difference is that instead of using setTimeout(), a more efficient alternative is used:
The process.nextTick() function executes the given callback function once the end of the execution flow of the program has been reached. Because it does not use the timer, it's more efficient.
To save users the burden of calling process.nextTick() and ending the execution flow each time something may block the system, most library functions in Node.js are asynchronous by default (with some having a synchronous alternative). Asynchronous functions have the following properties:
They return (almost) immediately.
A callback function (provided as function parameter) gets invoked when the task is done, which can be used to retrieve its result or status.
They automatically generate tick events and reach the end of their control flow to prevent the system being blocked.
For example, to read a file and store its contents in a string without blocking the system, one could do:
The above example opens /etc/hosts asynchronously. The fs.readFile() function returns immediately, then the string "Hello" is printed and the program reaches the end of its control flow. Once the file has been read, the callback function is invoked displaying either an error message (if the file cannot be opened) or the contents of the file.
Although asynchronous functions prevent the system being blocked and allow multitasking, one of its big drawbacks is that the callback mechanism makes it much more difficult to structure a program properly. For example, if I want to create a folder called "out", then a subfolder called "test" in the previous folder, inside the "test" subfolder a file called "hello.txt" containing "Hello world!", and finally verify whether the written text file contains "Hello world!", we may end up writing:
var fs = require('fs');
var path = require('path');
fs.mkdir("out", 0755, function(err) {
if(err) throw err;
fs.mkdir(path.join("out, "test"), 0755, function(err) {
if (err) throw err;
var filename = path.join("out", "test", "hello.txt");
fs.writeFile(filename, "Hello world!", function(err) {
if(err) throw err;
fs.readFile(filename, function(err, data) {
if(err) throw err;
if(data == "Hello world!")
process.stderr.write("File is correct!\n");
else
process.stderr.write("File is incorrect!\n");
});
});
});
});
For each step in the above sequential program structure, we end up one indentation level deeper. For the above example it's probably not so bad, but what if we have to execute 10 asynchronous functions sequentially?
Moreover, it is also non-trivial to modify the above piece of code. What, for example, if we want to write another text file, before the function that creates the test folder? It requires us to weave another function callback in the middle of the program and to refactor the entire indentation structure.
Fortunately, there are also solutions to cope with that. For example, the async library contains a collection of functions supporting control flow patterns and functions supporting data collections that can be used to deal with complexities of asynchronous functions.
The waterfall pattern, for example, can be used to execute a collection of asynchronous function sequentially and stops if any of the callbacks returns an error. This can be used to flatten the structure of our previous code example and also makes it better maintainable:
Besides the waterfall pattern, the async library supports many other control flow and collection patterns, such as executing functions in parallel or asynchronous loops over items in an array.
Mutable state
There is another important aspect that we have to keep in mind while executing stuff concurrently. Shared mutable state may influence the result of an execution of a task. For example, in our last example using async.waterfall(), I have declared the filename variable as a global variable, because I (naively) thought that it was a good way to share its value among the callback functions.
However, since it's possible to run multiple tasks concurrently, another task may change this global variable to something else. As a consequence, when we have reached the callback using it, it may has been changed to a different value and thus our expected result may differ. Moreover, it's also extremely hard to debug these kind of problems.
To protect ourselves from these kind of problems, it's better to scope a variable, which can be done with the var keyword. However, although most programming languages, such as Java (which influenced JavaScript) use block-level scoping, in JavaScript variables are function-level scoped.
For example, the following Java example shows the effect of block-level scoping:
public class Test {
public static void main(String[] args) {
int i = 1;
int j = 0;
if(j == 0) {
int i = 2;
}
return i; // 1;
}
}
In the example above we have two i variables. The first one is declared inside the main() body, the second inside the if-statement block. Because the variables are scoped within the nearest block boundary, the value of i inside the if-block is discarded after it terminates.
However, a direct port of the above code to JavaScript yields a different result, because i is scoped to the nearest function boundary:
function test() {
var i = 1;
var j = 0;
if(j == 0) {
var i = 2;
}
return i; // 2;
}
So in our JavaScript code fragment, the second var i; declaration inside the if-block is also bound to the scope of the test() function, therefore yielding a different result. Fortunately, block level scoping can be easily simulated by declaring a function without parameters and calling it without parameters (function() { var x; ... } ();):
function test() {
var i = 1;
var j = 0;
if(j == 0) {
function() {
var i = 2; /* i is scoped within the surrounding function */
}();
}
return i; // 1;
}
Similarly this trick also allows us to scope the filename variable shown in our earlier example with async.waterfall():
function() {
var filename; /* filename is scoped within the surrounding function */
async.waterfall(...);
}();
In the above example, the filename variable is not global anymore. The callback functions inside async.waterfall() will not refer to the global variable named filename any longer, but to the scoped variable inside the function block. As a result, other tasks running concurrently adapting global variables cannot influence this task any longer.
Conclusion
In this blog post, I have described some of the pains I have experienced with multitasking inside a JavaScript environment. From this, I have learned the following lessons:
If a script is being executed, then nothing else can be done, such as screen updates, processing user events, handling client connections etc.
To allow multitasking (and prevent the system being blocked entirely) we have to cooperate by generating timeout or tick events and ending the control flow of the program.
Asynchronous functions use callbacks to retrieve their results and/or statuses making it extremely hard to structure your program properly. In other words, it easily becomes a mess. To cope with this, you need to think about patterns to combine them. Libraries, such as async, may help you with this.
Shared state is evil, error prone and makes a program hard/impossible to debug. Therefore, only use global variables that are immutable or refer to singleton instances (such as CommonJS modules). Otherwise, scope them.
JavaScript does not support block level scoping, but it can be simulated by encapsulating a block inside a function that gets called without parameters. Fortunately, the ECMAScript 6 standard defines the let keyword supporting block-level scoping natively, but it's not supported in most implementations, such as in Node.js.
Although I'm a little happy about the fact that I know how to handle these problems, I'm very sad at the same time because of the following reasons:
JavaScript was originally advertised as "a lightweight and simpler variant of Java". The fact that every JavaScript developer must cope with all these concurrency issues, which Java mostly solves with a threading API, definitely makes JavaScript NOT simpler than Java in this respect IMHO.
Cooperative multitasking is a technique that was very common/useful in the early 1980s (and probably invented years before it), but has been replaced by preemptive multitasking in modern operating systems. One of the reasons to take responsibility away from the processes (besides the fact that it makes the lives of programmers easier) is because they cannot be trusted. However, we still seem to "trust" JavaScript code embedded in malicious web pages.
We're using a programming language for all kinds of purposes for which it has not been designed. Moreover, we require solutions for these issues that are not part of the language, API or runtime, such as (ab)using the timer and relying on external libraries implementing patterns to combine asynchronous functions.
Sometimes these new use cases of JavaScript are called innovation, but the fact that we use techniques that are over 30 years old (and intentionally not using better modern alternatives), does not really look like innovation to me IMHO.
So are solutions using JavaScript that bad? I see that there is practical use and value in it (e.g. the V8 runtime compiles to efficient native code and there is a big eco-system of JavaScript libraries), but dealing with concurrency like this is definitely something I consider a big drawback and a huge step backwards.
I have written quite some Nix-related articles on this blog. Nix is typically advertised as the core component of the NixOS Linux distribution. However, it can also be used separately on conventional Linux distributions and other UNIX-like systems, such as FreeBSD and Mac OS X.
Using Nix on conventional systems makes it possible to use the interesting features of Nix and its applications, such as Hydra and Disnix, while still being able to use your favorite distribution.
Single-user Nix installations
I have noticed that on non-NixOS systems, the Nix package manager is often installed for only one single user, as performing single-user installations is relatively simple. For example, in my blog post describing how to build apps for iOS with Nix, I perform a Nix installation that can only be used by my personal user account.
For most of the simple use cases, single user installations are sufficient. However, they have a number of issues besides the fact that only one user of a machine (in addition to the super-user) is able to use it:
Although Nix creates build environments that remove several important side-effects, e.g. by clearing environment variables and storing all packages in isolation in the Nix store, builds can still refer to executables and other files in global directories by having hardcoded references, such as /usr/bin or /var, which may influence the result of the build.
On NixOS these references are typically not an issue since these directories do not exist, but on conventional distributions these do exist, which may cause (purity) problems.
Many packages have hard-coded references to the default Bourne-compatible shell in: /bin/sh. Some of these packages assume that this shell is bash and use bash-specific features.
However, some Linux distributions, such as Debian and Ubuntu, use dash as default /bin/sh causing some builds to break. Moreover, BSDs such as FreeBSD also use a simpler Bourne shell implementation by default.
Some build processes may unknowingly try to download stuff from the Internet causing impurities.
Package builds have the same privileges as the calling user, allowing external processes run by the user to interfere with the build process. As a consequence, impurities may sneak in when executing multiple package builds in parallel.
Multi-user Nix installations
As a remedy for the issues just described, Nix is also capable of executing each build with separate user privileges in a nearly clean chroot environment in which we bind mount the Nix store. However, in order to use these features, a multi-user Nix installation is required, as these operations require super-user privileges. In NixOS, a multi-user Nix installation comes for free.
In multi-user Nix installations, builds are not executed directly by each individual user, since they cannot be trusted. Instead we run a server, called the nix-daemon that builds packages on behalf of a user. This daemon also takes care of running processes as a unique unprivileged user and setting up chroot environments.
Although the Nix manual provides some pointers to set up a multi-user installation, it turned out to be a bit trickier than I thought. Moreover, I have noticed that a few practical bits were missing in the manual and the Nix distribution.
In this blog post, I have investigated these issues and implemented a few improvements that provide a solution for these missing parts. I have performed these steps on a Ubuntu 12.04 LTS machine.
Installing Nix from source
As a first step, I installed Nix from source by running the following commands:
$ ./configure --prefix=/usr --sysconfdir=/etc
$ make
$ sudo make install
Adding the Nix build group and users
Since every concurrent build must be run as a separate user, we have to define a common group of which these users should be a member:
$ sudo groupadd -g 20000 nixbld
Then we need to add user accounts for each build that gets executed simultaneously:
$ for i in `seq 1 10`
do
sudo useradd -u `expr 20000 + $i` -g nixbld \
-c "Nix build user $i" -d /var/empty -s /noshell
done
In the code fragment above, we assume that 10 users are sufficient, but if you want/need to utilise more processors/cores this number needs to be raised.
Finally, we have to specify the build users group in the Nix configuration:
In ordinary installations the user is made owner of /nix/store. In multi-user installations, it must be owned by root and group owned by the nixbld user. The following shell commands grant the Nix store the right permissions:
Creating per-user profile and garbage collection root folders
In single user installations, we only have one system-wide default profile (/nix/var/nix/profiles/default) owned by the user. In multi-user installations, each user should be capable of creating their own profiles and garbage collector roots. The following shell commands ensure that it can be done:
We must also run the nix-daemon that executes builds on behalf of a user. To be able to start and stop it, I have created an init.d script for the Nix daemon. The interesting part of this script is the start operation:
DAEMON=/usr/bin/nix-daemon
NAME=nix-daemon
if test -f /etc/default/nix-daemon; then
. /etc/default/nix-daemon
fi
...
case "$1" in
start)
if test "$NIX_DISTRIBUTED_BUILDS" = "1"; then
NIX_BUILD_HOOK=$(dirname $DAEMON)/../libexec/nix/build-remote.pl
if test "$NIX_REMOTE_SYSTEMS" = "" ; then
NIX_REMOTE_SYSTEMS=/etc/nix/remote-systems.conf
fi
# Set the current load facilities
NIX_CURRENT_LOAD=/var/run/nix/current-load
if test ! -d $NIX_CURRENT_LOAD; then
mkdir -p $NIX_CURRENT_LOAD
fi
fi
start-stop-daemon -b --start --quiet \
--exec /usr/bin/env \
NIX_REMOTE_SYSTEMS=$NIX_REMOTE_SYSTEMS \
NIX_BUILD_HOOK=$NIX_BUILD_HOOK \
NIX_CURRENT_LOAD=$NIX_CURRENT_LOAD \
$DAEMON -- $DAEMON_OPTS
echo "$NAME."
;;
...
esac
For the start operation, we have to spawn the nix-daemon in background mode. Moreover, to allow Nix to perform distributed builds, we must set a number of environment variables that provide the locations of the build hook script, the configuration file containing the properties of the external machines and a directory containing files keeping track of the load of the machines. Moreover, some directories may also have to be created if they don't exist.
To ensure that it's automatically launched on startup, we must add the following symlinks for the relevant runlevels:
$ sudo -i
# cd /etc/rc2.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc3.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc4.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# cd ../rc5.d
# ln -s ../init.d/nix-daemon S60nix-daemon
# exit
I think the above init.d script can be trivially ported to other distributions.
Setting up user profiles
To allow users to install software through Nix and allow them to refer to their installed programs from a simple command-line invocation, we need to add some stuff to the user's shell profile, such as setting the PATH environment variable pointing to certain Nix profiles.
However, the nix.shprofile.d script in the Nix distribution only performs the necessary steps for single user installations. For example, it only adds to system-wide Nix profile and assumes that the user has all the rights to configure a channel.
I have ported all the relevant features from NixOS to create /etc/profile.d/nix-multiuser.sh, supporting all required features to set up a shell profile for multi-user installations:
First, we have to set up a user's profile directory in the per-user profile directory, if it doesn't exist:
export NIX_USER_PROFILE_DIR=/nix/var/nix/profiles/per-user/$USER
mkdir -m 0755 -p $NIX_USER_PROFILE_DIR
if test "$(stat --printf '%u' $NIX_USER_PROFILE_DIR)" != "$(id -u)"; then
echo "WARNING: bad ownership on $NIX_USER_PROFILE_DIR" >&2
fi
In single user installations, a ~/.nix-profile symlink is created pointing to the system-wide default Nix profile. For multi-user installations, we must create a ~/.nix-profile symlink pointing to the per-user profile. For the root user, we can still use the system wide Nix profile providing software for all users of the system:
if ! test -L $HOME/.nix-profile; then
echo "creating $HOME/.nix-profile" >&2
if test "$USER" != root; then
ln -s $NIX_USER_PROFILE_DIR/profile $HOME/.nix-profile
else
# Root installs in the system-wide profile by default.
ln -s /nix/var/nix/profiles/default $HOME/.nix-profile
fi
fi
In single user installations, we add the bin directory of the system-wide Nix profile to PATH. In multi-user installations, we have to do this both for the system-wide and the user profile:
export NIX_PROFILES="/nix/var/nix/profiles/default $HOME/.nix-profile"
for i in $NIX_PROFILES; do
export PATH=$i/bin:$PATH
done
In single user installations, the user can subscribe itself to the Nixpkgs unstable channel providing pre-built substitutes for packages. In multi-user installations only the super-user can do this (as ordinary users cannot be trusted). Although root can only subscribe to a channel, ordinary users can still install from the subscribed channels:
if [ "$USER" = root -a ! -e $HOME/.nix-channels ]; then
echo "http://nixos.org/channels/nixpkgs-unstable nixpkgs" \
> $HOME/.nix-channels
fi
We have to create a garbage collector root folder for the user, if it does not exists:
NIX_USER_GCROOTS_DIR=/nix/var/nix/gcroots/per-user/$USER
mkdir -m 0755 -p $NIX_USER_GCROOTS_DIR
if test "$(stat --printf '%u' $NIX_USER_GCROOTS_DIR)" != "$(id -u)"; then
echo "WARNING: bad ownership on $NIX_USER_GCROOTS_DIR" >&2
fi
We must also set the default Nix expression, so that we can conveniently install packages from Nix channels:
if [ ! -e $HOME/.nix-defexpr -o -L $HOME/.nix-defexpr ]; then
echo "creating $HOME/.nix-defexpr" >&2
rm -f $HOME/.nix-defexpr
mkdir $HOME/.nix-defexpr
if [ "$USER" != root ]; then
ln -s /nix/var/nix/profiles/per-user/root/channels \
$HOME/.nix-defexpr/channels_root
fi
fi
Unprivileged users do not have the rights to build package directly, since they cannot be trusted. Instead the daemon must do that on behalf of the user. The following shell code fragment ensures that:
if test "$USER" != root; then
export NIX_REMOTE=daemon
else
export NIX_REMOTE=
fi
Using multi-user Nix
After having performed the previous steps, we can start the Nix daemon by running the init.d script as root user:
$ sudo /etc/init.d/nix-daemon start
Then if we login as root, we can update the Nix channels and install packages that are supposed to be available system-wide:
We should also be able to log in as an unprivileged user and capable of installing software:
$ nix-env -i wget
$ wget # Only available to the user that installed it
$ hello # Also works because it's in the system-wide profile
Enabling parallel builds
With a multi-user installation, we should also be able to safely run multiple builds concurrently. The following change can be made to allow 4 builds to be run in parallel:
The above allows us to delegate builds for Mac OS X to a Mac OS X machine.
Enabling chroot builds
On Linux, we can also enable chroot builds allowing us to remove many undesired side-effects that single-user Nix installations have. Chroot environments require some directories of the host system to be bind mounted, such as /dev, /dev/pts, /proc and /sys.
Moreover, we need a default Bourne shell in /bin/sh that must be bash, as other more primitive Bourne compatible shells may give us trouble. Unfortunately, we cannot bind mount the host's system /bin folder, as it's filled with all kinds of executables causing impurities. Moreover, these executables have requirements on shared libraries residing in /lib, which we do not want to expose in the chroot environment.
I know two ways to have bash as /bin/sh in our chroot environment:
We can install bash through Nix and expose that in the chroot environment:
The last line exposes /dev, /dev/pts and /nix-bin (mounted on /bin, containing only sh) of the host system to the chroot environment allowing us to build stuff purely. If we have installed bash through Nix, then we also have to add the Nix closure of bash, which we can obtain through the nix-store -qR command. For a static bash this invocation has to be omitted.
Conclusion
In this blog post, I have described everything I did to set up a multi-user Nix installation supporting distributed and chroot builds on a conventional Linux distribution (Ubuntu) which was a bit tricky.
I'm planning to push some of things I did upstream, so that others can benefit from it. This is a good thing, because I have the feeling that most non-NixOS Nix users will lose their interest if they have figure out the same stuff I just did.
Another important aspect is to know that most of the facilities of Hydra are publicly accessible by anyone, except for the administration tasks, such as maintaining projects and jobsets for which it is required to have a user account.
A way to prevent Hydra from having public user access is to secure the reverse proxy that is in front of it. In this blog post, I will describe how to set up an HTTPS virtual domain with password authentication for this purpose. Moreover, the approach is not restricted to Hydra -- it can be used to secure any web application by means of a reverse proxy.
Generating a self-signed SSL certificate
To be able to set up HTTPS connections we need an SSL certificate. The easiest way to get one is to create a self-signed SSL certificate. The following command-line instruction suffices for me to generate a private key:
The above command asks you to provide a passphrase. Then we need to generate a certificate signing request (CSR) file to sign the certificate with our own identity:
The above command asks you some details about your identity, such as the company name, state or province and country. After the CSR has been created, we can generate a certificate by running:
Now we have successfully generated a self-signed certificate allowing us to encrypt the remote connection to our Hydra instance.
Obviously, if you really care about security, it's better to obtain a cross-signed SSL certificate, since that provides you (some sort of) guarantee that a remote host can be trusted, whereas our self-signed SSL certificate forces users to create a security exception in their browsers.
Creating user accounts
A simple way to secure an Apache HTTP server with user authentication is by using the htpasswd facility. I did the following to create a user account for myself:
$ htpasswd -c /etc/nixos/htpasswd sander
The above command creates a user account named: sander, asks me for a password and stores the resulting htpasswd file in /etc/nixos. The above command can be repeated to add more user accounts.
Adapting the NixOS system configuration
As a final step, the reverse proxy must be reconfigured to accept HTTPS connections. The following fragment shows how the Apache HTTPD NixOS service can be configured to act as a reverse proxy having an unsecured HTTP virtual domain, and a secured HTTPS virtual domain requiring users to authenticate with a username and password:
services.httpd = {
enable = true;
adminAddr = "sander@example.com";
hostName = "hydra.example.com";
sslServerCert = "/etc/nixos/hydra.example.com.crt";
sslServerKey = "/etc/nixos/hydra.example.com.key";
virtualHosts = [
{ hostName = "localhost";
extraConfig = ''
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyRequests Off
ProxyPreserveHost On
ProxyPass / http://localhost:3000/ retry=5 disablereuse=on
ProxyPassReverse / http://localhost:3000/
'';
}
{ hostName = "localhost";
extraConfig = ''
<Proxy *>
Order deny,allow
Allow from all
AuthType basic
AuthName "Hydra"
AuthBasicProvider file
AuthUserFile /etc/nixos/htpasswd
Require user sander
</Proxy>
ProxyRequests Off
ProxyPreserveHost On
ProxyPass / http://localhost:3000/ retry=5 disablereuse=on
ProxyPassReverse / http://localhost:3000/
RequestHeader set X-Forwarded-Proto https
RequestHeader set X-Forwarded-Port 443
'';
enableSSL = true;
}
];
};
There is one important aspect that I had to take into account in the configuration of the HTTPS virtual host. Without providing the RequestHeader properties, links in Hydra are not properly generated, as Catalyst (the MVC framework used by Hydra) does not know that links should use the https:// scheme instead of http://.
Restricting the unsecured HTTP virtual domain
It may also desired to prevent others from having access to the insecure HTTP virtual host. In a NixOS system configuration, you can set the firewall configuration to only accept HTTPS connections by adding the following lines:
The tricks described in this blog post allowed me to secure Hydra with an HTTPS connection and password authentication, so that I can open a web browser and use: https://hydra.example.com to access the secured virtual host.
The major disadvantage of this approach is that you cannot use Nix's channel mechanism to automatically obtain substitutes from Hydra. Perhaps, in the future Nix can be adapted to also support connections with user authentication.
As a follow up story on the previous two blog posts, I can tell you that I have defended my PhD thesis last Monday, which went successfully. Now I can officially call myself a Doctor.
It was quite a tense, busy and interesting day. I'm not really used to days in which you are the centre of attention for most of the time. In the morning, our foreign guests: Sam Malek and Roberto di Cosmo (who are members of my committee) arrived to tell a bit about their ongoing research. We had some sort of a small software deployment workshop with quite some interesting ideas and discussions. Having them visiting us, gave me some renewed excitement about ongoing research and some interesting future ideas, since software deployment (as I have explained) is typically a misunderstood and under appreciated research subject.
After our small workshop, I had to quickly pick up our suits, and then return to the university. In the early afternoon, I had to give a laymen's talk to my family and friends in which I tried to explain the background and the goal of my PhD thesis. Then the actual defence started lasting for exactly one hour (no more, no less) in which the committee members asked me questions about my thesis and the accompanied propositions.
At Delft University of Technology and any other Dutch university, there is a lot of ceremonial stuff involved with the PhD defence. Professors have to be dressed up in gowns. Me and my paranimphs (the two people sitting in front of me who help me and take over my defence if I pass out) had to be dressed up in suits. I have to address to committee members formally depending on their roles, such as: "hooggeleerde opponent" and the committee members had to formally address me with "waarde promovendus".
Although I was experiencing some nervousness at the beginning of the day, as I didn't know what exactly was going to happen and what kind of questions I would receive, in the end it was fun and I liked it very much. After the defence I received my PhD diploma:
For people that want to know what my thesis is exactly about:
Apart from the contents of my PhD thesis, theses at our university are usually accompanied by a list of propositions. According to our university's Doctorate regulations, they must be defendable and opposable, at least six of the propositions are not supposed to be directly related to the research subject and two of them may be slightly playful. Besides the contents of my thesis, committee members are also allowed to ask questions about the propositions.
A colleague of mine: Felienne Hermans, has covered her propositions on her blog to elaborate about them. I have decided to do the same thing, although I'm not planning to create separate blog posts for each individual proposition. Instead, I cover all of them in a single blog post.
Propositions
Many of the non-functional requirements of a deployed service-oriented system can be realized by selecting an appropriate software deployment system.
As I have explained earlier in a blog post about software deployment complexity, systems are rarely self-contained but composed of components. An important property of a component is that it can be deployed independently, significantly complicating a software deployment process.
Components of service-oriented systems are called "services". It's a bit of an open debate to exactly tell what they are, since people from industry often think in terms of web services (things that use SOAP, WSDL, UDDI), while I have also seen the description "autonomous platform independent entities that can be loosely coupled" in the academic literature.
Although web services are some sort of platform independent entities, they still have implementations behind their interfaces depending on certain technology and can be deployed to various machines in a network. We have seen that deployment on a single machine is hard and that deploying components into networks of machines is even more complicated, time consuming and error prone.
Besides deployment activities, there are many important non-functional requirements a system has to meet. Many of them can be achieved by designing an architecture, e.g. components, connectors and the way they interact through architectural patterns/styles. Architectural patterns (e.g. layers, pipes and filters, blackboard etc.) implement certain quality attributes.
For service-oriented systems, it's required to deploy components properly into a network of machines to be able to compose systems. In other words: we have to design and implement a proper deployment architecture. This has several technical challenges, such as the fact that we have to deploy components in such a way that they exactly match the deployment architecture, the deployment activities themselves, and the fact that we have to determine whether a system is capable of running a certain component (i.e. a service using Windows technology cannot run on a Linux machine or vice-versa).
In addition to technical constraints, there are also many non-functional issues related to deployment that require attention, i.e. where to place components and how to combine them to achieve certain non-functional requirements? For example, privacy could be achieved by placing services providing access to privacy-sensitive data in a restricted zone and robustness by deploying multiple redundant instances of the same service.
It can also be hard to manually find a deployment architecture that satisfies all non-functional requirements. In such cases, deployment planning algorithms are very helpful. In some cases it's even too hard or impossible to find an optimal solution.
Because of all these cases, an automated deployment solution taking all relevant issues into account is very helpful in achieving many non-functional requirements of a deployed service-oriented system. This is what I have been trying to do in my PhD thesis.
The intention of the Filesystem Hierarchy Standard (FHS) is to provide portability among Linux systems.
However, due to ambiguity, over-specification, and legacy support, this standard limits adoption and innovation. This phenomenon applies to several other software standards as well.
There are many standards in the software engineering domain. In fact, it's dominated by them. One standard that is particularly important in my research is the Filesystem Hierarchy Standard (FHS) defining the overall filesystem structure of Linux systems. I have written a blog post on the FHS some time ago.
In short: the FHS defines the purposes of directories, it makes a distinction between static and variable parts of a system, it defines hierarchies (e.g. / is for boot/recovery, /usr is for user software and for /usr/local nobody really knows (ambiguity)). Moreover, it also defines the contents of certain directories, e.g. /bin should contain /bin/sh (over-specification).
I have problems with the latter two aspects -- the hierarchies do not provide support for isolation, allowing side-effects to easily manifest themselves while deploying and enabling destructive upgrades. I also have a minor problem with strict requirements of the contents of directories, as they easily allow builds to trigger side-effects while assuming that certain tools are always present.
For all these reasons, we deviate on some aspects of the FHS in NixOS. Some people consider this unacceptable, and therefore they will not be able to incorporate most of our techniques to improve the quality of deployment processes.
Moreover, as the FHS itself has issues, we observe that although the filesystem structure is standardized, file system layouts in many Linux distributions are still slightly different and portability issues still arise.
In other domains, I have also observed various issues with standards:
Operating systems: nearly every operating system is more or less forced to implement POSIX and/or the Single UNIX Specification, taking a lot of effort. Furthermore, by implementing these standards they UNIX is basically reimplemented. These standards have many strict requirements on how certain library calls should be implemented, although it also specifies undefined behaviour at the same time. Apart from the fact that it's difficult and time consuming to implement these standards, there is little room to implement an operating system that is conceptually different from UNIX as it conflicts with portability.
The Web (e.g. HTML, CSS, DOM etc.): First, a draft is written in a natural language (which is inherently ambiguous) and sometimes underspecified. Then vendors start implementing these drafts. As initially these standards are ambiguous and underspecified, implementations behave very differently. Slowly these implementations converge into something that is uniform by collaborating with other vendors and the W3C to improve the draft version of a standard. Some vendors intentionally or accidentally implement conformance bugs, which don't get fixed for quite some time.
These buggy implementations may become the de-facto standard, which has happened in the past, e.g. with Internet Explorer 6, requiring web developers to implement quirks code. Since the release of Internet Explorer 6 in 2001, Microsoft had 95% market share and did not release a new version until 2006. This was seriously hindering innovation in web technology. It also took many years before other implementations with better web standards conformance and more features gained acceptance.
So are standards bad? Not necessarily, but I think we have to critically evaluate them and not consider them as holy books. Moreover, standards need to be developed with some formality and elegance in mind. If junk gets standardized, it will remain junk and requires everybody to cope with junk for quite some time.
One of the things that may help is using good formalisms. For example, a good one I can think of is BNF that was used in the ALGOL 60 specification.
To move the software engineering community as a whole forward, industry and academia should collaborate. Unfortunately, their Key Performance Indicators (KPIs) drive them apart, resulting in a prisoner's dilemma.
It's obvious that, if both fractions would let themselves go a bit from their KPIs, i.e. academia does a bit more in engineering tools and transferring knowledge, while industry spends some of their effort in experimenting and paying attention to "secondary tasks", that both parties would benefit. However, in practice often the opposite happens (although there are exceptions of course).
This is analogous to a prisoner's dilemma, which is a peculiar phenomenon. Visualize the following situation: two prisoners have jointly committed a crime and got busted. If both confess their crime then the amount of time they have to spend in prison are five years. If one prisoner confesses while the other does not, then the prisoner that confesses goes ten years into jail and the other remains free. If none of them confess, they both have to spent twenty years in prison.
In this kind of situation the (obvious) win-win situation for both criminals is that they both confess. However, because they both give priority to their self-interests, none of them confesses as they assume that they remain free. But instead, the situation has the worst outcome: both have to spent twenty years in prison.
In software engineering, the use of social media, such as blogging and twitter, are an effective and efficient way to strengthen collaboration between industry and academia.
This proposition is related to the previous one. How can we create a win-win situation? Often I hear people saying: "Well collaboration is interesting, but it costs time and money, which we don't have right now and we have other stuff to do".
I don't think the barrier has to be that high. Social media, such as blogging and twitter, can be used for free and allows one to easily share stories, thoughts, results and so on. Moreover, recipients can also share these with people they know.
My blog for example, has attracted many more readers and has given me much more feedback then all my research papers combined. Moreover, I'm not limited by all kinds of contraints that program committee members impose on me.
However these observations are not unique to me. Many years ago, a famous Dutch computer scientist named Edsger Dijkstra wrote many manuscripts that he sent to his peers directly. He wrote about subjects that he found relevant. His peers spread these manuscripts through their colleagues allowing him to reach a wide range of people, eventually reaching thousands of people.
While the vision behind the definition of free software as described by the Free Software Foundation
to promote freedom is compelling, the actual definition is ambiguous and inadequately promoted.
The free software definition defines four freedoms. I can rephrase them in one sentence: "Software that can be used, studied, adapted and shared for any purpose". An essential precondition for this is the availability of the source code. I think this definition is clear and makes sense.
However, there is a minor issue with the definition. The word 'free' is ambiguous in English. In the definition, it refers to free as in freedom not free in price (gratis). In Dutch or French it's not a problem. In these languages free software translates to 'vrije software' and 'libre software'.
Moreover, although (almost) all free software is gratis, it's also allowed to sell free software for any price, which is often misunderstood.
I have seen that the ambiguity of the word free is often used as an argument why the definition is not attracting a general audience.
I think there is a bigger problem: the way free software is advertised. Most advertisements are not about what's good about free software, but about what's bad about proprietary software and how evil certain practices of certain companies are.
Although I don't want to say that they are not right and we should tolerate such bad practices, I think it would also help to pay more attention to the good aspects of free software. The open source definition has much more care for this. For example, being able to improve quality of software. That's something I think that would attract people from the other side, whereas negative campaigning does not.
Compared to the definition of free software provided by the Free Software Foundation, the definition of Open Source as provided by the Open Source Initiative, fails to improve on freedom. While it has been more effectively promoted, it lacks a vision and does not solve ambiguity.
The open source definition lists ten pragmatic points with the intention of having software that is free (as in freedom), e.g. availability of source code, means to share modified versions, and so on. However, it does not explain why it's desired for others to respect these ten pragmatic points and what their rationale is (although there is an annotated definition that does).
Because of these reasons, I have seen that sometimes software is incorrectly advertised as being open-source, while in reality they are not.
For example, there is also software available with source code, for which it is not allowed to do commercial redistributions, such as the LCC compiler. That's not open source (nor free software). Another prominent example is Microsoft's Shared Source initiative, only allowing someone to look at code, but not to modify or redistribute it.
A very useful aspect of open source is the way it's advertised. It pays a lot of attention in selling its good points. For example, that everyone is able to improve its quality, and allowed to collaborate etc. Companies (even those that sell proprietary software) acknowledge these positive aspects and are sometimes willing to work with open-source people on certain aspects or to "open-source" pieces of their proprietary products. Examples of this are the Eclipse platform and the Quake 3 arena video game.
Just like making music is more than translating notes and rests into tones and pauses with specific durations, developing software systems is more than implementing functional requirements. In both domains, details, collaboration and listening to others are most important.
I have observed that in both domains we make estimations. In software development, we try to estimate how much effort it takes to develop something and in music we try to estimate how much effort it takes to practice and master a composition.
In software development, we often look at functional requirements (describing what a system should do) to estimate. I have seen that sometimes functional requirements may look ridiculously simple, such as displaying tabular data on a web page. Nearly every software developer would say: "that's easy".
But even if functional requirements are simple, certain non-functional requirements (describing how and where) could make it very difficult. For example, properly implementing security facilities, a certain quality standard (such as ISO 9126) or to provide scalability. These kind of aspects may be much more complicated than the features of a system itself.
Moreover, software is often developed in a team. Good communication and being able to divide work properly is important. In practice, you will almost always see that something goes wrong with that, because people have assumptions that all details are known by others or there is no clear architecture of a system so that work can be properly divided among team members.
These are all kinds of reasons that may result in development times that are significantly longer than expected and failure to properly deliver what clients have asked.
In my spare time I'm a musician and in music I have observed similar things. People make effort estimations by looking at the notes written on paper. Essentially, you could see those as functional requirements as they tell you what to play.
However, besides playing notes and pausing, there are many more aspects that are important, such as tempo, dynamics (sudden and gradual loudness) and articulation. You could compare these aspects to non-functional requirements in software, as they tell somebody how to play (series of) notes.
Moreover, making music can also be a group effort, such as a band or an orchestra, requiring people to properly interact with each other. If others make mistakes they may confuse you as well.
I vividly remember a classical composition of 15 years ago. I just joined an orchestra and we were practising: "Land of the Long White Cloud" by Philip Sparke. In the middle of the composition, there is a snare drum passage consisting of only sixteenth notes. I already learned playing patterns like these in my first drum lesson, so I thought that it would be easy.
However, I had to play these notes in a very fast tempo and very quietly, which are usually conflicting constraints for percussionists. Furthermore, I had to keep up the right tempo and don't let the other members distract me. Unfortunately, I couldn't cope with all these additional constraints, and that particular passage had to be performed by somebody else. I felt like an idiot and I was very disappointed. However, we did win the contest in which we had to perform that particular composition.
As a side note: "The land of the long white cloud" is also known as Aotearoa or New Zealand.
Multilingualism is a good quality for a software engineer as it raises awareness that in natural languages as well as in software languages and techniques, there are things that cannot be translated literally.
It's well known that some words or sentences cannot be literally translated from one natural language to another. In such cases, we have to reformulate a sentence into something that has an equivalent meaning, which is not always trivial.
For example, non-native English speakers, like Dutch people, tend to make (subtle) mistakes now and then, which sometimes have hilarious outcomes. Make that the cat wise is a famous website that elaborates on this, calling the Dutch variant of English: Dunglish.
Although we are aware of the fact that we cannot always translate things literally in natural languages, I have observed that in the software domain the same phenomenon occurs. One particular programming language may be more useful for a certain goal, than another programming language. Eventually, code written in a programming language gets compiled into machine code or another programming language (having an equivalent reformulation in a different language) or interpreted by an interpreter.
However, I have also observed that in the software engineering domain there is a lot of programming language conservatism. Most conventional programming languages used nowadays (Python, C++, Java, C# etc.) use structured and imperative programming concepts in combination with class-based OO techniques. Unconventional languages such as purely functional programming languages (Haskell) or declarative languages (Prolog, Erlang) only get little mainstream acceptance, although they have very powerful features. For example, programs implemented in a purely functional language easily scale across multiple cores/processors.
Instead, many developers use conventional languages to achieve the same, imposing many additional problems that need to be solved and more chances on errors. Our research based on the purely functional deployment model also suffers from conservatism. Therefore, I think multilingualism is very powerful asset for an engineer, as he is not limited by a solution set that is too narrow.
Stubbornness is both a positive as well as a negative trait of a researcher.
I think that when you do research and if you discover something that is uncommon, others may reject it or tell you to do something that they consider more relevant. Some researchers choose to comply and give up stuff that they think is relevant. If every scientist would do that, then I think certain things would have never been discovered. I think it's a scientist's duty to properly defend important discoveries.
In the middle ages it was even worse. For example, a famous scientist named Galileo Galilei revealed that not the Earth but the Sun was the centre of our solar system. He was sentenced to house arrest for the rest of his life by the catholic church.
However, stubbornness also has a negative aspect. It often comes with ignorance and sometimes that's bad. For example, I have "ignored" some advice about properly studying related work and taking evaluation seriously, resulting in a paper that was badly rejected.
Conclusion
In this blog post, I have described some thoughts on my PhD propositions. The main reason of writing this down is to prepare myself for my defence. I know this blog post is lengthy, but that's good. This will probably prevent my committee members to read all the details, so that they cannot use everything I have just written against me :-) (I'm very curious to see if anyone has notice that I just said this :P).
