Friday, August 24, 2012

A review of conferences in 2010

Previously, I wrote a review of the conferences I have attended in 2008-2009. In this blog post, I will cover those I have visited in 2010. 2010 was a very busy conference year for me and in order to keep things short, I have decided to dedicate a full blog post to this year.

FOSDEM: Free and Open Source Software Developers' European Meeting (2010)


The first conference that I attended in 2010 was FOSDEM, which I visited for the second time. As with my visit last year, the experience was nearly identical - the same building with the same "peculiar atmosphere" and the same kind of enthusiastic people. I also slept in the same hotel in Brussels (whose staff is unable to speak Dutch, for some reason).

During this years' edition I have met quite a number of interesting people. Two of them were Nix developers I have frequently spoken with on the IRC, but never met in person.

Nicolas Pierron gave a talk about NixOS and the NixOS configuration system, which he had implemented in his spare time.

Apart from a number of Nix developers, we have also seen some members of the competing Macoosi project. Eelco and I met the RPM5 maintainer (also member of Mancoosi). He had some interesting ideas about package management and he proposed to merge our approaches (i.e. the purely functional model), although he did not clearly explain how. Although some attempts have been made by the Mancoosi members, it did not result into anything useful, unfortunately.

SEAA: EUROMICRO Conference on Software Engineering and Advanced Applications (2010)


The next conference of 2010 was another academic one - SEAA, which was a conference I did not know about. I ended up visiting this conference, because my CBSE paper submission was rejected.

The rejection was actually not a surprise. I really did a poor job. Parts of the paper were incomplete, I tried to put too many technical details in it and I did not study related work too well. Most of these flaws were due to time pressure.

Although it was disappointing to have a paper rejected, the good thing about it was that the reviewers listed all my flaws, suggested how certain sections of the paper could be improved and they gave me a number of references that I should study. A few days later I received an e-mail in which they encouraged me to submit my revised paper to SEAA, which I did.

As a sidenote: I think this is something that more reviewers of conference papers should do. Nearly all my other rejections always resulted in a few vague comments, which were not useful to me at all, nor gave me any insights in what to revise.


The SEAA conference was located in France, in a city called Lille (which apparently has a Dutch translation: Rijsel) and held at the Polytech' Lille. I had no idea what to expect from this conference, but my experience was quite positive. The conference had very friendly and enthusiastic organizers. Some of them apparently have a long organizing reputation and knew many participants quite well.


Moreover, the conference was co-located with the EUROMICRO DSD (Digital System Design) conference and the keynote sessions of SEAA and DSD were shared. During these keynotes, I also learned a thing or two about embedded systems and VHDL. A few keynotes were given by industry people.


The social event was a dinner at Chateau de Bourgogne in Estaimbourg and the wine tasted good :-).

Unfortunately, I had to present my paper the next day as the second presenter, in the first presentation slot (somewhere at 9:00 in the morning). I really had a hard time getting up and to get myself motivated, but it turned out that I was well prepared - nobody was sleeping during my presentation and I received a number interesting questions.

WASDeTT: Workshop on Academic Software Development Tools and Techniques (2010)


The next event that I attended was WASDeTT, which was co-located with ASE and held at the University of Antwerp.

As explained earlier, my CBSE submission was rejected partially because I included too much implementation details. Some time later I "discovered" this workshop that is focused on the development of tools within academia and development aspects in general. For that reason, I have decided to write a tool-oriented paper for Disnix, covering the latest version, implementation details and extensions. Because the page limit was 25 pages, I could write down all the stuff that I could not do in the SEAA paper.

The workshop was not about the tools themselves, but rather about their development aspects. So instead of presenting the concepts of Disnix again, I have explained all its related tool development aspects and development choices. We have had some interesting discussions, most notably - how to make research tools available to other researchers, which (of course) is a deployment issue.

One of the ideas was making a tool available as a web application. I also suggested Nix, as it provides complete dependencies and reproducible deployment. The suggestion, however, was slammed by a fellow workshop participant stating that Nix is nice, but a "conquer-the-world approach".

ASE: International Conference on Automated Software Engineering (2010)



I have also attended ASE, the host conference of the WASDeTT workshop. ASE is a conference that was known as KBSA (Knowledge Based Software Assistant) in the past, as it purpose was to use artificial intelligence techniques to automate all phases of a software engineering process. Nowadays, the domain has been broadened to software engineering in general and ASE is more or less a general conference on nearly the same level as ICSE, with a comparable paper acceptance rate.

Before I left, I considered Antwerp a very "boring" place compared to all the previous conference locations (as I have visited Antwerp many times and I can speak to its residents in Dutch, my native language), but the entire experience was much much better than I expected.


One of the good things about ASE compared to ICSE, is that it is less crowded and it has fewer parallel sessions (at ASE there were only 2 parallel sessions, while at ICSE there are 6 or even more, I'm not sure :-) ). At ASE, it was easier for me to meet new people and to have discussions.

Two papers were very interesting at ASE. One of them was from the Mancoosi project (our "competitors"), in which they have used SAT-solvers for solving dependency resolution problems. The other one was about a tool called Ninka, which can be used to derive licenses from source files using sentence matching techniques. We have used Ninka in our research as well.


ASE included two social events - a beer tasting event in the city hall (which was interesting, because I have seen the city hall many times, but I have never seen it from the inside) and a banquet at the Antwerp Zoo. After both events, I noticed that several conference attendees were unable to walk back to their hotels in a straight line. I have to admit that it took me a bit of effort too :-)

ISSRE: International Symposium on Software Reliability Engineering (2010)


ISSRE was the third academic conference that I have attended in 2010. ISSRE is a conference about reliability engineering and related subjects, such as testing. We ended up there, because our ICSE submission was rejected (although we made it very close). We have turned the testing part of this rejected paper into a paper for ISSRE.

ISSRE was also an unknown conference to me and held in a suburb of San Jose, California, which was basically composed of a large number of Cisco buildings. The conference itself was also held at Cisco.


I've noticed that the conference also had very friendly and enthusiastic organizers and a loyal group of visitors that knew each other quite well. Furthermore, the organizers and session chairs were eager to meet all the new/unknown participants including me and Eelco.

A lot of the papers covered topics that were completely unknown to me. For example, I still remember a paper covering a software aging analysis of the Linux kernel. Before attending the presentation, I had no clue what software aging was. In turned out that it was about "progressive performance degradation or a sudden hang/crash of a software system due to exhaustion of operating system resources, fragmentation and accumulation of errors".

There were many more interesting papers. Another paper I liked was about fixing certain undefined features in the C programming language and the "political process" of the C standards committee.

There was also another presentation in which (apparently) something went wrong, and the speaker cried some kind of F-word out loud :-)


Another good aspect of this conference was the high industry participation degree of which the organizers were very proud. I've talked to one of them about other conferences that I've attended, such as ICSE, and he said to me that these conferences are "so highly academic", which turned out that this is indeed true, unfortunately (which does not implicate that they are useless, of course).


The social event was held at the Tech Museum in San Jose, which had some exciting attractions, such as an earthquake simulator and an interesting exercise in which you had to construct a road by using a number of random objects, so that the ball can safely roll from the entrance to the exit. As you may see in the picture above, Eelco is a smart guy and knew how to do it.

LAC: Landelijk Architectuur Congres (translation: 'National Architecture Congress') (2010)


The LAC (Dutch National Architecture Congress) was the last conference I have attended in 2010, which was basically an industry oriented conference with a bit of research participation. Furthermore, nearly all the presentations were in Dutch.

I ended up visiting this conference, because our funding agency: NWO/Jacquard had its own session and I was asked to give a presentation about PDS, the research project I'm participating in.

Apart from giving a presentation about my project, I also took some time to explore the conference venue. There were a lot of companies there presenting themselves (probably to attract new potential employees), showing all kinds of pictures representing software architectures.

One of those pictures captured my interest, because it looked like a painting and it did not make sense to me.

I asked one of the presenters what the picture is supposed to mean and I heard a story that I did not understand. After a while, he started talking about his projects, such as the traffic light systems, and suddenly the conversation became much more interesting.

Another funny thing is that the conference was dominated by certain buzz words, such as Agile and TOGAF (which I have never heard of previously). It turned out that TOGAF is an framework developed by The Open Group providing a comprehensive approach for designing, planning, implementing, and governing an enterprise information architecture.

One of the major differences I have noticed between this conference and an academic conference covering software architectures, is that industrial conferences are dominated by tools and standards, whereas academic conferences are mostly about concepts.

Conclusion


In this blog post I have described all conferences that I have attended in 2010. Stay tuned, as there are two more years to come...

Monday, July 30, 2012

A review of conferences in 2008-2009

As some of you may know (and some others probably don't :-) ), I have visited several conferences, which I'm required to attend as part of my job as PhD student. Most of the conferences are academic conferences where I have to present papers that I publish. In contrast to many other research fields - in software engineering - we typically publish at conferences instead of journals.

Apart from academic conferences, I have visited a number of non-academic conferences as well, because apart from being a researcher, I'm also a person interested in technology and development.

In this blog post, I'm going to list the conferences I have visited the past few years and I'm going to share some experiences. The reason that I have written this blog post is basically because of what I have written earlier about software engineering fractions. It's not to have a rant about these conferences, but to share my personal experiences about the topics, audience, peculiarities and other traits, because I think it's important to know how to reach several types of audiences instead of living on an isolated island.

The conference experiences here are in chronological order. In order to keep blog posts short, I start with the first two years (2008-2009):

DSM: Workshop on Domain-Specific Modeling (2008)


This is the first academic event I have ever attended. DSM is a workshop about Domain-Specific Modeling and is typically co-located with OOPSLA (nowadays known as SPLASH). I have attended the 2008 edition in Nashville, Tennessee, USA, which was also my first trip to the USA.

I didn't have to present anything here, but the main reason of attending was that students were allowed to attend any co-located event at OOPSLA for free and one of my colleagues had to present a paper there.

The workshop covers short papers about both theoretical and practical aspects of domain specific modeling. I have seen many kinds of submissions, ranging from simple ideas to a number of tools that have a significant practical use. I have even seen some submissions that me and my colleagues still laugh about today (including their paper).

One of the nice aspects of this workshop is that there is a lot of room for discussion with the attendants and a relatively low barrier of getting your idea published (this is both positive as well as a negative thing).

The attendants of this workshop are mostly academic people, but also quite a number of industry people, showing industrial applications. Below I have included a picture taken of the attendants (people who know me will surely recognise me :-) )


HotSWUp: Workshop on Hot Topics in Software Upgrades (2008)


HotSWUp is the second academic event I have attended. HotSWUp is a cross-disciplinary workshop about software upgrades in any sub field, such as programming languages, operating systems, distributed systems and so on. This event was also co-located with OOPSLA and held in Nashville.

In this workshop, I had to present my first research paper, covering distributed atomic upgrades, an important aspect of my masters thesis.

HotSWUp was a very great workshop and a great experience for me. There were many nice papers covering great ideas, such as distributed upgrades, package management upgrades and how to safely mirror software components with untrusted parties. Even though many of these papers are not directly applicable to my research, they still have a fair amount of practical usage that I liked very much.

I received interesting questions, suggestions and feedback. Furthermore, there was also a lot of room for discussion that gave me many interesting ideas and several new research directions.

I have also met a number of interesting people close to my research. I met two "competitors" from the Mancoosi project (one of them is now the current Debian project leader), who also investigate package management related issues. We had a very nice opportunity to get to acquainted and share ideas about package management. I also met two other people who did similar kind of research with distributed systems.


OOPSLA: Object-Oriented Programming, Systems, Languages & Applications (2008)


OOPSLA is the first academic conference I have attended (which was basically the host conference of the two previous workshops). Actually, OOPSLA was the only conference I knew about before I started my PhD and I very much liked the fact that I was able to attend it. Moreover, another colleague of mine had to present a full paper there and several others had a poster submission about WebDSL.

The papers of OOPSLA were mostly about practical aspects of Object-Oriented programming languages or related issues, such as garbage collection. There were quite a number of interesting papers showing good practical usage, such as the fact that using inverse lookups is better way of refactoring code. I saw an interesting paper using Robocode as a case-study, to which I was addicted when I started learning programming in Java.

There were also some presentations that were a bit hard to follow. For example, I still remember this presentation showing ovals with curly lines attached to it. When I saw those figures it reminded me of something totally different (which was probably not something that the presenter has intended :-) )

Another funny thing I have noticed during the presentations, was some guy in the audience, who was always sleeping during the presentations and could not keep his eyes open.

Apart from attending presentations, I also attended two tutorials, which were also free for students. I learned a thing or two about the Scala programming language, which combines object-oriented, functional programming and several other concepts in a new programming language which builts on top of the Java virtual machine. I also did a Smalltalk tutorial involving the Squeak implementation, which can also be used to develop web applications in Smalltalk.

Nowadays, OOPSLA no longer specifically focuses itself on object-oriented programming languages anymore. It's even not focused on programming languages, but on software engineering in general. To reflect the broadened scope, the new name of the conference is SPLASH: Systems, Programming, Languages and Applications: Software for Humanity.

FOSDEM: Free and Open Source Software Developers' European Meeting (2009)


FOSDEM was the next and a non-academic that I have visited. I was not required to visit it nor I had to present something. The main reason of attending it, was because I have always been involved with free and open source software.

FOSDEM is the biggest free and open-source event in Europe in which virtually every well known free and open-source project is represented, such as the X Window System, the Linux kernel, KDE, Mozilla, GNOME etc. FOSDEM is always held in Brussels at the campus of the Université libre de Bruxelles and can be visited by anyone interested for free (gratis).

When I arrived there for the first time, I was actually a bit "surprised" about the conference venue (the quality of the buildings were a bit "different" than I expected, after the impression I got when looking at their website :-) ).


I also noticed that there were an overwhelming amount of attendants. At the keynote presentation, the entire Janson auditorium is completely filled with free and open source enthusiasts, as shown in the picture above. Some of them were really, really enthusiastic about the stuff they are working, as may see in the picture when looking at the headgear of some of the attendants.

Apart from a filled Janson auditorium room, it was also very difficult to move from one room to another or to look for somebody you know, because the hallways were always full of people.

The quality of the talks is -- as with many conferences -- of varying quality. I liked the talks about GEM/KMS and Ext4. There were also a number of talks, that were hard to follow and not well prepared. In contrast to academic conferences, where the session chair typically has a question for you (when nobody else has one), here at FOSDEM, the session chair has no mercy and immediately announces the next speaker.

ICSE: International Conference on Software Engineering (2009)



ICSE was the second top general software engineering conference I have attended. The main reason of attending it was because I had to present a paper at the co-located cloud computing workshop.

This conference was held in Vancouver, British Columbia, Canada and my visit was actually a bit special, as it was the first time for me to travel by plane alone and to travel such a long distance. :-)

The ICSE conference is a very broad conference covering topics in the entire software engineering spectrum, ranging from requirements engineering, testing, model checking and collaboration. Although software deployment is also listed on their call for papers, I haven't encountered any papers covering this subject in the last few years, unfortunately.

This years' ICSE edition had a very good atmosphere and I've met quite a number of interesting people there as well as some familiar ones from OOPSLA and HotSWUp. One of the papers presented there resulted in a collaboration many years later.

I also had to cheer for my former colleague: Ali Mesbah who presented his paper: 'Invariant-Based Automatic Testing of Ajax User Interfaces' for which he won the ACM SIGSOFT Distinguished paper award.

The social event was held at the Vancouver Aquarium and included a nice show with dolphins. Apart from the conference, I also took a few days off to explore the environment and I went to many interesting places, such as Stanley Park, Dr. Sun-Yat Sen Garden and Granville Island.

I also ended up in an infamous street called: 'East Hastings', for which I have been warned not visiting it, a few days later (so you have been warned now!!). I have some interesting pictures included below:


ICSE-Cloud: ICSE Workshop on Software Engineering Challenges in Cloud Computing (2009)



The ICSE Cloud workshop was the last event I have attended in 2009 in which I had to present a paper. The picture above, is taken by Jan S. Rellermeyer who also attended this workshop and HotSWUp, which I have described earlier.

Apart from a very good ICSE experience, the ICSE cloud shop was also a very nice workshop, with lots of discussions. Another important aspect was the high industry participation degree and especially the first keynote, given by somebody from Amazon, the company who basically pioneered the 'cloud computing' term.

Although I have to admit that the paper I had to present there was not one of my strongest contributions, it was actually quite well received there. Furthermore, for some unknown reasons, it also my most downloaded and cited paper.

Conclusion


This blog post reports about my conference experiences from the begin period of my career as a PhD researcher. In a future blog post I'll report about the next years, so stay tuned...

Friday, June 22, 2012

IFF file format experiments

I haven't written any blog post for a while, but don't worry... I'm still alive, though very busy with writing my PhD thesis. There is still a fun project that showed some interesting results a while ago, but so far I never allowed myself to take the time to report about it.

A while ago, I wrote a blog post about my second computer, the Commodore Amiga. I also mentioned that one of my favourite Amiga programs was Deluxe Paint, which stores images in a so-called "IFF file format", which was ubiquitous on the AmigaOS and supported by many Amiga programs.

Nowadays, this file format is rarely used and also poorly supported in modern applications. Most common viewers cannot open it, although a number of more advanced programs can. However, the quality of their implementations typically differ as well as the features that they support.

What is the IFF file format?


IFF is an abbreviation for Interchange File Format. Quite often, people think that it is just a format to store images, as the most common IFF application format is the InterLeaved BitMap (ILBM) format used by Deluxe Paint and many other programs.

In fact, IFF is a generic purpose container format for structuring data. Apart from pictures, it is also used to store 8-bit audio samples (8SVX), musical scores (SMUS), animations (ANIM) and several other formats. The IFF file format as well as a number of application file formats were designed by Electronic Arts, nowadays a well-known game publishing company, and described in several public domain specifications, which everybody was allowed to implement.

The confusion with the IFF file format is similar to the OGG file format, which are quite often mistakenly identified as Vorbis audio files, as Vorbis is the most common OGG application file format. In fact, OGG is a container format for bitstreams, while Vorbis is an application format to provide lossy audio compression and decompression. There are many other OGG application formats, such as Theora (for video) and Speex (for speech).

IFF concepts


Conceptually, IFF files have a very simple structure. Every IFF file is divided into chunks. Each chunk consists of a 4 character identifier followed by a signed 32-bit integer describing the chunk size, followed by the given amount of bytes representing data:



The picture above shows a very simple example chunk with identifier: 'BODY' which contains 24000 bytes of data. The data in the chunk body represents pixel data.

Although the concept of IFF files using chunks is ridiculously simple, it immediately offers a number of useful features for handling file formats. By looking at a chunk identifier, a program can determine whether it contains useful information to present to end users or whether a chunk is irrelevant and can be skipped. Furthermore, the chunk sizes indicate how much data has to be read or how many bytes must be skipped to reach the next chunk. Using these attributes make it possible to implement a robust parser capable of properly retrieving the data that we want to present.

In principle, every chunk captures custom data. Apart from data chunks, the IFF standard defines a number of special group chunks, which can be used to structure data in a meaningful way. The IFF standard defines three types of group chunks:


  • The FORM chunk, contains an arbitrary collection of data chunks or other group chunks, as shown in the picture above. Our example, defines a FORM which has the type ILBM. In principle, every application file format is essentially a form in which the form type refers to the application file format identifier. In the body of the FORM several data chunks can be found:
    • The BMHD defines the bitmap header containing various general settings, such as the width, height and the amount of colors used.
    • The CMAP defines the red, green and blue color channel values of each color in the palette.
    • The BODY chunk contains pixel data.
  • The CAT chunk may only contain a collection of group chunks, that is only FORM, CAT or LIST chunks.
  • The LIST chunk is an extended CAT chunk that also contains a number of PROP chunks. PROP chunks are group chunks which may only reside in a list and contain a collection of data chunks. These data chunks are shared properties of all group chunks inside the LIST. For example, a LIST containing ILBM FORM chunks, may use a PROP chunk containing a CMAP chunk, which purpose is to share the same palette over a number of bitmap images.

Application file formats can define their own application specific chunks and their attributes. For example, the ILBM file format defines the BMHD data as BitMap Header chunk, containing important attributes of an image, such as the width, height and the amount of colors used and the BODY chunk that stores the actual graphics data.

Apart from these basic concepts, IFF has a number of other small requirements:
  • If a chunk size is odd, then the chunk data must be padded with an extra 0 byte, so that the next chunk is always stored on an even address in memory (as shown in the example form). This requirement was introduced, because the 68000 processor (which the Amiga uses) processes integers much faster on even addresses in memory. In our example form shown earlier, the CMAP chunk is padded.
  • Also application file format attributes of word and long word sizes, must be word aligned (stored on even addresses in memory).
  • All integers must be big-endian, because the Amiga was a big-endian system. This means that on little-endian systems, such as PCs, the byte order of integers has to be reversed.

IFF file format support


The IFF file format is yet simple, but also powerful and served it purpose really well when the Amiga was still alive. For a very large and cool experiment (which I will keep secret for a while) I wanted to open ILBM images in a SDL application (a cross-platform library frequently used to develop games and multimedia applications), as well as modifying ILBM files and saving them. I ran into several issues:

  • Support for most IFF application formats is not present in many common viewers and players. However, some more advanced programs support it. For example, Paint Shop Pro and the SDL_image library have support for viewing ILBM images.
  • These applications all have their own implementation of a specific IFF application format. Some implementations are good, others lack certain features. For example, I have seen several viewers not supporting the special Amiga screen modes, such as Extra HalfBrite (EHB) and Hold-and-Modify (HAM) or the color range cycle chunks.
  • Applications can open simple IFF files that consist of a single FORM, but do not know how to deal with IFF scrap files, i.e. CATs/LISTs containing multiple FORMs of various types, possibly with shared options.
  • Most applications can view IFF application formats, but cannot write them or check for their validity, which may result in crashes if invalid IFF files are opened.
  • A number of open file formats have generic parser libraries, e.g. PNG (libpng), JPEG (libjpeg), GIF (giflib), Ogg (libogg), Vorbis (libvorbis) etc. that applications use to open, parse and save files. There is no equivalent for ILBM and other IFF application formats.

IFF libraries experiment


So after I ran into these issues I've decided to take a look at the IFF specification to see how hard it could be to implement the stuff I needed. After reading the standard, I started appreciating the IFF file format more and more, because of the simplicity and the practical purpose.

Furthermore, instead of implementing yet another crappy parser that only supports a subset, I have decided to do it right and to develop a set of general, good quality, reusable and portable libraries for this purpose, with similar goals to the other file format libraries so that application programs can support IFF application file formats as easy as the common file formats that we use nowadays.

I also think it's good to have file formats which used to be widely used, properly supported on modern platforms. Finally, it looks like fun, so why not doing it?? I did a few experiments that resulted in a number of interesting software packages.

Implementing a SDL ILBM viewer


First, I have decided to implement support for my primary use case: Proper ILBM image support in SDL applications. I have implemented a SDL-based viewer program, having the following architecture:


In the picture above, several components are shown:

  • libiff. This library implements the properties defined in the IFF specification, such as parsing data chunks and groups chunks. Furthermore, it also supports writing IFF files as well as conformance checking.
  • libilbm. This library implements the application chunks as well as the byte run compression algorithm defined in the ILBM specification. Furthermore, it supports several extension chunks and the file format used by the PC version of Deluxe Paint (which has several minor differences compared to Amiga version). Application chunks can be parsed, by defining a table with function pointers to the ILBM functions that handle these and to pass the table to the IFF library functions.
  • libamivideo. This library acts as a conversion library for Amiga graphics data. As explained earlier, the Amiga uses bitplanes to organise graphics and has several special screen modes (Extra-Halfbrite (EHB) and Hold-and-Modify (HAM)) to display more colors out of the predefined color registers. In the SDL viewer we use the libamivideo library to convert Amiga graphics data to chunky or RGB graphics and to emulate the special screen modes.

    Images saved by the PC version of Deluxe Paint however (which have the PBM form type instead of ILBM), do not use bitplanes but chunky graphics, and thus conversion is not necessary.
  • SDL_ILBM. This package contains a high level SDL library as well as the ilbmviewer command-line tool, directly generating SDL surfaces from IFF files containing ILBM images as well as performing the required conversions automatically.

Usage of the SDL ILBM viewer is straight forward:

$ ilbmviewer picture.IFF

The viewer can also be used view IFF scrap files. For example, it may be possible to combine several ILBM images as well as other formats (such as a 8SVX file) into a single IFF file. By passing the combined file to the viewer, you can switch between images using the 'Page Up' and 'Page Down' keys. For example:

$ iffjoin Champagne Venus.lores Sample.8SVX > join.IFF 
$ ilbmviewer join.IFF

Below I have included some screenshots of the SDL ILBM viewer. The picture on the top left is an image included in Graphicraft, which defines a color range cycle to animate the bird and the bunny. By pressing the 'TAB' key, the viewer cycles the color range to show you the animation. The other screenshots are images included with Deluxe Paint V. As you can see, the viewer also knows how to view HAM images (the Aquarium) and AGA images (the desk).



Implementing a SDL 8SVX player


To see how well my IFF library implementation is designed, I have decided to implement a second IFF application format, namely the 8SVX format used to store 8-bit audio samples. The architecture of the SDL 8SVX player is quite similar to the SDL ILBM viewer, with the following differences:

  • lib8svx. This library implements the application chunks as well as the fibonacci-delta compression method defined in the 8SVX specification. As with libilbm, it also defines a table with function pointers handling application specific chunks to the IFF parser.
  • libresample. This library is used to convert sample rates. 8SVX samples have variable sample rates, while on the PC hardware samples are typically passed to audio buffers with a fixed sample rate. Therefore, we have to convert them.
  • SDL_8SVX. This package contains a library as well as the 8svxplayer command-line tool. Sample rate conversion is automatically done by the SDL library.

As with the SDL ILBM viewer, the SDL 8SVX player can also play samples from scrap IFF files:
$ iffjoin Picture.ILBM Sample.8SVX > join.IFF
$ 8svxplayer join.IFF 

Backporting the ILBM viewer to AmigaOS


The third experiment I did was a really crazy one. I have backported the libraries and tools to the AmigaOS. People probably wonder why I want to do something like this, but hey: it is fun, so why shouldn't I do it? The reasons are the same why people want to backport WINE to Windows or AROS back to the Motorola 68000 platform.

Another reason is that I wanted to know how well these libraries perform on the original platform were these file formats were designed for. The Nix AmigaOS build function I have developed previously, helped me a lot in achieving this goal. Apart from a few small fixes, mainly because getopt_long() is not supported, I could easily port the codebase in a straight forward manner without implementing any workarounds.



The architecture shown above is nearly identical to the SDL ILBM viewer. The only difference is the role of the libamivideo library. In the AmigaOS viewer application, it serves the opposite goal compared to the SDL version; it converts images saved by the PC version of Deluxe Paint in chunky graphics format to bitplanes.

It was also nice to write a Intuition GUI application for AmigaOS. Back in the old days, I have never programmed in C and I never wrote a GUI application (apart from a few small experiments in Amiga BASIC), simply because I did not have the knowledge and tools available back then. The AmigaOS libraries were not very difficult to understand and to use.

Below I have included some screenshots of the UAE emulator running the viewer using my own libraries. As you can see, the GUI application has implemented Intuition menus allowing you to open other IFF files using a file picker and to navigate through IFF scrap files:



Conclusion


In this blog post I have described several software packages that resulted from my IFF file format experiments, because I could not find any IFF libraries that have all the features that I want. The purpose of these packages is to provide a set of high quality, complete, portable libraries to display, parse, write and check several IFF application formats.

All the software packages can be obtained from the IFF file format experiments subpage of my homepage and used under various free and non-copylefted software licenses, such as the MIT license and the zlib license.

I haven't made any official releases yet, nor I have defined a roadmap, so don't consider these libraries production ready. Also, the API may still evolve. Probably, at some time in the future I will make it more stable.

I have also found two other projects implementing the IFF standard:

  • The IFF project on Sourceforge, is a C++ library using uSTL, which deviates on some aspects. For example, it stores integers in little-endian format. Furthermore, I haven't seen any application file formats using this library.
  • I also found a project named libiff on Google Code. It seems to have no releases and very little documentation. I have no clue about its capabilities and features.

It is also interesting to point out that I have more stuff on my hard drive, such as libraries supporting several other file formats, which utilise several packages described here. When I can find the time, I'll make these available as well.

Thursday, April 26, 2012

Dynamic analysis of build processes to discover license constraints

As I have explained earlier in the blog post about software deployment complexity, software is rarely self-contained nowadays, but typically use many off the shelf components. Reuse has advantages, such as the fact that productivity increases and products can be finished more quickly. One of the disadvantages is an increasingly more complicated deployment process.

Apart from productivity and deployment aspects, the usage of components under Free and Open Source licenses is very popular. This is probably due to the fact that the source code is available, can be adapted and most software packages are available for free through the internet (free in price, a.k.a. gratis).

What a lot of vendors don't realize, is that most Free and Open-Source components are not in the public domain. They are in fact copyrighted and distributed under licenses ranging from simple permissive ones (allowing you to do almost everything you want including keeping modifications secret) to more complicated copyleft licenses imposing requirements on derived works. The GNU General Public License (GPL) is the most famous copyleft license around.

Because of the obligations that these licenses impose on users, licenses have become a very important non-functional requirement of software systems. Not obeying these licenses could result in costly lawsuits by copyright holders. Busybox is a well-know example of a software package which has been defended successfully in court several times.

Some clarification


Before you read on, I first want to give some clarification to readers unfamiliar with Free and Open Source software. I have written an earlier blog post about Free and Open-Source software explaining what it is and what it is not. In this blog post I also try to clarify a number of common misconceptions.

Most outsiders think that these lawsuits are about money, which is not true. These lawsuits are not held because people include FOSS components in their commercial products and ask money for these products. As I have explained earlier, selling free and open-source software is fine.

These lawsuits are held because some copyleft licenses require that the source code of the derived products or parts thereof are available under the conditions of the same license, which includes access to the source code. Typically, many vendors refrain from publishing source code and do not obey the obligations that these licenses specify. In many cases, vendors are unaware of this.

Background


The research I have done about this subject has a bit of history, which I'd like to explain here :-)


A couple of years ago, when I was in my first year as a PhD student, I've attended ICSE 2009 and there was one talk that I found very inspiring and gave me a lot of ideas. The paper was titled: 'License Integration Patterns: Addressing License Mismatches in Component-Based Development' and presented by Ahmed E. Hassan, which basically covered a large number of FOSS licenses and described patterns how to combine components governed under various licenses in a proper way.

Although their paper covers a great amount of license issues, they were still looking into automation of their patterns, for example to automatically verify derived works. This process turns out te be quite challenging because automating such processes require you to have powerful deployment tools and a complete notion of all dependencies involved in producing an artifact, such as a binary. Fortunately, deployment research is our expertise and our tools are designed for such purposes. For a while, I had several ideas about a possible solution in mind, but I never implemented anything.


Some time later at ASE 2010, which I also attended, there was another talk related to this subject titled: 'A sentence-matching method for automatic license identification of source code files' and given by Yuki Manabe. In this paper a tool was developed, called Ninka, which can be used to analyse sentences in comments of the source code, to determine under which license a source file is governed. I asked Yuki whether the tool was available somewhere, but unfortunately the idea quickly appeared on the bottom of my todo list and I forgot about this (which is a shame).

A while later Daniel German, who is involved in all the publications I have mentioned, was invited by Eelco Dolstra to visit our group in Delft. That visit resulted in an eventual collaboration between me, Eelco Dolstra (from our group), Daniel German, Julius Davies (from University of Victoria) and Armijn Hemel (who is from the gpl-violations.org project as well as owner of Tjaldur, a company specialised in software governance and license compliance engineering).

Motivation


In order to say something about the rights and obligations of software systems, you must know the following things:

  • How are source files combined into a final executable (e.g. static linking, dynamic linking)?
  • What licenses govern the (re)use of source files that were used to build the binary?
  • How can we derive the license of the resulting binary from the build graph containing the source files?

We together wrote a paper to provide an answer for the first question.

Approach


To provide a good answer for that question, we have crafted a method which traces system calls of build processes (essentially the involved processes and what files go in and out) and we produce build graphs out of these traces. Furthermore, the traces of each package are stored in a central database, so that inter package-dependencies can be determined.

We have used the Nix package manager to manage all the build processes. Nix is a very convenient instrument, as it has a number of good features, such as the fact that builds are pure (so no undeclared dependencies can affect the reliability of our results), that it guarantees dependency completeness (so that we are certain that no crucial dependencies affecting the license of the result binary are missed) and because Nix stores all packages in isolation in separate directories, we can easily identify inter-package relationships by looking at absolute file names. Furthermore, the Nix expression language allows us to modify the standard builder environment, without changing any package build specifications.

Tracing system calls


We trace the following system calls:
  • File related system calls, e.g. open(), execve()
  • Process related system calls, e.g. fork(), clone(), vfork()

Apart from capturing traces, there were a number issues we had to deal with:

  • We have to translate all relative paths to absolute paths
  • In Linux, pids wrap around if they exceed 32767, so we have to use a different attribute to distinguish between processes
  • Cycles appear in the graph, if files are read/written multiple times, which we have to remove
  • The are coarse grained processes, such as the install process, which install multiple files in one run. In the resulting graph it looks like the resulting artifact is dependent on all other artifiacts installed by the same process, which is not true. We have to identify these processes and rewrite them.
  • We don't want to know anything about the dependencies of the build tools themselves, because these are not considered derived work.

I have kept the mechanics intentionally brief here, because I don't want to explain them again here. The exact details can be found in the paper.

Build trace graphs


Below I have included a graph of cupsd, an executable belonging to the CUPS package, which we have derived with our tool:



The SVG pictures of this graph as well as several other graphs, can be found here.

By using a graph such as the one of cupsd and by using Ninka to analyse the source files in this graph, one can say something about the license under which the resulting binary is covered. In the paper, we have found an interesting problem with a well known free/open-source package, which I'm not going to reveal in this blog post :-)

Discussion


A reader with Nix experience may probably wonder why we have implemented an additional tracing approach, next to the Nix package manager. The answer is that Nix works on package level, but licenses do not always cover complete packages. There are packages in which individual files are covered under several licenses. Therefore, a more fine-grained tracing approach is required.

Unfortunately, the paper was rejected from ICSE 2012, which I was a bit disappointed about a while ago (although I'm still there anyway because I have to present another paper at HotSWUp). The fact that a paper is not "good enough" is not really what bothers me, but what bothers me is that it is a bit unclear whether this contribution is useful or useless and the fact that the solution is seen as 'too simple' (which is NOT a bad thing IMHO).

Perhaps it may indeed be too simple for a top general conference, but I also have no idea to what other type of conference or journal I could send this. And if this solution is too "practical", would it then perhaps be useful for a 'Software Engineering in Practice Track / Experience Track' at some conference? Although I have heard somebody talking about "engineering perspective", I haven't heard any reviewers suggesting about submitting to another track type.

The only thing that becomes clear to me from the reviews I have received, is that they are not really critical about the contents (although certain details can be strengthened of course), but rather about the significance of the contribution.

I have also noticed that the goal of this paper is generally misunderstood. People think that we are actually solving the complete licensing problem, but instead we provide an important ingredient, which is not there yet. Realising these build graphs, which cover complete build processes are already complicated enough, although the idea of using system call tracing for various purposes is not new. Nobody, however, has used system call tracing for this purpose yet (and therefore had to solve several problems as well). And furthermore, because we're using Nix, the process of experimenting with builds, suddenly becomes much simpler, which with conventional solutions will take significantly more effort.

If you look to the three questions I have given earlier, the paper is about the first question. The ASE 2010 paper provides a solution for the second. The third question is still future work, for an eventual license calculus. But in order to develop such a license calculus the ingredient of complete build trace graphs is required. I'm pretty sure that if I would talk to software deployment people about this, that this story will be appreciated. Unfortunately, as I have explained before, software deployment is a very cold research subject, without a real community.

I'm still thinking what to do with this paper, but I have no idea yet. Furthermore, the amount of time that I have left, is pretty limited. I have decided to put it online and announce it through this blog anyway. Normally, I always report about papers after they have been accepted, but not everything in research can be a 'success story'. Of course, I'm always open for all suggestions.

References


The paper is titled: 'Discovering Software License Constraints: Identifying a Binary's Sources by Tracing Build Processes'. As always, papers can be obtained from my publications page.

The techniques described in this blog post are becoming part of the service portfolio of Tjaldur, a company specialised in software governance. Furthermore, one day I expect that this tool is also going to be integrated in the Nix project.

UPDATE: Never give up! In the meantime, an updated version of this paper titled: "Tracing software build processes to uncover license compliance inconsistencies" has been accepted for ASE 2014! I owe a big thanks to Shane McIntosh, who did some major efforts in improving the paper, and he showed me that there are always new possibilities. Sometimes, it's good to be wrong about something! :-)

Tuesday, April 17, 2012

Software engineering fractions, collaborations and "The System"

As I have explained many times on this blog, software engineering is complicated. It happens quite often, that in order to solve problems or to gain knowledge, people collaborate with each other by various means. Nowadays, we have a wide range of means to share information and to collaborate. A few examples are:



As you may notice, some of these means are quite common in certain fractions of the software engineering community and very uncommon in others. I can roughly divide the software engineering community into three fractions, having a number of distinct characteristics, interests and peculiarities (beware that I'm using stereotypes here and these fractions are not necessarily mutually exclusive):

Academic fraction


This is the group where I currently belong to. Academics are people working for a university and their main goal is doing scientific research. As I have explained earlier, scientific research within the software engineering domain is a bit strange, because there is no clear consensus what this actually means. Earlier, I have dived into literature and I have found a definition, which I will rephrase here once more:
Research in engineering is directed towards the efficient accomplishment of specific tasks and towards the development of tools that will enable classes of tasks to be accomplished more efficiently.

The "deliverables" that academic people produce are, in principle, papers published in peer-reviewed academic conference proceedings and scientific journals. Paper submissions are typically in competition with each other and only papers that are good enough are eligible for publication. The software engineering domain is a bit exceptional compared to many other scientific disciplines, because conference papers are more popular than journal papers.

Academic conferences are primarily about presenting research papers. Most of the attendees of these conferences are other academic people. According to some sources, there used to be a high participation degree of industry people in the past, but this is no longer true, unfortunately.

Industry fraction


The primary goal of industry is (not surprisingly) to develop and sell software (as a product or as a service) or to provide IT services and make profit, which is (preferably) as high as possible.

In order to achieve that goal, industry typically want to be as cost efficient as possible. Therefore, they don't want to invest too much time and effort in secondary goals, such as developing software tools, as these tools cost money and do not immediately give them any profits. Rather, they want to focus themselves as much on their primary goals as possible.

The conferences that industry people attend, are often related to the technology they are using. For example, companies using Java typically attend JavaOne, or companies using Microsoft technology may attend Microsoft TechEd. Companies may also participate in "trade show" conferences, such as CeBIT, to advertise their products and to attract potential new employees.

Community project fraction


Another fraction is what I call the "community project" fraction. Typically, a lot of people refer to this fraction as "Open-Source projects", but I don't want to refer to them like that. While most community projects are distributing their software under free and open-source licenses, there are also commercial parties doing this, without outside involvement. I have written an earlier blog post about Free and Open Source software explaining what this is all about.

Community projects are usually formed by various individuals having various affiliations, share a common interest and work together on common goals. There are many prominent examples of community projects around, such as The Linux Kernel and KDE.

Another notable trait of community projects is that nearly all contributors are also users of the software. Many community projects have an informal organization structure and copyright is owned by each individual contributor. Some community projects are also governed by a legal entity owning the entire codebase, such as the Free Software Foundation, Apache Software Foundation and the Eclipse Foundation.

There are also a number of free and open source conferences around, such as FOSDEM and LinuxCon. Most attendees of these conferences are either users or contributors to community projects and very much interested in new capabilities of software.

"The System"


The industrial and academic fractions have "targets" which they have to reach within a certain period of time. Usually this period is short term. These fractions also want to grow, improve and perform better than the competition.

People in both fractions are periodically assessed by some kind of measurement standard, indicating whether the results (according to this standard) are satisfiable and have increased enough. As a consequence, people in these fractions have a tendency to do as much as possible to improve these numbers according to this standard, rather than doing what have to be done. I call this phenomenon "The System".

Implications of "The System" on the academic fraction


In the academic world, publication records are used as the main productivity / efficiency measurement unit. This often means that the more papers you produce the better you are as a researcher. Furthermore, various other publication quality attributes are typically taken into account, such as the ranking of the conference or journal and the amount of citations that you have. Some metrics that measure a researcher's productivity are the G-index and the H-index.

Because publications are the primary (or sometimes the only) measurement unit of research, many researchers primarily work to improve these numbers. In my opinion, this is a bad thing.

As a consequence, many researchers spent most of their time aiming at a collection of academic conferences and journals. Each conference and journal have particular requirements, boundaries, traits and peculiarities, such as the allowed subjects, page limits, evaluation methods, stuff of which you know that is going to be well-received by the Program Committee members and stuff which don't. Some of the "tactics" researchers use to get their paper accepted is 'identify the champion', which means that you have a look at the list of Program Committee members and write your paper to please at least one of them, so that he will probably vote in favor of your acceptance of your paper.

Sometimes, I get the impression that doing research this way, looks like a darts game, in which you keep aiming at a fixed number of sections, and keep modifying your arrows until you hit the right score.

If you look at the definition of 'research in engineering' I have given earlier, writing publications and improving publication records is not the only thing that needs to be done. For example, sometimes also "uncommon" aspects have to be investigated, which do not necessarily produce great results, but are nonetheless worth knowing.

Furthermore, new software engineering concepts typically result in tools which have to be developed. Eventually, ideas developed in the academic world have to reach a broader audience and I think tools are the primary way to achieve that goal.

A lot of researchers refrain from doing these steps, because "The System" enforces them to do so. I have heard some people saying: "You shouldn't spent so much time on development. Just develop a prototype and then move on to your next goal!". As a consequence, lots of papers tend to become "forgotten knowledge" and the rest of the software engineering community doesn't care and perhaps reinvent the wheel some day, but implement it in a much crappier way.

Implications of "The System" on the industry fraction


In industry, quite often developers are seen as "code production units" and they have to be used as efficiently as possible. In order to be as efficient as possible, it is desired to reduce as many costs as possible and employees should focus themselves on the primary goals of the company as much as possible. One of the 'solutions' companies implement is to outsource labour to countries in which salaries are lower, to delegate certain tasks to other companies, or completely relying on an external vendor to provide a solution.

In my opinion this is a bad thing for the following reasons (I have taken these arguments from my colleague Rini van Solingen's video log: vlog episode 2 (in Dutch) ):

  • Developing software is not necessarily a process that merely costs some amount of money. Software development is an investment and also gives benefits. The benefits are often overlooked by a lot of managers. Reducing costs, e.g. by hiring cheaper employees with less knowledge, may typically result in fewer benefits.
  • In order to reach your primary goals, you also have to reach secondary goals. For example, banks are not really software companies, but they have to become software companies because their organization depend completely on software.

    I think the same is true for software companies doing software engineering. People developing financial software, may not want to think about build management complexity issues, but they have to, because otherwise it is impossible to properly engineer systems for end-users.

Many companies have the tendency to delegate secondary problems as much as possible, with the assumption that others can do it better, cheaper and more efficiently. In my opinion this is not always necessarily true. Sometimes secondary problems are so specific to a particular industry that there is no general solution provided by an external vendor or by somebody else willing to provide a solution for this.

In such cases, you have to solve these secondary yourself, but many organizations refrain from doing so. They decide to keep living with the burden and prefer to be inefficient. I have worked for several companies (which I'm not going to mention here) and I can speak from experience. I have had several unconventional ideas in mind back then, and the only thing I was doing was fighting resistance, while I could have already solved many secondary problems already.

Another trait I have frequently encountered is that some companies are afraid to participate with other communities, because of the potential advantages the competitors could get. I think for most secondary goals this is not really an issue.

Implications of "The System" on community projects


As far as I can tell, there is no "System" for community projects, as these projects are typically not bound to deadlines or formal assessment procedures. Community projects basically have to keep themselves and the community as a whole happy. Furthermore, these projects are not composed by members of a single organization, but from various individuals all over the world. However, community projects also have a few peculiarities that I'd like to mention.

Quite often, because the developers are also the users of a particular application, it is difficult for non-technical users to get involved and have their problems solved. Sometimes applications delivered by community projects are seen as very unfriendly by non-techinal users.

Also, there are "social-issues" in community projects, such as developers who receive criticism (either about them in personally or about the project in general) immediately feel themselves offended and developers pissing of non-technical users claiming that they are stupid and they don't need a particular feature. The Linux Haters Blog often elaborates on this issue. Another famous phenomenon is "bikeshedding", in which big discussions are held over relatively minor problems, while important bigger problems are overlooked.

Discussion


"The System" of each of these fractions have arised with the intention of improving themselves, but personally I think that these "Systems" actually conflict with other and make things worse, not better.

In principle, the academic fraction (which investigates software engineering techniques) solve secondary problems for the industry fraction. But in order to completely "solve" a secondary problem, usable tools have to be developed, which academic people refrain to do so because it is a waste of time, according to their "System".

Second, industry has to focus themselves on their primary goal (because their "System" requires that) and they don't want to spent too much effort in secondary goals, such as working together with academics to successfully apply research or get involved with community projects sharing knowledge.

It almost looks a bit like a prisoners dilemma to me. Collaboration between these fractions obviously requires several small sacrifices, but it also offers all parties benefits. I also see community projects as a good means to collaborate between academia and industry (and anyone else who is interested). Although this is obvious, all fractions stick to their "System" and, as a consequence, they diverge from each other and don't benefit from each other at all.

I have a few concrete examples of this:

  • The Dutch government as well as the industry in the Netherlands, spent a relatively small amount of money in research, while they want our country to be a 'knowledge economy'. It's actually the least of all countries in the European Union. The government is planning to reduce these investments even more. They expect that companies invest more in research, but in my experience, apart from a few notable exceptions, most of companies are reluctant or have no clue what is going on in the research world. We have very good researchers in the Netherlands, but their work isn't applied that well in industry. In my opinion, that's a shame and a waste. (I have used this blog post as a reference (In Dutch) )
  • As mentioned in an earlier blog post, industry participation at academic conferences is low. Industry people often have different interests than academic people. Nowadays, they rather attend technology related conferences. For industry people, application and benefits of tools and technology is important. More important than a mathematical proof or evaluation showing numbers, which they don't understand.

    I have encountered several boring presentations at academic conferences, showing lots of "greek symbols" and all kinds of complicated things I didn't understand. I'm pretty sure that if the presenter would have attendees from industry, they have no clue what they are talking about and they quickly lose interest.
  • Sometimes people reinvent the wheel, but in a crappier way, which they have to maintain themselves. In my research, for example, I have seen many custom build systems that have a significant maintenance burden. People usually stick to these suboptimal solutions for a long time, while there are many solutions available that are more convenient, more powerful and easier to maintain.

Recommendations



In order to improve the struggle of diverging fractions, I think all fractions have to cooperate better with each other (which is obvious of course) and let themselves go of their "System" in some degree (or better: make sure that the "System" changes). I have a few recommendations:
  • I think "The System" of academic researchers shouldn't be merely about publications. Furthermore, they shouldn't be merely about these fixed collection of conferences/journals each having their own "borders". Publishing stuff is not a bad thing in my opinion, as concepts must be properly explained, evaluated and peer-reviewed. But concepts are useless without any deliverables that can be applied.

    "Playing darts" by only aiming at "sections on a dart board" is bad for research. For example, Edsger Dijkstra, one of the most famous computer scientists published a lot, but most of his publications were his "EWDs"; manuscripts, which he wrote about any subject he want, whenever he wanted, without aiming at anything or keeping some kind of "System" satisfied.
  • As an academic software engineering researcher, you are typically investigating stuff for some kind of "audience" (e.g. developers or testers) and it may possibly be related to certain kind of technology (e.g. Java, Eclipse, Linux etc.). Therefore, I think it's also important to directly work with people from these fractions, address them regularly (e.g. visit them and participate in their technology-related conferences) and see whether you can make your work interesting for them.
  • It's also a good thing to make your tools available by some means. Perhaps joining a community project or start your own community project can be good thing.
  • Companies must know that developing software is an investment and that secondary goals have to be reached in order to achieve primary goals. Some secondary goals cannot be solved efficiently by third parties, as it's too specific for the domain of the company.
  • Companies should not be afraid in participating in other fractions' community means. In fact, they should be more eager in finding out what other fractions have to offer them.

These recommendations may look challenging, but I think the barrier to build better "bridges" between software engineering fractions isn't that high. I have outlined a list various means in the beginning of this blog post, that may help you and I think they are relatively easy to apply without any additional costs.

For example, besides publishing academic papers, I also have this blog and I use Twitter to regularly report about results, findings and other stuff. Furthermore, the tools we are developing are made available to everyone through a community project, called the Nix project.

Apart from academic conferences, I have also presented at an industrial conference as well as FOSDEM, the biggest Free and Open-Source conference in Europe. Actually, our work has been very well received there. Far better than any academic conference I have attended so far.

Why am I writing this?


With the work that I'm doing as a PhD student, I'm trying to address the complete software engineering community, not just a small subset. It's also a shame if 4 years of hard work becomes forgotten knowledge that nobody cares about.

Furthermore, I think that for many reasons, building "bridges" between these software engineering fractions is essential and gives all fractions benefits. But the "Systems" of all these fractions (which are basically there to improve them) drive these fractions apart.

Besides publishing papers, I have spent I considerable amount of time in development of tools, case studies and examples. For example, according to the COCOMO estimation method of ohloh.net, I have spent 2 man-years of effort in Disnix (and I'm the sole author). Furthermore, I have also developed several extensions to Disnix and I also did many contributions to other Nix projects. Apart from development, I'm also maintaining this blog in which, apart from my research, I report about several other technical issues and fun projects.

While all this work is very much appreciated by the people I work with and talk with, it's actually is a waste of time according to "The System" of my fraction. If I would have sticked to "The System", then this blog would never exist. Moreover, I would have never produced the following blog posts, because I don't know how to submit them to any academic conference or journal:

These blog posts are very useful and appreciated. Perhaps not for academic people, but certainly for the other fractions! Finally, these blog posts have attracted many more readers than all my academic papers combined.

P.S. If anyone knows how to "sell" this stuff "scientifically" and knows to make a "dart" out of this which I can throw in a good "section", please let me know! I'd like to integrate this stuff in my PhD thesis, which is primarily about research! ;)

Conclusion


In this blog post, I have identified three fractions within the software engineering community. Each fraction have their distinct characteristics. The academic and industrial fractions have a "System", which have arised to improve the individual fractions, but drive the fractions apart from each other. I have proposed a few recommendations to "bridge" these fractions, but in order to achieve that goal, they have to let themselves go of their "System", which is not easy.

As a final point, I'd like to point out that I have used stereotypes to describe these fractions. These descriptions do not always accurately reflect what happens the real world. In practice, not every academic researcher is completely focused on writing papers. I know many researchers besides me, who write tools and make them publicly available. Actually, my supervisors encourage me to work on tooling and they appreciate the work I'm doing. Furthermore, many of my colleagues also have very good collaborations with other fractions, and often present at non-academic events, which I'm very happy about.

I also said that academic presentations are boring. While I have attended quite a number of boring presentations, I have also seen many good ones, which I liked very much. It's a shame that the rest of the software community does not know about these.

Furthermore, companies aren't necessarily completely driven by making profits. They also care about their customers in some degree and thinking about improvements in technology. And yes: there are collaborations between these fractions which sometimes produce good results.

But nonetheless, although the real word is a bit better than the "stereotype" world I have described, I still see a lot of room for improvement in "bridging" these fractions.

Wednesday, March 21, 2012

Deployment of mutable components

It's been quiet here for a while. Lately, I'm very busy with all kinds of research related things (including working on my thesis). So I thought it's time for a new research related episode here on this blog... Yesterday, a paper of mine got accepted.

Purely functional deployment


As you may know, the Nix package manager is also known as the purely functional deployment model (which happens to be the title of Eelco Dolstra's PhD thesis), because it borrows concepts from purely functional programming languages, such as Haskell. In Nix, packages are derived from purely functional build functions, also called derivations:

  • The result of a derivation exclusively depends on the specified inputs of a derivation. If a dependency is not specified, it cannot be found and thus a build fails.
  • In a Nix build environment, many side effects are removed. For example, most environment variables are cleared or set to non-existing values, such as HOME=/homeless-shelter. Also certain commands cannot be used because they have side effects, such as date or hostname.
  • Various common build tools (e.g. GCC and binutils) have been patched to ignore impure directories, such as /usr/lib and /usr/include which may accidentally allow a build to succeed, with improper dependency specifications.
  • Nix stores every component in a so-called Nix store using a special filename, such as: /nix/store/a92pq...-firefox-9.0.1 in which the former part: a92pq... is a hash code derived from all build-time dependencies. This filesystem organisation prevents that dependencies can be implicitly found unless they are explicitly specified in a build process, e.g. through configuration flags or environment variables.
  • To improve purity, builds can be optionally performed in a chroot() environment.
  • Nix components which have been successfully built, are made immutable by removing the write permission bits, analogous to purely functional language objects which are immutable.

As explained in many earlier blog posts, although this approach is unconventional, it has a number of distinct advantages, such as the fact that multiple versions or variants of components can be safely stored next to each other, upgrades are always safe, dependencies are always complete and so on.

Mutable components


Although Nix has several powerful features, one aspect that it is lacking is supporting the deployment of (which I call) mutable components, such as databases. Apart from the fact that the state of mutable components changes continuously (which cannot be supported in Nix, because components are made immutable), they also have several other important properties:

  • Mutable components are typically hosted inside a container, such as a DBMS or application server. In order deploy a mutable component filesystem operations are not always enough; also the container must be notified.
  • The state of mutable components is very often stored in an unisolated directory and in a format close to the underlying implementation of the container. In order to transfer a mutable component from one machine to another, copying state files may (in many cases) not work, because they are incompatible with the target architecture or temporarily inconsistent, because of locks and/or unfinished write operations.

Because of these issues, many containers include tools which allow you to capture state in a portable format and in a consistent manner, e.g. mysqladmin. In the paper, I call the actual files representing state, physical state whereas the dump captured by a tool, such as mysqladmin the logical representation of state.

Dysnomia


To support automated deployment of mutable components using concepts similar to Nix, I have developed a tool called Dysnomia, which originates from the Disnix activation scripts package.

Dysnomia provides a generic interface for mutable components. Basically, it's an extended version of the activation scripts, implementing a snapshot and restore operation, which capture the state of a mutable component and restores them (respectively) by invoking tools that can consistently capture state in a portable manner. Also, for certain types it supports incremental snapshots.

Apart from a generic interface to support deployment of mutable components, we also have to identify the logical representation of mutable components in a unique manner, similar to hash codes for immutable Nix components. This makes it possible to perform efficient upgrades, because components with the same name do not have to be deployed again.

In Dysnomia, we address components using the following naming convention:

/dysnomia/<type>/<container identifier>/<component name>/<generation number>

The latter component, the generation number, is derived from a container property, such as a MySQL binlog sequence number or Subversion repository number. It may also be possible to use a timestamp, although this has some practical issues.

Extending Disnix


In a few days time, I have created a very hacky version of Disnix using Dysnomia. I have adapted the deployment process of Disnix and constructed two new variants:

  • The simple version. In the transition phase, we capture all state of mutable components, transfer the state and restore the state. During this phase access may be blocked and can be very expensive for large datasets.
  • The optimised version. In this version, we transfer the state outside the transition phase so that the system is still accessible. After transferring the state, we enter the transition phase and we transfer an incremental dump of the mutable state components to take changes into account which have been made meanwhile. In this version, the blocking time window is reduced to a minimum reducing the costs of blocking as much as possible.

Discussion


After reading all this, you may think that the problem of dealing with state in conjunction with Nix has been solved and the world has become a much better place. Unfortunately, there are a number of drawbacks, which may be significant:

  • The filesystem is used as an intermediate layer, which may be very expensive for very large databases. Database replication tools can solve these problems much more efficiently.
  • It is not part of Disnix yet. I'm still thinking of integrating it, but it radically changes the behaviour of Disnix. Perhaps it may become an optional feature in the future.
  • Also, the costs of managing state may be higher than the benefits that it gives.
  • The number of supported mutable components is still limited, and not all types of mutable components can be efficiently snapshotted.

Where comes the name from?


One of the aspects not covered in the paper is where the name Dysnomia comes from and it probably sounds very strange to you. Unfortunately, the space in the paper is very limited to explain it :-)

This is how I came to it:

  • Nix is derived from the Dutch word for nothing: 'Niks' (because if you don't specify dependencies, Nix cannot find anything) combined with UNIX, hence: Nix. Coincidentally, Nix is also a mythological figure as well as a name for a moon of the dwarf planet Pluto which is part of the Kuiper belt.
  • Disnix and NixOS also use the original naming convention. For example, Disnix is DIStributed NIX.
  • Hydra, our continuous integration server and Charon, our cloud deployment tool are names based on the other moons of Pluto.
  • Apparently, Pluto has a fourth moon, which is recently discovered, but it does not have a name yet. Furthermore, I don't think this hacky tool is worthy to bear that new name.
  • I took another dwarf planet from the Kuiper belt: Eris, which happens to be the largest dwarf planet discovered so far. Apparently it has a moon named: Dysnomia. So that explains where the name comes from. Furthermore, I also think it's better to take a moon of a different dwarf planet, because Dysnomia could also be used without Nix (and related applications).


Conclusion


Although I have a very experimental extension supporting deployment of mutable components, it has a number of drawbacks. When I started thinking about this tool, I thought it was a great idea. However, after implementing it and doing some experiments, I think that in some cases the costs are actually higher than the benefits. But nonetheless, I still think it's important to have a solution for this problem and some knowledge, which could lead to interesting discussions.

More details can be found in the paper titled: 'A Generic Approach for Deploying and Upgrading Mutable Software Components' which can be found on my publications page. As usual, my slides can be found on my talks page, once I have prepared them.

I'm going to present this paper at the HotSWUp workshop in Zurich (which is co-located with ICSE 2012). In my last blog post, I was a bit disappointed about my ICSE rejection, but it turns out that I'm going to be there anyway, just in case you want to meet up :-)

To prove once more that I have nothing against ICSE and that I actually like it, here is some shameless promotion: