Dynamic analysis of build processes to discover license constraints

As I have explained earlier in the blog post about software deployment complexity, software is rarely self-contained nowadays, but typically use many off the shelf components. Reuse has advantages, such as the fact that productivity increases and products can be finished more quickly. One of the disadvantages is an increasingly more complicated deployment process.

Apart from productivity and deployment aspects, the usage of components under Free and Open Source licenses is very popular. This is probably due to the fact that the source code is available, can be adapted and most software packages are available for free through the internet (free in price, a.k.a. gratis).

What a lot of vendors don't realize, is that most Free and Open-Source components are not in the public domain. They are in fact copyrighted and distributed under licenses ranging from simple permissive ones (allowing you to do almost everything you want including keeping modifications secret) to more complicated copyleft licenses imposing requirements on derived works. The GNU General Public License (GPL) is the most famous copyleft license around.

Because of the obligations that these licenses impose on users, licenses have become a very important non-functional requirement of software systems. Not obeying these licenses could result in costly lawsuits by copyright holders. Busybox is a well-know example of a software package which has been defended successfully in court several times.

Some clarification

Before you read on, I first want to give some clarification to readers unfamiliar with Free and Open Source software. I have written an earlier blog post about Free and Open-Source software explaining what it is and what it is not. In this blog post I also try to clarify a number of common misconceptions.

Most outsiders think that these lawsuits are about money, which is not true. These lawsuits are not held because people include FOSS components in their commercial products and ask money for these products. As I have explained earlier, selling free and open-source software is fine.

These lawsuits are held because some copyleft licenses require that the source code of the derived products or parts thereof are available under the conditions of the same license, which includes access to the source code. Typically, many vendors refrain from publishing source code and do not obey the obligations that these licenses specify. In many cases, vendors are unaware of this.

Background

The research I have done about this subject has a bit of history, which I'd like to explain here :-)

A couple of years ago, when I was in my first year as a PhD student, I've attended ICSE 2009 and there was one talk that I found very inspiring and gave me a lot of ideas. The paper was titled: 'License Integration Patterns: Addressing License Mismatches in Component-Based Development' and presented by Ahmed E. Hassan, which basically covered a large number of FOSS licenses and described patterns how to combine components governed under various licenses in a proper way.

Although their paper covers a great amount of license issues, they were still looking into automation of their patterns, for example to automatically verify derived works. This process turns out te be quite challenging because automating such processes require you to have powerful deployment tools and a complete notion of all dependencies involved in producing an artifact, such as a binary. Fortunately, deployment research is our expertise and our tools are designed for such purposes. For a while, I had several ideas about a possible solution in mind, but I never implemented anything.

Some time later at ASE 2010, which I also attended, there was another talk related to this subject titled: 'A sentence-matching method for automatic license identification of source code files' and given by Yuki Manabe. In this paper a tool was developed, called Ninka, which can be used to analyse sentences in comments of the source code, to determine under which license a source file is governed. I asked Yuki whether the tool was available somewhere, but unfortunately the idea quickly appeared on the bottom of my todo list and I forgot about this (which is a shame).

A while later Daniel German, who is involved in all the publications I have mentioned, was invited by Eelco Dolstra to visit our group in Delft. That visit resulted in an eventual collaboration between me, Eelco Dolstra (from our group), Daniel German, Julius Davies (from University of Victoria) and Armijn Hemel (who is from the gpl-violations.org project as well as owner of Tjaldur, a company specialised in software governance and license compliance engineering).

Motivation

In order to say something about the rights and obligations of software systems, you must know the following things:

• How are source files combined into a final executable (e.g. static linking, dynamic linking)?
• What licenses govern the (re)use of source files that were used to build the binary?
• How can we derive the license of the resulting binary from the build graph containing the source files?

We together wrote a paper to provide an answer for the first question.

Approach

To provide a good answer for that question, we have crafted a method which traces system calls of build processes (essentially the involved processes and what files go in and out) and we produce build graphs out of these traces. Furthermore, the traces of each package are stored in a central database, so that inter package-dependencies can be determined.

We have used the Nix package manager to manage all the build processes. Nix is a very convenient instrument, as it has a number of good features, such as the fact that builds are pure (so no undeclared dependencies can affect the reliability of our results), that it guarantees dependency completeness (so that we are certain that no crucial dependencies affecting the license of the result binary are missed) and because Nix stores all packages in isolation in separate directories, we can easily identify inter-package relationships by looking at absolute file names. Furthermore, the Nix expression language allows us to modify the standard builder environment, without changing any package build specifications.

Tracing system calls

We trace the following system calls:

• File related system calls, e.g. open(), execve()
• Process related system calls, e.g. fork(), clone(), vfork()

Apart from capturing traces, there were a number issues we had to deal with:

• We have to translate all relative paths to absolute paths
• In Linux, pids wrap around if they exceed 32767, so we have to use a different attribute to distinguish between processes
• Cycles appear in the graph, if files are read/written multiple times, which we have to remove
• The are coarse grained processes, such as the install process, which install multiple files in one run. In the resulting graph it looks like the resulting artifact is dependent on all other artifiacts installed by the same process, which is not true. We have to identify these processes and rewrite them.
• We don't want to know anything about the dependencies of the build tools themselves, because these are not considered derived work.

I have kept the mechanics intentionally brief here, because I don't want to explain them again here. The exact details can be found in the paper.

Build trace graphs

Below I have included a graph of cupsd, an executable belonging to the CUPS package, which we have derived with our tool:

The SVG pictures of this graph as well as several other graphs, can be found here.

By using a graph such as the one of cupsd and by using Ninka to analyse the source files in this graph, one can say something about the license under which the resulting binary is covered. In the paper, we have found an interesting problem with a well known free/open-source package, which I'm not going to reveal in this blog post :-)

Discussion

A reader with Nix experience may probably wonder why we have implemented an additional tracing approach, next to the Nix package manager. The answer is that Nix works on package level, but licenses do not always cover complete packages. There are packages in which individual files are covered under several licenses. Therefore, a more fine-grained tracing approach is required.

Unfortunately, the paper was rejected from ICSE 2012, which I was a bit disappointed about a while ago (although I'm still there anyway because I have to present another paper at HotSWUp). The fact that a paper is not "good enough" is not really what bothers me, but what bothers me is that it is a bit unclear whether this contribution is useful or useless and the fact that the solution is seen as 'too simple' (which is NOT a bad thing IMHO).

Perhaps it may indeed be too simple for a top general conference, but I also have no idea to what other type of conference or journal I could send this. And if this solution is too "practical", would it then perhaps be useful for a 'Software Engineering in Practice Track / Experience Track' at some conference? Although I have heard somebody talking about "engineering perspective", I haven't heard any reviewers suggesting about submitting to another track type.

The only thing that becomes clear to me from the reviews I have received, is that they are not really critical about the contents (although certain details can be strengthened of course), but rather about the significance of the contribution.

I have also noticed that the goal of this paper is generally misunderstood. People think that we are actually solving the complete licensing problem, but instead we provide an important ingredient, which is not there yet. Realising these build graphs, which cover complete build processes are already complicated enough, although the idea of using system call tracing for various purposes is not new. Nobody, however, has used system call tracing for this purpose yet (and therefore had to solve several problems as well). And furthermore, because we're using Nix, the process of experimenting with builds, suddenly becomes much simpler, which with conventional solutions will take significantly more effort.

If you look to the three questions I have given earlier, the paper is about the first question. The ASE 2010 paper provides a solution for the second. The third question is still future work, for an eventual license calculus. But in order to develop such a license calculus the ingredient of complete build trace graphs is required. I'm pretty sure that if I would talk to software deployment people about this, that this story will be appreciated. Unfortunately, as I have explained before, software deployment is a very cold research subject, without a real community.

I'm still thinking what to do with this paper, but I have no idea yet. Furthermore, the amount of time that I have left, is pretty limited. I have decided to put it online and announce it through this blog anyway. Normally, I always report about papers after they have been accepted, but not everything in research can be a 'success story'. Of course, I'm always open for all suggestions.

References

The paper is titled: 'Discovering Software License Constraints: Identifying a Binary's Sources by Tracing Build Processes'. As always, papers can be obtained from my publications page.

The techniques described in this blog post are becoming part of the service portfolio of Tjaldur, a company specialised in software governance. Furthermore, one day I expect that this tool is also going to be integrated in the Nix project.

UPDATE: Never give up! In the meantime, an updated version of this paper titled: "Tracing software build processes to uncover license compliance inconsistencies" has been accepted for ASE 2014! I owe a big thanks to Shane McIntosh, who did some major efforts in improving the paper, and he showed me that there are always new possibilities. Sometimes, it's good to be wrong about something! :-)

Software engineering fractions, collaborations and "The System"

As I have explained many times on this blog, software engineering is complicated. It happens quite often, that in order to solve problems or to gain knowledge, people collaborate with each other by various means. Nowadays, we have a wide range of means to share information and to collaborate. A few examples are:

• Academic research publication libraries: ACM Digital Library, IEEE eXplore, SpringerLink
• Conferences:
  • Academic: ICSE, ASE, ISSRE, SPLASH
  • Industrial:
  • Technology-related: Oracle JavaOne, Microsoft TechEd
  • "Trade shows": CeBIT
  • Free and Open-Source software related: FOSDEM, EclipseCon, LinuxCon
• Internet services:
  • Question and answers websites: Stack overflow
  • Technology related websites: Slashdot, Phoronix, Ars Technica
  • Social news websites: Reddit
  • Source code sharing and collaboration: Sourceforge, Github
  • (Micro)blogs: Twitter, Blogger
  • Messaging: IRC, Mailing Lists

As you may notice, some of these means are quite common in certain fractions of the software engineering community and very uncommon in others. I can roughly divide the software engineering community into three fractions, having a number of distinct characteristics, interests and peculiarities (beware that I'm using stereotypes here and these fractions are not necessarily mutually exclusive):

Academic fraction

This is the group where I currently belong to. Academics are people working for a university and their main goal is doing scientific research. As I have explained earlier, scientific research within the software engineering domain is a bit strange, because there is no clear consensus what this actually means. Earlier, I have dived into literature and I have found a definition, which I will rephrase here once more:

Research in engineering is directed towards the efficient accomplishment of specific tasks and towards the development of tools that will enable classes of tasks to be accomplished more efficiently.

The "deliverables" that academic people produce are, in principle, papers published in peer-reviewed academic conference proceedings and scientific journals. Paper submissions are typically in competition with each other and only papers that are good enough are eligible for publication. The software engineering domain is a bit exceptional compared to many other scientific disciplines, because conference papers are more popular than journal papers.

Academic conferences are primarily about presenting research papers. Most of the attendees of these conferences are other academic people. According to some sources, there used to be a high participation degree of industry people in the past, but this is no longer true, unfortunately.

Industry fraction

The primary goal of industry is (not surprisingly) to develop and sell software (as a product or as a service) or to provide IT services and make profit, which is (preferably) as high as possible.

In order to achieve that goal, industry typically want to be as cost efficient as possible. Therefore, they don't want to invest too much time and effort in secondary goals, such as developing software tools, as these tools cost money and do not immediately give them any profits. Rather, they want to focus themselves as much on their primary goals as possible.

The conferences that industry people attend, are often related to the technology they are using. For example, companies using Java typically attend JavaOne, or companies using Microsoft technology may attend Microsoft TechEd. Companies may also participate in "trade show" conferences, such as CeBIT, to advertise their products and to attract potential new employees.

Community project fraction

Another fraction is what I call the "community project" fraction. Typically, a lot of people refer to this fraction as "Open-Source projects", but I don't want to refer to them like that. While most community projects are distributing their software under free and open-source licenses, there are also commercial parties doing this, without outside involvement. I have written an earlier blog post about Free and Open Source software explaining what this is all about.

Community projects are usually formed by various individuals having various affiliations, share a common interest and work together on common goals. There are many prominent examples of community projects around, such as The Linux Kernel and KDE.

Another notable trait of community projects is that nearly all contributors are also users of the software. Many community projects have an informal organization structure and copyright is owned by each individual contributor. Some community projects are also governed by a legal entity owning the entire codebase, such as the Free Software Foundation, Apache Software Foundation and the Eclipse Foundation.

There are also a number of free and open source conferences around, such as FOSDEM and LinuxCon. Most attendees of these conferences are either users or contributors to community projects and very much interested in new capabilities of software.

"The System"

The industrial and academic fractions have "targets" which they have to reach within a certain period of time. Usually this period is short term. These fractions also want to grow, improve and perform better than the competition.

People in both fractions are periodically assessed by some kind of measurement standard, indicating whether the results (according to this standard) are satisfiable and have increased enough. As a consequence, people in these fractions have a tendency to do as much as possible to improve these numbers according to this standard, rather than doing what have to be done. I call this phenomenon "The System".

Implications of "The System" on the academic fraction

In the academic world, publication records are used as the main productivity / efficiency measurement unit. This often means that the more papers you produce the better you are as a researcher. Furthermore, various other publication quality attributes are typically taken into account, such as the ranking of the conference or journal and the amount of citations that you have. Some metrics that measure a researcher's productivity are the G-index and the H-index.

Because publications are the primary (or sometimes the only) measurement unit of research, many researchers primarily work to improve these numbers. In my opinion, this is a bad thing.

As a consequence, many researchers spent most of their time aiming at a collection of academic conferences and journals. Each conference and journal have particular requirements, boundaries, traits and peculiarities, such as the allowed subjects, page limits, evaluation methods, stuff of which you know that is going to be well-received by the Program Committee members and stuff which don't. Some of the "tactics" researchers use to get their paper accepted is 'identify the champion', which means that you have a look at the list of Program Committee members and write your paper to please at least one of them, so that he will probably vote in favor of your acceptance of your paper.

Sometimes, I get the impression that doing research this way, looks like a darts game, in which you keep aiming at a fixed number of sections, and keep modifying your arrows until you hit the right score.

If you look at the definition of 'research in engineering' I have given earlier, writing publications and improving publication records is not the only thing that needs to be done. For example, sometimes also "uncommon" aspects have to be investigated, which do not necessarily produce great results, but are nonetheless worth knowing.

Furthermore, new software engineering concepts typically result in tools which have to be developed. Eventually, ideas developed in the academic world have to reach a broader audience and I think tools are the primary way to achieve that goal.

A lot of researchers refrain from doing these steps, because "The System" enforces them to do so. I have heard some people saying: "You shouldn't spent so much time on development. Just develop a prototype and then move on to your next goal!". As a consequence, lots of papers tend to become "forgotten knowledge" and the rest of the software engineering community doesn't care and perhaps reinvent the wheel some day, but implement it in a much crappier way.

Implications of "The System" on the industry fraction

In industry, quite often developers are seen as "code production units" and they have to be used as efficiently as possible. In order to be as efficient as possible, it is desired to reduce as many costs as possible and employees should focus themselves on the primary goals of the company as much as possible. One of the 'solutions' companies implement is to outsource labour to countries in which salaries are lower, to delegate certain tasks to other companies, or completely relying on an external vendor to provide a solution.

In my opinion this is a bad thing for the following reasons (I have taken these arguments from my colleague Rini van Solingen's video log: vlog episode 2 (in Dutch) ):

• Developing software is not necessarily a process that merely costs some amount of money. Software development is an investment and also gives benefits. The benefits are often overlooked by a lot of managers. Reducing costs, e.g. by hiring cheaper employees with less knowledge, may typically result in fewer benefits.
• In order to reach your primary goals, you also have to reach secondary goals. For example, banks are not really software companies, but they have to become software companies because their organization depend completely on software.
  
  I think the same is true for software companies doing software engineering. People developing financial software, may not want to think about build management complexity issues, but they have to, because otherwise it is impossible to properly engineer systems for end-users.

Many companies have the tendency to delegate secondary problems as much as possible, with the assumption that others can do it better, cheaper and more efficiently. In my opinion this is not always necessarily true. Sometimes secondary problems are so specific to a particular industry that there is no general solution provided by an external vendor or by somebody else willing to provide a solution for this.

In such cases, you have to solve these secondary yourself, but many organizations refrain from doing so. They decide to keep living with the burden and prefer to be inefficient. I have worked for several companies (which I'm not going to mention here) and I can speak from experience. I have had several unconventional ideas in mind back then, and the only thing I was doing was fighting resistance, while I could have already solved many secondary problems already.

Another trait I have frequently encountered is that some companies are afraid to participate with other communities, because of the potential advantages the competitors could get. I think for most secondary goals this is not really an issue.

Implications of "The System" on community projects

As far as I can tell, there is no "System" for community projects, as these projects are typically not bound to deadlines or formal assessment procedures. Community projects basically have to keep themselves and the community as a whole happy. Furthermore, these projects are not composed by members of a single organization, but from various individuals all over the world. However, community projects also have a few peculiarities that I'd like to mention.

Quite often, because the developers are also the users of a particular application, it is difficult for non-technical users to get involved and have their problems solved. Sometimes applications delivered by community projects are seen as very unfriendly by non-techinal users.

Also, there are "social-issues" in community projects, such as developers who receive criticism (either about them in personally or about the project in general) immediately feel themselves offended and developers pissing of non-technical users claiming that they are stupid and they don't need a particular feature. The Linux Haters Blog often elaborates on this issue. Another famous phenomenon is "bikeshedding", in which big discussions are held over relatively minor problems, while important bigger problems are overlooked.

Discussion

"The System" of each of these fractions have arised with the intention of improving themselves, but personally I think that these "Systems" actually conflict with other and make things worse, not better.

In principle, the academic fraction (which investigates software engineering techniques) solve secondary problems for the industry fraction. But in order to completely "solve" a secondary problem, usable tools have to be developed, which academic people refrain to do so because it is a waste of time, according to their "System".

Second, industry has to focus themselves on their primary goal (because their "System" requires that) and they don't want to spent too much effort in secondary goals, such as working together with academics to successfully apply research or get involved with community projects sharing knowledge.

It almost looks a bit like a prisoners dilemma to me. Collaboration between these fractions obviously requires several small sacrifices, but it also offers all parties benefits. I also see community projects as a good means to collaborate between academia and industry (and anyone else who is interested). Although this is obvious, all fractions stick to their "System" and, as a consequence, they diverge from each other and don't benefit from each other at all.

I have a few concrete examples of this:

• The Dutch government as well as the industry in the Netherlands, spent a relatively small amount of money in research, while they want our country to be a 'knowledge economy'. It's actually the least of all countries in the European Union. The government is planning to reduce these investments even more. They expect that companies invest more in research, but in my experience, apart from a few notable exceptions, most of companies are reluctant or have no clue what is going on in the research world. We have very good researchers in the Netherlands, but their work isn't applied that well in industry. In my opinion, that's a shame and a waste. (I have used this blog post as a reference (In Dutch) )
• As mentioned in an earlier blog post, industry participation at academic conferences is low. Industry people often have different interests than academic people. Nowadays, they rather attend technology related conferences. For industry people, application and benefits of tools and technology is important. More important than a mathematical proof or evaluation showing numbers, which they don't understand.
  
  I have encountered several boring presentations at academic conferences, showing lots of "greek symbols" and all kinds of complicated things I didn't understand. I'm pretty sure that if the presenter would have attendees from industry, they have no clue what they are talking about and they quickly lose interest.
• Sometimes people reinvent the wheel, but in a crappier way, which they have to maintain themselves. In my research, for example, I have seen many custom build systems that have a significant maintenance burden. People usually stick to these suboptimal solutions for a long time, while there are many solutions available that are more convenient, more powerful and easier to maintain.

Recommendations

In order to improve the struggle of diverging fractions, I think all fractions have to cooperate better with each other (which is obvious of course) and let themselves go of their "System" in some degree (or better: make sure that the "System" changes). I have a few recommendations:

• I think "The System" of academic researchers shouldn't be merely about publications. Furthermore, they shouldn't be merely about these fixed collection of conferences/journals each having their own "borders". Publishing stuff is not a bad thing in my opinion, as concepts must be properly explained, evaluated and peer-reviewed. But concepts are useless without any deliverables that can be applied.
  
  "Playing darts" by only aiming at "sections on a dart board" is bad for research. For example, Edsger Dijkstra, one of the most famous computer scientists published a lot, but most of his publications were his "EWDs"; manuscripts, which he wrote about any subject he want, whenever he wanted, without aiming at anything or keeping some kind of "System" satisfied.
• As an academic software engineering researcher, you are typically investigating stuff for some kind of "audience" (e.g. developers or testers) and it may possibly be related to certain kind of technology (e.g. Java, Eclipse, Linux etc.). Therefore, I think it's also important to directly work with people from these fractions, address them regularly (e.g. visit them and participate in their technology-related conferences) and see whether you can make your work interesting for them.
• It's also a good thing to make your tools available by some means. Perhaps joining a community project or start your own community project can be good thing.
• Companies must know that developing software is an investment and that secondary goals have to be reached in order to achieve primary goals. Some secondary goals cannot be solved efficiently by third parties, as it's too specific for the domain of the company.
• Companies should not be afraid in participating in other fractions' community means. In fact, they should be more eager in finding out what other fractions have to offer them.

These recommendations may look challenging, but I think the barrier to build better "bridges" between software engineering fractions isn't that high. I have outlined a list various means in the beginning of this blog post, that may help you and I think they are relatively easy to apply without any additional costs.

For example, besides publishing academic papers, I also have this blog and I use Twitter to regularly report about results, findings and other stuff. Furthermore, the tools we are developing are made available to everyone through a community project, called the Nix project.

Apart from academic conferences, I have also presented at an industrial conference as well as FOSDEM, the biggest Free and Open-Source conference in Europe. Actually, our work has been very well received there. Far better than any academic conference I have attended so far.

Why am I writing this?

With the work that I'm doing as a PhD student, I'm trying to address the complete software engineering community, not just a small subset. It's also a shame if 4 years of hard work becomes forgotten knowledge that nobody cares about.

Furthermore, I think that for many reasons, building "bridges" between these software engineering fractions is essential and gives all fractions benefits. But the "Systems" of all these fractions (which are basically there to improve them) drive these fractions apart.

Besides publishing papers, I have spent I considerable amount of time in development of tools, case studies and examples. For example, according to the COCOMO estimation method of ohloh.net, I have spent 2 man-years of effort in Disnix (and I'm the sole author). Furthermore, I have also developed several extensions to Disnix and I also did many contributions to other Nix projects. Apart from development, I'm also maintaining this blog in which, apart from my research, I report about several other technical issues and fun projects.

While all this work is very much appreciated by the people I work with and talk with, it's actually is a waste of time according to "The System" of my fraction. If I would have sticked to "The System", then this blog would never exist. Moreover, I would have never produced the following blog posts, because I don't know how to submit them to any academic conference or journal:

• Techniques and lessons for improvement of deployment processes
• An evaluation and comparison of GoboLinux
• On Nix, NixOS and the Filesystem Hierarchy Standard (FHS)
• Deploying .NET services with Disnix
• Deploying .NET applications with the Nix package manager

These blog posts are very useful and appreciated. Perhaps not for academic people, but certainly for the other fractions! Finally, these blog posts have attracted many more readers than all my academic papers combined.

P.S. If anyone knows how to "sell" this stuff "scientifically" and knows to make a "dart" out of this which I can throw in a good "section", please let me know! I'd like to integrate this stuff in my PhD thesis, which is primarily about research! ;)

Conclusion

In this blog post, I have identified three fractions within the software engineering community. Each fraction have their distinct characteristics. The academic and industrial fractions have a "System", which have arised to improve the individual fractions, but drive the fractions apart from each other. I have proposed a few recommendations to "bridge" these fractions, but in order to achieve that goal, they have to let themselves go of their "System", which is not easy.

As a final point, I'd like to point out that I have used stereotypes to describe these fractions. These descriptions do not always accurately reflect what happens the real world. In practice, not every academic researcher is completely focused on writing papers. I know many researchers besides me, who write tools and make them publicly available. Actually, my supervisors encourage me to work on tooling and they appreciate the work I'm doing. Furthermore, many of my colleagues also have very good collaborations with other fractions, and often present at non-academic events, which I'm very happy about.

I also said that academic presentations are boring. While I have attended quite a number of boring presentations, I have also seen many good ones, which I liked very much. It's a shame that the rest of the software community does not know about these.

Furthermore, companies aren't necessarily completely driven by making profits. They also care about their customers in some degree and thinking about improvements in technology. And yes: there are collaborations between these fractions which sometimes produce good results.

But nonetheless, although the real word is a bit better than the "stereotype" world I have described, I still see a lot of room for improvement in "bridging" these fractions.

Deployment of mutable components

It's been quiet here for a while. Lately, I'm very busy with all kinds of research related things (including working on my thesis). So I thought it's time for a new research related episode here on this blog... Yesterday, a paper of mine got accepted.

Purely functional deployment

As you may know, the Nix package manager is also known as the purely functional deployment model (which happens to be the title of Eelco Dolstra's PhD thesis), because it borrows concepts from purely functional programming languages, such as Haskell. In Nix, packages are derived from purely functional build functions, also called derivations:

• The result of a derivation exclusively depends on the specified inputs of a derivation. If a dependency is not specified, it cannot be found and thus a build fails.
• In a Nix build environment, many side effects are removed. For example, most environment variables are cleared or set to non-existing values, such as HOME=/homeless-shelter. Also certain commands cannot be used because they have side effects, such as date or hostname.
• Various common build tools (e.g. GCC and binutils) have been patched to ignore impure directories, such as /usr/lib and /usr/include which may accidentally allow a build to succeed, with improper dependency specifications.
• Nix stores every component in a so-called Nix store using a special filename, such as: /nix/store/a92pq...-firefox-9.0.1 in which the former part: a92pq... is a hash code derived from all build-time dependencies. This filesystem organisation prevents that dependencies can be implicitly found unless they are explicitly specified in a build process, e.g. through configuration flags or environment variables.
• To improve purity, builds can be optionally performed in a chroot() environment.
• Nix components which have been successfully built, are made immutable by removing the write permission bits, analogous to purely functional language objects which are immutable.

As explained in many earlier blog posts, although this approach is unconventional, it has a number of distinct advantages, such as the fact that multiple versions or variants of components can be safely stored next to each other, upgrades are always safe, dependencies are always complete and so on.

Mutable components

Although Nix has several powerful features, one aspect that it is lacking is supporting the deployment of (which I call) mutable components, such as databases. Apart from the fact that the state of mutable components changes continuously (which cannot be supported in Nix, because components are made immutable), they also have several other important properties:

• Mutable components are typically hosted inside a container, such as a DBMS or application server. In order deploy a mutable component filesystem operations are not always enough; also the container must be notified.
• The state of mutable components is very often stored in an unisolated directory and in a format close to the underlying implementation of the container. In order to transfer a mutable component from one machine to another, copying state files may (in many cases) not work, because they are incompatible with the target architecture or temporarily inconsistent, because of locks and/or unfinished write operations.

Because of these issues, many containers include tools which allow you to capture state in a portable format and in a consistent manner, e.g. mysqladmin. In the paper, I call the actual files representing state, physical state whereas the dump captured by a tool, such as mysqladmin the logical representation of state.

Dysnomia

To support automated deployment of mutable components using concepts similar to Nix, I have developed a tool called Dysnomia, which originates from the Disnix activation scripts package.

Dysnomia provides a generic interface for mutable components. Basically, it's an extended version of the activation scripts, implementing a snapshot and restore operation, which capture the state of a mutable component and restores them (respectively) by invoking tools that can consistently capture state in a portable manner. Also, for certain types it supports incremental snapshots.

Apart from a generic interface to support deployment of mutable components, we also have to identify the logical representation of mutable components in a unique manner, similar to hash codes for immutable Nix components. This makes it possible to perform efficient upgrades, because components with the same name do not have to be deployed again.

In Dysnomia, we address components using the following naming convention:

/dysnomia/<type>/<container identifier>/<component name>/<generation number>

The latter component, the generation number, is derived from a container property, such as a MySQL binlog sequence number or Subversion repository number. It may also be possible to use a timestamp, although this has some practical issues.

Extending Disnix

In a few days time, I have created a very hacky version of Disnix using Dysnomia. I have adapted the deployment process of Disnix and constructed two new variants:

• The simple version. In the transition phase, we capture all state of mutable components, transfer the state and restore the state. During this phase access may be blocked and can be very expensive for large datasets.
• The optimised version. In this version, we transfer the state outside the transition phase so that the system is still accessible. After transferring the state, we enter the transition phase and we transfer an incremental dump of the mutable state components to take changes into account which have been made meanwhile. In this version, the blocking time window is reduced to a minimum reducing the costs of blocking as much as possible.

Discussion

After reading all this, you may think that the problem of dealing with state in conjunction with Nix has been solved and the world has become a much better place. Unfortunately, there are a number of drawbacks, which may be significant:

• The filesystem is used as an intermediate layer, which may be very expensive for very large databases. Database replication tools can solve these problems much more efficiently.
• It is not part of Disnix yet. I'm still thinking of integrating it, but it radically changes the behaviour of Disnix. Perhaps it may become an optional feature in the future.
• Also, the costs of managing state may be higher than the benefits that it gives.
• The number of supported mutable components is still limited, and not all types of mutable components can be efficiently snapshotted.

Where comes the name from?

One of the aspects not covered in the paper is where the name Dysnomia comes from and it probably sounds very strange to you. Unfortunately, the space in the paper is very limited to explain it :-)

This is how I came to it:

• Nix is derived from the Dutch word for nothing: 'Niks' (because if you don't specify dependencies, Nix cannot find anything) combined with UNIX, hence: Nix. Coincidentally, Nix is also a mythological figure as well as a name for a moon of the dwarf planet Pluto which is part of the Kuiper belt.
• Disnix and NixOS also use the original naming convention. For example, Disnix is DIStributed NIX.
• Hydra, our continuous integration server and Charon, our cloud deployment tool are names based on the other moons of Pluto.
• Apparently, Pluto has a fourth moon, which is recently discovered, but it does not have a name yet. Furthermore, I don't think this hacky tool is worthy to bear that new name.
• I took another dwarf planet from the Kuiper belt: Eris, which happens to be the largest dwarf planet discovered so far. Apparently it has a moon named: Dysnomia. So that explains where the name comes from. Furthermore, I also think it's better to take a moon of a different dwarf planet, because Dysnomia could also be used without Nix (and related applications).

Conclusion

Although I have a very experimental extension supporting deployment of mutable components, it has a number of drawbacks. When I started thinking about this tool, I thought it was a great idea. However, after implementing it and doing some experiments, I think that in some cases the costs are actually higher than the benefits. But nonetheless, I still think it's important to have a solution for this problem and some knowledge, which could lead to interesting discussions.

More details can be found in the paper titled: 'A Generic Approach for Deploying and Upgrading Mutable Software Components' which can be found on my publications page. As usual, my slides can be found on my talks page, once I have prepared them.

I'm going to present this paper at the HotSWUp workshop in Zurich (which is co-located with ICSE 2012). In my last blog post, I was a bit disappointed about my ICSE rejection, but it turns out that I'm going to be there anyway, just in case you want to meet up :-)

To prove once more that I have nothing against ICSE and that I actually like it, here is some shameless promotion:

'Engineering' versus 'Science'

As part of my work as a PhD student, I do research in software engineering, in particular software deployment. My research consists of all kind of activities: I talk to all kinds of people (academic people, people from companies), I write and maintain various software tools, such as Disnix, I give presentations and I do various experiments. Most importantly, I write papers which report about various pieces of my research. Usually these papers are supposed to be presented at academic conferences.

Writing papers is not always an easy job. Conferences typically only accept high quality submissions for presentation. Papers are peer-reviewed by members of a program committee and submissions are in competition with each other. Usually, a small percentage of good papers are eligible for presentation. Some papers of mine got accepted immediately, others got rejected and received criticism.

In some of the reviews I have received so far, I get complimented with things like:

This is a remarkable piece of engineering work.

Although this may sound like a compliment, this typically means something bad, because conferences want valuable scientific research and not engineering work. In all these cases I've received such "compliments", the paper was rejected.

Although I understand that a paper has to conform to a certain quality level and not every story is suitable for presentation on a software engineering conference, such compliments bother me a bit, because I don't know exactly what they mean, nor I have an exact idea what the reviewers want me to do with these kind of criticisms. And another big question that I have is: When do we consider something engineering and when is it really science?

Software engineering conferences

There are many software engineering conferences around. We have sub-field conferences about specific domains, such as software testing, reliability engineering, programming languages, reverse engineering and repository mining. Furthermore, we have a number of top general conferences in which every software engineering task is potentially eligible for presentation. The International Conference on Software Engineering (ICSE) probably is the most well known top general conference in our field.

Most of the attendants of these conferences are academic people, although there is also some industry participation. For example, I have heard that the industry participation of ICSE is less than 20%, while this used to be around 50% in the past.

Difference between engineering and science

In the software engineering domain, the difference between science and engineering is not so clear to me. A common perception is that scientific research is what we should do in academia, engineering is what industry does.

Lately I have been looking for answers and (for fun) I dived into the history of the ICSE, the most popular software engineering conference in our field. I've looked into the proceedings of ICSE 1976, which was the first edition of the real ICSE (it seems that ICSE 1975 was actually NCSE: National Conference on Software Engineering). Furthermore, in this year the industry participation (if what I've heard is correct) is around 50%.

When I looked at the proceedings, the following paper caught my interest: 'Research Paradigms in Computer Science' by Peter Wegner from Brown University. In this paper, the author explores 4 influential definitions of computer science. Interestingly enough, this paper also reports about research in engineering:

Research in engineering is directed towards the efficient accomplishment of specific tasks and towards the development of tools that will enable classes of tasks to be accomplished more efficiently.

Furthermore, the paper also makes a distinction between a research engineer and a practicing engineer:

The problem-solving paradigm of the practicing engineer generally involves a sequence of systematic selection or design decisions which progressively narrow down alternative options for accomplishing the task until a unique realization of the task is determined.

The research engineer may use the paradigms of mathematics and physics in the development of tools for the practicing engineer, but is much more concerned with the practical implications of his research than the empirical scientist or mathematician.

Academic contributions

After reading this paper, assuming that this properly defines what research in engineering is about, I have the following idea about academic software engineering research contributions:

• You shouldn't merely solve a problem for which you have to engineer something. It should be related to software engineering processes.
• Your contribution should fill a gap within in a software engineering process. For example, it should offer an improvement or a new insight everybody should know about. Furthermore, this contribution must be novel.
• The contribution should be significant enough.
• You need strong motivation why this contribution is needed, which is not always obvious. You also need to look at what has been done before.
• You need to prove what you claim is valid (or at least strong enough) and you need to describe how it compares to (possibly) related work.
• And of course, you need to explain what you did

If you just solve an arbitrary problem by means of software engineering, without filling a gap, it is engineering.

Reflection

If I take what I have written in the previous section into account and look back to what I've done before, I don't think I have written a pure engineering paper so far. I think that in these cases probably something else went wrong. This is what I think that could have go wrong:

• The motivation is unclear or misunderstood.
• Perhaps the paper may contain too many implementation details.
• There is something wrong with the evaluation method or the evaluation is not strong enough.
• Reviewers have different expectations about submissions. Maybe this is also due to the fact that software deployment is a very uncommon research subject.
• A solution is/looks too simple

I find it a bit disappointing that I have to guess about these reasons myself. If I get a bad review, I would appreciate it if reviewers also make themselves clear why they think it's engineering, so that I can do something with their feedback (or preferably: use concrete criticisms as I have shown above). I think that's what peer-reviewing is about, right?

Engineering

I'd also like to elaborate a bit more on engineering in this blog post. Although academic conferences want scientific contributions, is doing engineering work a bad thing? In my opinion it's also a good thing to do engineering work (even as a researcher), for the following reasons:

• Typically, many software engineering problems arise in practice. Being in the field gives you a lot of experience and a better understanding of particular software engineering problems.
• Many software engineering contributions are implemented as tools. Tools need to be constructed from various components by using various means. So in order to fill a gap in software engineering processes, you also have to do some software engineering.
• Tools need to be applied in real world scenarios. In such cases, you may have to study existing systems, extend/integrate your tooling, or even implement work-arounds because you're tooling assumes ideal behaviour.
• There may be lessons that you have learned in your research that every software engineer should know about.

In my period as a PhD researcher, I have done many engineering contributions, next to the scientific papers I have written. Typically, I use my blog to report about these, such as applying Disnix for deploying .NET services or comparing NixOS with GoboLinux. Although these blog posts aren't really useful scientific contributions, they are if fact more appreciated by the people I work with. Furthermore, these blog posts have also attracted many more readers than all my academic papers combined.

Last year I have also presented pieces of my work at FOSDEM, the Free/Open-Source Developers' European Meeting attracting thousands of developers from all over the world. For some reason, my talk attracted quite a lot of people and I also received much more interesting questions than I have ever received on any academic software engineering conference.

So what's the big deal anyway?

Next to the discussion of the meaning of science and engineering, I'd like to point out that engineering is not something bad, although it may not always be something reviewers of academic conferences are interested in. Engineering aspects, however, are very useful for industry people. They typically care more about what benefits new tools could offer and how they can apply these techniques in their environment instead of seeing a mathematical proof.

There have been many discussions among academic conference organisers about the question why the industry participation is so low these days and how this could be improved. I think one of the big reasons is a mismatch of interests. There are of course more reasons, but this is probably not something you could blame academic people for.

Conclusion

In this blog post, I have tried to discover what scientific research in software engineering is and what ordinary engineering is. I have also expressed my opinion about engineering and concluded that it's not useless, although it may not be something that program committee members of academic conferences are interested in.

Finally, I want to make myself clear that I have nothing against ICSE or any other academic conference. This blog post is not an ICSE rant. I find the ICSE a good conference, which I have attended with pleasure in 2009 and 2011. I have attended a number of very good presentations and I have also met quite a number of great people. Also I'm currently collaborating with a few researchers that I have met there.

(Ok! Ok!... I'm not going to lie about the fact I'm a bit disappointed that our paper for this years' edition has been rejected and I agree that certain parts of the paper can be improved. Furthermore, this blog post is not elaborate about this rejection, but about a phenomenon occurring within the academic software engineering research community that bothers me a bit.)

Porting software to AmigaOS (unconventional style)

As I have mentioned in my previous blog post (in which I reflected over the last year), the Amiga is still not quite dead yet and (for some reason) my blog post about my good ol' Amiga has attracted much more visitors than I expected. In fact, for a long time this was the most interesting story I have written, while most of my other blog posts are mostly about the research that I'm currently doing.

Nowadays, there is still an active community developing Amiga software. For that reason, I have decided to pay some attention to this subject once more. This time I'm going to make it a bit more related to the research I'm currently doing. If you are curious to see how I achieve that goal, please read on...

Geek Gadgets

One of the interesting efforts done in the past is a project called Geek Gadgets, in which the GNU toolchain and many other UNIX utilities are ported to AmigaOS. You could see this effort as something similar to Cygwin, providing a UNIX environment and GNU toolchain on Microsoft Windows.

Interesting packages included in this project are:

• ixemul.library. A BSD 4.3 kernel running under AmigaOS, which internally translates a number system API calls to AmigaOS kernel calls. The API is specifically modelled after NetBSD.
• libnix. An ANSI C API library directly using the AmigaOS libraries, instead of the Unix compatibility layer. This library is better suited for developing native AmigaOS applications not requiring any UNIX functionality.
• GNU Compiler Collection. The free/open-source/ubiquitous compiler suite used on many platforms, such as Linux, Cygwin, FreeBSD, OpenBSD, NetBSD and Mac OS X.
• Korn Shell. Geek Gadgets provides the pdksh shell as a Bourne-compatible shell instead of bash, which is commonly used on other GNU systems. Probably because it's smaller and faster.

These tools make it possible to port Unix-like applications to AmigaOS as well as writing native AmigaOS applications, using development tools commonly found on free software systems, such as Linux.

Quite a number of applications are ported using this toolset such as TeX, Emacs, the X Window System and SDL, a library which is commonly used on Linux to develop games and other multimedia applications.

Using Geek Gadgets

Below, I have included a screenshot showing the Amiga Workbench and an Amiga Shell. In this Amiga shell, I have opened a Korn Shell session to show the contents of the Geek Gadgets toolset and the version of the GCC compiler.

As you may notice in the screenshot above, the /gg directory is the Geek Gadgets root directory in which a typical UNIX-like filesystem hierarchy is defined with its associated files and utilities.

The Amiga filesystem is organized quite differently as a UNIX filesystem. Whereas a UNIX filesystem is a tree with only one root directory, an Amiga filesystem has many roots. For example, on the Amiga a path can be identified by Assign:RootDirectory/Subdirectory/File, in which an assignment can represent a disk drive identifier, disk label or a random assignment which you can attach to any folder in the filesystem.

Like Cygwin, the ixemul kernel also provides a UNIX to native filesystem mapping, by defining assignments as root directories. For example the /gg directory represents the GG: assignment in which the Geek Gadgets files are stored.

Deploying Geek Gadgets

Deploying a Geek Gadgets environment was a bit trickier than I expected:

• The latest stable version is much too old, dating from 1998. The Geek Gadgets documentation is quite outdated and does not accurately describe what packages you need from the latest snapshot.
• In order to use any Geek Gadgets tools, I need to manually set up the Geek Gadgets environment each time, by executing the GG:Sys/S/GG-Startup script and by raising the stack size, because otherwise I can get mysterious lock-ups/crashes. Of course, I could also add these instructions to S:User-Startup, but I don't want the Geek Gadgets to spoil up my regular system when I use native AmigaOS applications.
• Geek Gadgets uses a traditional UNIX filesystem organisation, with all its disadvantages. For example, packages are deployed in an impure manner because their contents are scattered across the filesystem. I don't like it very much to let development packages pollute my filesystem.

Most importantly, my research as a PhD student is about software deployment in which I try to deal with such inconveniences. For Amiga readers not knowing anything about it, one of our key aspects is the Nix package manager, which we use to build software components from declarative build specifications. Furthermore, Nix has several distinct features such as the ability to store components in isolation, atomic upgrades and rollbacks and garbage collector which safely removes components no longer in use.

Nowadays, I use the Nix package manager for virtually everything that I develop and I have to compile, because it stores all builds isolated and it never pollutes my system. Furthermore, I can also easily use the Nix package manager to build several variants of the same package (e.g. for i686-linux and x86_64-linux), just by modifying several build parameters in the Nix expression for that particular package.

Building AmigaOS applications with Nix

So an interesting question that I have raised to myself is: "Can I use my research about software deployment to deal with these deployment inconveniences of Geek Gadgets on AmigaOS?".

An answer that you probably would expect me to give is that I have ported the Nix package manager to AmigaOS, but unfortunately that isn't possible because Nix requires much more memory than an Amiga has. Furthermore, the ixemul library does not completely support the POSIX API and does not support symlinks, which makes it impossible to run the Nix package manager properly.

Instead, I have developed a Nix function, which should be invoked on a Linux machine. This function launches UAE, the Ultimate Amiga Emulator, to perform a build and stores the build result in the Nix store of the host system. To develop this Nix function I had to create a basic disk image and a function that starts UAE with this disk image in which the actual build is performed.

Creating a disk image

Creating a basic disk image is a straightforward process:

• UAE must be configured to use a host filesystem directory as a hard drive partition. In the UAE GUI this can be enabled by clicking on 'New filesystem...' button in the 'Harddisks' tab.
• The Amiga Workbench 3.1 diskettes are required to perform a hard drive installation of the Amiga Workbench. The install disk contains a wizard which takes care of these steps. I performed an 'Immediate user installation' in which I only installed the English language locale, no printers and an American keymap.
• I have also installed the LhA archiver, because this is the most widely used archiver on the Amiga. I have obtained this version from Aminet: http://aminet.net/package/util/arc/lha. This file is a self extracting archive, which can be installed by running the following instructions from the command-line interface:
  

  T:
  Protect lha.run +e
  lha.run
  Copy lha_68k C:lha
  

Installing the Geek Gadgets environment is a little trickier though:

• I used this simple README file from Aminet as a reference describing the base packages I need: http://de4.aminet.net/dev/gg/0README-GG.txt
• I did not download the Geek Gadgets packages from Aminet, but instead I downloaded the latest snapshot, which I found here: ftp://ftp.back2roots.org/pub/geekgadgets/amiga/m68k/snapshots/990529/bin.
• Instead of gcc, I have downloaded egcs. Additionally, I have downloaded a2ixlibrary and gawk, which are frequently required to build packages. libg++ is not included in the latest snapshot and is apparently also not required.
• I've extracted all the tarballs on the host system into the GG subfolder of the Amiga drive, because this is much faster. I had to fix to bin/sh symlink to the ksh manually, because the symlink in the tarball is incorrect.
• A GG: assignment is required so that Geek Gadgets knows where to find its files. This assignment can be made by adding the following line to the S:User-Startup file:
  

  Assign >NIL: GG: DH0:GG
  

• If you want to create native AmigaOS applications, you need to copy the Amiga system headers into GG:os-include. I have found a collection of Geek Gadgets compatible os-include headers here: http://ftp.back2roots.org/back2roots/cds/fred_fish/geekgadgets_vol1_9610/ade-bin
• In order to automatically perform builds from Nix expressions, we need to start the build process right after the system is booted. To make this possible I commented out the last two lines from the S:Startup-Sequence, to disable the Workbench and to boot into command-line mode:
  

  ; C:LoadWB
  ; EndCLI >NIL:
  

  
  I have added the following lines to the S:User-Startup script to initialize the Geek Gadgets environment and to automatically start the build process:
  

  Execute GG:Sys/S/GG-Startup
  Stack 200000
  
  DH0:T
  sh build.sh
  

Writing a Nix function

Now that we have a basic disk image containing the Amiga Workbench and the Geek Gadgets utilities, we need to write a Nix function that automatically performs the build steps. I will list pieces of the code of this Nix function here, with some explanation.

The AmigaOS build function, takes 4 arguments where 1 argument is optional:

{name, src, buildCommand, buildInputs ? []}:

The name parameter specifies the name of the component, which also appears in the resulting Nix store path. The src parameter is used to specify the source code that must be compiled. This parameter could refer to a tarball, or a directory containing source code. You could also bind this parameter to a function, such as fetchurl to remotely obtain a source tarball. The buildCommand parameter is a string which specifies how the source code can be built. The buildInputs parameter can be used to install additional packages into the Geek Gadgets environment, such as libraries or other build tools which aren't included in the basic disk image.

The next line imports all the Nix package build functions into the scope of our AmigaOS build function, so that we can conveniently access UAE and other utilities:

with import <nixpkgs> {};

We also need to specify where we can find the disk image containing the Workbench and Geek Gadgets. The UAE emulator also needs a kickstart ROM image. The following code fragment specifies where to find them (you need to replace these paths to match your situation):

let
  amigaBase = ./amigabase;
  kickRom = ./kickrom/kick.rom;
in

The following lines invoke the stdenv.mkDerivation function, which we need to specify how we can actually perform a build. The following code fragment inherits the name from our function header. Furthermore, it adds the UAE emulator in our environment so that we can boot our disk image and perform the compilation. We need procps to kill the emulator, once the build has finished:

stdenv.mkDerivation {
  inherit name;
  
  buildInputs = [ uae procps ];
 
  buildCommand = ''

The remainder of the function specifies how the build should be performed. As we're not allowed to mess up the original base image (it's read-only anyway during a Nix build), we need to create a new one in our temp directory of our build. Fortunately, most parts of the AmigaOS filesystem can be used in read-only mode, so we only need to symlink these system directories. The following code fragment creates symlinks to all static AmigaOS directories:

    mkdir hd
    cd hd
       
    for i in ${amigaBase}/{C,Classes,Expansion,Fonts,L,Libs,Locale} \
      ${amigaBase}/{Prefs,Rexxc,S,Storage,System,Tools,Utilities}
    do
        ln -s "$i"
    
        # Symlink icons
        if [ -f "$i.info" ]
        then
            ln -sf "$i.info"
        fi
    done

The Geek Gadgets environment however, must be copied to our new temporary hard drive, because we may need to install other packages in this environment. Additionally, we need to fix the permissions of the copied GG directory, because in the Nix store it has been made read-only and we require a writeable GG directory:

    cp -av ${amigaBase}/GG .
    chmod 755 GG
    
    cd GG
    
    for i in `find . -type d` `find . -type f`
    do
        chmod 755 $i
    done

If we have any build inputs (such as library packages), we need to unpack them into the Geek Gadgets environment:

    for i in ${toString buildInputs}
    do
        cp -av $i/* .
    done

The next step is copying the source code into a location in which a build can be performed. For this purpose we create a temp T directory in our AmigaOS filesystem. If the source code is a directory, we need to strip the hash off it and we need to make it writable again:

    cd ..
    
    mkdir T
    cd T
    
    stripHash ${src}
    cp -av ${src} $strippedName
    
    if [ -d "$strippedName" ]
    then
        chmod -R 755 "$strippedName"
    fi

Then we have to execute the instructions defined in the buildCommand somehow. We generate a shell script called buildinstructions.sh containing the contents of the buildCommand string. This script is executed at boot time by the Korn shell.

    echo "src=$strippedName" > buildinstructions.sh
    
    cat >> buildinstructions.sh << "EOF"
    ${buildCommand}
    EOF

We can't directly invoke the build instructions, because we also have to determine whether a build has succeeded or failed. The following code fragments creates an additional script called: build.sh, which invokes the buildinstructions.sh script. After the build instructions script has finished, it determines whether it has succeeded or failed and writes the status into a file called done which is stored in the resulting Nix store path:

    cat > build.sh << "EOF"
    ( sh -e buildinstructions.sh
      
      if [ $? = 0 ]
      then
          echo "success" > /OUT/done
      else
          echo "failure" > /OUT/done
      fi
    ) 2>&1 | tee /OUT/log.txt
    EOF

We now configured a bootable disk image, which automatically performs a build. The next step is to provide a number of settings to UAE so that our temporary disk image can be used and that a build is performed properly:

    cd ../..
    
    # Create UAE config file
    export HOME=$(pwd)
    
    cat > .uaerc <<EOF
    config_description=UAE default configuration
    use_gui=no
    kickstart_rom_file=${kickRom}
    sound_output=none
    fastmem_size=8
    chipmem_size=4
    cpu_speed=max
    cpu_type=68ec020
    gfx_linemode_windowed=double
    filesystem2=rw,:HD0:$(pwd)/hd,1
    filesystem=rw,HD0:$(pwd)/hd
    filesystem2=rw,:OUT:$out,0
    filesystem=rw,OUT:$out
    EOF

The code fragment above defines our temporary build directory as HOME directory (In a Nix build environment this variable is set to a non-existent path: /homeless-shelter in order to remove side effects). Furthermore, we generate a UAE configuration file with the following properties:

• We disable the GUI, because this prevents us running UAE non-interactively.
• We need to specify where the Kickstart ROM image can be found. This property is defined earlier in the kickRom variable.
• We disable sound support, because we don't need it and it may cause lock ups in some cases.
• The Geek Gadgets tools require quite an amount of RAM to run properly, so we've configured the UAE instance to use 2 MiB of Chip RAM and 8 MiB of Fast RAM.
• Normally the UAE emulator tries to accurately match the speed of a 7 MHz Motorla 68000 chip, which makes building software very slow compared to nowadays' standards. Fortunately, UAE can also use the host CPU as efficiently as possible, which we do in our build function to make builds significantly faster.
• We also configured two hard drive instances. The HD0: drive corresponds to the disk image we have created earlier containing the Workbench and Geek Gadgets. The OUT: drive corresponds to the Nix store output path, in which we have to store our build result.

After configuring UAE, we have to run it. Since UAE is a GUI application requiring a X11 display server, we have to specify the DISPLAY environment variable:

    mkdir -p $out

    export DISPLAY=:0
    
    # Start UAE
    uae &

There is no way to shut UAE down from an Amiga session, so we wait until we see the done file in our output directory. Once this file exists we kill the UAE process:

    while [ ! -f $out/done ]
    do
        sleep 1
    done
    
    # Kill the emulator
    pkill uae
    sleep 1

    # Check the build status
    [ "$(cat $out/done)" = "success" ]

After executing all the previous steps, the build has either succeeded or failed. If a build has succeeded, we can remove the done file, as well as some temp files created by UAE:

    cd $out
    rm done
    
    rm -f `find . -name _UAEFSDB.___`
  '';
}

Using the Nix function

The following code fragment shows how this Nix function can be used to build GNU Hello for AmigaOS:

import ./amigaosbuild.nix {
  name = "hello";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
  };
  buildCommand = ''
    tar xfvz $src
    cd hello-2.1.1
    ./configure --prefix=/OUT
    make || true
    make install
  '';
}

The expression above looks almost identical to the regular Nix expression building GNU Hello. Important to point out is that we store the output of the GNU Hello world in the /OUT directory, which corresponds to the OUT: drive. The GNU Hello build apparently requires help2man to generate a manual page. This tool is not present in our environment, and to make the build succeed I've added || true to the make instruction, so that it always succeeds. By running the following command line instruction:

$ nix-build hello.nix

The GNU Hello component gets build for AmigaOS, which looks like this:

I have also tried compiling a native AmigaOS application, which creates a very simple Intuition GUI window. I wrote the following test program for this purpose:

#include <exec/types.h>
#include <intuition/intuition.h>
#include <intuition/intuitionbase.h>
#include <intuition/screens.h>

#include <clib/exec_protos.h>
#include <clib/dos_protos.h>
#include <clib/intuition_protos.h>

#include <stdio.h>

#define INTUITION_VERSION 37

struct Library *IntuitionBase = NULL;

struct TagItem win_tags[] = {
    {WA_Left, 20},
    {WA_Top, 20},
    {WA_Width, 300},
    {WA_Height, 120},
    {WA_CloseGadget, TRUE},
    {WA_Title, "Hello world!"},
    {WA_IDCMP, IDCMP_CLOSEWINDOW},
    {TAG_DONE, NULL}
};

void handle_window_events(struct Window *window)
{
    WaitPort(window->UserPort);
}

int main(int argc, char *argv[])
{
    IntuitionBase = OpenLibrary("intuition.library",
      INTUITION_VERSION);
    
    if(!IntuitionBase)
    {
        fprintf(stderr, "Error opening intuition library!\n");
        return 1;
    }
    else
    {
        struct Window *window = OpenWindowTagList(NULL, win_tags);
 
        if(window == NULL)
            fprintf(stderr, "Window failed to open!\n");
        else
        {
            handle_window_events(window);
            CloseWindow(window);
        }
    
        CloseLibrary(IntuitionBase);
        return 0;
    }
}

And this is what the result looks like:

So apparently it works fine!

Conclusion

This blog post covers one of the craziest experiments I have done so far. I have developed a Nix function, which builds software for AmigaOS using the Geek Gadgets toolset by creating a disk image and by using UAE.

After reading this blog post (in case you're still interested :-) ), you may wonder if I actually have a real use case for this. The answer is: YES, although I'm not going to reveal what it is yet. Probably it will take a while, because I currently don't have that much spare time for fun projects like these.

I have decided not to commit this function to Nixpkgs, because it's quite hacky and it depends on stuff (like the Amiga Workbench) which I cannot distribute freely. It's probably not so hard to recreate this function yourself. You can freely use the code I have written under the MIT license.

References

• In order to use this build function you need an Amiga Kickstart ROM image and the Amiga Workbench diskettes. If you don't have these, the legal way to obtain these is by ordering one of the Cloanto's Amiga Forever discs. Alternatively, you could try AROS, a free/open-source operating system reimplementing the AmigaOS APIs, although I have no experience with it so far.
• Performing builds in virtual machines inside a Nix expression is nothing new. We have similar approaches in our Hydra build farm, in which we generate OpenSUSE, Ubuntu and Fedora machines to build RPM and Debian packages from Nix expressions. NixOS virtualization provides additional advantages, which I have described here.

Availability

UPDATE: I didn't expect this, but it seems that there are people around who actually want to use this. I have recently put some effort in it to make it a bit usable. The source code can be found on the nix-amigaosenv GitHub page, which contains updated instructions. Furthermore, I have made some enhancements, so that builds can be performed much faster and cheaper.

One year later

Today, it's my blog's first anniversary, so I thought this is a good opportunity to do some reflection about it.

Motivation

So why have I started this blog in the first place? The main reason is that as a software engineer and researcher, I don't necessarily only write code or write academic papers:

• Sometimes I have to solve engineering problems, instead of doing science (whatever that means). In my opinion, this is also interesting and useful to report about. For example, while most of my research is about Nix, NixOS, Disnix, sometimes I have to apply my work in specific scenarios/environments which is challenging enough to get it to work. A notable example is the deployment .NET software. Without proper application of the stuff I'm investigating, the usefulness of my research isn't so great either.
• Sometimes I have to express ideas and opinions. For example, what others say about the research we're doing or about general software engineering issues.
• Sometimes I need to do knowledge transfer about issues I have to keep explaining over and over again, like the blog post about software deployment in general.
• Sometimes I have some fun projects also well.

The main purpose of my blog is to fill this gap with academic paper publishing. Apparently, it seems that some of my blog posts have raised quite some attention, which I'm very happy about. Moreover, some blog posts (like the .NET related stuff) also gave me some early feedback which helped me solving a problem which I was struggling with for a long time.

Writing for a blog

For some reason, I find writing blog posts more convenient than writing academic papers. In order to write an academic paper I have to take a lot of stuff into account next to the scientific contribution I want to make. For example:

• Because I'm doing research in software deployment and because this is a neglected research topic, I have to adapt my paper to fit in the scope of a conference I have to submit to, because they are typically about something else. This is not always easy.
• I have to identify the audience of the particular conference and learn about the subjects and concepts they talk about.
• I have to read a lot of related work papers to determine the exact scientific contribution and to describe what the differences are.
• I have to integrate my idea into the concepts of the given conference.
• I have to explain the same concepts over and over again, because they are not generally understood. What is software deployment? Why is software deployment difficult and important? What is Nix? Why is Nix conceptually different compared to conventional package managers? etc. etc.
• In each paragraph, I have to convince the reader over and over again.
• It should fit within the page limit
• I have to keep myself aware of the fact that my paper must be scientific paper and not an engineering paper.

Writing for my blog is actually much easier for me, because I don't have to spent so much time on these other issues next to the contribution I want to make. Furthermore, because I can link to earlier topics and I know my audience a bit, I don't have to explain the same things over and over again.

Although these observations are quite negative, this does not mean that I want to claim that I shouldn't write any academic papers and that academic papers are useless, but nonetheless it's still a funny observation I have.

Blog posts

Another funny observation is the top 10 of most popular blog posts. The top 10 (at this time) is as follows:

1. Software deployment complexity. I'm happy that my general blog post about software deployment complexity has raised so much attention, because within the software engineering research community it's typically an ignored subject. In just one week, it surpassed the number of hits of all my other blog posts.
2. Second computer. For some reason my blog post about my good ol' Commodore Amiga is very popular. For a long time, this was actually my most popular blog post and it's not even research related! It seems that 19 years after Commodore's demise, the Amiga is far from dead.
3. First blog post. This is the first blog post in which I briefly introduce myself. Apparently, people want to know who I am :-)
4. NixOS: A purely functional Linux distribution. This is a general blog post explaining the idea and features of NixOS, the Linux distribution built on top of the Nix package manager, which we use as a foundation for our current research project.
5. First computer. Apart from my good ol' Commodore Amiga, my first computer the: Commodore 128 also still lives on!
6. The Nix package manager. This blog post covers the Nix package manager, on which NixOS is built. Apparently people are interested in the main Nix concepts as well.
7. Concepts of programming languages. This blog post is about the 'Concepts of programming languages' course taught at the TU Delft, for which I was a teaching assistent. This course covers various programming languages, which are conceptually different from each other. I actually have no idea why this blog post is so popular.
8. Using NixOS for declarative deployment and testing. This blog post covers two very nice applications of NixOS in which you can automatically deploy a distributed network of NixOS machines and use the same specification to generate a network of virtual machines for testing. I have presented this topic at FOSDEM, Europe's biggest free and open source software event, and my presentation was well received there :-).
9. On Nix, NixOS and the Filesystem Hierarchy Standard (FHS). This blog post responds to one of the common criticisms I have received about Nix and NixOS from other distributions. In this blog post I explain why we are different and that it's for a very good reason.
10. Self-adaptive deployment with Disnix. This blog post is about one of my biggest research contributions, which I have presented at the SEAMS symposium (co-located with ICSE) in Hawaii. In this blog post I have built a framework on top of Disnix and the underlying purely functional deployment model of Nix to make systems self-adaptable by redeployment in case of an event.

My biggest surprise is the fact that the Amiga stuff is so popular. Actually, I have some fun projects that I'm working on, so if I can find any time for it, there may be more to report about.

Conclusion

It seems for me that it pays off to have blog, so hopefully next year there will be much more interesting things to report about. I have one more thing to say and that is:

HAPPY NEW YEAR!!!!!!! (I took the fireworks picture above during the ICSE in Hawaii, just in case you wanted to know ;) )