Writing papers is not always an easy job. Conferences typically only accept high quality submissions for presentation. Papers are peer-reviewed by members of a program committee and submissions are in competition with each other. Usually, a small percentage of good papers are eligible for presentation. Some papers of mine got accepted immediately, others got rejected and received criticism.
In some of the reviews I have received so far, I get complimented with things like:
This is a remarkable piece of engineering work.
Although this may sound like a compliment, this typically means something bad, because conferences want valuable scientific research and not engineering work. In all these cases I've received such "compliments", the paper was rejected.
Although I understand that a paper has to conform to a certain quality level and not every story is suitable for presentation on a software engineering conference, such compliments bother me a bit, because I don't know exactly what they mean, nor I have an exact idea what the reviewers want me to do with these kind of criticisms. And another big question that I have is: When do we consider something engineering and when is it really science?
Software engineering conferences
There are many software engineering conferences around. We have sub-field conferences about specific domains, such as software testing, reliability engineering, programming languages, reverse engineering and repository mining. Furthermore, we have a number of top general conferences in which every software engineering task is potentially eligible for presentation. The International Conference on Software Engineering (ICSE) probably is the most well known top general conference in our field.
Most of the attendants of these conferences are academic people, although there is also some industry participation. For example, I have heard that the industry participation of ICSE is less than 20%, while this used to be around 50% in the past.
Difference between engineering and science
In the software engineering domain, the difference between science and engineering is not so clear to me. A common perception is that scientific research is what we should do in academia, engineering is what industry does.
Lately I have been looking for answers and (for fun) I dived into the history of the ICSE, the most popular software engineering conference in our field. I've looked into the proceedings of ICSE 1976, which was the first edition of the real ICSE (it seems that ICSE 1975 was actually NCSE: National Conference on Software Engineering). Furthermore, in this year the industry participation (if what I've heard is correct) is around 50%.
When I looked at the proceedings, the following paper caught my interest: 'Research Paradigms in Computer Science' by Peter Wegner from Brown University. In this paper, the author explores 4 influential definitions of computer science. Interestingly enough, this paper also reports about research in engineering:
Research in engineering is directed towards the efficient accomplishment of specific tasks and towards the development of tools that will enable classes of tasks to be accomplished more efficiently.
Furthermore, the paper also makes a distinction between a research engineer and a practicing engineer:
The problem-solving paradigm of the practicing engineer generally involves a sequence of systematic selection or design decisions which progressively narrow down alternative options for accomplishing the task until a unique realization of the task is determined.
The research engineer may use the paradigms of mathematics and physics in the development of tools for the practicing engineer, but is much more concerned with the practical implications of his research than the empirical scientist or mathematician.
After reading this paper, assuming that this properly defines what research in engineering is about, I have the following idea about academic software engineering research contributions:
- You shouldn't merely solve a problem for which you have to engineer something. It should be related to software engineering processes.
- Your contribution should fill a gap within in a software engineering process. For example, it should offer an improvement or a new insight everybody should know about. Furthermore, this contribution must be novel.
- The contribution should be significant enough.
- You need strong motivation why this contribution is needed, which is not always obvious. You also need to look at what has been done before.
- You need to prove what you claim is valid (or at least strong enough) and you need to describe how it compares to (possibly) related work.
- And of course, you need to explain what you did
If you just solve an arbitrary problem by means of software engineering, without filling a gap, it is engineering.
If I take what I have written in the previous section into account and look back to what I've done before, I don't think I have written a pure engineering paper so far. I think that in these cases probably something else went wrong. This is what I think that could have go wrong:
- The motivation is unclear or misunderstood.
- Perhaps the paper may contain too many implementation details.
- There is something wrong with the evaluation method or the evaluation is not strong enough.
- Reviewers have different expectations about submissions. Maybe this is also due to the fact that software deployment is a very uncommon research subject.
- A solution is/looks too simple
I find it a bit disappointing that I have to guess about these reasons myself. If I get a bad review, I would appreciate it if reviewers also make themselves clear why they think it's engineering, so that I can do something with their feedback (or preferably: use concrete criticisms as I have shown above). I think that's what peer-reviewing is about, right?
I'd also like to elaborate a bit more on engineering in this blog post. Although academic conferences want scientific contributions, is doing engineering work a bad thing? In my opinion it's also a good thing to do engineering work (even as a researcher), for the following reasons:
- Typically, many software engineering problems arise in practice. Being in the field gives you a lot of experience and a better understanding of particular software engineering problems.
- Many software engineering contributions are implemented as tools. Tools need to be constructed from various components by using various means. So in order to fill a gap in software engineering processes, you also have to do some software engineering.
- Tools need to be applied in real world scenarios. In such cases, you may have to study existing systems, extend/integrate your tooling, or even implement work-arounds because you're tooling assumes ideal behaviour.
- There may be lessons that you have learned in your research that every software engineer should know about.
In my period as a PhD researcher, I have done many engineering contributions, next to the scientific papers I have written. Typically, I use my blog to report about these, such as applying Disnix for deploying .NET services or comparing NixOS with GoboLinux. Although these blog posts aren't really useful scientific contributions, they are if fact more appreciated by the people I work with. Furthermore, these blog posts have also attracted many more readers than all my academic papers combined.
Last year I have also presented pieces of my work at FOSDEM, the Free/Open-Source Developers' European Meeting attracting thousands of developers from all over the world. For some reason, my talk attracted quite a lot of people and I also received much more interesting questions than I have ever received on any academic software engineering conference.
So what's the big deal anyway?
Next to the discussion of the meaning of science and engineering, I'd like to point out that engineering is not something bad, although it may not always be something reviewers of academic conferences are interested in. Engineering aspects, however, are very useful for industry people. They typically care more about what benefits new tools could offer and how they can apply these techniques in their environment instead of seeing a mathematical proof.
There have been many discussions among academic conference organisers about the question why the industry participation is so low these days and how this could be improved. I think one of the big reasons is a mismatch of interests. There are of course more reasons, but this is probably not something you could blame academic people for.
In this blog post, I have tried to discover what scientific research in software engineering is and what ordinary engineering is. I have also expressed my opinion about engineering and concluded that it's not useless, although it may not be something that program committee members of academic conferences are interested in.
Finally, I want to make myself clear that I have nothing against ICSE or any other academic conference. This blog post is not an ICSE rant. I find the ICSE a good conference, which I have attended with pleasure in 2009 and 2011. I have attended a number of very good presentations and I have also met quite a number of great people. Also I'm currently collaborating with a few researchers that I have met there.
(Ok! Ok!... I'm not going to lie about the fact I'm a bit disappointed that our paper for this years' edition has been rejected and I agree that certain parts of the paper can be improved. Furthermore, this blog post is not elaborate about this rejection, but about a phenomenon occurring within the academic software engineering research community that bothers me a bit.)