On reading research papers and maintaining knowledge

Ten years ago I have obtained my PhD degree and made my (somewhat gradual) transition from academia to industry. Despite the fact that I made this transition a long time ago, I still often get questions from people who are considering doing a PhD.

Most of the discussions that I typically have with such people are about writing -- I have already explained plenty about writing in the past, including a recommendation to start a blog so that writing becomes a habit. Having a blog allows you to break up your work into manageable pieces and build up an audience for your work.

Recently, I have been elaborately reorganizing files on my hard-drive, a tedious task that I often do at the end of the year. This year, I have also been restructuring my private collection of research papers.

Reading research papers became a habit while working on my master's thesis and doing my PhD. Although I have left academia a long time ago, I have retained the habit, although the amount of papers and articles that I read today is much lower than my PhD days. I no longer need to study research works much, but I have retained the habit to absorb existing knowledge and put things into context whenever I intend to do something new, for example for my blog posts or software projects.

In 2020, during the first year of the COVID pandemic, I have increased my interest in research papers somewhat, because I had to revise some of the implementations of algorithms in the Dynamic Disnix framework that were based on work done by other researchers. Fortunately, the ACM temporarily opened their entire digital library to the public for free so that I could get access to quite an amount of interesting papers, without requiring to pay.

In addition to writing, reading in academic research is also very important, for the following reasons:

• To expand your knowledge about your research domain.
• To put your own work into context. If you want to publish a paper about your work, having a cool idea is not enough -- you have to explain what your research contributes: what the innovation is. As a result, you need to relate to earlier work and (optionally) to studies that motivate the relevance of your work. Furthermore, you can also not just take credit for work that has already been done by others. As a consequence, you need to very carefully investigate what is out there.
• You may have to peer review papers for acceptance in conference proceedings and journals.

Reading papers is not an easy job -- it often takes me quite a bit of time and dedication to fully grasp a paper.

Moreover, when studying a research paper, you may also have to dive into related work (by examining a paper's references, and the references of these) to fully get an understanding. You may have to dive several levels deep to gain enough understanding, which is not a straightforward job.

In this blog post, I want to share my personal experiences with reading papers and maintaining knowledge.

My personal history with reading

I have an interesting relationship with reading. Already at young age, I used to study programming books to expand my programming knowledge.

With only limited education and knowledge, I was very practically minded -- I would relentlessly study books and magazines to figure out how to get things done, but as soon as I figured out enough to get something done I stopped reading, something that I consider a huge drawback of the younger version of me.

For example, I still vividly remember how I used to program 2D side scroller games for the Commodore Amiga 500 using the AMOS BASIC programming language. I figured out many basic concepts by reading books, magazines and the help pages, such as how to load IFF/ILBM pictures as backgrounds, load ProTracker modules as background music, using blitter objects for moving actors, using a double buffer to smoothly draw graphics, side scrolling, responding to joystick input etc.

Although I have managed to make somewhat playable games by figuring out these concepts, the games were often plagued by bugs and very slow performance. A big reason that contributed to these failures is because I stopped reading after mastering the basics.

For example, to improve performance, I should have disabled the autoback feature that automatically swaps the physical and logical screens on every drawing command and do the screen swap manually after all drawing instructions were completed. I knew that using a double screen buffer would take graphics glitches away, but I never bothered to study the concepts behind it.

As I grew older and entered middle school, I became more critical of myself. For example, I learned that it is essential to properly cite where you get your knowledge from rather than "pretending" that you are pulling something out of your own hat. :)

Fast forwarding to my studies at the university: reading papers from the academic literature became something that I had to commonly do. For example, I still remember the real-time systems and software architecture courses.

The end goal of the former course was to write your own research paper about a subject in the real-time systems domain. In this course, I learned, in addition to real-time system concepts, how academic research works: writing a research paper is not just about writing down a cool idea (with references that you used as an inspiration), but you also need to put your work into context -- a research paper is typically based on work already done by others, and your paper typically serves as an ingredient that can be picked up by other researchers.

In the latter course, I had to read quite a few papers in the software architecture domain, write summaries and discuss my findings with other students. Here, I learned that reading papers is all but a trivial job:

• Papers are often densely written. As a result, I get overwhelmed with information and it requires quite a bit of energy from my side to consume all of it.
• The formatting of many papers is not always helpful. Papers are typically written for print, not for reading from a screen. Also, the formatting of papers are not always good for displaying code fragments or diagrams.
• There is often quite a bit of unexplained jargon in a paper. To get a better understanding you need to dive deeper into the literature, such as also studying the references of the papers or books that are related to the subject.
• Sometimes authors frequently use multi-syllable words.
• It is also not uncommon for authors to use logic and formulas to formalize concepts and mathematically prove their contributions. Although formalization helps to do this, reading formulas is often a tough job for me -- there is typically a huge load of information and Greek symbols. These symbols IMO are not always very helpful to relate to what concepts they represent.
• Authors often tend to elaborately stress out the caveats of their contributions, making things hard to read.

Despite having read many papers in the last 16 years and I got better at it, reading still remains a tough job because of the above reasons.

In the final year of my master's, I had to do a literature survey before starting the work on my master's thesis. The first time I heard about this, I felt scared, because of my past experiences with reading papers.

Fortunately, my former supervisor: Eelco Visser, was very practically minded about the process -- he wanted us to first work on practical aspects of their research projects, such as WebDSL: a domain-specific language for developing web applications with a rich data model and related tools, such as Stratego/XT and the Nix package manager.

After mastering the practical concepts of these projects, doing a literature survey felt much easier -- instinctively, while using these tools in practice, I became more interested in learning about the concepts behind them. Many of their underlying concepts were described in research papers published my my colleagues in the same research department. While studying these papers, I also got more motivated/interested into diving deeper in the academic literature by studying the papers' references and searching for related subjects in the digital libraries of the ACM, IEEE, USENIX, Springer, Elsevier etc.

During my PhD reading research papers became even more important. In the first six months of my PhD, I had a very good start. I published a paper about an important aspect of my master's thesis: atomic upgrading of the static parts of a distributed system, and a paper about the overall objective of the research project that I was in. I have to admit that, despite having these papers instantly accepted, I still had the wrong mindset -- I was basically just "selling my cool ideas" and finding support in the academic literature, rather than critically studying what is out there.

For my third paper, that covers a new implementation of Disnix (the third major revision to be precise), I learned an important/hard lesson. The first version of the paper got badly rejected by the program committee, because of my "advertising cool ideas mindset" that I always used to have -- I failed to study the academic literature well enough to explain what the innovation of my paper is in comparison to other deployment solutions. As a consequence, I got some very hard criticisms from the reviewers.

Fortunately, they gave me good feedback. For example, I had to study papers from the Working Conference on Component Deployment. I have addressed their criticisms and the revised paper got accepted. I learned what I had to do in the future -- it is a requirement to also study the academic literature well enough to explain what your contribution is and demonstrate its relevance.

This rejection also changed my attitude how I deal with research papers. Previously, after my work for a paper was done, I would typically discard the artifacts that I no longer needed, including the papers that I used as a reference. After this rejection, I learned that I need to build my own personal knowledge base so that for future work, I could always relate to the things that I have read previously.

Reading research papers

I have already explained that for various reasons, reading research papers is all but an easy job. For some papers, in particular the ones in my former research domain: software deployment, I got better as I grew more familiar to the research domain.

Nonetheless, I still sometimes find reading papers challenging. For example, studying algorithmic papers is extremely hard IMO. In 2021, I had to revise my implementations of approximation solutions for the multi-way cut, and graph coloring problems in the Dynamic Disnix framework. I had to re-read the corresponding papers again. Because they were so hard to grasp, I wrote a blog post that explains how I practically applied them.

To fully grasp a paper, reading it a single time is often not enough. In particular the algorithmic papers that I mentioned earlier, I had to read them many times.

Interestingly enough, I learned that reading papers is also a subject of study. A couple of years ago I discovered a paper titled: "How to Read a Paper" that explains a strategy for reading research papers using a three-pass approach:

• First pass: bird's eye view. Study the title, abstract, introduction, headings, conclusions. A single pass is often already enough to decide whether a paper is relevant to read or not.
• Second pass: study in greater detail, but ignore the big details, such as mathematical proofs.
• Third pass: read everything in detail by attempting to virtually re-implement the paper.

After discovering this paper, I have also been using the three pass approach. I have studied most of my papers in my collection in two passes, and some of them in detail in three passes.

Another thing that I discovered by accident is that to extensively study literature, a continuous approach works better for me (e.g. reserving certain timeslots in a week) than just reserving longer periods of time that consist of only reading papers.

Also, regularly discussing papers with your colleagues helps. During my PhD days, I did not do it that often (we had no formal "process" for it) but there were several good sessions, such as a program committee simulation organized by Arie van Deursen, head of our research group.

In this simulation, we organized a program committee meeting of the ICSE conference in which the members of the department represented program committee members. We have discussed submitted papers and voted for acceptance or rejection. Moreover, we also had to leave the room if there was a conflict of interest.

I also learned that Edsger Dijkstra, a famous Dutch computer scientist, organized the ETAC (Eindhoven Tuesday Afternoon Club) and ATAC (Austin Tuesday Afternoon Club) in which amongst other activities, reading and discussing research papers was a recurring activity.

Building up your personal knowledge base

As I have explained earlier, I used to throw away my downloaded papers when the work for a paper was done, but I changed that habit after that hard paper rejection.

There are many good reasons to keep and organize the papers that you have read, even if they do not seem to be directly relevant to your work:

• As I have already explained, in addition to reading a single paper and writing your own research papers, you need to maintain your knowledge base so that you can put them into context.
• It is not always easy to obtain papers. Many of them are behind a paywall. Without a subscription you cannot access them, so once you have obtained them it is better to think twice before you discard them. Fortunately, open access becomes more common but it still remains a challenge. Arie van Deursen has written a variety of blog posts about open access.
• Although many papers are challenging to read, I also started to appreciate certain research papers.

My own personal paper collection has evolved in an interesting way. In the beginning, I just used to put any paper that I have obtained into a single folder called: papers until it grew large enough that I had to start classifying them.

Initially, there was a one-level folder structure, consisting of categories such as: deployment, operating systems, programming languages, DSL engineering etc. At some point, the content of some of these folders grew large enough and I introduced a second level directory structure.

For example, the sub folder for my former research domain: software deployment (the process that consists of all activities to make a software system available for use) contains the largest amount of papers. Currently, I have collected 168 deployment papers that I have divided over the following sub categories:

• Deployment models. Papers whose main contribution is a means to model various deployment aspects of a system, such as the structure of a system and deployment activities.
• Deployment planning. Papers whose main contribution are algorithms that decide a suitable/optimal deployment architecture based on functional and non-functional requirements of a system.
• Empirical studies. Papers containing empirical studies about deployment in practice.
• Execution. Papers in which the main contribution is executing deployment activities. I have also sub categorized this folder into technology-specific solutions (e.g. a solution is specific to a programming language, such as Java or component technology, such as CORBA) and generic solutions.
• Practice reports. Papers that report on the use of deployment technologies in practice.
• Surveys. Papers that analyse the literature and draw conclusions from them.

A hierarchical directory structure is not perfect for organizing papers -- for many papers there is an overlap between multiple sub domains in the software engineering domain. For example, deployment may also be related to a certain component technology, in service of optimizing the architecture of a system, related to other configuration management activities (versioning, status accounting, monitoring etc.) or an ingredient in integration testing. If there is an overlap, I typically look at the strongest kind of contribution that the paper makes.

For example, in the deployment domain, Eelco Dolstra wrote a paper about maximal laziness, an important implementation aspect of the Nix expression language. The Nix package manager is a deployment solution, but the contribution of the paper is not deployment, but making the implementation of a purely functional DSL efficient. As a result, I have categorized the paper under DSL engineering rather than deployment.

The organization of my paper collection is always in motion. Sometimes I gain new insights causing me to adjust the classifications, or when a collection of papers for a sub domain grows, I may introduce a second-level classification.

Some practical tips to get familiar with a certain research subject

So what is my recommended way to get familiar with a certain research subject in the software engineering domain?

I would start by doing something practical first. In software engineering research domain, often the goal is to develop or examine tools. Start by using these tools first and see if you can contribute to them from a practical point of view -- for example, by improving features, fixing bugs etc.

As soon as I have mastered the practical aspects, I may typically already get motivated to dive into their underlying concepts by studying the papers that cover them. Then I will apply the three pass reading strategy and eventually study the references of the papers to get a better understanding.

After my orientation phase has finished, the next thing I would typically look at is the conferences/venues that are directly related to the subject. For software deployment, for example, there used to be only one subject-related conference: the Working Conference On Component Deployment (that unfortunately was no longer organized after 2005). It is typically a good thing to have examined all the papers of the related conferences/venues, by at least using a first-pass approach.

Then a potential next step is to search for "early defining papers" in that research area. In my experience, many research papers are improving on concepts pioneered by these papers, so it is IMO a good thing to know where it all started.

For example, in the software deployment domain the paper: "A Characterization Framework for Software Deployment Technologies" is such an early defining paper, covering a deployment solution called "The Software Dock". The paper comes with a definition for the term: "software deployment" that is considered the canonical definition in academic research.

Alternatively, the paper: "Software Deployment, Past, Present and Future" is a more recent yet defining paper covering newer deployment technologies and also offers its own definition of the term software deployment.

For unknown reasons, I always seem to like early defining papers in various software engineering domains. These are some of my recommendations of early defining papers in other software engineering domains:

• General purpose programming languages: The IBM 701 Speedcoding System, The FORTRAN Automatic Coding System, Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I, Modified Report on the Algorithmic Language ALGOL 60.
• Domain-specific languages: Programming Pearls: Little Languages, Domain-Specific Languages: An Annotated Bibliography
• Programming: A case against the GO TO statement, Notes on Structured Programming
• Operating systems: The Structure of the "THE"-multiprogramming System, The Multics Input/Output System, A General-Purpose File System for Secondary Storage, The UNIX Time-Sharing System
• Software engineering in general: Research paradigms in computer science

After studying all these kinds of papers, your knowledge level should already be decent enough to find your way to study the remaining papers that are out there.

Literature surveys

In addition to research papers that need to put themselves into context, extensive literature surveys can also be quite valuable to the research community. During my PhD, I learned that it is also possible to publish a paper about a literature survey.

For example, some of my former colleagues did an extensive and systematic literature survey in the dynamic analysis domain. In addition to the results, the authors also explain their methodology, that consists of searching on keywords, looking for appropriate conferences and journals and following the papers' references. From these results they have derived an attribute framework and classified all the papers into this attribute framework.

I have kept the paper as a reference for myself, because I like the methodology. I am not so interested in dynamic analysis or program comprehension from a research perspective.

Literature surveys also exist in my former research domain, such as a survey of deployment solutions for distributed systems.

Conclusions

In this blog post, I have shared my experiences with reading papers and maintaining knowledge. In research, it is quite important and you need to take it seriously.

Fortunately, during my PhD I have learned a lot. In summary, my recommendations are:

• Archive your papers and build up a personal knowledge base
• Start with something practical
• Follow paper references
• Study early defining papers
• Find people to discuss with
• Study continuously in small steps

Although I never did an extensive literature survey in the software deployment domain (it is not needed for submitting papers that contribute new techniques) I can probably even write a paper about software deployment literature myself. The only problem is that I am not quite up to date with work that has been published in the last few years, because I no longer have access to these digital libraries.

Moreover, I also need to find the time and energy to do it, if I really want to :)

On motivation and purpose

In this blog post, I will discuss an important and recurring non-technical subject that is common in our field of expertise: motivation. Pretty much everybody in the industry that I know (including myself) have motivational problems once in a while. There are a variety of reasons why people get motivated or demotivated. In my experience, lack of motivation is one of the more important reasons why people quit and change jobs.

I will elaborate about one of the ingredients that is important to me: purpose.

A common misunderstanding: technology

What I have noticed is that quite a few people think that software developers are generally motivated by technology -- for instance, they will be motivated if they can use the latest computer models and the latest and greatest software technologies to develop software. At the time of writing this blog post, I see many companies that have vacancies advertising with technologies, such as: Node.js, Angular, React, JavaScript, Python, Go, Docker, Kubernetes etc.

While it is true that my skills are stronger with one particular class of technology than another, and that I actually do have preferences, such as which kind of tool is best for deployment, technology alone is not something that gives me motivation.

Instead: development is done for a purpose. Typically, we develop software systems to accomplish certain goals for a certain audience. As developers, we want to reach these goals and offer good quality -- technology is in service of reaching these goals. This sometimes also means that I have to work with technology that is not my primary choice or something I am not familiar with.

In addition to using technology, we may have to do things that are not technical at all. Typically, developing software systems is team work. To work effectively as a team, communication is also very important. For example:

• You may need to write good documentation so that ideas, requirements and other technical considerations are clear among team members. For this you also need proper writing skills and mastery of the English language, if your native language is different.
• You may need to give trainings or knowledge transfer presentations to end users or fellow team members for which you need good presentation skills.

Examples

I have a lot of interesting anecdotes of unorthodox choices that I made in the past, that I can rationalize by explaining the purpose. Some of them are:

• I did a PhD, which has number of implications for your career -- as a PhD student you are (fortunately!) employed in the Netherlands, so you will get a salary and additional benefits such as a pension. The biggest disadvantage is that you only get a temporary employment contract and your salary is much lower that an industry job in the same field. Without a permanent employment contract, for example, it is very difficult to get a mortgage to buy a house.
  
  Because of this "disadvantage", quite a few people think that the PhD degree is the reason that motivated me (because supposedly it provides you better job perspectives, which is generally not the case in software technology).
  
  My motivation was actually much different: the subject of my PhD research was software deployment. Before I started my PhD research, I already knew how difficult it was to construct software systems from source code, how to package components and how to deploy service-oriented systems that are distributed and facilitate technology diversity. To have the possibility to dedicate yourself to a subject for a few years and construct tools to automate and improve such deployment processes was the real reason why I wanted to do this.
  
  (As a sidenote: although constructing tools was my primary motivation and the main objective of research in software engineering, I also struggled a bit with motivation on a few occasions. In a blog post that I wrote in my final year, I explained what got me demotivated).
  
  I learned a lot of things while I was a PhD student. In addition to technology, I also considerably improved my writing skills, presentations skills, I started this blog, and I did quite a lot of traveling alone, which gives you all kinds of interesting experiences. All of these learning experiences were in service of reaching my main goal.
  
• During my PhD, I was also a visiting researcher at Philips Healthcare. At Philips Healthcare, I was applying my research and tooling to medical systems.
  
  Most of the technology stack used at Philips were Microsoft technologies: Windows as an operating system, Internet Information Services as web server, SQL server as DBMS, .NET/C# as an implementation language, and so on.
  
  I was (and still am) a Linux/Free software/Open source person. As a result, I was always avoiding these technologies as much as possible -- what I basically did not like about them is that were all proprietary and strongly tied to the Windows operating system. Furthermore, I did not like Internet Information Services because of its bad security reputation.
  
  At the same time, the deployment tool I was working on (Disnix), was also designed to facilitate technology diversity, including technologies that I did not like per se. As part of my work at Philips, I managed to automate the build processes of C#/.NET projects with the Nix package manager and I created Dysnomia plugins so that services implemented with Microsoft technology could be deployed with Disnix.
  
  I also learned quite a few things about the .NET packaging internals, such as strong names and the global assembly cache. Because I wanted to facilitate technology diversity, I was motivated to learn these concepts.
  
• At Conference Compass, I developed a variety of Nix functions to build mobile applications (native Android, native iOS and Titanium). I was highly motivated for native Android, because of two reasons: I have an Android device myself and I consider it quite valuable to automate the build processes of complex Android apps including company applications.
  
  The iOS and Titanium build functions were less interesting. In particular, what I disliked the most about iOS is that I do not have such a device myself, and I really do not like the fact that app delivery to iOS devices (e.g. iPhone, iPad) rely on one single distribution channel: Apple. It is not even possible to deploy an app that you have developed yourself for a device that you own, without obtaining a certificate and provisioning profile from Apple!
  
  Still, I considered a conference app to be quite valuable. Our audience uses both iOS and Android devices. This means that iOS cannot be avoided, because that would disqualify a significant chunk of our audience.
  
  Furthermore, I also wanted to support technology diversity again -- to be able to build apps with the Nix package manager for any mobile platform is useful. The domain and technology diversity of Nix motivated me to also learn about these areas that I initially did not find interesting.
  
• For my Nix and Disnix related work, I have developed several small utility libraries, e.g. for concurrency and data exchange, and I explored underlying concepts, such as layered build function abstractions. The primary reason to do these things is not because I was directly interested in concepts, but they significantly contribute to quality improvement of the deployment tools -- they make the infrastructure faster, more robust and easier to maintain.

What I typically do

To get motivated, I basically need to know my purpose, and then define and align goals. This is typically easier said than done and it requires quite a bit of exploration.

Basically, I have adopted the following habits whenever I am new to some organization:

• Learn about the company: IMO is important to know (at least from a high level perspective) the kinds of products and/or services a company offers, because the work you do is primarily focused on improving business value. For example, when I had the feeling that I learned enough about the Mendix product and service, I wrote a small article about it on my blog and I am grateful to Mendix that I am allowed to do this.
• Learn about the domain: in addition to the company product and/or service, it is also important to know in what domain it is active. You will get a better understanding about the customers, what they want and what kind of challenges you might face in reaching them.
  
  For example, at Philips I learned a lot about medical regulations, at Conference Compass I learned the true value of having digital versions of traditional paper programs (that cannot change after they have been printed) and at Mendix it is interesting to continuously think about what kinds of value can low-code development offer (in terms of speed, quality and target audience, such as non-technical developers).
• Learn about your team and their contributions. Typically, in large organizations with big/complex services or products, you typically work on a particular component (or stack of components), not the product as a whole. For me, it was also interesting to see what my team's contribution is and what value it offers to end users.
  
  To fully answer that question, I wrote a simple tutorial page that explains how end users use our team's product -- it helped a lot to understand what my changes will contribute to and I noticed that it has been particularly helpful for on-boarding new team members.
• Define and align goals. Typically, after learning about the company and the teams' products and/or services, you will probably see opportunities. For example, you may notice that there is something that can be improved with technology that you are familiar with. It is good to remember them and work with the team to address them. Be proactive.
• Keep learning. In addition to opportunities, you may also probably experience confusion or realize that there are things that you do not know yet. I always try to allocate time to learn new things, both technical (e.g. new programming languages, frameworks, technologies) and non-technical (e.g. communication, the domain). From my experience, in software engineering there is only one constant and that constant is change.
• Try to be pragmatic. This is an important personal lesson for me: since you are working in a team and every person is different (different opinions and priorities), you must sometimes accept that you can not always (directly) accomplish everything you want and that things will work out the way you have intended.

What organizations can do

In addition to the things you can do as an individual, you also need support from the organization. I highly value the following traits:

• Transparency. It is very useful to inform teams about the impact of the work they do: both positive and negative. I have seen in the past that it is quite tempting, for example, that after a failure things get covered up. I personally believe it is important for developers to know about the strengths and weaknesses of the software they work on, so that they can make meaningful contributions to make something a success.
• Opportunities to get in touch with end users and other relevant stakeholders. In many organizations, developers are rarely in touch with end users and that is typically for a good reason: they should not get distracted from their work.
  
  Although I consider preventing distraction a good thing, I personally believe it would not be a bad thing to get in touch with end users sometimes: it gives a developer direct insights in how well the product works and what a customer needs or struggles with.
  
  At Mendix, we sometimes have clients that will visit us to explain what they do with Mendix and what kinds of challenges they face -- everybody in the company is invited and has the opportunity to ask questions. This will not happen on a very frequent basis, but having these sessions once in a while is something I consider to be very valuable.
• Offer time to learn and explore. Typically to reach goals, quite frequently developers need to expand their skill set by e.g. by learning new technologies or simply explore the possibilities. They should be offered the time to do this.
• Taking technical debt seriously. Technical debt, a phenomenon that hinders evolution of a software system, by postponing/skipping certain kind of work (i.e. taking shortcuts) that should have been done (e.g. testing, documentation, refactoring) should also be taken seriously.
  
  When a system has a lot of technical debt, making changes and improving quality can be enormously (and unnecessarily) difficult and time consuming. In extreme cases, even a subtle change takes too much time. As a result, it becomes quite easy to lose track of the original purpose and easily causes developers lose their motivation.
• Taking feedback from developers seriously. Developers typically raise concerns (such as quality issues) that may not always look urgent -- as a result, it is very tempting for organizations to always give priority to new features than quality improvements. This may sometimes cause quality to degrade significantly over time.
  
  Whilst developers are typically not against developing new features, they are very concerned about poor quality and high technical debt. If the latter grows out of hand too much, it is quite easy for developers to lose track of the original purpose of the product or service and lose motivation.

Conclusion

In this blog post, I have shared my experiences with motivation in relation to purpose. Although it may sound conceptually simple, learning about the purpose of a product and service and aligning goals is actually quite difficult -- it requires you to learn about an organization, a product or service, the people in the company, your team, and about yourself. It typically is quite a journey with interesting and, sometimes, a few non-interesting steps.

Finally, purpose is not the only factor that motivates or demotivates me as a developer, but it is an important one.

A minimalistic discovery and architecture documentation process

In a blog post written a couple of years ago, I have described how to set up a basic configuration management process in a small organization that is based on the process framework described in the IEEE 828-2012 configuration management standard. The most important prerequisite for setting up such a process is identifying all configuration items (CIs) and storing them in a well-organized repository.

There are many ways to organize configuration items ranging from simple to very sophisticated solutions. I used a very small set of free and open source tools, and a couple of simple conventions to set up a CI repository:

• A Git repository with an hierarchical directory structure referring to configurations items. Each path component in the directory structure serves a specific purpose to group configuration items. The overall strategy was to use a directory structure with a maximum of three levels: environment/machine/application. Using Git makes it possible to version configuration items and share the repository with team members.
• Using markdown to write down the purposes of the configuration items and descriptions how they can be reproduced. Markdown works well for two reasons: it can be nicely formatted in a browser, but also read from a terminal when logged in to remote servers via SSH.
• Using Dia for drawing diagrams of systems consisting of more complex applications components. Dia is not the most elegant program around, but it works well enough, it is free and open source, and supported on Linux, Windows and macOS.

My main motivation to formalize configuration management (but only lightly), despite being in a small organization, is to prevent errors and minimize delays and disruptions while remaining flexible by not being bound to all kinds of complex management procedures.

I wrote this blog post while I was still employed at a small-sized startup company with only one development team. In the meantime, I have joined a much bigger organization (Mendix) that has many cross-disciplinary development teams that work concurrently on various aspects of our service and product portfolio.

About microservices

When I just joined, the amount of information I had to absorb was quite overwhelming. I also learned that we heavily adopted the microservices architecture paradigm for our entire online service platform.

According to Martin Fowler's blog post on microservices, using microservices offers the following benefits:

• Strong module boundaries. You can divide the functionality of a system into microservices and make separate teams responsible for the development of each service. This makes it possible to iterate faster and offer better quality because teams can focus on themselves on a subset of features only.
  
• Independent deployment. Microservices can be deployed independently making it possible to ship features when they are done, without having complex integration cycles.
• Technology diversity. Microservices are language and technology agnostic. You can pick almost any programming language (e.g. Java, Python, Mendix, Go), data storage solution (e.g. PostgreSQL, MongoDB, InfluxDB) or operating system (e.g. Linux, FreeBSD, Windows) to implement a microservice making it possible pick the most suitable combination of technologies and use them at their full advantage.

However, decomposing a system into a collection of collaborating services also comes at a (sometimes substantial!) price:

• There is typically much more operational complexity. Because there are many components and typically a large infrastructure to manage, activities such as deploying, upgrading, and monitoring the condition of a system is much more time consuming and complex. Furthermore, because of technology diversity, there are also many kinds of specialized deployment procedures that you need to carry out.
• Data is eventually consistent. You have to live with the fact that (temporary) inconsistencies could end up in your data, and you must invest in implementing facilities that keep your data is consistent.
• Because of distribution development is harder in general -- it is more difficult to diagnose errors (e.g. a failure in one service could trigger a chain reaction of errors, without having proper error traces), it is harder to test a system because of additional deployment complexity. The network links between services may be slow and subject to failure, causing all kinds of unpredictable problems. Also machines that host critical services may crash.

Studying the architecture

When applied properly, e.g. functionality is well separated and there is strong cohesion and weak coupling between services, while investing in solutions to cope with the challenges listed above, the benefits of microservices can be reaped, resulting in a scalable systems that can be developed my multiple teams working on features concurrently.

However, an important prerequisite for making changes in such an environment, and maintaining or improving the quality properties of a system, requires discipline and a relatively good understanding of the environment -- in the beginning, I faced all kinds of practical problems when I wanted to make even a subtle change -- some areas of our platform where documented, while others were not. Some documentation was also outdated, slightly incomplete and sometimes inconsistent with the actual implementation.

Certain areas of our platform were also highly complex resulting in very complex architectural views, with many boxes and arrows. Furthermore, information was also scattered around many different places.

As part of my on-boarding process, and as a means to cope with some of my practical problems, I have created a documentation repository of the platform that our team develops by extending the (minimalistic) principles for configuration management described in the earlier blog post.

I realized that simply identifying the service components of which the system consists, is not enough to get an understanding of the system -- there are many items and complex details that need to be captured.

In addition to the identification of all configuration items, I also want:

• Proper guidance. To understand a particular piece of functionality, I should not need to study every component in detail. Instead, I want to know the full context and only the details of the relevant components.
• Completeness. I want all service components to be visible. I do not want any details to be covered up. For example, I have also seen quite a few diagrams that hide complex implementation details. I much rather want flaws to be visible so that they can be resolved at a later point in time.
• Clear boundaries. Our platform is not self contained, but relies on services provided by other teams. I want to know what components are our responsibility and what is managed by external teams.
• Clarity. I want to know what the purpose of a component is. Their names may not always necessarily reflect or explain what they do.
• Consistency. No matter how nicely a diagram is drawn, it should match the actual implementation or it is of very little use.
• References to the actual implementation. I also want to know where I can find the implementation of a component, such as its Git repository.

Documenting the architecture

To visualize the architecture of our platform and organize all relevant information, I followed a strategy:

• I took the components (typically their source code repositories) as the basis for everything else -- every component translates to a box in the architecture diagram.
• I analyzed the dependency relationships between the components and denoted them as arrows. When a box points to another box by means of an arrow, this means that the other box is a dependency that should be deployed first. When a dependency is absent, the service will (most likely) not work.
• I also discovered that the platform diagram easily gets cluttered by the sheer amount of components -- I decided to combine components that have very strongly correlated functionality in feature groups (that have dashed borders). Every feature group in architecture diagrams refers to another sub architecture diagram that provides a more specialized view of the feature group.
• To clearly illustrate the difference between components that are our responsibility and those that are maintained by others teams, I make all external dependencies visible in the top-level architecture diagram.

The notation I used for these diagrams is not something I have entirely invented from scratch -- it is inspired by graph theory, package management and service management concepts. Disnix, for example, can visualize deployment architectures by using a similar notation.

To find all relevant information to create the diagrams, I consulted various sources:

• I studied existing documents and diagrams to get a basic understanding of the system and an idea of the details I should look at.
• I talked to a variety of people from various teams.
• I looked inside the configuration settings of all deployment solutions used, e.g. the Amazon AWS console, Docker, CloudFoundry, Kubernetes, Nix configuration files.
• Peek inside the source code repositories and look for settings that are references to other systems, such as configuration values that store URLs.
• When I am in doubt: I consider the deployment configuration files and source code the "ultimate source of truth", because no matter how nice a diagram looks, it is useless if it is implemented differently.

Finally, just drawing diagrams will not completely suffice when the goal is provide clarity. I also observed that I need to document some leftover details.

Foremost, having a diagram without the semantics not explained will typically leave too many details open to interpretation to the user, so you need to explain the notation.

Second, you need to provide additional details about the services. I typically enumerate the following properties in a table for every component:

• The name of the component.
• A one line description stating its purpose.
• The type of project (e.g. a Python/Java/Go project, Docker container, AWS Lambda function, etc.). This is useful to determine the kind of deployment procedure for the component.
• A reference to the source code repository, e.g. a Git repository. The README of the corresponding repository should provide more detailed information about the project.

Benefits

Although it is quite a bit of work to set up, having a well documented architecture provides us the following benefits:

• More effective deployment. Because of the feature groups and dividing the architecture into multiple layers, general concepts and details are separated. This makes it easier for developers to focus and absorb the right detailed knowledge to change a service.
• More consensus in the team about the structure of the system and general quality attributes, such as scalability and security.
• Better on-boarding for new team members.

Discussion

Writing architecture documentation IMO is not rocket science, just discipline. Obviously, there are much more sophisticated tools available to organize and visualize architectures (even tools that can generate code and reverse engineer code), but this is IMO not a hard requirement to start documenting.

However, you can not take all confusion away -- even if you have the best possible architecture documentation, people's thinking habits are shaped by the concepts they know and there will always be a slight mismatch (which is documented in academic research: 'Why Is It So Hard to Define Software Architecture?' written by Jason Baragry and Karl Reed).

Finally, architecture documentation is only a first good step to improve the quality of service-oriented systems. To make it a success, much more is needed, such as:

• Automated (and reproducible) deployment processes.
• More documentation (such as the APIs, end-user documentation, project documentation).
• Automated unit, integration and acceptance testing.
• Monitoring.
• Measuring and improving code quality, test coverage, etc.
• Using design patterns, architectural patterns, good programming abstractions.
• And many more aspects.

But to do these things properly, having proper architecture documentation is an important prerequisite.

Related work

UPDATE: after publishing this blog post and giving an internal presentation at Mendix about this subject, I received a couple of questions about architectural approaches that share similarities with my approach, such as the C4 model.

The C4 model also uses a layered approach in which the top-level diagram displays the context of the system (the relation of the system with external users and systems), and deeper layers gradually reveal more details of the inner workings of the system while limiting the view to a subset of components to prevent details obfuscating the purpose.

I did not use this approach as an example reference, but my work is basically built on top of the same underlying principles that the C4 model builds on -- creating abstractions.

Creating abstractions in modeling is popularized already in the 70s by various computer scientists, such as Edward Yourdon and Tom DeMarco, who bring concepts from structural programming to other domains, such as modeling (as explained in the paper: 'The Choice of New Software Development Methodologies for Software Development Projects' by Edward Yourdon).

One of the mental aids in structured programming is abstraction, so that it "only matters what something does, disregarding how it works". I took data flow modeling (DFD) as an example technique (that also facilitates layers of abstractions including a top-level context DFD), but I replaced the data flow notation by dependency modeling.

Furthermore, the C4 model also provides a number of diagram types for each abstraction layer with specific purposes, but in my approach the notation and purposes of each layer are left abstract.

My introduction to Mendix and low-code application development

As explained in my previous blog post, I started a new challenge two months ago and joined Mendix, a company that develops a low-code application development platform. Because I am receiving many questions about the platform and low-code application development in general, I have decided to write a blog post about it.

Low-code application development

With Mendix, the idea is that instead of writing (textual) code, applications are developed by modeling (and configuring) various kinds of application aspects, such as the data model and graphical user interfaces.

Why would that be useful you might ask? It makes you a developer more productive.

As many seasoned software developers may probably already know, developing applications is typically a costly investment and time consuming process. Aside from implementing all desired functionality and making it work correctly, a developer typically also needs to solve many kinds of secondary issues, such as:

• Memory management.
• Interacting with a database for storage.
• Pagination: Displaying a fixed amount of rows on a page for a collection of data
• Making screens work on various kinds of displays, e.g. monitors, tablets, phones.
• Deploying the application to a production environment so that it can be used by (possibly) a large number of end users.

For certain classes of systems, such as information systems, these secondary issues are frequently implemented over and over again, wasting precious development time and increasing the likelihood on errors.

With Mendix, most of these secondary concerns have been addressed in a reasonable manner, and primary functionality can be implemented by modeling and writing none to just a tiny amount of code, providing an incredible boost in developer productivity.

In addition to productivity, another objective of the Mendix platform is to reach a broader audience than just developers, such as business consultants.

A simple development scenario

What does the development process of a very tiny Mendix application look like? Typically, an application development process starts by creating a data model that specifies what kind of data needs to be stored (either in a database or in memory).

As shown in the picture above, in Mendix, a data model is created by drawing an ER (Entity-Relationship) diagram. For example, the example above defines a 'Contact' entity representing a contact person (with properties: first name and last name) and a 'Message' entity representing chat messages. A contact can send zero or more messages, as denoted by the 'Message_Contact' relationship.

After defining a data model, you may want to develop screens containing user interface components allowing users to inspect and change the data. In Mendix, these screens can be generated with just a few simple mouse clicks: creating a blank page, dragging and dropping a data grid to the page, and dragging and dropping the 'Contact' entity to the data grid:

In addition to an overview page displaying a collection of data items, we also want a page allowing us to change a record or to create a new record. This screen can be created by right clicking on the 'New' button and picking the 'Generate page...' option. The result is a page allowing us to edit an individual contact record:

Finally, to make the screen available to end users, we must add it to the navigation layout by creating a button that redirects the user to the overview page.

The result

With just a few simple clicks, we have already constructed a very small, but working application. For example, when clicking on the 'Run locally...' button in the IDE, we can start an instance of our test app and run it in a web browser.

The overview page looks as follows:

As may be observed by looking at the image above, the screen offers all functionality needed to work with a collection of records: navigation buttons, a search function, pagination, and edit functionality.

By clicking on an item in the list or clicking on the 'Edit' button, we can navigate to a page allowing us to edit the properties of a record:

After changing any of the attributes and clicking on the 'Save' button, the changes will be committed to the database.

Implementing all this functionality did not require me to write a single line of code.

Programming additional functionality

In addition to defining the domain model and generating views for all data elements, we may also want to program additional functionality, for example, a method that counts the amount of contacts having a first name that starts with an 'S'.

In Mendix, server-side functionality can be programmed by constructing microflows. Microflows use a graphical notation based on the Business Process Model and Notation (BPMN), a standardized graphical notation.

The above microflow retrieves all contacts from the database, sets an initial counter variable to 0 and iterates over all contacts. When a first name that starts with an 'S' has been encountered, the counter will be increased. Finally, it will return the counter value to the caller.

Microflows can be used to program many kinds of aspects in an application -- for example, they can be launched on startup, attached to buttons as event handlers, or attached as pre- or post commit hooks to data entities.

Other features

Besides the features described in this blog post, the Mendix platform has many additional features to offer, such as access control, internationalization, custom page layouts and collaboration tools (with a Subversion-based team server, and sprintr application managing the development workflow).

Furthermore, the deployment process of a Mendix application is completely automated in the Mendix cloud -- with just a few simple mouse clicks, your application becomes publicly available, without having to worry about managing a database or application server.

Finally, the platform is extensible -- you can implement custom actions in Java (server-side) and your own widgets in JavaScript (client-side) and there is an app store allowing you to download all kinds of pre-built third-party extensions.

Availability

You can sign up for a free Mendix account to start experimenting.

A new challenge

It has been quiet on my blog for a while, but don't worry, I'm still alive. In my absence, many things have happened. Most importantly, I have decided to embark on a new challenge.

Almost two months ago, I joined Mendix, a Rotterdam-based company (with its headquarters located in Boston) that ships a low-code application development platform. I will be working on their deployment infrastructure and I will be thinking about software modularity.

Some reflection

I vividly remember the days when I completed my PhD and left academia -- I still had to wait a couple of months for the defence ceremony, mainly because of the availability of the committee members.

Although I was quite happy with some of the work I had done during my research, I also wanted to leave my ivory tower and be more connected to the "real world" and my "research audience": developers. I joined a small startup company named Conference Compass (located in the YES!Delft incubator centre) that consisted of fewer than 10 people around the time I joined.

They had been looking into a setting up a product-line for their mobile conference apps, which sounded like an interesting challenge.

In the years that I was employed at Conference Compass, quite a few things happened. Most notably, the size of the company, the product and the service portfolio have grown considerably. Aside from these developments, I am particularly proud that I made quite a number of impacting open source contributions as part of my daily work.

The app building infrastructure

The biggest contribution I made by far is the mobile app building infrastructure. Most of its components have been part of Nixpkgs (the ecosystem of packages that can be deployed with the Nix package manager) for several years, such as:

• The Android app build environment that can deploy the Android SDK and its plugins, build Android apps from source code and spawn emulator instances.
• The iOS app build environment that can build iOS apps (for both a simulator or a real device) and spawn iOS simulator instances.
• A layer built around the two previous environments for building apps constructed with the Titanium app development framework.

To carry out all the builds on a large and timely scale, I installed a Hydra cluster: a Nix-based continuous integration service. I also developed an NPM module and command-line tool that we could use to remotely control a Hydra server from our custom built applications.

Chat functionality

Another interesting development area was enriching the app's product line with chat functionality built around the XMPP protocol/ejabberd service. I have ported the Simple XMPP library from the Node.js package ecosystem to Titanium by using a zero-forking strategy and I made a simple test application that somewhat resembles the Pidgin chat application.

I also ran into a number of practical issues while trying to keep the architecture of the test app clean. I did an in-depth study on the MVC paradigm and wrote a blog post about my findings.

Node.js

I also learned quite a lot about Node.js and its underlying concepts, such as asynchronous programming. Before joining Conference Compass, my JavaScript knowledge was limited to browser usage only, and I was not using JavaScript on a very extensive scale.

My learning experiences resulted in the following blog posts elaborating about various kinds of related concepts:

• Yet another blog post about Object Oriented Programming and JavaScript.
• Asynchronous programming with JavaScript.
• Asynchronous programming with JavaScript (part 2).
• Structured asynchronous programming.
• Integrating callback and promise based function invocation patterns.

As of today, some of these blog posts are still in my all-time top 10 of most frequently read blog posts.

As a result of having to work with Node.js and being involved with the Nix project, I became the maintainer of node2nix, a tool that can generate Nix expressions from NPM package configurations after the maintainer of npm2nix decided to hand over the project.

Building a service deployment platform

In the first two years of my employment, my chief responsibility was the app building infrastructure. Another thing I am particularly proud of is the deployment infrastructure for the service platform that I built from scratch, that grew from just a single virtual machine hosted in the Amazon EC2 cloud to a platform managing the data and configuration services for 100+ apps per year, with some events attracting tens of thousands of app users.

I used variety of solutions, such as various Amazon web services, e.g. EC2, Route 53, S3. Most importantly, I used NixOps for infrastructure deployment and Disnix (the tool I created as part of my research) for service deployment.

Although Disnix already supported all the features I needed before I actually started using it at Conference Compass, my company experiences helped to substantially improve Disnix from a usability perspective -- in academia, Disnix was mostly used to validate my research objectives. To make it suitable for using it in a company with non-specialized deployment people, you need iron out many additional issues. I added more helpful error messages, assistance in recovering from errors and additional utilities to make diagnosing problems and carrying out maintenance tasks more convenient.

At the end of 2015, after using Disnix for almost one year in production, I gave a talk about Disnix's deployment concepts at NixCon 2015, the first edition of a conference fully centered around Nix and its related technologies.

Conclusion

I am grateful to my previous employer: Conference Compass who gave me the opportunity to do all the things described in this blog post.

At Mendix, there will be many new interesting challenges for me -- I will be working with different kinds of technologies, a new platform and new people. Stay tuned, for more information...

Setting up a basic software configuration management process in a small organization

I have been responsible for many things in my past and current career. Besides research and development, I have also been responsible for software and systems configuration management in small/medium sized companies, such as my current employer.

I have observed that in organizations like these, configuration management (CM) is typically nobody's (full) responsibility. Preferably, people want to stick themselves to their primary responsibilities and typically carry out change activities in an ad-hoc and unstructured way.

Not properly implementing changes have a number of serious implications. For example, some problems I have encountered are:

• Delays. There are many factors that will unnecessarily increase the time it will take to implement a change. Many of my big delays were caused by the fact that I always have to search for all the relevant installation artifacts, such as documentation, installation discs, and so on. I have also encountered many times that artifacts were missing requiring me to obtain copies elsewhere.
• Errors. Any change could potentially lead to errors for many kinds of reasons. For example, implementing a set of changes in the wrong order could break a system. Also, components of which a system consist may have complex dependencies on other components that have to be met. Quite often, it is not fully clear what the dependencies of a component or system are, especially when documentation is incomplete or lacking.
  
  Moreover, after having solved an error, you need to remember many arbitrary things, such as workarounds, that tend to become forgotten knowledge over time.
  
• Disruptions. When implementing changes, a system may be partially or fully unavailable until all changes have been implemented. Preferably this time window should be as short as possible. Unfortunately, the inconsistency time window most likely becomes quite big when the configuration management process is not optimal or subject to errors.

It does not matter if an organization is small or big, but these problems cost valuable time and money. To alleviate these problems, it is IMO unavoidable to have a structured way of carrying out changes so that a system maintains its integrity.

Big organizations typically have information systems, people and management procedures to support structured configuration management, because failures are typically too costly for them. There are also standards available (such as the IEEE 828-2012 Standard for Configuration Management in Systems and Software Engineering) that they may use as a reference for implementing a configuration management process.

However, in small organizations people typically refrain from thinking about a process at all while they keep suffering from the consequences, because they think they are too small for it. Furthermore, they find it too costly to invest in people or an information system supporting configuration management procedures. Consulting a standard is something that is generally considered a leap too far.

In this blog post, I will describe a very basic software configuration management process I have implemented at my current employer.

The IEEE Standard for Configuration Management

As crazy as this may sound, I have used the IEEE 828-2012 standard as a reference for my implementation. The reason why I consulted this standard besides the fact that using an existing and reasonably well-established reference is good, is that I was already familiar with it, because of my previous background as a PhD researcher in software deployment.

The IEEE standard defines a framework of so-called "lower-level processes" from which a configuration management process can be derived. The most important lower-level processes are:

• CM identification, which concerns identifying, naming, describing, versioning, storing and retrieving configuration items (CIs) and their baselines.
• CM change control is about planning, requesting, approval and verification of change activities.
• CM status accounting is about identifying the status of CIs and change requests.
• CM auditing concerns identifying, tracing and reporting discrepancies with regards to the CIs and their baselines.
• CM release management is about planning, defining a format for distribution, delivery, and archival of configuration items.

All the other lower-level processes have references to the lower-level processes listed above. CM planning is basically about defining a plan how to carry out the above activities. CM management is about actually installing the tools involved, executing the process, monitoring its progress, and status and revising/optimizing the plan if any problem occurs.

The remaining two lower-level processes concern outside involvement -- Supplier configuration item control concerns CIs that are provided by external parties. Interface control concerns the implications of configuration changes that concern external parties. I did not take the traits of these lower-level processes into account in the implementation.

Implementing a configuration management process

Implementing a configuration management process (according to the IEEE standard) starts by developing configuration management plan. The standard mentions many planning aspects, such as identifying the information needs, reporting needs, the reporting frequency, and the information needed to manage CM activities. However, from my perspective, in a small organization many of these aspects are difficult to answer in advance, in particular the information needs.

As a rule of thumb, I think that when you do not exactly know what is needed, consider that the worst thing could happen -- the office burns down and everything gets lost. What does it take to reproduce the entire configuration from scratch?

This is how I have implemented the main lower-level processes:

CM identification

A configuration item is any structural unit that is distinguishable and configurable. In our situation, the most important kind of configuration item is a machine configuration (e.g. a physical machine or a virtual machine hosted in an IaaS environment, such as Amazon EC2), or a container configuration (such as an Apache Tomcat container or PaaS service, such as Elastic Beanstalk).

Machines/containers belong to an environment. Some examples of environments that we currently maintain are: the production environment containing the configurations of the production machines of our service, test contains the configurations of the test environment, and internal contains the configurations of our internal IT infrastructure, such as our internal Hydra build cluster, and other peripherals, such as routers and printers.

Machines run a number of applications that may have complex installation procedures. Moreover, identical/similar application configurations may have to be deployed to multiple machines.

For storing the configurations of the CIs, I have set up a Git repository that follows a specific directory structure of three levels:

<environment>/<machine | container>/<application>

Each directory contains all artifacts (e.g. keys, data files, configuration files, scripts, documents etc.) required to reproduce a CI's configuration from scratch. Moreover, each directory has a README.md markdown file:

• The top-level README.md describes which environments are available and what their purposes are.
• The environment-level README.md describes which machines/containers are part of it, a brief description of their purpose, and a picture showing how they are connected. I typically use Dia to draw them, because the tool is simple and free.
• The machine-level README.md describes the purpose of the machine and the activities that must be carried out to reproduce its configuration from scratch.
• The application-level README.md captures the steps that must be executed to reproduce an application's configuration.

When storing artifacts and writing README.md files, I try to avoid duplication as much as possible, because it makes it harder to keep the repository consistent and maintainable:

• README.md files are not supposed to be tool manuals. I mention the steps that must be executed and what their purposes are, but I avoid explaining how a tool works. That is the purpose of the tool's manual.
• When there are common files used among machines, applications or environments, I do not duplicate them. Instead, I use a _common/ folder that I put one directory level higher. For example, the _common/ folder in the internal/ directory contains shared artifacts that are supposed to reused among all machines belonging to our internal IT infrastructure.
• I also capture common configuration steps in a separate README.md and refer to it, instead of duplicating the same steps in multiple README.md files.

Because I use Git, versioning, storage and retrieval of configurations is implicit. I do not have to invent something myself or think too much about it. For example, I do not have to manually assign version numbers to CIs, because Git already computes them for each change. Moreover, because I use textual representations of most of the artifacts I can also easily compare versions of the configurations.

Furthermore, besides capturing and storing all the prerequisites to reproduce a configuration, I also try to automate this process as much as possible. For most of the automation aspects, I use tools from the Nix project, such as the Nix package manager for individual packages, NixOS for system configurations, Disnix for distributed services, and NixOps for networks of machines.

Tools in the Nix project are driven by declarative specifications -- a specification captures the structure of a system, e.g. their components and their dependencies. From this specification the entire deployment process will be derived, such as building the components from source code, distributing them to the right machines in the network, and activating them in the right order.

Using a declarative deployment approach prevents me writing down the activities to carry out, because they are implicit. Also, there is no need describing the structure of the system because it is already captured in the deployment specification.

Unfortunately, not all machine's deployment processes can be fully automated with Nix deployment tools (e.g. non-Linux machines and special purpose peripherals, such as routers) still requiring me to carry out some configuration activities manually.

CM change control

Implementing changes may cause disruptions costing time and money. That is why the right people must be informed and approval is needed. Big organizations typically have sophisticated management procedures including request and approval forms, but in a small organization it typically suffices to notify people informally before implementing a change.

Besides notifying people, I also take the following things into account while implementing changes:

• Some configuration changes require validation including review and testing before they can be actually implemented in production. I typically keep the master Git branch in a state releasable state, meaning that it is ready to be deployed into production. Any changes that require explicit validation go into a different branch first.
  
  Moreover, when using tools from the Nix project it is relatively easy to reliably test changes first by deploying a system into a test environment, or by spawning virtual NixOS machines in which integration tests can be executed.
• Sometimes you need to make an analysis of the impact and costs that a change would bring. Up-to-date and consistent documentation of the CIs including their dependencies makes this process more reliable.
  
  Furthermore, with tools from the Nix project you can also make better estimations by executing a dry-run deployment process -- the dry run shows what activities will be carried out without actually executing them or bringing the system down.
• After a change has been deployed, we also need to validate whether the new configuration is correct. Typically, this requires testing.
  
  Tools from the Nix project support congruent deployment, meaning that if the deployment has succeeded, the actual configuration is guaranteed to match the deployment specification for the static parts of a system, giving better guarantees about its validity.
  
• Also you have to pick the right moment to implement potentially disrupting changes. For example, it is typically a bad idea to do this while your service is under peak load.

CM status accounting

It is also highly desirable to know what the status of the CIs and the change activities are. The IEEE standard goes quite far in this. For example, the overall state of the system may converge into a certain direction (e.g. in terms of features complete, error ratios etc.), which you continuously want to measure and report about. I think that in a small organization these kinds of status indicators are typically too complex to define and measure, in particular in the early stages of a development process.

However, I think the most important status indicator that you never want to lose track of is the following: does everything (still) work?

There are two facilities that help me out a lot in keeping a system in working state:

• Automating deployment with tools from the Nix project ensure that the static parts of a deployed system are congruent with the deployment configuration specification and atomic -- either a deployment is in the old configuration or the new configuration but never in an inconsistent mix of the two. As a result, we have fewer broken configurations as a result of (re)deploying a system.
• We must also observe a system's runtime behavior and take action if things will grow out of hand. For example, when a machine runs out of system resources.
  
  Using a monitoring service, such as Zabbix or Datadog, helps me a lot in accomplishing this. They can also be used to configure alarms that warn you when things become critical.

CM auditing

Another important aspect is the integrity of the configurations repository. How can we be sure that what is stored inside the repository matches the actual configurations and that the configuration procedures still work?

Fortunately, because we use tools from the Nix project, there is relatively little audit work we need to do. With Nix-related tools the deployment process is fully automated. As a consequence, we need to adapt the deployment specification when we intend to make changes. Moreover, since the deployment specifications of Nix-related tools are congruent, we know that the static parts of a system are guaranteed to match the actual configurations if the (re)deployment process succeeded.

However, for non-NixOS machines and other peripherals, we must still manually check once in a while whether the indented configuration matches. I made it a habit to go through them once a month and to adjust the documentation if any discrepancies were found.

CM release management

When updating a configuration file, we must also release the new corresponding configuration items. The IEEE standard describes many concerns, such as approval procedures, requirements on the distribution mechanism and so on.

For me, most of these concerns are unimportant, especially in a small organization. The only thing that matters to me is that a release process is fully automated, reliable, reproducible. Fortunately, the deployment tools from the Nix project support these properties quite well.

Discussion

In this blog post, I have described a basic configuration management process that I have implemented in a small organization.

Some people will probably argue that defining a CM process in a small organization looks crazy. Some people think they do not need a process and that it is too much of an investment. Following an IEEE standard is generally considered a leap too far.

In my opinion, however, the barrier of implementing a CM process is not actually not that high. From my experience, the biggest investment is setting up a configuration management repository. Although big organizations typically have sophisticated information systems, I have also shown that using a simple filesystem layout and collection of free and open source tools (e.g. Git, Dia, Nix) a simple variant of such a repository can be set up with relatively little effort.

I also observed that automating CM tasks helps a lot, in particular using a declarative and congruent deployment approach, such as Nix. With a declarative approach, configuration activities are implicit (they are a consequence of applying a change in the deployment specification) and do not have to be documented. Furthermore, because Nix's deployment models are congruent, the static aspects of a configuration are guaranteed to match the deployment specifications. Moreover, the deployment model serves as the documentation, because it captures the structure of a system.

So how beneficial is setting up a CM process in a small organization? I observed many benefits. For example, a major benefit is that I can carry out many CM tasks much faster. I no longer have to waste much of my time looking for configuration artifacts and documentation. Also, because the steps to carry out are documented or automated, there are fewer things I need to (re)discover or solve while implementing a change.

Another benefit is that I can more reliably estimate the impact of implementing changes, because the CIs and their relationships are known. More knowledge, also causes fewer errors.

Although a simple CM approach provides benefits and many aspects can be automated, it always requires discipline from all people involved. For example, when errors are discovered and configurations must be modified in a stressful situation, it is very tempting to bypass updating the documentation.

Moreover, communication is also an important aspect. For example, when notifying people of a potentially disrupting change, clear communication is required. Typically, also non-technical stakeholders must be informed. Eventually, you have to start developing formalized procedures to properly handle decision processes.

Finally, the CM approach described in this blog post is obviously too limited if a company grows. If an organization gets bigger, a more sophisticated and more formalized CM approach will be required.