Thursday, February 28, 2019

Generating functional architecture documentation from Disnix service models

In my previous blog post, I have described a minimalistic architecture documentation approach for service-oriented systems based on my earlier experiences with setting up basic configuration management repositories. I used this approach to construct a documentation catalog for the platform I have been developing at Mendix.

I also explained my motivation -- it improves developer effectiveness, team consensus and the on-boarding of new team members. Moreover, it is a crucial ingredient in improving the quality of a system.

Although we are quite happy with the documentation, my biggest inconvenience is that I had to derive it entirely by hand -- I consulted various kinds of sources, but since existing documentation and information provided by people may be incomplete or inconsistent, I considered the source code and deployment configuration files the ultimate source of truth, because no matter how elegantly a diagram is drawn, it is useless if it does not match the actual implementation.

Because a manual documentation process is very costly and time consuming, a more ideal situation would be to have an automated approach that automatically derives architecture documentation from deployment specifications.

Since I am developing a deployment framework for service-oriented systems myself (Disnix), I have decided to extend it with a generator that can derive architecture diagrams and supplemental descriptions from the deployment models using the conventions I have described in my previous blog post.

Visualizing deployment architectures in Disnix


As explained in my previous blog post, the notation that I used for the diagrams was not something I invented from scratch, but something I borrowed from Disnix.

Disnix already has a feature (for quite some time) that can visualize deployment architectures referring to a description that shows how the functional parts (the services/components) are mapped to physical resources (e.g. machines/containers) in a network.

For example, after deploying a service-oriented system, such as my example web application system, by running:

$ disnix-env -s services.nix -i infrastructure.nix \
  -d distribution-bundles.nix

You can visualize the corresponding deployment architecture of the system, by running:

$ disnix-visualize > out.dot

The above command-line instruction generates a directed graph in the DOT language. The resulting dot file can be converted into a displayable image (such as a PNG or SVG file) by running:

$ dot -Tpng out.dot > out.png

Resulting in a diagram of the deployment architecture that may look as follows:


The above diagram uses the following notation:

  • The light grey boxes denote machines in a network. In the above deployment scenario, we have two them.
  • The ovals denote services (more specifically: in a Disnix-context, they reflect any kind of distributable deployment unit). Services can have almost any shape, such as web services, web applications, and databases. Disnix uses a plugin system called Dysnomia to make sure that the appropriate deployment steps are carried out for a particular type of service.
  • The arrows denote inter-dependencies. When a service points to another service means that the latter is an inter-dependency of the former service. Inter-dependency relationships ensure that the dependent service gets all configuration properties so that it knows how to reach the dependency and the deployment system makes sure that inter-dependencies of a specific service are deployed first.

    In some cases, enforcing the right activation order of activation may be expensive. It is also possible to drop the ordering requirement, as denoted by the dashed arrows. This is acceptable for redirects from the portal application, but not acceptable for database connections.
  • The dark grey boxes denote containers. Containers can be any kind of runtime environment that hosts zero or more distributable deployment units. For example, the container service of a MySQL database is a MySQL DBMS, whereas the container service of a Java web application archive can be a Java Servlet container, such as Apache Tomcat.

Visualizing the functional architecture of service-oriented systems


The services of which a service-oriented systems is composed are flexible -- they can be deployed to various kinds of environments, such a test environment, a second fail-over production environment or a local machine.

Because services can be deployed to a variety of targets, it may also be desired to get an architectural view of the functional parts only.

I created a new tool called: dydisnix-visualize-services that can be used to generate functional architecture diagrams by visualizing the services in the Disnix services model:


The above diagram is a visual representation of the services model of the example web application system, using a similar notation as the deployment architecture without showing any environment characteristics:

  • Ovals denote services and arrows denote inter-dependency relationships.
  • Every service is annotated with its type, so that it becomes clear what kind of a shape a service has and what kind of deployment procedures need to be carried out.

Despite the fact that the above diagram is focused on the functional parts, it may still look quite detailed, even from a functional point of view.

Essentially, the architecture of my example web application system is a "system of sub systems" -- each sub system provides an isolated piece of functionality consisting of a database backend and web application front-end bundle. The portal sub system is the entry point and responsible for guiding the users to the sub systems implementing the functionality that they want to use.

It is also possible to annotate services in the Disnix services model with a group and description property:

{distribution, invDistribution, pkgs, system}:

let
  customPkgs = import ../top-level/all-packages.nix {
    inherit pkgs system;
  };

  groups = {
    homework = "Homework";
    literature = "Literature";
    ...
  };
in
{
  homeworkdb = {
    name = "homeworkdb";
    pkg = customPkgs.homeworkdb;
    type = "mysql-database";
    group = groups.homework;
    description = "Database backend of the Homework subsystem";
  };

  homework = {
    name = "homework";
    pkg = customPkgs.homework;
    dependsOn = {
      inherit usersdb homeworkdb;
    };
    type = "apache-webapplication";
    appName = "Homework";
    group = groups.homework;
    description = "Front-end of the Homework subsystem";
  };

  ...
}

In the above services model, I have grouped every database and web application front-end bundle in a group that represents a sub system (such as Homework). By adding the --group-subservices parameter to the dydisnix-visualize-services command invocation, we can simplify the diagram to only show the sub systems and how these sub systems are inter-connected:

$ dydisnix-visualize-services -s services.nix -f png \
  --group-subservices

resulting in the following functional architecture diagram:


As may be observed in the picture above, all services have been grouped. The service groups are denoted by ovals with dashed borders.

We can also query sub architecture diagrams of every group/sub system. For example, the following command generates a sub architecture diagram for the Homework group:

$ dydisnix-visualize-services -s services.nix -f png \
  --group Homework --group-subservices

resulting in the following diagram:


The above diagram will only show the the services in the Homework group and their context -- i.e. non-transitive dependencies and services that have a dependency on any service in the requested group.

Services that exactly fit the group or any of its parent groups will be displayed verbatim (e.g. the homework database back-end and front-end). The other services will be categorized into in the lowest common sub group (the Users and Portal sub systems).

For more complex architectures consisting of many layers, you may probably want to generate all available architecture diagrams in one command invocation. It is also possible to run the visualization tool in batch mode. In batch mode, it will recursively generate diagrams for the top-level architecture and every possible sub group and stores them in a specified output folder:

$ dydisnix-visualize-services --batch -s services.nix -f svg \
  --output-dir out

Generating supplemental documentation


Another thing I have explained in my previous blog post is that providing diagrams is useful, but they cannot clear up all confusion -- you also need to document and clarify additional details, such as the purposes of the services.

It also possible to generate a documentation page for each group showing a table of services with their descriptions and types:

The following command generates a documentation page for the Homework group:

$ dydisnix-document-services -s services.nix --group Homework

It is also possible to adjust the generation process by providing a documentation configuration file (by using the --docs parameter):

$ dydisnix-document-services -f services.nix --docs docs.nix \
  --group Homework

The are a variety of settings that can be provided in a documentation configuration file:

{
  groups = {
    Homework = "Homework subsystem";
    Literature = "Literature subsystem";
    ...
  };

  fields = [ "description" "type" ];

  descriptions = {
    type = "Type";
    description = "Description";
  };
}

The above configuration file specifies the following properties:

  • The descriptions for every group.
  • Which fields should be displayed in the overview table. It is possible to display any property of a service.
  • A description of every field in the services model.

Like the visualization tool, the documentation tool can also be used in batch mode to generate pages for all possible groups and sub groups.

Generating a documentation catalog


In addition to generating architecture diagrams and descriptions, it is also possible to combine both tools to automatically generate a complete documentation catalog for a service-oriented system, such as the web application example system:

$ dydisnix-generate-services-docs -s services.nix --docs docs.nix \
  -f svg --output-dir out

By opening the entry page in the output folder, you will get an overview of the top-level architecture, with a description of the groups.


By clicking on a group hyperlink, you can inspect the sub architecture of the corresponding group, such as the 'Homework' sub system:


The above page displays the sub architecture diagram of the 'Homework' subsystem and a description of all services belonging to that group.

Another particularly interesting aspect is the 'Portal' sub system:


The portal's purpose is to redirect users to functionality provided by the other sub systems. The above architecture diagram displays all the sub systems in grouped form to illustrate that there is a dependency relationship, but without revealing all their internal details that clutters the diagram with unnecessary implementation details.

Other features


The tools support more use cases than those described in this blog post -- it is also possible, for example, to create arbitrary layers of sub groups by using the '/' character as a delimiter in the group identifier. I also used the company platform as an example case, that can be decomposed into four layers.

Availability


The tools described in this blog post are part of the latest development version of Dynamic Disnix -- a very experimental extension framework built on top of Disnix that can be used to make service-oriented systems self-adaptive by redeploying their services in case of events.

The reason why I have added these tools to Dynamic Disnix (and not the core Disnix toolset) is because the extension toolset has an infrastructure to parse and reflect over individual Disnix models.

Although I promised to make an official release of Dynamic Disnix a very long time ago, this still has not happened yet. However, the documentation feature is a compelling reason to stabilize the code and make the framework more usable.

Thursday, January 31, 2019

A minimalistic discovery and architecture documentation process

In a blog post written a couple of years ago, I have described how to set up a basic configuration management process in a small organization that is based on the process framework described in the IEEE 828-2012 configuration management standard. The most important prerequisite for setting up such a process is identifying all configuration items (CIs) and storing them in a well-organized repository.

There are many ways to organize configuration items ranging from simple to very sophisticated solutions. I used a very small set of free and open source tools, and a couple of simple conventions to set up a CI repository:

  • A Git repository with an hierarchical directory structure referring to configurations items. Each path component in the directory structure serves a specific purpose to group configuration items. The overall strategy was to use a directory structure with a maximum of three levels: environment/machine/application. Using Git makes it possible to version configuration items and share the repository with team members.
  • Using markdown to write down the purposes of the configuration items and descriptions how they can be reproduced. Markdown works well for two reasons: it can be nicely formatted in a browser, but also read from a terminal when logged in to remote servers via SSH.
  • Using Dia for drawing diagrams of systems consisting of more complex applications components. Dia is not the most elegant program around, but it works well enough, it is free and open source, and supported on Linux, Windows and macOS.

My main motivation to formalize configuration management (but only lightly), despite being in a small organization, is to prevent errors and minimize delays and disruptions while remaining flexible by not being bound to all kinds of complex management procedures.

I wrote this blog post while I was still employed at a small-sized startup company with only one development team. In the meantime, I have joined a much bigger organization (Mendix) that has many cross-disciplinary development teams that work concurrently on various aspects of our service and product portfolio.

About microservices


When I just joined, the amount of information I had to absorb was quite overwhelming. I also learned that we heavily adopted the microservices architecture paradigm for our entire online service platform.

According to Martin Fowler's blog post on microservices, using microservices offers the following benefits:

  • Strong module boundaries. You can divide the functionality of a system into microservices and make separate teams responsible for the development of each service. This makes it possible to iterate faster and offer better quality because teams can focus on themselves on a subset of features only.
  • Independent deployment. Microservices can be deployed independently making it possible to ship features when they are done, without having complex integration cycles.
  • Technology diversity. Microservices are language and technology agnostic. You can pick almost any programming language (e.g. Java, Python, Mendix, Go), data storage solution (e.g. PostgreSQL, MongoDB, InfluxDB) or operating system (e.g. Linux, FreeBSD, Windows) to implement a microservice making it possible pick the most suitable combination of technologies and use them at their full advantage.

However, decomposing a system into a collection of collaborating services also comes at a (sometimes substantial!) price:

  • There is typically much more operational complexity. Because there are many components and typically a large infrastructure to manage, activities such as deploying, upgrading, and monitoring the condition of a system is much more time consuming and complex. Furthermore, because of technology diversity, there are also many kinds of specialized deployment procedures that you need to carry out.
  • Data is eventually consistent. You have to live with the fact that (temporary) inconsistencies could end up in your data, and you must invest in implementing facilities that keep your data is consistent.
  • Because of distribution development is harder in general -- it is more difficult to diagnose errors (e.g. a failure in one service could trigger a chain reaction of errors, without having proper error traces), it is harder to test a system because of additional deployment complexity. The network links between services may be slow and subject to failure, causing all kinds of unpredictable problems. Also machines that host critical services may crash.

Studying the architecture


When applied properly, e.g. functionality is well separated and there is strong cohesion and weak coupling between services, while investing in solutions to cope with the challenges listed above, the benefits of microservices can be reaped, resulting in a scalable systems that can be developed my multiple teams working on features concurrently.

However, an important prerequisite for making changes in such an environment, and maintaining or improving the quality properties of a system, requires discipline and a relatively good understanding of the environment -- in the beginning, I faced all kinds of practical problems when I wanted to make even a subtle change -- some areas of our platform where documented, while others were not. Some documentation was also outdated, slightly incomplete and sometimes inconsistent with the actual implementation.

Certain areas of our platform were also highly complex resulting in very complex architectural views, with many boxes and arrows. Furthermore, information was also scattered around many different places.

As part of my on-boarding process, and as a means to cope with some of my practical problems, I have created a documentation repository of the platform that our team develops by extending the (minimalistic) principles for configuration management described in the earlier blog post.

I realized that simply identifying the service components of which the system consists, is not enough to get an understanding of the system -- there are many items and complex details that need to be captured.

In addition to the identification of all configuration items, I also want:

  • Proper guidance. To understand a particular piece of functionality, I should not need to study every component in detail. Instead, I want to know the full context and only the details of the relevant components.
  • Completeness. I want all service components to be visible. I do not want any details to be covered up. For example, I have also seen quite a few diagrams that hide complex implementation details. I much rather want flaws to be visible so that they can be resolved at a later point in time.
  • Clear boundaries. Our platform is not self contained, but relies on services provided by other teams. I want to know what components are our responsibility and what is managed by external teams.
  • Clarity. I want to know what the purpose of a component is. Their names may not always necessarily reflect or explain what they do.
  • Consistency. No matter how nicely a diagram is drawn, it should match the actual implementation or it is of very little use.
  • References to the actual implementation. I also want to know where I can find the implementation of a component, such as its Git repository.

Documenting the architecture


To visualize the architecture of our platform and organize all relevant information, I followed a strategy:

  • I took the components (typically their source code repositories) as the basis for everything else -- every component translates to a box in the architecture diagram.
  • I analyzed the dependency relationships between the components and denoted them as arrows. When a box points to another box by means of an arrow, this means that the other box is a dependency that should be deployed first. When a dependency is absent, the service will (most likely) not work.
  • I also discovered that the platform diagram easily gets cluttered by the sheer amount of components -- I decided to combine components that have very strongly correlated functionality in feature groups (that have dashed borders). Every feature group in architecture diagrams refers to another sub architecture diagram that provides a more specialized view of the feature group.
  • To clearly illustrate the difference between components that are our responsibility and those that are maintained by others teams, I make all external dependencies visible in the top-level architecture diagram.

The notation I used for these diagrams is not something I have entirely invented from scratch -- it is inspired by graph theory, package management and service management concepts. Disnix, for example, can visualize deployment architectures by using a similar notation.

To find all relevant information to create the diagrams, I consulted various sources:

  • I studied existing documents and diagrams to get a basic understanding of the system and an idea of the details I should look at.
  • I talked to a variety of people from various teams.
  • I looked inside the configuration settings of all deployment solutions used, e.g. the Amazon AWS console, Docker, CloudFoundry, Kubernetes, Nix configuration files.
  • Peek inside the source code repositories and look for settings that are references to other systems, such as configuration values that store URLs.
  • When I am in doubt: I consider the deployment configuration files and source code the "ultimate source of truth", because no matter how nice a diagram looks, it is useless if it is implemented differently.

Finally, just drawing diagrams will not completely suffice when the goal is provide clarity. I also observed that I need to document some leftover details.

Foremost, having a diagram without the semantics not explained will typically leave too many details open to interpretation to the user, so you need to explain the notation.

Second, you need to provide additional details about the services. I typically enumerate the following properties in a table for every component:

  • The name of the component.
  • A one line description stating its purpose.
  • The type of project (e.g. a Python/Java/Go project, Docker container, AWS Lambda function, etc.). This is useful to determine the kind of deployment procedure for the component.
  • A reference to the source code repository, e.g. a Git repository. The README of the corresponding repository should provide more detailed information about the project.

Benefits


Although it is quite a bit of work to set up, having a well documented architecture provides us the following benefits:

  • More effective deployment. Because of the feature groups and dividing the architecture into multiple layers, general concepts and details are separated. This makes it easier for developers to focus and absorb the right detailed knowledge to change a service.
  • More consensus in the team about the structure of the system and general quality attributes, such as scalability and security.
  • Better on-boarding for new team members.

Discussion


Writing architecture documentation IMO is not rocket science, just discipline. Obviously, there are much more sophisticated tools available to organize and visualize architectures (even tools that can generate code and reverse engineer code), but this is IMO not a hard requirement to start documenting.

However, you can not take all confusion away -- even if you have the best possible architecture documentation, people's thinking habits are shaped by the concepts they know and there will always be a slight mismatch (which is documented in academic research: 'Why Is It So Hard to Define Software Architecture?' written by Jason Baragry and Karl Reed).

Finally, architecture documentation is only a first good step to improve the quality of service-oriented systems. To make it a success, much more is needed, such as:

  • Automated (and reproducible) deployment processes.
  • More documentation (such as the APIs, end-user documentation, project documentation).
  • Automated unit, integration and acceptance testing.
  • Monitoring.
  • Measuring and improving code quality, test coverage, etc.
  • Using design patterns, architectural patterns, good programming abstractions.
  • And many more aspects.

But to do these things properly, having proper architecture documentation is an important prerequisite.

Related work


UPDATE: after publishing this blog post and giving an internal presentation at Mendix about this subject, I received a couple of questions about architectural approaches that share similarities with my approach, such as the C4 model.

The C4 model also uses a layered approach in which the top-level diagram displays the context of the system (the relation of the system with external users and systems), and deeper layers gradually reveal more details of the inner workings of the system while limiting the view to a subset of components to prevent details obfuscating the purpose.

I did not use this approach as an example reference, but my work is basically built on top of the same underlying principles that the C4 model builds on -- creating abstractions.

Creating abstractions in modeling is popularized already in the 70s by various computer scientists, such as Edward Yourdon and Tom DeMarco, who bring concepts from structural programming to other domains, such as modeling (as explained in the paper: 'The Choice of New Software Development Methodologies for Software Development Projects' by Edward Yourdon).

One of the mental aids in structured programming is abstraction, so that it "only matters what something does, disregarding how it works". I took data flow modeling (DFD) as an example technique (that also facilitates layers of abstractions including a top-level context DFD), but I replaced the data flow notation by dependency modeling.

Furthermore, the C4 model also provides a number of diagram types for each abstraction layer with specific purposes, but in my approach the notation and purposes of each layer are left abstract.