Monday, May 27, 2019

A Nix-friendly XML-based data exchange library for the Disnix toolset

In the last few months, I have been intensively working on a variety of internal improvements to the Disnix toolset.

One of the more increasingly complex and tedious aspects in the Disnix toolset is data exchange -- Disnix implements declarative deployment in the sense that it takes three specifications written in the Nix expression language as inputs: a services model that specifies the deployable units, their properties and how they depend on each other, an infrastructure model specifies the available target machines and their properties, and a distribution model that specifies the mapping between services in the services models and target machines in the infrastructure model.

From these three declarative models, Disnix derives all the activities that need to be carried out to get a system in a running state: building services from source, distributing services (and their dependencies) to target machines, activating the services, and (optionally) restoring state snapshots.

Using the Nix expression language for these input models is useful for a variety of reasons:

  • We can use the Nix package manager's build infrastructure to reliably build services from source code, including all their required dependencies, and store them in isolation in the Nix store. The Nix store ensures that multiple variants and versions can co-exist and that we can always roll back to previous versions.
  • Because the Nix expression language is a purely functional domain-specific language (that in addition to data structures supports functions), we can make all required configuration parameters (such as the dependencies of the services that we intend to deploy) explicit by using functions so that we know that all mandatory settings have been specified.

Although the Nix expression language is a first class-citizen concept for tasks carried out by the Nix package manager, we also want to use the same specifications to instruct tools that carry out activities that Nix does not implement, such as the tools that activate the services and restore state snapshots.

The Nix expression language is not designed to be consumed by other tools than Nix (as a sidenote: despite this limitation, it is still somewhat possible to use the Nix expression language independently of the package manager in experimental setups, such as this online tutorial, but the libexpr component of Nix does not have a stable interface or commitment to make the language portable across tools).

As a solution, I convert objects in the Nix expression language to XML, so that they can be consumed by any of the tools that implement the activities that Nix does not support.

Although this may sound conceptually straight forward, the amount of data that needs to be converted, and code that needs to be written to parse that data is growing bigger and more complex, and becomes increasingly harder to adjust and maintain.

To cope with this growing complexity, I have standardized a collection of Nix-XML conversion patterns, and wrote a library named: libnixxml that can be used to make data interchange in both directions more convenient.

Converting objects in the Nix expression language to XML


The Nix expression language supports a variety of language integrations. For example, it can export Nix objects to XML and JSON, and import from JSON and TOML data.

The following Nix attribute set:
{
  message = "This is a test";
  tags = [ "test" "example" ];
}

can be converted to XML (with the builtins.toXML primop) or by running:

$ nix-instantiate --eval-only --xml --strict example.nix

resulting in the following XML data:

<?xml version='1.0' encoding='utf-8'?>
<expr>
  <attrs>
    <attr column="3" line="2" name="message" path="/home/sander/example.nix">
      <string value="This is a test" />
    </attr>
    <attr column="3" line="3" name="tags" path="/home/sander/example.nix">
      <list>
        <string value="test" />
        <string value="example" />
      </list>
    </attr>
  </attrs>
</expr>

Although the above XML code fragment is valid XML, it is basically also just a literal translation of the underlying abstract syntax tree (AST) to XML.

An AST dump is not always very practical for consumption by an external application -- it is not very "readable", contains data that we do not always need (e.g. line and column data), and imposes (due to the structure) additional complexity on a program to parse the XML data to a domain model. As a result, exported XML data almost always needs to be converted to an XML format that is more practical for consumption.

For all the input models that Disnix consumes, I was originally handwriting XSL stylesheets converting the XML data to a format that can be more easily consumed and handwriting all the parsing code. Eventually, I derived a number of standard patterns.

For example, a more practical XML representation of the earlier shown Nix expression could be:

<?xml version="1.0"?>
<expr>
  <message>This is a test</message>
  <tags>
    <elem>test</elem>
    <elem>example</elem>
  </tags>
</expr>

In the above expression, the type and meta information is discarded. The attribute set is translated to a collection of XML sub elements in which the element names correspond to the attribute keys. The list elements are translated to generic sub elements (the above example uses elem, but any element name can be picked). The above notation is IMO, more readable, more concise and easier to parse by an external program.

Attribute keys may be identifiers, but can also be strings containing characters that invalidate certain XML element names (e.g. < or >). It is also possible to use a slightly more verbose notation in which a generic element name is used and the name property is used for each attribute set key:

<?xml version="1.0"?>
<expr>
  <attr name="message">This is a test</attr>
  <attr name="tags">
    <elem>test</elem>
    <elem>example</elem>
  </attr>
</expr>

When an application has a static domain model, it is not necessary to know any types (e.g. this conversion can be done in the application code using the application domain model). However, it may also be desired to construct data structures dynamically.

For dynamic object construction, type information needs to be known. Optionally, XML elements can be annotated with type information:

<?xml version="1.0"?>
<expr type="attrs">
  <attr name="message" type="string">This is a test</attr>
  <attr name="tags" type="list">
    <elem type="string">test</elem>
    <elem type="string">example</elem>
  </attr>
</expr>

To automatically convert data to XML format following the above listed conventions, I have created a standardized XSL stylesheet and command-line tool that can automatically convert Nix expressions.

The following command generates the first XML code fragment:

$ nixexpr2xml --attr-style simple example.nix

We can use the verbose notation for attribute sets, by running:

$ nixexpr2xml --attr-style verbose example.nix

Type annotations can be enabled by running:

$ nixexpr2xml --attr-style verbose --enable-types example.nix

The root, attribute and list element representations as well as the attribute set and types properties use generic element and property names. Their names can also be adjusted, if desired:

$ nixexpr2xml --root-element-name root \
  --list-element-name item \
  --attr-element-name property \
  --name-attribute-name key \
  --type-attribute-name mytype
  example.nix

Parsing a domain model


In addition to producing more "practical" XML data, I have also implemented utility functions that help me consuming the XML data to construct a domain model in the C programming language, that consists values (strings, integers etc.), structs, list-like data structures (e.g. arrays, linked lists) and table-like data structures, such as hash tables.

For example, the following XML document only containing a string:

<expr>hello</expr>

can be parsed to a string in C as follows:

#include <nixxml-parse.h>

xmlNodePtr element;
/* Open XML file and obtain root element */
xmlChar *value = NixXML_parse_value(element, NULL);
printf("value is: %s\n"); // value is: hello

We can also use functions to parse (nested) data structures. For example, to parse the following XML code fragment representing an attribute set:

<expr>
  <attr name="firstName">Sander</attr>
  <attr name="lastName">van der Burg</attr>
</expr>

We can use the following code snippet:

#include <stdlib.h>
#include <nixxml-parse.h>

xmlNodePtr element;

typedef struct
{
    xmlChar *firstName;
    xmlChar *lastName;
}
ExampleStruct;

void *create_example_struct(xmlNodePtr element, void *userdata)
{
    return calloc(1, sizeof(ExampleStruct));
}

void parse_and_insert_example_struct_member(xmlNodePtr element, void *table, const xmlChar *key, void *userdata)
{
    ExampleStruct *example = (ExampleStruct*)table;

    if(xmlStrcmp(key, (xmlChar*) "firstName") == 0)
        example->firstName = NixXML_parse_value(element, userdata);
    else if(xmlStrcmp(key, (xmlChar*) "lastName") == 0)
        example->lastName = NixXML_parse_value(element, userdata);
}

/* Open XML file and obtain root element */

ExampleStruct *example = NixXML_parse_verbose_heterogeneous_attrset(element, "attr", "name", NULL, create_example_struct, parse_and_insert_example_struct_member);

To parse the attribute set in the XML code fragment above (that uses a verbose notation) and derive a struct from it, we invoke the NixXML_parse_verbose_heterogeneous_attrset() function. The parameters specify that the XML code fragment should be parsed as follows:

  • It expects the name of the XML element of each attribute to be called: attr.
  • The property that refers to the name of the attribute is called: name.
  • To create a struct that stores the attributes in the XML file, the function: create_example_struct() will be executed that allocates memory for it and initializes all fields with NULL values.
  • The logic that parses the attribute values and assigns them to the struct members is in the parse_and_insert_example_member() function. The implementation uses NixXML_parse_value() (as shown in the previous example) to parse the attribute values.

In addition to parsing values as strings and attribute sets as structs, it is also possible to:

  • Parse lists, by invoking: NixXML_parse_list()
  • Parse uniformly typed attribute sets (in which every attribute set member has the same type), by invoking: NixXML_parse_verbose_attrset()
  • Parse attribute sets using the simple XML notation for attribute sets (as opposed to the verbose notation): NixXML_parse_simple_attrset() and NixXML_parse_simple_heterogeneous_attrset()

Printing Nix or XML representation of a domain model


In addition to parsing NixXML data to construct a domain model, the inverse process is also possible -- the API also provides convenience functions to print an XML or Nix representation of a domain model.

For example, the following string in C:

char *greeting = "Hello";

can be displayed as a string in the Nix expression language as follows:

#include <nixxml-print-nix.h>

NixXML_print_string_nix(stdout, greeting, 0, NULL); // outputs: "Hello"

or as an XML document, by running:

#include <nixxml-print-xml.h>

NixXML_print_open_root_tag(stdout, "expr");
NixXML_print_string_xml(stdout, greeting, 0, NULL, NULL);
NixXML_print_close_root_tag(stdout, "expr");

producing the following output:

<expr>Hello</expr>

The example struct shown in the previous section can be printed as a Nix expression with the following code:

#include <nixxml-print-nix.h>

void print_example_attributes_nix(FILE *file, const void *value, const int indent_level, void *userdata, NixXML_PrintValueFunc print_value)
{
    ExampleStruct *example = (ExampleStruct*)value;
    NixXML_print_attribute_nix(file, "firstName", example->firstName, indent_level, userdata, NixXML_print_string_nix);
    NixXML_print_attribute_nix(file, "lastName", example->lastName, indent_level, userdata, NixXML_print_string_nix);
}

NixXML_print_attrset_nix(stdout, &example, 0, NULL, print_example_attributes_nix, NULL);

The above code fragment executes the function: NixXML_print_attrset_nix() to print the example struct as an attribute set. The attribute set printing function invokes the function: print_example_attributes_nix() to print the attribute set members.

The print_example_attributes_nix() function prints each attribute assignment. It uses the NixXML_print_string_nix() function (shown in the previous example) to print each member as a string in the Nix expression language.

The result of running the above code is the following Nix expression:

{
  "firstName" = "Sander";
  "lastName" = "van der Burg";
}

the same struct can be printed as XML (using the verbose notation for attribute sets) with the following code:

#include <nixxml-print-xml.h>

void print_example_attributes_xml(FILE *file, const void *value, const char *child_element_name, const char *name_property_name, const int indent_level, const char *type_property_name, void *userdata, NixXML_PrintXMLValueFunc print_value)
{
    ExampleStruct *example = (ExampleStruct*)value;
    NixXML_print_verbose_attribute_xml(file, child_element_name, name_property_name, "firstName", example->firstName, indent_level, NULL, userdata, NixXML_print_string_xml);
    NixXML_print_verbose_attribute_xml(file, child_element_name, name_property_name, "lastName", example->lastName, indent_level, NULL, userdata, NixXML_print_string_xml);
}

NixXML_print_open_root_tag(stdout, "expr");
NixXML_print_verbose_attrset_xml(stdout, &example, "attr", "name", 0, NULL, NULL, print_example_attributes_xml, NULL);
NixXML_print_close_root_tag(stdout, "expr");

The above code fragment uses a similar strategy as the previous example (by invoking NixXML_print_verbose_attrset_xml()) to print the example struct as an XML file using a verbose notation for attribute sets.

The attribute set members are printed by the print_example_attributes_xml() function.

The result of running the above code is the following XML output:

<expr>
  <attr name="firstName">Sander</attr>
  <attr name="lastName">van der Burg</attr>
</expr>

In addition to printing values and attribute sets, it is also possible to:

  • Print lists in Nix and XML format: NixXML_print_list_nix(), NixXML_print_list_xml()
  • Print attribute sets in simple XML notation: NixXML_print_simple_attrset_xml()
  • Print strings as int, float or bool: NixXML_print_string_as_*_xml.
  • Print integers: NixXML_print_int_xml()
  • Disable indentation by setting the indent_level parameter to -1.
  • Print type annotated XML, by setting the type_property_name parameter to a string that is not NULL.

Using abstract data structures


There is no standardized library for abstract data structures in C, e.g. lists, maps, trees etc. As a result, each framework provides their own implementations of them. To parse lists and attribute sets (that have arbitrary structures), you need generalized data structures that are list-like or table-like.

libnixxml provides two sub libraries to demonstrate how integration with abstract data structures can be implemented. One sub library is called libnixxml-data that uses pointer arrays for lists and xmlHashTable for attribute sets, and another is called libnixxml-glib that integrates with GLib using GPtrArray structs for lists and GHashTables for attribute sets.

The following XML document:

<expr>
  <elem>test</elem>
  <elem>example</elem>
</expr@>

can be parsed as a pointer array (array of strings) as follows:

#include <nixxml-ptrarray.h>

xmlNodePtr element;
/* Open XML file and obtain root element */
void **array = NixXML_parse_ptr_array(element, "elem", NULL, NixXML_parse_value);

and printed as a Nix expression with:

NixXML_print_ptr_array_nix(stdout, array, 0, NULL, NixXML_print_string_nix);

and as XML with:

NixXML_print_open_root_tag(stdout, "expr");
NixXML_print_ptr_array_xml(stdout, array, "elem", 0, NULL, NULL, NixXML_print_string_xml);
NixXML_print_close_root_tag(stdout, "expr");

Similarly, there is a module that works with xmlHashTables providing a similar function interface as the pointer array module.

Working with generic NixXML nodes


By using generic data structures to represent lists and tables, type annotated NixXML data and a generic NixXML_Node struct (that indicates what kind of node we have, such as a value, list or attribute set) we can also automatically parse an entire document by using a single function call:

#include <nixxml-ptrarray.h>
#include <nixxml-xmlhashtable.h>
#include <nixxml-parse-generic.h>

xmlNodePtr element;
/* Open XML file and obtain root element */
NixXML_Node *node = NixXML_generic_parse_expr(element,
    "type",
    "name",
    NixXML_create_ptr_array,
    NixXML_create_xml_hash_table,
    NixXML_add_value_to_ptr_array,
    NixXML_insert_into_xml_hash_table,
    NixXML_finalize_ptr_array);

The above function composes a generic NixXML_Node object. The function interface uses function pointers to compose lists and tables. These functions are provided by the pointer array and xmlHashTable modules in the libnixxml-data library.

We can also print an entire NixXML_Node object structure as a Nix expression:

#include <nixxml-print-generic-nix.h>

NixXML_print_generic_expr_nix(stdout,
    node,
    0,
    NixXML_print_ptr_array_nix,
    NixXML_print_xml_hash_table_nix);

as well as XML (using simple or verbose notation for attribute sets):

#include <nixxml-print-generic-xml.h>

NixXML_print_generic_expr_verbose_xml(stdout,
    node,
    0,
    "expr",
    "elem",
    "attr",
    "name",
    "type",
    NixXML_print_ptr_array_xml,
    NixXML_print_xml_hash_table_verbose_xml);

Summary


The following table summarizes the concepts described in this blog post:

Concept Nix expression representation XML representation C application domain model
value "hello" hello char*
list [ "hello" "bye" ] <elem>hello</elem><elem>bye</elem> void**, linked list, ...
attribute set { a = "hello"; b = "bye"; } <a>hello</a><b>bye</b> xmlHashTablePtr, struct, ...
attribute set { a = "hello"; b = "bye"; } <attr name="a">hello</attr><attr name="b">bye</attr> xmlHashTablePtr, struct, ...

The above table shows the concepts that the NixXML defines, and how they can be represented in the Nix expression language, XML and in a domain model of a C application.

The representations of these concepts can be translated as follows:

  • To convert a raw AST XML representation of a Nix expression to NixXML, we can use the included XSL stylesheet or run the nixexpr2xml command.
  • XML concepts can be parsed to a domain model in a C application by invoking NixXML_parse_* functions for the appropriate concepts and XML representation.
  • Domain model elements can be printed as XML by invoking NixXML_print_*_xml functions.
  • Domain model elements can be printed in the Nix expression language by invoking NixXML_print_*_nix functions.

Benefits


I have re-engineered the current development versions of Disnix and the Dynamic Disnix toolsets to use libnixxml for data exchange. For Disnix, there is much fewer boilerplate code that I need to write for the parsing infrastructure, making it significantly easier to maintain it.

In the Dynamic Disnix framework, libnixxml provides even more benefits beyond a simpler parsing infrastructure. The Dynamic Disnix toolset provides deployment planning methods, and documentation and visualization tools. These concerns are orthogonal to the features of the core Disnix toolset -- there is first-class Nix/Disnix integration, but the features of Dynamic Disnix should work with any service-oriented system (having a model that works with services and dependencies) regardless of what technology is used to carry out the deployment process itself.

With libnixxml it is now quite easy to make all these tools both accept Nix and XML representations of their input models, and make them output data in both Nix and XML. It is now also possible to use most features of Dynamic Disnix, such as the visualization features described in the previous blog post, independently of Nix and Disnix.

Moreover, the deployment planning methods should now also be able to more conveniently invoke external tools, such as SAT-solvers.

Related work


libnixxml is not the only Nix language integration facility I wrote. I also wrote NiJS (that is JavaScript-based) and PNDP (that is PHP-based). Aside from the language (C programming language), the purpose of libnixxml is not to replicate the functionality of these two libraries in C.

Basically, libnixxml has the inverse purpose -- NiJS and PNDP are useful for systems that already have a domain model (e.g. a domain-specific configuration management tool), and make it possible to generate the required Nix expression language code to conveniently integrate with Nix.

In libnixxml, the Nix expression representation is the basis and libnixxml makes it more convenient for external programs to consume such a Nix expression. Moreover, libnixxml only facilitates data interchange, and not all Nix expression language features.

Conclusion


In this blog post, I have described libnixxml that makes XML-based data interchange with configurations in the Nix expression language and domain models in the C programming language more convenient. It is part of the current development version of Disnix and can be obtained as a separate project from my GitHub page.

Thursday, February 28, 2019

Generating functional architecture documentation from Disnix service models

In my previous blog post, I have described a minimalistic architecture documentation approach for service-oriented systems based on my earlier experiences with setting up basic configuration management repositories. I used this approach to construct a documentation catalog for the platform I have been developing at Mendix.

I also explained my motivation -- it improves developer effectiveness, team consensus and the on-boarding of new team members. Moreover, it is a crucial ingredient in improving the quality of a system.

Although we are quite happy with the documentation, my biggest inconvenience is that I had to derive it entirely by hand -- I consulted various kinds of sources, but since existing documentation and information provided by people may be incomplete or inconsistent, I considered the source code and deployment configuration files the ultimate source of truth, because no matter how elegantly a diagram is drawn, it is useless if it does not match the actual implementation.

Because a manual documentation process is very costly and time consuming, a more ideal situation would be to have an automated approach that automatically derives architecture documentation from deployment specifications.

Since I am developing a deployment framework for service-oriented systems myself (Disnix), I have decided to extend it with a generator that can derive architecture diagrams and supplemental descriptions from the deployment models using the conventions I have described in my previous blog post.

Visualizing deployment architectures in Disnix


As explained in my previous blog post, the notation that I used for the diagrams was not something I invented from scratch, but something I borrowed from Disnix.

Disnix already has a feature (for quite some time) that can visualize deployment architectures referring to a description that shows how the functional parts (the services/components) are mapped to physical resources (e.g. machines/containers) in a network.

For example, after deploying a service-oriented system, such as my example web application system, by running:

$ disnix-env -s services.nix -i infrastructure.nix \
  -d distribution-bundles.nix

You can visualize the corresponding deployment architecture of the system, by running:

$ disnix-visualize > out.dot

The above command-line instruction generates a directed graph in the DOT language. The resulting dot file can be converted into a displayable image (such as a PNG or SVG file) by running:

$ dot -Tpng out.dot > out.png

Resulting in a diagram of the deployment architecture that may look as follows:


The above diagram uses the following notation:

  • The light grey boxes denote machines in a network. In the above deployment scenario, we have two them.
  • The ovals denote services (more specifically: in a Disnix-context, they reflect any kind of distributable deployment unit). Services can have almost any shape, such as web services, web applications, and databases. Disnix uses a plugin system called Dysnomia to make sure that the appropriate deployment steps are carried out for a particular type of service.
  • The arrows denote inter-dependencies. When a service points to another service means that the latter is an inter-dependency of the former service. Inter-dependency relationships ensure that the dependent service gets all configuration properties so that it knows how to reach the dependency and the deployment system makes sure that inter-dependencies of a specific service are deployed first.

    In some cases, enforcing the right activation order of activation may be expensive. It is also possible to drop the ordering requirement, as denoted by the dashed arrows. This is acceptable for redirects from the portal application, but not acceptable for database connections.
  • The dark grey boxes denote containers. Containers can be any kind of runtime environment that hosts zero or more distributable deployment units. For example, the container service of a MySQL database is a MySQL DBMS, whereas the container service of a Java web application archive can be a Java Servlet container, such as Apache Tomcat.

Visualizing the functional architecture of service-oriented systems


The services of which a service-oriented systems is composed are flexible -- they can be deployed to various kinds of environments, such a test environment, a second fail-over production environment or a local machine.

Because services can be deployed to a variety of targets, it may also be desired to get an architectural view of the functional parts only.

I created a new tool called: dydisnix-visualize-services that can be used to generate functional architecture diagrams by visualizing the services in the Disnix services model:


The above diagram is a visual representation of the services model of the example web application system, using a similar notation as the deployment architecture without showing any environment characteristics:

  • Ovals denote services and arrows denote inter-dependency relationships.
  • Every service is annotated with its type, so that it becomes clear what kind of a shape a service has and what kind of deployment procedures need to be carried out.

Despite the fact that the above diagram is focused on the functional parts, it may still look quite detailed, even from a functional point of view.

Essentially, the architecture of my example web application system is a "system of sub systems" -- each sub system provides an isolated piece of functionality consisting of a database backend and web application front-end bundle. The portal sub system is the entry point and responsible for guiding the users to the sub systems implementing the functionality that they want to use.

It is also possible to annotate services in the Disnix services model with a group and description property:

{distribution, invDistribution, pkgs, system}:

let
  customPkgs = import ../top-level/all-packages.nix {
    inherit pkgs system;
  };

  groups = {
    homework = "Homework";
    literature = "Literature";
    ...
  };
in
{
  homeworkdb = {
    name = "homeworkdb";
    pkg = customPkgs.homeworkdb;
    type = "mysql-database";
    group = groups.homework;
    description = "Database backend of the Homework subsystem";
  };

  homework = {
    name = "homework";
    pkg = customPkgs.homework;
    dependsOn = {
      inherit usersdb homeworkdb;
    };
    type = "apache-webapplication";
    appName = "Homework";
    group = groups.homework;
    description = "Front-end of the Homework subsystem";
  };

  ...
}

In the above services model, I have grouped every database and web application front-end bundle in a group that represents a sub system (such as Homework). By adding the --group-subservices parameter to the dydisnix-visualize-services command invocation, we can simplify the diagram to only show the sub systems and how these sub systems are inter-connected:

$ dydisnix-visualize-services -s services.nix -f png \
  --group-subservices

resulting in the following functional architecture diagram:


As may be observed in the picture above, all services have been grouped. The service groups are denoted by ovals with dashed borders.

We can also query sub architecture diagrams of every group/sub system. For example, the following command generates a sub architecture diagram for the Homework group:

$ dydisnix-visualize-services -s services.nix -f png \
  --group Homework --group-subservices

resulting in the following diagram:


The above diagram will only show the the services in the Homework group and their context -- i.e. non-transitive dependencies and services that have a dependency on any service in the requested group.

Services that exactly fit the group or any of its parent groups will be displayed verbatim (e.g. the homework database back-end and front-end). The other services will be categorized into in the lowest common sub group (the Users and Portal sub systems).

For more complex architectures consisting of many layers, you may probably want to generate all available architecture diagrams in one command invocation. It is also possible to run the visualization tool in batch mode. In batch mode, it will recursively generate diagrams for the top-level architecture and every possible sub group and stores them in a specified output folder:

$ dydisnix-visualize-services --batch -s services.nix -f svg \
  --output-dir out

Generating supplemental documentation


Another thing I have explained in my previous blog post is that providing diagrams is useful, but they cannot clear up all confusion -- you also need to document and clarify additional details, such as the purposes of the services.

It also possible to generate a documentation page for each group showing a table of services with their descriptions and types:

The following command generates a documentation page for the Homework group:

$ dydisnix-document-services -s services.nix --group Homework

It is also possible to adjust the generation process by providing a documentation configuration file (by using the --docs parameter):

$ dydisnix-document-services -f services.nix --docs docs.nix \
  --group Homework

The are a variety of settings that can be provided in a documentation configuration file:

{
  groups = {
    Homework = "Homework subsystem";
    Literature = "Literature subsystem";
    ...
  };

  fields = [ "description" "type" ];

  descriptions = {
    type = "Type";
    description = "Description";
  };
}

The above configuration file specifies the following properties:

  • The descriptions for every group.
  • Which fields should be displayed in the overview table. It is possible to display any property of a service.
  • A description of every field in the services model.

Like the visualization tool, the documentation tool can also be used in batch mode to generate pages for all possible groups and sub groups.

Generating a documentation catalog


In addition to generating architecture diagrams and descriptions, it is also possible to combine both tools to automatically generate a complete documentation catalog for a service-oriented system, such as the web application example system:

$ dydisnix-generate-services-docs -s services.nix --docs docs.nix \
  -f svg --output-dir out

By opening the entry page in the output folder, you will get an overview of the top-level architecture, with a description of the groups.


By clicking on a group hyperlink, you can inspect the sub architecture of the corresponding group, such as the 'Homework' sub system:


The above page displays the sub architecture diagram of the 'Homework' subsystem and a description of all services belonging to that group.

Another particularly interesting aspect is the 'Portal' sub system:


The portal's purpose is to redirect users to functionality provided by the other sub systems. The above architecture diagram displays all the sub systems in grouped form to illustrate that there is a dependency relationship, but without revealing all their internal details that clutters the diagram with unnecessary implementation details.

Other features


The tools support more use cases than those described in this blog post -- it is also possible, for example, to create arbitrary layers of sub groups by using the '/' character as a delimiter in the group identifier. I also used the company platform as an example case, that can be decomposed into four layers.

Availability


The tools described in this blog post are part of the latest development version of Dynamic Disnix -- a very experimental extension framework built on top of Disnix that can be used to make service-oriented systems self-adaptive by redeploying their services in case of events.

The reason why I have added these tools to Dynamic Disnix (and not the core Disnix toolset) is because the extension toolset has an infrastructure to parse and reflect over individual Disnix models.

Although I promised to make an official release of Dynamic Disnix a very long time ago, this still has not happened yet. However, the documentation feature is a compelling reason to stabilize the code and make the framework more usable.

Thursday, January 31, 2019

A minimalistic discovery and architecture documentation process

In a blog post written a couple of years ago, I have described how to set up a basic configuration management process in a small organization that is based on the process framework described in the IEEE 828-2012 configuration management standard. The most important prerequisite for setting up such a process is identifying all configuration items (CIs) and storing them in a well-organized repository.

There are many ways to organize configuration items ranging from simple to very sophisticated solutions. I used a very small set of free and open source tools, and a couple of simple conventions to set up a CI repository:

  • A Git repository with an hierarchical directory structure referring to configurations items. Each path component in the directory structure serves a specific purpose to group configuration items. The overall strategy was to use a directory structure with a maximum of three levels: environment/machine/application. Using Git makes it possible to version configuration items and share the repository with team members.
  • Using markdown to write down the purposes of the configuration items and descriptions how they can be reproduced. Markdown works well for two reasons: it can be nicely formatted in a browser, but also read from a terminal when logged in to remote servers via SSH.
  • Using Dia for drawing diagrams of systems consisting of more complex applications components. Dia is not the most elegant program around, but it works well enough, it is free and open source, and supported on Linux, Windows and macOS.

My main motivation to formalize configuration management (but only lightly), despite being in a small organization, is to prevent errors and minimize delays and disruptions while remaining flexible by not being bound to all kinds of complex management procedures.

I wrote this blog post while I was still employed at a small-sized startup company with only one development team. In the meantime, I have joined a much bigger organization (Mendix) that has many cross-disciplinary development teams that work concurrently on various aspects of our service and product portfolio.

About microservices


When I just joined, the amount of information I had to absorb was quite overwhelming. I also learned that we heavily adopted the microservices architecture paradigm for our entire online service platform.

According to Martin Fowler's blog post on microservices, using microservices offers the following benefits:

  • Strong module boundaries. You can divide the functionality of a system into microservices and make separate teams responsible for the development of each service. This makes it possible to iterate faster and offer better quality because teams can focus on themselves on a subset of features only.
  • Independent deployment. Microservices can be deployed independently making it possible to ship features when they are done, without having complex integration cycles.
  • Technology diversity. Microservices are language and technology agnostic. You can pick almost any programming language (e.g. Java, Python, Mendix, Go), data storage solution (e.g. PostgreSQL, MongoDB, InfluxDB) or operating system (e.g. Linux, FreeBSD, Windows) to implement a microservice making it possible pick the most suitable combination of technologies and use them at their full advantage.

However, decomposing a system into a collection of collaborating services also comes at a (sometimes substantial!) price:

  • There is typically much more operational complexity. Because there are many components and typically a large infrastructure to manage, activities such as deploying, upgrading, and monitoring the condition of a system is much more time consuming and complex. Furthermore, because of technology diversity, there are also many kinds of specialized deployment procedures that you need to carry out.
  • Data is eventually consistent. You have to live with the fact that (temporary) inconsistencies could end up in your data, and you must invest in implementing facilities that keep your data is consistent.
  • Because of distribution development is harder in general -- it is more difficult to diagnose errors (e.g. a failure in one service could trigger a chain reaction of errors, without having proper error traces), it is harder to test a system because of additional deployment complexity. The network links between services may be slow and subject to failure, causing all kinds of unpredictable problems. Also machines that host critical services may crash.

Studying the architecture


When applied properly, e.g. functionality is well separated and there is strong cohesion and weak coupling between services, while investing in solutions to cope with the challenges listed above, the benefits of microservices can be reaped, resulting in a scalable systems that can be developed my multiple teams working on features concurrently.

However, an important prerequisite for making changes in such an environment, and maintaining or improving the quality properties of a system, requires discipline and a relatively good understanding of the environment -- in the beginning, I faced all kinds of practical problems when I wanted to make even a subtle change -- some areas of our platform where documented, while others were not. Some documentation was also outdated, slightly incomplete and sometimes inconsistent with the actual implementation.

Certain areas of our platform were also highly complex resulting in very complex architectural views, with many boxes and arrows. Furthermore, information was also scattered around many different places.

As part of my on-boarding process, and as a means to cope with some of my practical problems, I have created a documentation repository of the platform that our team develops by extending the (minimalistic) principles for configuration management described in the earlier blog post.

I realized that simply identifying the service components of which the system consists, is not enough to get an understanding of the system -- there are many items and complex details that need to be captured.

In addition to the identification of all configuration items, I also want:

  • Proper guidance. To understand a particular piece of functionality, I should not need to study every component in detail. Instead, I want to know the full context and only the details of the relevant components.
  • Completeness. I want all service components to be visible. I do not want any details to be covered up. For example, I have also seen quite a few diagrams that hide complex implementation details. I much rather want flaws to be visible so that they can be resolved at a later point in time.
  • Clear boundaries. Our platform is not self contained, but relies on services provided by other teams. I want to know what components are our responsibility and what is managed by external teams.
  • Clarity. I want to know what the purpose of a component is. Their names may not always necessarily reflect or explain what they do.
  • Consistency. No matter how nicely a diagram is drawn, it should match the actual implementation or it is of very little use.
  • References to the actual implementation. I also want to know where I can find the implementation of a component, such as its Git repository.

Documenting the architecture


To visualize the architecture of our platform and organize all relevant information, I followed a strategy:

  • I took the components (typically their source code repositories) as the basis for everything else -- every component translates to a box in the architecture diagram.
  • I analyzed the dependency relationships between the components and denoted them as arrows. When a box points to another box by means of an arrow, this means that the other box is a dependency that should be deployed first. When a dependency is absent, the service will (most likely) not work.
  • I also discovered that the platform diagram easily gets cluttered by the sheer amount of components -- I decided to combine components that have very strongly correlated functionality in feature groups (that have dashed borders). Every feature group in architecture diagrams refers to another sub architecture diagram that provides a more specialized view of the feature group.
  • To clearly illustrate the difference between components that are our responsibility and those that are maintained by others teams, I make all external dependencies visible in the top-level architecture diagram.

The notation I used for these diagrams is not something I have entirely invented from scratch -- it is inspired by graph theory, package management and service management concepts. Disnix, for example, can visualize deployment architectures by using a similar notation.

To find all relevant information to create the diagrams, I consulted various sources:

  • I studied existing documents and diagrams to get a basic understanding of the system and an idea of the details I should look at.
  • I talked to a variety of people from various teams.
  • I looked inside the configuration settings of all deployment solutions used, e.g. the Amazon AWS console, Docker, CloudFoundry, Kubernetes, Nix configuration files.
  • Peek inside the source code repositories and look for settings that are references to other systems, such as configuration values that store URLs.
  • When I am in doubt: I consider the deployment configuration files and source code the "ultimate source of truth", because no matter how nice a diagram looks, it is useless if it is implemented differently.

Finally, just drawing diagrams will not completely suffice when the goal is provide clarity. I also observed that I need to document some leftover details.

Foremost, having a diagram without the semantics not explained will typically leave too many details open to interpretation to the user, so you need to explain the notation.

Second, you need to provide additional details about the services. I typically enumerate the following properties in a table for every component:

  • The name of the component.
  • A one line description stating its purpose.
  • The type of project (e.g. a Python/Java/Go project, Docker container, AWS Lambda function, etc.). This is useful to determine the kind of deployment procedure for the component.
  • A reference to the source code repository, e.g. a Git repository. The README of the corresponding repository should provide more detailed information about the project.

Benefits


Although it is quite a bit of work to set up, having a well documented architecture provides us the following benefits:

  • More effective deployment. Because of the feature groups and dividing the architecture into multiple layers, general concepts and details are separated. This makes it easier for developers to focus and absorb the right detailed knowledge to change a service.
  • More consensus in the team about the structure of the system and general quality attributes, such as scalability and security.
  • Better on-boarding for new team members.

Discussion


Writing architecture documentation IMO is not rocket science, just discipline. Obviously, there are much more sophisticated tools available to organize and visualize architectures (even tools that can generate code and reverse engineer code), but this is IMO not a hard requirement to start documenting.

However, you can not take all confusion away -- even if you have the best possible architecture documentation, people's thinking habits are shaped by the concepts they know and there will always be a slight mismatch (which is documented in academic research: 'Why Is It So Hard to Define Software Architecture?' written by Jason Baragry and Karl Reed).

Finally, architecture documentation is only a first good step to improve the quality of service-oriented systems. To make it a success, much more is needed, such as:

  • Automated (and reproducible) deployment processes.
  • More documentation (such as the APIs, end-user documentation, project documentation).
  • Automated unit, integration and acceptance testing.
  • Monitoring.
  • Measuring and improving code quality, test coverage, etc.
  • Using design patterns, architectural patterns, good programming abstractions.
  • And many more aspects.

But to do these things properly, having proper architecture documentation is an important prerequisite.

Related work


UPDATE: after publishing this blog post and giving an internal presentation at Mendix about this subject, I received a couple of questions about architectural approaches that share similarities with my approach, such as the C4 model.

The C4 model also uses a layered approach in which the top-level diagram displays the context of the system (the relation of the system with external users and systems), and deeper layers gradually reveal more details of the inner workings of the system while limiting the view to a subset of components to prevent details obfuscating the purpose.

I did not use this approach as an example reference, but my work is basically built on top of the same underlying principles that the C4 model builds on -- creating abstractions.

Creating abstractions in modeling is popularized already in the 70s by various computer scientists, such as Edward Yourdon and Tom DeMarco, who bring concepts from structural programming to other domains, such as modeling (as explained in the paper: 'The Choice of New Software Development Methodologies for Software Development Projects' by Edward Yourdon).

One of the mental aids in structured programming is abstraction, so that it "only matters what something does, disregarding how it works". I took data flow modeling (DFD) as an example technique (that also facilitates layers of abstractions including a top-level context DFD), but I replaced the data flow notation by dependency modeling.

Furthermore, the C4 model also provides a number of diagram types for each abstraction layer with specific purposes, but in my approach the notation and purposes of each layer are left abstract.