Disnix 0.5 release announcement and some reflection

In this blog post, I'd like to announce the next Disnix release. At the same time, I noticed that it has been eight years ago that I started developing it, so this would also be a nice opportunity to do some reflection.

Some background information

The idea was born while I was working on my master's thesis. A few months prior, I got familiar with Nix and NixOS -- I read Eelco Dolstra's PhD thesis, managed to package some software, and wrote a couple of services for NixOS.

Most of my packing work was done to automate the deployment of WebDSL applications, a case study in domain-specific language engineering, that is still an ongoing research project in my former research group. WebDSL's purpose is to be a domain-specific language for developing dynamic web applications with a rich data model.

Many aspects in Nix/NixOS were quite "primitive" compared to today's implementations -- there was no NixOS module system, making it less flexible to create additions. Many packages that I needed were missing and I had to write Nix expressions for them myself, such as Apache Tomcat, MySQL, and Midnight Commander. Also the desktop experience, such as KDE, was quite primitive, as only the base package was supported.

As part of my master's thesis project, I did an internship at the Healthcare Systems Architecture group at Philips Research. They had been developing a platform called SDS2, which purpose was to provide asset tracking and utilization analysis services for medical equipment.

SDS2 qualifies itself as a service-oriented system (a term that people used to talk frequently about in the past, but not anymore :) ). As such, it can be decomposed into a set of distributable components (a.k.a. services) that interact with each other through "standardized protocols" (e.g. SOAP), sometimes through network links.

There are a variety of reasons why SDS2 has a distributed architecture. For example, data that has been gathered from medical devices may have to be physically stored inside a hospital for privacy reasons. The analysis components may require a lot of computing power and would perform better if they run in a data center with a huge amount of system resources.

Being able to distribute services is good for many reasons (e.g. in meeting certain non-functional requirements such as privacy), but it also has a big drawback -- services are software components, and one of their characteristics is that they are units of deployment. Deploying a single service without any (or proper) automation to one machine is already complicated and time consuming, but deploying a network of machines is many times as complex.

The goal of my thesis assignment was to automate SDS2's deployment in distributed environments using the Nix package manager as a basis. Nix provides a number of unique properties compared to many conventional deployment solutions, such as fully automated deployment from declarative specifications, and reliable and reproducible deployment. However, it was also lacking a number of features to provide the same or similar kinds of quality properties to deployment processes of service-oriented systems in networks of machines.

The result of my master's thesis project was the first prototype of Disnix that I never officially released. After my internship, I started my PhD research and resumed working on Disnix (as well as several other aspects). This resulted in a second prototype and two official releases eventually turning Disnix into what it is today.

Prototype 1

This was the prototype resulting from my master's thesis and was primarily designed for deploying SDS2.

The first component that I developed was a web service (using similar kinds of technologies as SDS2, such as Apache Tomcat and Apache Axis2) exposing a set of deployment operations to remote machines (most of them consulting the Nix package manager).

To cope with permissions and security, I decided to make the web service just an interface around a "core service" that was responsible for actually executing the deployment activities. The web service used the D-Bus protocol to communicate with the core.

On top of the web service layer, I implemented a collection of tools each executing a specific deployment activity in a network of machines, such as building, distributing and activating services. There were also a number of tools combining deployment activities, such as the "famous" disnix-env command responsible for executing all the activities required to deploy a system.

The first prototype of disnix-env, in contrast to today's implementation, provided two deployment procedure variants: building on targets and building on the coordinator.

The first variant was basically inspired by the manual workflow I used to carry out to get SDS2 deployed -- I manually installed a couple of NixOS machines, then used SSH to remotely connect to them, there I would do a checkout of Nixpkgs and all the other Nix expressions that I need, then I would deploy all packages from source and finally I modified the system configuration (e.g. Apache Tomcat) to run the web services.

Unfortunately, transferring Nix expressions is not an easy process, as they are rarely self contained and typically rely on other Nix expression files scattered over the file system. While thinking about a solution, I "discovered" that the Nix expression evaluator creates so-called store derivation files (low-level build specifications) for each package build. Store derivations are also stored in the Nix store next to ordinary packages, including their dependencies. I could instead instantiate a Nix expression on the coordinator, transfer the closure of store derivation files to a remote machine, and build them there.

After some discussion with my company supervisor Merijn de Jonge, I learned that compiling on target machines was undesired, in particular in production environments. Then I learned more about Nix's purely functional nature, and "discovered" that builds are referentially transparent -- for example, it should not matter where a build has been performed. As long as the dependencies remain the same, the outcome would be the same as well. With this "new knowledge" in mind, I implemented a second deployment procedure variant that would do the package builds on the coordinator machine, and transfer their closures (dependencies) to the target machines.

As with the current implementation, deployment in Disnix was driven by three kinds of specifications: the services model, infrastructure model and distribution model. However, their notational conventions were a bit different -- the services model already knew about inter-dependencies, but propagating the properties of inter-dependencies to build functions was an ad-hoc process. The distribution model was a list of attribute sets also allowing someone to specify the same mappings multiple times (which resulted in undefined outcomes).

Another primitive aspect was the activation step, such as deploying web applications inside Apache Tomcat. It was basically done by a hardcoded script that only knew about Java web applications and Java command-line tools. Database activation was completely unsupported, and had to be done by hand.

I also did a couple of other interesting things. I studied the "two-phase commit protocol" for upgrading distributed systems atomically and mapped its concepts to Nix operations, to support (almost) atomic upgrades. This idea resulted in a research paper that I have presented at HotSWUp 2008.

Finally, I sketched a simple dynamic deployment extension (and wrote a partial implementation for it) that would calculate a distribution model, but time did not permit me to finish it.

Prototype 2

The first Disnix prototype made me quite happy in the early stages of my PhD research -- I gave many cool demos to various kinds of people, including our industry partner: Philips Healthcare and NWO/Jacquard: the organization that was funding me. However, I soon realized that the first prototype became too limited.

The first annoyance was my reliance on Java. Most of the tools in the Disnix distribution were implemented in Java and depended on the Java Runtime Environment, which is quite a big dependency for a set of command-line utilities. I reengineered most of the Disnix codebase and rewrote it in C. I only kept the core service (which was implemented in C already) and the web service interface, that I separated into an external package called DisnixWebService.

I also got rid of the reliance on a web service to execute remote deployment operations, because it was quite tedious to deploy it. I made the communication aspect pluggable and implemented an SSH plugin that became the default communication protocol (the web service protocol could still be used as an external plugin).

For the activation and deactivation of services, I developed a plugin system (Disnix activation scripts) and a set of modules supporting various kinds of services replacing the hardcoded script. This plugin system allowed me to activate and deactivate many kinds of components, including databases.

Finally, I unified the two deployment procedure variants of disnix-env into one procedure. Building on the targets became simply an optional step that was carried out before building on the coordinator.

Disnix 0.1

After my major reengineering effort, I was looking into publishing something about it. While working on a paper (which first version got badly rejected), I realized that services in a SOA-context are "platform independent" because of their interfaces, but they still have implementations underneath that could depend on many kinds of technologies. Heterogeneity makes deployment extra complicated.

There was still one piece missing to bring service-oriented systems to their full potential -- there was no multiple operating systems support in Disnix. The Nix package manager could also be used on several other operating systems besides Linux, but Disnix was bound to one operating system only (Linux).

I did another major reengineering effort to make the system architecture of the target systems configurable requiring me to change many things internally. I also developed new notational conventions for the Disnix models. Each service expression became a nested function in which the outer function corresponds to the intra-dependencies and the inner function to the inter-dependencies, and look quite similar to expressions for ordinary Nix packages. Moreover, I removed the ambiguity problem in the distribution model by making it an attribute set.

The resulting Disnix version was first described in my SEAA 2010 paper. Shortly after the paper got accepted, I decided to officially release this version as Disnix 0.1. Many external aspects of this version are still visible in the current version.

Disnix 0.2

After releasing the first of Disnix, I realized that there were still a few pieces missing while automating deployment processes of service-oriented systems. One of the limitations of Disnix is that it expects machines to be present already that may have to run a number of preinstalled system services, such as MySQL, Apache Tomcat, and the Disnix service exposing remote deployment operations. These machines had to be deployed by other means first.

Together with Eelco Dolstra I had been working on declarative deployment and testing of networked NixOS configurations, resulting in a tool called nixos-deploy-network that deploys networks of NixOS machines and a NixOS test driver capable of spawning networks of NixOS virtual machines in which system integration tests can be run non-interactively. These contributions were documented in a tech report and the ISSRE 2010 paper.

I made Disnix more modular so that extensions could be built on top of it. The most prominent extension was DisnixOS that integrates NixOS deployment and the NixOS test driver's features with Disnix service deployment so that a service oriented system's deployment process could be fully automated.

Another extension was Dynamic Disnix, a continuation of the dynamic deployment extension that I never finished during my internship. Dynamic Disnix extends the basic toolset with an infrastructure discovery tool and a distribution generator using deployment planning algorithms from the academic literature to map services to machines. The extended architecture is described in the SEAMS 2011 paper.

The revised Disnix architecture has been documented in both the WASDeTT 2010 and SCP 2014 papers and was released as Disnix 0.2.

Disnix 0.3

After the 0.2 release I got really busy, which was partly caused by the fact that I had to write my PhD thesis and yet another research paper for an unfinished chapter.

The last Disnix-related research contribution was a tool called Dysnomia, which I had based on the Disnix activation scripts package. I augmented the plugins with experimental state deployment operations and changed the package into a new tool, that in (theory) could be combined with other tools as well, or used independently.

Unfortunately, I had to quickly rush out a paper for HotSWUp 2012 and the code was in a barely usable state. Moreover, the state management facilities had some huge drawbacks, so I was not that eager to get them integrated into the mainstream version.

Then I had to fully dedicate myself to completing my PhD thesis and for more than six months, I hardly wrote any code.

After finishing my first draft of my PhD thesis and waiting for feedback from my committee, I left academia and switched jobs. Because I had no use practical use cases for Disnix, and other duties in my new job, its development was done mostly in my spare time at a very low pace -- some things that I accomplished in that period is creating a 'slim' version of Dysnomia that supported all the activities in the HotSWUp paper without any snapshotting facilities.

Meanwhile, nixops-deploy-network got replaced by a new tool named Charon, that later became NixOps. In addition to deployment, NixOps could also instantiate virtual machines in IaaS environments, such as Amazon EC2. I modified DisnixOS to also integrate with NixOps to use its capabilities.

Three and a half years after the previous release (late 2014), my new employer wanted to deploy their new microservices-based system to a production environment, which made me quite motivated to work on Disnix again. I did some huge refactorings and optimized a few aspects to make it work for larger systems. Some interesting optimizations were concurrent data transfers and concurrent service activations.

I also implemented multi-connection protocol support. For example, you could use SSH to connect to one machine and SOAP to another.

After implementing the optimizations, I realized that I had reached a stable point and decided that it was a good time to announce the next release, after a few years of only little development activity.

Disnix 0.4

Despite being happy with the recent Disnix 0.3 release and using it to deploy many services to production environments, I quickly ran into another problem -- the services that I had to manage store data in their own dedicated databases. Sometimes I had to move services from one machine to another. Disnix (like the other Nix tools) does not manage state, requiring me to manually migrate data, which was quite painful.

I decided to dig up the state deployment facilities from the HotSWUp 2012 paper to cope with this problem. Despite having a number of limitations, the databases that I had to manage were relatively small (tens of megabytes), so the solution was still a good fit.

I integrated the state management facilities described in the paper from the prototype into the "production" version of Dysnomia, and modified Disnix to use them. I left out the incremental snapshot facilities described in the paper, because there was no practical use for them. When the work was done, I announced the next release.

Disnix 0.5

With Disnix 0.4, all my configuration management work was automated. However, I spotted a couple of inefficiencies, such as many unnecessary redeployments while upgrading. I solved this issue by making the target-specific services concept a first class citizen in Disnix. Moreover, I regularly had to deal with RAM issues and added on-demand activation support (by using the operating system's service manager, such as systemd).

There were also some user-unfriendly aspects that I improved -- better and more concise logging, more helpful error messages, --rollback, --switch-generation options for disnix-env, and some commands that work on the deployment manifest were extended to take the last deployed manifest into account when no parameters have been provided (e.g. disnix-visualize).

Conclusion

This long blog post describes how the current Disnix version (0.5) came about after nearly eight years of development. I'd like to announce its immediate availability! Consult the Disnix homepage for more information.

Integrating callback and promise based function invocation patterns (Asynchronous programming with JavaScript part 4)

It has been quiet for a while on my blog in the programming language domain. Over two years ago, I started writing a series of blog posts about asynchronous programming with JavaScript.

In the first blog post, I explained some general asynchronous programming issues, code structuring issues and briefly demonstrated how the async library can be used to structure code more properly. Later, I have written a blog post about promises, another abstraction mechanism dealing with asynchronous programming complexities. Finally, I have developed my own abstraction functions by investigating how JavaScript's structured programming language constructs (that are synchronous) translate to the asynchronous programming world.

In these blog posts, I have used two kinds of function invocation styles -- something that I call the Node.js-function invocation style, and the promises invocation style. As the name implies, the former is used by the Node.js standard library, as well as many Node.js-based APIs. The latter is getting more common in the browser world. As a matter of fact, many modern browsers, provide a Promise prototype as part of their DOM API allowing others to construct their own Promise-based APIs with it.

In this blog post, I will compare both function invocation styles and describe some of their differences. Additionally, there are situations in which I have to mix APIs using both styles and I have observed that it is quite annoying to combine them. I will show how to alleviate this pain a bit by developing my own generically applicable adapter functions.

Two example invocations

The most frequently used invocation style in my blog posts is something that I call the Node.js-function invocation style. An example code fragment that uses such an invocation is the following:

fs.readFile("hello.txt", function(err, data) {
    if(err) {
        console.log("Error while opening file: "+err);
    } else {
        console.log("File contents is: "+data);
    }
});

As you may see in the code fragment above, when we invoke the readFile() function, it returns immediately (to be precise: it returns, but it returns no value). We use a callback function (that is typically the last function parameter) to retrieve the results of the invocation (or the error if something went wrong) at a later point in time.

By convention, the first parameter of the callback is an error parameter that is not null if some error occurs. The remaining parameters are optional and can be used to retrieve the corresponding results.

When using promises (more specifically: promises that conform to the Promises/A and Promises/A+ specifications), we use a different invocation pattern that may look as follows:

Task.findAll().then(function(tasks) {
    for(var i = 0; i < tasks.length; i++) {
        var task = tasks[i];
        console.log(task.title + ": "+ task.description);
    }
}, function(err) {
    console.log("An error occured: "+err);
});

As with the previous example, the findAll() function invocation shown above also returns immediately. However, it also does something different compared to the Node.js-style function invocation -- it returns an object called a promise whereas the invocation in the previous example never returns anything.

By convention, the resulting promise object provides a method called then() in which (according the Promises/A and A+ standards) the first parameter is a callback that gets invoked when the function invocation succeeds and the second callback gets invoked when the function invocation fails. The parameters of the callback functions represent result objects or error objects.

Comparing the invocation styles

At first sight, you may probably notice that despite having different styles, both function invocations return immediately and need an "artificial facility" to retrieve the corresponding results (or errors) at a later point in time, as opposed to directly returning a result in a function.

The major difference is that in the promises invocation style, you will always get a promise as a result of an invocation. A promise provides a reference to something which corresponding result will be delivered in the future. For example, when running:

var tasks = Task.findAll();

I will obtain a promise that, at some point in the future, provides me an array of tasks. I can use this reference to do other things by passing the promise around (for example) as a function argument to other functions.

For example, I may want to construct a UI displaying the list of tasks. I can already construct pieces of it without waiting for the full list of tasks to be retrieved:

displayTasks(tasks);

The above function could, for example, already start rendering a header, some table cells and buttons without the results being available yet. The display function invokes the then() function when it really needs the data.

By contrast, in the Node.js-callback style, I have no reference to the pending invocation at all. This means that I always have to wait for its completion before I can render anything UI related. Because we are forced to wait for its completion, it will probably make the application quite unresponsive, in particular when we have to retrieve many task records.

So in general, in addition to better structured code, promises support composability whereas Node.js-style callbacks do not. Because of this reason, I consider promises to be more powerful.

However, there is also something that I consider a disadvantage. In my first blog post, I have shown the following Node.js-function invocation style pyramid code example as a result of nesting callbacks:

var fs = require('fs');
var path = require('path');

fs.mkdir("out", 0755, function(err) {
    if(err) throw err;
    
    fs.mkdir(path.join("out, "test"), 0755, function(err) {
        if (err) throw err;        
        var filename = path.join("out", "test", "hello.txt");

        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) throw err;
                    
            fs.readFile(filename, function(err, data) {
                if(err) throw err;
                
                if(data == "Hello world!")
                    process.stderr.write("File is correct!\n");
                else
                    process.stderr.write("File is incorrect!\n");
            });
        });
    });
});

I have also shown in the same blog post, that I can use the async.waterfall() abstraction to flatten its structure:

var fs = require('fs');
var path = require('path');

filename = path.join("out", "test", "hello.txt");

async.waterfall([
    function(callback) {
        fs.mkdir("out", 0755, callback);
    },

    function(callback) {
        fs.mkdir(path.join("out, "test"), 0755, callback);
    },

    function(callback) {
        fs.writeFile(filename, "Hello world!", callback);
    },

    function(callback) {
        fs.readFile(filename, callback);
    },

    function(data, callback) {
        if(data == "Hello world!")
            process.stderr.write("File is correct!\n");
        else
            process.stderr.write("File is incorrect!\n");
    }

], function(err, result) {
    if(err) throw err;
});

As you may probably notice, the code fragment above is much more readable and better maintainable.

In my second blog post, I implemented a promises-based variant of the same example:

var fs = require('fs');
var path = require('path');
var Promise = require('rsvp').Promise;

/* Promise object definitions */

var mkdir = function(dirname) {
    return new Promise(function(resolve, reject) {
        fs.mkdir(dirname, 0755, function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var writeHelloTxt = function(filename) {
    return new Promise(function(resolve, reject) {
        fs.writeFile(filename, "Hello world!", function(err) {
            if(err) reject(err);
            else resolve();
        });
    });
};

var readHelloTxt = function(filename) {
    return new Promise(function(resolve, reject) {
        fs.readFile(filename, function(err, data) {
            if(err) reject(err);
            else resolve(data);
        });
    });
};

/* Promise execution chain */

var filename = path.join("out", "test", "hello.txt");

mkdir(path.join("out"))
.then(function() {
    return mkdir(path.join("out", "test"));
})
.then(function() {
    return writeHelloTxt(filename);
})
.then(function() {
    return readHelloTxt(filename);
})
.then(function(data) {
    if(data == "Hello world!")
        process.stderr.write("File is correct!\n");
    else
        process.stderr.write("File is incorrect!\n");
}, function(err) {
    console.log("An error occured: "+err);
});

As you may notice, because the then() function invocations can be chained, we also have a flat structure making the code better maintainable. However, the code fragment is also considerably longer than the async library variant and the unstructured variant -- for each asynchronous function invocation, we must construct a promise object, adding quite a bit of overhead to the code.

From my perspective, if you need to do many ad-hoc steps (and not having to compose complex things), callbacks are probably more convenient. For reusable operations, promises are typically a nicer solution.

Mixing function invocations from both styles

It may happen that function invocations from both styles need to be mixed. Typically mixing is imposed by third-party APIs -- for example, when developing a Node.js web application we may want to use express.js (callback based) for implementing a web application interface in combination with sequelize (promises based) for accessing a relational database.

Of course, you could write a function constructing promises that internally only use Node.js-style invocations or the opposite. But if you have to regularly intermix calls, you may end up writing a lot of boilerplate code. For example, if I would use the async.waterfall() abstraction in combination with promise-style function invocations, I may end up writing:

async.waterfall([
    function(callback) {
        Task.sync().then(function() {
            callback();
        }, function(err) {
            callback(err);
        });
    },
    
    function(callback) {
        Task.create({
            title: "Get some coffee",
            description: "Get some coffee ASAP"
        }).then(function() {
            callback();
        }, function(err) {
            callback(err);
        });
    },
    
    function(callback) {
        Task.create({
            title: "Drink coffee",
            description: "Because I need caffeine"
        }).then(function() {
            callback();
        }, function(err) {
            callback(err);
        });
    },
    
    function(callback) {
        Task.findAll().then(function(tasks) {
            callback(null, tasks);
        }, function(err) {
            callback(err);
        });
    },
    
    function(tasks, callback) {
        for(var i = 0; i < tasks.length; i++) {
            var task = tasks[i];
            console.log(task.title + ": "+ task.description);
        }
    }
], function(err) {
    if(err) {
        console.log("An error occurred: "+err);
        process.exit(1);
    } else {
        process.exit(0);
    }
});

For each Promise-based function invocation, I need to invoke the then() function and in the corresponding callbacks, I must invoke the callback of each function block to propagate the results or the error. This makes the amount of code I have to write unnecessary long, tedious to write and a pain to maintain.

Fortunately, I can create a function that abstracts over this pattern:

function chainCallback(promise, callback) {
    promise.then(function() {
        var args = Array.prototype.slice.call(arguments, 0);
        
        args.unshift(null);
        callback.apply(null, args);
    }, function() {
        var args = Array.prototype.slice.call(arguments, 0);
        
        if(args.length == 0) {
            callback("Promise error");
        } else if(args.length == 1) {
            callback(args[0]);
        } else {
            callback(args);
        }
    });
}

The above code fragment does the following:

• We define a function takes a promise and a Node.js-style callback function as parameters and invokes the then() method of the promise.
• When the promise has been fulfilled, it sets the error parameter of the callback to null (to indicate that there is no error) and propagates all resulting objects as remaining parameters to the callback.
• When the promise has been rejected, we propagate the resulting error object. Because the Node.js-style-callback requires a single defined object, we compose one ourselves if no error object was returned, and we return an array as an error object, if multiple error objects were returned.

Using this abstraction function, we can rewrite the earlier pattern as follows:

async.waterfall([
    function(callback) {
        prom2cb.chainCallback(Task.sync(), callback);
    },
    
    function(callback) {
        prom2cb.chainCallback(Task.create({
            title: "Get some coffee",
            description: "Get some coffee ASAP"
        }), callback);
    },
    
    function(callback) {
        prom2cb.chainCallback(Task.create({
            title: "Drink coffee",
            description: "Because I need caffeine"
        }), callback);
    },
    
    function(callback) {
        prom2cb.chainCallback(Task.findAll(), callback);
    },
    
    function(tasks, callback) {
        for(var i = 0; i < tasks.length; i++) {
            var task = tasks[i];
            console.log(task.title + ": "+ task.description);
        }
    }
], function(err) {
    if(err) {
        console.log("An error occurred: "+err);
        process.exit(1);
    } else {
        process.exit(0);
    }
});

As may be observed, this code fragment is more concise and significantly shorter.

The opposite mixing pattern also leads to issues. For example, we can first retrieve the list of tasks from the database (through a promise-style invocation) and then write it as a JSON file to disk (through a Node.js-style invocation):

Task.findAll().then(function(tasks) {
    fs.writeFile("tasks.txt", JSON.stringify(tasks), function(err) {
        if(err) {
            console.log("error: "+err);
        } else {
            console.log("everything is OK");
        }
    });
}, function(err) {
    console.log("error: "+err);
});

The biggest annoyance is that we are forced to do the successive step (writing the file) inside the callback function, causing us to write pyramid code that is harder to read and tedious to maintain. This is caused by the fact that we can only "chain" a promise to another promise.

Fortunately, we can create a function abstraction that wraps an adapter around any Node.js-style function taking the same parameters (without the callback) that returns a promise:

function promisify(Promise, fun) {
    return function() {
       var args = Array.prototype.slice.call(arguments, 0);
           
       return new Promise(function(resolve, reject) {
            function callback() {
                var args = Array.prototype.slice.call(arguments, 0);
                var err = args[0];
                args.shift();
                    
                if(err) {
                    reject(err);
                } else {
                    resolve(args);
                }
            }
           
            args.push(callback);
                
            fun.apply(null, args);
        });
    };
}

In the above code fragment, we do the following:

• We define a function that takes two parameters: a Promise prototype that can be used to construct promises and a function representing any Node.js-style function (which the last parameter is a Node.js-style callback).
• In the function, we construct (and return) a wrapper function that returns a promise.
• We construct an adapter callback function, that invokes the Promise toolkit's reject() function in case of an error (with the corresponding error object provided by the callback), and resolve() in case of success. In case of success, it simply propagates any result object provided by the Node.js-style callback.
• Finally, we invoke the Node.js-function with the given function parameters and our adapter callback.

With this function abstraction we can rewrite the earlier example as follows:

Task.findAll().then(function(tasks) {
    return prom2cb.promisify(Promise, fs.writeFile)("tasks.txt", JSON.stringify(tasks));
})
.then(function() {
    console.log("everything is OK");
}, function(err) {
    console.log("error: "+err);
});

as may be observed, we can convert the writeFile() Node.js-style function invocation into an invocation returning a promise, and nicely structure the find and write file invocations by chaining then() invocations.

Conclusions

In this blog post, I have explored two kinds of asynchronous function invocation patterns: Node.js-style and promise-style. You may probably wonder which one I like the most?

I actually hate them both, but I consider promises to be the more powerful of the two because of their composability. However, this comes at a price of doing some extra work to construct them. The most ideal solution to me is still a facility that is part of the language, instead of "forgetting" about existing language constructs and replacing them by custom-made abstractions.

I have also explained that we may have to combine both patterns, which is often quite tedious. Fortunately, we can create function abstractions that convert one into another to ease the pain.

Related work

I am not the first one comparing the function invocation patterns described in this blog post. Parts of this blog post are inspired by a blog post titled: "Callbacks are imperative, promises are functional: Node’s biggest missed opportunity". In this blog post, a comparison between the two invocation styles is done from a programming language paradigm perspective, and is IMO quite interesting to read.

I am also not the first to implement conversion functions between these two styles. For example, promises constructed with the bluebird library implement a method called .asCallback() allowing a user to chain a Node.js-style callback to a promise. Similarly, it provides a function: Promise.promisify() to wrap a Node.js-style function into a function returning a promise.

However, the downside of bluebird is that these facilities can only be used if bluebird is used as a toolkit in an API. Some APIs use different toolkits or construct promises themselves. As explained earlier, Promises/A and Promises/A+ are just interface specifications and only the purpose of then() is defined, whereas the other facilities are extensions.

My function abstractions only make a few assumptions and should work with many implementations. Basically it only requires a proper .then() method (which should be obvious) and a new Promise(function(resolve, reject) { ... }) constructor.

Besides the two function invocation styles covered in this blog post, there are others as well. For example, Zef's blog post titled: "Callback-Free Harmonious Node.js" covers a mechanism called 'Thunks'. In this pattern, an asynchronous function returns a function, which can be invoked to retrieve the corresponding error or result at a later point in time.

References

The two conversion abstractions described in this blog post are part of a package called prom2cb. It can be obtained from my GitHub page and the NPM registry.