One of the reasons that I wrote so many JavaScript-related blog posts is because the language used to have many catches, such as:
- Scoping. Contrary to many other mainstream programming languages, JavaScript uses function-level scoping as opposed to block-level scoping. Syntactically, function-level scoping looks very similar to block-level scoping.
Function-level scoping has a number of implications that could have severe consequences. For example, you may unintentionally re-assign values. - Simulating class-based inheritance. JavaScript supports Object Oriented programming with prototypes rather than classes (that most mainstream Object Oriented programming languages use). It is possible to use prototypes to simulate classes and class-based inheritance.
Although I consider prototypes to be conceptually simple, using them in JavaScript used to be quite confusing. As a consequence, simulating class inheritance also used to be quite difficult. - Asynchronous programming. JavaScript is originally designed for use in web browsers, but has also become quite popular outside the browser, such as Node.js, to write server/command-line applications. For both browser usage as well as server applications, it is often desired to do multiple tasks concurrently.
Unfortunately, most of JavaScript's language constructs are synchronous, making such tasks quite difficult without using any software abstractions.
In particular about the last topic: asynchronous programming, I wrote many blog posts. I have elaborated about callbacks in Node.js and abstraction libraries to do coordination and another popular abstraction: promises.
I have also argued that most of JavaScript's language constructs, that implement structured programming concepts, are synchronous and cannot be used with asynchronous functions that return (almost) immediately, and resume their executions with callbacks.
I have built my own library: slasp, that implements asynchronous equivalents for the synchronous language constructs that you should not use.
Fortunately, much has happened since I wrote that blog post. The JavaScript language has many new features (part of the ECMAScript 6 standard) that have become a standard practice nowadays, such as:
- Block level scoping. Block scoped immutable values can be declared with: const and mutable values with: let.
- An object with a custom prototype can now be directly created with: Object.create.
- A class construct was added that makes it possible to define classes (that are simulated with prototypes).
Moreover, modern JavaScript also has new constructs and APIs for asynchronous programming, making most of the software abstractions that I have elaborated about obsolete for the most part.
Recently, I have been using these new constructs quite intensively and learned that my previous blog posts (that I wrote several years ago) are still mostly about old practices.
In this blog post, I will revisit the structured programming topic and explain how modern language constructs can be used to implement these concepts.
Asynchronous programming in JavaScript
As explained in my previous blog posts, asynchronous programming in JavaScript is important for a variety of reasons. Some examples are:
- Animating objects and providing other visual effects in a web browser, while keeping the browser responsive so that it can respond to user events, such as mouse clicks.
- The ability to serve multiple clients concurrently in a Node.js server application.
Multi-tasking in JavaScript is (mainly) cooperative. The idea is that JavaScript code runs in a hidden main loop that responds to events in a timely manner, such as user input (e.g. mouse clicks) or incoming connections.
To keep your application responsive, it is required that the execution of a code block does not take long (to allow the application to respond to other events), and that an event is generated to allow the execution to be resumed at a later point in time.
Not meeting this requirement may cause the web browser or your server application to block, which is often undesirable.
Because writing non-blocking code is so important, many functions in the Node.js API are asynchronous by default: they return (almost) immediately and use a callback function parameter that gets invoked when the work is done.
For example, reading a file from disk while keeping the application responsive can be done as follows:
const fs = require('fs'); fs.readFile("hello.txt", function(err, contents) { if(err) { console.error(err); process.exit(1); } else { console.log(contents); } });
Note that in the above code fragment, instead of relying on the return value of the fs.readFile call, we provide a callback function as a parameter that gets invoked when the operation has finished. The callback is responsible for displaying the file's contents or the resulting error message.
While the file is being read (that happens in the background), the event loop is still able to process other events making it possible for the application to work on other tasks concurrently.
To ensure that an application is responsive and scalable, I/O related functionality in the Node.js API is asynchronous by default. For some functions there are also synchronous equivalents for convenience, but as a rule of thumb they should be avoided as much as possible.
Asynchronous I/O is an important ingredient in making Node.js applications scalable -- because I/O operations are typically several orders of magnitude slower than CPU operations, the application should remain responsive as long as no callback takes long to complete. Furthermore, because there is no thread per connection model, there is no context-switching and memory overhead for each concurrent task.
However, asynchronous I/O operations and a callback-convention does not guarantee that the main loop never gets blocked.
When implementing tasks that are heavily CPU-bound (such as recursively computing a Fibonacci number), the programmer has to make sure that the execution does not block the main loop too long (for example, by dividing it into smaller tasks that generate events, or using threads).
Code structuring issues
Another challenge that comes with asynchronous functions is that it becomes much harder to keep your code structured and maintainable.
For example, if we want to create a directory, then write a text file to it, and then read the text file, and only use non-blocking functions to keep the application responsive, we may end up writing:
const fs = require('fs'); fs.mkdir("test", function(err) { if(err) { console.error(err); process.exit(1); } else { fs.writeFile("hello.txt", "Hello world!", function(err) { if(err) { console.error(err); process.exit(1); } else { fs.readFile("hello.txt", function(err, contents) { if(err) { console.error(err); process.exit(1); } else { // Displays: "Hello world!" console.log(contents); } }); } }); } });
As may be observed, for each function call, we define a callback responsible for checking the status of the call and executing the next step. For each step, we have to nest another callback function, resulting in pyramid code.
The code above is difficult to read and maintain. If we want to add another step in the middle, we are forced to refactor the callback nesting structure, which is labourious and tedious.
Because code structuring issues are so common, all kinds of software abstractions have been developed to coordinate the execution of tasks. For example, we can use the async library to rewrite the above code fragment as follows:
const async = require('async'); async.waterfall([ function(callback) { fs.mkdir("test", callback); }, function(callback) { fs.writeFile("hello.txt", "Hello world!", callback); }, function(callback) { fs.readFile("hello.txt", callback); }, function(contents, callback) { // Displays: "Hello world!" console.log(contents); callback(); } ], function(err) { if(err) { console.error(err); process.exit(1); } });
The async.waterfall abstraction flattens the code, allows us to conveniently add additional asynchronous steps and change the order, if desired.
Promises
In addition to Node.js-style callback functions and abstraction libraries for coordination, a more powerful software abstraction was developed: promises (to be precise: there are several kinds of promise abstractions developed, but I am referring to the Promises/A+ specification).
Promises have become very popular, in particular for APIs that are used in the browser. As a result, they have been accepted into the core JavaScript API.
With promises the idea is that every asynchronous function quickly returns a promise object that can be used as a reference to a value that will be delivered at some point in the future.
For example, we can wrap the function invocation that reads a text file into a function that returns promise:
const fs = require('fs'); function readHelloFile() { return new Promise((resolve, reject) => { fs.readFile("hello.txt", function(err, contents) { if(err) { reject(err); } else { resolve(contents); } }); }); }
The above function: readHelloFile invokes fs.readFile from the Node.js API to read the hello.txt file and returns a promise. In case the file was successfully read, the promise is resolved and the file's contents is propagated as a result. In case of an error, the promise is rejected with the resulting error message.
To retrieve and display the result, we can invoke the above function as follows:
readHelloFile().then(function(contents) { console.log(contents); }, function(err) { console.error(err); process.exit(1); });
Invoking the then method causes the main event loop to invoke either the resolve (first parameter) or reject callback function (second parameter) when the result is available.
Because promises have become part of the ECMAScript standard, Node.js has introduced alternative APIs that are promise-based, instead of callback based (such as for filesystem operations: fs.promises).
By using the promises-based API for filesystem operations, we can simplify the previous example to:
const fs = require('fs').promises; fs.readFile("hello.txt").then(function(contents) { console.log(contents); }, function(err) { console.error(err); process.exit(1); });
As described in my old blog post about promises -- they are considered more powerful than callbacks. A promise provides you a reference to a value that is in the process of being delivered. Callbacks can only give you insights in the status of a background task as soon as it completes.
Although promises have an advantage over callbacks, both approaches still share the same drawback -- we are forced to avoid most JavaScript language constructs and use alternative function abstractions.
In modern JavaScript, it is also no longer necessary to always explicitly create promises. Instead, we can also declare a function as async. Simply returning a value or throwing exception in such a function automatically ensures that a promise is returned:
async function computeSum(a, b) { return a + b; }
The function above returns the sum of the provided input parameters. The JavaScript runtime automatically wraps its execution into a promise that can be used to retrieve the return value at some point in the future.
(As a sidenote: the function above returns a promise, but is still blocking. It does not generate an event that can be picked up by the event loop and a callback that can resume its execution at a later point in time.)
The result of executing the following code:
const result = computeSum(1, 1); console.log("The result is: " + result);
is a promise object, not a numeric value:
Result is: [object Promise]
When a function returns a promise, we also no longer have to invoke .then() and provide callbacks as parameters to retrieve the result or any thrown errors. The await keyword can be used to automatically wait for a promise to yield its return value or an exception, and then move to the next statement:
(async function() { const result = await computeSum(1, 1); console.log("The result is: " + result); // The result is: 2 })();
The only catch is that you can only use await in the scope of an asynchronous function. The default scope of a Node.js program is synchronous. As a result, we have to wrap the code into an asynchronous function block.
By using the promise-based fs API and the new language features, we can rewrite our earlier callback-based example (that creates a directory, writes and reads a file) as follows:
const fs = require('fs').promises; (async function() { try { await fs.mkdir("test"); await fs.writeFile("hello.txt", "Hello world!"); const contents = await fs.readFile("hello.txt"); } catch(err) { console.error(err); process.exit(1); } })();
As may be observed, the code is much simpler than manually orchestrating promises.
Structured programming concepts
In my previous blog post, I have argued that most of JavaScript's language constructs (that implement structured programming concepts) cannot be directly used in combination with non-blocking functions (that return almost immediately and require callback functions as parameters).
As a personal exercise, I have created function abstractions that are direct asynchronous equivalents for all these structured programming concepts that should be avoided and added them to my own library: slasp.
By combining promises, async functions and await statements, these function abstractions have mostly become obsolete.
In this section, I will go over the structured programming concepts I covered in my old blog post and show their direct asynchronous programming equivalents using modern JavaScript language constructs.
Function definitions
As I have already explained in my previous blog post, the most basic thing one can do in JavaScript is executing statements, such as variable assignments or function invocations. This used to be already much different when moving from a synchronous programming to an asynchronous programming world.
As a trivial example, I used a synchronous function whose only purpose is to print text on the console:
function printOnConsole(value) { console.log(value); }
The above example is probably too trivial, but it is still possible to make it non-blocking -- we can generate a tick event so that the function returns immediately and use a callback parameter so that the task will be resumed at a later point in time:
function printOnConsole(value) { return new Promise((resolve, reject) => { process.nextTick(function() { console.log(value); resolve(); }); }); }
To follow modern JavaScript practices, the above function is wrapped into a constructor that immediately returns a promise that can be used as a reference to determine when the task was completed.
(As a sidenote: we compose a regular function that returns a promise. We cannot define an async function, because the process.nextTick is an asynchronous function that requires a callback function parameter. The callback is responsible for propagating the end result. Using a return only causes the callback function to return and not the enclosing function.)
I have also shown that for functions that return a value, the same principle can be applied. As an example, I have used a function that translates a numeric digit into a word:
function generateWord(digit) { const words = [ "zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine" ]; return words[digit]; }
We can also make this function non-blocking by generating a tick event and wrapping it into a promise:
function generateWord(digit) { return new Promise((resolve, reject) => { process.nextTick(function() { const words = [ "zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine" ]; resolve(words[digit]); }); }); }
Sequential decomposition
The first structured programming concept I elaborated about was sequential decomposition in which a number of statements are executed in sequential order.
I have shown a trivial example that adds 1 to a number, then converts the resulting digit into a word, and finally prints the word on the console:
const a = 1; const b = a + 1; const number = generateWord(b); printOnConsole(number); // two
With the introduction of the await keyword, converting the above code to use the asynchronous implementations of all required functions has become straight forward:
(async function() { const a = 1; const b = a + 1; const number = await generateWord(b); await printOnConsole(number); // two })();
The above example is a one-on-one port of its synchronous counterpart -- we just have to use the await keyword in combination with our asynchronous function invocations (that return promises).
The only unconventional aspect is that we need to wrap the code inside an asynchronous function block to allow the await keyword to be used.
Alteration
The second programming concept that I covered is alteration that is used to specify conditional statements.
I gave a simple example that checks whether a given name matches my own name:
function checkMe(name) { return (name == "Sander"); } const name = "Sander"; if(checkMe(name)) { printOnConsole("It's me!"); printOnConsole("Isn't it awesome?"); } else { printOnConsole("It's someone else!"); }
It is also possible to make the checkMe function non-blocking by generating a tick event and wrapping it into a promise:
function checkMe(name) { return new Promise((resolve, reject) => { process.nextTick(function() { resolve(name == "Sander"); }); }); }
To invoke the asynchronous function shown above inside the if-statement, we only have to write:
(async function() { const name = "Sander"; if(await checkMe(name)) { await printOnConsole("It's me!"); await printOnConsole("Isn't it awesome?"); } else { await printOnConsole("It's someone else!"); } })();
In my previous blog post, I was forced to abolish the regular if-statement and use an abstraction (slasp.when) that invokes the non-blocking function first, then uses the callback to retrieve the result for use inside an if-statement. In the above example, the only subtle change I need to make is to use await inside the if-statement.
I can also do the same thing for the other alteration construct: the switch -- just using await in the conditional expression and the body should suffice.
Repetition
For the repetition concept, I have shown an example program that implements the Gregory-Leibniz formula to approximate PI up to 6 digits:
function checkTreshold(approx) { return (approx.toString().substring(0, 7) != "3.14159"); } let approx = 0; let denominator = 1; let sign = 1; while(checkTreshold(approx)) { approx += 4 * sign / denominator; printOnConsole("Current approximation is: "+approx); denominator += 2; sign *= -1; }
As with the previous example, we can also make the checkTreshold function non-blocking:
function checkTreshold(approx) { return new Promise((resolve, reject) => { process.nextTick(function() { resolve(approx.toString().substring(0, 7) != "3.14159"); }); }); }
In my previous blog post, I have explained that the while statement is unfit for executing non-blocking functions in sequential order, because they return immediately and have to resume their execution at a later point in time.
As with the alteration language constructs, I have developed a function abstraction that is equivalent to the while statement (slasp.whilst), making it possible to have a non-blocking conditional check and body.
With the introduction of the await statement, this abstraction function also has become obsolete. We can rewrite the code as follows:
(async function() { let approx = 0; let denominator = 1; let sign = 1; while(await checkTreshold(approx)) { approx += 4 * sign / denominator; await printOnConsole("Current approximation is: "+approx); denominator += 2; sign *= -1; } })();
As can be seen, the above code is a one-on-one port of its synchronous counterpart.
The function abstractions for the other repetition concepts: doWhile, for, for-in have also become obsolete by using await for evaluating the non-blocking conditional expressions and bodies.
Implementing non-blocking recursive algorithms still remains tricky, such as the following (somewhat inefficient) recursive algorithm to compute a Fibonacci number:
function fibonacci(n) { if (n < 2) { return 1; } else { return fibonacci(n - 2) + fibonacci(n - 1); } } const result = fibonacci(20); printOnConsole("20th element in the fibonacci series is: "+result);
The above algorithm is mostly CPU-bound and takes some time to complete. As long as it is computing, the event loop remains blocked causing the entire application to become unresponsive.
To make sure that the execution does not block for too long by using cooperative multi-tasking principles, we should regularly generate events, suspend its execution (so that the event loop can process other events) and use callbacks to allow it to resume at a later point in time:
function fibonacci(n) { return new Promise((resolve, reject) => { if (n < 2) { setImmediate(function() { resolve(1); }); } else { let first; let second; fibonacci(n - 2) .then(function(result) { first = result; return fibonacci(n - 1); }) .then(function(result) { second = result; resolve(first + second); }); } }); } (async function() { const result = await fibonacci(20); await printOnConsole("20th element in the fibonacci series is: "+result); })();
In the above example, I made the algorithm non-blocking by generating a macro-event with setImmediate for the base step. Because the function returns a promise, and cannot be wrapped into an async function, I have to use the promises' then methods to retrieve the return values of the computations in the induction step.
Extensions
In my previous blog post, I have also covered the extensions to structured programming that JavaScript provides.
Exceptions
I have also explained that with asynchronous functions, we cannot use JavaScript's throw, and try-catch-finally language constructs, because exceptions are typically not thrown instantly but at a later point in time.
With await, using these constructs is also no longer a problem.
For example, I can modify our generateWord example to throw an exception when the provided number is not between 0 and 9:
function generateWord(num) { if(num < 0 || num > 9) { throw "Cannot convert "+num+" into a word"; } else { const words = [ "zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine" ]; return words[num]; } } try { let word = generateWord(1); printOnConsole("We have a: "+word); word = generateWord(10); printOnConsole("We have a: "+word); } catch(err) { printOnConsole("Some exception occurred: "+err); } finally { printOnConsole("Bye bye!"); }
We can make generateWord an asynchronous function by converting it in the usual way:
function generateWord(num) { return new Promise((resolve, reject) => { process.nextTick(function() { if(num < 0 || num > 9) { reject("Cannot convert "+num+" into a word"); } else { const words = [ "zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine" ]; resolve(words[num]); } }); }); } (async function() { try { let word = await generateWord(1); await printOnConsole("We have a: "+word); word = await generateWord(10); await printOnConsole("We have a: "+word); } catch(err) { await printOnConsole("Some exception occurred: "+err); } finally { await printOnConsole("Bye bye!"); } })();
As can be seen in the example above, thanks to the await construct, we have not lost our ability to use try/catch/finally.
Objects
Another major extension is object-oriented programming. As explained in an old blog post about object oriented programming in JavaScript, JavaScript uses prototypes rather than classes, but prototypes can still be used to simulate classes and class-based inheritance.
Because simulating classes is such a common use-case, a class construct was added to the language that uses prototypes to simulate it.
The following example defines a Rectangle class with a method that can compute a rectangle's area:
class Rectangle { constructor(width, height) { this.width = width; this.height = height; } calculateArea() { return this.width * this.height; } } const r = new Rectangle(2, 2); printOnConsole("Area is: "+r.calculateArea());
In theory, it is also possible that the construction of an object takes a long time and should be made non-blocking.
Although JavaScript does not have a language concept to do asynchronous object construction, we can still do it by making a couple of small changes:
class Rectangle { asyncConstructor(width, height) { return new Promise((resolve, reject) => { process.nextTick(() => { this.width = width; this.height = height; resolve(); }); }); } calculateArea() { return this.width * this.height; } } (async function() { const r = new Rectangle(); await r.asyncConstructor(2, 2); await printOnConsole("Area is: "+r.calculateArea()); })();
As can be seen in the above example, the constructor function has been replaced with an asyncConstructor method that implements the usual strategy to make it non-blocking.
To asynchronously construct the rectangle, we first construct an empty object using the Rectangle class object as its prototype. Then we invoke the asynchronous constructor to initialize the object in a non-blocking way.
In my previous blog post, I have developed an abstraction function that could be used as an asynchronous replacement for JavaScript's new operator (slasp.novel) that performs the initialization of an empty object and then invokes the asynchronous constructor.
Due to the fact that JavaScript introduced a class construct (that replaces all the obscure instructions that I had to perform to simulate an empty object instance with the correct class object as prototype) my abstraction function has mostly lost its value.
Summary of concepts
In my previous blog post, I have given an overview of all covered synchronous programming language concepts and corresponding replacement function abstractions that should be used with non-blocking asynchronous functions.
In this blog post, I will do the same with the concepts covered:
Concept | Synchronous | Asynchronous |
---|---|---|
Function interface | function f(a) { ... } |
async function f(a) { ... } function f(a) { return new Promise(() => {...}); } |
Return statement | return val; |
return val; resolve(val); |
Sequence | a(); b(); ... |
await a(); await b(); ... |
if-then-else | if(condFun()) { thenFun(); } else { elseFun(); } |
if(await condFun()) { await thenFun(); } else { await elseFun(); } |
switch | switch(condFun()) { case "a": funA(); break; case "b": funB(); break; ... } |
switch(await condFun()) { case "a": await funA(); break; case "b": await funB(); break; ... } |
Recursion | function fun() { fun(); } |
function fun(callback) { return new Promise((res, rej) => { setImmediate(function() { return fun(); }); }); } |
while | while(condFun()) { stmtFun(); } |
while(await condFun()) { await stmtFun(); } |
doWhile | do { stmtFun(); } while(condFun()); |
do { await stmtFun(); } while(await condFun()); |
for | for(startFun(); condFun(); stepFun() ) { stmtFun(); } |
for(await startFun(); await condFun(); await stepFun() ) { await stmtFun(); } |
for-in | for(const a in arrFun()) { stmtFun(); } |
for(const a in (await arrFun())) { await stmtFun(); } |
throw | throw err; |
throw err; reject(err); |
try-catch-finally | try { funA(); } catch(err) { funErr(); } finally { funFinally(); } |
try { await funA(); } catch(err) { await funErr(); } finally { await funFinally(); } |
constructor | class C { constructor(a) { this.a = a; } } |
class C { asyncConstructor(a) { return new Promise((res, rej) => { this.a = a; res(); } } } |
new | const obj = new C(a); |
const obj = new C(); await obj.asyncConstructor(a); |
The left column in the table shows all language constructs that are synchronous by default and the right column shows their equivalent asynchronous implementations.
Note that compared to the overview given in my previous blog post, the JavaScript language constructs are not avoided, but used.
With the exceptions of wrapping callback-based function invocations in promises and implementing recursive algorithms, using await usually suffices to retrieve the results of all required sub expressions.
Discussion
In all my previous blog posts that I wrote about asynchronous programming, I was always struck by the fact that most of JavaScript's language constructs were unfit for asynchronous programming. The abstractions that were developed to cope with this problem (e.g. callbacks, coordination libraries, promises etc.) make it possible to get the job done in a reasonable manner, but IMO they always remain somewhat tedious to use and do not prevent you from making many common mistakes.
Using these abstractions remained a common habit for years. With the introduction of the async and await concepts, we finally have a solution that is decent IMO.
Not all problems have been solved with the introduction of these new language features. Callback-based APIs have been a common practice for a very long time, as can be seen in some of my examples. Not all APIs have been converted to promise-based solutions and there are still many APIs and third-party libraries that keep following old practices. Most likely, not all old-fashioned APIs will ever go away.
As a result, we sometimes still have to manually compose promises and do the appropriate conversions from callback APIs. There are also nice facilities that make it possible to conveniently convert callback invocations into promises, but it still remains a responsibility of the programmer.
Another problem (that I often see with programmers that are new to JavaScript), is that they believe that using the async keyword automatically makes their functions non-blocking.
Ensuring that a function does not block still remains the responsibility of the programmer. For example, by making sure that only non-blocking I/O functions are called, or CPU-bound instructions are broken up into smaller pieces.
The async keyword is only an interface solution -- making sure that a function returns a promise and that the await keyword can be used so that a function can stop (by returning) and be resumed at a later point in time.
The JavaScript language does not natively support threads or processes, but APIs have been added to Node.js (worker threads) and web browsers (web workers) to allow code to be executed in a thread. Using these facilities somewhat relieve programmers of the burden to divide long running CPU-bound operations into smaller tasks. Moreover, context-switching is typically much more efficient than cooperative multi-tasking (by generating events and invoking callbacks).
Another problem that remains is calling asynchronous functions from synchronous functions -- there is still no facility in JavaScript that makes it possible to wait for the completion of an asynchronous function in a synchronous function context.
I also want to make a general remark about structured programming -- although the patterns shown in this blog post can be used to prevent that a main loop blocks, structured programming is centered around the idea that you need to execute a step after the completion of another. For long running tasks that do not have a dependency on each other this may not always be the most efficient way of doing things. You may end up waiting for an unnecessary amount of time.
The fact that a promise gives you a reference to a value that will be delivered in the future, also gives you many new interesting abilities. For example, in an application that consists of a separate model and view, you could already start composing a view and provide the promises for the model objects as parameters to the views. Then it is no longer necessary to wait for all model objects to be available before the views can be constructed -- instead, the views can already be rendered and the data can be updated dynamically.
Structured programming patterns are also limiting the ability to efficiently process collections of data -- the repetition patterns in this blog post expect that all data is retrieved before we can iterate over the resulting data collection. It may be more efficient to work with asynchronous iterators that can retrieve data on an element-by-element basis, in which the element that comes first is processed first.
No comments:
Post a Comment