Showing posts with label JavaScript. Show all posts
Showing posts with label JavaScript. Show all posts

Tuesday, July 4, 2023

Using a site map for generating dynamic menus in web applications

In the last few weeks, I have been playing around with a couple of old and obsolete web applications that I have developed in the past with my own web framework. Much of the functionality that these custom web applications offer are facilitated by my framework, but sometimes these web applications also contain significant chunks of custom code.

One of the more interesting features provided by custom code is folding menus (also known as dropdown and dropright menus etc.), that provide a similar experience to the Windows start menu. My guess is that because the start menu experience is so familiar to many users, it remains a frequently used feature by many web applications as of today.

When I still used to actively develop my web framework and many custom web applications (over ten years ago), implementing such a feature heavily relied on JavaScript code. For example, I used the onmouseover attribute on a hyperlink to invoke a JavaScript function that unfolds a panel and the onmouseout attribute to fold a panel again. The onmouseover event handler injects a menu section into the DOM using CSS absolute positioning to put it in the right position on the screen.

I could not use the standard menu rendering functionality of my layout framework, because it deliberately does not rely on the usage of JavaScript. As a consequence, I had to write a custom menu renderer for a web application that requires dynamic menu functionality.

Despite the fact that folding menus are popular and I have implemented them as custom code, I never made it a feature of my layout framework for the following two reasons:

  • I want web applications built around my framework to be as declarative as possible -- this means that I want to concisely express as much as possible what I want to render (a paragraph, an image, a button etc. -- this is something HTML mostly does), rather than specifying in detail how to do it (in JavaScript code). As a result, the usage of JavaScript code in my framework is minimized and non-essential.

    All functionality of the web applications that I developed with my framework must be accessible without JavaScript as much possible.
  • Another property that I appreciate of web technology is the ability to degrade gracefully: the most basic and primary purpose of web applications is to provide information as text.

    Because this property is so important, many non-textual elements, such as an image (img element), provide fallbacks (such as an alt attribute) that simply renders alternative text when graphics capabilities are absent. As a result, it is possible to use more primitive browsers (such as text-oriented browsers) or alternative applications to consume information, such as a text-to-speech system.

    When essential functionality is only exposed as JavaScript code (which more primitive browsers cannot interpret), this property is lost.

Recently, I have discovered that there is a way to implement folding menus that does not rely on the usage of JavaScript.

Moreover, there is also another kind of dynamic menu that has become universally accepted -- the mobile navigation menu (or hamburger menu) making navigation convenient on smaller screens, such as mobile devices.

Because these two types of dynamic menus have become so common, I want to facilitate the implementation of such dynamic menus in my layout framework.

I have found an interesting way to make such use cases possible while retaining the ability to render text and degrade gracefully -- we can use an HTML representation of a site map consisting of a root hyperlink and a nested unordered list as a basis ingredient.

In this blog post, I will explain how implementing these use cases are possible.

The site map feature


As already explained, the basis for implementing these dynamic menus is a textual representation of a site map. Generating site maps is a feature that is already supported by the layout framework:


The above screenshot shows an example page that renders a site map of the entire example web application. In HTML, the site map portion has the following structure:

<a href="/examples/simple/index.php">Home</a>

<ul>
    <li>
        <a href="/examples/simple/index.php/home">Home</a>
    </li>
    <li>
        <a href="/examples/simple/index.php/page1">Page 1</a>
        <ul>
            <li>
                <a href="/examples/simple/index.php/page1/page11">Subpage 1.1</a>
            </li>
            <li>
                <a href="/examples/simple/index.php/page1/page12">Subpage 1.2</a>
            </li>
        </ul>
    </li>
    <li>
        <a href="/examples/simple/index.php/page2">Page 2</a>
        ...
    </li>
    ...
</ul>

The site map, shown in the screenshot and code fragment above, consists of three kinds of links:

  • On top, the root link is displayed that brings the user to the entry page of the web application.
  • The unordered list displays links to all the pages visible in the main menu section that are reachable from the entry page.
  • The nested unordered list displays links to all the pages visible in the sub menu section that are reachable from the selected sub page in the main menu.

With a few simple modifications to my layout framework, I can use a site map as an alternative type of menu section:

  • I have extended the site map generator with the ability to mark selected sub pages and as active, similar to links in menu sections. By adding the active CSS class as an attribute to a hyperlink, a link gets marked as active.
  • I have introduced a SiteMapSection to the layout framework that can be used as a replacement for a MenuSection. A MenuSection displays reachable pages as hyperlinks from a selected page on one level in the page hierarchy, whereas a SiteMap section renders the selected page as a root link and all its visible sub pages and transitive sub pages.

With the following model of a layout:

$application = new Application(
    /* Title */
    "Site map menu website",

    /* CSS stylesheets */
    array("default.css"),

    /* Sections */
    array(
        "header" => new StaticSection("header.php"),
        "menu" => new SiteMapSection(0),
        "contents" => new ContentsSection(true)
    ),

    ...
);

We may render an application with pages that have the following look:


As can be seen in the above screenshot and code fragment, the application layout defines three kinds of sections: a header (a static section displaying a logo), a menu (displaying links to sub pages) and a contents section that displays the content based on the sub page that was selected by the user (in the menu or by opening a URL).

The menu section is displayed as a site map. This site map will be used as the basis for the implementation of the dynamic menus that I have described earlier in this blog post.

Implementing a folding menu


Turning a site map into a folding menu, by using only HTML and CSS, is a relatively straight forward process. To explain the concepts, I can use the following trivial HTML page as a template:


The above page only contains a root link and nested unordered list representing a site map.

In CSS, we can hide the root link and the nested unordered lists by default with the following rules:

/* This rule hides the root link */
body > a
{
    display: none;
}

/* This rule hides nested unordered lists */
ul li ul
{
    display: none;
}

resulting in the following page:


With the following rule, we can make a nested unordered list visible when a user hovers over the surrounding list item:

ul li:hover ul
{
    display: block;
}

Resulting in a web page that behaves as follows:


As can be seen, the unordered list that is placed under the Page 2 link became visible because the user hovers over the surrounding list item.

I can make the menu a bit more fancy if I want to. For example, I can remove the bullet points with the following CSS rule:

ul
{
    list-style-type: none;
    margin: 0;
    padding: 0;
}

I can add borders around the list items to make them appear as buttons:

ul li
{
    border-style: solid;
    border-width: 1px;
    padding: 0.5em;
}

I can horizontally align the buttons by adopting a flexbox layout using the row direction property:

ul
{
    display: flex;
    flex-direction: row;
}

I can position the sub menus right under the buttons of the main menu by using a combination of relative and absolute positioning:

ul li
{
    position: relative;
}

ul li ul
{
    position: absolute;
    top: 2.5em;
    left: 0;
}

Resulting in a menu with the following behaviour:


As can be seen, the trivial example application provides a usable folding menu thanks to the CSS rules that I have described.

In my example application bundled with the layout framework, I have applied all the rules shown above and combined them with the already existing CSS rules, resulting in a web application that behaves as follows:


Displaying a mobile navigation menu


As explained in the introduction, another type of dynamic menu that has been universally accepted is the mobile navigation menu (also known as a hamburger menu). Implementing such a menu, despite its popularity, is challenging IMHO.

Although there seem to be ways to implement such a menu without JavaScript (such as this example using a checkbox) the only proper way to do it IMO is still to use JavaScript. Some browsers have trouble accepting such HTML+CSS-only implementations and it requires the use of an HTML element (an input element) that is not designed for that purpose.

In my example web application, I have implemented a custom JavaScript module, that dynamically transforms a site map (that may have already been displayed as a folding menu) into a mobile navigation menu by performing the following steps:

  • We query the root link of the site map and transform it into a mobile navigation menu button by replacing the text of the root link by an icon image. Clicking on the menu button makes the navigation menu visible or invisible.
  • The first level sub menu becomes visible by adding the CSS class: navmenu_active to the unordered list.
  • The menu button becomes active by adding the CSS class: navmenu_icon_active to the image of the root link.
  • Nested menus can be unfolded or folded. The JavaScript code adds fold icons to each list item of the unordered lists that embed a nested unordered list.
  • Clicking on the fold icon makes the nested unordered list visible or invisible.
  • A nested unordered list becomes visible by adding the CSS class: navsubmenu_active to the unordered list
  • A fold button becomes active by adding the CSS class: navmenu_unfold_active to the fold icon image

It was quite a challenge to implement this JavaScript module, but it does the trick. Moreover, the basis remains a simple HTML-rendered site map that can still be used in text-oriented browsers.

The result of using this JavaScript module is the following navigation menu that has unfoldable sub menus:


Concluding remarks


In this blog post, I have explained a new feature addition to my layout framework: the SiteMapSection that can be used to render menu sections as site maps. Site maps can be used as a basis to implement dynamic menus, such as folding menus and mobile navigation menus.

The benefit of using a site map as a basis ingredient is that a web page still remains useful in its most primitive form: text. As a result, I retain two important requirements of my web framework: declarativity (because a nested unordered list describes concisely what I want) and the ability to degrade gracefully (because it stays useful when it is rendered as text).

Developing folding/navigation menus in the way I described is not something new. There are plenty of examples on the web that show how such features can be developed, such as these W3Schools dropdown menu and mobile navigation menu examples.

Compared to many existing solutions, my approach is somewhat puristic -- I do not abuse HTML elements (such as a check box), I do not rely on using helper elements (such as divs and spans) or helper CSS classes/ids. The only exception is to support dynamic features that are not part of HTML, such as "active links" and the folding/unfolding buttons of the mobile navigation menu.

Although it has become possible to use my framework to implement mobile navigation menus, I still find it sad that I have to rely on JavaScript code to do it properly.

Folding menus, despite their popularity, are nice but the basic one-level menus (that only display a collection of links/buttons of sub pages) are in my opinion fine too and much simpler -- the same implementation is usable on desktops, mobile devices and text-oriented browsers.

With folding menus, I have to test multiple resolutions and devices to check whether they provide the right user experience. Folding menus are useless on mobile devices --- you cannot separately trigger a hover event without generating a click event, making it impossible to unfold a sub menu and peek what is inside.

When it is also desired to provide an optimal mobile device experience, you also need to implement an alternative menu. This requirement makes the implementation of a web application significantly more complex.

Availability


The SiteMapSection has become a new feature of the Java, PHP and JavaScript implementations of my layout framework and can be obtained from my GitHub page.

In addition, I have added a sitemapmenu example web application that displays a site map section in multiple ways:

  • In text mode, it is just displayed as a (textual) site map
  • In graphics mode, when the screen width is 1024 pixels or greater, it displays a horizontal folding menu.
  • In graphics mode, when the screen width is smaller than 1024 pixels and JavaScript is disabled, it displays a vertical folding menu.
  • In graphics mode, when the screen width is smaller than 1024 pixels and JavaScript is enabled, it displays a mobile navigation menu.

Friday, December 30, 2022

A summary of my layout framework improvements

It has been quiet for a while on my blog. In the last couple of months, I have been improving my personal web application framework, after several years of inactivity.

The reason why I became motivated to work on it again, is because I wanted to improve the website of the musical society that I am a member of. This website is still one of the few consumers of my personal web framework.

One of the areas for improvement is the user experience on mobile devices, such as phones and tablets.

To make these improvements possible, I wanted to get rid of complex legacy functionality, such as the "One True Layout" method, that heavily relies on all kinds of interesting hacks that are no longer required in modern browsers. Instead, I wanted to use a flexbox layout that is much more suitable for implementing the layout aspects that I need.

As I have already explained in previous blog posts, my web application framework is not monolithic -- it consists of multiple components each addressing a specific concern. These components can be used and deployed independently.

The most well-explored component is the layout framework that addresses the layout concern. It generates pages from a high-level application model that defines common layout aspects of an application and the pages of which an application consists including their unique content parts.

I have created multiple implementations of this framework in three different programming languages: Java, PHP, and JavaScript.

In this blog post, I will give a summary of all the recent improvements that I made to the layout framework.

Background


As I have already explained in previous blog posts, the layout framework is very straight forward to use. As a developer, you need to specify a high-level application model and invoke a view function to render a sub page belonging to the application. The layout framework uses the path components in a URL to determine which sub page has been selected.

The following code fragment shows an application model for a trivial test web application:

use SBLayout\Model\Application;
use SBLayout\Model\Page\StaticContentPage;
use SBLayout\Model\Page\Content\Contents;
use SBLayout\Model\Section\ContentsSection;
use SBLayout\Model\Section\MenuSection;
use SBLayout\Model\Section\StaticSection;

$application = new Application(
    /* Title */
    "Simple test website",

    /* CSS stylesheets */
    array("default.css"),

    /* Sections */
    array(
        "header" => new StaticSection("header.php"),
        "menu" => new MenuSection(0),
        "contents" => new ContentsSection(true),
    ),

    /* Pages */
    new StaticContentPage("Home", new Contents("home.php"), array(
        "page1" => new StaticContentPage("Page 1", new Contents("page1.php")),
        "page2" => new StaticContentPage("Page 2", new Contents("page2.php")),
        "page3" => new StaticContentPage("Page 3", new Contents("page3.php"))
    ))
);

The above application model captures the following application layout properties:

  • The title of the web application is: "Simple test website" and displayed as part of the title of any sub page.
  • Every page references the same external CSS stylesheet file: default.css that is responsible for styling all pages.
  • Every page in the web application consists of the same kinds of sections:
    • The header element refers to a static header section whose purpose is to display a logo. This section is the same for every sub page.
    • The menu element refers to a MenuSection whose purpose is to display menu links to sub pages that can be reached from the entry page.
    • The contents element refers to a ContentsSection whose purpose is to display contents (text, images, tables, itemized lists etc.). The content is different for each selected page.
  • The application consists of a number of pages:
    • The entry page is a page called: 'Home' and can be reached by opening the root URL of the web application: http://localhost
    • The entry page refers to three sub pages: page1, page2 and page3 that can be reached from the entry page.

      The array keys refer to the path component in the URL that can be used as a selector to open the sub page. For example, http://localhost/page1 will open the page1 sub page and http://localhost/page2 will open the page2 sub page.

The currently selected page can be rendered with the following function invocation:

\SBLayout\View\HTML\displayRequestedPage($application);

By default, the above function generates a simple HTML page in which each section gets translated to an HTML div element:


The above screenshot shows what a page in the application could look like. The grey panel on top is the header that displays the logo, the blue bar is menu section (that displays links to sub pages that are reachable from the entry page), and the black area is the content section that displays the selected content.

One link in the menu section is marked as active to show the user which page in the page hierarchy (page1) has been selected.

Compound sections


Although the framework's functionality works quite well for most of my old use cases, I learned that in order to support flexbox layouts, I need to nest divs, which is something the default HTML code generator: displayRequestedPage() cannot do (as a sidenote: it is possible to create nestings by developing a custom generator).

For example, I may want to introduce another level of pages and add a submenu section to the layout, that is displayed on the left side of the screen.

To make it possible to position the menu bar on the left, I need to horizontally position the submenu and contents sections, while the remaining sections: header and menu must be vertically positioned. To make this possible with flexbox layouts, I need to nest the submenu and contents in a container div.

Since flexbox layouts have become so common nowadays, I have introduced a CompoundSection object, that acts as a generic container element.

With a CompoundSection, I can nest divs:

/* Sections */
array(
    "header" => new StaticSection("header.php"),
    "menu" => new MenuSection(0),
    "container" => new CompoundSection(array(
        "submenu" => new MenuSection(1),
        "contents" => new ContentsSection(true)
    ))
),

In the above code fragment, the container section will be rendered as a container div element containing two sub div elements: submenu and contents. I can use the nested divs structure to vertically and horizontally position the sections in the way that I described earlier.


The above screenshot shows the result of introducing a secondary page hierarchy and a submenu section (that has a red background).

By introducing a container element (through a CompoundSection) it has become possible to horizontally position the submenu next to the contents section.

Easier error handling


Another recurring issue is that most of my applications have to validate user input. When user input is incorrect, a page needs to be shown that displays an error message.

Previously, error handling and error page redirection was entirely the responsibility of the programmer -- it had to be implemented in every controller, which is quite a repetitive process.

In one of my test applications of the layout framework, I have created a page with a form that asks for the user's first and last name:


I wanted to change the example application to return an error message when any of these mandatory attributes were not provided.

To ease that burden, I have made framework's error handling mechanism more generic. Previously, the layout manager only took care of two kinds of errors: when an invalid sub page is requested, a PageNotFoundException is thrown redirecting the user to the 404 error page. When the accessibility criteria have not been met (e.g. a user is not authenticated) a PageForbiddenException is thrown directing the user to the 403 error page.

In the revised version of the layout framework, the PageNotFoundException and PageForbiddenException classes have become sub classes of the generic PageException class. This generic error class makes it possible for the error handler to redirect users to error pages for any HTTP status code.

Error pages should be added as sub pages to the entry page. The numeric keys should match the corresponding HTTP status codes:

/* Pages */
new StaticContentPage("Home", new Contents("home.php"), array(
    "400" => new HiddenStaticContentPage("Bad request", new Contents("error/400.php")),
    "403" => new HiddenStaticContentPage("Forbidden", new Contents("error/403.php")),
    "404" => new HiddenStaticContentPage("Page not found", new Contents("error/404.php"))
    ...
))
I have also introduced a BadRequestException class (that is also a sub class of PageException) that can be used for handling input validation errors.

PageExceptions can be thrown from controllers with a custom error message as a parameter. I can use the following controller implementation to check whether the first and last names were provided:

use SBLayout\Model\BadRequestException;

if($_SERVER["REQUEST_METHOD"] == "POST") // This is a POST request
{
    if(array_key_exists("firstname", $_POST) && $_POST["firstname"] != ""
        && array_key_exists("lastname", $_POST) && $_POST["lastname"] != "")
        $GLOBALS["fullname"] = $_POST["firstname"]." ".$_POST["lastname"];
    else
        throw new BadRequestException("This page requires a firstname and lastname parameter!");
}

The side effect is that if the user forgets to specify any of these mandatory attributes, he gets automatically redirected to the bad request error page:


This improved error handling mechanism significantly reduces the amount of boilerplate code that I need to write in applications that use my layout framework.

Using the iterator protocol for sub pages


As can be seen in the application model examples, some pages in the example applications have sub pages, such as the entry page.

In the layout framework, there are three kinds of pages that may provide sub pages:

  • A StaticContentPage object is a page that may refer to a fixed/static number of sub pages (as an array object).
  • A PageAlias object, that redirects the user to another sub page in the application, also offers the ability to refer users to a fixed/static number of sub pages (as an array object).
  • There is also a DynamicContentPage object in which a sub page can interpret the path component as a dynamic value. That dynamic value can, for example, be used as a parameter for a query that retrieves a record from a database.

In the old implementation of my framework, the code that renders the menu sections always has to treat these objects in a special way to render links to their available sub pages. As a result, I had to use the instanceof operator a lot, which is in a bad code smell.

I have changed the framework to use a different mechanism for stepping over sub pages: iterators or iterables (depending on the implementation language).

The generic Page class (that is the parent class of all page objects) provides a method called: subPageIterator() that returns an iterator/iterable that yields no elements. The StaticContentPage and PageAlias classes override this method to return an interator/iterable that steps over the elements in the array of sub pages.

Using iterators/iterables has a number of nice consequences -- I have eliminated two special cases and a bad code smell (the intensive use of instanceof), significantly improving the quality and readability of my code.

Another nice property is that it is also possible to override this method with a custom iterator, that for example, fetches sub page configurations from a database.

The pagemanager framework (another component in my web framework) offers a content management system giving end-users the ability to change the page structure and page contents. The configuration of the pages is stored in a database.

Although the pagemanager framework uses the layout framework for the construction of pages, it used to rely on custom code to render the menu sections.

By using the iterator protocol, it has become possible to re-use the menu section functionality from the layout framework eliminating the need for custom code. Moreover, it has also become much easier to integrate the pagemanager framework into an application because no additional configuration work is required.

I have also created a gallery application that makes it possible to expose the albums as items in the menu sections. Rendering the menu sections also used to rely on custom code, but thanks to using the iterator protocol that custom code was completely eliminated.

Flexible presentation of menu items


As I have already explained, an application layout can be divided into three kinds of sections. A StaticSection remains the same for any requested sub page, and a ContentSection is filled with content that is unique for the selected page.

In most of my use-cases, it is only required to have a single dynamic content section.

However, the framework is flexible enough to support multiple content sections as well. For example, the following screenshot shows the advanced example application (included with the web framework) in which both the header and the content sections change for each sub page:


The presentation of the third kind of section: MenuSection still used to remain pretty static -- they are rendered as div elements containing hyperlinks. The page that is currently selected is marked as active by using the active class property.

For most of my use-cases, just rendering hyperlinks suffices -- with CSS you can still present them in all kinds of interesting ways, e.g. by changing their colors, adding borders, and changing some its aspects when the user hovers with the mouse cursor over it.

In some rare cases, it may also be desired to present links to sub pages in a completely different way. For example, you may want to display an icon or add extra styling properties to an individual button.

To allow custom presentations of hyperlinks, I have added a new parameter: menuItem to the constructors of page objects. The menuItem parameter refers to a code snippet that decides how to render the link in a menu section:

new StaticContentPage("Icon", new Contents("icon.php"), "icon.php")

In the above example, the last parameter to the constructor, refers to an external file: menuitem/icon.php:

<span>
	<?php
	if($active)
	{
		?>
		<a class="active" href="<?= $url ?>">
			<img src="<?= $GLOBALS["baseURL"] ?>/image/menu/go-home.png" alt="Home icon">
			<strong><?= $subPage->title ?></strong>
		</a>
		<?php
	}
	else
	{
		?>
		<a href="<?= $url ?>">
			<img src="<?= $GLOBALS["baseURL"] ?>/image/menu/go-home.png" alt="Home icon">
			<?= $subPage->title ?>
		</a>
		<?php
	}
	?>
</span>

The above code fragment specifies how a link in the menu section should be displayed when the page is active or not active. We use the custom rendering code to display a home icon before showing the hyperlink.

In the advanced test application, I have added an example page in which every sub menu item is rendered in a custom way:


In the above screenshot, we should see two custom presented menu items in the submenu section on the left. The first has the home icon added and the second uses a custom style that deviates from the normal page style.

If no menuItem parameter was provided, the framework just renders a menu item as a normal hyperlink.

Other functionality


In addition to the new functionality explained earlier, I also made a number of nice small feature additions:


  • A function that displays bread crumbs (the route from the entry page to the currently opened page). The route is derived automatically from the requested URL and application model.
  • A function that displays a site map that shows the hierarchy of pages.
  • A function that makes it possible to embed a menu section in arbitrary sections of a page.

Conclusion


I am quite happy with the recent feature changes that I made to the layout framework. Although I have not done any web front-end development for quite some time, I had quite a bit of fun doing it.

In addition to the fact that useful new features were added, I have also simplified the codebase and improved its quality.

Availability


The Java, PHP and JavaScript implementations of my layout framework can be obtained from my GitHub page. Use them at your own risk!

Monday, February 14, 2022

A layout framework experiment in JavaScript

It has been a while since I wrote a blog post about front-end web technology. The main reason is that I am not extensively doing front-end development anymore, but once in a while I still tinker with it.

In my Christmas break, I wanted to expand my knowledge about modern JavaScript programming practices. To make the learning process more motivating, I have been digging up my old web layout framework project and ported it to JavaScript.

In this blog post, I will explain the rationale of the framework and describe the features of the JavaScript version.

Background


Several years ago, I have elaborated about some of the challenges that I faced while creating layouts for web applications. Although front-end web technology (HTML and CSS) were originally created for pages (not graphical user interfaces), most web applications nowadays are complex information systems that typically have to present collections of data to end-users in a consistent manner.

Although some concepts of web technology are powerful and straight forward, a native way to isolate layout from a page's content and style is still virtually non-existent (with the exception of frames that have been deprecated a long time ago). As a consequence, it has become quite common to rely on custom abstractions and frameworks to organize layouts.

Many years ago, I also found myself repeating the same patterns to implement consistent layouts. To make my life easier, I have developed my own layout framework that allows you to define a model of your application layout, that captures common layout properties and all available sub pages and their dynamic content.

A view function can render a requested sub page, using the path in the provided URL as a selector.

I have created two implementations of the framework: one in Java and another in PHP. The Java version was the original implementation but I ended up using the PHP version the most, because nearly all of the web applications I developed were hosted at shared web hosting providers only offering PHP as a scripting language.

Something that I consider both an advantage and disadvantage of my framework is that it has to generate pages on the server-side. The advantage of this approach is that pages rendered by the framework will work in many browsers, even primitive text-oriented browsers that lack JavaScript support.

A disadvantage is that server-side scripting requires a more complex server installation. Although PHP is relatively simple to set up, a Java Servlet container install (such as Apache Tomcat) is typically more complex. For example, you typically want to put it behind a reverse proxy that serves static content more efficiently.

Furthermore, executing server-side code for each request is also significantly more expensive (in terms of processing power) than serving static files.

The interesting aspect of using JavaScript as an implementation language is that we can use the framework both on the client-side (in the browser) as well as on the server-side (with Node.js). The former aspect makes it possible to host applications on web servers that only serve static content, making web hosting considerably easier and cheaper.

Writing an application model


As explained earlier, my layout framework separates the model from a view. An application layout model can be implemented in JavaScript as follows:

import { Application } from "js-sblayout/model/Application.mjs";

import { StaticSection } from "js-sblayout/model/section/StaticSection.mjs";
import { MenuSection } from "js-sblayout/model/section/MenuSection.mjs";
import { ContentsSection } from "js-sblayout/model/section/ContentsSection.mjs";

import { StaticContentPage } from "js-sblayout/model/page/StaticContentPage.mjs";
import { HiddenStaticContentPage } from "js-sblayout/model/page/HiddenStaticContentPage.mjs";
import { PageAlias } from "js-sblayout/model/page/PageAlias.mjs";

import { Contents } from "js-sblayout/model/page/content/Contents.mjs";

/* Create an application model */

export const application = new Application(
    /* Title */
    "My application",

    /* Styles */
    [ "default.css" ],

    /* Sections */
    {
        header: new StaticSection("header.html"),
        menu: new MenuSection(0),
        submenu: new MenuSection(1),
        contents: new ContentsSection(true)
    },

    /* Pages */
    new StaticContentPage("Home", new Contents("home.html"), {
        "404": new HiddenStaticContentPage("Page not found", new Contents("error/404.html")),

        home: new PageAlias("Home", ""),

        page1: new StaticContentPage("Page 1", new Contents("page1.html"), {
            page11: new StaticContentPage("Subpage 1.1", new Contents("page1/subpage11.html")),
            page12: new StaticContentPage("Subpage 1.2", new Contents("page1/subpage12.html")),
            page13: new StaticContentPage("Subpage 1.3", new Contents("page1/subpage13.html"))
        }),

        page2: new StaticContentPage("Page 2", new Contents("page2.html"), {
            page21: new StaticContentPage("Subpage 2.1", new Contents("page2/subpage21.html")),
            page22: new StaticContentPage("Subpage 2.2", new Contents("page2/subpage22.html")),
            page23: new StaticContentPage("Subpage 2.3", new Contents("page2/subpage23.html"))
        }),
    }),

    /* Favorite icon */
    "favicon.ico"
);

The above source code file (appmodel.mjs) defines an ECMAScript module exporting an application object. The application object defines the layout of a web application with the following properties:

  • The title of the web application is: "My application".
  • All pages use: default.css as a common stylesheet.
  • Every page consists of a number of sections that have a specific purpose:
    • A static section (header) provides content that is the same for every page.
    • A menu section (menu, submenu) display links to sub pages part of the web application.
    • A content section (contents) displays variable content, such as text and images.
  • An application consists of multiple pages that display the same sections. Every page object refers to a file with static HTML code providing the content that needs to be displayed in the content section.
  • The last parameter refers to a favorite icon that is the same for every page.

Pages in the application model are organized in a tree-like data structure. The application constructor only accepts a single page parameter that refers to the entry page of the web application. The entry page can be reached by opening the web application from the root URL or by clicking on the logo displayed in the header section.

The entry page refers to two sub pages: page1, page2. The menu section displays links to the sub pages that are reachable from the entry page.

Every sub page can also refer to their own sub pages. The submenu section will display links to the sub pages that are reachable from a selected the sub page. For example, when page1 is selected the submenu section will display links to: page11, page12.

In addition to pages that are reachable from the menu sections, the application model also has hidden error pages and a home link that is an alias for the entry page. In many web applications, it is a common habit that in addition to clicking on the logo, a home button can also be used to redirect a user to the entry page.

Besides using the links in the menu sections, any sub page in the web application can be reached by using the URL as a selector. A common convention is to use the path components in the URL to determine which page and sub page need to be displayed.

For example, by opening the following URL in a web browser:

http://localhost/page1/page12

Brings the user to the second sub page of the first sub page.

When providing an invalid selector in the URL, such as http://localhost/page4, the framework automatically redirects the user to the 404 error page, because the page cannot be found.

Displaying sub pages in the application model


As explained earlier, to display any of the sub pages that the application model defines, we must invoke a view function.

A reasonable strategy (that should suit most needs) is to generate an HTML page, with a title tag composed the application and page's title, globally include the application and page-level stylesheets, and translate every section to a div using the section identifier as its id. The framework provides a view function that automatically performs this translation.

As a sidenote: for pages that require a more complex structure (for example, to construct a layout with more advanced visualizations), it is also possible to develop a custom view function.

We can create a custom style sheet: default.css to position the divs and give each section a unique color. By using such a stylesheet, the application model shown earlier may be presented as follows:


As can be seen in the screenshot above, the header section has a gray color and displays a logo, the menu section is blue, the submenu is red and the contents section is black.

The second sub page from the first sub page was selected (as can be seen in the URL as well as the selected buttons in the menu sections). The view functions that generate the menu sections automatically mark the selected sub pages as active.

With the Java and PHP versions (described in my previous blog post), it is a common practice to generate all requested pages server-side. With the JavaScript port, we can also use it on the client-side in addition to server-side.

Constructing an application that generates pages server-side


For creating web applications with Node.js, it is a common practice to create an application that runs its own web server.

(As a sidenote: for production environments it is typically recommended to put a more mature HTTP reverse proxy in front of the Node.js application, such as nginx. A reverse proxy is often more efficient for serving static content and has more features with regards to security etc.).

We can construct an application that runs a simple embedded HTTP server:

import { application } from "./appmodel.mjs";
import { displayRequestedPage } from "js-sblayout/view/server/index.mjs";
import { createTestServer } from "js-sblayout/testhttpserver.mjs";

const server = createTestServer(function(req, res, url) {
    displayRequestedPage(req, res, application, url.pathname);
});
server.listen(process.env.PORT || 8080);

The above Node.js application (app.mjs) performs the following steps:

  • It includes the application model shown in the code fragment in the previous section.
  • It constructs a simple test HTTP server that serves well-known static files by looking at common file extensions (e.g. images, stylesheets, JavaScript source files) and treats any other URL pattern as a dynamic request.
  • The embedded HTTP server listens to port 8080 unless a PORT environment variable with a different value was provided.
  • Dynamic URLs are handled by a callback function (last parameter). The callback invokes a view function from the framework that generates an HTML page with all properties and sections declared in the application layout model.

We can start the application as follows:

$ node app.mjs

and then use the web browser to open the root page:

http://localhost:8080

or any sub page of the application, such as the second sub page of the first sub page:

http://localhost:8080/page1/page12

Although Node.js includes a library and JavaScript interface to run an embedded HTTP server, it is very low-level. Its only purpose is to map HTTP requests (e.g. GET, POST, PUT, DELETE requests) to callback functions.

My framework contains an abstraction to construct a test HTTP server with reasonable set of features for testing web applications built with the framework, including serving commonly used static files (such as images, stylesheets and JavaScript files).

For production deployments, there is much more to consider, which is beyond the scope of my HTTP server abstraction.

It is also possible to use the de-facto web server framework for Node.js: express in combination with the layout framework:

import { application } from "./appmodel.mjs";
import { displayRequestedPage } from "js-sblayout/view/server/index.mjs";
import express from "express";

const app = express();
const port = process.env.PORT || 8080;

// Configure static file directories
app.use("/styles", express.static("styles"));
app.use("/image", express.static("image"));

// Make it possible to parse form data
app.use(express.json());
app.use(express.urlencoded({ extended: true }));

// Map all URLs to the SB layout manager
app.get('*', (req, res) => {
    displayRequestedPage(req, res, application, req.url);
});

app.post('*', (req, res) => {
    displayRequestedPage(req, res, application, req.url);
});

// Configure listening port
app.listen(port, () => {
    console.log("Application listening on port " + port);
});

The above application invokes express to construct an HTTP web server that listens to port 8080 by default.

In addition, express has been configured to serve static files from the styles and image folders, and maps all dynamic GET and POST requests to the displayRequestedPage view function of the layout framework.

Using the model client-side and dynamically updating the DOM


As already explained, using JavaScript as an implementation language also makes it possible to directly consume the application model in the browser and dynamically generate pages from it.

To make this possible, we only have to write a very minimal static HTML page:

<!DOCTYPE html>

<html>
    <head>
        <title>My page</title>
        <script type="module">
import { application } from "./appmodel.mjs";
import { initRequestedPage, updateRequestedPage } from "js-sblayout/view/client/index.mjs";

document.body.onload = function() {
    initRequestedPage(application);
};

document.body.onpopstate = function() {
    updateRequestedPage(application);
};
        </script>
    </head>

    <body>
    </body>
</html>

The above HTML page has the following properties:

  • It contains the bare minimum of HTML code to construct a page that is still valid HTML5.
  • We include the application model (shown earlier) that is identical to the application model that we have been using to generate pages server-side.
  • We configure two event handlers. When the page is loaded (onload) we initially render all required page elements in the DOM (including the sections that translate to divs). Whenever the user clicks on a link (onpopstate), we update the affected sections in the DOM.

To make the links in the menu sections work, we have to compose them in a slightly different way -- rather than using the path to derive the selected sub page, we have to use hashes instead.

For example, the second sub page of the first page can be reached by opening the following URL:

http://localhost/index.html#/page1/page21

The popstate event triggers whenever the browser's history changes, and makes it possible for the user to use the back and forward navigation buttons.

Generating dynamic content


In the example application model shown earlier, all sections are made out of static HTML code fragments. Sometimes it may also be desired to generate the sections' content dynamically, for example, to respond to user input.

In addition to providing a string with static HTML code as a parameter, it is also possible to provide a function that generates the content of the section dynamically.

new StaticContentPage("Home", new Contents("home.html"), {
    ...
    hello: new StaticContentPage("Hello 10 times", new Contents(displayHello10Times))
})

In the above code fragment, we have added a new sub page the to entry page that refers to the function: displayHello10Times to dynamically generate content. The purpose of this function is to display the string: "Hello" 10 times:


When writing an application that generates pages server-side, we could implement this function as follows:

function displayHello10Times(req, res) {
    for(let i = 0; i < 10; i++) {
        res.write("<p>Hello!</p>\n");
    }
}

The above function follows a convention that is commonly used by applications using Node.js internal HTTP server:

  • The req parameter refers to the Node.js internal HTTP server's http.IncomingMessage object and can be used to retrieve HTTP headers and other request parameters.
  • The req.sbLayout parameter provides parameters that are related to the layout framework.
  • The res parameter refers to the Node.js internal HTTP server's http.ServerResponse object and can be used to generate a response message.

It is also allowed to declare the function above async or let it return a Promise so that asynchronous APIs can be used.

When developing a client-side application (that dynamically updates the browser DOM), this function should have a different signature:

function displayHello10Times(div, params) {
    let response = "";

    for(let i = 0; i < 10; i++) {
        response += "<p>Hello!</p>\n";
    }

    div.innerHTML = response;
}

In the browser, a dynamic content generation function accepts two parameters:

  • div refers to an HTMLDivElement in the DOM that contains the content of the section.
  • params provides layout framework specific properties (identical to req.sbLayout in the server-side example).

Using a templating engine


Providing functions that generate dynamic content (by embedding HTML code in strings) may not always be the most intuitive way to generate dynamic content. It is also possible to configure template handlers: the framework can invoke a template handler function for files with a certain extension.

In the following server-side example, we define a template handler for files with an .ejs extension to use the EJS templating engine:

import { application } from "./appmodel.mjs";
import { displayRequestedPage } from "js-sblayout/view/server/index.mjs";
import { createTestServer } from "js-sblayout/testhttpserver.mjs";

import * as ejs from "ejs";

function renderEJSTemplate(req, res, sectionFile) {
    return new Promise((resolve, reject) => {
        ejs.renderFile(sectionFile, { req: req, res: res }, {}, function(err, str) {
            if(err) {
                reject(err);
            } else {
                res.write(str);
                resolve();
            }
        });
    });
}

const server = createTestServer(function(req, res, url) {
    displayRequestedPage(req, res, application, url.pathname, {
        ejs: renderEJSTemplate
    });
});
server.listen(process.env.PORT || 8080);

In the above code fragment, the renderEJSTemplate function is used to open an .ejs template file and uses ejs.renderFile function to render the template. The resulting string is propagated as a response to the user.

To use the template handlers, we invoke the displayRequestedPage with an additional parameter that maps the ejs file extension to the template handler function.

In a client-side/browser application, we can define a template handler as follows:

<!DOCTYPE html>

<html>
    <head>
        <title>My page</title>
        <script type="text/javascript" src="ejs.js"></script>
        <script type="module">
import { application } from "./appmodel.mjs";
import { initRequestedPage, updateRequestedPage } from "js-sblayout/view/client/index.mjs";

const templateHandlers = {
  ejs: function(div, response) {
      return ejs.render(response, {});
  }
}

document.body.onload = function() {
    initRequestedPage(application, templateHandlers);
};

document.body.onpopstate = function() {
    updateRequestedPage(application, templateHandlers);
};
        </script>
    </head>

    <body>
    </body>
</html>

In the above code fragment, we define a templateHandlers object that gets propagated to the view function that initially renders the page (initRequestedPage) and dynamically updates the page (updateRequestedPage).

By adding the following sub page to the entry page, we can use an ejs template file to dynamically generate a page rather than a static HTML file or function:

new StaticContentPage("Home", new Contents("home.html"), {
    ...
    stats: new StaticContentPage("Stats", new Contents("stats.ejs"))
})

In a server-side application, we can use stats.ejs to display request variables:

<h2>Request parameters</h2>

<table>
    <tr>
        <th>HTTP version</th>
        <td><%= req.httpVersion %></td>
    </tr>
    <tr>
        <th>Method</th>
        <td><%= req.method %></td>
    </tr>
    <tr>
        <th>URL</th>
        <td><%= req.url %></td>
    </tr>
</table>

resulting in a page that may have the following look:


In a client-side application, we can use stats.ejs to display browser variables:

<h2>Some parameters</h2>

<table>
    <tr>
        <th>Location URL</th>
        <td><%= window.location.href %></td>
    </tr>
    <tr>
        <th>Browser languages</th>
        <td>
        <%
        navigator.languages.forEach(language => {
            %>
            <%= language %><br>
            <%
        });
        %>
        </td>
    </tr>
    <tr>
        <th>Browser code name</th>
        <td><%= navigator.appCodeName %></td>
    </tr>
</table>

displaying the following page:


Strict section and page key ordering


In all the examples shown previously, we have used an Object to define sections and sub pages. In JavaScript, the order of keys in an object is somewhat deterministic but not entirely -- for example, numeric keys will typically appear before keys that are arbitrary strings, regardless of the insertion order.

As a consequence, the order of the pages and sections may not be the same as the order in which the keys are declared.

When the object key ordering is a problem, it is also possible to use iterable objects, such as a nested array, to ensure strict key ordering:

import { Application } from "js-sblayout/model/Application.mjs";

import { StaticSection } from "js-sblayout/model/section/StaticSection.mjs";
import { MenuSection } from "js-sblayout/model/section/MenuSection.mjs";
import { ContentsSection } from "js-sblayout/model/section/ContentsSection.mjs";

import { StaticContentPage } from "js-sblayout/model/page/StaticContentPage.mjs";
import { HiddenStaticContentPage } from "js-sblayout/model/page/HiddenStaticContentPage.mjs";
import { PageAlias } from "js-sblayout/model/page/PageAlias.mjs";

import { Contents } from "js-sblayout/model/page/content/Contents.mjs";

/* Create an application model */

export const application = new Application(
    /* Title */
    "My application",

    /* Styles */
    [ "default.css" ],

    /* Sections */
    [
        [ "header", new StaticSection("header.html") ],
        [ "menu", new MenuSection(0) ],
        [ "submenu", new MenuSection(1) ],
        [ "contents", new ContentsSection(true) ],
        [ 1, new StaticSection("footer.html") ]
    ],

    /* Pages */
    new StaticContentPage("Home", new Contents("home.html"), [
        [ 404, new HiddenStaticContentPage("Page not found", new Contents("error/404.html")) ],

        [ "home", new PageAlias("Home", "") ],

        [ "page1", new StaticContentPage("Page 1", new Contents("page1.html"), [
            [ "page11", new StaticContentPage("Subpage 1.1", new Contents("page1/subpage11.html")) ],
            [ "page12", new StaticContentPage("Subpage 1.2", new Contents("page1/subpage12.html")) ],
            [ "page13", new StaticContentPage("Subpage 1.3", new Contents("page1/subpage13.html")) ]
        ])],

        [ "page2", new StaticContentPage("Page 2", new Contents("page2.html"), [
            [ "page21", new StaticContentPage("Subpage 2.1", new Contents("page2/subpage21.html")) ],
            [ "page22", new StaticContentPage("Subpage 2.2", new Contents("page2/subpage22.html")) ],
            [ "page23", new StaticContentPage("Subpage 2.3", new Contents("page2/subpage23.html")) ]
        ])],
        
        [ 0, new StaticContentPage("Last page", new Contents("lastpage.html")) ]
    ]),

    /* Favorite icon */
    "favicon.ico"
);

In the above example, we have rewritten the application model example to use strict key ordering. We have added a section with numeric key: 1 and a sub page with key: 0. Because we have defined a nested array (instead of an object), these section and page will come last (if we would have used an object, then they will appear first, which is undesired).

Internally, the Application and Page objects use a Map to ensure strict ordering.

More features


The framework has full feature parity with the PHP and Java implementations of the layout framework. In addition to the features described in the previous sections, it can also do the following:

  • Work with multiple content sections. In our examples, we only have one content section that changes when picking a menu item, but it is also possible to have multiple content sections.
  • Page specific stylesheets and JavaScript includes. Besides including CSS stylesheets and JavaScript files globally it can also be done on page level.
  • Using path components as parameters. Instead of selecting a sub page, it is also possible to treat a path component as a parameter and dynamically generate a response.
  • Internationalized pages. Each sub page uses an ISO localization code and the framework will pick the most suitable language in which the page should be displayed by default.
  • Security handlers. Every page can implements its own method that checks whether it should be accessible or not according to a custom security policy.
  • Controllers. It is also possible to process GET or POST parameters before the page gets rendered to decide what to do with them, such as validation.

Conclusion


In this blog post, I have described the features of the JavaScript port of my layout framework. In addition to rendering pages server-side, it can also be directly used in the web browser to dynamically update the DOM. For the latter aspect, it is not required to run any server-side scripting language making application deployments considerably easier.

One of the things I liked about this experiment is that the layout model is sufficiently high-level so that it can be used in a variety of application domains. To make client-side rendering possible, I only had to develop another view function. The implementation of the model aspect is exactly the same for server-side and client-side rendering.

Moreover, the newer features of the JavaScript language (most notably ECMAScript modules) make it much easier to reuse code between Node.js and web browsers. Before ECMAScript modules were adopted by browser vendors, there was no module system in the browser at all (Node.js has CommonJS) forcing me to implement all kinds of tricks to make a reusable implementation between Node.js and browsers possible.

As explained in the introduction of this blog post, web front-end technologies do not have a separated layout concern. A possible solution to cope with this limitation is to generate pages server-side. With the JavaScript implementation this is no longer required, because it can also be directly done in the browser.

However, this does still not fully solve my layout frustrations. For example, dynamically generated pages are poorly visible to search engines. Moreover, a dynamically rendered web application is useless to users that have JavaScript disabled, or a web browser that does not support JavaScript, such as text browsers.

Using JavaScript also breaks the declarative nature of web applications -- HTML and CSS allow you to write what the structure and style of a page without specifying how to render it. This has all kinds of advantages, such as the ability to degrade gracefully when certain features cannot be used, such as graphics. With JavaScript some of these properties are lost.

Still, this project was a nice distraction -- I already had the idea to explore this for several years. During the COVID-19 pandemic, I have read quite a few technical books, such as JavaScript: The Definitive Guide and learned that with the introduction of new language JavaScript features, such as ECMAScript modules, it would be possible to exactly the same implementation of the model both server-side and client-side.

As explained in my blog reflection over 2021, I have been overly focused on a single goal for almost two years and it started to negatively affect my energy level. This project was a nice short distraction.

Future work


I have also been investigating whether I could use my framework to create offline web applications with a consistent layout. Unfortunately, it does not seem to be very straight forward to do that.

It seems that it is not allowed to do any module imports from local files for security reasons. In theory, this restriction can be bypassed by packing up all the modules into a single JavaScript include with webpack.

However, it turns out that there is another problem -- it is also not possible to open any files from the local drive for security reasons. There is a file system access API in development, that is still not finished or mature yet.

Some day, when these APIs have become more mature, I may revisit this problem and revise my framework to also make offline web applications possible.

Availability


The JavaScript port of my layout framework can be obtained from my GitHub page. To use this framework client-side, a modern web browser is required, such as Mozilla Firefox or Google Chrome.

Tuesday, January 11, 2022

Structured asynchronous programming revisited (Asynchronous programming with JavaScript part 5)

It has been a while since I wrote a JavaScript related blog post. In my previous job, I was using it on a daily basis, but in the last few years I have been using it much less frequently.

One of the reasons that I wrote so many JavaScript-related blog posts is because the language used to have many catches, such as:

  • Scoping. Contrary to many other mainstream programming languages, JavaScript uses function-level scoping as opposed to block-level scoping. Syntactically, function-level scoping looks very similar to block-level scoping.

    Function-level scoping has a number of implications that could have severe consequences. For example, you may unintentionally re-assign values.
  • Simulating class-based inheritance. JavaScript supports Object Oriented programming with prototypes rather than classes (that most mainstream Object Oriented programming languages use). It is possible to use prototypes to simulate classes and class-based inheritance.

    Although I consider prototypes to be conceptually simple, using them in JavaScript used to be quite confusing. As a consequence, simulating class inheritance also used to be quite difficult.
  • Asynchronous programming. JavaScript is originally designed for use in web browsers, but has also become quite popular outside the browser, such as Node.js, to write server/command-line applications. For both browser usage as well as server applications, it is often desired to do multiple tasks concurrently.

    Unfortunately, most of JavaScript's language constructs are synchronous, making such tasks quite difficult without using any software abstractions.

In particular about the last topic: asynchronous programming, I wrote many blog posts. I have elaborated about callbacks in Node.js and abstraction libraries to do coordination and another popular abstraction: promises.

I have also argued that most of JavaScript's language constructs, that implement structured programming concepts, are synchronous and cannot be used with asynchronous functions that return (almost) immediately, and resume their executions with callbacks.

I have built my own library: slasp, that implements asynchronous equivalents for the synchronous language constructs that you should not use.

Fortunately, much has happened since I wrote that blog post. The JavaScript language has many new features (part of the ECMAScript 6 standard) that have become a standard practice nowadays, such as:

  • Block level scoping. Block scoped immutable values can be declared with: const and mutable values with: let.
  • An object with a custom prototype can now be directly created with: Object.create.
  • A class construct was added that makes it possible to define classes (that are simulated with prototypes).

Moreover, modern JavaScript also has new constructs and APIs for asynchronous programming, making most of the software abstractions that I have elaborated about obsolete for the most part.

Recently, I have been using these new constructs quite intensively and learned that my previous blog posts (that I wrote several years ago) are still mostly about old practices.

In this blog post, I will revisit the structured programming topic and explain how modern language constructs can be used to implement these concepts.

Asynchronous programming in JavaScript


As explained in my previous blog posts, asynchronous programming in JavaScript is important for a variety of reasons. Some examples are:

  • Animating objects and providing other visual effects in a web browser, while keeping the browser responsive so that it can respond to user events, such as mouse clicks.
  • The ability to serve multiple clients concurrently in a Node.js server application.

Multi-tasking in JavaScript is (mainly) cooperative. The idea is that JavaScript code runs in a hidden main loop that responds to events in a timely manner, such as user input (e.g. mouse clicks) or incoming connections.

To keep your application responsive, it is required that the execution of a code block does not take long (to allow the application to respond to other events), and that an event is generated to allow the execution to be resumed at a later point in time.

Not meeting this requirement may cause the web browser or your server application to block, which is often undesirable.

Because writing non-blocking code is so important, many functions in the Node.js API are asynchronous by default: they return (almost) immediately and use a callback function parameter that gets invoked when the work is done.

For example, reading a file from disk while keeping the application responsive can be done as follows:

const fs = require('fs');

fs.readFile("hello.txt", function(err, contents) {
    if(err) {
        console.error(err);
        process.exit(1);
    } else {
        console.log(contents);
    }
});

Note that in the above code fragment, instead of relying on the return value of the fs.readFile call, we provide a callback function as a parameter that gets invoked when the operation has finished. The callback is responsible for displaying the file's contents or the resulting error message.

While the file is being read (that happens in the background), the event loop is still able to process other events making it possible for the application to work on other tasks concurrently.

To ensure that an application is responsive and scalable, I/O related functionality in the Node.js API is asynchronous by default. For some functions there are also synchronous equivalents for convenience, but as a rule of thumb they should be avoided as much as possible.

Asynchronous I/O is an important ingredient in making Node.js applications scalable -- because I/O operations are typically several orders of magnitude slower than CPU operations, the application should remain responsive as long as no callback takes long to complete. Furthermore, because there is no thread per connection model, there is no context-switching and memory overhead for each concurrent task.

However, asynchronous I/O operations and a callback-convention does not guarantee that the main loop never gets blocked.

When implementing tasks that are heavily CPU-bound (such as recursively computing a Fibonacci number), the programmer has to make sure that the execution does not block the main loop too long (for example, by dividing it into smaller tasks that generate events, or using threads).

Code structuring issues


Another challenge that comes with asynchronous functions is that it becomes much harder to keep your code structured and maintainable.

For example, if we want to create a directory, then write a text file to it, and then read the text file, and only use non-blocking functions to keep the application responsive, we may end up writing:

const fs = require('fs');

fs.mkdir("test", function(err) {
    if(err) {
        console.error(err);
        process.exit(1);
    } else {
        fs.writeFile("hello.txt", "Hello world!", function(err) {
            if(err) {
                console.error(err);
                process.exit(1);
            } else {
                fs.readFile("hello.txt", function(err, contents) {
                    if(err) {
                        console.error(err);
                        process.exit(1);
                    } else {
                        // Displays: "Hello world!"
                        console.log(contents);
                    }
                });
            }
        });
    }
});

As may be observed, for each function call, we define a callback responsible for checking the status of the call and executing the next step. For each step, we have to nest another callback function, resulting in pyramid code.

The code above is difficult to read and maintain. If we want to add another step in the middle, we are forced to refactor the callback nesting structure, which is labourious and tedious.

Because code structuring issues are so common, all kinds of software abstractions have been developed to coordinate the execution of tasks. For example, we can use the async library to rewrite the above code fragment as follows:

const async = require('async');

async.waterfall([
    function(callback) {
        fs.mkdir("test", callback);
    },
    
    function(callback) {
        fs.writeFile("hello.txt", "Hello world!", callback);
    },
    
    function(callback) {
        fs.readFile("hello.txt", callback);
    },
    
    function(contents, callback) {
        // Displays: "Hello world!"
        console.log(contents);
        callback();
    }
], function(err) {
    if(err) {
        console.error(err);
        process.exit(1);
    }
});

The async.waterfall abstraction flattens the code, allows us to conveniently add additional asynchronous steps and change the order, if desired.

Promises


In addition to Node.js-style callback functions and abstraction libraries for coordination, a more powerful software abstraction was developed: promises (to be precise: there are several kinds of promise abstractions developed, but I am referring to the Promises/A+ specification).

Promises have become very popular, in particular for APIs that are used in the browser. As a result, they have been accepted into the core JavaScript API.

With promises the idea is that every asynchronous function quickly returns a promise object that can be used as a reference to a value that will be delivered at some point in the future.

For example, we can wrap the function invocation that reads a text file into a function that returns promise:

const fs = require('fs');

function readHelloFile() {
    return new Promise((resolve, reject) => {
       fs.readFile("hello.txt", function(err, contents) {
           if(err) {
               reject(err);
           } else {
               resolve(contents);
           }
       });
    });
}

The above function: readHelloFile invokes fs.readFile from the Node.js API to read the hello.txt file and returns a promise. In case the file was successfully read, the promise is resolved and the file's contents is propagated as a result. In case of an error, the promise is rejected with the resulting error message.

To retrieve and display the result, we can invoke the above function as follows:

readHelloFile().then(function(contents) {
    console.log(contents);
}, function(err) {
    console.error(err);
    process.exit(1);
});

Invoking the then method causes the main event loop to invoke either the resolve (first parameter) or reject callback function (second parameter) when the result is available.

Because promises have become part of the ECMAScript standard, Node.js has introduced alternative APIs that are promise-based, instead of callback based (such as for filesystem operations: fs.promises).

By using the promises-based API for filesystem operations, we can simplify the previous example to:

const fs = require('fs').promises;

fs.readFile("hello.txt").then(function(contents) {
    console.log(contents);
}, function(err) {
    console.error(err);
    process.exit(1);
});

As described in my old blog post about promises -- they are considered more powerful than callbacks. A promise provides you a reference to a value that is in the process of being delivered. Callbacks can only give you insights in the status of a background task as soon as it completes.

Although promises have an advantage over callbacks, both approaches still share the same drawback -- we are forced to avoid most JavaScript language constructs and use alternative function abstractions.

In modern JavaScript, it is also no longer necessary to always explicitly create promises. Instead, we can also declare a function as async. Simply returning a value or throwing exception in such a function automatically ensures that a promise is returned:

async function computeSum(a, b) {
    return a + b;
}

The function above returns the sum of the provided input parameters. The JavaScript runtime automatically wraps its execution into a promise that can be used to retrieve the return value at some point in the future.

(As a sidenote: the function above returns a promise, but is still blocking. It does not generate an event that can be picked up by the event loop and a callback that can resume its execution at a later point in time.)

The result of executing the following code:

const result = computeSum(1, 1);
console.log("The result is: " + result);

is a promise object, not a numeric value:

Result is: [object Promise]

When a function returns a promise, we also no longer have to invoke .then() and provide callbacks as parameters to retrieve the result or any thrown errors. The await keyword can be used to automatically wait for a promise to yield its return value or an exception, and then move to the next statement:

(async function() {
    const result = await computeSum(1, 1);
    console.log("The result is: " + result); // The result is: 2
})();

The only catch is that you can only use await in the scope of an asynchronous function. The default scope of a Node.js program is synchronous. As a result, we have to wrap the code into an asynchronous function block.

By using the promise-based fs API and the new language features, we can rewrite our earlier callback-based example (that creates a directory, writes and reads a file) as follows:

const fs = require('fs').promises;

(async function() {
    try {
        await fs.mkdir("test");
        await fs.writeFile("hello.txt", "Hello world!");
        const contents = await fs.readFile("hello.txt");
    } catch(err) {
        console.error(err);
        process.exit(1);
    }
})();

As may be observed, the code is much simpler than manually orchestrating promises.

Structured programming concepts


In my previous blog post, I have argued that most of JavaScript's language constructs (that implement structured programming concepts) cannot be directly used in combination with non-blocking functions (that return almost immediately and require callback functions as parameters).

As a personal exercise, I have created function abstractions that are direct asynchronous equivalents for all these structured programming concepts that should be avoided and added them to my own library: slasp.

By combining promises, async functions and await statements, these function abstractions have mostly become obsolete.

In this section, I will go over the structured programming concepts I covered in my old blog post and show their direct asynchronous programming equivalents using modern JavaScript language constructs.

Function definitions


As I have already explained in my previous blog post, the most basic thing one can do in JavaScript is executing statements, such as variable assignments or function invocations. This used to be already much different when moving from a synchronous programming to an asynchronous programming world.

As a trivial example, I used a synchronous function whose only purpose is to print text on the console:

function printOnConsole(value) {
    console.log(value);
}

The above example is probably too trivial, but it is still possible to make it non-blocking -- we can generate a tick event so that the function returns immediately and use a callback parameter so that the task will be resumed at a later point in time:

function printOnConsole(value) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            console.log(value);
            resolve();
        });
    });
}

To follow modern JavaScript practices, the above function is wrapped into a constructor that immediately returns a promise that can be used as a reference to determine when the task was completed.

(As a sidenote: we compose a regular function that returns a promise. We cannot define an async function, because the process.nextTick is an asynchronous function that requires a callback function parameter. The callback is responsible for propagating the end result. Using a return only causes the callback function to return and not the enclosing function.)

I have also shown that for functions that return a value, the same principle can be applied. As an example, I have used a function that translates a numeric digit into a word:

function generateWord(digit) {
    const words = [ "zero", "one", "two", "three", "four",
        "five", "six", "seven", "eight", "nine" ];
    return words[digit];
}

We can also make this function non-blocking by generating a tick event and wrapping it into a promise:

function generateWord(digit) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            const words = [ "zero", "one", "two", "three", "four", "five",
                "six", "seven", "eight", "nine" ];
            resolve(words[digit]);
        });
    });
}

Sequential decomposition


The first structured programming concept I elaborated about was sequential decomposition in which a number of statements are executed in sequential order.

I have shown a trivial example that adds 1 to a number, then converts the resulting digit into a word, and finally prints the word on the console:

const a = 1;
const b = a + 1;
const number = generateWord(b);
printOnConsole(number); // two

With the introduction of the await keyword, converting the above code to use the asynchronous implementations of all required functions has become straight forward:

(async function() {
    const a = 1;
    const b = a + 1;
    const number = await generateWord(b);
    await printOnConsole(number); // two
})();

The above example is a one-on-one port of its synchronous counterpart -- we just have to use the await keyword in combination with our asynchronous function invocations (that return promises).

The only unconventional aspect is that we need to wrap the code inside an asynchronous function block to allow the await keyword to be used.

Alteration


The second programming concept that I covered is alteration that is used to specify conditional statements.

I gave a simple example that checks whether a given name matches my own name:

function checkMe(name) {
    return (name == "Sander");
}
    
const name = "Sander";
    
if(checkMe(name)) {
    printOnConsole("It's me!");
    printOnConsole("Isn't it awesome?");
} else {
    printOnConsole("It's someone else!");
}

It is also possible to make the checkMe function non-blocking by generating a tick event and wrapping it into a promise:

function checkMe(name) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            resolve(name == "Sander");
        });
    });
}

To invoke the asynchronous function shown above inside the if-statement, we only have to write:

(async function() {
    const name = "Sander";

    if(await checkMe(name)) {
        await printOnConsole("It's me!");
        await printOnConsole("Isn't it awesome?");
    } else {
        await printOnConsole("It's someone else!");
    }
})();

In my previous blog post, I was forced to abolish the regular if-statement and use an abstraction (slasp.when) that invokes the non-blocking function first, then uses the callback to retrieve the result for use inside an if-statement. In the above example, the only subtle change I need to make is to use await inside the if-statement.

I can also do the same thing for the other alteration construct: the switch -- just using await in the conditional expression and the body should suffice.

Repetition


For the repetition concept, I have shown an example program that implements the Gregory-Leibniz formula to approximate PI up to 6 digits:

function checkTreshold(approx) {
    return (approx.toString().substring(0, 7) != "3.14159");
}

let approx = 0;
let denominator = 1;
let sign = 1;

while(checkTreshold(approx)) {
    approx += 4 * sign / denominator;
    printOnConsole("Current approximation is: "+approx);

    denominator += 2;
    sign *= -1;
}

As with the previous example, we can also make the checkTreshold function non-blocking:

function checkTreshold(approx) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            resolve(approx.toString().substring(0, 7) != "3.14159");
        });
    });
}

In my previous blog post, I have explained that the while statement is unfit for executing non-blocking functions in sequential order, because they return immediately and have to resume their execution at a later point in time.

As with the alteration language constructs, I have developed a function abstraction that is equivalent to the while statement (slasp.whilst), making it possible to have a non-blocking conditional check and body.

With the introduction of the await statement, this abstraction function also has become obsolete. We can rewrite the code as follows:

(async function() {
    let approx = 0;
    let denominator = 1;
    let sign = 1;

    while(await checkTreshold(approx)) {
        approx += 4 * sign / denominator;
        await printOnConsole("Current approximation is: "+approx);

        denominator += 2;
        sign *= -1;
    }
})();

As can be seen, the above code is a one-on-one port of its synchronous counterpart.

The function abstractions for the other repetition concepts: doWhile, for, for-in have also become obsolete by using await for evaluating the non-blocking conditional expressions and bodies.

Implementing non-blocking recursive algorithms still remains tricky, such as the following (somewhat inefficient) recursive algorithm to compute a Fibonacci number:

function fibonacci(n) {
    if (n < 2) {
        return 1;
    } else {
        return fibonacci(n - 2) + fibonacci(n - 1);
    }
}

const result = fibonacci(20);
printOnConsole("20th element in the fibonacci series is: "+result);

The above algorithm is mostly CPU-bound and takes some time to complete. As long as it is computing, the event loop remains blocked causing the entire application to become unresponsive.

To make sure that the execution does not block for too long by using cooperative multi-tasking principles, we should regularly generate events, suspend its execution (so that the event loop can process other events) and use callbacks to allow it to resume at a later point in time:

function fibonacci(n) {
    return new Promise((resolve, reject) => {
        if (n < 2) {
            setImmediate(function() {
                resolve(1);
            });
        } else {
            let first;
            let second;

            fibonacci(n - 2)
            .then(function(result) {
                first = result;
                return fibonacci(n - 1);
            })
            .then(function(result) {
                second = result;
                resolve(first + second);
            });
        }
    });
}

(async function() {
    const result = await fibonacci(20);
    await printOnConsole("20th element in the fibonacci series is: "+result);
})();

In the above example, I made the algorithm non-blocking by generating a macro-event with setImmediate for the base step. Because the function returns a promise, and cannot be wrapped into an async function, I have to use the promises' then methods to retrieve the return values of the computations in the induction step.

Extensions


In my previous blog post, I have also covered the extensions to structured programming that JavaScript provides.

Exceptions


I have also explained that with asynchronous functions, we cannot use JavaScript's throw, and try-catch-finally language constructs, because exceptions are typically not thrown instantly but at a later point in time.

With await, using these constructs is also no longer a problem.

For example, I can modify our generateWord example to throw an exception when the provided number is not between 0 and 9:

function generateWord(num) {
    if(num < 0 || num > 9) {
        throw "Cannot convert "+num+" into a word";
    } else {
        const words = [ "zero", "one", "two", "three", "four", "five",
            "six", "seven", "eight", "nine" ];
        return words[num];
    }
}

try {
    let word = generateWord(1);
    printOnConsole("We have a: "+word);
    word = generateWord(10);
    printOnConsole("We have a: "+word);
} catch(err) {
    printOnConsole("Some exception occurred: "+err);
} finally {
    printOnConsole("Bye bye!");
}

We can make generateWord an asynchronous function by converting it in the usual way:

function generateWord(num) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            if(num < 0 || num > 9) {
                reject("Cannot convert "+num+" into a word");
            } else {
                const words = [ "zero", "one", "two", "three", "four", "five",
                    "six", "seven", "eight", "nine" ];
                resolve(words[num]);
            }
        });
    });
}

(async function() {
    try {
        let word = await generateWord(1);
        await printOnConsole("We have a: "+word);
        word = await generateWord(10);
        await printOnConsole("We have a: "+word);
    } catch(err) {
        await printOnConsole("Some exception occurred: "+err);
    } finally {
        await printOnConsole("Bye bye!");
    }
})();

As can be seen in the example above, thanks to the await construct, we have not lost our ability to use try/catch/finally.

Objects


Another major extension is object-oriented programming. As explained in an old blog post about object oriented programming in JavaScript, JavaScript uses prototypes rather than classes, but prototypes can still be used to simulate classes and class-based inheritance.

Because simulating classes is such a common use-case, a class construct was added to the language that uses prototypes to simulate it.

The following example defines a Rectangle class with a method that can compute a rectangle's area:

class Rectangle {
    constructor(width, height) {
        this.width = width;
        this.height = height;
    }

    calculateArea() {
        return this.width * this.height;
    }
}

const r = new Rectangle(2, 2);

printOnConsole("Area is: "+r.calculateArea());

In theory, it is also possible that the construction of an object takes a long time and should be made non-blocking.

Although JavaScript does not have a language concept to do asynchronous object construction, we can still do it by making a couple of small changes:

class Rectangle {
    asyncConstructor(width, height) {
        return new Promise((resolve, reject) => {
            process.nextTick(() => {
                this.width = width;
                this.height = height;
                resolve();
            });
        });
    }

    calculateArea() {
        return this.width * this.height;
    }
}

(async function() {
    const r = new Rectangle();
    await r.asyncConstructor(2, 2);
    await printOnConsole("Area is: "+r.calculateArea());
})();

As can be seen in the above example, the constructor function has been replaced with an asyncConstructor method that implements the usual strategy to make it non-blocking.

To asynchronously construct the rectangle, we first construct an empty object using the Rectangle class object as its prototype. Then we invoke the asynchronous constructor to initialize the object in a non-blocking way.

In my previous blog post, I have developed an abstraction function that could be used as an asynchronous replacement for JavaScript's new operator (slasp.novel) that performs the initialization of an empty object and then invokes the asynchronous constructor.

Due to the fact that JavaScript introduced a class construct (that replaces all the obscure instructions that I had to perform to simulate an empty object instance with the correct class object as prototype) my abstraction function has mostly lost its value.

Summary of concepts


In my previous blog post, I have given an overview of all covered synchronous programming language concepts and corresponding replacement function abstractions that should be used with non-blocking asynchronous functions.

In this blog post, I will do the same with the concepts covered:

Concept Synchronous Asynchronous
Function interface
function f(a) { ... }
async function f(a) { ... }

function f(a) {
    return new Promise(() => {...});
}
Return statement
return val;
return val;
resolve(val);
Sequence
a(); b(); ...
await a(); await b(); ...
if-then-else
if(condFun()) {
    thenFun();
} else {
    elseFun();
}
if(await condFun()) {
    await thenFun();
} else {
    await elseFun();
}
switch
switch(condFun()) {
    case "a":
        funA();
        break;
    case "b":
        funB();
        break;
    ...
}
switch(await condFun()) {
    case "a":
        await funA();
        break;
    case "b":
        await funB();
        break;
    ...
}
Recursion
function fun() {
    fun();
}
function fun(callback) {
    return new Promise((res, rej) => {
        setImmediate(function() {
            return fun();
        });
    });
}
while
while(condFun()) {
    stmtFun();
}
while(await condFun()) {
    await stmtFun();
}
doWhile
do {
    stmtFun();
} while(condFun());
do {
    await stmtFun();
} while(await condFun());
for
for(startFun();
    condFun();
    stepFun()
) {
    stmtFun();
}
for(await startFun();
    await condFun();
    await stepFun()
) {
    await stmtFun();
}
for-in
for(const a in arrFun()) {
    stmtFun();
}
for(const a in (await arrFun())) {
    await stmtFun();
}
throw
throw err;
throw err;
reject(err);
try-catch-finally
try {
    funA();
} catch(err) {
    funErr();
} finally {
    funFinally();
}
try {
    await funA();
} catch(err) {
    await funErr();
} finally {
    await funFinally();
}
constructor
class C {
    constructor(a) {
        this.a = a;
    }
}
class C {
    asyncConstructor(a) {
        return new Promise((res, rej) => {
            this.a = a;
            res();
        }
    }
}
new
const obj = new C(a);
const obj = new C();
await obj.asyncConstructor(a);

The left column in the table shows all language constructs that are synchronous by default and the right column shows their equivalent asynchronous implementations.

Note that compared to the overview given in my previous blog post, the JavaScript language constructs are not avoided, but used.

With the exceptions of wrapping callback-based function invocations in promises and implementing recursive algorithms, using await usually suffices to retrieve the results of all required sub expressions.

Discussion


In all my previous blog posts that I wrote about asynchronous programming, I was always struck by the fact that most of JavaScript's language constructs were unfit for asynchronous programming. The abstractions that were developed to cope with this problem (e.g. callbacks, coordination libraries, promises etc.) make it possible to get the job done in a reasonable manner, but IMO they always remain somewhat tedious to use and do not prevent you from making many common mistakes.

Using these abstractions remained a common habit for years. With the introduction of the async and await concepts, we finally have a solution that is decent IMO.

Not all problems have been solved with the introduction of these new language features. Callback-based APIs have been a common practice for a very long time, as can be seen in some of my examples. Not all APIs have been converted to promise-based solutions and there are still many APIs and third-party libraries that keep following old practices. Most likely, not all old-fashioned APIs will ever go away.

As a result, we sometimes still have to manually compose promises and do the appropriate conversions from callback APIs. There are also nice facilities that make it possible to conveniently convert callback invocations into promises, but it still remains a responsibility of the programmer.

Another problem (that I often see with programmers that are new to JavaScript), is that they believe that using the async keyword automatically makes their functions non-blocking.

Ensuring that a function does not block still remains the responsibility of the programmer. For example, by making sure that only non-blocking I/O functions are called, or CPU-bound instructions are broken up into smaller pieces.

The async keyword is only an interface solution -- making sure that a function returns a promise and that the await keyword can be used so that a function can stop (by returning) and be resumed at a later point in time.

The JavaScript language does not natively support threads or processes, but APIs have been added to Node.js (worker threads) and web browsers (web workers) to allow code to be executed in a thread. Using these facilities somewhat relieve programmers of the burden to divide long running CPU-bound operations into smaller tasks. Moreover, context-switching is typically much more efficient than cooperative multi-tasking (by generating events and invoking callbacks).

Another problem that remains is calling asynchronous functions from synchronous functions -- there is still no facility in JavaScript that makes it possible to wait for the completion of an asynchronous function in a synchronous function context.

I also want to make a general remark about structured programming -- although the patterns shown in this blog post can be used to prevent that a main loop blocks, structured programming is centered around the idea that you need to execute a step after the completion of another. For long running tasks that do not have a dependency on each other this may not always be the most efficient way of doing things. You may end up waiting for an unnecessary amount of time.

The fact that a promise gives you a reference to a value that will be delivered in the future, also gives you many new interesting abilities. For example, in an application that consists of a separate model and view, you could already start composing a view and provide the promises for the model objects as parameters to the views. Then it is no longer necessary to wait for all model objects to be available before the views can be constructed -- instead, the views can already be rendered and the data can be updated dynamically.

Structured programming patterns are also limiting the ability to efficiently process collections of data -- the repetition patterns in this blog post expect that all data is retrieved before we can iterate over the resulting data collection. It may be more efficient to work with asynchronous iterators that can retrieve data on an element-by-element basis, in which the element that comes first is processed first.