Tuesday, January 11, 2022

Structured asynchronous programming revisited (Asynchronous programming with JavaScript part 5)

It has been a while since I wrote a JavaScript related blog post. In my previous job, I was using it on a daily basis, but in the last few years I have been using it much less frequently.

One of the reasons that I wrote so many JavaScript-related blog posts is because the language used to have many catches, such as:

  • Scoping. Contrary to many other mainstream programming languages, JavaScript uses function-level scoping as opposed to block-level scoping. Syntactically, function-level scoping looks very similar to block-level scoping.

    Function-level scoping has a number of implications that could have severe consequences. For example, you may unintentionally re-assign values.
  • Simulating class-based inheritance. JavaScript supports Object Oriented programming with prototypes rather than classes (that most mainstream Object Oriented programming languages use). It is possible to use prototypes to simulate classes and class-based inheritance.

    Although I consider prototypes to be conceptually simple, using them in JavaScript used to be quite confusing. As a consequence, simulating class inheritance also used to be quite difficult.
  • Asynchronous programming. JavaScript is originally designed for use in web browsers, but has also become quite popular outside the browser, such as Node.js, to write server/command-line applications. For both browser usage as well as server applications, it is often desired to do multiple tasks concurrently.

    Unfortunately, most of JavaScript's language constructs are synchronous, making such tasks quite difficult without using any software abstractions.

In particular about the last topic: asynchronous programming, I wrote many blog posts. I have elaborated about callbacks in Node.js and abstraction libraries to do coordination and another popular abstraction: promises.

I have also argued that most of JavaScript's language constructs, that implement structured programming concepts, are synchronous and cannot be used with asynchronous functions that return (almost) immediately, and resume their executions with callbacks.

I have built my own library: slasp, that implements asynchronous equivalents for the synchronous language constructs that you should not use.

Fortunately, much has happened since I wrote that blog post. The JavaScript language has many new features (part of the ECMAScript 6 standard) that have become a standard practice nowadays, such as:

  • Block level scoping. Block scoped immutable values can be declared with: const and mutable values with: let.
  • An object with a custom prototype can now be directly created with: Object.create.
  • A class construct was added that makes it possible to define classes (that are simulated with prototypes).

Moreover, modern JavaScript also has new constructs and APIs for asynchronous programming, making most of the software abstractions that I have elaborated about obsolete for the most part.

Recently, I have been using these new constructs quite intensively and learned that my previous blog posts (that I wrote several years ago) are still mostly about old practices.

In this blog post, I will revisit the structured programming topic and explain how modern language constructs can be used to implement these concepts.

Asynchronous programming in JavaScript


As explained in my previous blog posts, asynchronous programming in JavaScript is important for a variety of reasons. Some examples are:

  • Animating objects and providing other visual effects in a web browser, while keeping the browser responsive so that it can respond to user events, such as mouse clicks.
  • The ability to serve multiple clients concurrently in a Node.js server application.

Multi-tasking in JavaScript is (mainly) cooperative. The idea is that JavaScript code runs in a hidden main loop that responds to events in a timely manner, such as user input (e.g. mouse clicks) or incoming connections.

To keep your application responsive, it is required that the execution of a code block does not take long (to allow the application to respond to other events), and that an event is generated to allow the execution to be resumed at a later point in time.

Not meeting this requirement may cause the web browser or your server application to block, which is often undesirable.

Because writing non-blocking code is so important, many functions in the Node.js API are asynchronous by default: they return (almost) immediately and use a callback function parameter that gets invoked when the work is done.

For example, reading a file from disk while keeping the application responsive can be done as follows:

const fs = require('fs');

fs.readFile("hello.txt", function(err, contents) {
    if(err) {
        console.error(err);
        process.exit(1);
    } else {
        console.log(contents);
    }
});

Note that in the above code fragment, instead of relying on the return value of the fs.readFile call, we provide a callback function as a parameter that gets invoked when the operation has finished. The callback is responsible for displaying the file's contents or the resulting error message.

While the file is being read (that happens in the background), the event loop is still able to process other events making it possible for the application to work on other tasks concurrently.

To ensure that an application is responsive and scalable, I/O related functionality in the Node.js API is asynchronous by default. For some functions there are also synchronous equivalents for convenience, but as a rule of thumb they should be avoided as much as possible.

Asynchronous I/O is an important ingredient in making Node.js applications scalable -- because I/O operations are typically several orders of magnitude slower than CPU operations, the application should remain responsive as long as no callback takes long to complete. Furthermore, because there is no thread per connection model, there is no context-switching and memory overhead for each concurrent task.

However, asynchronous I/O operations and a callback-convention does not guarantee that the main loop never gets blocked.

When implementing tasks that are heavily CPU-bound (such as recursively computing a Fibonacci number), the programmer has to make sure that the execution does not block the main loop too long (for example, by dividing it into smaller tasks that generate events, or using threads).

Code structuring issues


Another challenge that comes with asynchronous functions is that it becomes much harder to keep your code structured and maintainable.

For example, if we want to create a directory, then write a text file to it, and then read the text file, and only use non-blocking functions to keep the application responsive, we may end up writing:

const fs = require('fs');

fs.mkdir("test", function(err) {
    if(err) {
        console.error(err);
        process.exit(1);
    } else {
        fs.writeFile("hello.txt", "Hello world!", function(err) {
            if(err) {
                console.error(err);
                process.exit(1);
            } else {
                fs.readFile("hello.txt", function(err, contents) {
                    if(err) {
                        console.error(err);
                        process.exit(1);
                    } else {
                        // Displays: "Hello world!"
                        console.log(contents);
                    }
                });
            }
        });
    }
});

As may be observed, for each function call, we define a callback responsible for checking the status of the call and executing the next step. For each step, we have to nest another callback function, resulting in pyramid code.

The code above is difficult to read and maintain. If we want to add another step in the middle, we are forced to refactor the callback nesting structure, which is labourious and tedious.

Because code structuring issues are so common, all kinds of software abstractions have been developed to coordinate the execution of tasks. For example, we can use the async library to rewrite the above code fragment as follows:

const async = require('async');

async.waterfall([
    function(callback) {
        fs.mkdir("test", callback);
    },
    
    function(callback) {
        fs.writeFile("hello.txt", "Hello world!", callback);
    },
    
    function(callback) {
        fs.readFile("hello.txt", callback);
    },
    
    function(contents, callback) {
        // Displays: "Hello world!"
        console.log(contents);
        callback();
    }
], function(err) {
    if(err) {
        console.error(err);
        process.exit(1);
    }
});

The async.waterfall abstraction flattens the code, allows us to conveniently add additional asynchronous steps and change the order, if desired.

Promises


In addition to Node.js-style callback functions and abstraction libraries for coordination, a more powerful software abstraction was developed: promises (to be precise: there are several kinds of promise abstractions developed, but I am referring to the Promises/A+ specification).

Promises have become very popular, in particular for APIs that are used in the browser. As a result, they have been accepted into the core JavaScript API.

With promises the idea is that every asynchronous function quickly returns a promise object that can be used as a reference to a value that will be delivered at some point in the future.

For example, we can wrap the function invocation that reads a text file into a function that returns promise:

const fs = require('fs');

function readHelloFile() {
    return new Promise((resolve, reject) => {
       fs.readFile("hello.txt", function(err, contents) {
           if(err) {
               reject(err);
           } else {
               resolve(contents);
           }
       });
    });
}

The above function: readHelloFile invokes fs.readFile from the Node.js API to read the hello.txt file and returns a promise. In case the file was successfully read, the promise is resolved and the file's contents is propagated as a result. In case of an error, the promise is rejected with the resulting error message.

To retrieve and display the result, we can invoke the above function as follows:

readHelloFile().then(function(contents) {
    console.log(contents);
}, function(err) {
    console.error(err);
    process.exit(1);
});

Invoking the then method causes the main event loop to invoke either the resolve (first parameter) or reject callback function (second parameter) when the result is available.

Because promises have become part of the ECMAScript standard, Node.js has introduced alternative APIs that are promise-based, instead of callback based (such as for filesystem operations: fs.promises).

By using the promises-based API for filesystem operations, we can simplify the previous example to:

const fs = require('fs').promises;

fs.readFile("hello.txt").then(function(contents) {
    console.log(contents);
}, function(err) {
    console.error(err);
    process.exit(1);
});

As described in my old blog post about promises -- they are considered more powerful than callbacks. A promise provides you a reference to a value that is in the process of being delivered. Callbacks can only give you insights in the status of a background task as soon as it completes.

Although promises have an advantage over callbacks, both approaches still share the same drawback -- we are forced to avoid most JavaScript language constructs and use alternative function abstractions.

In modern JavaScript, it is also no longer necessary to always explicitly create promises. Instead, we can also declare a function as async. Simply returning a value or throwing exception in such a function automatically ensures that a promise is returned:

async function computeSum(a, b) {
    return a + b;
}

The function above returns the sum of the provided input parameters. The JavaScript runtime automatically wraps its execution into a promise that can be used to retrieve the return value at some point in the future.

(As a sidenote: the function above returns a promise, but is still blocking. It does not generate an event that can be picked up by the event loop and a callback that can resume its execution at a later point in time.)

The result of executing the following code:

const result = computeSum(1, 1);
console.log("The result is: " + result);

is a promise object, not a numeric value:

Result is: [object Promise]

When a function returns a promise, we also no longer have to invoke .then() and provide callbacks as parameters to retrieve the result or any thrown errors. The await keyword can be used to automatically wait for a promise to yield its return value or an exception, and then move to the next statement:

(async function() {
    const result = await computeSum(1, 1);
    console.log("The result is: " + result); // The result is: 2
})();

The only catch is that you can only use await in the scope of an asynchronous function. The default scope of a Node.js program is synchronous. As a result, we have to wrap the code into an asynchronous function block.

By using the promise-based fs API and the new language features, we can rewrite our earlier callback-based example (that creates a directory, writes and reads a file) as follows:

const fs = require('fs').promises;

(async function() {
    try {
        await fs.mkdir("test");
        await fs.writeFile("hello.txt", "Hello world!");
        const contents = await fs.readFile("hello.txt");
    } catch(err) {
        console.error(err);
        process.exit(1);
    }
})();

As may be observed, the code is much simpler than manually orchestrating promises.

Structured programming concepts


In my previous blog post, I have argued that most of JavaScript's language constructs (that implement structured programming concepts) cannot be directly used in combination with non-blocking functions (that return almost immediately and require callback functions as parameters).

As a personal exercise, I have created function abstractions that are direct asynchronous equivalents for all these structured programming concepts that should be avoided and added them to my own library: slasp.

By combining promises, async functions and await statements, these function abstractions have mostly become obsolete.

In this section, I will go over the structured programming concepts I covered in my old blog post and show their direct asynchronous programming equivalents using modern JavaScript language constructs.

Function definitions


As I have already explained in my previous blog post, the most basic thing one can do in JavaScript is executing statements, such as variable assignments or function invocations. This used to be already much different when moving from a synchronous programming to an asynchronous programming world.

As a trivial example, I used a synchronous function whose only purpose is to print text on the console:

function printOnConsole(value) {
    console.log(value);
}

The above example is probably too trivial, but it is still possible to make it non-blocking -- we can generate a tick event so that the function returns immediately and use a callback parameter so that the task will be resumed at a later point in time:

function printOnConsole(value) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            console.log(value);
            resolve();
        });
    });
}

To follow modern JavaScript practices, the above function is wrapped into a constructor that immediately returns a promise that can be used as a reference to determine when the task was completed.

(As a sidenote: we compose a regular function that returns a promise. We cannot define an async function, because the process.nextTick is an asynchronous function that requires a callback function parameter. The callback is responsible for propagating the end result. Using a return only causes the callback function to return and not the enclosing function.)

I have also shown that for functions that return a value, the same principle can be applied. As an example, I have used a function that translates a numeric digit into a word:

function generateWord(digit) {
    const words = [ "zero", "one", "two", "three", "four",
        "five", "six", "seven", "eight", "nine" ];
    return words[digit];
}

We can also make this function non-blocking by generating a tick event and wrapping it into a promise:

function generateWord(digit) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            const words = [ "zero", "one", "two", "three", "four", "five",
                "six", "seven", "eight", "nine" ];
            resolve(words[digit]);
        });
    });
}

Sequential decomposition


The first structured programming concept I elaborated about was sequential decomposition in which a number of statements are executed in sequential order.

I have shown a trivial example that adds 1 to a number, then converts the resulting digit into a word, and finally prints the word on the console:

const a = 1;
const b = a + 1;
const number = generateWord(b);
printOnConsole(number); // two

With the introduction of the await keyword, converting the above code to use the asynchronous implementations of all required functions has become straight forward:

(async function() {
    const a = 1;
    const b = a + 1;
    const number = await generateWord(b);
    await printOnConsole(number); // two
})();

The above example is a one-on-one port of its synchronous counterpart -- we just have to use the await keyword in combination with our asynchronous function invocations (that return promises).

The only unconventional aspect is that we need to wrap the code inside an asynchronous function block to allow the await keyword to be used.

Alteration


The second programming concept that I covered is alteration that is used to specify conditional statements.

I gave a simple example that checks whether a given name matches my own name:

function checkMe(name) {
    return (name == "Sander");
}
    
const name = "Sander";
    
if(checkMe(name)) {
    printOnConsole("It's me!");
    printOnConsole("Isn't it awesome?");
} else {
    printOnConsole("It's someone else!");
}

It is also possible to make the checkMe function non-blocking by generating a tick event and wrapping it into a promise:

function checkMe(name) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            resolve(name == "Sander");
        });
    });
}

To invoke the asynchronous function shown above inside the if-statement, we only have to write:

(async function() {
    const name = "Sander";

    if(await checkMe(name)) {
        await printOnConsole("It's me!");
        await printOnConsole("Isn't it awesome?");
    } else {
        await printOnConsole("It's someone else!");
    }
})();

In my previous blog post, I was forced to abolish the regular if-statement and use an abstraction (slasp.when) that invokes the non-blocking function first, then uses the callback to retrieve the result for use inside an if-statement. In the above example, the only subtle change I need to make is to use await inside the if-statement.

I can also do the same thing for the other alteration construct: the switch -- just using await in the conditional expression and the body should suffice.

Repetition


For the repetition concept, I have shown an example program that implements the Gregory-Leibniz formula to approximate PI up to 6 digits:

function checkTreshold(approx) {
    return (approx.toString().substring(0, 7) != "3.14159");
}

let approx = 0;
let denominator = 1;
let sign = 1;

while(checkTreshold(approx)) {
    approx += 4 * sign / denominator;
    printOnConsole("Current approximation is: "+approx);

    denominator += 2;
    sign *= -1;
}

As with the previous example, we can also make the checkTreshold function non-blocking:

function checkTreshold(approx) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            resolve(approx.toString().substring(0, 7) != "3.14159");
        });
    });
}

In my previous blog post, I have explained that the while statement is unfit for executing non-blocking functions in sequential order, because they return immediately and have to resume their execution at a later point in time.

As with the alteration language constructs, I have developed a function abstraction that is equivalent to the while statement (slasp.whilst), making it possible to have a non-blocking conditional check and body.

With the introduction of the await statement, this abstraction function also has become obsolete. We can rewrite the code as follows:

(async function() {
    let approx = 0;
    let denominator = 1;
    let sign = 1;

    while(await checkTreshold(approx)) {
        approx += 4 * sign / denominator;
        await printOnConsole("Current approximation is: "+approx);

        denominator += 2;
        sign *= -1;
    }
})();

As can be seen, the above code is a one-on-one port of its synchronous counterpart.

The function abstractions for the other repetition concepts: doWhile, for, for-in have also become obsolete by using await for evaluating the non-blocking conditional expressions and bodies.

Implementing non-blocking recursive algorithms still remains tricky, such as the following (somewhat inefficient) recursive algorithm to compute a Fibonacci number:

function fibonacci(n) {
    if (n < 2) {
        return 1;
    } else {
        return fibonacci(n - 2) + fibonacci(n - 1);
    }
}

const result = fibonacci(20);
printOnConsole("20th element in the fibonacci series is: "+result);

The above algorithm is mostly CPU-bound and takes some time to complete. As long as it is computing, the event loop remains blocked causing the entire application to become unresponsive.

To make sure that the execution does not block for too long by using cooperative multi-tasking principles, we should regularly generate events, suspend its execution (so that the event loop can process other events) and use callbacks to allow it to resume at a later point in time:

function fibonacci(n) {
    return new Promise((resolve, reject) => {
        if (n < 2) {
            setImmediate(function() {
                resolve(1);
            });
        } else {
            let first;
            let second;

            fibonacci(n - 2)
            .then(function(result) {
                first = result;
                return fibonacci(n - 1);
            })
            .then(function(result) {
                second = result;
                resolve(first + second);
            });
        }
    });
}

(async function() {
    const result = await fibonacci(20);
    await printOnConsole("20th element in the fibonacci series is: "+result);
})();

In the above example, I made the algorithm non-blocking by generating a macro-event with setImmediate for the base step. Because the function returns a promise, and cannot be wrapped into an async function, I have to use the promises' then methods to retrieve the return values of the computations in the induction step.

Extensions


In my previous blog post, I have also covered the extensions to structured programming that JavaScript provides.

Exceptions


I have also explained that with asynchronous functions, we cannot use JavaScript's throw, and try-catch-finally language constructs, because exceptions are typically not thrown instantly but at a later point in time.

With await, using these constructs is also no longer a problem.

For example, I can modify our generateWord example to throw an exception when the provided number is not between 0 and 9:

function generateWord(num) {
    if(num < 0 || num > 9) {
        throw "Cannot convert "+num+" into a word";
    } else {
        const words = [ "zero", "one", "two", "three", "four", "five",
            "six", "seven", "eight", "nine" ];
        return words[num];
    }
}

try {
    let word = generateWord(1);
    printOnConsole("We have a: "+word);
    word = generateWord(10);
    printOnConsole("We have a: "+word);
} catch(err) {
    printOnConsole("Some exception occurred: "+err);
} finally {
    printOnConsole("Bye bye!");
}

We can make generateWord an asynchronous function by converting it in the usual way:

function generateWord(num) {
    return new Promise((resolve, reject) => {
        process.nextTick(function() {
            if(num < 0 || num > 9) {
                reject("Cannot convert "+num+" into a word");
            } else {
                const words = [ "zero", "one", "two", "three", "four", "five",
                    "six", "seven", "eight", "nine" ];
                resolve(words[num]);
            }
        });
    });
}

(async function() {
    try {
        let word = await generateWord(1);
        await printOnConsole("We have a: "+word);
        word = await generateWord(10);
        await printOnConsole("We have a: "+word);
    } catch(err) {
        await printOnConsole("Some exception occurred: "+err);
    } finally {
        await printOnConsole("Bye bye!");
    }
})();

As can be seen in the example above, thanks to the await construct, we have not lost our ability to use try/catch/finally.

Objects


Another major extension is object-oriented programming. As explained in an old blog post about object oriented programming in JavaScript, JavaScript uses prototypes rather than classes, but prototypes can still be used to simulate classes and class-based inheritance.

Because simulating classes is such a common use-case, a class construct was added to the language that uses prototypes to simulate it.

The following example defines a Rectangle class with a method that can compute a rectangle's area:

class Rectangle {
    constructor(width, height) {
        this.width = width;
        this.height = height;
    }

    calculateArea() {
        return this.width * this.height;
    }
}

const r = new Rectangle(2, 2);

printOnConsole("Area is: "+r.calculateArea());

In theory, it is also possible that the construction of an object takes a long time and should be made non-blocking.

Although JavaScript does not have a language concept to do asynchronous object construction, we can still do it by making a couple of small changes:

class Rectangle {
    asyncConstructor(width, height) {
        return new Promise((resolve, reject) => {
            process.nextTick(() => {
                this.width = width;
                this.height = height;
                resolve();
            });
        });
    }

    calculateArea() {
        return this.width * this.height;
    }
}

(async function() {
    const r = new Rectangle();
    await r.asyncConstructor(2, 2);
    await printOnConsole("Area is: "+r.calculateArea());
})();

As can be seen in the above example, the constructor function has been replaced with an asyncConstructor method that implements the usual strategy to make it non-blocking.

To asynchronously construct the rectangle, we first construct an empty object using the Rectangle class object as its prototype. Then we invoke the asynchronous constructor to initialize the object in a non-blocking way.

In my previous blog post, I have developed an abstraction function that could be used as an asynchronous replacement for JavaScript's new operator (slasp.novel) that performs the initialization of an empty object and then invokes the asynchronous constructor.

Due to the fact that JavaScript introduced a class construct (that replaces all the obscure instructions that I had to perform to simulate an empty object instance with the correct class object as prototype) my abstraction function has mostly lost its value.

Summary of concepts


In my previous blog post, I have given an overview of all covered synchronous programming language concepts and corresponding replacement function abstractions that should be used with non-blocking asynchronous functions.

In this blog post, I will do the same with the concepts covered:

Concept Synchronous Asynchronous
Function interface
function f(a) { ... }
async function f(a) { ... }

function f(a) {
    return new Promise(() => {...});
}
Return statement
return val;
return val;
resolve(val);
Sequence
a(); b(); ...
await a(); await b(); ...
if-then-else
if(condFun()) {
    thenFun();
} else {
    elseFun();
}
if(await condFun()) {
    await thenFun();
} else {
    await elseFun();
}
switch
switch(condFun()) {
    case "a":
        funA();
        break;
    case "b":
        funB();
        break;
    ...
}
switch(await condFun()) {
    case "a":
        await funA();
        break;
    case "b":
        await funB();
        break;
    ...
}
Recursion
function fun() {
    fun();
}
function fun(callback) {
    return new Promise((res, rej) => {
        setImmediate(function() {
            return fun();
        });
    });
}
while
while(condFun()) {
    stmtFun();
}
while(await condFun()) {
    await stmtFun();
}
doWhile
do {
    stmtFun();
} while(condFun());
do {
    await stmtFun();
} while(await condFun());
for
for(startFun();
    condFun();
    stepFun()
) {
    stmtFun();
}
for(await startFun();
    await condFun();
    await stepFun()
) {
    await stmtFun();
}
for-in
for(const a in arrFun()) {
    stmtFun();
}
for(const a in (await arrFun())) {
    await stmtFun();
}
throw
throw err;
throw err;
reject(err);
try-catch-finally
try {
    funA();
} catch(err) {
    funErr();
} finally {
    funFinally();
}
try {
    await funA();
} catch(err) {
    await funErr();
} finally {
    await funFinally();
}
constructor
class C {
    constructor(a) {
        this.a = a;
    }
}
class C {
    asyncConstructor(a) {
        return new Promise((res, rej) => {
            this.a = a;
            res();
        }
    }
}
new
const obj = new C(a);
const obj = new C();
await obj.asyncConstructor(a);

The left column in the table shows all language constructs that are synchronous by default and the right column shows their equivalent asynchronous implementations.

Note that compared to the overview given in my previous blog post, the JavaScript language constructs are not avoided, but used.

With the exceptions of wrapping callback-based function invocations in promises and implementing recursive algorithms, using await usually suffices to retrieve the results of all required sub expressions.

Discussion


In all my previous blog posts that I wrote about asynchronous programming, I was always struck by the fact that most of JavaScript's language constructs were unfit for asynchronous programming. The abstractions that were developed to cope with this problem (e.g. callbacks, coordination libraries, promises etc.) make it possible to get the job done in a reasonable manner, but IMO they always remain somewhat tedious to use and do not prevent you from making many common mistakes.

Using these abstractions remained a common habit for years. With the introduction of the async and await concepts, we finally have a solution that is decent IMO.

Not all problems have been solved with the introduction of these new language features. Callback-based APIs have been a common practice for a very long time, as can be seen in some of my examples. Not all APIs have been converted to promise-based solutions and there are still many APIs and third-party libraries that keep following old practices. Most likely, not all old-fashioned APIs will ever go away.

As a result, we sometimes still have to manually compose promises and do the appropriate conversions from callback APIs. There are also nice facilities that make it possible to conveniently convert callback invocations into promises, but it still remains a responsibility of the programmer.

Another problem (that I often see with programmers that are new to JavaScript), is that they believe that using the async keyword automatically makes their functions non-blocking.

Ensuring that a function does not block still remains the responsibility of the programmer. For example, by making sure that only non-blocking I/O functions are called, or CPU-bound instructions are broken up into smaller pieces.

The async keyword is only an interface solution -- making sure that a function returns a promise and that the await keyword can be used so that a function can stop (by returning) and be resumed at a later point in time.

The JavaScript language does not natively support threads or processes, but APIs have been added to Node.js (worker threads) and web browsers (web workers) to allow code to be executed in a thread. Using these facilities somewhat relieve programmers of the burden to divide long running CPU-bound operations into smaller tasks. Moreover, context-switching is typically much more efficient than cooperative multi-tasking (by generating events and invoking callbacks).

Another problem that remains is calling asynchronous functions from synchronous functions -- there is still no facility in JavaScript that makes it possible to wait for the completion of an asynchronous function in a synchronous function context.

I also want to make a general remark about structured programming -- although the patterns shown in this blog post can be used to prevent that a main loop blocks, structured programming is centered around the idea that you need to execute a step after the completion of another. For long running tasks that do not have a dependency on each other this may not always be the most efficient way of doing things. You may end up waiting for an unnecessary amount of time.

The fact that a promise gives you a reference to a value that will be delivered in the future, also gives you many new interesting abilities. For example, in an application that consists of a separate model and view, you could already start composing a view and provide the promises for the model objects as parameters to the views. Then it is no longer necessary to wait for all model objects to be available before the views can be constructed -- instead, the views can already be rendered and the data can be updated dynamically.

Structured programming patterns are also limiting the ability to efficiently process collections of data -- the repetition patterns in this blog post expect that all data is retrieved before we can iterate over the resulting data collection. It may be more efficient to work with asynchronous iterators that can retrieve data on an element-by-element basis, in which the element that comes first is processed first.

Thursday, December 30, 2021

11th annual blog reflection

Today it is my blog's 11th anniversary. As with previous years, this is a nice opportunity to reflect over last year's writings.

Nix process management framework


In the first few months of the year, I have dedicated quite a bit of time on the development of the experimental Nix process framework that I started in 2019.

As explained in my blog reflection over 2020, I have reached all of my original objectives. However, while developing these features and exploring their underlying concepts, I discovered that there were still a number side issues that I needed to address to make the framework usable.

s6-rc backend


The first thing I did was developing a s6-rc backend. Last year, I did not know anything about s6 or s6-rc , but it was provided to me as a feature suggestion by people from the Nix community. Aside from the fact that it is a nice experiment to evaluate how portable the framework is, I also learned a great deal about s6, its related tools, and its ancestor: daemontools from which many of s6's ideas are inspired.

Mutable multi-process containers


I also worked on a mutable multi-process container deployment approach. Last year, I have developed a Docker backend for the Nix process management framework making it possible to expose each running process instance as a container. Furthermore, I also made it possible to conveniently construct multi-process containers in which any capable process manager that the Nix process management framework supports can be used as a root process.

Unfortunately, multi-process containers have a big drawback: they are immutable, and when any of the processes need to be changed or upgraded, the container as a whole needs to be discarded and redeployed from a new image, causing all processes to be terminated.

To cope with this limitation, I have developed an extension that makes it possible to deploy mutable multi-process containers, in which most of the software in containers can be upgraded by the Nix package manager.

As long as the root process is not affected, a container does not have to be discarded when a process is upgraded. This extension also makes it possible to run Hydra: the Nix-based continuous integration service in a Docker container.

Using the Nix process management framework as an infrastructure deployment solution


I was also able to use the Nix process management framework to solve the bootstrap problem for Disnix on non-NixOS systems -- in order to use Disnix, every target machine needs to run the Disnix service and a number of container provider services.

For a NixOS machine this process is automated, but on non-NixOS systems a manual installation is still required, which is quite cumbersome. The Nix process management framework can automatically deploy Disnix and all required container provider services on any system capable of running the Nix package manager and the Nix process management framework.

Test framework


Finally, I have developed a test framework for the Nix process management framework. As I have already explained, the framework makes it possible to use multiple process managers, multiple operating systems, multiple instances of all kinds of services, and run services as an unprivileged user, if desired.

Although the framework facilitates all these features, it cannot guarantee that a service will work with under all possible conditions. The framework makes it possible to conveniently reproduce all these conditions so that a service can be validated.

With the completion of the test framework, I consider the Nix process management framework to be quite practical. I have managed to automate the deployments of all services that I frequently use (e.g. web servers, databases, application services etc.) and they seem to work quite well. Even commonly used Nix projects are packaged, such as the Nix daemon for multi-user installations and Hydra: the Nix-based continuous integration server.

Future work


There are still some open framework issues that I intend to address at some point in the future. We still cannot test any services on non-Linux systems such as FreeBSD, which requires a more generalized test driver.

I also still need to start writing an RFC that identifies the concepts of the framework so that these ideas can be integrated into Nixpkgs. The Nix process management framework is basically a prototype to explore ideas, and it has always been my intention to push the good parts upstream.

Home computing


After the completion of the test framework, I have shifted my priorities and worked on improving my home computing experience.

For many years, I have been using a custom script that uses rsync to exchange files, but implements a Git-like workflow, to make backups of my personal files and exchange files between multiple machines, such as my desktop machine and laptop. I have decided to polish the script, release it, and write a blog post that explains how it came about.

Last summer, I visited the Home Computer Museum, that gave me inspiration to check if both of my vintage computers: the Commodore 128 and Commodore Amiga 500 still work. I have not touched the Amiga since 2011 (the year that I wrote a blog post about it) and it was lying dormant in a box on the attic every since.

Unfortunately, a few peripherals were broken or in a bad condition (such as the hard drive). I have decided to bring it to the museum for repairs and order replacement peripherals. It turns out that it was quite a challenge to have it all figured out, in particular the installation process of the operating system.

Because not all information that I needed is available on the Internet, I have decided to write a blog post about my experiences.

I am still in the process of figuring out all the details for my Commodore 128 and I hope I can publish about it soon.

Revising the NPM package deployment infrastructure for Nix


Aside from doing a nice "home computing detour", I have also been shifting my priorities to a new major development area: improving the NPM deployment infrastructure for Nix. Although node2nix is doing its job pretty well in most cases, its design is heavily dated, and giving me more and more problems in correctly supporting the new features of NPM.

As a first step, I have revised what I consider the second most complicated part of node2nix: the process that populates the node_modules/ folder and makes all necessary modifications so that npm install will not attempt to download source artifacts from their original locations.

This is an important requirement -- the fact that NPM and Nix do not play well together is because dependency management is conflicting -- Nix's purity principles are more strict. As a result, NPM's dependency management needs to be bypassed.

The result is a companion tool that I call: placebo-npm that will replace most of the complicated shell code in the node-env.nix module.

I am still working on revising many other parts of node2nix. This should eventually lead to a new and more modular design, that will support NPM's newer features and should be much easier to maintain.

Next year, I hope to report more about my progress.

Some thoughts


As with 2020, I consider 2021 an exceptional year for the record books.

Development


Compared to last year, I am much less productive from a blogging perspective. Partially, this is caused by the fact that there are still many things I have worked on that I could not properly finish.

I have also noticed that there was a considerable drop in my energy level after I completed the test framework for the Nix process management framework. I think this can be attributed to the fact that the process management framework has basically been my only spare time project for over two years.

For a small part, this kind of work is about exploring ideas, but is even more about the execution of those ideas -- unfortunately, being in execution mode for such a long time (while ignoring the exploration of ideas you come up in other areas) gradually made it more difficult to keep enjoying the work.

Despite struggling with my energy levels, I remained motivated to complete all of it, because I know that I am also a very bad multi-tasker. Switching to something else makes it even more difficult to complete it in a reasonable time span.

After I reached all my goals, for a while, it became extremely difficult to get myself focused on any technical challenge.

Next year, I have another big project that I am planning to take on (node2nix), but at the same I will try to schedule proper "breaks" in between to keep myself in balance.

The COVID-19 pandemic


In my annual reflection from last year, I have also elaborated about the COVID-19 pandemic that reached my home country (The Netherlands) in March 2020. Many things have happened that year, and at the time writing my reflection blog post over 2020, we were in our second lockdown.

The second lockdown felt much worse than the first, but I was still mildly optimistic because of the availability of the first vaccine: Pfizer/BioNTech that looked like our "way out". Furthermore, I was also quite fascinated by the mRNA technique making it possible to produce a vaccine so quickly.

Last year, I was hoping that next year I could report that the problem was under control or at least partially solved, but sadly that does not seem to be the case.

Many things have happened: the variant that appeared in England eventually became dominant (Alpha variant) and was considerably more contagious than the original variant, causing the second lockdown to last another three months.

In addition, two more contagaious mutations appeared at the same time in South Africa and Brazil (Beta and Gamma), of which the Alpha variant became the dominant.

Several months later, there was another huge outbreak in India introducing the Delta variant, that was even more contagious than the Alpha variant. Eventually, that variant became the most dominant in the world. Fortunately, the mRNA vaccines that were developed were still effective enough, but the Delta variant was so contagious that it was considered impossible to build up herd immunity to eradicate the virus.

In the summer, the problem seemed to be mostly under control because many people have been vaccinated in my home country. Nonetheless, we have learned in a painful way that relaxing restrictions too much could still lead to very big rapid outbreaks.

We have also learned in a painful way that the virus spreads more easily in the winter. As a result, we also observed that, despite the fact that over 85% of all adults are fully vaccinated, there are still enough clusters of people that have not built up any resistance against the virus (either by vaccination or contracting the virus), again leading to significant problems in the hospitals and ICs.

Furthermore, the effectiveness of vaccines also drops over time, causing vaccinated people with health problems to still end up in hospitals.

As a result of all these hospitalizations and low IC capacity, we are now in yet another lockdown.

What has always worried me the most is the fact that in so many areas in the world, people hardly have any access to vaccines and medicines causing the virus to easily spread on a massive scale and mutate. I knew because of these four variants, it would only be a matter of time before a new dominant mutation will appear.

And that fear eventually became reality -- in November, in Botswana and South Africa, a new and even more contagious mutation appeared: the Omicron variant with a spike-protein that is much different than the delta variant, reducing the effectiveness of our vaccines.

At the moment, we are not sure the implications are. In the Netherlands, as well as many other countries in the world, the Omicron variant has now become the most dominant virus mutation.

The only thing that may be good news is that the Omicron variant could also potentially cause less severe sickness, but so far we have no clarity on that yet.

I think next year we still have much to learn -- to me it has become very clear that this problem will not go away any time soon and that we have to find more innovative/smarter ways to cope with it.

Furthermore, the mutations have also demonstrated that we should probably do more about inequality in the world. As a consequence, the virus could still significantly spread and mutate becoming a problem to everybody in the world.

Blog posts


Last year I forgot about it, but every year I also typically reflect over my top 10 of most frequently read blog posts:

  1. Managing private Nix packages outside the Nixpkgs tree. As with previous years, this blog post remains the most popular because it is very practical and unanswered in official Nix documentation.
  2. On Nix and GNU Guix. This blog post used to be my most popular blog post for a long time, and still remains the second most popular. I believe this can be attributed to the fact that this comparison is still very relevant.
  3. An evaluation and comparison of Snappy Ubuntu. Also a very popular blog post since 2015. It seems that the comparison with Snappy and Flatpak (a tool with similar objectives) remains relevant.
  4. Disnix 0.5 release announcement and some reflection. This is a blog post that I wrote in 2016 and suddenly appeared in the overall top 10 this year. I am not sure why this has become so relevant all of a sudden.
  5. On using Nix and Docker as deployment solutions: similarities and differences. This is a blog post that I wrote last year to compare Nix and Docker and explain in what ways they are similar and different. It seems to be very popular despite the fact that it was not posted on discussion sites such as Reddit and Hackernews.
  6. Yet another blog post about Object Oriented Programming and JavaScript. This explanation blog post is pretty old but seems to stay relevant, despite the fact that modern JavaScript has a class construct.
  7. Setting up a multi-user Nix installation on non-NixOS systems. Setting up multi-user Nix installations on non-NixOS machines used to be very cumbersome, but fortunately that has been improved in the recent versions. Still, the discussion seems to remain relevant.
  8. An alternative explanation of the Nix package manager. An alternative explanation that I consider to be better of two that I wrote. It seems to remain popular because I refer to it a lot.
  9. On NixOps, Disnix, service deployment and infrastructure deployment. A very popular blog post, that has dropped somewhat in popularity. It still seems that the tools and the discussion is relevant.
  10. Composing FHS-compatible chroot environments with Nix (or deploying steam in NixOS). An old blog post, but it remains relevant because it addresses a very important compatibility concern with binary-only software and a common Nix-criticism that it is not FHS-compatible.

Conclusion


As with 2020, 2021 has been quite a year. I hope everybody stays safe and healthy.

The remaining thing I'd like to say is: HAPPY NEW YEAR!!!

Tuesday, October 19, 2021

Using my Commodore Amiga 500 in 2021


Due to the high number of new COVID-19 infections in my home country last summer, I had to "improvise" yet another summer holiday. As a result, I finally found the time to tinker with my old computers again after a very long time of inactivity.

As I have explained in two blog posts that I wrote over ten years ago, the first computer (a Commodore 128 bought by my parents in 1985) and second computer (Commodore Amiga 500 bought by my parents in 1992) that I ever used, are still in my possession.

In the last few years, I have used the Commodore 128 a couple of times, but I have not touched the Commodore Amiga 500 since I wrote my blog post about it ten years ago.

It turns out that the Commodore Amiga 500 still works, but I ran into a number of problems:

  • A black and white display. I used to have a monitor, but it broke down in 1997. Since then, I have been using Genlock device to attach the Amiga to a TV screen. Unfortunately, in 2021 the Genlock device no longer seems to work.

    The only display option I had left is to attach the Amiga to a TV with an RCA to SCART cable by using the monochrome video output. The downside is that it is only capable of displaying a black and white screen.
  • No secondary disk drive. I used to have two 3.5-inch double density disk drives: an internal disk drive (inside the case) and an external disk drive that you can attach to the disk drive port.

    The external disk drive still seems to respond when I insert a floppy disk (the led blinks), but it no longer seems to be capable of reading any disks.
  • Bad hard drive and expansion slot problems. The expansion board (that contains the hard drive) seems to give me all kinds of problems.

    Sometimes the Amiga completely fails to detect it. In other occasions, I ran into crashes causing the filesystem to return me write errors. Attempting to repair them typically results in new errors.

    After thoroughly examining the disk with DiskSalv, I learned that the drive has physical damage and needs to be replaced.

I also ran into an interesting problem from a user point of view -- exchanging data to and from my Amiga (such as software downloaded from the Internet and programs that I used to write) is quite a challenge. In late 1996, when my parents switched to the PC, I used floppy disks to exchange data.

In 2021, floppy drives have completely disappeared from all modern computers. In the rare occasion that I still need to read a floppy disk, I have an external USB floppy drive at my disposal, but it is only capable of reading high density 3.5-inch floppy disks. A Commodore Amiga's standard floppy drive (with the exception of the Amiga 4000) is only capable of reading double density disks.

Fortunately, I have discovered that there are still many things possible with old machines. I brought both my Commodore 128 and Commodore 500 to the Home Computer Museum in Helmond for repairs. Furthermore, I have ordered all kinds of replacement peripherals.

Getting it all to work, turned out to be quite a challenge. Eventually, I have managed to overcome all my problems and the machine works like a charm again.

In this blog post, I will describe what problems I faced and how I solved them.

Some interesting properties of the Amiga


I often receive many questions from all kinds of people who want to know why it is so interesting to use such an old machine. Aside from nostalgic reasons, I think the machine is an interesting piece of computer history. At the time the first model was launched: the Amiga 1000 in 1985, the machine was far ahead of its time and provided unique multimedia capabilities.

Back in the late 80s, system resources were very limited (such as CPU, RAM and storage) compared to modern machines, but there were all kinds of interesting facilities to overcome their design limitations.

For example, the original Amiga 500 model only had 512 KiB of RAM and 32 configurable color registers. Colors can be picked out of a range of 4096 possible colors.

Despite only having the ability to configure a maximum 32 distinct colors, it could still display photo-realistic images:


As can be seen, the screen shot above clearly has more than 32 distinct colors. This is made possible by using a special screen mode called Hold-and-Modify (HAM).

In HAM mode, a pixel's color can be picked from a palette of 16 base colors, or a color component (red, green or blue) of the adjacent pixel can be changed. The HAM screen mode makes it possible to use all possible 4096 colors, albeit with some restrictions on the adjacent color values.

Another unique selling point of the Amiga were its sound capabilities. It could mix 4 audio channels in hardware, and easily combined with graphics, animations and games. The Amiga has all kinds of interesting music productivity software, such as ProTracker, that I used a lot.

To make all these multimedia features possible, the Amiga has its own unique hardware architecture:


The above diagram provides a simplified view of the most important chips in the Amiga 500 and how they are connected:

  • On the left, the CPU is shown: a Motorola 68000 that runs at approximately 7 MHz (the actual clock speeds differ somewhat on a PAL and NTSC display). The CPU is responsible for doing calculations and executing programs.
  • On the right, the unique Amiga chips are shown. Each of them has a specific purpose:
    • Denise (Display ENabler) is responsible for producing the RGB signal for the display, provides bitplane registers for storing graphics data, and is responsible for displaying sprites.
    • Agnus (Address GeNerator UnitS) provides a blitter (that is responsible for quick transfers of data in chip memory, typically graphics data), and a copper: a programmable co-processor that is aligned with the video beam.

      The copper makes all kinds of interesting graphical features possible, while keeping the CPU free for work. For example, the following screenshot of the game Trolls:


      clearly contains more than 32 distinct colors. For example, the rainbow-like background provides a unique color on each scanline. The copper is used in such a way that the value of the background color register is changed on each scanline, while the screen is drawn.

      The copper also makes it possible to switch between screen modes (low resolution, high resolution) on the same physical display, such as in the Workbench:


      As can be seen in the above screenshot, the upper part of the screen shows Deluxe Paint in low-res mode with its own unique set of colors, while the lower part shows the workbench in high resolution mode (with a different color palette). The copper can change the display properties while the screen is rendered, while keeping the CPU free to do work.
    • Paula is a multi-functional chip that provides sound support, such as processing sample data from memory and mixing 4 audio channels. Because it does mixing in hardware, the CPU is still free to do work.

      It also controls the disk drive, serial port, mouse and joysticks.
  • All the chips in the above diagram require access to memory. Chip RAM is memory that is shared between all chips. As a consequence, they share the same memory bus.

    A shared bus imposes speed restrictions -- on even clock cycles the CPU can access chip memory, while on the uneven cycles the chips have memory access.

    Many Amiga programs are optimized in such a way that all CPU's memory access operations are at even clock cycles as much as possible. When the CPU needs to access memory on uneven clock cycles, it is forced to wait, losing execution speed.
  • An Amiga can also be extended with Fast RAM that does not suffer from any speed limitations. Fast RAM is on a different memory bus that can only be accessed by the CPU and not by any of the chips.

    (As a sidenote: there is also Slow RAM that is not shown in the diagram. It falls in between chip and fast RAM. Slow RAM is memory that is exclusive to the CPU, but cannot be used on uneven clock cycles).

Compared to other computer architectures used at the same time, such as the PC, 7 MHz of CPU clock speed does not sound all that impressive, but the combination of all these autonomous chips working together is what makes many incredible multimedia properties possible.

My Amiga 500 specs



When my parents bought my Commodore Amiga 500 machine in 1992, it still had the original chipset and 512 KiB of Chip RAM. The only peripherals were an external 3.5-inch floppy drive and a kickstart switcher allowing me switch between Kickstart 1.3 and 2.0. (The kickstart are portions of the Amiga operating system residing in the ROM).

Some time later, the Agnus and Denise chips were upgraded (we moved from the Original Chipset to the Enchanced Chipset), extending the amount of chip RAM to 1 MiB and making it possible to use super high resolution screen modes.

At some point, we bought a KCS PowerPC board making it possible to emulate a PC and run MS-DOS applications. Although the product calls itself an emulator, it is also provides a board that extends the hardware with a number of interesting features:

  • A 10 MHz NEC V30 CPU that is pin and instruction-compatible with an Intel 8086/8088 CPU. Moreover, it implements some 80186 instructions, some of its own instructions, and is between 10-30% faster.
  • 1 MiB of RAM that can be used by the NEC V30 CPU for conventional and upper memory. In addition, the board's memory can also be used by the Amiga as additional chip RAM, fast RAM and as a RAM disk.
  • A clock (powered by a battery) so that you do not have reconfigure the date and time on startup. This PC clock can also be used in Amiga mode.

Eventually, we also obtained a hard drive. The Amiga 500 does not include any hard drive, nor has it an internal hard drive connector.

Nonetheless, it can be extended through the Zorro expansion slot with an extension board. We obtained this extension board: MacroSystem evolution providing a SCSI connector, a whopping 8 MiB of fast RAM and an additional floppy drive connector. To the SCSI connector, a 120 MiB Maxtor 7120SR hard-drive was attached.

Installing new and replacement peripherals


In this section, I will describe my replacement peripherals and what I did to make them work.

RCB to SCART cable


As explained in the introduction, I no longer have a monitor and the Genlock device is broken, only making it possible to have a black and white display.

Fortunately, all kinds of replacement options seem to be available to connect an Amiga to a more modern display.

I have ordered an RGB to SCART cable. It can be attached to the RGB and audio output of the Amiga and to the SCART input on my LCD TV.

GoTek floppy emulator


Another problem is that the secondary floppy drive is broken and could not be repaired.

Even if I could find a suitable replacement drive, floppy disks are very difficult media to use for data exchange these days.

Even with an old PC that still has an internal floppy drive (capable of reading both high and double density floppy disks), exchanging information remains difficult -- due to limitations of a PC floppy controller, a PC is incapable of reading Amiga disks, but an Amiga can read and write to PC floppy disks. A PC formatted floppy disk has less storage capacity than an Amiga formatted disk.

There is also an interesting alternative to a real floppy drive: the GoTek floppy emulator.

The GoTek floppy emulator works with disk image files stored on a USB memory stick. The numeric digit on the display indicates which disk image is currently inserted into the drive. With the rotating switch you can switch between disk images. It operates at the same speed as a real disk drive and produces similar sounds.

Booting from floppy disk 0 starts a program that allows you to configure disk images for the remaining numeric entries:


The GoTek floppy emulator can act both as a replacement for the internal floppy drive as well as an external floppy drive and uses the same connectors.

I have decided to buy an external model, because the internal floppy drive still works and I want to keep the machine as close to the original as possible. I can turn the GoTek floppy drive into the primary disk drive, by using the DF0 switch on the right side of the Amiga case.

Because all disk images are stored on a FAT filesystem-formatted USB stick, makes exchanging information with a PC much easier. I can transfer the same disk files that I can use in the Amiga emulator to the USB memory stick on my PC and then natively use them on a real Amiga.

SCSI2SD


As explained earlier, the 29-year old SCSI hard drive connected to the expansion board is showing all kinds of age-related problems. Although I could search for a compatible second-hand hard drive that was built in the same era, it is probably not going to last very long either.

Fortunately, for retro-computing purposes, an interesting replacement device was developed: the SCSI2SD, that can be used as drop-in replacement for a SCSI hard drive and other kinds of SCSI devices.

This device can be attached to the same SCSI and power connector cables that the old hard drive uses. As the name implies, its major difference is that is uses a (modern) SD-card for storage.


The left picture (shown above) shows the interior of the MacroSystem evolution board's case with the original Maxtor hard drive attached. On the right, I have replaced the hard drive with a SCSI2SD board (that uses a 16 GiB SD-card for storage).

Another nice property of the SCSI2SD is that an SD card offers much more storage capacity. The smallest SD card that I could buy offers 16 GiB of storage, which is a substantially more than the 120 MiB that the old Maxtor hard drive from 1992 used to offer.

Unfortunately, the designers of the original Amiga operating system did not forsee that people would use devices with so much storage capacity. From a technical point of view, AmigaOS versions 3.1 and older are incapable of addressing more than 4 GiB of storage per device.

In addition to the operating system's storage addressing limit, I discovered that there is another limit -- the SCSI controller on the MacroSystem evolution extension board is unable to address more than 1 GiB of storage space per SCSI device. Trying to format a partition beyond this 1 GiB boundary results in a "DOS disk not found" error. This limit does not seem to be documented anywhere in the MacroSystem evolution manual.

To cope with these limitations, the SCSI2SD device can be configured in such a way that it stays within the boundaries of the operating system. To do this, it needs to be connected to a PC with a micro USB cable and configured with the scsi2sd-util tool.

After many rounds of trial and error, I ended up using the following settings:

  • Enable SCSI terminator (V5.1 only): on
  • SCSI Host Speed: Normal
  • Startup Delay (seconds): 0
  • SCSI Selection Delay: 255
  • Enable Parity: on
  • Enable Unit Attention: off
  • Enable SCSI2 Mode: on
  • Disable glitch filter: off
  • Enable disk cache (experimental): off
  • Enable SCSI Disconnect: off
  • Respond to short SCSI selection pulses: on
  • Map LUNS to SCSI IDs: off

Furthermore, the SCSI2SD allows you to configure multiple SCSI devices and put restrictions on how much storage from the SD card can be used per device.

I have configured one SCSI device (representing a 1 GiB hard drive) with the following settings:

  • Enable SCSI Target: on
  • SCSI ID: 0
  • Device Type: Hard Drive
  • Quirks Mode: None
  • SD card start sector: 0
  • Sector size (bytes): 512
  • Sector count: leave it alone
  • Device size: 1 GB

I left the Vendor, ProductID, Revision and Serial Number values untouched. The Sector count is derived automatically from the start sector and device size.

Before using the SD card, I recommend to erase it first. Strictly speaking, this is not required, but I have learned in a very painful way that DiskSalv, a tool that is frequently used to fix corrupted Amiga file systems, may get confused if there are traces of a previous filesystem left behind. As a result, it may incorrectly treat files as invalid file references causing further corruption.

On Linux, I can clear the memory of the SD card with the following command (/dev/sdb refers to the device file of my SD-card reader):

$ dd if=/dev/zero of=/dev/sdb bs=1M status=progress

After clearing the SD card, I can insert it into the SCSI2SD device, do the partitioning and perform the installation of the Workbench. This process turns out to be more tricky than I thought -- the MacroSystem evolution board seems to only include a manual that is in German, requiring me to brush up my German reading skills.

The first step is to use the HDToolBox tool (included with the Amiga Workbench 2.1 installation disk) to detect the hard disk.

(As a sidenote: check if the SCSI cable is properly attached to both the SCSI2SD device, as well as the board. In my first attempt, the firmware was able to detect that there was a SCSI device with LUN 0, but it could not detect that it was a hard drive. After many rounds of trial and error, I discovered that the SCSI cable was not properly attached to the extension board!).

By default, HDToolBox works with the standard SCSI driver bundled with the Amiga operating system (scsi.device) which is not compatible with the SCSI controller on the MacroSystem Evolution board.

To use the correct driver, I had to configure HDToolBox to use a different driver, by opening a shell session and running the following command-line instructions:

Install2.1:HDTools
HDToolBox evolution.device

In the above code fragment, I pass the driver name: evolution.device as a command-line parameter to HDToolBox.

With the above configuration setting, the SCSI2SD device gets detected by HDToolBox:


I did the partitioning of my SD-card hard drive as follows:


Partition Device Name Capacity Bootable
DH0 100 MiB yes
KCS 100 MiB no
DH1 400 MiB no
DH2 400 MiB no

I did not change any advanced file system settings. I have configured all partitions to use mask: 0xfffffe and max transfer: 0xffffff.

Beyond creating partitions, there was another tricky configuration aspect I had to take into account -- I had to reserve the second partition (the KCS partition) as a hard drive for the KCS PowerPC emulator.

In my first partitioning attempt, I configured the KCS partition as the last partition, but that seems to cause problems when I start the KCS PowerPC emulator, typically resulting in a very slow startup followed by a system crash.

It appears that this problem is a caused by a memory addressing problem. Putting the KCS partition under the 200 MiB limit seems to fix the problem. Since most addressing boundaries are power of 2 values, my guess is that the KCS PowerPC emulator expects a hard drive partition to reside below the 256 MiB limit.

After creating the partitions and rebooting the machine, I can format them. For some unknown reason, a regular format does not seem to work, so I ended up doing a quick format instead.

Finally, I can install the workbench on the DH0: partition by running the Workbench installer (that resides in the: Install2.1 folder on the installation disk):


Null modem cable


The GoTek floppy drive and SCSI2SD already make it much easier to exchange data with my Amiga, but they are still somewhat impractical for exchanging small files, such as Protracker modules or software packages (in LhA format) downloaded from Aminet.

I have also bought a good old-fashioned null modem cable that can be used to link two computers through their serial ports. Modern computers no longer have a RS-232 serial port, but you can still use an USB to RS-232 converter that indirectly makes it possible to link up with a USB connection.

To link up, the serial port settings on both ends need to be the same and the baud rate should not be to high. I have configured the following settings on my Amiga (configured with the SYS:Prefs/Serial preferences program):

  • Baud rate: 19,200
  • Input buffer size: 512
  • Handshaking: RTS/CTS
  • Parity: None
  • Bits/Char: 8
  • Stop Bits: 1

With a terminal client, such as NComm, I can make a terminal connection to my Linux machine. By installing lrzsz on my Linux machine, I can exchange files by using the Zmodem protocol.

There are a variety of ways to link my Amiga with a Linux PC. A quick and easy way to exchange files, is by starting picocom on the Linux machine with the following parameters:

$ picocom --baud 19200 \
  --flow h \
  --parity n \
  --databits 8 \
  --stopbits 1 \
  /dev/ttyUSB0

After starting Picocom, I can download files from my Linux PC by selecting: Transfer -> Download in the NComm menu. This action opens a file dialog on my Linux machine that allows me to pick the files that I want to download.

Similarly, I can upload files to my Linux machine by selecting Transfer -> Upload. On my Linux machine, a file dialog appears that allows me to pick the target directory where the uploaded files need to be stored.

In addition to simple file exchange, I can also expose a Linux terminal over a serial port and use my Amiga to remotely provide command-line instructions:

$ agetty --flow-control ttyUSB0 19200


To keep the terminal screen formatted nicely (e.g. a fixed number of rows and columns) I should run the following command in the terminal session:

stty rows 48 cols 80

By using NComm's upload function, I can transfer files to the current working directory.

Downloading a file from my Linux PC can be done by running the sz command:

$ sz mod.cool

The above command allows me to download the ProTracker module file: mod.cool from the current working directory.

It is also possible to remotely administer an Amiga machine from my Linux machine. Running the following command starts a shell session exposed over the serial port:

> NewShell AUX:

With a terminal client on my Linux machine, such as Minicom, I can run Amiga shell instructions remotely:

$ minicom -b 19200 -D /dev/ttyUSB0

showing me the following output:


Usage


All these new hardware peripherals open up all kinds of new interesting possibilities.

Using the SD card in FS-UAE


For example, I can detach the SD card from the SCSI2SD device, put it in my PC, and then use the hard drive in the emulator (both FS-UAE and WinUAE seem to work).

By giving the card reader's device file public permissions:

$ chmod 666 /dev/sdb

FS-UAE, that runs as an ordinary user, should be able to access it. By configuring a hard drive that refers to the device file:

hard_drive_0 = /dev/sdb

we have configured FS-UAE to use the SD card as a virtual hard drive (allowing me to use the exact same installation):


An advantage of using the SD card in the emulator is that we can perform installations of software packages much faster. I can temporarily boost the emulator's execution and disk drive speed, saving me quite a bit of installation time.

I can also more conveniently transfer large files from my host system to the SD card. For example, I can create a temp folder and expose it in FS-UAE as a secondary virtual hard drive:

hard_drive_1 = /home/sander/temp
hard_drive_1_label = temp

and then copy all files from the temp: drive to the SD card:


Using the KCS PowerPC board with the new peripherals


The GoTek floppy emulator and the SCSI2SD device can also be used in the KCS PowerPC board emulator.

In addition to Amiga floppy disks, the GoTek floppy emulator can also be used for emulating double density PC disks. The only inconvenience is that it is impossible to format an empty disk on the Amiga for a PC with CrossDOS.

However, on my Linux machine, it is possible to create an empty 720 KiB disk image, format it as a DOS disk, and put the image file on the USB stick:

$ dd if=/dev/zero of=./mypcdisk.img bs=1k count=720
$ mkdosfs -n mydisk ./mypcdisk.img

The KCS PowerPC emulator also makes it possible to use Amiga's serial and parallel ports. As a result, I can also transfer files from my Linux PC by using a PC terminal client, such as Telix:


To connect to my Linux PC, I am using almost the same serial port settings as in the Workbench preferences. The only limitation is that I need to lower my baud rate -- it seems that Telix no longer works reliably for baud rates higher than 9600 bits per second.

The KCS PowerPC board is a very capable PC emulator. Some PC aspects are handled by real hardware, so that there is no speed loss -- the board provides a real 8086/8088 compatible CPU and 1 MiB of memory.

It also provides its own implementation of a system BIOS and VGA BIOS. As a result, text-mode DOS applications work as well as their native XT-PC counterparts, sometimes even slightly better.

One particular aspect that is fully emulated in software is CGA/EGA/VGA graphics. As I have explained in a blog written several years ago, the Amiga uses bitplane encoding for graphics whereas PC hardware uses chunky graphics. To allow graphics to be displayed, the data needs to be translated into planar graphics format, making graphics rendering very slow.

For example, it is possible to run Microsoft Windows 3.0 (in real mode) in the emulator, but the graphics are rendered very very slowly:


Interestingly enough, the game: Commander Keen seems to work at an acceptable speed:


I think Commander Keen runs so fast in the emulator (despite its slow graphics emulation), because of the adaptive tile refresh technique (updating the screen by only redrawing the necessary parts).

File reading problems and crashes


Although all these replacement peripherals are nice, such as the SCSI2SD, I was also running into a very annoying recurring problem.

I have noticed that after using the SCSI2SD for a while, sometimes a file may get incorrectly read.

Incorrectly read files lead to all kinds of interesting problems. For example, unpacking an LhA or Zip archive from the hard drive may sometimes result in one or more CRC errors. I have also noticed subtle screen and audio glitches while playing games stored on the SD card.

A really annoying problem is when an executable is incorrectly read -- this typically results in program failure crashes with error codes 8000 0003 or 8000 0004. The former error is caused by executing a wrong CPU instruction.

These read errors do not seem to happen all the time. For example, reading a previously incorrectly read file may actually open it successfully, so it appears that files are correctly written to disk.

After some investigation and comparing my SD card configuration with the old SCSI hard drive, I have noticed that the read speeds were a bit poor. SysInfo shows me a read speed of roughly 698 KiB per second:


By studying the MacroSystem Evolution manual (in German) and comparing the configuration with the Workbench installation on the old hard drive, I discovered that there is a burst mode option that can boost read performance.

To enable burst mode, I need to copy the Evolution utilities from the MacroSystem evolution driver disk to my hard drive (e.g. by copying DF0:Evolution3 to DH0:Programs/Evolution3). and add the following command-line instruction to S:User-Startup:

DH0:Programs/Evolution3/Utilities/HDParms 0 NOCHANGE NOFORMAT NOCACHE BURST

Resulting in read speeds that are roughly 30% faster:


Unfortunately, faster read speeds also seem to dramatically increase the likelyhood on read errors making my system quite unreliable.

I am still not completely sure what is causing these incorrect reads, but from my experiments I know that read speeds definitely have something to do with it. Restoring the configuration to no longer use burst mode (and slower reads), seems to make my system much more stable.

I also learned that these read problems are very similar to problems reported about a wrong MaxTransfer value. According to this page, setting it to 0x1fe00 should be a safe value. I tried adjusting the MaxTransfer value, but it does not seem to change anything.

Although my system seems to be stable enough after making these modifications, I would still like to expand my knowledge about this subject so that I can fully explain what is going on.

Conclusion



It took me several months to figure out all these details, but with my replacement peripherals, my Commodore Amiga 500 works great again. The machine is more than 29 years old and I can still run all applications and games that I used to work with in the mid 1990s and more. Furthermore, data exchange with my Linux PC has become much easier.

Back in the early 90s, I did not have the luxury to download software and information from Internet.

I also learned many new things about terminal connections. It seems that Linux (because of its UNIX heritage) has all kinds of nice facilities to expose itself as a terminal server.

After visiting the home computer museum, I became more motivated to preserve my Amiga 500 in the best possible way. It seems that as of today, there are still replacement parts for sale and many things can be repaired.

My recommendation is that if you still own a classic machine, do not just throw it away. You may regret it later.

Future work


Aside from finding a proper explanation for the file reading problems, I am still searching for a real replacement floppy drive. Moreover, I still need to investigate whether the Genlock device can be repaired.