Porting software to AmigaOS (unconventional style)

As I have mentioned in my previous blog post (in which I reflected over the last year), the Amiga is still not quite dead yet and (for some reason) my blog post about my good ol' Amiga has attracted much more visitors than I expected. In fact, for a long time this was the most interesting story I have written, while most of my other blog posts are mostly about the research that I'm currently doing.

Nowadays, there is still an active community developing Amiga software. For that reason, I have decided to pay some attention to this subject once more. This time I'm going to make it a bit more related to the research I'm currently doing. If you are curious to see how I achieve that goal, please read on...

Geek Gadgets

One of the interesting efforts done in the past is a project called Geek Gadgets, in which the GNU toolchain and many other UNIX utilities are ported to AmigaOS. You could see this effort as something similar to Cygwin, providing a UNIX environment and GNU toolchain on Microsoft Windows.

Interesting packages included in this project are:

• ixemul.library. A BSD 4.3 kernel running under AmigaOS, which internally translates a number system API calls to AmigaOS kernel calls. The API is specifically modelled after NetBSD.
• libnix. An ANSI C API library directly using the AmigaOS libraries, instead of the Unix compatibility layer. This library is better suited for developing native AmigaOS applications not requiring any UNIX functionality.
• GNU Compiler Collection. The free/open-source/ubiquitous compiler suite used on many platforms, such as Linux, Cygwin, FreeBSD, OpenBSD, NetBSD and Mac OS X.
• Korn Shell. Geek Gadgets provides the pdksh shell as a Bourne-compatible shell instead of bash, which is commonly used on other GNU systems. Probably because it's smaller and faster.

These tools make it possible to port Unix-like applications to AmigaOS as well as writing native AmigaOS applications, using development tools commonly found on free software systems, such as Linux.

Quite a number of applications are ported using this toolset such as TeX, Emacs, the X Window System and SDL, a library which is commonly used on Linux to develop games and other multimedia applications.

Using Geek Gadgets

Below, I have included a screenshot showing the Amiga Workbench and an Amiga Shell. In this Amiga shell, I have opened a Korn Shell session to show the contents of the Geek Gadgets toolset and the version of the GCC compiler.

As you may notice in the screenshot above, the /gg directory is the Geek Gadgets root directory in which a typical UNIX-like filesystem hierarchy is defined with its associated files and utilities.

The Amiga filesystem is organized quite differently as a UNIX filesystem. Whereas a UNIX filesystem is a tree with only one root directory, an Amiga filesystem has many roots. For example, on the Amiga a path can be identified by Assign:RootDirectory/Subdirectory/File, in which an assignment can represent a disk drive identifier, disk label or a random assignment which you can attach to any folder in the filesystem.

Like Cygwin, the ixemul kernel also provides a UNIX to native filesystem mapping, by defining assignments as root directories. For example the /gg directory represents the GG: assignment in which the Geek Gadgets files are stored.

Deploying Geek Gadgets

Deploying a Geek Gadgets environment was a bit trickier than I expected:

• The latest stable version is much too old, dating from 1998. The Geek Gadgets documentation is quite outdated and does not accurately describe what packages you need from the latest snapshot.
• In order to use any Geek Gadgets tools, I need to manually set up the Geek Gadgets environment each time, by executing the GG:Sys/S/GG-Startup script and by raising the stack size, because otherwise I can get mysterious lock-ups/crashes. Of course, I could also add these instructions to S:User-Startup, but I don't want the Geek Gadgets to spoil up my regular system when I use native AmigaOS applications.
• Geek Gadgets uses a traditional UNIX filesystem organisation, with all its disadvantages. For example, packages are deployed in an impure manner because their contents are scattered across the filesystem. I don't like it very much to let development packages pollute my filesystem.

Most importantly, my research as a PhD student is about software deployment in which I try to deal with such inconveniences. For Amiga readers not knowing anything about it, one of our key aspects is the Nix package manager, which we use to build software components from declarative build specifications. Furthermore, Nix has several distinct features such as the ability to store components in isolation, atomic upgrades and rollbacks and garbage collector which safely removes components no longer in use.

Nowadays, I use the Nix package manager for virtually everything that I develop and I have to compile, because it stores all builds isolated and it never pollutes my system. Furthermore, I can also easily use the Nix package manager to build several variants of the same package (e.g. for i686-linux and x86_64-linux), just by modifying several build parameters in the Nix expression for that particular package.

Building AmigaOS applications with Nix

So an interesting question that I have raised to myself is: "Can I use my research about software deployment to deal with these deployment inconveniences of Geek Gadgets on AmigaOS?".

An answer that you probably would expect me to give is that I have ported the Nix package manager to AmigaOS, but unfortunately that isn't possible because Nix requires much more memory than an Amiga has. Furthermore, the ixemul library does not completely support the POSIX API and does not support symlinks, which makes it impossible to run the Nix package manager properly.

Instead, I have developed a Nix function, which should be invoked on a Linux machine. This function launches UAE, the Ultimate Amiga Emulator, to perform a build and stores the build result in the Nix store of the host system. To develop this Nix function I had to create a basic disk image and a function that starts UAE with this disk image in which the actual build is performed.

Creating a disk image

Creating a basic disk image is a straightforward process:

• UAE must be configured to use a host filesystem directory as a hard drive partition. In the UAE GUI this can be enabled by clicking on 'New filesystem...' button in the 'Harddisks' tab.
• The Amiga Workbench 3.1 diskettes are required to perform a hard drive installation of the Amiga Workbench. The install disk contains a wizard which takes care of these steps. I performed an 'Immediate user installation' in which I only installed the English language locale, no printers and an American keymap.
• I have also installed the LhA archiver, because this is the most widely used archiver on the Amiga. I have obtained this version from Aminet: http://aminet.net/package/util/arc/lha. This file is a self extracting archive, which can be installed by running the following instructions from the command-line interface:
  

  T:
  Protect lha.run +e
  lha.run
  Copy lha_68k C:lha
  

Installing the Geek Gadgets environment is a little trickier though:

• I used this simple README file from Aminet as a reference describing the base packages I need: http://de4.aminet.net/dev/gg/0README-GG.txt
• I did not download the Geek Gadgets packages from Aminet, but instead I downloaded the latest snapshot, which I found here: ftp://ftp.back2roots.org/pub/geekgadgets/amiga/m68k/snapshots/990529/bin.
• Instead of gcc, I have downloaded egcs. Additionally, I have downloaded a2ixlibrary and gawk, which are frequently required to build packages. libg++ is not included in the latest snapshot and is apparently also not required.
• I've extracted all the tarballs on the host system into the GG subfolder of the Amiga drive, because this is much faster. I had to fix to bin/sh symlink to the ksh manually, because the symlink in the tarball is incorrect.
• A GG: assignment is required so that Geek Gadgets knows where to find its files. This assignment can be made by adding the following line to the S:User-Startup file:
  

  Assign >NIL: GG: DH0:GG
  

• If you want to create native AmigaOS applications, you need to copy the Amiga system headers into GG:os-include. I have found a collection of Geek Gadgets compatible os-include headers here: http://ftp.back2roots.org/back2roots/cds/fred_fish/geekgadgets_vol1_9610/ade-bin
• In order to automatically perform builds from Nix expressions, we need to start the build process right after the system is booted. To make this possible I commented out the last two lines from the S:Startup-Sequence, to disable the Workbench and to boot into command-line mode:
  

  ; C:LoadWB
  ; EndCLI >NIL:
  

  
  I have added the following lines to the S:User-Startup script to initialize the Geek Gadgets environment and to automatically start the build process:
  

  Execute GG:Sys/S/GG-Startup
  Stack 200000
  
  DH0:T
  sh build.sh
  

Writing a Nix function

Now that we have a basic disk image containing the Amiga Workbench and the Geek Gadgets utilities, we need to write a Nix function that automatically performs the build steps. I will list pieces of the code of this Nix function here, with some explanation.

The AmigaOS build function, takes 4 arguments where 1 argument is optional:

{name, src, buildCommand, buildInputs ? []}:

The name parameter specifies the name of the component, which also appears in the resulting Nix store path. The src parameter is used to specify the source code that must be compiled. This parameter could refer to a tarball, or a directory containing source code. You could also bind this parameter to a function, such as fetchurl to remotely obtain a source tarball. The buildCommand parameter is a string which specifies how the source code can be built. The buildInputs parameter can be used to install additional packages into the Geek Gadgets environment, such as libraries or other build tools which aren't included in the basic disk image.

The next line imports all the Nix package build functions into the scope of our AmigaOS build function, so that we can conveniently access UAE and other utilities:

with import <nixpkgs> {};

We also need to specify where we can find the disk image containing the Workbench and Geek Gadgets. The UAE emulator also needs a kickstart ROM image. The following code fragment specifies where to find them (you need to replace these paths to match your situation):

let
  amigaBase = ./amigabase;
  kickRom = ./kickrom/kick.rom;
in

The following lines invoke the stdenv.mkDerivation function, which we need to specify how we can actually perform a build. The following code fragment inherits the name from our function header. Furthermore, it adds the UAE emulator in our environment so that we can boot our disk image and perform the compilation. We need procps to kill the emulator, once the build has finished:

stdenv.mkDerivation {
  inherit name;
  
  buildInputs = [ uae procps ];
 
  buildCommand = ''

The remainder of the function specifies how the build should be performed. As we're not allowed to mess up the original base image (it's read-only anyway during a Nix build), we need to create a new one in our temp directory of our build. Fortunately, most parts of the AmigaOS filesystem can be used in read-only mode, so we only need to symlink these system directories. The following code fragment creates symlinks to all static AmigaOS directories:

    mkdir hd
    cd hd
       
    for i in ${amigaBase}/{C,Classes,Expansion,Fonts,L,Libs,Locale} \
      ${amigaBase}/{Prefs,Rexxc,S,Storage,System,Tools,Utilities}
    do
        ln -s "$i"
    
        # Symlink icons
        if [ -f "$i.info" ]
        then
            ln -sf "$i.info"
        fi
    done

The Geek Gadgets environment however, must be copied to our new temporary hard drive, because we may need to install other packages in this environment. Additionally, we need to fix the permissions of the copied GG directory, because in the Nix store it has been made read-only and we require a writeable GG directory:

    cp -av ${amigaBase}/GG .
    chmod 755 GG
    
    cd GG
    
    for i in `find . -type d` `find . -type f`
    do
        chmod 755 $i
    done

If we have any build inputs (such as library packages), we need to unpack them into the Geek Gadgets environment:

    for i in ${toString buildInputs}
    do
        cp -av $i/* .
    done

The next step is copying the source code into a location in which a build can be performed. For this purpose we create a temp T directory in our AmigaOS filesystem. If the source code is a directory, we need to strip the hash off it and we need to make it writable again:

    cd ..
    
    mkdir T
    cd T
    
    stripHash ${src}
    cp -av ${src} $strippedName
    
    if [ -d "$strippedName" ]
    then
        chmod -R 755 "$strippedName"
    fi

Then we have to execute the instructions defined in the buildCommand somehow. We generate a shell script called buildinstructions.sh containing the contents of the buildCommand string. This script is executed at boot time by the Korn shell.

    echo "src=$strippedName" > buildinstructions.sh
    
    cat >> buildinstructions.sh << "EOF"
    ${buildCommand}
    EOF

We can't directly invoke the build instructions, because we also have to determine whether a build has succeeded or failed. The following code fragments creates an additional script called: build.sh, which invokes the buildinstructions.sh script. After the build instructions script has finished, it determines whether it has succeeded or failed and writes the status into a file called done which is stored in the resulting Nix store path:

    cat > build.sh << "EOF"
    ( sh -e buildinstructions.sh
      
      if [ $? = 0 ]
      then
          echo "success" > /OUT/done
      else
          echo "failure" > /OUT/done
      fi
    ) 2>&1 | tee /OUT/log.txt
    EOF

We now configured a bootable disk image, which automatically performs a build. The next step is to provide a number of settings to UAE so that our temporary disk image can be used and that a build is performed properly:

    cd ../..
    
    # Create UAE config file
    export HOME=$(pwd)
    
    cat > .uaerc <<EOF
    config_description=UAE default configuration
    use_gui=no
    kickstart_rom_file=${kickRom}
    sound_output=none
    fastmem_size=8
    chipmem_size=4
    cpu_speed=max
    cpu_type=68ec020
    gfx_linemode_windowed=double
    filesystem2=rw,:HD0:$(pwd)/hd,1
    filesystem=rw,HD0:$(pwd)/hd
    filesystem2=rw,:OUT:$out,0
    filesystem=rw,OUT:$out
    EOF

The code fragment above defines our temporary build directory as HOME directory (In a Nix build environment this variable is set to a non-existent path: /homeless-shelter in order to remove side effects). Furthermore, we generate a UAE configuration file with the following properties:

• We disable the GUI, because this prevents us running UAE non-interactively.
• We need to specify where the Kickstart ROM image can be found. This property is defined earlier in the kickRom variable.
• We disable sound support, because we don't need it and it may cause lock ups in some cases.
• The Geek Gadgets tools require quite an amount of RAM to run properly, so we've configured the UAE instance to use 2 MiB of Chip RAM and 8 MiB of Fast RAM.
• Normally the UAE emulator tries to accurately match the speed of a 7 MHz Motorla 68000 chip, which makes building software very slow compared to nowadays' standards. Fortunately, UAE can also use the host CPU as efficiently as possible, which we do in our build function to make builds significantly faster.
• We also configured two hard drive instances. The HD0: drive corresponds to the disk image we have created earlier containing the Workbench and Geek Gadgets. The OUT: drive corresponds to the Nix store output path, in which we have to store our build result.

After configuring UAE, we have to run it. Since UAE is a GUI application requiring a X11 display server, we have to specify the DISPLAY environment variable:

    mkdir -p $out

    export DISPLAY=:0
    
    # Start UAE
    uae &

There is no way to shut UAE down from an Amiga session, so we wait until we see the done file in our output directory. Once this file exists we kill the UAE process:

    while [ ! -f $out/done ]
    do
        sleep 1
    done
    
    # Kill the emulator
    pkill uae
    sleep 1

    # Check the build status
    [ "$(cat $out/done)" = "success" ]

After executing all the previous steps, the build has either succeeded or failed. If a build has succeeded, we can remove the done file, as well as some temp files created by UAE:

    cd $out
    rm done
    
    rm -f `find . -name _UAEFSDB.___`
  '';
}

Using the Nix function

The following code fragment shows how this Nix function can be used to build GNU Hello for AmigaOS:

import ./amigaosbuild.nix {
  name = "hello";
  src = fetchurl {
    url = mirror://gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
  };
  buildCommand = ''
    tar xfvz $src
    cd hello-2.1.1
    ./configure --prefix=/OUT
    make || true
    make install
  '';
}

The expression above looks almost identical to the regular Nix expression building GNU Hello. Important to point out is that we store the output of the GNU Hello world in the /OUT directory, which corresponds to the OUT: drive. The GNU Hello build apparently requires help2man to generate a manual page. This tool is not present in our environment, and to make the build succeed I've added || true to the make instruction, so that it always succeeds. By running the following command line instruction:

$ nix-build hello.nix

The GNU Hello component gets build for AmigaOS, which looks like this:

I have also tried compiling a native AmigaOS application, which creates a very simple Intuition GUI window. I wrote the following test program for this purpose:

#include <exec/types.h>
#include <intuition/intuition.h>
#include <intuition/intuitionbase.h>
#include <intuition/screens.h>

#include <clib/exec_protos.h>
#include <clib/dos_protos.h>
#include <clib/intuition_protos.h>

#include <stdio.h>

#define INTUITION_VERSION 37

struct Library *IntuitionBase = NULL;

struct TagItem win_tags[] = {
    {WA_Left, 20},
    {WA_Top, 20},
    {WA_Width, 300},
    {WA_Height, 120},
    {WA_CloseGadget, TRUE},
    {WA_Title, "Hello world!"},
    {WA_IDCMP, IDCMP_CLOSEWINDOW},
    {TAG_DONE, NULL}
};

void handle_window_events(struct Window *window)
{
    WaitPort(window->UserPort);
}

int main(int argc, char *argv[])
{
    IntuitionBase = OpenLibrary("intuition.library",
      INTUITION_VERSION);
    
    if(!IntuitionBase)
    {
        fprintf(stderr, "Error opening intuition library!\n");
        return 1;
    }
    else
    {
        struct Window *window = OpenWindowTagList(NULL, win_tags);
 
        if(window == NULL)
            fprintf(stderr, "Window failed to open!\n");
        else
        {
            handle_window_events(window);
            CloseWindow(window);
        }
    
        CloseLibrary(IntuitionBase);
        return 0;
    }
}

And this is what the result looks like:

So apparently it works fine!

Conclusion

This blog post covers one of the craziest experiments I have done so far. I have developed a Nix function, which builds software for AmigaOS using the Geek Gadgets toolset by creating a disk image and by using UAE.

After reading this blog post (in case you're still interested :-) ), you may wonder if I actually have a real use case for this. The answer is: YES, although I'm not going to reveal what it is yet. Probably it will take a while, because I currently don't have that much spare time for fun projects like these.

I have decided not to commit this function to Nixpkgs, because it's quite hacky and it depends on stuff (like the Amiga Workbench) which I cannot distribute freely. It's probably not so hard to recreate this function yourself. You can freely use the code I have written under the MIT license.

References

• In order to use this build function you need an Amiga Kickstart ROM image and the Amiga Workbench diskettes. If you don't have these, the legal way to obtain these is by ordering one of the Cloanto's Amiga Forever discs. Alternatively, you could try AROS, a free/open-source operating system reimplementing the AmigaOS APIs, although I have no experience with it so far.
• Performing builds in virtual machines inside a Nix expression is nothing new. We have similar approaches in our Hydra build farm, in which we generate OpenSUSE, Ubuntu and Fedora machines to build RPM and Debian packages from Nix expressions. NixOS virtualization provides additional advantages, which I have described here.

Availability

UPDATE: I didn't expect this, but it seems that there are people around who actually want to use this. I have recently put some effort in it to make it a bit usable. The source code can be found on the nix-amigaosenv GitHub page, which contains updated instructions. Furthermore, I have made some enhancements, so that builds can be performed much faster and cheaper.