Mounting a KCS PowerPC board emulated PC hard-drive partition in Linux (or: an exercise in writing a simple network block device server)

In my previous blog post, I have explained that AmigaOS is a flexible operating system supporting many kinds of filesystems through custom DOS drivers. For example, in the 90s, I frequently used the CrossDOS driver to get access to PC floppy disks.

In addition, my Amiga 500 was extended with a KCS PowerPC board making it possible to emulate an XT PC and run PC applications. This PC emulator integrates with all kinds of Amiga peripherals, such as the keyboard, mouse, floppy drives, and hard drives.

Although I could easily read PC floppies from AmigaOS, it was not possible for me to mount partitions from my KCS PowerPC board emulated PC hard drive. The reason is that the hard drive data is modified for efficiency -- each pair of bytes is reversed for more efficient processing on the Motorola 68000 CPU, which is a big endian CPU. For example, the typical signature of a Master Boot Record (MBR) is stored in reversed order: rather than 0x55AA, it is stored as 0xAA55.

In my previous blog post, I have shown a custom developed AmigaOS device driver that unreverses bytes and translates offsets in such a way that the addressing of the PC hard drive starts at offset 0. By using this device driver, Amiga filesystem DOS drivers can work with a virtual PC hard-drive.

Although it is convenient to be able to manage the files of my PC partition from AmigaOS, not all my inconveniences have been solved. With my SCSI2SD replacement for a physical hard drive, I can also conveniently transfer data to my Amiga hard drive from my Linux PC using my PC's card reader.

Sometimes I also want to transfer files from my Linux PC to my emulated PC partition, such as custom developed software. Unfortunately, on Linux I ran into the same problem as AmigaOS -- Linux has a FAT filesystem driver, but it cannot be used because the partition's raw data is not in a format that the filesystem module understands.

In addition to AmigaOS, I have also worked on a solution that makes it possible to cope with this problem on Linux. In this blog post, I will explain the solution.

Looking for a solution

Linux is a UNIX-like operating system. One of the concepts that UNIX is built on is that everything is a file. In addition to regular files, access to many kinds of storage media are also provided through files, namely: block device files.

Mounting a file representing a storage medium (this can be a device file, but also a regular file containing a dump of a medium, such as a floppy or CD-ROM) to a directory, makes it possible to browse the contents of a storage medium from the file system.

Linux and other kinds of UNIX-like operating systems (such as FreeBSD, NetBSD and OpenBSD) support many kinds of file systems, including the FAT filesystem that is commonly used on MS-DOS and many versions of the Windows operating system.

Because for efficiency reasons pairs of bytes have been reversed on my PC emulated Amiga partition, the raw data coming from the storage medium is not in a format that the FAT filesystem module understands.

The question that I came up with that is: how to cope with that? How can we provide a file that actually does provide the raw data in the right format, in which the bytes have been unreversed?

I have been thinking about various kinds of solution directions. Some impractical directions I have been thinking about were the following:

• Working with hard-drive dumps. A simple but crude way to get access to the data is to make a dump (e.g with the dd command) of the emulate PC hard-drive partition and to run a program that unreverses the pair of bytes. I can mount the modified dump as a loopback device and then read the data.
  
  The downside of this approach is that it requires me to store a copy of the entire drive (although 100 MiB for nowadays standards' is not much) and that modifying data requires me write back the entire dump back to the SD card, which is costly and inconvenient.
• Using a named pipe (also known as a FIFO). I could write a program that unreverses the bytes, using the SD card's device file as an input and a named pipe (FIFO) as an output. A named pipe can be accessed as a file from the filesystem.
  
  Unfortunately, random access is not possible with a named pipe -- it is only possible to traverse a collection data from the beginning to the end. As a result, I cannot mount a FIFO to a directory, because filesystem modules require random access to data.
• Developing a custom kernel module. To be able to mount a device file file that provides the raw data in the right format I could develop a kernel module. Device files are a means to communicate with a kernel module.
  
  Although I have some basic experience developing Linux kernel modules, I consider this approach to be quite impractical for my use case:
  
  • Linux does not have a stable kernel API, so custom module development requires maintenance each time a new Linux version is released.
  • Deploying a Linux kernel module is not very convenient -- you need special user privileges to do that.
  • Code running in the kernel is privileged -- incorrectly implemented code could crash the entire system.
  • It is very unlikely that a kernel module for such a niche use case gets accepted into the mainline kernel.

After some more thinking and talking to people, I learned that I am not the only one struggling with raw hard-drive data that is not in the right format. For example, QEMU is a modern machine emulator that also emulates hard-drives. Hard drive data is typically stored in a space efficient way -- the qcow format, in which data is only allocated when it is actually needed.

There are also facilities to get access to these virtual QEMU hard drives from the host system or a remote system in a network. QEMU provides a utility named: qemu-nbd to make this possible. With this tool, it is possible, for example, to mount the contents of a qcow2 hard-drive image on the host system.

After learning about this tool, I also learned more about NBD. NBD is an abbreviation for Network Block Device and offers the following features:

• It makes block driver access possible over the network. NBD is realized by three components: a server, a client and the network between them. A server exports data (in blocks of a fixed size) and is typically implemented as a userspace process rather than functionality that runs in the kernel.
• A client runs on a client machine and connects to an NBD server over a network using the NBD protocol. It makes a network block device accessible through an ndb device file, which is also a block device file.
• Networking also uses file concepts on UNIX-like operating systems -- as a result, I can also efficiently connect to a local NBD server (without using a network interface) by using a Unix domain socket.

Because NBD makes it possible to use a userspace process to export data and this data can be exported at runtime, it looks like the ideal ingredient in solving my problem.

As a result, I have decided to look into developing a custom network block device server for my use case.

Developing a nbdkit plugin

How hard is to implement your own NDB server? After some searching, I discovered that developing your own NDB server is actually pretty easy: nbdkit can be used for this.

ndbkit manages many low-level aspects for you. As a developer, you need to construct an instance of struct ndbkit_plugin that defines an NDB configuration and register it as a plugin.

Most of the members of this struct are callback functions that get invoked during the life-cycle of the server:

static struct nbdkit_plugin plugin = {
    .name = "kcshdproxy-plugin",
    .version = VERSION,
    .longname = "KCS HD Proxy plugin",
    .description = "NBD plugin for accessing KCS PowerPC board emulated hard drives",
    .config = kcshdproxy_config,
    .config_complete = kcshdproxy_config_complete,
    .open = kcshdproxy_open,
    .get_size = kcshdproxy_get_size,
    .pread = kcshdproxy_pread,
    .pwrite = kcshdproxy_pwrite,
    .close = kcshdproxy_close
};

NBDKIT_REGISTER_PLUGIN(plugin);

The above plugin configuration defines some metadata (e.g. name, version, long name and description) and a collection of callbacks that have the following purpose:

• The kcshdproxy_config callback is invoked for each command-line configuration parameter that is provided. This callback function is used to accept the targetFile parameter that specifies the device file representing my hard-drive and the offset defining the offset of the Amiga partition that stores the emulated PC hard drive data.
• The kcshdproxy_config_complete callback is invoked when all configuration parameters have been processed. This function checks whether the configuration properties are valid.
• The kcshdproxy_open function opens a file handler to the target device file
• The kcshdproxy_get_size function returns the size of the file representing the hard-drive. This function was the most complicated aspect to implement and requires a bit of explanation.
• The kcshdproxy_pread callback fast forwards to a specified offset, then reads a block of data and finally unreverses each pair of bytes. Implementing this function is straight forward.
• The kcshdproxy_pwrite callback fast forwards to a specified offset, reverses the input block data, writes it, and finally unreverses it again. As with the AmigaOS device driver, unreversing is required because the driver does not re-read a previously written block.
• The kcshdproxy_close function closes the file handler for the target device.

The most tricky part was implementing the kcshdproxy_get_size function that determines the size of the block device. At first, I thought using the stat() function call would suffice -- but it does not. For hard-drive dumps, it (sort of) works because I use the offsets provided by GRUB to create the dump file -- in this usage scenario, the file size exactly matches the Amiga partition size.

It is also possible to use the NDB server with a full dump of the entire hard-drive. Unfortunately, if I do this then the file size no longer matches the partition size, but the size of the entire hard drive.

For device files it is not possible to use stat() -- it will always return 0. The reason why this happens is that size information comes from the inode data structure of a file system. For device files, sizes are not maintained.

Linux has a non-standardized function to determine the size of a block device, but using this function has the same limitation as ordinary files -- this value may not always reflect the partition size, but it could also be something else, such as the size of the entire hard drive.

I ended up using the partition entry information from the Master Boot Record (MBR) to determine the size of the virtual PC hard-drive. Each partition entry contains various kinds of attributes including the offset of the first absolute sector, and the number of sectors that the partition consists of.

I can determine the size of the virtual PC hard-drive by computing the provisional sizes for each partition (a classic MBR allows up to four partitions to be defined) with the following formula:

Provisional size = (Offset of first absolute sector + Number of sectors)
    * Sector size

As explained in my previous blog post, the sector size is always 512.

By computing the provisional sizes for each partition and taking the maximum value of these provisional sizes, I have a pretty reliable indication of the true hard drive size.

Usage

Using the NDB server with a local connection is straight forward.

First, I must determine the offset of the Amiga partition that stores the emulated PC hard-drive data (/dev/sdd is a block device file referring to my card reader). I can use GNU Parted to retrieve this information by changing the offset to bytes and printing the partition table:

$ parted /dev/sdd
unit B
print
Model: Generic- USB3.0 CRW-SD (scsi)
Disk /dev/sdd: 15931539456B
Sector size (logical/physical): 512B/512B
Partition Table: amiga
Disk Flags: 

Number  Start       End          Size        File system  Name  Flags
 1      557056B     104726527B   104169472B  affs1        DH0   boot
 2      104726528B  209174527B   104448000B               KCS
 3      209174528B  628637695B   419463168B  affs1        DH1
 4      628637696B  1073725439B  445087744B  affs1        DH2

In the above output, the second partition (with name: KCS) represents my emulated PC hard-drive. Its offset is: 104726528.

As explained in my previous blog post, in most of the cases, the master boot record (MBR) can be found at the beginning of the emulated PC partition, but sometimes there are exceptions. For example, on my Kickstart 1.3 drive, it is moved somewhat.

I have also created a Linux-equivalent of the searchmbr tool determine the exact offset of the MBR. Running the following command scans for the presence of the MBR by using the partition's offset as a start offset:

$ searchmbr /dev/sdd 104726528
MBR found at offset: 104726528

As can be seen in the output, the MBR's offset is identical to the Amiga partition offset confirming that the MBR is at the beginning of the partition.

I can start an NDB server using a UNIX domain socket: $HOME/kcshdproxy.socket for local connectivity, as follows:

$ nbdkit --unix $HOME/kcshdproxy.socket \
  --foreground \
  --verbose /usr/lib/kcshdproxy-plugin.so \
  offset=104726528 targetFile=/dev/sdd

In the above command-line instruction, I have provided two configuration settings as parameters:

• offset specifies the offset of the emulated PC hard drive. In the example, the value corresponds to our previously discovered partition offset. If no offset is given, then it defaults to 0.
• targetFile specifies the file that refers to the hard disk image. In the example: /dev/sdd refers to my card reader.

To connect to the server and configure a network block device, I can run the following command:

$ nbd-client -unix $HOME/kcshdproxy.socket -block-size 512

After running the above command: /dev/nbd0 refers to the block device file giving me access to the block data provided by my NDB server. /dev/ndb0p1 refers to the first partition.

I can mount the first partition with the following command:

$ mount /dev/nbd0p1 /mnt/kcs-pc-partition

And inspect its contents as follows:

$ cd /mnt/kcs-pc-partition
$ ls
 COMMAND.COM    DRVSPACE.BIN  GAMES
 AUTOEXEC.BAT   CONFIG.SYS    IO.SYS
 AUTOEXEC.OLD   DOS           MSDOS.SYS

As can be seen in the output above, the data is accessible from my Linux system. Isn't it awesome? :-)

Conclusion

In this blog post I have shown a custom developed NDB server that makes it possible to mount a KCS PowerPC board emulated PC hard drive partition in Linux.

Availability

My ndbkit plugin can be obtained from my GitHub page and used under the terms and conditions of the MIT license.

Mounting a KCS PowerPC board emulated PC hard-drive partition in AmigaOS (or: an exercise in writing a simple AmigaOS device driver)

In my previous two blog posts, I have shown that my Amiga 4000 has the ability to run multiple operating systems -- in addition to AmigaOS, it can also run Linux and NetBSD, albeit with some challenges.

Something that these three operating systems have in common is that their file system capabilities are very powerful -- they are flexible enough to support many kinds of file systems through custom modules.

For example, both Linux and NetBSD have an Amiga Fast Filesystem module making it possible for me to conveniently exchange data with my Amiga partition. The opposite scenario is also possible -- there are custom file system handlers available for AmigaOS allowing me to exchange data with my Linux Ext2 and NetBSD Berkey Fast Filesystem partitions.

The ability to exchange files with different operating systems is not a new experience to me. When my Amiga 500 was still mainstream, I was already accustomed to working with PC floppy disks. Amiga Workbench 2.0 bundles a file system driver: CrossDOS that makes it possible to exchange data with PC floppy disks.

Although exchanging data with PC floppy disks is possible, there was something that I have been puzzled by for a long time. My Amiga 500 contains a KCS PowerPC board making it possible to emulate an XT-based PC. The user experience is shown in the picture on the top right. The KCS PowerPC board integrates with all kinds of Amiga peripherals, such as its keyboard, mouse, floppy drive, and hard drive (attached to an expansion board).

Unfortunately, I have never been able to exchange files with my emulated PC hard-drive from AmigaOS. Already in the mid 90s, I knew that this should be possible somehow. I have dedicated a great amount of effort in configuring a mount entry for my hard drive, but despite all my efforts I could not make it work. As a result, I had to rely on floppy disks or a null modem cable to exchange data, making data exchange very slow and inconvenient.

Recently, I have decided to revisit this very old problem. It turns out that the problem is simple, but the solution is a bit more complicated than I thought.

In this blog post, I will explain the problem, describe my solution and demonstrate how it works.

The KCS PowerPC board

As I have already explained in a previous blog post, the KCS PowerPC board is an extension card for the Amiga. In an Amiga 500, it can be installed as a trapdoor expansion.

It is not a full emulator, but it offers a number of additional hardware features making it possible to run PC software:

• A 10 MHz NEC V30 CPU that is pin and instruction-compatible with an Intel 8086/8088 CPU. Moreover, it implements some 80186 instructions, some of its own instructions, and is between 10-30% faster.
• 1 MiB of RAM that can be used by the NEC V30 CPU for conventional and upper memory. In addition, the board's memory can also be used by the Amiga as additional chip RAM, fast RAM and as a RAM disk.
• A clock (powered by a battery) so that you do not have reconfigure the date and time on startup. This PC clock can also be used in Amiga mode.

The KCS PowerPC board integrates with all kinds of Amiga peripherals, such as the keyboard, mouse, RAM, joysticks, floppy drives and hard drives (provided by an expansion board).

It can also emulate various kinds of displays controllers, such as CGA, EGA and VGA, various kinds of graphics modes, and Soundblaster and Adlib audio.

Video and audio is completely emulated by using Amiga's own chips and software. Because of the substantial differences between PC graphics controllers and the Amiga chips, graphics emulation is often quite slow.

Although graphics are quite costly to emulate, text mode applications generally work quite well and sometimes even better than a real XT PC. Fortunately, many PC applications in the 80s and early 90s were text based, so this was, aside for games, not a big issue.

Configuring DOS drivers in AmigaOS

AmigaOS has the ability to configure custom DOS drivers. Already in the 90s, I discovered how this process works.

In Amiga Workbench 2.0 or newer, DOS driver configurations can be found in the DEVS:DOSDrivers folder. For example, if I open this folder in the Workbench then I can see configurations for a variety of devices:

(As a sidenote: in Amiga Workbench 1.3 and older, we can configure DOS drivers in a centralized configuration file: DEVS:MountList).

In the above screenshot, the PC0 and PC1 configurations refer to devices that can be used to exchange data with PC floppy disks. When a PC floppy disk is inserted into the primary or secondary disk drive, the PC0: or PC1: devices provide access to its contents (rather than the standard DF0: or DF1:).

I eventually figured out that these DOS driver entries are textual configuration files -- I could, for example, use the PC0: configuration as a template for creating a configuration to mount my hard drive partition (PCDH0:). If I open this configuration file in a text editor, then I see the following settings:

FileSystem     = L:CrossDOSFileSystem
Device         = mfm.device
Unit           = 0
Flags          = 1
Surfaces       = 2
BlocksPerTrack = 9
Reserved       = 1
Interleave     = 0
LowCyl         = 0
HighCyl        = 79
Buffers        = 5
BufMemType     = 0
StackSize      = 600
Priority       = 5
GlobVec        = -1
DosType        = 0x4D534400

With my very limited knowledge in the 90s, I was already able to determine that if I want to open my hard drive partition, some settings need to be changed:

• CrossDOSFileSystem is a filesystem driver that understands MS-DOS FAT file systems and makes them accessible through AmigaDOS function calls. This setting should remain unchanged.
• The device driver: mfm.device is used to carry out I/O operations on a floppy drive. Many years later, I learned that MFM means Modified frequency modulation, a common method for encoding data on floppy disks.
  
  To be able to access my SCSI hard drive, I need to replace this device driver with a device driver can carry out I/O operations for my hard drive: evolution.scsi. This device file is available from the ROM thanks to the autoconfig facility of my expansion board: the MacroSystem Evolution.
• The geometry of the hard drive is different than a floppy drive -- for example, it offers much more storage and the PC partition is typically somewhere in the middle of the hard drive, not at the beginning. Aside from the fact that I knew that the LowCyl and HighCyl settings can be used to specify the boundaries of a partition, I knew that I had to change other configuration properties as well. In the 90s, I had no idea what most of them were supposed to mean, but I was able to determine them by using a utility program: SysInfo.

Fast forwarding many years later, I learned that it is a common habit to measure offsets and sizes of classic storage media in cylinders, heads and sectors. The following picture (taken from Wikipedia) provides a good overview of the concepts:

By requesting the properties of my KCS: partition (the Amiga partition storing the data of my emulated PC drive) in SysInfo, I am able to derive most of the properties that I need to adjust my configuration:

Some of the above properties can be translated to DOS driver configuration properties as follows:

• Unit number translates to: Unit
• Device name translates to: Device
• Surfaces translates to: Surfaces
• Sectors per side translates to: BlocksPerTrack. I must admit that I find the term: "sectors per side" a bit confusing so I am not quite sure how it relates to concepts in the picture shown earlier, Anyway, I discovered by experimentation that this metric is equal to blocks per track setting.
• Reserved blocks translates to Reserved.
• Lowest cylinder translates to LowCyl.
• Highest cylinder translates to HighCyl.
• Number of buffers translates to Buffers.

Some DOS driver configurations also need to know the Mask and MaxTransfer properties of a hard-drive partition. I can discover these settings by using HDToolBox and selecting the following option: Advanced options -> Change File System for Partition:

Finally, to be able to know how to address the offsets in sizes in bytes, I need to know the block size. On the Amiga, 512 is a common value.

If I take the discovered properties into account and make the required changes, I will get the following DOS driver configuration for my hard drive partition (PCDH0:):

FileSystem     = L:CrossDOSFileSystem
Device         = evolution.device
Unit           = 0
Flags          = 1
Surfaces       = 1
BlocksPerTrack = 544
Reserved       = 2
Interleave     = 0
LowCyl         = 376
HighCyl        = 750
Buffers        = 130
BufMemType     = 0
StackSize      = 600
Priority       = 5
GlobVec        = -1
DosType        = 0x4D534400
Mask           = 0xffffff
MaxTransfer    = 0xfffffe
BlockSize      = 512

Although the configuration looks complete, it does not work -- if I try to mount the PCDH0: device and list its contents, then I will see the following error message:

This is as far as I could get in the 1990s. Eventually, I gave up trying because I had no additional resources to investigate this problem.

A deeper dive into the issue

Fast forwarding many years later, there is much I have learned.

Foremost, I learned a couple of things about PC hard drives -- I now know that the first block of every PC hard-drive (consisting of 512 bytes) contains a Master Boot Record (MBR).

A master boot record contains various kinds of data, such as bootstrap code (a prerequisite to boot an operating system from a hard drive partition), a partition table and a fixed signature (the last two bytes) that are supposed to always contain the values: 0x55, 0xAA.

With my SCSI2SD replacement for my physical hard drive, I can easily check the raw contents of the partitions by putting the SD card in the card reader of my Linux PC and inspecting it with a hex editor.

In addition, I found a nice hex editor for the Amiga: AZap, that can also be used to inspect the raw data stored on an AmigaDOS drive. For example, if I open a CLI and run the following command-line instruction:

AZap KCS:

Then I can inspect the raw data of my emulated PC drive (the KCS: partition):

The above screenshot shows the first block (of 512 bytes) of the KCS partition. This block corresponds to the hard drive's MBR.

If I look at its contents, I have noticed something funny. For example, the last two bytes (the typical signature) seems to be reversed -- instead of 55AA, it is AA55.

Furthermore, if I look at the bootstrap code, I see a number of somewhat human-readable strings. As an example, take the string:

"b seutirgnsssyetme"

It should probably not take long to observe what is going on -- the order of each pair of bytes is reversed. If I unreverse them, we will see the following string:

" besturingssysteem"

the word in the middle: besturingssysteem translates to: operating system (I am using a Dutch version of MS-DOS).

So why is the order of each pair of bytes reversed? I suspect that this has something to do with performance -- as I have already explained, the KCS PowerPC board integrates with a variety of Amiga peripherals, including hard-drives. To use Amiga hard drives the emulation software uses AmigaOS device drivers.

Amiga software uses the Motorola 68000 CPU (or newer variants) for executing instructions. This CPU is a big-endian CPU, while the NEC V30 CPU (pin-compatible with an Intel 8086/8088) is a little endian CPU. The order of bytes is reversed in little endian numbers. Because PCs use little-endian CPUs, data formats used on the PC, such as the MBR and the FAT file system typically store numbers in little endian format.

To let run software properly on the Motorola 68000 CPU, it needs to reverse the bytes that numbers consists of. Most likely, the emulation performs better if the byte order does not have to be reversed at runtime.

After this observation I realized that a big ingredient in solving my puzzle is to make sure that each pair of bytes gets unreversed at runtime so that a file system handler knows how to interpret the hard-drive's raw data.

Idea: developing a proxy device driver

As I have already explained, a file system handler typically uses a device driver to perform low-level I/O operations. To cope with the byte reversing problem, I came with the idea to develop a proxy driver providing the following features:

• It relays I/O requests from the proxy driver to the actual hard-disk driver. For requests that work with hard disk data, the driver intercepts the call and reverses the bytes so that the client gets it in a format that it understands.
• Some filesystem handlers, such as fat95, know how to auto detect partitions by inspecting a hard-drive's MBR. Unfortunately, auto detection only works if the MBR is on the first block. An emulated PC hard drive is typically never the first partition on an Amiga hard drive. The proxy driver can translate offsets in such a way that the caller can refer to the MBR at offset 0.

In a nutshell, the proxy device driver offers us a virtual PC hard drive that is accessible from AmigaOS.

A high level overview of some of the AmigaOS components

Implementing an Amiga device driver for the first time requires some learning about the concepts of the operating system. As a result, it is good to have a basic understanding of some of the components of which the operating system consists:

• The kernel is named: Exec. It is (somewhat) a microkernel that is responsible for task management, memory allocation, interrupt handling, shared library and device management, and inter-process communication through message passing.
  
  Compared to more modern/common microkernel operating systems, it does message passing through pointers rather than serializing data. As a result, it is very fast and efficient, but not as secure as true microkernels.
• One of the responsibilities of the kernel is device management -- this is done through device drivers that can be loaded from disk when they are needed, or by device drivers that are already in ROM or RAM. Device drivers use .device as a filename extension.
  
  From the file system, they are typically loaded from the DEVS: assignment. By default, this assignment is an alias for the Devs/ directory residing on the system partition.
  
  A driver is a module typically representing a shared resource, such as a physical device (e.g. a floppy drive). The functionality of a device driver can be used by client applications by sending I/O request objects using the kernel's message passing system.
  
  I/O requests are basically function calls that specify a command number and some data payload. A device driver may also modify the data payload to send an answer, such a block of data that was read.
• AmigaDOS is a sub system providing file system abstractions, a process management infrastructure and a command-line interface. Due to time pressure, it was not developed from scratch but taken from another operating system called: TRIPOS under a license from MetaComCo. AmigaDOS was written in BCPL and interacting with it from the C programming language was not always very efficient. In AmigaOS 2.0, the AmigaDOS sub system was rewritten in C.
• AmigaDOS also has its own drivers (they are often called DOS drivers rather than Exec drivers). DOS drivers can integrate with the file system operations that AmigaDOS provides making it possible work with many kinds of file systems.
  
  Similar to UNIX-like systems (e.g. Linux), also non-storage devices can be accessed through a sub set of file system operations. DOS drivers providing functionality for non-storage devices are typically called handlers. For example, it is possible to redirect output to the serial port (e.g. using SER: as a file output path) or opening a console window (by using CON:) through custom handlers.
  
  DOS drivers are typically loaded from the L: assignment, that by default, refers to the L/ folder on the system partition.
• AmigaOS consists of more parts, such as Intuition: a windowing system and the Workbench: a graphical desktop environment. For this blog post, their details are irrelevant.

Developing a custom driver

I have plenty of C development experience, but that experience came many years after abandoning the Amiga as a mainstream platform. I gained most of my C development experience while using Linux. Back in the 90s, I had neither a C compiler at my disposal nor any C development resources.

In 2012 I learned that a distribution of the GNU toolchain and many common Linux tools exists for AmigaOS: Geek Gadgets. I have used Geek Gadgets to port the tools of my IFF file format experiments to AmigaOS. I have also used it to build a native viewer application for AmigaOS.

After some searching, I discovered a simple AmigaOS driver development example using GCC. Unfortunately, I learned that compiling this example driver with the GCC version included with the Geek Gadgets distribution does not work -- these tutorials rely on compiler extensions (for example, to implement the right calling convention and ordering of functions) that only seem to be supported by much newer versions of GCC. These tutorials recommend people to use amiga-gcc -- a cross compiling toolchain that needs to run on modern computers.

Although I can set up such a cross compilation toolchain, it is way too much complexity for my use case -- the main reason why I previously used GCC (in Geek Gadgets) is to port some of my Linux applications to AmigaOS.

Then I ended up searching for conventional resources that were commonly used when the Amiga was still a mainstream platform. The Amiga Development Reference explains quite well how to use device drivers from client applications. Unfortunately, it does not provide much information that explains how to develop a device driver yourself -- the only information that it includes is an example driver implemented in assembly. I am not scared of a technical challenge, but for a simple driver that just needs to relay I/O requests and reverse data, assembly is overkill IMO. C is a more useful language to do that IMO.

I ended up using the SAS C Compiler (originally it was called the Lattice C compiler) version 6.58. I have managed to obtain a copy several years ago. The compiler also includes a very simple device driver example implemented in C (it resides in the example/example_device sub folder).

A device driver has a very simple structure:

int  __saveds __asm __UserDevInit(register __d0 long unit,
                                  register __a0 struct IORequest *ior,
                                  register __a6 struct MyLibrary *libbase)
{
    /* .... */
}

void __saveds __asm __UserDevCleanup(register __a0 struct IORequest *ior,
                                     register __a6 struct MyLibrary *libbase)
{
    /* .... */
}

void __saveds __asm DevBeginIO(register __a1 struct IORequest *ior)
{
    /* .... */
}

void __saveds __asm DevAbortIO(register __a1 struct IORequest *ior)
{
    /* .... */
}

Only four functions need to be implemented:

• The UserDevInit function is called by the kernel when the device driver is loaded. In my proxy device driver, it opens a connection the actual hard disk device driver. This function returns 0 if the initialization succeeds and 1 if it fails.
• The UserDevCleanup function is called by the kernel when the device driver is unloaded. It closes the connection to the actual device driver and cleans up all related resources.
• The DevBeginIO function is invoked when a device driver receives an I/O request from a client application. This function checks the command type and reverses the data payload, if needed.
• The DevAbortIO function is invoked when a device driver receives an abort request from a client application. This function does nothing in my proxy driver.

In addition to implementing these functions, these functions also require the compiler to follow a certain calling convention. Normally, a compiler has some freedom to choose how function parameters are propagated, but for an Exec device driver, CPU registers need to be used in a very specific way.

The implementation process

As I have explained earlier, device drivers accept I/O requests. I/O requests are function calls consisting of a command number and some data payload. Device driver commands are somewhat standardized -- there are commands that all devices accept, but also device-specific commands. The Amiga development reference provides an overview of device drivers that are bundled with AmigaOS and their commands.

Custom drivers typically follow the same kinds of conventions. For example, the driver for my SCSI controller that comes with my expansion board (evolution.device) accepts the same commands that Commodore's SCSI driver does (scsi.device). This driver is bundled with Kickstart 2.0 and newer.

To implement my proxy driver, I have implemented all the SCSI driver commands described in the manual. For the CMD_READ, CMD_WRITE and TD_FORMAT commands I am using a routine that reverses the order of the bytes in the data field.

Although the implementation process looked simple, I ran into a number of issues. In my first attempt to read my PC partition, the system crashed. To diagnose the problem, I have augmented the driver code with some conditional debugging print statements, such as:

#ifdef DEBUG
    KPrintF("Target device: %s opened successfully at unit: %ld\n", config.device, unit);
#endif

The KPrintF function call sends its output over a serial port. In FS-UAE, I can redirect the output that is sent over a serial port to a TCP socket, by adding the following setting to the emulator's configuration:

serial_port = tcp://127.0.0.1:5000/wait

Then by using a telnet to connect to the TCP socket (listening on port 5000) I can inspect the output of the driver:

$ telnet localhost 5000

I can also connect my null modem cable to my real Amiga 500 and inspect the output in minicom. I need to use a baud rate setting of 9600 bits per second.

With my debugging information, I discovered a number of problems that I had to correct:

• It seems that the CrossDOS and fat95 DOS drivers do not only send SCSI driver commands. They also seem to be sending a number of trackdisk-specific device commands. I had to relay these commands through my proxy driver as well.
• The fat95 driver also works with New Style Devices (NSD). An important feature addition of NSD devices is that they are not restricted by the 32-bit integer limit. As a result, they can handle devices providing more than 4 GiB of storage capacity. They seem to use their own commands -- one of them checks whether a device is a new style device. I was not properly relaying a "command not implemented" error for this case, causing a crash.
• There is a trackdisk command: TD_ADDCHANGEINT to configure a software interrupt that activates on a disk change. It seems that this command never sends an answer message. My driver ran into an infinite loop because it was still expecting an answer. I also learned that in UAE, the uaehf.device driver seems to work with this command, but my real hard drive's SCSI driver (evolution.device) seems to crash when this command is called. I ended up ignoring the command -- it seems to do no harm.
• For write operations, I need to reverse bytes before writing a block, but I also need to unreverse them again after the write operation has completed. It turns out that file system drivers do not re-read previously written blocks.

Although I prefer to avoid debugging strategies, I also know that it is unavoidable for this particular development scenario -- specifications are never complete and third-party drivers may not strictly follow them. You really need to figure out how all of it works in practice.

I am very fortunate to have an emulator at my disposal -- it makes it very easy to have a safe test setup. I also do not have to be afraid to mess up my PC hard-drive -- I can simply make a dump of the entire contents of my hard drive and restore it once my system has been messed up.

In the 90s, I did not have such facilities. Driver development would probably have been much harder and more time consuming.

Usage

Earlier in this blog post, I have shown a non-working mount configuration for my PC hard-drive partition (PCDH0:). With my proxy driver and some additional settings, I can produce a working configuration.

First, I must configure the KCS HD proxy device in such a way that it becomes a virtual PC hard drive device for my emulated PC drive.

I can determine the offset (in bytes) of my KCS partition, by using the following formula:

Offset = Surfaces * Sectors per side * Lowest cylinder * Block size

Applying the formula with the properties discovered in SysInfo results in:

Offset = 1 * 544 * 376 * 512 = 104726528

After determining the partition offset, we must determine the exact offset of the MBR. Typically, it is at the beginning of the Amiga partition storing the PC hard drive data but there may be situations (e.g. when using Kickstart 1.3) that the MBR is moved somewhat.

I have developed a utility called: searchmbr can help you to determine the exact offset (what is does is searching for a 512 bytes block that ends with the MBR signature: 0x55AA):

> searchmbr evolution.device 0 104726528
Checking block at offset: 104726528
MBR found at offset: 104726528

In the output above, we see that the MBR is at the beginning of the Amiga partition.

With this information, I can create a configuration file: S:KCDHDProxy-Config for the proxy driver that has the following contents:

104726528
evolution.device

In the above configuration, the first line refers to the offset of the PC partition (in bytes) and the second line to the device driver of the hard drive.

It turns out that CrossDOS does not know how to interpret an MBR -- it needs to know the exact position of the first PC partition.

(As a sidenote: the fat95 DOS driver does have a feature to autodetect the position of a partition. If you set the LowCyl value to 0, then the fat95 driver will attempt to auto-detect a partition).

After configuring the proxy driver, the querypcparts tool can be used to retrieve the offset and size of the partitions on the PC hard drive:

$ querypcparts 0
Partition: 1, first sector offset: 34, number of sectors: 202878

In my example case, there is only one MS-DOS partition -- it resides at offset 34. Its size is 202878. Both of these properties are measured in sectors.

With the above information, I can make adjustments to our previous mount configuration (of the PCDH0: device). Rather than using the real SCSI driver for our hard drive, we use the KCS HD proxy driver (that reverses the bytes and translates the offsets):

Device = kcshdproxy.device

As explained earlier, partition offsets and sizes are typically specified in cylinders, heads and sectors. A PC hard drive typically does not use the same value for the amount of blocks per track that an Amiga partition does -- as a result, the boundaries of a PC partition are typically not aligned to cylinders.

To cope with this, we can change the following properties to 1, so that we can specify a PC partition's offsets in sectors:

BlocksPerTrack = 1
Surfaces       = 1

Because the driver translates offsets in such a way that the beginning of the virtual PC hard drive starts at 0 and the offsets are measured in cylinders, the LowCyl property should correspond to the offset of the first sector reported by the querypcparts tool:

LowCyl = 34

We can compute the HighCyl boundary with the following formula:

HighCyl = First sector offset + Number of sectors - 1

Applying the formula with the values from the querymbr tool results in the following value:

HighCyl = 202911

After making the above modifications, I can successfully mount the PCDH0: device in AmigaOS and browse it contents in the Amiga Workbench:

Isn't it awesome? :-)

The driver also works in Kickstart 1.3, giving me the classic Amiga 500 experience:

Availability

The KCS HD proxy driver can be obtained from my GitHub page. Currently, it has been tested with the CrossDOS and fat95 filesystem DOS drivers.

The KCS PowerPC board supports many kinds of SCSI controllers -- unfortunately, beyond uaehf.device and my expansion board's SCSI driver: evolution.device, I have not been able to test any other configurations. I believe it should work with any driver that is scsi.device compatible, but I have do not have the means to test this. Use this driver at your own risk!

Future work

Although it is nice to have a solution to easily exchange files with my PC hard drive partition from AmigaOS, not all my problems have been solved -- I also sometimes download stuff from the Internet to my Linux PC that I may want to transfer to my emulated PC partition. With the SCSI2SD device as a replacement hard drive, transferring data to my Amiga "hard drive" is very easy -- I can insert the SD card into the card reader of my PC and get access to its data.

Unfortunately, on Linux, I have the same problem -- Linux has a FAT filesystem driver, but it does not understand the raw data because each pair of bytes is reversed. In the future, I may also have to develop a tool that implements the same kind of solution on Linux.

Running NetBSD on my Amiga 4000

In my last blog post, I have shown how I have been using Linux on my Amiga 4000. Running Linux on an Amiga has always been a fascinating use case to me -- I have developed strong interests in both areas from a very young age.

Shortly after buying my second-hand Amiga 4000 in 2022, I discovered an interesting Reddit article stating that NetBSD (another UNIX-like operating system) supports the Amiga. The title of this article was IMO somewhat confusing, because it suggests that support for the Amiga was recently added, but that statement is false -- NetBSD already supports the Amiga since its 1.0 release (done in 1994). Amiga support was heavily improved in 2022.

NetBSD is an interesting operating system -- it is the first BSD derivative that was forked from 386BSD by a group of developers in 1992 that were dissatisfied with its development process. FreeBSD is a fork started by another group of developers in the same year. OpenBSD was forked from NetBSD in 1995 and DragonFly BSD from FreeBSD in 2003 due to disagreements.

Each of these BSD derivatives have different goals and standards. NetBSD's focus is portability and simplicity. Its slogan is "of course it runs NetBSD" and facilitates a design and infrastructure that makes porting to new platforms relatively easy. As a result, it has been ported to many kinds of CPUs and hardware architectures, including the PC and the Amiga.

The other BSD derivatives have different goals -- OpenBSD is focused on security and correctness and FreeBSD's purpose is to be a more balanced/general purpose BSD derivative. DragonFly BSD's focus is threading and symmetric multiprocessing.

Another interesting story about NetBSD and the Amiga is its porting process. In order to port NetBSD to the Amiga, the build tool chain had to be ported to AmigaOS first. The result of this porting process is a library project called: ixemul.library providing a UNIX system call interface on top of AmigaOS.

The ixemul.library is the main ingredient of the Geek Gadgets distribution that offers a large collection of UNIX utilities that can be used on top of AmigaOS. Some time ago, I have used the Nix package manager to automate package builds with these tools to port my IFF file formats project from Linux to AmigaOS and creating a viewer program for AmigaOS.

Although NetBSD is an interesting operating system, I have never used any of the BSD derivatives as much as I have used Linux. For example, I have never used any of them as a desktop operating system.

I have the most experience with FreeBSD -- I have used it for web servers, as a porting target for the Nix package manager so that we could add another build target to our university's build farm (that uses Hydra: the Nix-based continuous integration system), and as a porting target for my my experimental Nix process management framework.

My experience with OpenBSD and NetBSD is more limited -- the only notable use case I have with them is to use them as porting targets for the Nix package manager, similar to FreeBSD. Apart from a few simple experiments, I never did anything meaningful with DragonFly BSD.

Because of the impact of the article and my past experiences with NetBSD, trying it out on my Amiga 4000 was high on my TODO list. It was a nice experience -- beyond installing NetBSD it was also the first time for me to use it as an operating system to "do work with", rather than a development target.

In this blog post, I will report about my experiences.

Getting started with NetBSD for the Amiga

In my previous blog post (covering Linux on Amiga), I explained that it was a bit of a challenge to start, because much of the information that I need is outdated and scattered.

With NetBSD, the starting process was much easier for me -- NetBSD for the Amiga has IMO a very well written web page from which I can easily obtain the installation CD-ROM ISO file and Amiga-specific installation instructions.

Moreover, contrary to Debian, I can use the latest version of NetBSD (version 10.1 at the time writing this blog post) on my Amiga 4000. With Debian, I was not able to use any version newer than 3.1.

(To be precise: the m68k port was discontinued after Debian 3.1, but re-introduced in Debian 9.0. Unfortunately, 9.0 and newer versions did not work for me. The latest version of Debian is 12.9 at the time writing this blog post).

Installing NetBSD on the Amiga 4000

The installation procedure does not appear to be very user friendly, but is IMO not too difficult if you already have some experience with AmigaOS and a UNIX-like operating system.

To produce a working NetBSD installation on an Amiga I followed their documentation. From a high-level point of view it consists of the following steps:

• Downloading an ISO file and burning it to a writable CD-ROM.
• Booting up my Amiga 4000 and creating a partition layout suitable for using NetBSD. Similar to Linux, NetBSD can be installed next to an existing AmigaOS installation on a separate partition. There are a couple things that need to be kept in mind:
  
  
  • Partitioning must be done by using native AmigaOS tools, such as HDToolBox. NetBSD does not provide its own partitioning tool.
  • We must use HDToolBox to manually specify the correct partition identifiers for the NetBSD swap and root partition. Manual specification is a bit cumbersome, but necessary -- HDToolBox does not recognize these partition types by itself
  • We must make the swap and root partitions bootable. Furthermore, we need to allow custom boot blocks for these partitions. More information about this later.
  • The swap and root partitions must be under the 4 GiB limit. This limit turns out to be a hard limit for my machine as well -- even though my AmigaOS 3.9 homebrew kickstart ROM has a SCSI driver that is not restricted by the 4 GiB limitation, the boot menu does not seem to recognize bootable partitions above this limit.
    
    To address the 4 GiB requirement, I ended up creating a 1 GiB AmigaOS partition containing the Amiga Workbench, followed by a 128 MiB NetBSD swap partition, followed by a 2 GiB NetBSD root partition. The remaining disk space is used for partitions used by AmigaOS.
• Transferring the miniroot filesystem image from the CD-ROM to the swap partition. The miniroot filesystem contains a minimal NetBSD installation capable of performing the actual installation of the NetBSD distribution.
• Booting from the swap partition to set the NetBSD installation procedure in motion. The installer is text-based and asks a number of questions. The package installation process takes several hours on my Amiga to complete.
• Rebooting the machine from the root partition and completing the NetBSD installation.

The NetBSD boot process

Similar to Linux, NetBSD has an interesting boot process -- using these operating systems require the presence of a running kernel. A kernel is not magically there, but needs to be loaded from an external location first. A boot loader is a program that is responsible for facilitating this process.

In my previous blog post, I have explained that are variety of ways to do boot loading. Most importantly, it is possible to load a kernel as quickly as possible after powerup using as few facilities from the ROM or the native operating system. This is a process that I will call cold booting.

In addition, it is possible to first boot into an existing operating system and load a kernel image while the existing operating system is running. Once the kernel is loaded it can take control of the system. This is a process that I called warm booting.

On the Amiga, it is a common practice to warm boot Linux (contrary to Linux on PCs, where cold booting is the normal practice). With NetBSD on Amiga it is actually the opposite -- cold booting is the preferred practice.

As I have already explained in the previous section, we must make the swap and root partitions bootable in HDToolBox. Furthermore, we must allow these partitions to use custom bootblocks.

When powering up my Amiga, I can hold both mouse buttons to enter the boot menu. In the boot menu, I can select the partition that I want to boot from (by default, it traverses the list of bootable drives/partitions top down and takes the first workable option):

In the above screenshot, our NetBSD partitions are also visible as possible boot partitions.

When I select: netbsd-root I can boot from my NetBSD root partition that contains a custom boot block providing a boot loader. The bootloader can be used to load a NetBSD kernel with my desired command-line parameters:

Although cold booting is the preferred practice according to the documentation, it is also possible to warm boot a NetBSD kernel. The loadbsd command can be used for this -- it is an Amiga tool that can load a kernel image from an Amiga partition with a number of kernel parameters. Once the kernel has been loaded, it can take control of the system.

There is yet another way to start NetBSD -- there is also a command-line tool called runbootblock. This tool can be used to automatically execute the code stored in a bootblock of a partition from a running AmigaOS session.

I guess this approach falls in between warm and cold booting -- it is warm booting a kernel from a running operating system using the facilities of a cold booting process. I guess this method can be called lukewarm booting.

Although I am fine with using the Amiga Kickstart's boot menu, I also find it convenient to boot into NetBSD when I am already in the Amiga Workbench. To do that, I have created a desktop icon called StartNetBSD to lukewarm boot into NetBSD:

The icon in the bottom right corner above executes the following script:

runbootblock -dscsi.device -u0 -pnetbsd-root

The above command specifies that we want to run the bootblock of the partition netbsd-root that resides on the first drive unit. We use the scsi.device driver to get access to the partition.

Lukewarm booting seems to work fine if I use the Amiga's chipset for the framebuffer. Unfortunately, when using my Cybervision 64/3D RTG card, the screen looks messed up. I discovered that AmigaOS' initialization of the VGA display is the culprit -- as a result, if I want to use NetBSD in combination with my RTG card, I always need to do a full cold boot.

Post installation steps

After completing the NetBSD installation, there are number of additional configuration steps that I had to perform to get all of my use cases supported. I took heavy inspiration from this NetBSD 9.2 Amiga post installation guide.

Switching off unneeded services

The first thing I noticed after first startup is that my NetBSD installation boots quite slowly. After logging in, it still behaves slowly.

By running ps I figured out that there is a background process: makemandb that consumes quite a bit of my CPU's time. Moreover, I noticed that postfix takes quite a bit of time to start on boot up.

Because I do not need these services, I have disabled them at startup by adding the following lines to /etc/rc.conf:

makemandb=NO
postfix=NO

Configuring user accounts

By default, there is only one user account -- the root user, that has full privileges. Moreover, the root user has no password set.

We can configure the root password by running the following command:

$ passwd

We can configure an unprivileged user account and set a password as follows:

$ useradd -m -G wheel sander
$ passwd sander

Enabling wscons

In the default installation, the command-line experience is somewhat primitive -- I only have a gray background with black colored text and a single terminal window.

To enhance the NetBSD command-line experience, I can enable wscons: NetBSD's platform-independent workstation console driver -- it handles complete abstraction of keyboards and mice and it can multiplex terminals.

To enable wscons, we first need to deploy a kernel that supports it -- the default kernel does not have that functionality built in.

A wscons-enabled kernel can be found on the NetBSD CD-ROM. I can replace the default kernel by logging into my NetBSD installation and running the following commands:

$ mount /dev/cd0a /cdrom
$ cd /
$ cp /cdrom/amiga/binary/kernel/netbsd-WSCONS.gz /
$ gunzip netbsd-WSCONS.gz
$ mv netbsd netbsd-ORIG
$ mv netbsd-WSCONS netbsd

In the above code fragment, I copy the wscons-enabled kernel from the NetBSD CD-ROM to the root directory, uncompress it, make a backup of the old kernel and finally replace the default kernel with the new kernel.

We must also enable the wscons service on startup. This can be done by adding the following line to /etc/rc.conf:

wscons=YES

To use multiple terminals, we must open /etc/ttys in a text editor and enable the ttyE1, ttyE2 and ttyE3 consoles by switching their 'off' flags to 'on':

Finally, we must reboot the system so that we can use the wscons-enabled kernel.

Configuring frame buffer video modes

Another thing I was curious about is how I can configure the output device and resolutions for the framebuffer. For example, when I boot up the kernel, NetBSD automatically uses my Cybervision 64/3D RTG card's display using a 640x480 resolution in 8-bit color mode.

Although I am happy to use my RTG card, I also want to have the ability to use Amiga's AGA chipset for graphics. Moreover, I want to have ability to switch to different graphics modes.

On Linux this can be done by passing the appropriate video parameter to the kernel. On NetBSD, this turns out to be a very tricky configuration aspect.

According to the boot manual page (that describes the NetBSD kernel's boot parameters), there is only one display-related kernel parameter: -A. The documentation says that this parameter enables AGA display mode, but it actually does two things -- it indeed enables AGA mode so that the display uses 256 colors, but it also configures the display to use a "double NTSC" screen mode (640x400, non-interlaced).

Without the -A parameter, the NetBSD kernel is instructed to use 8 colors and an NTSC high resolution interlaced screen mode (640x400). Although my Amiga has an AGA chipset, my LED TV does not seem to accept double NTSC or double PAL screen modes.

In addition to very limited native Amiga chipset configuration abilities, I did not see any kernel parameters that allow me to select my preferred framebuffer device. After some searching, I learned that it is not possible to provide more screen mode settings.

While studying the NetBSD post installation notes GitHub gist, I learned that I am not the only person running into problems with the display. In the gist, the author describes how he adjusted the display settings for the AGA display mode to match the settings of a VGA output by copying the settings from the Linux amifb driver and cross compiling the NetBSD kernel for the Amiga.

I ended up cross compiling a kernel myself as well by following the same instructions. To do the cross compilation, I have downloaded a NetBSD 10.1 distribution ISO file for x64 machines and installed it in a VirtualBox virtual machine on my PC.

In this VirtualBox virtual machine, I have downloaded the NetBSD 10.1 source code by running:

$ cvs -d anoncvs@anoncvs.NetBSD.org:/cvsroot checkout -P -r netbsd-10-1-RELEASE src

I can set up the cross compilation toolchain by running:

$ cd src
$ ./build.sh -m amiga tools

The above command automatically downloads the required dependencies (e.g. the source tarballs for the cross compiler, cross linker etc.) and compiles them.

My first goal was to see if I can make the required adjustments to force the kernel to use the Amiga chips to display the framebuffer. I ended up editing two files and disabling a number of options.

The first file that I have changed is the following: sys/arch/amiga/conf/GENERIC. I have commented out the following properties:

#options 	RETINACONSOLE	# enable code to allow retina to be console
#options 	CV64CONSOLE	# CyberVision console
#options 	TSENGCONSOLE	# Tseng console
#options 	CV3DCONSOLE	# CyberVision 64/3D console

#options 	GRF_AGA_VGA	# AGA VGAONLY timing
#options 	GRF_SUPER72	# AGA Super-72

#grfrt0		at zbus0		# retina II
#grfrh0		at zbus0		# retina III

#grfcv0		at zbus0		# CyberVision 64
#grfet*		at zbus0		# Tseng (oMniBus, Domino, Merlin)
#grfcv3d0	at zbus0		# CyberVision 64/3D

#grf1		at grfrt0
#grf2		at grfrh0

#grf5		at grfcv0
#grf6		at grfet?
#grf7		at grfcv3d0

#ite1		at grf1			# terminal emulators for grfs
#ite2		at grf2			# terminal emulators for grfs

#ite5		at grf5			# terminal emulators for grfs
#ite6		at grf6			# terminal emulators for grfs
#ite7		at grf7			# terminal emulators for grfs

The most important property to comment out is the CV3DCONSOLE. Disabling the Cybervision 64/3D console ensures that my Cybervision 64/3D card never gets detected. As a result, NetBSD is forced to use the native Amiga chipset.

As a side effect of disabling the Cybervision 64/3D console option, I must also disable the corresponding color graphics driver (grfcv3d0), framebuffer device (grf7) and terminal emulator device (ite7).

In addition to Cybergraphics 64/3D, I have been disabling a number of additional features that I do not need. For example, I do not need any support for unorthodox resolutions (VGA, Super72). I can also disable support for additional RTG cards because I do not have them.

As a side effect of disabling their consoles, I must also disable their corresponding color graphics drivers, framebuffer devices and terminal emulation devices.

Another file that I need to modify is the wscons configuration (sys/arch/amiga/conf/WSCONS):

#no grfrt0	at zbus0
#no grfrh0	at zbus0

#no grf1		at grfrt0
#no grf2		at grfrh0

#no ite1		at grf1
#no ite2		at grf2

#no ite5		at grf5
#no ite6		at grf6
#no ite7		at grf7

In the above file, I also need to disable the color graphics, framebuffer and terminal emulation devices that I have commented out in the previous configuration file.

After configuring the kernel, I can compile it with the following command-line instruction:

$ ./build.sh -m amiga kernel=WSCONS

The resulting kernel image can be found in: src/sys/arch/amiga/compile/obj/WSCONS/netbsd. I have copied the resulting kernel image to my NetBSD root partition and named it: netbsd-NORTG.

If I want to use a NetBSD session using my Amiga's chips (with a 8-color interlaced high resolution screen mode), then I can provide the following command to the bootloader:

netbsd-NORTG -Sn2

Resulting in the following NetBSD session:

As can be seen in the picture above, I have successfully instructed the bootloader to load my custom NORTG NetBSD kernel forcing the operating system to use the Amiga display driver.

After compiling my custom NORTG kernel, I have also been experimenting with the kernel source code to see if I can somehow avoid using the incompatible "double NTSC" screen mode. Unfortunately, I discovered that screen modes are hardcoded in the kernel (to be precise, I found them in this file: src/sys/arch/amiga/dev/grfabs_cc.c).

After trying out a few display settings (including the VGA monitor settings described in the GitHub gist) I discovered that if I cut the amount of scan lines in half for the AGA screen mode, I can get an acceptable display in AGA mode (at least in the console).

Unfortunately, despite this achievement I realized that my customized AGA mode is useless -- if I want to use the X Window System, the display server does not seem to know how to use my hacked screen mode. As a result, it fails to start.

I eventually gave up investigating this problem and decided to simply use the non-AGA graphics mode.

Using the X Window System

A nice property of the X Window System on NetBSD/Amiga is that it integrates with wscons. As a result, it has a very minimal configuration process.

Something that appears to be missing is the presence of a pseudo terminal device file. Without it, xterm refuses to start. I can create this missing device file by running the following commands:

$ cd /dev
$ ./MAKEDEV pty0

Another nice property is that, in contrast to the X Window System for Debian Linux/Amiga, the NetBSD version also supports the Amiga chipset for displaying graphics.

When using a NORTG kernel console session, I can simply run: startx and after a while, it shows me this:

The picture above shows a CTWM session with some running applications. The only two downsides of using the X Window System on the Amiga is that it takes quite a bit of time to boot up and I can only use monochrome graphics.

To have a color display, I need to enable AGA mode. As already explained, I cannot use the AGA screen mode because my display cannot handle double PAL or double NTSC screen modes.

Adjusting screen modes

As far as I could see, it is not possible to change screen modes at runtime while using the Amiga chipset.

With my Cybervision 64/3D RTG card it is actually possible to change screen modes at runtime. To do that, I need to load a screen mode definition file. I made an addition to the /etc/rc.local script to load such a definition file at boot time:

$ grfconfig /dev/grf7 /etc/gfxmodes

In the above command-line instruction, the /dev/grf7 parameter corresponds to the Cybervision 64/3D framebuffer device (this device file can be determined by looking at the output of the dmesg command) and /etc/gfxmodes to a screen mode definition file.

Writing a screen mode definition file is a bit cumbersome. Fortunately, from my previous experiments with the Picasso96Mode Amiga preferences tool, I discovered that this tool also has a feature to generate NetBSD 1.2 and 1.3 compatible mode definitions. I learned that NetBSD 10.1 still uses the same format as NetBSD 1.3

To automatically save these mode settings to a file, I need to open a CLI in my Amiga Workbench and run the following command:

SYS:Prefs/Picasso96Mode >T:gfxmodes

The above command opens the Picasso96Mode preferences program and redirects the standard output to T:gfxmodes.

In the Picasso96 GUI, I must select all relevant screen modes and select the menu option: Mode -> Print Mode to export the screen mode (as shown in the picture above).

To get a reasonable coverage, I start with the lowest resolution and color mode (8-bit). Then I move to higher color modes, then higher resolutions etc. until all the screen modes that I need are covered.

After exporting all relevant screen modes, I need to open the T:gfxmodes file in a text editor and remove all comments and NetBSD 1.2 entries. What remains is a file that has the following structure:

x 26249996 640 480 8 640 688 768 792 479 489 492 518 default
x 43977267 800 600 8 800 832 876 976 599 610 620 638 +hsync +vsync
x 67685941 1024 768 8 1024 1072 1184 1296 767 772 777 798 default

In the above file, each line represents a screen mode. The first defines a 640x480 resolution screen, the second an 800x600 resolution screen and the third a 1024x768 resolution screen. All three screen modes use an 8-bit color palette.

We need to slightly adjust this file to allow it to work with NetBSD -- the first column (that contains an 'x' character) represents a mode number. We need to give it a unique numeric value or a 'c' to define the screen mode of the console.

If I want my console to use a 640x480 screen mode, and keep the remaining screen modes as an option, I can change the above mode definition file into:

c 26249996 640 480 8 640 688 768 792 479 489 492 518 default
1 26249996 640 480 8 640 688 768 792 479 489 492 518 default
2 43977267 800 600 8 800 832 876 976 599 610 620 638 +hsync +vsync
3 67685941 1024 768 8 1024 1072 1184 1296 767 772 777 798 default

In the above file, I have replaced the 'x' characters by unique numeric values and I have duplicated the 640x480 mode to be the console screen mode.

By copying the above file to /etc/gfxlogin on my NetBSD root partition, I can switch between screen modes if desired.

I learned that to use the X Window System in combination with my Cybervision 64/3D card, I also require a modes defintion file. By default, the X Window System uses the first screen mode (screen mode: 1). When I start the X server, I eventually get to see the following display:

Installing custom packages

Contrary to Linux, which is a kernel that needs to be combined with other software packages (for example, from the GNU project) to become a functional distribution, NetBSD is a complete system. However, I do need some extra software to make my life more convenient, such as the Midnight Commander.

NetBSD includes a package manager: pkgsrc and a package repository with a variety of software packages. Automatically downloading and installing prebuilt binaries and their dependencies can be conveniently done with a front-end for pkgsrc: pkgin.

Unfortunately, as I have already explained in my previous blog post, my Amiga 4000 machine does not have a network card. The only means I have to link it up to the Internet is a null-modem cable which is too slow for downloading packages.

Fortunately, the web-based interface of pkgsrc is easy enough to manually download the packages I want and their dependencies. For example, to obtain Midnight Commander, I can open the following page: https://ftp.netbsd.org/pub/pkgsrc/current/pkgsrc/sysutils/mc46/index.html, download the m68k tarball, follow the runtime dependencies, download their tarballs, and then the transitive dependencies etc.

To install the downloaded packages on NetBSD, I just copy them to a directory on my NetBSD drive. Then I can install them with the pkg_add command:

$ pkg_add /root/packages/mc-4.6.1nb28.tgz

The result is that I can use Midnight Commander on NetBSD on my Amiga:

Setting up a terminal connection between the Amiga and PC with a null-modem cable

In my previous blog post about running Linux on my Amiga, I have been using a null-modem cable to link the Amiga to my PC. With NetBSD, I can do the same thing. For example, on my Linux PC I can start a terminal session over the serial port:

$ agetty --flow-control ttyUSB0 19200

and use Minicom on my NetBSD/Amiga installation to remotely connect to it.

I can also do the opposite -- I can enable a serial console on NetBSD by editing /etc/ttys and enabling a terminal for the /dev/tty00 device:

tty00    "/usr/libexec/getty std.9600"    unknown    on    secure

Then I can use minicom on my Linux PC to remotely connect to my Amiga:

$ minicom -b 9600 -D /dev/ttyUSB0

The result can be seen in the following picture:

Connecting to the Internet

Similar to my Linux setup, I can also connect my NetBSD/Amiga installation to the Internet. The recipe is exactly the same. I took inspiration from the Linux PPP HOWTO for this.

First, I need to set up a link end point on my Amiga 4000, with the following command:

$ pppd -detach crtscts lock noauth defaultroute 192.168.1.2:192.168.1.1 /dev/tty00 19200

Then I can configure my desktop PC (which has a connection to the Internet by using an ethernet card):

$ pppd -detach crtscts lock noauth proxyarp 192.168.1.1:192.168.1.2 /dev/ttyUSB0 19200

Then I should be able to ping my desktop PC from the Amiga by running:

$ ping 192.168.1.1

With some additional steps, I can connect my Amiga 4000 to the Internet by using my desktop PC as a gateway.

First, I need to enable IP forwarding on my desktop PC:

echo 1 > /proc/sys/net/ipv4/ip_forward

Then, on my desktop PC, I can enable network address translation (NAT) as follows:

INTERNAL_INTERFACE_ID="ppp0"
EXTERNAL_INTERFACE_ID="enp6s0"

iptables -t nat -A POSTROUTING -o $EXTERNAL_INTERFACE_ID -j MASQUERADE
iptables -A FORWARD -i $EXTERNAL_INTERFACE_ID -o $INTERNAL_INTERFACE_ID -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i $INTERNAL_INTERFACE_ID -o $EXTERNAL_INTERFACE_ID -j ACCEPT

In the above example: ppp0 refers to the PPP link interface and enp6s0 to the ethernet interface that is connected to the Internet.

In order to resolve domain names on the Amiga 4000, I need to copy the nameserver settings from the /etc/resolv.conf file of the desktop PC to the Amiga 4000.

After doing these configuration steps I can, for example, use w3m to visit my homepage:

Exchanging files

Similar to my Amiga/Linux set up, I also want to the option to exchange files to and from my NetBSD/Amiga setup, for example, to try out stuff that I have downloaded from the Internet. For my NetBSD setup I can use the same two kinds of approaches.

Exchanging files with the memory card on my Linux PC

In my previous blog post I have shown that it is possible to take the Compact Flash card from my Amiga's CF2IDE device and put it in the card reader of my PC. Linux on my PC does not recognize an Amiga RDB partition table, but I can use GNU Parted to determine the partitions' offsets and use a loopback device to mount it:

$ parted /dev/sdb
GNU Parted 3.6
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit B                                                           
(parted) print                                                            
Model: Generic- USB3.0 CRW-CF/MD (scsi)
Disk /dev/sdb: 32019111936B
Sector size (logical/physical): 512B/512B
Partition Table: amiga
Disk Flags: 

Number  Start         End           Size          File system  Name         Flags
 1      56899584B     1130766335B   1073866752B   asfs         DH0          boot
 2      1130766336B   1265338367B   134572032B                 netbsd-swap  boot
 3      1265338368B   3953166335B   2687827968B   sun-ufs      netbsd-root  boot
 4      3953166336B   6100899839B   2147733504B   asfs         DH1
 5      6100899840B   8248633343B   2147733504B   asfs         DH2
 6      8248633344B   16838664191B  8590030848B   asfs         DH3
 7      16838664192B  32019111935B  15180447744B  asfs         DH4

(parted) quit

In the above code fragment, I have invoked GNU parted to read the partition table of my Amiga drive (/dev/sdb), switched the units to bytes, and printed the partition table. The third partition entry (Number: 3) refers to the NetBSD root partition on my Amiga 4000.

Linux is capable of mounting UFS partitions. I can use the start offset and size fields in the above table to configure a loopback device (/dev/loop0) referring to my Amiga drive's NetBSD partition:

$ losetup --offset 1265338368 --sizelimit 2687827968 /dev/loop0 /dev/sdb

Then I can mount the partition (through the loopback device file) to a directory:

$ mount /dev/loop0 /mnt/amiga-netbsd-partition

And then I can read files from the Amiga's NetBSD partition by visiting /mnt/amiga-netbsd-partition. Unfortunately, my Linux kernel's UFS module does not seem to have write support enabled -- it appears that UFS write functionality is experimental and disabled by default. As a consequence, I can only read files from the partition.

When the work is done, I must unmount the partition and detach the loopback device:

$ umount /mnt/amiga-netbsd-partition
$ losetup -d /dev/loop0

I have also been investigating how I could exchange files with my PC if NetBSD were my desktop operating system. I learned that there is the vndconfig command to configure a pseudo disk device (which is mostly feature comparable to a Linux loopback device), but I could not find any facilities to configure the partition offets.

Exchanging files between the AmigaOS and NetBSD operating systems

Similar to my Amiga Workbench 3.1/Linux hard drive configuration, I also want the ability to exchange files between my AmigaOS and NetBSD installation on the same system.

NetBSD on Amiga natively supports mounting Amiga Fast Filesystem partitions. For example, if I have such a partition I can automatically mount it on startup by adding the following line to /etc/fstab:

/dev/wd0d /mnt/harddisk ados rw 0 0

As with my other Compact Flash card, I have picked Smart Filesystem (SFS) over Amiga's Fast File system for the same reasons -- I need a file system that is more suitable for large hard drives. Unfortunately, it seems that there are no facilities for NetBSD to read from or write to Smart Filesystem partitions.

However, I can do the opposite -- a Berkeley Fast File System handler for AmigaOS seems to exist that offers read and write support.

Installing the package is similar to the Ext2 handler described in the previous blog post:

• Unpacking the LhA file into a temp directory
• Copying the l/BFFSFileSystem file to L:
• Creating a DOSDriver mount entry to get the partition mounted.

Creating the DOSDriver mount entry was a bit tricky. I did the following steps:

• I used the doc/MountList.BFFS file from the LhA archive as a template by copying it to DEVS:DOSDrivers/BH0. In the destination file, I need to adjust a number of properties that we must look up first.
• I can use HDToolBox to determine the start and end cylinders of the NetBSD root partition.
• I can use SysInfo to determine the amount of surfaces and sectors per side, by opening the "DRIVES" function and selecting a random hard drive partition:
  
  
  
• Then I need to adjust the partition's DOSDriver file: DEVS:DOSDrivers/BH0:
  • I must remove all examples, except the BH0: example
  • The LowCyl and HighCyl need to match the properties that we have discovered with HDToolBox.
  • The Surfaces property must match the value from SysInfo
  • The BlocksPerTrack property must match the Sectors per side property in SysInfo.
• Finally I can make an icon for the BH0 DOS Driver, by copying the PC1.info file to BH0.info and removing the UNIT=1 tooltip (by using the icon information function).

The result is that I can access my NetBSD root partition from AmigaOS:

The screenshot above shows an Amiga Workbench 3.9 session in which I have opened the NetBSD root partition.

Running NetBSD/Amiga in FS-UAE

In my previous blog post, I have shown that it is also possible to install and run Linux in FS-UAE, an Amiga emulator. Using an emulator is convenient for experimentation and running software at greater speeds than the real machine.

To run a different operating system, we must edit the FS-UAE configuration in a text editor and change the configuration to use a hard file. Moreover, we need to enable IDE emulation for the hard-drive and CD-ROM devices:

hard_drive_0 = mydrive.hdf
hard_drive_0_controller = ide
hard_drive_0_type = rdb
cdrom_drive_0_controller = ide1

The above configuration properties specify the following:

• We want to use a hard file: mydrive.hdf
• The controller property specifies that we need IDE controller emulation -- this property is required because NetBSD does not work with Amiga drivers. Instead, it wants to control hardware itself.
• The type property specifies that the hard drive image contains an RDB partition table. This property is only required when the hard file is empty. Normally, the type of a hard file is auto detected.

The result is that I can run NetBSD conveniently in FS-UAE:

As with my Linux installation, I can also take the Compact Flash card in my Amiga's CF2IDE drive (containing the NetBSD installation), put it in the card reader of my PC and use it in FS-UAE.

I need to change the hard drive setting to use the device file that corresponds to my card reader:

hard_drive_0 = /dev/sdb

In the above code fragment: /dev/sdb corresponds to the device file representing the card reader. I also need to make sure that the device file is accessible by an unprivileged user by giving it public permissions, by running the following command (as a root user):

$ chmod 666 /dev/sdb

Conclusion

In this blog post I have shown how I have been using NetBSD on my Amiga 4000. I hope this information can be of use to anyone planning to run NetBSD on their Amigas.

Compared to running Linux on Amiga, there are pros and cons. I definitely consider the homepage and documentation of NetBSD's Amiga port to be of great quality -- all required artifacts can be easily obtained from a single location and the installation procedure is well documented.

Moreover, NetBSD 10.1 (released in December 2024) is considerably newer than Debian 3.1r8 (released in 2008) and provides more modern software packages. Furthermore, the X Window System in NetBSD supports the Amiga chipset for displaying graphics -- the Debian Linux version does not.

NetBSD also has a number of disadvantages. Foremost, I consider its biggest drawback its speed -- it is considerably slower than my Debian Linux installation. This is probably due to the fact that modern software is slower than older software, not because Linux is superior to NetBSD when it comes to speed. Most likely, if I can manage to make a modern Linux distribution work on my Amiga, then it would probably also be much slower than Debian 3.1r8.

Another drawback is that I have much fewer options to control the framebuffer in NetBSD. With Linux I can easily select the video output device, resolutions and color modes at boot time and run time. In NetBSD the options are much more limited.

Beyond these observable Amiga differences between NetBSD and Linux, there are many more differences between NetBSD and Debian Linux, but I will not cover them in this blog post -- this blog post is not a NetBSD vs Linux debate.

Finally, I want to say that I highly appreciate the efforts of the NetBSD developers to have improved their Amiga port. NetBSD is definitely usable and useful for the Amiga. It would be nice if some areas could be improved even further.