In my previous two blog posts, I have shown that my Amiga 4000 has the ability to run multiple operating systems -- in addition to AmigaOS, it can also run Linux and NetBSD, albeit with some challenges.
Something that these three operating systems have in common is that their file system capabilities are very powerful -- they are flexible enough to support many kinds of file systems through custom modules.
For example, both Linux and NetBSD have an Amiga Fast Filesystem module making it possible for me to conveniently exchange data with my Amiga partition. The opposite scenario is also possible -- there are custom file system handlers available for AmigaOS allowing me to exchange data with my Linux Ext2 and NetBSD Berkey Fast Filesystem partitions.
The ability to exchange files with different operating systems is not a new experience to me. When my Amiga 500 was still mainstream, I was already accustomed to working with PC floppy disks. Amiga Workbench 2.0 bundles a file system driver: CrossDOS that makes it possible to exchange data with PC floppy disks.
Although exchanging data with PC floppy disks is possible, there was something that I have been puzzled by for a long time. My Amiga 500 contains a KCS PowerPC board making it possible to emulate an XT-based PC. The user experience is shown in the picture on the top right. The KCS PowerPC board integrates with all kinds of Amiga peripherals, such as its keyboard, mouse, floppy drive, and hard drive (attached to an expansion board).
Unfortunately, I have never been able to exchange files with my emulated PC hard-drive from AmigaOS. Already in the mid 90s, I knew that this should be possible somehow. I have dedicated a great amount of effort in configuring a mount entry for my hard drive, but despite all my efforts I could not make it work. As a result, I had to rely on floppy disks or a null modem cable to exchange data, making data exchange very slow and inconvenient.
Recently, I have decided to revisit this very old problem. It turns out that the problem is simple, but the solution is a bit more complicated than I thought.
In this blog post, I will explain the problem, describe my solution and demonstrate how it works.
The KCS PowerPC board
As I have already explained in a previous blog post, the KCS PowerPC board is an extension card for the Amiga. In an Amiga 500, it can be installed as a trapdoor expansion.
It is not a full emulator, but it offers a number of additional hardware features making it possible to run PC software:
- A 10 MHz NEC V30 CPU that is pin and instruction-compatible with an Intel 8086/8088 CPU. Moreover, it implements some 80186 instructions, some of its own instructions, and is between 10-30% faster.
- 1 MiB of RAM that can be used by the NEC V30 CPU for conventional and upper memory. In addition, the board's memory can also be used by the Amiga as additional chip RAM, fast RAM and as a RAM disk.
- A clock (powered by a battery) so that you do not have reconfigure the date and time on startup. This PC clock can also be used in Amiga mode.
The KCS PowerPC board integrates with all kinds of Amiga peripherals, such as the keyboard, mouse, RAM, joysticks, floppy drives and hard drives (provided by an expansion board).
It can also emulate various kinds of displays controllers, such as CGA, EGA and VGA, various kinds of graphics modes, and Soundblaster and Adlib audio.
Video and audio is completely emulated by using Amiga's own chips and software. Because of the substantial differences between PC graphics controllers and the Amiga chips, graphics emulation is often quite slow.
Although graphics are quite costly to emulate, text mode applications generally work quite well and sometimes even better than a real XT PC. Fortunately, many PC applications in the 80s and early 90s were text based, so this was, aside for games, not a big issue.
Configuring DOS drivers in AmigaOS
AmigaOS has the ability to configure custom DOS drivers. Already in the 90s, I discovered how this process works.
In Amiga Workbench 2.0 or newer, DOS driver configurations can be found in the DEVS:DOSDrivers folder. For example, if I open this folder in the Workbench then I can see configurations for a variety of devices:
(As a sidenote: in Amiga Workbench 1.3 and older, we can configure DOS drivers in a centralized configuration file: DEVS:MountList).
In the above screenshot, the PC0 and PC1 configurations refer to devices that can be used to exchange data with PC floppy disks. When a PC floppy disk is inserted into the primary or secondary disk drive, the PC0: or PC1: devices provide access to its contents (rather than the standard DF0: or DF1:).
I eventually figured out that these DOS driver entries are textual configuration files -- I could, for example, use the PC0: configuration as a template for creating a configuration to mount my hard drive partition (PCDH0:). If I open this configuration file in a text editor, then I see the following settings:
FileSystem = L:CrossDOSFileSystem Device = mfm.device Unit = 0 Flags = 1 Surfaces = 2 BlocksPerTrack = 9 Reserved = 1 Interleave = 0 LowCyl = 0 HighCyl = 79 Buffers = 5 BufMemType = 0 StackSize = 600 Priority = 5 GlobVec = -1 DosType = 0x4D534400
With my very limited knowledge in the 90s, I was already able to determine that if I want to open my hard drive partition, some settings need to be changed:
- CrossDOSFileSystem is a filesystem driver that understands MS-DOS FAT file systems and makes them accessible through AmigaDOS function calls. This setting should remain unchanged.
- The device driver: mfm.device is used to carry out I/O operations on a floppy drive. Many years later, I learned that MFM means Modified frequency modulation, a common method for encoding data on floppy disks.
To be able to access my SCSI hard drive, I need to replace this device driver with a device driver can carry out I/O operations for my hard drive: evolution.scsi. This device file is available from the ROM thanks to the autoconfig facility of my expansion board: the MacroSystem Evolution. - The geometry of the hard drive is different than a floppy drive -- for example, it offers much more storage and the PC partition is typically somewhere in the middle of the hard drive, not at the beginning. Aside from the fact that I knew that the LowCyl and HighCyl settings can be used to specify the boundaries of a partition, I knew that I had to change other configuration properties as well. In the 90s, I had no idea what most of them were supposed to mean, but I was able to determine them by using a utility program: SysInfo.
Fast forwarding many years later, I learned that it is a common habit to measure offsets and sizes of classic storage media in cylinders, heads and sectors. The following picture (taken from Wikipedia) provides a good overview of the concepts:
By requesting the properties of my KCS: partition (the Amiga partition storing the data of my emulated PC drive) in SysInfo, I am able to derive most of the properties that I need to adjust my configuration:
Some of the above properties can be translated to DOS driver configuration properties as follows:
- Unit number translates to: Unit
- Device name translates to: Device
- Surfaces translates to: Surfaces
- Sectors per side translates to: BlocksPerTrack. I must admit that I find the term: "sectors per side" a bit confusing so I am not quite sure how it relates to concepts in the picture shown earlier, Anyway, I discovered by experimentation that this metric is equal to blocks per track setting.
- Reserved blocks translates to Reserved.
- Lowest cylinder translates to LowCyl.
- Highest cylinder translates to HighCyl.
- Number of buffers translates to Buffers.
Some DOS driver configurations also need to know the Mask and MaxTransfer properties of a hard-drive partition. I can discover these settings by using HDToolBox and selecting the following option: Advanced options -> Change File System for Partition:
Finally, to be able to know how to address the offsets in sizes in bytes, I need to know the block size. On the Amiga, 512 is a common value.
If I take the discovered properties into account and make the required changes, I will get the following DOS driver configuration for my hard drive partition (PCDH0:):
FileSystem = L:CrossDOSFileSystem Device = evolution.device Unit = 0 Flags = 1 Surfaces = 1 BlocksPerTrack = 544 Reserved = 2 Interleave = 0 LowCyl = 376 HighCyl = 750 Buffers = 130 BufMemType = 0 StackSize = 600 Priority = 5 GlobVec = -1 DosType = 0x4D534400 Mask = 0xffffff MaxTransfer = 0xfffffe BlockSize = 512
Although the configuration looks complete, it does not work -- if I try to mount the PCDH0: device and list its contents, then I will see the following error message:
This is as far as I could get in the 1990s. Eventually, I gave up trying because I had no additional resources to investigate this problem.
A deeper dive into the issue
Fast forwarding many years later, there is much I have learned.
Foremost, I learned a couple of things about PC hard drives -- I now know that the first block of every PC hard-drive (consisting of 512 bytes) contains a Master Boot Record (MBR).
A master boot record contains various kinds of data, such as bootstrap code (a prerequisite to boot an operating system from a hard drive partition), a partition table and a fixed signature (the last two bytes) that are supposed to always contain the values: 0x55, 0xAA.
With my SCSI2SD replacement for my physical hard drive, I can easily check the raw contents of the partitions by putting the SD card in the card reader of my Linux PC and inspecting it with a hex editor.
In addition, I found a nice hex editor for the Amiga: AZap, that can also be used to inspect the raw data stored on an AmigaDOS drive. For example, if I open a CLI and run the following command-line instruction:
AZap KCS:
Then I can inspect the raw data of my emulated PC drive (the KCS: partition):
The above screenshot shows the first block (of 512 bytes) of the KCS partition. This block corresponds to the hard drive's MBR.
If I look at its contents, I have noticed something funny. For example, the last two bytes (the typical signature) seems to be reversed -- instead of 55AA, it is AA55.
Furthermore, if I look at the bootstrap code, I see a number of somewhat human-readable strings. As an example, take the string:
"b seutirgnsssyetme"
It should probably not take long to observe what is going on -- the order of each pair of bytes is reversed. If I unreverse them, we will see the following string:
" besturingssysteem"
the word in the middle: besturingssysteem translates to: operating system (I am using a Dutch version of MS-DOS).
So why is the order of each pair of bytes reversed? I suspect that this has something to do with performance -- as I have already explained, the KCS PowerPC board integrates with a variety of Amiga peripherals, including hard-drives. To use Amiga hard drives the emulation software uses AmigaOS device drivers.
Amiga software uses the Motorola 68000 CPU (or newer variants) for executing instructions. This CPU is a big-endian CPU, while the NEC V30 CPU (pin-compatible with an Intel 8086/8088) is a little endian CPU. The order of bytes is reversed in little endian numbers. Because PCs use little-endian CPUs, data formats used on the PC, such as the MBR and the FAT file system typically store numbers in little endian format.
To let run software properly on the Motorola 68000 CPU, it needs to reverse the bytes that numbers consists of. Most likely, the emulation performs better if the byte order does not have to be reversed at runtime.
After this observation I realized that a big ingredient in solving my puzzle is to make sure that each pair of bytes gets unreversed at runtime so that a file system handler knows how to interpret the hard-drive's raw data.
Idea: developing a proxy device driver
As I have already explained, a file system handler typically uses a device driver to perform low-level I/O operations. To cope with the byte reversing problem, I came with the idea to develop a proxy driver providing the following features:
- It relays I/O requests from the proxy driver to the actual hard-disk driver. For requests that work with hard disk data, the driver intercepts the call and reverses the bytes so that the client gets it in a format that it understands.
- Some filesystem handlers, such as fat95, know how to auto detect partitions by inspecting a hard-drive's MBR. Unfortunately, auto detection only works if the MBR is on the first block. An emulated PC hard drive is typically never the first partition on an Amiga hard drive. The proxy driver can translate offsets in such a way that the caller can refer to the MBR at offset 0.
In a nutshell, the proxy device driver offers us a virtual PC hard drive that is accessible from AmigaOS.
A high level overview of some of the AmigaOS components
Implementing an Amiga device driver for the first time requires some learning about the concepts of the operating system. As a result, it is good to have a basic understanding of some of the components of which the operating system consists:
- The kernel is named: Exec. It is (somewhat) a microkernel that is responsible for task management, memory allocation, interrupt handling, shared library and device management, and inter-process communication through message passing.
Compared to more modern/common microkernel operating systems, it does message passing through pointers rather than serializing data. As a result, it is very fast and efficient, but not as secure as true microkernels. - One of the responsibilities of the kernel is device management -- this is done through device drivers that can be loaded from disk when they are needed, or by device drivers that are already in ROM or RAM. Device drivers use .device as a filename extension.
From the file system, they are typically loaded from the DEVS: assignment. By default, this assignment is an alias for the Devs/ directory residing on the system partition.
A driver is a module typically representing a shared resource, such as a physical device (e.g. a floppy drive). The functionality of a device driver can be used by client applications by sending I/O request objects using the kernel's message passing system.
I/O requests are basically function calls that specify a command number and some data payload. A device driver may also modify the data payload to send an answer, such a block of data that was read. - AmigaDOS is a sub system providing file system abstractions, a process management infrastructure and a command-line interface. Due to time pressure, it was not developed from scratch but taken from another operating system called: TRIPOS under a license from MetaComCo. AmigaDOS was written in BCPL and interacting with it from the C programming language was not always very efficient. In AmigaOS 2.0, the AmigaDOS sub system was rewritten in C.
- AmigaDOS also has its own drivers (they are often called DOS drivers rather than Exec drivers). DOS drivers can integrate with the file system operations that AmigaDOS provides making it possible work with many kinds of file systems.
Similar to UNIX-like systems (e.g. Linux), also non-storage devices can be accessed through a sub set of file system operations. DOS drivers providing functionality for non-storage devices are typically called handlers. For example, it is possible to redirect output to the serial port (e.g. using SER: as a file output path) or opening a console window (by using CON:) through custom handlers.
DOS drivers are typically loaded from the L: assignment, that by default, refers to the L/ folder on the system partition. - AmigaOS consists of more parts, such as Intuition: a windowing system and the Workbench: a graphical desktop environment. For this blog post, their details are irrelevant.
Developing a custom driver
I have plenty of C development experience, but that experience came many years after abandoning the Amiga as a mainstream platform. I gained most of my C development experience while using Linux. Back in the 90s, I had neither a C compiler at my disposal nor any C development resources.
In 2012 I learned that a distribution of the GNU toolchain and many common Linux tools exists for AmigaOS: Geek Gadgets. I have used Geek Gadgets to port the tools of my IFF file format experiments to AmigaOS. I have also used it to build a native viewer application for AmigaOS.
After some searching, I discovered a simple AmigaOS driver development example using GCC. Unfortunately, I learned that compiling this example driver with the GCC version included with the Geek Gadgets distribution does not work -- these tutorials rely on compiler extensions (for example, to implement the right calling convention and ordering of functions) that only seem to be supported by much newer versions of GCC. These tutorials recommend people to use amiga-gcc -- a cross compiling toolchain that needs to run on modern computers.
Although I can set up such a cross compilation toolchain, it is way too much complexity for my use case -- the main reason why I previously used GCC (in Geek Gadgets) is to port some of my Linux applications to AmigaOS.
Then I ended up searching for conventional resources that were commonly used when the Amiga was still a mainstream platform. The Amiga Development Reference explains quite well how to use device drivers from client applications.
Unfortunately, it does not provide much information that explains how to develop a device driver yourself -- the only information that it includes is an example driver implemented in assembly. I am not scared of a technical challenge, but for a simple driver that just needs to relay I/O requests and reverse data, assembly is overkill IMO. C is a more useful language to do that IMO.
I ended up using the SAS C Compiler (originally it was called the Lattice C compiler) version 6.58. I have managed to obtain a copy several years ago. The compiler also includes a very simple device driver example implemented in C (it resides in the example/example_device sub folder).
A device driver has a very simple structure:
int __saveds __asm __UserDevInit(register __d0 long unit,
register __a0 struct IORequest *ior,
register __a6 struct MyLibrary *libbase)
{
/* .... */
}
void __saveds __asm __UserDevCleanup(register __a0 struct IORequest *ior,
register __a6 struct MyLibrary *libbase)
{
/* .... */
}
void __saveds __asm DevBeginIO(register __a1 struct IORequest *ior)
{
/* .... */
}
void __saveds __asm DevAbortIO(register __a1 struct IORequest *ior)
{
/* .... */
}
Only four functions need to be implemented:
- The UserDevInit function is called by the kernel when the device driver is loaded. In my proxy device driver, it opens a connection the actual hard disk device driver. This function returns 0 if the initialization succeeds and 1 if it fails.
- The UserDevCleanup function is called by the kernel when the device driver is unloaded. It closes the connection to the actual device driver and cleans up all related resources.
- The DevBeginIO function is invoked when a device driver receives an I/O request from a client application. This function checks the command type and reverses the data payload, if needed.
- The DevAbortIO function is invoked when a device driver receives an abort request from a client application. This function does nothing in my proxy driver.
In addition to implementing these functions, these functions also require the compiler to follow a certain calling convention. Normally, a compiler has some freedom to choose how function parameters are propagated, but for an Exec device driver, CPU registers need to be used in a very specific way.
The implementation process
As I have explained earlier, device drivers accept I/O requests. I/O requests are function calls consisting of a command number and some data payload. Device driver commands are somewhat standardized -- there are commands that all devices accept, but also device-specific commands. The Amiga development reference provides an overview of device drivers that are bundled with AmigaOS and their commands.
Custom drivers typically follow the same kinds of conventions. For example, the driver for my SCSI controller that comes with my expansion board (evolution.device) accepts the same commands that Commodore's SCSI driver does (scsi.device). This driver is bundled with Kickstart 2.0 and newer.
To implement my proxy driver, I have implemented all the SCSI driver commands described in the manual. For the CMD_READ, CMD_WRITE and TD_FORMAT commands I am using a routine that reverses the order of the bytes in the data field.
Although the implementation process looked simple, I ran into a number of issues. In my first attempt to read my PC partition, the system crashed. To diagnose the problem, I have augmented the driver code with some conditional debugging print statements, such as:
#ifdef DEBUG
KPrintF("Target device: %s opened successfully at unit: %ld\n", config.device, unit);
#endif
The KPrintF function call sends its output over a serial port. In FS-UAE, I can redirect the output that is sent over a serial port to a TCP socket, by adding the following setting to the emulator's configuration:
serial_port = tcp://127.0.0.1:5000/wait
Then by using a telnet to connect to the TCP socket (listening on port 5000) I can inspect the output of the driver:
$ telnet localhost 5000
I can also connect my null modem cable to my real Amiga 500 and inspect the output in minicom. I need to use a baud rate setting of 9600 bits per second.
With my debugging information, I discovered a number of problems that I had to correct:
- It seems that the CrossDOS and fat95 DOS drivers do not only send SCSI driver commands. They also seem to be sending a number of trackdisk-specific device commands. I had to relay these commands through my proxy driver as well.
- The fat95 driver also works with New Style Devices (NSD). An important feature addition of NSD devices is that they are not restricted by the 32-bit integer limit. As a result, they can handle devices providing more than 4 GiB of storage capacity. They seem to use their own commands -- one of them checks whether a device is a new style device. I was not properly relaying a "command not implemented" error for this case, causing a crash.
- There is a trackdisk command: TD_ADDCHANGEINT to configure a software interrupt that activates on a disk change. It seems that this command never sends an answer message. My driver ran into an infinite loop because it was still expecting an answer. I also learned that in UAE, the uaehf.device driver seems to work with this command, but my real hard drive's SCSI driver (evolution.device) seems to crash when this command is called. I ended up ignoring the command -- it seems to do no harm.
- For write operations, I need to reverse bytes before writing a block, but I also need to unreverse them again after the write operation has completed. It turns out that file system drivers do not re-read previously written blocks.
Although I prefer to avoid debugging strategies, I also know that it is unavoidable for this particular development scenario -- specifications are never complete and third-party drivers may not strictly follow them. You really need to figure out how all of it works in practice.
I am very fortunate to have an emulator at my disposal -- it makes it very easy to have a safe test setup. I also do not have to be afraid to mess up my PC hard-drive -- I can simply make a dump of the entire contents of my hard drive and restore it once my system has been messed up.
In the 90s, I did not have such facilities. Driver development would probably have been much harder and more time consuming.
Usage
Earlier in this blog post, I have shown a non-working mount configuration for my PC hard-drive partition (PCDH0:). With my proxy driver and some additional settings, I can produce a working configuration.
First, I must configure the KCS HD proxy device in such a way that it becomes a virtual PC hard drive device for my emulated PC drive.
I can determine the offset (in bytes) of my KCS partition, by using the following formula:
Offset = Surfaces * Sectors per side * Lowest cylinder * Block size
Applying the formula with the properties discovered in SysInfo results in:
Offset = 1 * 544 * 376 * 512 = 104726528
After determining the partition offset, we must determine the exact offset of the MBR. Typically, it is at the beginning of the Amiga partition storing the PC hard drive data but there may be situations (e.g. when using Kickstart 1.3) that the MBR is moved somewhat.
I have developed a utility called: searchmbr can help you to determine the exact offset (what is does is searching for a 512 bytes block that ends with the MBR signature: 0x55AA):
> searchmbr evolution.device 0 104726528 Checking block at offset: 104726528 MBR found at offset: 104726528
In the output above, we see that the MBR is at the beginning of the Amiga partition.
With this information, I can create a configuration file: S:KCSHDProxy-Config for the proxy driver that has the following contents:
104726528 evolution.device
In the above configuration, the first line refers to the offset of the PC partition (in bytes) and the second line to the device driver of the hard drive.
It turns out that CrossDOS does not know how to interpret an MBR -- it needs to know the exact position of the first PC partition.
(As a sidenote: the fat95 DOS driver does have a feature to autodetect the position of a partition. If you set the LowCyl value to 0, then the fat95 driver will attempt to auto-detect a partition).
After configuring the proxy driver, the querypcparts tool can be used to retrieve the offset and size of the partitions on the PC hard drive:
$ querypcparts 0 Partition: 1, first sector offset: 34, number of sectors: 202878
In my example case, there is only one MS-DOS partition -- it resides at offset 34. Its size is 202878. Both of these properties are measured in sectors.
With the above information, I can make adjustments to our previous mount configuration (of the PCDH0: device). Rather than using the real SCSI driver for our hard drive, we use the KCS HD proxy driver (that reverses the bytes and translates the offsets):
Device = kcshdproxy.device
As explained earlier, partition offsets and sizes are typically specified in cylinders, heads and sectors. A PC hard drive typically does not use the same value for the amount of blocks per track that an Amiga partition does -- as a result, the boundaries of a PC partition are typically not aligned to cylinders.
To cope with this, we can change the following properties to 1, so that we can specify a PC partition's offsets in sectors:
BlocksPerTrack = 1 Surfaces = 1
Because the driver translates offsets in such a way that the beginning of the virtual PC hard drive starts at 0 and the offsets are measured in cylinders, the LowCyl property should correspond to the offset of the first sector reported by the querypcparts tool:
LowCyl = 34
We can compute the HighCyl boundary with the following formula:
HighCyl = First sector offset + Number of sectors - 1
Applying the formula with the values from the querymbr tool results in the following value:
HighCyl = 202911
After making the above modifications, I can successfully mount the PCDH0: device in AmigaOS and browse it contents in the Amiga Workbench:
Isn't it awesome? :-)
The driver also works in Kickstart 1.3, giving me the classic Amiga 500 experience:
Availability
The KCS HD proxy driver can be obtained from my GitHub page and Aminet. Currently, it has been tested with the CrossDOS and fat95 filesystem DOS drivers.
The KCS PowerPC board supports many kinds of SCSI controllers -- unfortunately, beyond uaehf.device and my expansion board's SCSI driver: evolution.device, I have not been able to test any other configurations. I believe it should work with any driver that is scsi.device compatible, but I do not have the means to test this. Use this driver at your own risk!
Future work
Although it is nice to have a solution to easily exchange files with my PC hard drive partition from AmigaOS, not all my problems have been solved -- I also sometimes download stuff from the Internet to my Linux PC that I may want to transfer to my emulated PC partition. With the SCSI2SD device as a replacement hard drive, transferring data to my Amiga "hard drive" is very easy -- I can insert the SD card into the card reader of my PC and get access to its data.
Unfortunately, on Linux, I have the same problem -- Linux has a FAT filesystem driver, but it does not understand the raw data because each pair of bytes is reversed. In the future, I may also have to develop a tool that implements the same kind of solution on Linux.
