In addition, my Amiga 500 was extended with a KCS PowerPC board making it possible to emulate an XT PC and run PC applications. This PC emulator integrates with all kinds of Amiga peripherals, such as the keyboard, mouse, floppy drives, and hard drives.
Although I could easily read PC floppies from AmigaOS, it was not possible for me to mount partitions from my KCS PowerPC board emulated PC hard drive. The reason is that the hard drive data is modified for efficiency -- each pair of bytes is reversed for more efficient processing on the Motorola 68000 CPU, which is a big endian CPU. For example, the typical signature of a Master Boot Record (MBR) is stored in reversed order: rather than 0x55AA, it is stored as 0xAA55.
In my previous blog post, I have shown a custom developed AmigaOS device driver that unreverses bytes and translates offsets in such a way that the addressing of the PC hard drive starts at offset 0. By using this device driver, Amiga filesystem DOS drivers can work with a virtual PC hard-drive.
Although it is convenient to be able to manage the files of my PC partition from AmigaOS, not all my inconveniences have been solved. With my SCSI2SD replacement for a physical hard drive, I can also conveniently transfer data to my Amiga hard drive from my Linux PC using my PC's card reader.
Sometimes I also want to transfer files from my Linux PC to my emulated PC partition, such as custom developed software. Unfortunately, on Linux I ran into the same problem as AmigaOS -- Linux has a FAT filesystem driver, but it cannot be used because the partition's raw data is not in a format that the filesystem module understands.
In addition to AmigaOS, I have also worked on a solution that makes it possible to cope with this problem on Linux. In this blog post, I will explain the solution.
Looking for a solution
Linux is a UNIX-like operating system. One of the concepts that UNIX is built on is that everything is a file. In addition to regular files, access to many kinds of storage media are also provided through files, namely: block device files.
Mounting a file representing a storage medium (this can be a device file, but also a regular file containing a dump of a medium, such as a floppy or CD-ROM) to a directory, makes it possible to browse the contents of a storage medium from the file system.
Linux and other kinds of UNIX-like operating systems (such as FreeBSD, NetBSD and OpenBSD) support many kinds of file systems, including the FAT filesystem that is commonly used on MS-DOS and many versions of the Windows operating system.
Because for efficiency reasons pairs of bytes have been reversed on my PC emulated Amiga partition, the raw data coming from the storage medium is not in a format that the FAT filesystem module understands.
The question that I came up with that is: how to cope with that? How can we provide a file that actually does provide the raw data in the right format, in which the bytes have been unreversed?
I have been thinking about various kinds of solution directions. Some impractical directions I have been thinking about were the following:
- Working with hard-drive dumps. A simple but crude way to get access to the data is to make a dump (e.g with the dd command) of the emulate PC hard-drive partition and to run a program that unreverses the pair of bytes. I can mount the modified dump as a loopback device and then read the data.
The downside of this approach is that it requires me to store a copy of the entire drive (although 100 MiB for nowadays standards' is not much) and that modifying data requires me write back the entire dump back to the SD card, which is costly and inconvenient. - Using a named pipe (also known as a FIFO). I could write a program that unreverses the bytes, using the SD card's device file as an input and a named pipe (FIFO) as an output. A named pipe can be accessed as a file from the filesystem.
Unfortunately, random access is not possible with a named pipe -- it is only possible to traverse a collection data from the beginning to the end. As a result, I cannot mount a FIFO to a directory, because filesystem modules require random access to data. - Developing a custom kernel module. To be able to mount a device file file that provides the raw data in the right format I could develop a kernel module. Device files are a means to communicate with a kernel module.
Although I have some basic experience developing Linux kernel modules, I consider this approach to be quite impractical for my use case:
- Linux does not have a stable kernel API, so custom module development requires maintenance each time a new Linux version is released.
- Deploying a Linux kernel module is not very convenient -- you need special user privileges to do that.
- Code running in the kernel is privileged -- incorrectly implemented code could crash the entire system.
- It is very unlikely that a kernel module for such a niche use case gets accepted into the mainline kernel.
After some more thinking and talking to people, I learned that I am not the only one struggling with raw hard-drive data that is not in the right format. For example, QEMU is a modern machine emulator that also emulates hard-drives. Hard drive data is typically stored in a space efficient way -- the qcow format, in which data is only allocated when it is actually needed.
There are also facilities to get access to these virtual QEMU hard drives from the host system or a remote system in a network. QEMU provides a utility named: qemu-nbd to make this possible. With this tool, it is possible, for example, to mount the contents of a qcow2 hard-drive image on the host system.
After learning about this tool, I also learned more about NBD. NBD is an abbreviation for Network Block Device and offers the following features:
- It makes block driver access possible over the network. NBD is realized by three components: a server, a client and the network between them. A server exports data (in blocks of a fixed size) and is typically implemented as a userspace process rather than functionality that runs in the kernel.
- A client runs on a client machine and connects to an NBD server over a network using the NBD protocol. It makes a network block device accessible through an ndb device file, which is also a block device file.
- Networking also uses file concepts on UNIX-like operating systems -- as a result, I can also efficiently connect to a local NBD server (without using a network interface) by using a Unix domain socket.
Because NBD makes it possible to use a userspace process to export data and this data can be exported at runtime, it looks like the ideal ingredient in solving my problem.
As a result, I have decided to look into developing a custom network block device server for my use case.
Developing a nbdkit plugin
How hard is to implement your own NDB server? After some searching, I discovered that developing your own NDB server is actually pretty easy: nbdkit can be used for this.
ndbkit manages many low-level aspects for you. As a developer, you need to construct an instance of struct ndbkit_plugin that defines an NDB configuration and register it as a plugin.
Most of the members of this struct are callback functions that get invoked during the life-cycle of the server:
static struct nbdkit_plugin plugin = {
.name = "kcshdproxy-plugin",
.version = VERSION,
.longname = "KCS HD Proxy plugin",
.description = "NBD plugin for accessing KCS PowerPC board emulated hard drives",
.config = kcshdproxy_config,
.config_complete = kcshdproxy_config_complete,
.open = kcshdproxy_open,
.get_size = kcshdproxy_get_size,
.pread = kcshdproxy_pread,
.pwrite = kcshdproxy_pwrite,
.close = kcshdproxy_close
};
NBDKIT_REGISTER_PLUGIN(plugin);
The above plugin configuration defines some metadata (e.g. name, version, long name and description) and a collection of callbacks that have the following purpose:
- The kcshdproxy_config callback is invoked for each command-line configuration parameter that is provided. This callback function is used to accept the targetFile parameter that specifies the device file representing my hard-drive and the offset defining the offset of the Amiga partition that stores the emulated PC hard drive data.
- The kcshdproxy_config_complete callback is invoked when all configuration parameters have been processed. This function checks whether the configuration properties are valid.
- The kcshdproxy_open function opens a file handler to the target device file
- The kcshdproxy_get_size function returns the size of the file representing the hard-drive. This function was the most complicated aspect to implement and requires a bit of explanation.
- The kcshdproxy_pread callback fast forwards to a specified offset, then reads a block of data and finally unreverses each pair of bytes. Implementing this function is straight forward.
- The kcshdproxy_pwrite callback fast forwards to a specified offset, reverses the input block data, writes it, and finally unreverses it again. As with the AmigaOS device driver, unreversing is required because the driver does not re-read a previously written block.
- The kcshdproxy_close function closes the file handler for the target device.
The most tricky part was implementing the kcshdproxy_get_size function that determines the size of the block device. At first, I thought using the stat() function call would suffice -- but it does not. For hard-drive dumps, it (sort of) works because I use the offsets provided by GRUB to create the dump file -- in this usage scenario, the file size exactly matches the Amiga partition size.
It is also possible to use the NDB server with a full dump of the entire hard-drive. Unfortunately, if I do this then the file size no longer matches the partition size, but the size of the entire hard drive.
For device files it is not possible to use stat() -- it will always return 0. The reason why this happens is that size information comes from the inode data structure of a file system. For device files, sizes are not maintained.
Linux has a non-standardized function to determine the size of a block device, but using this function has the same limitation as ordinary files -- this value may not always reflect the partition size, but it could also be something else, such as the size of the entire hard drive.
I ended up using the partition entry information from the Master Boot Record (MBR) to determine the size of the virtual PC hard-drive. Each partition entry contains various kinds of attributes including the offset of the first absolute sector, and the number of sectors that the partition consists of.
I can determine the size of the virtual PC hard-drive by computing the provisional sizes for each partition (a classic MBR allows up to four partitions to be defined) with the following formula:
Provisional size = (Offset of first absolute sector + Number of sectors)
* Sector size
As explained in my previous blog post, the sector size is always 512.
By computing the provisional sizes for each partition and taking the maximum value of these provisional sizes, I have a pretty reliable indication of the true hard drive size.
Usage
Using the NDB server with a local connection is straight forward.
First, I must determine the offset of the Amiga partition that stores the emulated PC hard-drive data (/dev/sdd is a block device file referring to my card reader). I can use GNU Parted to retrieve this information by changing the offset to bytes and printing the partition table:
$ parted /dev/sdd unit B print Model: Generic- USB3.0 CRW-SD (scsi) Disk /dev/sdd: 15931539456B Sector size (logical/physical): 512B/512B Partition Table: amiga Disk Flags: Number Start End Size File system Name Flags 1 557056B 104726527B 104169472B affs1 DH0 boot 2 104726528B 209174527B 104448000B KCS 3 209174528B 628637695B 419463168B affs1 DH1 4 628637696B 1073725439B 445087744B affs1 DH2
In the above output, the second partition (with name: KCS) represents my emulated PC hard-drive. Its offset is: 104726528.
As explained in my previous blog post, in most of the cases, the master boot record (MBR) can be found at the beginning of the emulated PC partition, but sometimes there are exceptions. For example, on my Kickstart 1.3 drive, it is moved somewhat.
I have also created a Linux-equivalent of the searchmbr tool determine the exact offset of the MBR. Running the following command scans for the presence of the MBR by using the partition's offset as a start offset:
$ searchmbr /dev/sdd 104726528 MBR found at offset: 104726528
As can be seen in the output, the MBR's offset is identical to the Amiga partition offset confirming that the MBR is at the beginning of the partition.
I can start an NDB server using a UNIX domain socket: $HOME/kcshdproxy.socket for local connectivity, as follows:
$ nbdkit --unix $HOME/kcshdproxy.socket \ --foreground \ --verbose /usr/lib/kcshdproxy-plugin.so \ offset=104726528 targetFile=/dev/sdd
In the above command-line instruction, I have provided two configuration settings as parameters:
- offset specifies the offset of the emulated PC hard drive. In the example, the value corresponds to our previously discovered partition offset. If no offset is given, then it defaults to 0.
- targetFile specifies the file that refers to the hard disk image. In the example: /dev/sdd refers to my card reader.
To connect to the server and configure a network block device, I can run the following command:
$ nbd-client -unix $HOME/kcshdproxy.socket -block-size 512
After running the above command: /dev/nbd0 refers to the block device file giving me access to the block data provided by my NDB server. /dev/ndb0p1 refers to the first partition.
I can mount the first partition with the following command:
$ mount /dev/nbd0p1 /mnt/kcs-pc-partition
And inspect its contents as follows:
$ cd /mnt/kcs-pc-partition $ ls COMMAND.COM DRVSPACE.BIN GAMES AUTOEXEC.BAT CONFIG.SYS IO.SYS AUTOEXEC.OLD DOS MSDOS.SYS
As can be seen in the output above, the data is accessible from my Linux system. Isn't it awesome? :-)
Although it is not possible to run the KCS PowerPC board emulator program in any of the well known Amiga emulators, such as FS-UAE or WinUAE (they lack the hardware emulation to make this possible), I can use DOSBox (or improved versions of it, such as DOSBox-X) to run programs from my emulated PC partition, by starting DOSBox as follows:
$ dosbox -noautoexec -c 'mount C /mnt/kcs-pc-partition' -c 'C:'
The provided command parameters (-c parameters) automatically mount my KCS PC partition as the C: drive:
Conclusion
In this blog post I have shown a custom developed NDB server that makes it possible to mount a KCS PowerPC board emulated PC hard drive partition in Linux. By having access to this emulated partition from Linux, it becomes possible to conveniently exchange files with my Linux PC and manage my emulated PC configuration from DOSBox.
Availability
My ndbkit plugin can be obtained from my GitHub page and used under the terms and conditions of the MIT license.
