ATRFS: Mount ATR files as local directories under Linux or Mac

[pcrow]

1 hour ago, woj said:

1. Copying the boot sectors code works only if cp is used with --no-preserve=all or an equivalent when using a file manager (Midnight Commander in my case). Otherwise, there are no error messages returned, but the copy is effect-less.

Strange.  I do everything from the command line, and I don't have preserve on by default.  I expect the error has to do with not implementing the chown/chmod/utimes calls for the special files by simply returning success.  Some of the other code does that already specifically to silence pointless errors about trying to set values that don't exist on these file systems.

[pcrow]

On 1/2/2024 at 1:22 PM, woj said:

1. Copying the boot sectors code works only if cp is used with --no-preserve=all or an equivalent when using a file manager (Midnight Commander in my case). Otherwise, there are no error messages returned, but the copy is effect-less.

I just pushed a change that might help.  It will now return success on chmod calls for the boot sectors file.  I'm guessing that's where it failed.

[woj]

6 hours ago, pcrow said:

I just pushed a change that might help.  It will now return success on chmod calls for the boot sectors file.  I'm guessing that's where it failed.

I will check it out. I also posted the log of how to get the disk full on an ED disk before it is actually full above, you might have missed it.

[pcrow]

On 1/2/2024 at 9:14 AM, woj said:

So, attached is the file that is (way) below the 126249 threshold (it's filled with 'A', the contents is irrelevant). Then I did this:

 

woj@impreza:~/temp/atrfs $ mkdir disk; atrfs --create --name=tst.atr --sectors=1040 --fs=dos25 disk
woj@impreza:~/temp/atrfs $ cp tst.bin disk/
woj@impreza:~/temp/atrfs $ ls -la disk/
total 502
drwxr-xr-x 2 woj woj   1024 Jan  2 15:08 .
drwxrwxr-x 3 woj woj   4096 Jan  2 15:08 ..
-r--r--r-- 1 woj woj    305 Jan  2 15:08 .bootinfo
-r--r--r-- 1 woj woj    384 Jan  2 15:08 .bootsectors
-r--r--r-- 1 woj woj    302 Jan  2 15:08 .fsinfo
-rw-r--r-- 1 woj woj 116460 Jan  2 15:08 tst.bin
woj@impreza:~/temp/atrfs $ ls -la
total 260
drwxrwxr-x  3 woj woj   4096 Jan  2 15:08 .
drwxr-xr-x 31 woj woj   4096 Jan  2 15:04 ..
drwxr-xr-x  2 woj woj   1024 Jan  2 15:08 disk
-rw-r--r--  1 woj woj 133136 Jan  2 15:08 tst.atr
-rw-rw-r--  1 woj woj 124335 Jan  2 15:03 tst.bin

 

In this case the file is truncated silently, if I, however, create an image with dir2atr (also empty) and mount it, I get an explicit message saying there is no space left to copy the file (and it is copied partially).

tst.bin 121.42 kB · 0 downloads

Found, fixed, and pushed.  There was a bug in manipulating the extended bitmaps for DOS 2.5 where it was getting shifted from where it belonged.  I'm rather surprised I didn't notice this when I was testing originally.  Thanks for finding it.

[pcrow]

I'm working on two more options:

 

--upcase

This will make file accesses case insensitive, and will use upper case for all new file creations.  Files with mixed or lower-case filenames will still present in directory listings with the case as it is in the file system.

 

--lowcase

This is the same as --upcase, but all files are presented as lower case, regardless of how the filenames are stored in the disk image.

 

I've written the first, but not tested it.  (It compiles, so what could go wrong?)  I'll have the second shortly.

[woj]

So everything that I wanted and complained about now seems to work. I could copy boot sector data wihtout --no-preserve=all and could also copy the big file (along with other things) onto an ED disk, and also tested that my game is happy writing to this disk from the different DOSes. Thanks! 

[pcrow]

And I just pushed --upcase and --lowcase as described above.  I did some minimal testing, but it seems to work as I expected.

 

With both options, the file system is case insensitive and new files are stored in all caps.  With --lowcase, they're presented in lower case.

[pcrow]

The only feature request I haven't addressed is APT (Atari Partition Table) support.  I've looked at the specs, and it's fairly complicated, and it's not a format that I use.  I could do some work on it if someone wanted to work with me in providing examples and testing.  At least the initial pass would be to export each partition or disk image as a file, and then use a script to mount each of the included ATR files separately if desired.

 

I'm also open to supporting other Atari disk formats if anyone has some they would like supported.

[beqelessen]

5 hours ago, pcrow said:

The only feature request I haven't addressed is APT (Atari Partition Table) support.  I've looked at the specs, and it's fairly complicated, and it's not a format that I use.  I could do some work on it if someone wanted to work with me in providing examples and testing.  At least the initial pass would be to export each partition or disk image as a file, and then use a script to mount each of the included ATR files separately if desired.

 

I'm also open to supporting other Atari disk formats if anyone has some they would like supported.

Hi @pcrow, that was me! I've been following the conversation here since then. Thank you for the work you've done to expose ATR files via FUSE.

 

Based on my very limited understanding of the two formats, ATR is a representation of a floppy disk image and APT is a container for PBI hard disk images. Hopefully someone else can pitch in with a more accurate description of the latter. The APT spec is silent in that regard; it doesn't actually describe the problem to which the spec is the solution.

 

Initially I was thinking that it ought to be possible to expose the contents of an APT partition via FUSE, to allow reading and writing of the contents of the disks inside that partition from a FUSE-capable OS like Linux or macOS, but it seems like a very different task to exposing ATR files. I don't think APT partitions contain ATR files. We've got the APT spec: http://drac030.krap.pl/APT_spec.pdf which makes no reference to ATR files. The Atari 8-bit FAQ https://mcurrent.name/atari-8-bit/faq.txt only has two references to APT as features of the Firmware BIOS and SIDE Loader in INCOGNITO and SIDE3.

 

Here's a tour of @flashjazzcat's Atari FDISK utility for setting up an APT partition on a Compact Flash card, and then adding multiple virtual hard disks and allocating drive letters to them: https://atari8.co.uk/apt/toolkit/ (See the video on that page.) As far as I know, FDISK is written in 6502 assembly. I can't find any reference to FDISK on https://github.com/search?q=owner%3Aflashjazzcat FDISK&type=code though. Perhaps it's not available under any kind of free software license.

 

This all seems very daunting to me, with hobbyist-level coding abilities. What do you think?

 

All that said, I can provide you with sample APT partitions, and I have a background in software testing.

Edited January 6 by beqelessen

[phaeron]

ATR is a disk image format holding raw sectors, while APT is a partition table format. In theory you could have APT at the start of an ATR image, but that doesn't happen in practice because ATR is not used for hard disk images.

 

APT partitions do not contain ATR images, but they have logical sectors that map 1:1 to the same sectors you'd find in an equivalent ATR image. As such, the filesystem handling code can stay the same and just needs to run on top of a slightly different logical disk view. It's slightly more complicated than ATR, but that's expected given how simple ATR is. The main complexity is due to the different sector mapping modes, as there are about seven different ways that logical sectors are mapped to physical sectors, some of which interleave pairs of logical sectors together. This has to do with the various tricks that have been done in existing hard disk interfaces to reduce the 6502 overhead of emulating 128 byte floppy disk sectors on 256/512 byte hard drive sectors. Parsing the partition table is not too bad since some of the data is for mapping DOS drive mappings and can be ignored. Altirra's disk explorer supports viewing APT partitions as subfolders and the code to handle the partition table parsing and logical->physical sector mapping is about 550 lines total.

 

Additionally, though, atrfs would need support for raw hard disk image formats for APT support to actually be usable. BIN/IMG is trivial as it's just a headerless raw sector dump like XFD, but can be trivially mounted via loopback on many OSes. This would also likely be usable for mounting raw CF/SD devices directly, assuming automount is disabled. The VHD image format is considerably more complex, especially for writing to dynamic or differencing disks; Altirra's read/write implementation is ~900 lines.

 

APT can also contain a master boot record (MBR) that doubles both as protection for the Atari data from computer OSes and to house a separate FAT32 partition for transfer. There's less of a need to support FAT32 access given that any OS that could run atrfs would probably also support just mounting the whole disk image directly to access the FAT32 partition.

 

[woj]

4 hours ago, phaeron said:

Additionally, though, atrfs would need support for raw hard disk image formats for APT support to actually be usable. BIN/IMG is trivial as it's just a headerless raw sector dump like XFD, but can be trivially mounted via loopback on many OSes. This would also likely be usable for mounting raw CF/SD devices directly, assuming automount is disabled. The VHD image format is considerably more complex, especially for writing to dynamic or differencing disks; Altirra's read/write implementation is ~900 lines.

 

To me personally, the biggest value of such a development would be to be able to mount actual SD cards with APT data on them. Images are of lower priority (but raw images should be zero problem, see below). Also, VHD images are a pain to mount in Linux (not sure about MacOS), requires extra packages, long command lines, etc. (I wrote a tutorial somewhere here on that at some point, see here: https://forums.atariage.com/topic/347527-wine-altirra-working-with-vhd-files-on-linux/, was not a popular topic though). So, I'd say supporting whatever the host OS supports by default (real devices and raw images) would be a way to go, leaving all other formats to Altirra. Another way of saying this - it should be to support working with the real hardware

 

Just to motivate @pcrow  and thank him again for taking my suggestion up for supporting mounting / FUSE. I found ATRFS a huge improvement over dir2atr (which I had to hack for my purposes), in particular the ability to copy the files onto the images in a particular order, which is important for some cases. Altirra can do it too (though I have not seen the option to modify boot sectors directly), but only in interactive mode preventing any scripting for Makefile-s and such. It also seems that some code for APT is already available in Altirra, I am sure @phaeron will not complain if you use those bits  

 

[pcrow]

Yeah, I've looked through the specs on the APT format, and it pretty much makes sense, but at this point I need some real examples to do something with them.  If you're a little leary of posting them publicly, you can send them to me in a message.

[pcrow]

I just pushed two more minor features.

 

* I found some images that lacked the ATR header, so it now will assume it's a disk image if it's exactly 128*720 bytes long.

 

* If the file system type isn't found, there's now a table of known disk images.  The table is the md5sum of a defined range of sectors.  If it matches, then a directory listing will have a dummy file indicating what disk it is.  I put in the four sides of "Ultima IV" and "Alternate Reality: The City" to start with.  I don't intend to create a comprehensive database.

[beqelessen]

19 hours ago, pcrow said:

Yeah, I've looked through the specs on the APT format, and it pretty much makes sense, but at this point I need some real examples to do something with them.  If you're a little leary of posting them publicly, you can send them to me in a message.

Hi @pcrow -

 

Here's an image of a 1GB CF card I've ben using in INCOGNITO with my Atari 800 and which I've written to an SDHC card for use with SIDE3 in my Ultimate1MB-equipped 800 XL.

 

https://yakshimash.britomartis.bysh.me/nextcloud/index.php/s/3TmxnKLe97Yib4E

 

The image is partitioned pretty much as @flashjazzcat recommends in his tour of FDISK; there's an MBR on it. It has two roughly equally-sized partitions, one FAT16 and one APT partition. The FAT partition has a small number of folders with a handful of files in each. The APT partition has a number of (I think) 62MB hard disk partitions to which I've allocated the drive letters DJ: to DO: consecutively (keeping D1: to D8: available for FujiNet and DI: for the FAT partition, if I choose to load the SDX FAT driver.). Some (J: and K: in particular) have programs and data on them. The pic below is the output of SDX DF -A.

 

(The Nextcloud  folder in which the image is located is read-only for now, as this URL is on the public internet. I can give you an account on that Nextcloud instance for more fine-grained access control if it gets to the stage where we're exchanging images regularly.)

Edited January 7 by beqelessen

[pcrow]

16 minutes ago, beqelessen said:

Here's an image of a 1GB CF card I've ben using in INCOGNITO with my Atari 800 and which I've written to an SDHC card for use with SIDE3 in my Ultimate1MB-equipped 800 XL.

I've downloaded it.  Thanks.  I'll see what I can do.

[pcrow]

As I get into this, it looks like my initial strategy was wrong.

 

I was going to have a layer that parsed files with a MBR partition table, present each partition as a directory, and then handle each partition in those directories.  But as I look at examples, I see that the APT partition table uses absolute sector numbers allowing it to include entries for other partitions, so that a standard FAT file system in the MBR partition table can (and likely will) have an entry in the APT partition table.  So now I'm thinking to parse the MBR table, and if an APT is found, just parse that.  That would eliminate a layer of subdirectories.

 

Obviously I won't handle the FAT file systems, as it's not that difficult to mount them natively already.  I will export a raw file for just the partition, though, which could make such mounting simpler (and possibly done automatically by script.

[beqelessen]

Wow, nice one. That certainly sounds like a cleaner solution. Thank you.

[beqelessen]

51 minutes ago, pcrow said:

a standard FAT file system in the MBR partition table can (and likely will) have an entry in the APT partition table.

That makes sense. Perhaps that's how Sparta DOS X is able to use the PBI interface to find and mount a FAT partition via the FAT driver. Can anyone comment on that?

[flashjazzcat]

DOS doesn't concern itself with the PBI' or the partition table at all. The host device firmware takes take of presenting DOS partitions and external FAT partitions as block storage devices via the SIO on the appropriate DUNIT numbers.

Edited January 8 by flashjazzcat

[pcrow]

I've added *very* preliminary APT support.

 

If it detects that the file has a MBR partition table at the start, it will scan the primary MBR partitions looking for one of type 7f, and then look at the first sector of that partition for an APT partition table.  Or it will also look for an APT partition table at the start of the file (though I'm not sure how that would work, as the sector offsets in APT are absolute, not partition-relative).

 

Each partition is presented as a subdirectory.  Currently the only thing in that subdirectory is a single file, ".raw" that is the full contents of the partition with absolutely no processing.  The APT format allows for packing 256-byte or 128-byte sectors with padding into the real 512-byte sectors, and I currently don't do anything about that.  I also make no attempt to parse the partition as a file system (a recursive access like that is tricky).  Oh, and the partitions are currently read-only.

 

So, like I said, *very* preliminary.

 

I'm going to figure out how to get the file systems working next, likely as separate mount points.  Oh, and write support should be relatively simple.

[pcrow]

One nice feature is that it creates symlinks for the drive mappings to the partitions they map to.  And if the partitions have a metadata sector with a name, that name is used as part of the directory name (I could make it the full name).  That will get weird very quickly if you've used funny ATASCII characters in the names.

[beqelessen]

Wow, this is phenomenal, thank you!

 

I need to do some running around on Wednesday evening but I'll be free either on Friday evening or at the weekend to check out your work.

 

I'm also scraping up other Compact Flash cards and SD cards to assemble an image with no MBR and an APT partition only, again packed with disks but probably in a more interesting layout with different sector sizes and number of sectors per disk.

[pcrow]

I've thrown some stuff in for supporting chunks in a "floppy drawer" partition.  I don't see anything in the spec about naming the separate chunks, and I don't have any examples to work from yet.

 

I also don't correctly handle 256-byte or 128-byte sectors.

 

I'm still thinking about how to mount the individual file systems.  They'll probably be separate mount points using the ".raw" files that I'm already exposing.  I just added support for --secsize=512 for an existing file to use it without the ATR header, so that should work.  I was able to manually mount a SDX partition that way from an APT partition, but it should be automatic.

[beqelessen]

Thank you very much, @pcrow! I'll get to take a look at this at the weekend.

[drac030]

5 hours ago, pcrow said:

I've thrown some stuff in for supporting chunks in a "floppy drawer" partition.  I don't see anything in the spec about naming the separate chunks, and I don't have any examples to work from yet.

I do not think any such partitions exist. If I remember correctly, the APT specification was designed in the ancient times, when HDD interfaces were not able to run ATR images off the file system. Hence I guess the only partition types which can be found in the wild are $00 (DOS partition) and $03 (external DOS partition).