ATRFS: Mount ATR files as local directories under Linux or Mac

[pcrow]

7 hours ago, drac030 said:

I do not think any such partitions exist. If I remember correctly, the APT specification was designed in the ancient times, when HDD interfaces were not able to run ATR images off the file system. Hence I guess the only partition types which can be found in the wild are $00 (DOS partition) and $03 (external DOS partition).

That's interesting.  Reading the spec, the floppy drawer partitions don't have any way to name the floppies, which seems to be a huge shortcoming.  Also, that would be the partition type most likely to use the weird non-512-byte sectors with different packing methods (why not just choose one?).

 

So at this point, if I get it to mount SpartaDOS file systems in type 00 partitions, it sounds like I'll have 95% of the use cases, and I can look at others if people find them useful.

[drac030]

1 hour ago, pcrow said:

Reading the spec, the floppy drawer partitions don't have any way to name the floppies, which seems to be a huge shortcoming.

There are two bytes remaining for definition. It can be e.g. a relative number of a sector containing all the required information. Fortunately, it happened to be not really necessary.

 

1 hour ago, pcrow said:

Also, that would be the partition type most likely to use the weird non-512-byte sectors with different packing methods (why not just choose one?).

A specific type of hardware would surely choose one. The APT table information informs you, which packing method was chosen for the media being about to be mounted.

 

1 hour ago, pcrow said:

if I get it to mount SpartaDOS file systems in type 00 partitions, it sounds like I'll have 95% of the use cases

Probably, if the remaining 5% is MyDOS and friends.

[TheEditor]

Will this run on Ubuntu on WSL in Windows 10/11?

[pcrow]

1 hour ago, TheEditor said:

Will this run on Ubuntu on WSL in Windows 10/11?

A quick search says that fuse is not supported on WSL 1, but is in WSL 2, so I would think it would likely work.  I haven't heard of anyone testing it yet, so I would love to hear what you find.

[TheEditor]

50 minutes ago, pcrow said:

A quick search says that fuse is not supported on WSL 1, but is in WSL 2, so I would think it would likely work.  I haven't heard of anyone testing it yet, so I would love to hear what you find.

Are there instructions for installation somewhere on the github?  I didn't see them but may have missed them.

 

[pcrow]

1 hour ago, TheEditor said:

Are there instructions for installation somewhere on the github?  I didn't see them but may have missed them.

No, it's just type 'make' and it should produce a binary.

[beqelessen]

Hi @pcrow - I'm uploading another CF image to https://yakshimash.britomartis.bysh.me/nextcloud/index.php/s/3TmxnKLe97Yib4E - this time the image has no MBR and consists only of an ATR 'volume' with four disk partitions in it:

The first three are all 65,535 sectors, with sector sizes of 512, 256 and 128 bytes.

 

The fourth fills the rest of the disk with a disk that's apparently 7,552MB in size, but as you can see from the last screenshot, SpartaDOS X can only recognise 65,535 sectors. (See the output of `DIR` just before the output of `LSDSK`.)

 

The content of the disks is just random stuff I had on my FujiNet at the time - mostly the contents of the SpartaDOS X toolkit and an Action! disk or two. The data was mostly copied over with `COPY /R /S /V B:> L:>` for a fully recursive copy. I didn't attempt to format any of the disks with any DOS other than SDX.

[pcrow]

I just pushed a change where it should show file systems within APT partitions.

 

This only works with 512-byte sectors, and has had very minimal testing.  The only thing I know I don't support is moving files between partitions; you have to copy/remove them instead.

 

The changes touch a bunch of other code, but I made many of them using a script to modify the code, so I don't think I broke anything.  I also made a new middle layer to better support putting a file system inside an existing file system.

 

And I just downloaded the new image so I can start doing some testing on it and see what's broken.

[beqelessen]

Thanks very much! I didn't get to clone and build your project at the weekend, but I knocked this image together this evening.

 

Looking forward to testing your code.

[pcrow]

3 hours ago, beqelessen said:

The first three are all 65,535 sectors, with sector sizes of 512, 256 and 128 bytes.

That's very helpful  From the spec, there are four options for how to store smaller sectors:

• You can do one logical sector per 512-byte sector or pack them in
• You can put the data in interleaved by byte or by logical sector

In the examples you provided, it's doing byte interleaving with one logical sector per physical sector.  From the screen shots, I didn't see it giving you any options for the other formats, so I'm going to assume that what you provided covers most of what exists.

 

So the good news is that I think I have it so that the partitions will create a working ".raw" file that exports the contents of the partition, taking into account the sector mapping and data interleaving.  The bad news is that viewing the partitions as file systems won't work, as the file system code assumes it can view the data as a single linear block of bytes (I memory map the image file), so to get that to work, all the file system code would have to support the interleaving.  This can work if you run atrfs again to mount the .raw partition files as file systems separately.

[pcrow]

I think it's now fully working for the examples I have.  The performance will be poor with smaller sector sizes, as it makes copies in memory and does comparisons to see what's changed, but for simple interactive use, it should be good enough.

 

I'm only doing some *very* rudimentary testing, so there may be bugs.

 

I'm assuming that while the specs allow different sector packing methods, the method in the example I have is all that's used.  If anyone has information to the contrary, please provide an example image and I should be able to get it working.

[beqelessen]

This is amazing, @pcrow thank you. I was away this weekend, but I ought to be able to build and run your solution next weekend. (I'm at the computer all day four days a week and in the office the fifth day, so I tend to avoid retrocomputing activities during weekday evenings.)

[beqelessen]

On 1/13/2024 at 1:12 AM, TheEditor said:

Are there instructions for installation somewhere on the github?  I didn't see them but may have missed them.

On 1/13/2024 at 2:39 AM, pcrow said:

No, it's just type 'make' and it should produce a binary.

I'm running Xubuntu; it's "Ubuntu 22.04.3 LTS". I already had build-essential installed but I had to install a couple of extra development libraries to be able to get atrfs built:

# Install prerequisites
sudo apt install build-essential libfuse3-dev libssl-dev
# Checkout
git clone git@github.com:pcrow/atari_8bit_utils.git
# Build
cd atrfs
make

 

I'll outline my testing and issues found in the next post.

[beqelessen]

I've pulled the CF from my INCOGNITO board in my Atari 800, shoved it in a no-name (but reliable) CF USB reader:

And shoved that into my desktop machine:

 

[Mon Jan 29 18:34:52 2024] usb 1-4: new high-speed USB device number 42 using xhci_hcd
[Mon Jan 29 18:34:52 2024] usb 1-4: New USB device found, idVendor=14cd, idProduct=168a, bcdDevice= 0.01
[Mon Jan 29 18:34:52 2024] usb 1-4: New USB device strings: Mfr=1, Product=3, SerialNumber=2
[Mon Jan 29 18:34:52 2024] usb 1-4: Product: USB Mass Storage Device
[Mon Jan 29 18:34:52 2024] usb 1-4: Manufacturer: USB Device  
[Mon Jan 29 18:34:52 2024] usb 1-4: SerialNumber: 816820130806
[Mon Jan 29 18:34:52 2024] usb-storage 1-4:1.0: USB Mass Storage device detected
[Mon Jan 29 18:34:52 2024] scsi host4: usb-storage 1-4:1.0
[Mon Jan 29 18:34:52 2024] usbcore: registered new interface driver usb-storage
[Mon Jan 29 18:34:52 2024] usbcore: registered new interface driver uas
[Mon Jan 29 18:34:53 2024] scsi 4:0:0:0: Direct-Access     USB Mass  Storage Device  1.00 PQ: 0 ANSI: 0
[Mon Jan 29 18:34:53 2024] sd 4:0:0:0: Attached scsi generic sg0 type 0
[Mon Jan 29 18:34:53 2024] sd 4:0:0:0: [sda] 2003904 512-byte logical blocks: (1.03 GB/978 MiB)
[Mon Jan 29 18:34:53 2024] sd 4:0:0:0: [sda] Write Protect is off
[Mon Jan 29 18:34:53 2024] sd 4:0:0:0: [sda] Mode Sense: 03 00 00 00
[Mon Jan 29 18:34:53 2024] sd 4:0:0:0: [sda] No Caching mode page found
[Mon Jan 29 18:34:53 2024] sd 4:0:0:0: [sda] Assuming drive cache: write through
[Mon Jan 29 18:34:54 2024]  sda: sda1 sda2
[Mon Jan 29 18:34:54 2024] sd 4:0:0:0: [sda] Attached SCSI removable disk

 

The FAT partition /dev/sda1 automounts at /media/ben/5391-CD84. We can see from the dmesg output above that there's a sda2 partition as well - that's the APT partition.

 

Let's use atrfs to mount it:

 

$ mkdir ~/Desktop/APT
$ atrfs /dev/sda2 ~/Desktop/APT
Failed to open /dev/sda2; must have write permission

 

I've hit this check: https://github.com/pcrow/atari_8bit_utils/blob/140304b5bb97aef43c3940057cdcd189db886fca/atrfs/atrfs.c#L264.

 

The partition is sane:

 

$ sudo fdisk /dev/sda2 -l
Disk /dev/sda2: 489.44 MiB, 513212928 bytes, 1002369 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

 

Using sudo gives me a different failure:

 

$ sudo atrfs /dev/sda2 ~/Desktop/APT
[sudo] password for ben:
File is too small to be a valid ATR file: 0 bytes in /dev/sda2

 

https://github.com/pcrow/atari_8bit_utils/blob/140304b5bb97aef43c3940057cdcd189db886fca/atrfs/atrfs.c#L270

 

So it looks like I can only interact with images of CF cards for now.

 

Next post, I'll try to repro a FUSE idle timeout I saw when interacting with a CF image on disk. (One of the ones I shared upthread.)

 

Again, @pcrow, thank you for all your work on this excellent tool. You don't have to, but you do anyway, for precious little thanks. You'll never get rich and famous building fringe utilities to manage filesystems for long-obsolete computer hardware. Huge respect to you for this!

 

[beqelessen]

The use case above is the way I'm expecting to be using atrfs: do some work on the Atari, save it to a disk in the APT partition, pull the CF from INCOGNITO (or SD card in the SIDE3 on my 800XL), shove that in the CF card USB reader and use atrfs to mount the APT partition as a directory tree in order to fetch and further manipulate the files just created on the Atari (and potentially drop other files into one or other disks in the APT partition.)

 

OK, here's some usage of atrfs, a combination of typical command-line and graphical file management operations that I'd expect to be able to do:

 

$ atrfs Disk\ Image\ of\ 1GB\ CF\ INCOGNITO\ 2023-11-21.img 1GBIMG/
$ cd 1GBIMG/
$ ls -ltrh
total 0
drwxrwxr-x 1 ben ben 0 Jan 28 18:12 'partition 7 FIFTEEN'
drwxrwxr-x 1 ben ben 0 Jan 28 18:12 'partition 6 FOURTEEN'
drwxrwxr-x 1 ben ben 0 Jan 28 18:12 'partition 5 THIRTEEN'
drwxrwxr-x 1 ben ben 0 Jan 28 18:12 'partition 4 TWELVE'
drwxrwxr-x 1 ben ben 0 Jan 28 18:12 'partition 3 ELEVEN'
drwxrwxr-x 1 ben ben 0 Jan 28 18:12 'partition 2 TEN'
drwxrwxr-x 1 ben ben 0 Jan 28 18:12 'partition 1 FAT'
lrwxrwxrwx 1 ben ben 0 Jan 28 18:12 'DO FIFTEEN' -> 'partition 7 FIFTEEN'
lrwxrwxrwx 1 ben ben 0 Jan 28 18:12 'DN FOURTEEN' -> 'partition 6 FOURTEEN'
lrwxrwxrwx 1 ben ben 0 Jan 28 18:12 'DM THIRTEEN' -> 'partition 5 THIRTEEN'
lrwxrwxrwx 1 ben ben 0 Jan 28 18:12 'DL TWELVE' -> 'partition 4 TWELVE'
lrwxrwxrwx 1 ben ben 0 Jan 28 18:12 'DK ELEVEN' -> 'partition 3 ELEVEN'
lrwxrwxrwx 1 ben ben 0 Jan 28 18:12 'DJ TEN' -> 'partition 2 TEN'
lrwxrwxrwx 1 ben ben 0 Jan 28 18:12 'D9 FAT' -> 'partition 1 FAT'

 

$ thunar . &

 

In thunar (the Nautilus-derived XFCE file manager), double-click the folder "partition 2 TEN". Thunar pops up the following error dialog:

dmesg says:

 

[Mon Jan 29 19:20:30 2024] atrfs[586259]: segfault at 7f1a9c021000 ip 00007f1ae5962fb4 sp 00007f1aa7451100 error 6 in libc.so.6[7f1ae58fd000+195000]
[Mon Jan 29 19:20:30 2024] Code: 47 08 48 83 fb 08 0f 85 77 ff ff ff eb d3 0f 1f 84 00 00 00 00 00 31 c0 c3 0f 1f 44 00 00 31 c0 66 0f 1f 44 00 00 0f b6 14 01 <88> 14 07 48 83 c0 01 48 39 d8 75 f0 eb a9 66 66 2e 0f 1f 84 00 00

 

That's down in the guts of the standard library? ¯\_(ツ)_/¯

 

(This is not the same as the timeout I saw yesterday.)

Edited January 29 by beqelessen

[woj]

I recently had segfault with the older version of ATRFS, where I just tried to create a new ATR image with standard disk settings (90K SD disk) and Dos2.5 format, it just crashed.

[TheEditor]

On 1/29/2024 at 1:44 PM, beqelessen said:

I'm running Xubuntu; it's "Ubuntu 22.04.3 LTS". I already had build-essential installed but I had to install a couple of extra development libraries to be able to get atrfs built:

# Install prerequisites
sudo apt install build-essential libfuse3-dev libssl-dev
# Checkout
git clone git@github.com:pcrow/atari_8bit_utils.git
# Build
cd atrfs
make

 

I'll outline my testing and issues found in the next post.

When I run the "sudo apt install build-essential libfuse3-dev libssl-dev" command it returns

E: Unable to locate package libfuse3-dev.

 

I am running Minimal Ubuntu.  Do I need to run a more fleshed-out version?  How do I get around the error?

 

[beqelessen]

What's the output of

 

cat /etc/lsb*

?

You may be on an older version of Ubuntu.

 

Also, what's the output of

 

cat /etc/apt/sources.list

?

 

The minimal install may not have the full suite of package repositories that the full-fat version does; or it may use special alternative ones.

[TheEditor]

8 minutes ago, beqelessen said:

The minimal install may not have the full suite of package repositories that the full-fat version does; or it may use special alternative ones.

That was it.  I installed full Ubuntu and that error was resolved.  Now when I try to run the git command it returns the error in the attached image.

 

[beqelessen]

1 hour ago, TheEditor said:

That was it.  I installed full Ubuntu and that error was resolved.  Now when I try to run the git command it returns the error in the attached image.

 

Oh, that's probably because you haven't uploaded a public key to Github; you can generate one using ssh-keygen.

 

I'm going to be a pedant and recommend that you don't login as root to your Linux machine; it's too easy to mess things up accidentally. The last, kindest thing you should do as root is create an account for yourself, in this case a user named "atari", with the typical set of broad-but-not-too-broad permissions, with this one-liner:

 

(See the name at the end? Change that to whatever you want.) The command on the next line sets the password for that user.

 

useradd -m -U -G adm,cdrom,sudo,dip,plugdev,lpadmin,sambashare,libvirt atari
passwd atari

 

If you get any complaints about a group in that list not existing, chop it off and try again. (Remember the username stays on the end of the command, separated by a space!) The important groups are adm,cdrom,sudo,dip,plugdev,lpadmin.

 

Once the account's successfully created, either log out of your Linux machine and login again as the new user account, or switch from root to the new user with the command:

 

sudo -u atari bash --login -i

 

The useradd command also gave you sudo rights, so you can run individual commands with root privileges. You can have your cake and eat it! Prefix any command you want to run as root with sudo; for example:

 

sudo cat /etc/shadow

 

(Mandatory XKCD: https://xkcd.com/149/)

 

OK, now let's create a public key for use with Gitlab. ssh-keygen is interactive, so run it without any arguments. You'll first be prompted for a filename in which to save the generated key. Call it something like id_rsa_github_ubuntu-22., and save it into the default directory suggested. That'll give you a public and private key pair; the public one has a .pub extension. Never share the private one.

 

Paste the content of the public key in here: https://github.com/settings/ssh/new

 

(It's an "Authentication Key" - so stick with the default in the drop-down on that page.)

 

...and then try cloning the repo again:

 

git clone git@github.com:pcrow/atari_8bit_utils.git

 

Good luck!

[beqelessen]

You can of course also download a zip of the repo instead:

 

 

...but I still think you should create a user for yourself on your Linux system! 😜

 

 

[pcrow]

On 1/29/2024 at 2:11 PM, beqelessen said:

Let's use atrfs to mount it:

 

$ mkdir ~/Desktop/APT
$ atrfs /dev/sda2 ~/Desktop/APT
Failed to open /dev/sda2; must have write permission

 

I've hit this check: https://github.com/pcrow/atari_8bit_utils/blob/140304b5bb97aef43c3940057cdcd189db886fca/atrfs/atrfs.c#L264.

 

That error message is wrong (I'll change it).  The logic is to try opening for read/write, and then do read-only if that fails.  So it appears you don't have access to the file.  Since you're accessing a device file, it's likely that the permissions are like

brw-rw---- 1 root disk 259, 1 Feb  2 19:38 nvme0n1p1

With most Linux distributions, you can't access device files for storage as a regular user, but if you add your user to the "disk" group, then you get full raw access.  It should work in that case.

[pcrow]

On 1/29/2024 at 3:09 PM, woj said:

I recently had segfault with the older version of ATRFS, where I just tried to create a new ATR image with standard disk settings (90K SD disk) and Dos2.5 format, it just crashed.

If it's doing that consistently, run under gdb, and use the "bt" command to see where it crashed.  But do an update from git first to get the current version in case it's something I've already fixed.

[pcrow]

FYI: I just wrote a pair of blog posts about this project:

https://madcompiler.blogspot.com/2024/02/a-user-space-file-system-for-fun-and.html

https://madcompiler.blogspot.com/2024/02/lessons-from-ancient-file-systems.html

Comments on the blog posts would be appreciated.

[flashjazzcat]

'The APT documentation is poor'?

 

I think @drac030's documentation is usually of a high standard, and indeed I implemented my APT drivers based on his APT specification. Of course, I was able to ask questions about anything I didn't understand.

Edited February 5 by flashjazzcat