Atari DOS 4 Internal Layout and Technical Details

[pcrow]

Atari DOS 4 was the DOS that Atari was planning on releasing with the 1450 XLD, and was eventually published by Antic.  I found a few sources for technical information, but they were incomplete, inconsistent with each other, and in some case proved to be incorrect.  So I took what I could find, ran tests, and I think I've got the answers.

 

Usually when analyzing the disk structures for an Atari DOS, I start with the boot sector.  But that doesn't work for DOS 4.  It doesn't necessarily use boot sectors!  If you format a disk in DOS 4, it only writes the VTOC sector or sectors.

 

DOS 4 supports four formats:

•   SS/SD (90K: 720 128-byte sectors, also known as 810 mode)
•   SS/ED (130K: 1040 128-byte sectors, also known as 1050 mode)
•   SS/DD (180K: 720 256-byte sectors)
•   DS/DD (360K: 1440 256-byte sectors)

 

All allocation is done in clusters or blocks of multiple sectors.  For most formats, these are 768 bytes (6 SD sectors or 3 DD sectors), but so that the number clusters can fit in one byte, DS/DD disks use 1536 bytes (6 DD sectors).

 

One weird quirk is that clusters start with sector 1 being the beginning of cluster 8.  This sort-of makes sense, as the VTOC sector starts with 8 bytes of header information, so the first entry is at offset 8 from the start of the sector.

 

Two clusters are set aside for the directory and the VTOC.  The VTOC uses the last sector of the two clusters, except for SS/ED where it's the last two sectors.  They really wanted to keep the "optimization" of having the VTOC and directory in the middle of the disk, as the clusters reserved are different depending on the format:

• SS/SD: 66-67 (sectors 349-360)
• SS/ED: 92-93 (sectors 505-516)
• SS/DD: 126-127 (sectors 355-360)
• DS/DD: 66-67 (sectors 349-360)

 

The VTOC sector (or pair in 1050 mode) starts with an 8-byte structure:

•   Format: 'R' unless it's DS/DD, then it's 'C'
•   First Free Cluster: 00 if the disk is full, otherwise the number of the first free cluster
•   Unused: 00
•   Free Clusters: a count of the free clusters
•   Unused: 00 00 00 00

 

This is followed by entries for each cluster on the disk, starting with cluster 8.  The value in each byte is the next cluster in the file.  If it's the last cluster in the file, the number of full sectors in that cluster is saved.  This is conveniently always less than 8.  For free sectors, they are treated just like a file.  They are always kept in increasing order, with the entry for the last free sector being zero.

 

Note that on a freshly formatted disk, if it's single-density, the first free cluster will be 8.  If you write the DOS files to a freshly formatted single-density disk, QDOS.SYS will simply be written out sequentially starting at cluster 8, and it starts with the codes to have the OS load it, reporting it as having 33 boot sectors.  If it's a double-density disk, the first free cluster will be 9.  Cluster 8 is reserved on double-density disks because some drives (all?) will send 128 bytes for the first three sectors for boot compatibility.  If you write DOS files on a DD disk, a two-sector boot loader will be written to the first two sectors (or just the first sector if it's a true 256-byte sector from my testing in an emulator; the ATR format allows for either 256- or 128-byte sectors for the first three sectors on DD disk images).

 

Writing DOS files has a bug.  If you try to write DOS files to a single-density disk that already has files on it, it will hope that QDOS.SYS will go at the start, but it doesn't check that that's the case.  If any file already occupies any of the first six clusters, DOS won't boot.  Interestingly, if you have something that expects to do a raw boot, you can just save the boot sectors in a file, and use DOS 4 to write them as the first file after formatting a disk, and it will boot.

 

I haven't analyzed the two-sector boot code used on double-density disks.  It may simply boot starting at cluster 9, or it may do something smart.

 

Speaking of bugs, after formatting a disk, sometimes DOS 4 keeps the final write of the VTOC in a buffer, so the disk may be completely blank or in some cases missing the last eight entries in the free list chain in the VTOC.  Simply doing a directory listing is usually enough to get it to send the write to the disk.  This makes me suspicious that the same bug might show up in other situations where a final write might be left in memory.

 

The directory is pretty standard; very much like DOS 2.  You have 16 bytes per entry:

•   Status: open $01, locked $20, regular file $40, deleted, $80
•   Blocks: Number of clusters in the file
•   Bytes at end: The number of bytes in the last sector
•   Start: First cluster in the file
•   Reserved: 00
•   Name: 8.3 with spaces for padding

 

So a file size can be computed as:
  (blocks - 1) * CLUSTER_SIZE_IN_BYTES + last_vtoc_byte*sector_size + bytes_in_last_sector

 

With all the different values for cluster sizes and VTOC location being hard-coded, it seems it would require some hacking to get DOS 4 to work with anything other than the four disk formats it was designed for.

 

I believe the above is everything that is needed to write code that can read and write DOS 4 disk images.  I had fun figuring it all out.
 

[_The Doctor__]

That and if the bugs are real, maybe some fixes are in order.

[pcrow]

7 hours ago, _The Doctor__ said:

That and if the bugs are real, maybe some fixes are in order.

The bugs are consistently repeatable.  I am doing this in the atari800 emulator (in XE mode for what it's worth), not a real Atari, but I would be surprised if that's relevant.  The lack of validation on writing DOS files is more of a curiosity, as people usually do that immediately after formatting if they're doing it at all.  The writes that get held up are very concerning.

[Jfcatari]

I have been testing the fuse atr mount on linux and it works well, but I have some atr's it fails on.  These are normal dos 2.5 images.

[pcrow]

13 minutes ago, Jfcatari said:

I have been testing the fuse atr mount on linux and it works well, but I have some atr's it fails on.  These are normal dos 2.5 images.

Can I get a copy of a failing image?  Do they all fail in the same way?

[Jfcatari]

Yes I think unable to determine disk info

[pcrow]

1 hour ago, Jfcatari said:

Yes I think unable to determine disk info

I sent you a message.

[+kheller2]

The source code for both the Antic and Atari branded versions for DOS 4 was released.  

 

Edit:

Actually, only the source code for Atari DOS 4 (CORP) and QDOS FOUR (Barall) were released: 

 

I've not seen the sources for the Antic "Atari, INC" version.

 

[_The Doctor__]

I would test in a properly configure Altirra and real equipment, Atari800 has not always worked out for such tasks.

Edited September 8, 2023 by _The Doctor__

[kenames99]

On 9/8/2023 at 7:35 AM, Jfcatari said:

I have been testing the fuse atr mount on linux and it works well, but I have some atr's it fails on.  These are normal dos 2.5 images.

can you post some info on how to do this or a link? thanks.

[pcrow]

1 minute ago, kenames99 said:

can you post some info on how to do this or a link? thanks.

I just posted a new topic on my tool for this:

 

[pcrow]

Back to the original topic, I found in testing that the "bytes in last sector" field in the directory isn't exactly what I thought.  If it were the valid bytes in the last sector, then a value of zero would let you have an empty file (though it would consume one cluster).  Unfortunately, it is the byte index of the last valid byte in the last sector of the file, so a value of zero means one valid byte.  So each file is one byte longer than I had originally computed.  This also appears to mean that you can't have a zero-length file in DOS 4, though I need to do some testing to verify this.  (I suppose it could say zero blocks with a zero starting block and have an empty file with no reserved disk space; I'm guessing not, though.)

[drac030]

Perhaps this helps http://atariki.krap.pl/index.php/Format_AtariDOS_4

 

[pcrow]

8 minutes ago, drac030 said:

Perhaps this helps http://atariki.krap.pl/index.php/Format_AtariDOS_4

 

Yup, that has most of the information, but was missing a few bits that I've now filled in above.

[Jfcatari]

2 hours ago, kenames99 said:

can you post some info on how to do this or a link? thanks.

Here I mounted icet-2.76.atr on /cdr, did an ls, then umounted:

 

./atrfs -oro  --name=/net/backup2/mister/atari800/icet-2.76.atr /cdr
Warning: MyDOS total sectors reported 1027; should be 1040 - 15 = 1025
[neptune atrfs]$ ls -al /cdr
total 461
drwxr-xr-x   2 jceklosk jceklosk  1024 Sep  6  2021 .
dr-xr-xr-x. 30 root     root      4096 Aug  6 00:28 ..
-rw-r--r--   1 jceklosk jceklosk    88 Sep  6  2021 ATARI850.HND
-rw-r--r--   1 jceklosk jceklosk  1369 Sep  6  2021 AUTORUN.SYS
-r--r--r--   1 jceklosk jceklosk   305 Sep  6  2021 .bootinfo
-r--r--r--   1 jceklosk jceklosk   384 Sep  6  2021 .bootsectors
-rw-r--r--   1 jceklosk jceklosk  4464 Sep  6  2021 COL80.COM
-rw-r--r--   1 jceklosk jceklosk  4375 Sep  6  2021 DOS.SYS
-rw-r--r--   1 jceklosk jceklosk  6720 Sep  6  2021 DUP.SYS
-r--r--r--   1 jceklosk jceklosk   147 Sep  6  2021 .fsinfo
-rw-r--r--   1 jceklosk jceklosk   595 Sep  6  2021 HYPERE.ARR
-rw-r--r--   1 jceklosk jceklosk 16125 Sep  6  2021 ICET.COM
-rw-r--r--   1 jceklosk jceklosk  1620 Sep  6  2021 ICET.DAT
-rw-r--r--   1 jceklosk jceklosk 16125 Sep  6  2021 ICET.TXT
-rw-r--r--   1 jceklosk jceklosk  2518 Sep  6  2021 MPP.HND
-rw-r--r--   1 jceklosk jceklosk    94 Sep  6  2021 PRCONN.HND
-rw-r--r--   1 jceklosk jceklosk   220 Sep  6  2021 README.TXT
-rw-r--r--   1 jceklosk jceklosk  1369 Sep  6  2021 RS232.COM
-rw-r--r--   1 jceklosk jceklosk  1369 Sep  6  2021 RVERTER.HND
-rw-r--r--   1 jceklosk jceklosk 10091 Sep  6  2021 VT100.TXT
-rw-r--r--   1 jceklosk jceklosk  3236 Sep  6  2021 XM301.HND
[neptune atrfs]$ umount /cdr

 

[pcrow]

That looks normal, though this should really be on the ATRFS-specific topic.  The warning is probably a math bug on my part, and I know it was often giving wrong values; perhaps I should just turn it off until I fix it.

[+bf2k+]

As far as not finishing a write, I have seen the exact same behavior on the atari800win emulator using DOS 2.0 DD version.  This happens when I compile my utilities using the MMG compiler on a800win about 1 in every 5 tries... so often in fact, that I always drop to DOS after a compile and do a directory on the disk... it has become ingrained for me to do this as it must be that it finishes the write and closes the file.  

 

I have always attributed it to compiling on an emulator as I never saw this with real hardware back in the 80s.  I would not discount the emulator as a possible problem with your tests...

[drac030]

16 hours ago, pcrow said:

Yup, that has most of the information, but was missing a few bits that I've now filled in above.

Perhaps, I just posted that when I saw that you were re-inventing the hard way some bits which are already written there (like the "bytes in the last sector" being in fact an offset).

[pcrow]

36 minutes ago, bf2k+ said:

As far as not finishing a write, I have seen the exact same behavior on the atari800win emulator using DOS 2.0 DD version.  This happens when I compile my utilities using the MMG compiler on a800win about 1 in every 5 tries... so often in fact, that I always drop to DOS after a compile and do a directory on the disk... it has become ingrained for me to do this as it must be that it finishes the write and closes the file.  

 

I have always attributed it to compiling on an emulator as I never saw this with real hardware back in the 80s.  I would not discount the emulator as a possible problem with your tests...

That's good to know.  I wonder if I was using the image without exiting the emulator, and the emulator just hadn't synced its writes?  That would make sense

[pcrow]

On 9/11/2023 at 11:30 AM, pcrow said:

That's good to know.  I wonder if I was using the image without exiting the emulator, and the emulator just hadn't synced its writes?  That would make sense

I just tested again, and that was exactly the problem.  It appears that the atari800 emulator will cache writes, so don't try to look at images until you exit the emulator.  Everything looks consistent not.

[pcrow]

I'm comparing images that I generated with ones that DOS 4 created, and I found some inconsistencies that revealed some behaviors that I don't think have been documented.

 

DOS 4 keeps a sector chain of free sectors that makes finding a free sector trivial, and the chain uses the same logic as file chains.  Everything was fine with SS/SD images, but I was seeing some weird things on other sizes where the number of clusters was off by one.  After comparing different images and testing writes to fill them up, I finally noticed that cluster $80 isn't in the free list.  DOS 4 seems to occasionally put some random value there, by which I mean I haven't determined where it comes from or what causes it to write to it.  And all the conversions from cluster numbers to sector numbers take into account that there is no cluster number 128.  So cluster 129 follows cluster 127 directly with no sectors between them.

 

That was a fun one to find.

 

Now I'm creating images that are byte-for-byte identical to was DOS 4 creates except that it initializes the entry for cluster $80, and I'm leaving it blank.

 

Except for DS/DD:  DOS 4 just doesn't allocate the last cluster, so the last 6 sectors aren't used.

 

I should probably look at the source code and see what it says about cluster $80 and the last cluster on DS/DD disks.

[pcrow]

More testing: If I create a DS/DD image and add the last cluster to the free list that DOS 4 leaves off, it works just fine.  There must just be a math bug in the formatting code.

 

In fact, you can create images up to 1484 sectors in both SD and DD, and they work fine if you tell DOS 4 that they're SS/2D or DS/DD images that just happen to have more clusters (up through cluster $FF).  That would be the hard limit on the image sizes in DOS 4 unless there's a way to tell it to use more than 6 sectors per cluster, which would appear to require some hacking.

 

Overall, my biggest complaint about DOS 4 is that it can't figure out what kind of disk it's using without being told.  This would be a big pain switching between SS/SD and SS/ED on the same drive, for example.  They could have done the layout just a little differently to make this work, so that the VTOC was always sector 360, and on ED flip the order of the two sectors so the second VTOC sector was 359.  Trying to keep the main directory and VTOC in the physical center of the disk wasn't worth the pain of having to identify disks manually.

[pcrow]

I found out what DOS 4 is putting in byte 128 of the VTOC: It's a checksum for bytes 0--127.  Add them all together, plus all the carry bits, and that value is stored in byte 128.  There is no checksum on SS/SD disks.  I looked in the source, and it appears that the checksum is never validated, which agrees with my testing with images that didn't have it where it seemed to be ignored.

 

But now that I know that, when ATRFS creates DOS 4 images, they're byte-exact to what DOS 4 creates, except for DS/DD where I intentionally allocate one more cluster that it's ignoring.