Extending the ATR file format

[+bf2k+]

I'm curious...

 

If I have one of those ATR's with sec 1,2,3 = 256b each, what would happen if I tried to write this ATR to a real floppy? Would a Happy 1050 or USD 1050 be able to deal with the 1st three secs = 256b?

[cyco130]

I'm curious...

 

If I have one of those ATR's with sec 1,2,3 = 256b each, what would happen if I tried to write this ATR to a real floppy? Would a Happy 1050 or USD 1050 be able to deal with the 1st three secs = 256b?

 

I don't think there is a way to write to or read from a disk 256 bytes in the first 3 sectors using (semi-)standard hardware on Atari.

 

Depending on what software you use it will either reject the ATR image as invalid, correct it silently (or with a prompt) or will interpret it in a wrong way and will write garbage to the diskette, the last being the most common case in my experience, even my own software is in this category (it will be in the first whith the next release). Even if it tries to send 256 bytes to the drive, the drive would probably lose synchronization and it will trigger an error on both sides.

 

But there maybe some other ways that I'm not currently aware of to make it possible.

[HiassofT]

I'm curious...

 

If I have one of those ATR's with sec 1,2,3 = 256b each, what would happen if I tried to write this ATR to a real floppy? Would a Happy 1050 or USD 1050 be able to deal with the 1st three secs = 256b?

It won't work with the standard SIO commands R/P/W. The drive expects the data part to be 128 bytes and the 129th byte to be the checksum. So the drives would just return a checksum error (as your bytes 129..256 are data and byte 257 is the checksum).

 

You could upload code to the Happy and implement new SIO commands for disk access and store 256 bytes in every sector. But then, you'd have to use these special commands in your Atari code. If you replace the original R/P/W commands the Atari won't be able to boot the disk (the Atari OS also expects the first three sectors to contain 128 bytes and the 129th byte received from SIO to be the checksum).

 

so long,

 

Hias

[dmlloyd]

If someone does design a new disk format, it should include things like:

 

1. Minimum/maximum disk (or even per-track) rotational speed (in RPM) or duration (in milliseconds?) of one track rotation (for formats/titles which use copy protection in a way which is time-dependent), with an option for "don't care" to allow "maximum speed" when it doesn't matter

2. Sector layout, per track (to properly emulate disks with duplicate or missing sectors)

3. Sector length (not just in bytes but in milliseconds - an MFM sector is 2x more dense than an FM sector, for example) on a per-sector basis

4. Time offset of each sector from the "start" of the track (so that sector skew can be emulated)

5. Maybe minimum track seek time (to truly emulate the speed at which a sector can be accessed)?

 

In other words, if you emulate all the data on the disk itself, not just the formatted data of a disk, then you can cover all possible copy protection schemes.

 

In fact, maybe a new disk format should emulate tracks, not sectors at all, complete with header information and everything.

[drac030]

If someone does design a new disk format, it should include things like:

 

I disagree. Holding images of protected disks is rarely used, the vast majority of ATR files are ordinary, non-protected (or cracked) disk images. So the new format should be able to hold such an image in a way that is as simple and consitent as possible; BUT, as other pointed out, it should be also extensible, so that special software might include such information as you required above. Such a possibility shouldn't however make normal data images painful to handle.

[cyco130]

I disagree. Holding images of protected disks is rarely used, the vast majority of ATR files are ordinary, non-protected (or cracked) disk images. So the new format should be able to hold such an image in a way that is as simple and consitent as possible; BUT, as other pointed out, it should be also extensible, so that special software might include such information as you required above. Such a possibility shouldn't however make normal data images painful to handle.

 

Apparently there are two incompatible expectations from a future format: it should be easy to handle and it should emulate all possible copy protections methods.

 

Maybe there is a need two plainly different formats instead of two flavors.

[drac030]

it should be easy to handle and it should emulate all possible copy protections methods.

 

Let me reword that: it should be easy to handle and it should be CAPABLE OF emulating all possible copy protections methods.

 

Keeping such information as the length of a sector in milliseconds is useless most of the time, as most disks are simply a number of valid sectors and the exact method/time of accessing them is irrelevant for the software. Disk images, apart from holding historical software, are also in daily use for hard disk backups for example. Why a simple HD backup should be weightened down by such an useless information as a length of a (HD) sector in milliseconds?

 

At the other hand, a protected disk has also all features of an unprotected disk, i.e. it has a number of sides, tracks, sectors per track, sector size. So maybe the new format should be a container, that could provide these parameters for all images, and contain the image itself with separate header indicating its exact type and providing information to access it. For example, an image could look as follows:

 

I. Container header

 

1) signature indicating that this is an Atari disk image

 

2) 16-bit unsigned offset to the actual image header: makes the container header extensible

 

3) geometry

 

a) number of tracks (unsigned 8 bits), like: 0 - harddisk dump, 1 - partition dump, any other: floppy disk image

 

b) number of sectors per track (floppy) / total number of sectors (hard disk) (unsigned 32 bits)

 

c) sector size

 

II. Image header

 

1) signature indicating the image type

 

2) (image-type dependent data)

 

For a standard image the II.2 will be the beginning of the first sector (stored always in full extent). For other image types (protected etc.) any other required information.

 

This IMHO will make the image either simple AND capable of holding any information which may be required in the future.

Edited December 27, 2009 by drac030

[cyco130]

Let me reword that: it should be easy to handle and it should be CAPABLE OF emulating all possible copy protections methods.

 

Exactly.

 

I had a similar container format idea. But what I had in mind was chunk based, something like:

 

Container signature followed by chunks in the form of:

 

Chunk signature

Chunk size

Chunk data

 

And in the simplest model there would be a geometry chunk followed by a raw data chunk.

 

Why chunk based? So we can add information like CRC code, disk information (like copyright or created by etc.) and even application specific information if we allocate some part of the chunk signature for this purpose. i.e. computer emulators could store compatiblity info in there (OS revision, stereo pokey emulation etc.)

 

The simplest program would skip through the chunks until it finds the geometry and raw data. If it can't find raw data, it's probably either copy protected or compressed for example.

[drac030]

Yes, why not. But I am not sure if it should be so totally "chunky", for example, a disk image does not need more than one geometry information, so perhaps this should be a part of the general header (file-type signature + geometry), which in turn is followed by chunks. Because, for example, a disk image may need two data chunks = two sides of a floppy, and this is where this idea will work.

[cyco130]

Flippies in a single image. Very nice idea indeed

 

What about flippies with two sides formatted differently? I've just bought some 50 atari diskettes from a guy, there were plenty of flippies with SD on one side and ED on the other. Wouldn't it require separate geometry infos?

[drac030]

Hum, I have never seen such a program, which occupies two sides on a disk, and these two sides are in different densities. But it is even theoretically possible, so, yes, each data chunk should be able to have own geometry information, I agree.

[HiassofT]

I'd recommend separating standard (unprotected) images and protected images into 2 different formats. Personally, I really hate all-in-one container formats (like avi, for example) because you never now what's inside the container.

 

Concerning standard images: also think about low-end devices like the SIO2SD or SDrive. The microcontrollers only have 1-4k of RAM and handling ATR images is already quite painful because the 16 byte header makes sectors unaligned to physical sectors on the storage device.

 

Chunk based formats are really nice on large computers, but if designed improperly can be hard to handle on low-end microcontrollers.

 

As for disk geometry etc.: This is usually quite unimportant. The Atari OS/SIO protocol already abstracts storage devices and accesses are done using logical sector numbers (like SCSI is doing for decades and ATA LBA access is doing for 15-20 years). Geometry information was introduced later by Percom and IMO is only useful for floppy disk drives.

 

When loading an image without geometry information into an emulator it's sufficient to check if they contain 720 SD/DD sectors, 1040 SD sectors (and maybe 1440 DD sectors) and create a standard SD/ED/DD/QD percom block on the fly for compatibility with DOSes. All other images (eg 12345 DD sectors) can safely be represented as a "harddisk image", i.e. 1 track with 12345 sectors. For the other way round (when the Atari sends a Percom block to the emulator), just multiply sectors, tracks and sides and work with the total number of sectors.

 

Of course it doesn't hurt if the disk image contains an optional geometry block, but as already written it's not that important IMO.

 

As for copy protected disks: this is really a different can of worms. Back then I was involved in the ATP format design (and also wrote a reference implementation to handle the format with AtariSIO), but the project is quite dead. We spent too much time designing the format and thinking about possible protection schemes and how to represent them, but noone ever wrote an imaging tool.

 

The ATP format is quite OK, but I'm not too convinced anymore that it was really a good design choice for archival purposes. It's too much centered on how the 1050 (and 810 etc.) and known copy protection schemes work, but it doesn't really represent the data that's stored on the disk.

 

Some time after the ATP design process I discussed this topic with some other people and we had a completely different, IMO a lot better idea: Just connect a CPLD with some RAM or a microcontroller to the disk drive and store the flux transitions as they are fed into the FDC. Basically you just need to store the time between the transitions, which is (almost) constant for unprotected disks and can easily be compressed using a simple hoffman or arithmetic coder. Of course you also have to store the position of the index hole/pulse (to have a reference point), and if you add some further information (for example derived from an optical scan of the floppy disk) one could also represent disks protected by "laser lock" (i.e. a small hole in the disk).

 

Then, based on the flux transition one can (actually, has to) implement a model of the FDC. Although this is quite some work, it has the advantage that different FDCs (2793/2797/'765, ...) can be emulated.

 

For example: the "weak sector" protection scheme heavily relies on the FDC used in the drive. When writing a "weak sector" the write head is switched of for a short amount of time so there are no flux transitions for a few us. When reading this sector the PLL inside the FDC gets out of sync (because no flux transitions are received) and the FDC outputs some kind of "random data". Some time after the FDC receives flux transitions again, the PLL syncs and stable data is received (but the exact contents also depend on the FDC and it's PLL).

 

Another example: Tracks with interleaved sector header and sector data (aka "100 sector tracks"). It's possible to store a valid sector header in the data part of another sector and, if you do the math right, also store correct data checksums so that it appears that a track contains 50 or more correct sectors. Depending on the exact time when you start reading a sector you get different (but always valid and consistent) data. This scheme is really a mess when representing it with the ATP format and it's also next to impossible to re-create a real disk from the ATP data.

 

OTOH creating real disks from the "flux transition format" is really simple. One just needs a simple piece of hardware that interfaces with the disk drive. And, this also works for other disk formats (C64, CPM, Amiga, PC, ...) as well, no matter if they used FM, MFM or GCR encoding.

 

so long,

 

Hias

[drac030]

Conclusion? Maybe we stick to my suggestion above, slightly modifying it to be capable of holding two-side floppies where both sides are in different densities?

[+bf2k+]

When loading an image without geometry information into an emulator it's sufficient to check if they contain 720 SD/DD sectors, 1040 SD sectors (and maybe 1440 DD sectors) and create a standard SD/ED/DD/QD percom block on the fly for compatibility with DOSes. All other images (eg 12345 DD sectors) can safely be represented as a "harddisk image", i.e. 1 track with 12345 sectors. For the other way round (when the Atari sends a Percom block to the emulator), just multiply sectors, tracks and sides and work with the total number of sectors.

 

 

My 720k XF551 can format 2880 DD secs, 2 sides so you'd need OD (octo-density ) too!

 

I guess I am not sure why we'd need 2 images stored as one (flippies)... why not just 2 images?

Edited December 27, 2009 by bf2k+

[cyco130]

Hum, I have never seen such a program, which occupies two sides on a disk, and these two sides are in different densities. But it is even theoretically possible, so, yes, each data chunk should be able to have own geometry information, I agree.

 

The diskettes I was talking weren't realy originals. They were mostly a game on the single density A-side. The guy probably formatted the B-side in ED and used them for his own work, like magazine type-ins. And, thinking again, having two ATR images with appropriate file names works fine with flippies.

 

After reading hiassoft's post, I'm now leaning again towards two different formats for copy protected and simple images.

 

@hiassoft:

 

What would make sio2sd type devices' work easier? A 512-byte header followed by raw sectors? All sectors aligned to physical sectors?

 

I must say I'm quite inexperienced in copy protection. Back then, almost every title sold here in Turkey, even by the official Atari distributors were cracked copies. I would be happy to do the leg work, like writing imaging tools, for a new copy protected disk image format but unfortunately I have no way to test them. I don't own even a single original title!

[HiassofT]

My 720k XF551 can format 2880 DD secs, 2 sides so you'd need OD (octo-density ) too!

I've got an HDI with 2 1.44MB drives (5760 DD sectors) but almost never used it - HD floppy disks tend to be very unreliable.

 

So, well, we'd end up with maybe 10 different floppy formats (if we add all strange formats like 77track disks etc.). But I guess all other 120k other formats can savely be represented as "harddisks" without running into compatibility problems (the geometry information in the percom block is only important when formatting a disk, standard read/write access usually works fine even if the geometry information is way off).

 

I guess I am not sure why we'd need 2 images stored as one (flippies)... why not just 2 images?

I don't get the point either. If both sides are part of a double-sided disk (eg XF551 QD), just store the image as a single linear block - like the Atari sees it (or in case of proteced disk store separate information for side 0 and side 1). If it's a flippy, store it as 2 separate images.

 

so long,

 

Hias

[HiassofT]

What would make sio2sd type devices' work easier? A 512-byte header followed by raw sectors?

For example, yes.

 

If we create such a format, it's a good idea to put a version field inside the header and define that all unused bytes must be zero. If we later need to extend the format (for example add a jpeg of the cover), increase the version number and define some of the previously unused bytes to contain a file offset pointer (where the jpeg starts, eg immediately after the disk image data) plus it's length.

 

Older programs will the still be able to access the disk image (this part is unchanged), and newer programs will be able to display the embedded picture as well. Just as a very simple example.

 

All sectors aligned to physical sectors?

No, that'd be too much. But I think it's a good idea to use the same number of bytes for each sector (like KMK said). The format definition inside the header should then contain information how to use the data bytes (i.e. just use the first 128 bytes of sectors 1-3 of a 256 or 512 byte image, or if all data bytes should be used). BTW: it is also a good idea to support storing sector 0 (the SIO protocol allows accessing sector 0 of a drive) and to allow more than 65535 sectors (IIRC the KMK interface supports 24bit sector numbers).

 

I would be happy to do the leg work, like writing imaging tools, for a new copy protected disk image format but unfortunately I have no way to test them. I don't own even a single original title!

Similar situation here. I have a few copy protected original disks, but none of the important titles (concerning copy protection), for example EA games. So I can't really test either.

 

so long,

 

Hias

[+bf2k+]

[

I've got an HDI with 2 1.44MB drives (5760 DD sectors) but almost never used it - HD floppy disks tend to be very unreliable.

 

 

On my XF 720, I only use true 720k 3.5" disks which are very reliable - I've never tried any HD disks in it... maybe I will just for fun to see how well they work - it will only format them to 720k.

[+bf2k+]

 

Similar situation here. I have a few copy protected original disks, but none of the important titles (concerning copy protection), for example EA games. So I can't really test either.

 

 

I have few original EA disks that I could test as well as an array of 1050's (stock, Happy & USD).

[Hornpipe2]

Let me reword that: it should be easy to handle and it should be CAPABLE OF emulating all possible copy protections methods.

 

Exactly.

 

I had a similar container format idea. But what I had in mind was chunk based, something like:

 

Container signature followed by chunks in the form of:

 

Chunk signature

Chunk size

Chunk data

 

And in the simplest model there would be a geometry chunk followed by a raw data chunk.

 

Why chunk based? So we can add information like CRC code, disk information (like copyright or created by etc.) and even application specific information if we allocate some part of the chunk signature for this purpose. i.e. computer emulators could store compatiblity info in there (OS revision, stereo pokey emulation etc.)

 

The simplest program would skip through the chunks until it finds the geometry and raw data. If it can't find raw data, it's probably either copy protected or compressed for example.

 

Of the various suggestions offered so far I like this one best. 4-character signature ("HEAD", "DATA", "GEOM" etc), then uint32 length, and then raw data. I've worked with chunky file formats before and they were so easy to use.

[flashjazzcat]

Bumping this, the latest version of Aspeqt handles Draco's 32MB ATR very nicely. Sadly my SIO2SD doesn't recognize it, though...

 

Has any progress been made on the modified ATR format?

[russg]

I'm curious...

 

If I have one of those ATR's with sec 1,2,3 = 256b each, what would happen if I tried to write this ATR to a real floppy? Would a Happy 1050 or USD 1050 be able to deal with the 1st three secs = 256b?

I believe so. The disk drive is double, single or enhanced and all the sectors are of the specified size.

If you look at a double density disk sector one in a sector editor, such as Disk Wizard II, you will see it is

256 bytes, you have to specify double density in DW II, and all sectors are that size. In my opinion

ATRs should have 256 byte sectors for all sectors of a DD image, the 'extra' 384 bytes are no problem.

The A8 OS will load the first three sectors, but only 'pay attention' to the first 128 bytes.

(off to double check DW II).

I'm back. DW II only shows the first 128 bytes, you can't 'HALF' sectors 1 to 3.

However, I'm still certain that a real DD disk has 256 byte sectors for 1 to 3.

I verified this by dumping sectors 1 to 3 to a file using my own disk utilitly.

The file had 768 bytes (3*256) and the last 128 bytes of the sector dumps of

each sector were filled with zeros.

This actually goes back a long time ago that I verified that 1 to 3 have 256 bytes

and I don't remember the exact way I determined it. Somebody talked about DD

ATRs have 384 bytes tacked onto the end of the ATR, this isn't correct.

Here's my utility. It can be used, for instance, to read sectors to a file, like if you dumped memory

to sectors with Omnimon. It talks about SIOPLUS disks, you can ignore this, it was used to read/write

files to/from a PC drive and has a companion PC program.

sioplus.zip

Edited August 27, 2010 by russg

[HiassofT]

However, I'm still certain that a real DD disk has 256 byte sectors for 1 to 3.

This is correct. On the disk all sectors are formatted with 256 bytes.

 

But: the drive firmware always truncates them to 128 bytes (or pads them up to 256 byte when writing), and there's no way to access the upper 128 bytes of sectors 1-3 via standard SIO calls. The only possibility to do this from the Atari would be to use a Happy or Speedy drive and upload some special code to the drive. But this can be seen as highly non-standard :-)

 

so long,

 

Hias

[russg]

I'm curious...

 

If I have one of those ATR's with sec 1,2,3 = 256b each, what would happen if I tried to write this ATR to a real floppy? Would a Happy 1050 or USD 1050 be able to deal with the 1st three secs = 256b?

The ATR will write to secs 1 to 3, but the OS will substitute zeros for the last 128 bytes.

You can read and write 256 bytes to 1 to 3 using SIO commands, but the last 128 bytes are ignored, zeros.

In effect you can only write to the first 128 bytes. I don't know about any checksums.

I tried to get ANADISK to work for me to prove there are 256 bytes in a DD disk 1 to 3, but I don't have any clean 5.25" DD blank disks for Marslett's UTIL, or HIAS writeatr, or my own PCUTIL to work. Once a DD disk has been written on A8, you can't then use that disk in a PC, at least I can't.

Edited August 27, 2010 by russg

[flashjazzcat]

I hope we can get these 512 byte per sector ATRs working with SIO2SD (a firmware patch is required). And SIO2IDE for that matter. Just been testing Draco's 32MB ATR with Aspeqt at 3xSIO and SpartaDOS X and reads and writes are fast...

Edited August 30, 2010 by flashjazzcat