Is it a track copier?

[+mizapf]

We recently had a discussion about the techniques of copying disk contents, and there are some programs that seem to be very fast, calling themselves "track copier". I analyzed one example named "SCREAMER".

 

In summary: There is no "raw track copier". This is technically difficult for our controllers; you need special equipment. The standard WD controllers on the TI card and other cards do not have a track write capability*; they can only format the track with blank sectors. You may be able to read the raw track, but you cannot write it in the same way. Some controllers do not even support reading the raw track, like the HDC9234 on the HFDC.

 

Edit: *To clarify, the WD controllers can write a track with freely defined layout, and with some freedom to write data into the gaps and sector contents, but there is a set of reserved byte values that cannot be written in this way. Hence, by "write track" you cannot write arbitrary sector contents.

 

All programs that I found, SCREAMER, Turbo Copy, COPY-C, use a clever way to quickly read the sectors.

 

You can see the strategy of SCREAMER in the attached log file.

 

1. It first learns the sector sequence of the target disk.

2. Then it learns the sector sequence of the source disk.

3. It then issues read sector commands in the learned sequence. This allows it to read all sectors of a track in a single rotation (i.e. in 0.2 seconds for 300 RPM).

4. It reads five tracks in one go, where it shifts the sequence by one sector, i.e. it starts at the second sector of the learned sequence. This is to allow the floppy head to step to the next track, and then it will miss the first sector. To avoid an empty rotation, it reads the second, third, ... last one, and then the first one.

5. It writes the sectors in the learned sequence to the target disk.

This will be repeated until all tracks are read and written.

 

The interleave is not important, since SCREAMER learns the sector sequence and uses it for the read and write operations.

 

You will find some comments in the log from me.

 

screamer.log

[+9640News]

@mizapf

 

I can't attest to anything regarding the TI-99/4A programs, however the Geneve 9640 HyperCopy program, I thought it was about as close as one can get.  It was written for the TI/CorcComp/Myarc FDC's (not Myarc HFDC).  It would read a byte in a time, reading the entire track in one rotation, buffer everything in memory, and then write it back out in one spin of the disk.  With the skew setting, it would create an offset delay to the next track from the previous track so that the write would continue without spinning around for another rotation.

 

 

[+mizapf]

I can check the HyperCopy program in the same way; I'll see where I put it on my disk images. I'll give any bet that it does the same thing as SCREAMER or other programs of this kind.

[apersson850]

The copier I once wrote does indeed read an entire track in one fell swoop. The WD 2793 supports both Read track and Write track.

You don't even have to format a track into sectors. You can store data in it anyway.

Edited April 9, 2022 by apersson850

[+mizapf]

Would you mind sending me a copy of this program?

 

This is not to say that I don't believe you, but I cannot find this feature in the specifications of the WD279X-02 (https://ftp.whtech.com/datasheets and manuals/Datasheets - Misc/western_digital_WD279X-02.pdf).

 

The command set is basically identical to the WD179x; there is just a "WRITE TRACK FORMATTING THE DISK". You cannot write bytes like "F7" because this automatically creates a CRC and writes it to the disk. If you managed it somehow, this would be an undocumented feature of this chip.

[+9640News]

50 minutes ago, mizapf said:

Would you mind sending me a copy of this program?

 

This is not to say that I don't believe you, but I cannot find this feature in the specifications of the WD279X-02 (https://ftp.whtech.com/datasheets and manuals/Datasheets - Misc/western_digital_WD279X-02.pdf).

 

The command set is basically identical to the WD179x; there is just a "WRITE TRACK FORMATTING THE DISK". You cannot write bytes like "F7" because this automatically creates a CRC and writes it to the disk. If you managed it somehow, this would be an undocumented feature of this chip.

 

Source and programs at BeeryMiller/HyperCopy (github.com)

 

HC-CC / HC-CD Corcomp controller program

HC-MY / HC-MZ Myarc controller program

HC-TI / HC-TJ  TI controller program

 

• Quote
• Edit
• Options  

[+InsaneMultitasker]

I wonder if the fine line here is whether the authors considered their definition of 'track copy' to be reading/writing raw tracks or reading/writing a track at a time, by low-level sector IO.  The commont Disk Managers like DM2 and DM1000 and Disk Utilities all copied disks using the DSR's sector IO routine, and were quite slow.  By directly accessing the controller chip, the "track" copiers bypassed the controller's DSR to optimize read/write times.  Hypercopy, Rapidcopy, Screamer, and others used variations of skew, format timing, and alternating side 0/1 to improve performance.  Neat programs.. I only wish we had source code to peek under the hood at more of them.

[+mizapf]

So I had a look at HyperCopy now, for the Myarc DDCC1. We can assume that the program works in the same way for the TI controller and for the Corcomp.

 

Attached you find the log file for a copy process with HyperCopy (DSK1 to DSK2). I added some comments for the first track; the rest is working the same way.

 

As with SCREAMER, HyperCopy first learns the sector sequence of the source disk by reading the headers of the first two tracks. You have to specify the interleave (side note: "interlace" is the term used for video output, while "interleave" is used for the sector sequence in a track) and the skew.

 

Then, HyperCopy formats the track on the target disk by creating a track layout according to the selected interleave and skew. As it is not possible to write the sector contents in that process, they are filled with E5. After writing the track, the sector contents are filled in according to the sequence. This allows it to write all 18 sectors in one revolution (200 ms). However, it takes two revolutions for HyperCopy to write a track, while SCREAMER needs only one.

 

HyperCopy copies both tracks of each cylinder (side 0, side 1) while SCREAMER first works on side 0, then on side 1.

hypercopy.log

[+mizapf]

13 minutes ago, InsaneMultitasker said:

By directly accessing the controller chip, the "track" copiers bypassed the controller's DSR to optimize read/write times.  Hypercopy, Rapidcopy, Screamer, and others used variations of skew, format timing, and alternating side 0/1 to improve performance.  Neat programs.. I only wish we had source code to peek under the hood at more of them.

The clever idea of all those copiers is to learn the sequence of sectors first, and then to assume this sequence is the same for all tracks.

 

One might concede that you can read one sector in a go, but not several ones or all sectors of the track back-by-back, but the disk controllers do not buffer the sector contents (except for the HFDC). That is, if it is possible to read one sector (and it is, of course), it is possible to repeat that for every sector on the track. The time from one sector to the next one is sufficient to get ready (there is a gap in between), unless you want to copy the contents somewhere else - this may take too much time, and you miss the next sector. Here, you can profit from using the low-level controller commands by avoiding to write the bytes into the small VDP memory but instead copy them into normal RAM. Thus, there is no need to copy sector contents elsewhere.

 

Well, yes, what does "track copy" mean? These copiers do not copy the raw track, but they read each sector from the track in one revolution (in 200 ms!) and also write each sector in one revolution.

 

If you are familiar with the FM and MFM recording concept, you see that reading the gaps is highly unstable. When the disk controller writes a sector, it waits for the IDAM (ID address mark) and the following header which identifies the sector in the track, and then there is a small gap which I sometimes compare to a landing strip. The controller "touches down" near the end of the gap, writes another mark (the DAM, Data address mark), the sector contents, and the CRC. After that you have another gap, but depending on how precisely the head started writing on the medium, you may be ready with the CRC a bit early or a bit late. It mainly depends on how precisely the floppy motor spins the disk. If you are a bit late, and the CRC reaches out into the gap, the data in the complete gap is invalid because some bits are missing. For that reason, you cannot rely on the gap contents in any way, at least on a disk that is rewritten. The IDAMs and DAMs (and sometimes also the index hole) are the only ways for the controller to synchronize to the contents of the track.

[+mizapf]

I once prepared this document for my lecture to illustrate the FM encoding.

fmtrack.pdf

[+acadiel]

Every time I see threads like this, I fondly remember MG’s copy protection by only formatting some tracks ad the beginning and then later in the disk - and it would check that unformatted tracks errored out and that the later tracks did have the executable contents. :-)

[apersson850]

That's exactly the thing my copier was designed to handle.

@mizapf So what I mean by writing a track is that my copier can write a track with unformatted data, and also detect and read such a track. It doesn't write data into a normally formatted track using a track write command.

It does use read track to read a properly formatted track (one with sectors), then tries to analyze the data to figure out how the track is formatted. Like if the sector numbers start at 40 and increment by 5, instead of starting at zero and increment by one. If it figures out that a protection scheme is based on track ten having four sectors, numbered 40, 45, 50 and 55, then it uses track write to create such a track, and then uses sector write to store the data in these sectors. Other tracks can have other formats (usually the standard one).

 

Now looking in your first post again, I see there's a note about this very thing. So we agree: The controller can write a track, which is what it normally does when it formats the disk. But, as you point out, some bytes are used as commands to insert sector start indicators and similar stuff. The opposite is also true. When reading a normally formatted track (with sectors), sector start bytes are read back as data bytes. It takes some dechipering to understand the track that way.

 

My copier was not written to optimize copying performance, but to be able to deal with disks that have varying sector numbering in different tracks, or unformatted data in some tracks.

While doing it, I did find out that the standard sector interlacing in the 99/4A isn't the most efficient when using the p-system. It probably is, when using the more flexible file system in the standard operating system. The p-system, with its roots in machines with low-capacity drive systems, has a simpler file system. But less flexibility implies higher speed, so it benefits from a tighter interlacing scheme. It's ready to read/write the next sector one sector earlier. In the dformat program I wrote, to format disks under the p-system, I took advantage of this and optimized these disks for p-system use. That's one thing I learned from a desire to be able to make a backup of the Advanced Diagnostics and Explorer disks.

 

This kind of also comes down to semantics. Your headline question, Is it a track copier?, depends on what you mean. I see two different meanings:

1. A copier which faithfully can re-create the contents of the track, regardless of what kind of irregular formatting was used in the track.
2. A copier which can re-create a track in one single track write operation.

As should be clear by now, what I once created was the first one, and the second one is probably impossible directly on the TI 99/4A.

Edited April 10, 2022 by apersson850

[apersson850]

1 hour ago, acadiel said:

Every time I see threads like this, I fondly remember MG’s copy protection by only formatting some tracks ad the beginning and then later in the disk - and it would check that unformatted tracks errored out and that the later tracks did have the executable contents. ?

They stored data in unformatted tracks as well.

[sparkdrummer]

Here is a disk I put together with a lot of different copiers.

I do not think any of them work with a Myarc controller.

DSKCOPIERS.dsk

[+mizapf]

@apersson850Thanks for clarifying, looks like we fully agree on that point. ? There is a persistent myth that some copiers can read and write a track as such, and I just want to make clear that those programs do not work this way.

 

We had some discussion about the MG copy protection a few years ago, and I was able to get it running in MAME if the programs are stored on a HFE image (Lotharek/GoTek).

 

[+acadiel]

They stored data in unformatted tracks as well.

Yes, and it was in a strangely sized sector too as I recall. The “encrypted” executable would get the actual program code from this area.

[+acadiel]

[mention=31818]apersson850[/mention]Thanks for clarifying, looks like we fully agree on that point. [emoji846] There is a persistent myth that some copiers can read and write a track as such, and I just want to make clear that those programs do not work this way.
 
We had some discussion about the MG copy protection a few years ago, and I was able to get it running in MAME if the programs are stored on a HFE image (Lotharek/GoTek).
 

Oh yeah, I remember working on that with you! We improved not only the TI emulation to handle a direct image from a protected HFE file, but we fixed a bug in the disk emulation for the controller chip too!

[apersson850]

4 hours ago, acadiel said:

Yes, and it was in a strangely sized sector too as I recall. The “encrypted” executable would get the actual program code from this area.

Well, it's not any sector at all. It's possible to write data to a track without using sectors. That's what's called unformatted storage. But since some bytes are used as command bytes to create sectors, you can't write everything. But printable ASCII characters are fine.

I don't remember now if it was Advanced Diagnostics or Explorer that used unformatted tracks. One of them "only" used oddly numbered sectors. Whilst figuring out how to copy such a disk I learned a lot of details of disk controller handling.

Edited April 10, 2022 by apersson850

[apersson850]

5 hours ago, mizapf said:

Thanks for clarifying, looks like we fully agree on that point. ? There is a persistent myth that some copiers can read and write a track as such, and I just want to make clear that those programs do not work this way.

As you can understand, with my background, I didn't immediately understand what myth you were referring to, as I knew very well that writing a track in that way is impossible.

What I tried to accomplish with my program was that it should read a track from the source disk, analyze the content (which sectors are in the track), create an identical track on the destination disk and then copy the sectors from the source to the destination.

That way the track was copied, no matter what weird sector layout it may have, so that the track on the destination disk looked the same.

I did it in Pascal (or course), with support routines to handle sector read/write as well as track read/write. Understanding the track content was programmed in Pascal.

Edited April 10, 2022 by apersson850

[+acadiel]

Well, it's not any sector at all. It's possible to write data to a track without using sectors. That's what's called unformatted storage. But since some bytes are used as command bytes to create sectors, you can't write everything. But printable ASCII characters are fine.
I don't remember now if it was Advanced Diagnostics or Explorer that used unformatted tracks. One of them "only" used oddly numbered sectors. Whilst figuring out how to copy such a disk I learned a lot of details of disk controller handling.

Gotcha. It did visualize in the Kyroflux software like a super long sector, so maybe it was just a super long piece of raw data.

[apersson850]

When a track is formatted normally, it has this general structure in the magnetic information:

 

Beginning of track filler bytes

Track index mark

Gap

A number of sectors (normally 9 or 18 on the TI), consisting of:

Sector ID address mark

Sector number

Gap

Data mark

Data bytes in the sector

Gap

End of track filler

 

The sector positions are somewhat floating on the disk. The gaps allow a re-written sector to not end up exactly where it was to begin with, due to timing issues, variable rotational speed etc.

Data bytes in a sector can have any value from 00 to FF. A way to implement copy protection is to use unorthodox sector numbers, like 40-48 instead of 0-8, as an example.

 

If you store unformatted data, there will be no sectors. The data you can write is limited to 00 to F0, since some bytes in the range F1-FF have special meanings. Like create ID or data marks, create CRC etc. But you don't waste space on sector gaps and such stuff, so you can store more data on the disk. That's why sometimes disk capacity back in the days was given both as formatted (the number to use) and unformatted (more impressive number for marketing).

This is also a way to do copy protection, since a normal disk controller program would expect to find sectors on the disk in each track, but this track doesn't have any.

When a disk is formatted, a write track command is used, where the gap and filler bytes are specified to certain vaules, just like the commands to create specific identifiers on the track. Bytes like Data marks are written in a special way on the disk, with modified clock signals or similar, so they can always be distinguished by the floppy disk controller from normal data bytes with the same value. Sectors are filled with something. By (IBM) tradition often with E5.

 

When you read a sector, the floppy controller will handle all data marks, synchronization etc., so that what you get back is the actual data bytes in the sector.

If you read a track, you'll get all the fillers and gaps as well as data bytes representing the index, sector ID and data marks. But these marker bytes look the same as a data byte would do, since you don't get the different clock timing in your track read data buffer. To understand the track structure by looking at such a track data buffer takes some analysis, since the special mark bytes don't stand out as special.

 

That is also the particular reason for why you can't just read a track and write it back to get an identical copy. You have to read the track, analyze the content, create an equivalent track buffer with the correct mark commands in the correct places and then write that track. This will give you a track formatted like the original one, but with no valid data. Note that it's impossible to write the correct data at that same time, since the "forbidden" data bytes will be interpreted as commands by the controller. That's why you have to first format the track, then write each sector individually. But you can create an unformatted track in one operation, since the creator will have to make sure the data doesn't contain the forbidden command characters in the first place. Otherwise that track can't be created from the beginning, so there will not be anything to make a copy of.

 

I haven't studied any of these most speed optimized disk copiers. But as far as I understand, they'll adapt to the time it takes to write each sector and also to skip to the next track.

Normally, sectors aren't stored 0, 1, 2, 3, 4, 5, 6, 7, 8 on the disk. That's because it will take some time for the computer to prepare sector 1, after writing sector 0. So the tracks may be interlaced like 0, 3 , 6, 1, 4, 7, 2, 5, 8. This implies that after handling sector 0, the computer has the time it takes the disk to pass sectors 3 and 6, before it's time to handle sector 1. This means it takes three disk revolutions to write all sectors in a track.

If the computer is fast enough to have sector 1 ready alread when sector 6 hasn't passed, then an interleaving like 0, 5, 1, 6, 2, 7, 3, 8, 4 is more efficient. Now it takes two revolutions to write all sectors.

 

Now a smart disk copier will know if it's quick enough to write to every second sector, then it could study the interleave scheme first, then adapt to that. In the second case, it would use the interleaving as it is, and just write the sectors in incrementing order. But in the first case, it would write the sectors in order 0, 6, 4, 2, 8, 3, 1, 7, 5. Thus effectively it writes to every second physical sector on the disk, without changing the interleaving format.

It would also be possible to figure out how much passes by when you step one track. You may find that after writing sector 5 to the track, then stepping one track, that the first accesible sector in the next track isn't sector 0, but sector 3 (due to the interleaving), now assuming the data we want to write there is already in a buffer in memory. If we attempt to write our optimized sequence again, we have to wait one revolution, since it starts with sector 0. By wrapping the sequence around, to 3, 1, 7, 5, 0, 6, 4, 2, 8, we can start writing immediately, to avoid waiting for another spin of the disk.

 

This was just an alternative way of explaining what can be found in the screamer.log by @mizapf

 

Edited April 11, 2022 by apersson850

[+mizapf]

I fully agree with your text. Nicely written. ?

 

Some more technical comments, since we have already come that far.

 

The magnetic cells on the disk have magnetization directions, but they are read by the floppy head and the controller as "changed direction in this cell" or "no change in this cell". You can only sense magnetic changes. These cells are read as 0 if there was no change, and 1 if there was one. With each 1-cell, the phase-locked loop in the controller is adjusted, so if there are enough changes, reading is no problem. The 0-cells are the problem: If you have too many of them in a row, the mechanical tolerances will at some time grow so high that you may read one 0 too much or too few. It's just as if you tried to measure a room with a fixed-length stick, and you have to be very precise the smaller the stick is.

 

There is a condition called "RLL" (run-length limited) that magnetic recordings must comply with: (m,n)RLL means that between two ones, there are at least m zeros and at most n zeros.

 

Hence, there are several possible and used recordings. (0,1)RLL is the FM recording (single density), while (1,3)RLL is MFM recording (double density). (2,7)RLL was used in some harddisks and marketed as "RLL recording".

 

As for FM recording, there must never be two zeros in a row. You may have arbitrarily many ones.

 

To encode data bits, FM recording uses this recipe:

data 0 -> cells 10

data 1 -> cells 11

 

As you see, it is impossible to have two zeros back-to-back.

 

The IDAM is this cell sequence: 1111010101111110. As you see, it complies with the RLL condition (no two zeros meeting). However, it cannot be produced from data: 11.11.01.01.01.11.11.10, as you see from the 01 sequences.

 

The clever thing about it is that even when it is shifted by one position, it still cannot be produced from data:

 

xx.11.11.01.01.01.11.11.10.xx

xx.x1.11.10.10.10.11.11.11.0x

 

So it can always be safely detected.

 

The disk controller looks out for this IDAM, as it is the only chance for it to synchronize to the bitstream. It may have started by reading cells in one gap, but it does not know where it is exactly, and whether it is shifted by one position.

 

...

 

And now you may probably guess why you need a good CPU for MAME, because it is actually converting the DSK images to these cell streams and then doing all that stuff from above. And it is the reason why the DSK images are updated when you leave the emulation but not in the meantime, which means you cannot externally follow changes on your DSK image while the emulation is running.

[apersson850]

On 4/9/2022 at 10:52 PM, mizapf said:

I once prepared this document for my lecture to illustrate the FM encoding.

fmtrack.pdf 185.7 kB · 8 downloads

The track analysis you do there is what I tried to automate in my disk copier program.