Adventures in Floppy Disk Data Recovery

[+FarmerPotato]

Starting a new thread here.  I think this is of interest beyond the specifics of floppy disk formats used by TI  on the 990 minicomputers.

 

My goal is to recover the commented assembly source of Text-To-Speech, which may be on these disks (and more!).  It may be the Home Computer version, or the program that came from.

 

I have flux-level images of seven backup disks from the TI Speech Lab, thru Dec 83.  There are two undamaged disks, labeled KMG2 and KMG7.  The others show signs of de-gaussing, but probably have many recoverable sectors.  

 

The catch is, they were read by Applesauce, with no known format to go by, so it did not perform the necessary sync before data.  So I have to learn how floppy controllers work when they write.  Specifically, the Texas Instruments FD1000 TFDC and the "Texas Instruments Double Density Format" (TIDD).  

 

I imagine I'll tackle the 4A track formats afterward.   Some of my 5.25" floppies are unreadable now.  I blame early Myarc FDC.  

 

Here is an example of how Applesauce read some bytes (hex).   This disk is in TIDD format.  The encoding is MFM.  At >1C2B, the "2A" is a glitch that throws it off.
 

KMG2 Track 0.1 (Track 0 Side 1) Showing Glitch

 

00001bd0  55 55 55 55 55 55 55 55  55 55 55 55 55 55 55 55  |UUUUUUUUUUUUUUUU|
*
00001c10  0a 08 00 1a 12 01 20 46  9e 55 55 55 55 55 55 55  |...... F.UUUUUUU|
00001c20  55 55 55 55 55 55 55 55  55 55 55 2a aa aa aa aa  |UUUUUUUUUUU*....|
00001c30  aa aa aa aa aa aa 16 00  00 00 20 00 0c 54 54 54  |.......... ..TTT|
00001c40  40 82 88 ac 8a a4 84 a6  40 44 00 00 20 00 34 00  |@.......@D.. .4.|
00001c50  16 54 54 40 50 84 92 52  40 88 8a 8e a4 8a 8a 40  |.TT@P..R@......@|
00001c60  82 88 ac 8a a4 84 a6 3a  00 00 34 00 14 00 06 a2  |.......:..4.....|
00001c70  aa 92 a8 8a 40 4a 00 00  14 00 14 00 06 82 98 9a  |....@J..........|
00001c80  9e a6 a8 4a 00 00 14 00  10 00 04 ac 8a a4 b2 4c  |...J...........L|
00001c90  00 00 10 00 04 50 00 00  04 00 18 00 08 50 82 98  |.....P.......P..|
00001ca0  ae 82 b2 a6 52 48 00 00  18 00 34 00 16 54 54 40  |....RH....4..TT@|
00001cb0  50 84 8a a4 52 40 a0 98  82 86 8a 40 82 88 ac 8a  |P...R@.....@....|
00001cc0  a4 84 a6 3a 00 00 34 00  14 00 06 a8 90 8a a4 8a  |...:..4.........|
00001cd0  40 4a 00 00 14 00 10 00  04 90 8a a4 8a 4c 00 00  |@J...........L..|
00001ce0  10 00 1c 00 0a 8a ac 8a  a4 b2 ae 90 8a a4 8a 46  |...............F|
00001cf0  00 00 1c 00 18 00 08 9c  9e ae 90 8a a4 8a 40 48  |..............@H|
00001d00  00 00 18 00 30 00 14 54  54 40 88 92 a4 8a 86 a8  |....0..TT@......|
00001d10  92 9e 9c 40 82 88 ac 8a  a4 84 a6 3c 00 00 30 00  |...@.......<..0.|
00001d20  10 00 04 82 ae 82 b2 4c  00 00 10 00 10 00 04 84  |.......L........|
00001d30  82 86 96 4c 00 00 10 00  30 00 14 54 54 40 50 84  |...L....0..TT@P.|
00001d40  9c 5c 5c 5c 52 9c 8a 8e  82 a8 92 ac 8a a6 40 3c  |.\\\R.........@<|
00001d50  00 00 30 00 14 00 07 17  b6 06 3e aa aa aa aa aa  |..0.......>.....|
00001d60  aa aa aa aa aa aa aa aa  aa aa aa aa aa aa aa aa  |................|

 

After Correcting the "2A" Glitch

 

00001c00  aa aa aa aa aa aa aa aa  aa aa aa aa aa aa aa 85  |................|
00001c10  04 00 0d 09 00 90 23 4f  2a aa aa aa aa aa aa aa  |......#O*.......|
00001c20  aa aa aa aa aa aa aa aa  aa aa 95 55 55 55 55 55  |...........UUUUU|
00001c30  55 55 55 55 55 0b 00 00  00 10 00 06 2a 2a 2a 20  |UUUUU.......*** |
00001c40  41 44 56 45 52 42 53 20  22 00 00 10 00 1a 00 0b  |ADVERBS ".......|
00001c50  2a 2a 20 28 42 49 29 20  44 45 47 52 45 45 20 41  |** (BI) DEGREE A|
00001c60  44 56 45 52 42 53 1d 00  00 1a 00 0a 00 03 51 55  |DVERBS........QU|
00001c70  49 54 45 20 25 00 00 0a  00 0a 00 03 41 4c 4d 4f  |ITE %.......ALMO|
00001c80  53 54 25 00 00 0a 00 08  00 02 56 45 52 59 26 00  |ST%.......VERY&.|
00001c90  00 08 00 02 28 00 00 02  00 0c 00 04 28 41 4c 57  |....(.......(ALW|
00001ca0  41 59 53 29 24 00 00 0c  00 1a 00 0b 2a 2a 20 28  |AYS)$.......** (|
00001cb0  42 45 52 29 20 50 4c 41  43 45 20 41 44 56 45 52  |BER) PLACE ADVER|
00001cc0  42 53 1d 00 00 1a 00 0a  00 03 54 48 45 52 45 20  |BS........THERE |
00001cd0  25 00 00 0a 00 08 00 02  48 45 52 45 26 00 00 08  |%.......HERE&...|
00001ce0  00 0e 00 05 45 56 45 52  59 57 48 45 52 45 23 00  |....EVERYWHERE#.|
00001cf0  00 0e 00 0c 00 04 4e 4f  57 48 45 52 45 20 24 00  |......NOWHERE $.|
00001d00  00 0c 00 18 00 0a 2a 2a  20 44 49 52 45 43 54 49  |......** DIRECTI|
00001d10  4f 4e 20 41 44 56 45 52  42 53 1e 00 00 18 00 08  |ON ADVERBS......|
00001d20  00 02 41 57 41 59 26 00  00 08 00 08 00 02 42 41  |..AWAY&.......BA|
00001d30  43 4b 26 00 00 08 00 18  00 0a 2a 2a 20 28 42 4e  |CK&.......** (BN|
00001d40  2e 2e 2e 29 4e 45 47 41  54 49 56 45 53 20 1e 00  |...)NEGATIVES ..|
00001d50  00 18 00 0a 00 03 8b db  03 1f 55 55 55 55 55 55  |..........UUUUUU|
00001d60  55 55 55 55 55 55 55 55  55 55 55 55 55 55 55 55  |UUUUUUUUUUUUUUUU|

 

 

How any disk controller would remove the glitch

 

The "55 55 55 55" is a sync signal on which the read process depends.  The sync signal establishes a bit-clock for what follows.  Every 2 microseconds is one bit-cell.   After enough sync signal, "0A" identifies a sector ID, or "0B" identifies sector data.  (I'm ignoring the actual MFM encoding for a 0 or 1 here.)

 

A sector on the track is actually two sections:  the sector ID and the sector data.  Each begins with a sync signal.  A sector write operation will scan the sector ID, then jump in to re-write the data portion.

 

So when a disk controller re-writes the sector data, it begins by replacing part of the sync signal.  This never matches up exactly with the old pulses laid down by the original formatter.  In this case, it was late by a whole bit, in an obvious way (glossing over how it breaks the MFM encoding).    A TI controller would notice that and re-sync to 55 immediately.  No problem. 

 

 

Decoding the Sector ID

 

I will walk through every byte of the hex dumps above. 

 

00001bd0  55 55 55 55 55 55 55 55  55 55 55 55 55 55 55 55  |UUUUUUUUUUUUUUUU|
*
00001c10  0a 08 00 1a 12 01 20 46  9e 55 55 55 55 55 55 55  |...... F.UUUUUUU|
00001c20  55 55 55 55 55 55 55 55  55 55 55 2a aa aa aa aa  |UUUUUUUUUUU*....|
00001c30  aa aa aa aa aa aa 16 00  00 00 20 00 0c 54 54 54  |.......... ..TTT|

 

"55" are sync marks.  The read circuit synchronizes its timing to these. (signal PLLCLK).  IBM format would sync on AA, but TIDD used 55.  

 

>1c10:  "0a" is an address mark (AM1) for the sector ID.  (See Note 1 in references)

>1c11: "08 00 1a 12 01 20 46 9e" is the sector ID and CRC.  Byte by byte:

 

08 side 1 (?) should say 10...
00 track 0
1a sectors/track (yep, decimal 26)
12 sector number (decimal 18)
01 \
20 /  size is >0120 or decimal 288 (yep, not 256)
46 \
9e /  CRC for the address mark and ID

 

 

Decoding the Sector Data

 

>1c19: "55" are more sync marks.

>1c23: "2A" is a glitch!  This is  ... expected.  Typical.  Unavoidable.  It means the sector was re-written at some point.  It is one bit too late, leaving an extra 0.

>1c24: "AA" are the naive interpretation of syncs after the glitch.  A TIDD controller would have synchronized immediately to 55, but Applesauce doesn't know the TIDD format. 

>1c36: "16" should have been "0B", the address mark (AM2) for the data record.  

 

What results from using the glitch bit (0)

 

 0101 0101  (0)010 1010 1010 1010 ....  0001 0110
  5    5      2     A   A    A    ....   1    6     00 00 00 20 00 0c TTT@ ....

 

TIDD result ignoring the glitch bit (0)

 

 0101 0101 (0)0101 0101  0101 0101 .... 0000 1011
  5    5      5     5     5    5   ....  0    B     00 00 00 10 00 06 *** ADVERBS

 

 

 

What Next?

 

I have to copy and paste the hex out of the Applesauce window.  I wrote a program to manipulate it. So far, I can shift a whole track left or right to see what I get.  That's how I found the ADVERBS sector.  The next step is for it to interpret every bit of the track, ignoring the glitches.  We'll see if I get whole tracks of good sectors after that. 

 

References

 

Texas Instruments FD1000 TFDC

"Texas Instruments Double Density Format" (TIDD)

Page 78, Figure 1-23 and following 

 

TM990/303B Floppy Controller

on Colin Hinson's webpages

Table 4-13

Appendix C Figure C-6 (Figure title is wrong)

 

Notes

1. The FD1000 book says the address mark is encoded in an illegal way for MFM, which alerts the controller that it has found an address mark.  (You don't see the actual MFM here, just the decoded bytes.)  But the 303B manual shows the address mark with a legal clock signal, so, I don't know.  On these disks, I see the legal MFM encoding - clock-data 2244 for 0A and 2255 for 0B.

 

 

[+mizapf]

I should know quite a bit about floppy recording, as I had to learn it in MAME so that I could implement the HDC9234 controller on the HFDC. Also, processing the HFE images needs some knowledge about encodings.

 

However, I am a bit confused by the above description. Is that "normal" MFM? ID address marks and data address marks are encoded as A1A1A1FE or A1A1A1FB, respectively (with a special clock bit modification).

 

Some days ago I posted my "AnalyseHFE" Java tool here; this can be used to examine HFE images, which encode everything on cell level. Maybe you can compare your disk with HFE images.

[+FarmerPotato]

10 hours ago, mizapf said:

However, I am a bit confused by the above description. Is that "normal" MFM? ID address marks and data address marks are encoded as A1A1A1FE or A1A1A1FB, respectively (with a special clock bit modification).

 

Here, the main problem is that "Texas Instruments Double Density", or TIDD, is very different from IBM.  Somebody decided this in 1977 or earlier.  TIDD or TI-MFM is a format that exists only on Texas Instruments  disk controllers FD1000 (TFDC) and TM990/303B, not Home Computer. 

 

I started with this figure from TM990/303B:

 

 

I think this table gives the IBM ID address mark as "A1A1A1FE" and data as "A1A1A1FB" as you say.  Some clock bits that should be 1s are 0s, which is how address marks should be. 

 

Applesauce specifies the sync as flux pattern: "AAAA44894489"  which I separate to:

 

  AA        AA        44        89        44        89
 0 0 0 0   0 0 0 0   1 0 1 0   0 0 0 1   1 0 1 0   0 0 0 1 --> data 00A1A1
1 1 1 1   1 1 1 1   0 0 0 0   1 0 1 0   0 0 0 0   1 0 1 0  --> clock FF0A0A
                      '   '     '         '   '     '

' deleted clock bit

 

 

Edited June 24 by FarmerPotato

[Nightengale]

Are those the floppies I gave you PF?

[+FarmerPotato]

2 hours ago, Nightengale said:

Are those the floppies I gave you PF?

No, these are a lot older.  Those floppies were about 40% readable, posted somewhere here about them. 

[+FarmerPotato]

Here is a page from the FD1000 manual showing the TI-MFM version of address mark.

 

(Again, this is not a 99/4A disk controller format.  The FD1000 has its own 9900 and is an intelligent peripheral)

 

MFM Address Marks Figure 2-79.pdf

[+MarkB]

On 6/24/2024 at 9:57 AM, FarmerPotato said:

Here is an example of how Applesauce read some bytes (hex).   This disk is in TIDD format.  The encoding is MFM.  At >1C2B, the "2A" is a glitch that throws it off.

I just hacked together a quick-and-dirty A2R decoder.  It shows a bad timing value of 97 (= 6.1usec) where I think you say the "glitch" is (my address counter is off).  If I ignore it and carry on,  the rest seems to look ok?

 

1740 : 00 00 00 00 00 00 00 00 00 00 00 00 11 B9  X=23 03  X=23 95
1750 : 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55
1760 : 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 
1770 : 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55
1780 : 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 M 0A
1790 : 08 00 1A 12 01 20 46 9E 55 55 55 55 55 55 55 55 
17a0 : 55 55 55 55 55 55 55 55 55  X=97 55 55 55 55 55 55 55
17b0 : 55 55 55 55 M 0B 00 00 00 10 00 06 2A 2A 2A 20 41
17c0 : 44 56 45 52 42 53 20 22 00 00 10 00 1A 00 0B 2A
17d0 : 2A 20 28 42 49 29 20 44 45 47 52 45 45 20 41 44
17e0 : 56 45 52 42 53 1D 00 00 1A 00 0A 00 03 51 55 49
17f0 : 54 45 20 25 00 00 0A 00 0A 00 03 41 4C 4D 4F 53
1800 : 54 25 00 00 0A 00 08 00 02 56 45 52 59 26 00 00

 

(BTW it resets the bitcount when it encounters a "mark" (MFM 00 with no clock) shown with 'M' above.  I think this should be sufficient to realign the data bytes.)

Edited July 1 by khanivore

[+FarmerPotato]

Yep, that's exactly it.   It's explained in great detail in that FD1000 depot manual.  
 

I started out by looking where the syncs should be, then I had code to shift to align.  
 

I didn't read the a2r file entire, just one track which I exported from the GUI. 

 

 

[+MarkB]

FWIW here's the code if you don't want to cut-n-paste.

reada2r.cc