a8rawconv, a new raw disk conversion utility

[phaeron]

I tested a rev. 7.7 XF551 and it returns $FF status for a missing data field. According to the flowchart in the FDC docs, it's hard for the drive firmware to treat this any differently from a completely missing sector as on a DAM timeout it simply continues searching for the sector until timeout. HyperXF firmware returns $EF instead, as apparently the XF551's RNF handling was fixed.

 

Edited August 5, 2023 by phaeron

[ijor]

15 hours ago, phaeron said:

I tested a rev. 7.7 XF551 and it returns $FF status for a missing data field.

 

Seems $FF comes from the FDC status register, and I assume you tried a write enabled disk. Can you test again with a write protected disk? Depending on how the firmware manages the timeout, it is possible that $BF comes from the write protected status bit.

[phaeron]

You are correct:

[ijor]

57 minutes ago, phaeron said:

You are correct: ...

 

Good, another mystery solved But more seriously, it's unbelievable how buggy the original XF551 firmware is. Thanks for the test

Edited August 5, 2023 by ijor

[+DjayBee]

On 8/4/2023 at 5:35 AM, phaeron said:

Was that on an 810? You shouldn't be able to get $BF on a 1050 because the 2793/2797 only supports one of the data address mark bits. Unfortunately, ATX's encoding is modeled after the 1050 and reuses bit 6 for another purpose, so it cannot encode both bits either.

Dunno if it helps but @Yellowman tested Forms Generator on a 1050 and received $1C and $EF in DVSTAT and DVSTAT+1 for sector 84 but also for sector 85 which looks good in HXC.

[bani]

so i've been doing some testing and there definitely seems to be something wrong with a8rawconv's apple II support.

 

a8rawconv-x64.exe -if do .\dos33.do -of scp dos33.scp
A8 raw disk conversion utility v0.94
Copyright (C) 2014-2020 Avery Lee, All Rights Reserved.
Licensed under GNU General Public License, version 2 or later.

Reading Apple II disk image (DOS 3.3 ordering): .\dos33.do
Writing single-sided SuperCard Pro 40-track image with 5 revolutions: dos33.scp

 

Disk does not work in a real apple II.

 

The resulting image in HxC Visual Floppy Disk viewer:

 

 

If I use HxC to load dos33.do and export as SCP, then use a8rawconv to write it to disk, the disk works in a real apple II.

 

The resulting scp looks like this:

 

[bani]

This also seems incorrect:

 

a8rawconv-x64.exe -if do .\dos33.do -of scp-ss40 dos33.scp
A8 raw disk conversion utility v0.94
Copyright (C) 2014-2020 Avery Lee, All Rights Reserved.
Licensed under GNU General Public License, version 2 or later.

Reading Apple II disk image (DOS 3.3 ordering): .\dos33.do
Writing single-sided SuperCard Pro 40-track image with 5 revolutions: dos33.scp
Cannot write a 96tpi disk to a 48tpi SCP image.

 

[telengard]

This was the same thing I observed a few pages ago.  I could only write out working A2 disks if I converted the .do/.dsk to scp in the HxC software and wrote it with a8rawconf.

[bani]

looking at the GCR specifications and the a8rawconv source code, i'm not sure that a8rawconv has a working GCR encoder as required by the apple ii format.

[phaeron]

I have a fix pending, just have some other things to clean up before next release. The issue was a typo in one of the header epilogue bytes.

 

[pksw]

On 10/15/2023 at 4:14 PM, phaeron said:

I have a fix pending, just have some other things to clean up before next release. The issue was a typo in one of the header epilogue bytes.

 

Looking forward to it. I've never had it quite working with PRODOS but DOS3.3 seemed ok when I last tried. I think it's something with the sector remapping, but as I'm using an 80 track drive via a greaseweazle USB device, I've never quite known if it's my setup or something else.

Now that @bani is also trying, we might be able to have a few people report on their findings.

[phaeron]

0.95:

• KryoFlux image format
  • KryoFlux streams with index marks at the beginning of the stream are now supported.
  • Decoding side 2 of a KryoFlux stream set as a single sided disk is once again supported by specifying track00.1.raw as the base filename.
• SCP image format
  • Fixed incorrect error when attempting to write a 48tpi image to a 48tpi SCP file.
• Decoding
  • FM/MFM decoder no longer requires the first byte before a DAM to be $00. A 2793 FDC needs about four zero bits in FM to successfully read a sector, but not a full byte.
• Encoding
  • Fixed encoding of MFM sectors that have no data field.
  • Fixed incorrect header epilogue byte when encoding Apple II DOS 3.3 sectors.
  • Encoder now has an -e precise option to use original sector positions and also maintains bit alignment through the written track.
  • Write precompensation is now applied to the inner half of the disk when encoding MFM, to increase reliability of inner tracks.

a8rawconv-0.95.zip

[ijor]

3 hours ago, phaeron said:

Write precompensation is now applied to the inner half of the disk when encoding MFM, to increase reliability of inner tracks.

I'm not sure if by encoding you also mean when converting from higher to lower level, i.e., from ATR to SCP, or only when writing to physical floppies. If the former, IMHO it is wrong behavior.  Precompensation should be applied when writing to magnetic media, not when encoding.

[ijor]

Sorry, forgot to comment about that ...

 

3 hours ago, phaeron said:

FM/MFM decoder no longer requires the first byte before a DAM to be $00. A 2793 FDC needs about four zero bits in FM to successfully read a sector, but not a full byte.

This is also true for address mark detection, not only for data mark. The logic used to detect any FM mark requires an 8-bit data pattern == $0F.

[phaeron]

55 minutes ago, ijor said:

I'm not sure if by encoding you also mean when converting from higher to lower level, i.e., from ATR to SCP, or only when writing to physical floppies. If the former, IMHO it is wrong behavior.  Precompensation should be applied when writing to magnetic media, not when encoding.

Currently, it will apply to conversions to .SCP, yes. However, I am not sure if any of the current utilities for rewriting SCP images back to disk apply precompensation, so it would probably need to be an option. The write precomp that's currently implemented in 0.95 is very simple, it just moves adjacent MFM flux transitions closer by 125ns on track 20+ (48tpi). It's not sophisticated enough to handle arbitrary flux-level streams.

[ijor]

10 hours ago, phaeron said:

Currently, it will apply to conversions to .SCP, yes. However, I am not sure if any of the current utilities for rewriting SCP images back to disk apply precompensation, so it would probably need to be an option.

I don't really know, either. But it is always possible that they will implement precompensation at some point, if it is not implemented already. This might result in performing write precompensation twice, that might make matters worse.

 

Also SCP images might be used for other purposes than writing back to magnetic media. E.g., for using with a low level floppy emulator, like Gotek. Ideally, you don't want a precompensated stream in such a case.

 

I agree that as long as it is optional, and not by default, it is certainly useful.

[bani]

@phaeron have you considered putting a8rawconv up on github?

[phaeron]

9 hours ago, bani said:

@phaeron have you considered putting a8rawconv up on github?

Sorry, no, I have avoided doing this for some personal reasons.

[bani]

On 10/31/2023 at 12:02 PM, phaeron said:

Sorry, no, I have avoided doing this for some personal reasons.

would sourceforge work? gitlab? self hosted gitbucket?

 

i'd like to add native greaseweazle hardware support to a8rawconv like it supports supercard pro hardware natively. a public repo would make that easier.

[feldhamer]

I have this disk here called "In-Store Demo Program". When I imaged it with Kryoflux, it gave a comment that I don't think I've seen before - "Hard-sectored disk, the number of sectors is: 10". Despite this, it went on to read 40 sectors that it reported as OK using FM decoding (some required retries). However, when I run it through a8rawconv, I get nothing - "720 missing sectors, 0 phantom sectors, 0 sectors with errors".

 

I'm very confused. Any idea what is going on? I attached the Kryoflux output. Thanks. :)

AtariDemo.zip

[phaeron]

46 minutes ago, feldhamer said:

I have this disk here called "In-Store Demo Program". When I imaged it with Kryoflux, it gave a comment that I don't think I've seen before - "Hard-sectored disk, the number of sectors is: 10". Despite this, it went on to read 40 sectors that it reported as OK using FM decoding (some required retries). However, when I run it through a8rawconv, I get nothing - "720 missing sectors, 0 phantom sectors, 0 sectors with errors".

 

I'm very confused. Any idea what is going on? I attached the Kryoflux output. Thanks.

AtariDemo.zip 11.35 MB · 0 downloads

Now that's something I haven't seen in a few decades...

 

A hard-sectored disk is a disk that has an index hole per sector instead of once per rotation. It was used in old disk systems that actually used the index hole for sector timing. Most 8-bit systems, though, were soft-sectored and didn't need this, and the Atari drives were not only soft-sectored but also generally outright ignored the index hole. This means you can use a hard sectored disk in such a drive and it won't care.

 

The specific issue in this case is that a8rawconv has code to adjust the flux timing to compensate for the RPM of the reading drive, based on the detected rotational speed from the index timing. This is necessary since Atari drives rotated at 288 RPM while PC drives use 300 RPM or 360 RPM, and the pulse width timing depends on the rotational speed of the drive. The hard sectoring, however, confuses this logic and makes a8rawconv think the disk is spinning very fast, because it sees 11 index marks per rotation instead of 1. I'll need to put in an option to override this logic.

 

[feldhamer]

33 minutes ago, phaeron said:

Now that's something I haven't seen in a few decades...

 

A hard-sectored disk is a disk that has an index hole per sector instead of once per rotation. It was used in old disk systems that actually used the index hole for sector timing. Most 8-bit systems, though, were soft-sectored and didn't need this, and the Atari drives were not only soft-sectored but also generally outright ignored the index hole. This means you can use a hard sectored disk in such a drive and it won't care.

 

The specific issue in this case is that a8rawconv has code to adjust the flux timing to compensate for the RPM of the reading drive, based on the detected rotational speed from the index timing. This is necessary since Atari drives rotated at 288 RPM while PC drives use 300 RPM or 360 RPM, and the pulse width timing depends on the rotational speed of the drive. The hard sectoring, however, confuses this logic and makes a8rawconv think the disk is spinning very fast, because it sees 11 index marks per rotation instead of 1. I'll need to put in an option to override this logic.

 

Thanks for the explanation! I was wondering why I saw the index hole all the time, seemed like an odd coincidence.  Also, I'm dumb - not sure why I keep confusing sectors with tracks.

 

I look forward to the update.

 

And not sure if this has been dumped before....I didn't find a copy when I looked but my search was not exhaustive.

 

Thanks again!!!

Edited January 17 by feldhamer

[feldhamer]

@phaeron while you're making an update, is it possible to fix the behavior where a8rawconv gets confused if the input files have a number in them before the "00.0.raw"? For example, I ran:

 

"a8rawconv c:\utils\kryoflux\dtc\dtc_x64\Arcade200.0.raw Arcade2.atx"

 

and the result was:

 

"Unable to determine filename pattern for KryoFlux raw track streams. Expected pattern: track00.0.raw."

 

Thanks again for this great utility.

[+DjayBee]

I am currently fighting with a very weird Enhanced Density disk.

I can succcessfully convert the SCP dump to ATX and the ATX contains all data but it is not recognized as an MFM image afterwards.

If I try to dump directly from SCP to an ATR, the result is a 90K image. By parsing the ATX it is possible to extract all 1018 sectors and the resulting image then boots until it fails the duplicate sector 1 copy protection.

This is true for v0.92, v0.94 and v0.95 as well.

 

I suspect that the reason is the first track of the disk. It contains 27 sectors of which are 23*sector 00 (a8rawconv only reports 22), 2*sector 01, 1*sector 02 and 1*sector 03.

 

$  ./a8rawconv95.exe -l Guard\ \(1992\)\ Sonix\ \(PL\)\ Medium\ protected.scp gu.atx
A8 raw disk conversion utility v0.95
Copyright (C) 2014-2023 Avery Lee, All Rights Reserved.
Licensed under GNU General Public License, version 2 or later.

Reading SuperCard Pro image: Guard (1992) Sonix (PL) Medium protected.scp
WARNING: Track  0, sector  0: 22 phantom sectors found.
WARNING: Track  0, sector  1: 1 phantom sector found.
 0 (27) |  0 0  0 0  0 0  0 0 0  1 2  3 0  0 0  0 0  0 0  0 0 0    1 0 0  0 0
 1 (26) |   1416 1820 22 2426  1  3 5  7 9  1113 15 1719 2123 252  4  6 8  1012
 2 (26) | 18 2022 2426  1  3 5  7 9  1113 15 1719 2123 252  4  6 8  1012 1416
 3 (26) | 26 1  3  5 7  9 11 1315 1719 21 2325 2 4  6 8  10 1214 1618 2022 24
 4 (26) |   5 7  9 11 1315 17 1921 2325 2 4  6  8 10 1214 1618 20 2224 26 1  3
 5 (26) |   11 1315 1719 2123 252  4  6 8  1012 1416 18 2022 2426  1  3 5  7 9
 6 (26) | 1517 19 2123 252  4 6  8  1012 1416 1820 2224 26  1 3  5  7 9  1113
 7 (26) |  2123 25 2 4  6 8  1012 1416 18 2022 2426  1  3 5  7  9 11 1315 1719
 8 (26) |   2 4  6 8  1012 14 1618 2022 2426  1  3 5  7  9 11 1315 1719 21 2325
 9 (26) | 6 8  10 1214 1618 2022 2426  1  3 5  7  9 11 1315 1719 21 2325 2 4
10 (26) | 1416 1820 22 2426  1 3  5  7 9  1113 1517 19 2123 252  4 6  8  1012
11 (26) | 20 2224 26  1 3  5 7  9  1113 1517 1921 23 252  4 6  8 10 12 1416 18
12 (26) |  26  1 3  5 7  9 11 13 1517 1921 2325 2  4 6  8 10 1214 16 1820 2224
13 (26) |  3 5  7 9  1113 15 1719 2123 252  4  6 8  1012 1416 18 2022 2426  1
14 (26) |   9 11 1315 1719 2123 25 2 4  6 8  1012 14 1618 2022 2426  1  3 5  7
15 (26) | 1315 17 1921 2325 2 4  6 8  10 1214 1618 2022 24 26 1  3  5 7  9 11
16 (26) | 19 2123 25 2 4  6 8  1012 14 1618 2022 2426  1  3 5  7  9 11 1315 17
17 (26) | 25 2 4  6 8  10 1214 1618 2022 2426  1  3  5 7  9 11 1315 1719 21 23
18 (26) | 6  8 10 12 1416 1820 2224 26  1 3  5  7 9  1113 1517 19 2123 252  4
19 (26) |  12 1416 18 2022 2426  1  3 5  7 9  11 1315 1719 2123 25 2 4  6 8  10
20 (26) |  2022 2426  1  3 5  7  9 11 1315 1719 21 2325 2 4  6 8  10 1214 1618
21 (26) |  26  1  3 5  7 9  11 1315 1719 2123 25 2 4  6 8  1012 14 1618 2022 24
22 (26) |  3  5 7  9  1113 1517 1921 2325 2  4 6  8 10 1214 16 1820 2224 26  1
23 (26) | 7 9  1113 1517 1921 23 252  4 6  8 10 1214 16 1820 2224 26  1 3  5
24 (26) | 13 1517 19 2123 252  4  6 8  1012 1416 1820 22 2426  1  3 5  7 9  11
25 (26) |  1921 23 252  4 6  8 10 12 1416 1820 2224 26  1 3  5  7 9  1113 1517
26 (26) |  25 2  4 6  8 10 1214 16 1820 2224 26  1 3  5 7  9  1113 1517 1921 23
27 (26) |  6  8 10 1214 16 1820 2224 26  1 3  5 7  9  1113 1517 1921 23 252  4
28 (26) |   1214 1618 2022 2426  1  3 5  7  9 11 1315 1719 21 2325 2 4  6 8  10
29 (26) | 16 1820 2224 26  1 3  5 7  9 11 13 1517 1921 2325 2  4 6  8 10 1214
30 (26) |  6 8  1012 14 1618 2022 2426  1  3 5  7 9  11 1315 1719 2123 25 2 4
31 (26) |   1214 1618 20 2224 26  1 3  5 7  9 11 13 1517 1921 2325 2  4 6  8 10
32 (26) | 16 1820 2224 26  1 3  5 7  9  1113 1517 1921 23 252  4 6  8 10 12 14
33 (26) | 22 2426  1  3 5  7 9  11 1315 1719 2123 25 2 4  6 8  1012 14 1618 20
34 (26) |   1 3  5 7  9 11 13 1517 1921 2325 2 4  6  8 10 1214 16 1820 2224 26
35 (26) | 5 7  9 11 1315 17 1921 2325 2 4  6 8  10 1214 1618 20 2224 26 1  3
36 (26) | 11 13 1517 1921 2325 2  4 6  8 10 1214 1618 20 2224 26  1 3  5 7  9
37 (26) |  17 1921 2325 2 4  6 8  10 1214 1618 2022 24 26 1  3  5 7  9 11 1315
38 (26) |  2325 2 4  6  8 10 1214 1618 2022 24 26 1  3  5 7  9 11 13 1517 1921
39 (26) |  4  6 8  1012 14 1618 2022 2426  1  3 5  7 9  11 1315 1719 2123 25 2
Writing ATX file: gu.atx
WARNING: Track  0, sector  0: 22 phantom sectors found.
WARNING: Track  0, sector  1: 1 phantom sector found.
WARNING: Track  0: Missing sectors: 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18.
15 missing sectors, 1 phantom sector, 0 sectors with errors


$ ./atxMfmFix.exe gu.atx gu_.atx
ATX MFM fix 0.2
ATX image doesn't seem to be enhanced density

 

 

Guard (1992) Sonix (PL) Medium protected.zip gu.atx gu-parsed.atr

[phaeron]

8 hours ago, DjayBee said:

I am currently fighting with a very weird Enhanced Density disk.

I can succcessfully convert the SCP dump to ATX and the ATX contains all data but it is not recognized as an MFM image afterwards.

If I try to dump directly from SCP to an ATR, the result is a 90K image. By parsing the ATX it is possible to extract all 1018 sectors and the resulting image then boots until it fails the duplicate sector 1 copy protection.

This is true for v0.92, v0.94 and v0.95 as well.

 

I suspect that the reason is the first track of the disk. It contains 27 sectors of which are 23*sector 00 (a8rawconv only reports 22), 2*sector 01, 1*sector 02 and 1*sector 03.

Hmm... this is an interesting one.

 

You're correct that the issue is density detection. The easy issue is the disk not getting detected as ED in a8rawconv, which I can fix by forcing at least ED if the boot sectors are MFM. The trickier part is that there is also a density issue with Altirra trying to boot off of the disk since it has no non-boot sectors on track 0 -- and it turns out the reason why this happens is because of an accommodation I put in for the 1050 Duplicator, which uses true 128 byte MFM boot sectors on a double-density disk. Sure enough, in full emulation mode the 1050 Duplicator can't boot this disk because it gets stuck endlessly trying to buffer track 0. What I'll probably do is make the density detector default to an ED disk if it doesn't see any non-boot sectors as 1050 Duplicator DD disks are far rarer than copy protected ED disks.