TIImageTool 3

[Brufnus]

34 minutes ago, mizapf said:

I don't think that OmniFlop understands TDF (unless it explicitly says so) anyway.

No, probably not. Anyway, now it works fine! CYA, GPL, ABASIC and SYSTEM/SYS transferred with this format; all three programs ran fine and CYA confirmed the CRC of SYSTEM/SYS. 

 

I won't be needing it much, but it's always nice to have other options than just a serial cable. c".) haha

 

Next project: finalizing my collection and distribute it.  

[Asmusr]

I don't understand the thing about sectors per allocation unit. On the one hand we have (copied from the source code of tiimagetool):

 

    int getSectorsPerAU() {
		int total = getTotalSectors();
		if (total < 1600) return 1;
		else {
			if (total < 3200) return 2;
			else {
				if (total < 6400) return 4;
				else return 8;
			}
		}
	}

 

On the other we have:

int nEffectiveAUSize = (vol.getTotalSectors()>4095)? vol.getAUSize() : 1;

and

			// Data chain pointers
			int nlast = -1;
			for (int i=0; i < m_aCluster.length; i++) {
				int n = m_aCluster[i].end - m_aCluster[i].start + nlast + 1;
				int nStart = m_aCluster[i].start;
				if (vol.getTotalSectors() > 4095) nStart = nStart / vol.getAUSize();
				aFibNew[0x1c + i*3+0] = (byte)(nStart & 0x00ff);
				aFibNew[0x1c + i*3+1] = (byte)(((nStart & 0x0f00)>>8)|((n&0x000f)<<4));
				aFibNew[0x1c + i*3+2] = (byte)((n & 0x0ff0)>>4);
				nlast = n;
			}

 

Why is the AU size not used with fewer than 4096 sectors? This way the effective AU size is either 1, 4 or 8 (never 2). Shouldn't getAUSize() be changed to reflect this, or is it some peculiarity of the TIFiles format?

[Asmusr]

After looking more into this my conclusions are still the same: in DSK images bigger than 1600 sectors, the 'sectors per allocation unit' value used for the data chain pointers is different from the value used for other parts of the image (allocation bitmap, etc.). That's at least how it has to work for compatibility with TIImageTool, Ti99Dir, andexisting DSK files. For the data chain pointers the value jumps from 1 to 4 at 4094 sectors, and then apparently to 8 at 6400 sectors, which again is inconsistent (shouldn't it be 8192?). Perhaps @mizapf can take a look when you're back from holiday?

[+InsaneMultitasker]

A few comments that come to mind as we await Michael’s return.
 

The code seems to be testing the max cluster size (12 bits or 4096 sectors) as a boundary condition.  
 

For 720k images and 1.44mb images used with TIMT, this should work; however, in the DSR/real hardware - including ramdisks and emulated disks via the hfdc - the formatted disk size may be any value up to 12800 sectors.

 

I believe the calculation boundaries are as follows:

sectors per AU:

1-1600 sectors 1:1

1601-3200 sectors 2:1

3201-6400 sectors 4:1

6401-12800 sectors 8:1

(this ratio is a multiple of the bitmap size which is 200 bytes or 1600 bits) 

 

—-

Cluster representation:

1-3200 sectors - 1:1

3201-6400 sectors - 2:1

6401-12800 sectors - 4:1

 

iirc, only the sector pointer nybbles are adjusted. The sector count nybbles are never adjusted.  Which means a single cluster can never exceed 4096 sectors.  

Note that the adjustment boundaries are not multiples of 4096. 
 

—-

if testing the sector number, the above values would be decreased by 1.  (Base 0) 

——

The cluster ratio is not tied to the AU. Maybe it should have been, but it wasn’t.  When the Myarc cards offered 720K formats, the 12-bit cluster was sufficient.  

 

 

[Asmusr]

8 hours ago, Asmusr said:

For the data chain pointers the value jumps from 1 to 4 at 4094 sectors

 I meant 4096, of course.

7 hours ago, InsaneMultitasker said:

Note that the adjustment boundaries are not multiples of 4096. 

Thanks for the explanation. It starts to make sense when "sectors per AU" and "cluster representation" are treated as two different properties. 

[+mizapf]

I'll be back... next Thursday (now in Lithuania, after some days in Estonia and Latvia).

[+InsaneMultitasker]

8 hours ago, Asmusr said:

Thanks for the explanation. It starts to make sense when "sectors per AU" and "cluster representation" are treated as two different properties. 

I was thinking about this a bit more whilst sitting in the doctor's office.  I think Jim mirrored the AU ratio when the sector count was more than 3200 sectors, and skipped the 2:1 ratio for 720K (80 track double density) compatibility. Jim didn't document his code or process, so I always have to return to the source for code and/or any notes that I added later.

 

This is probably more accurate: 

 

Cluster representation:

1-3200 sectors - 1:1

3201-6400 sectors - 4:1

6401-12800 sectors - 8:1

 

Here is the code for the cluster allocation. It seems to follow the 1:1, 4:1, 8:1 approach.

https://github.com/BeeryMiller/MDOS/blob/b69a3fe6c9d38cd69f23442fb2f93e34c190275a/L8/GETAUS#L106

 

[+InsaneMultitasker]

Lines 126-129  https://github.com/BeeryMiller/MDOS/blob/b69a3fe6c9d38cd69f23442fb2f93e34c190275a/L8/GETAUS#L126C1-L129C22

seem to be adjusting the sector count based on the capacity of the disk. 

 

If true, this is contrary to what I remember and contrary to the N-value usage documented in Ninerpedia File systems - Ninerpedia where it is written, "Important comment to sector/AU usage: For disks with capacities up to 720 KiB (2880 sectors), the M-value refers to a sector. For disks with higher capacities (1.44 or 2.88 MiB, i.e. 5760 or 11520 sectors), M-values are AUs. The N-values are always sector counts, even when the M-value points to an AU".   

 

I don't have time this week to deep-dive the code. I will poke around my development notes for anything I wrote during my review of the HFDC DSR.

[+mizapf]

I'm not sure whether this issue has been resolved by you or not, so I'll add some short comments:

 

About the 1600 sectors: The floppy allocation bitmap is 200 bytes long (starting at offset 0x38 in sector 0 up to its end), so it contains 1600 bits. Available sectors are 0, while used and unavailable sectors are 1 bits (also for those above the disk capacity). Hence, the maximum AU count is 1600. If there are more sectors, the AU size must be increased. AU sizes are from {1, 2, 4, 8, 16}; for the floppy file system, 4 should be the maximum (delivers 1.5625 MiB, exceeding any usable floppy disk format).

 

As for the "Important notes", I think I got this from formatting a 1.4 MiB floppy disk in MAME using the HFDC and checking how the data block chain looks like. The thing to consider is that the N-values are always 12 bits long (3 nibbles), which means their range is from 0 to 4095, so they cannot address a sector with a higher number. Floppy disks with a capacity of 720 KiB contain 2880 sectors, but 1.4 MiB disks already have too many sectors (5760) to address.

[Asmusr]

16 hours ago, mizapf said:

I'm not sure whether this issue has been resolved by you or not, so I'll add some short comments:

 

About the 1600 sectors: The floppy allocation bitmap is 200 bytes long (starting at offset 0x38 in sector 0 up to its end), so it contains 1600 bits. Available sectors are 0, while used and unavailable sectors are 1 bits (also for those above the disk capacity). Hence, the maximum AU count is 1600. If there are more sectors, the AU size must be increased. AU sizes are from {1, 2, 4, 8, 16}; for the floppy file system, 4 should be the maximum (delivers 1.5625 MiB, exceeding any usable floppy disk format).

 

As for the "Important notes", I think I got this from formatting a 1.4 MiB floppy disk in MAME using the HFDC and checking how the data block chain looks like. The thing to consider is that the N-values are always 12 bits long (3 nibbles), which means their range is from 0 to 4095, so they cannot address a sector with a higher number. Floppy disks with a capacity of 720 KiB contain 2880 sectors, but 1.4 MiB disks already have too many sectors (5760) to address.

Whether the threshold for moving from 1 sector per cluster to 4 sectors per cluster is at 4096 (as in TIImageTool) or at 3201 as @InsaneMultitasker suggests is not so important (because real disks are either 2880 or 5760 sectors) as the fact that the cluster representation is different from the sectors per AU. The former is used for the M values in the data chain pointers, and the latter is used everywhere else. That could be more clear from https://www.ninerpedia.org/wiki/File_systems, which only mentions sectors per AU.

 

[+mizapf]

OK, I'll see how to add that to the explanation. It was more like an edge case for me at that point, since only 1.44 MB floppy disks show that issue.

[Asmusr]

27 minutes ago, mizapf said:

OK, I'll see how to add that to the explanation. It was more like an edge case for me at that point, since only 1.44 MB floppy disks show that issue.

Right. It all started because I wanted to load this disk into my emulator.

Funnelweb80.dsk

[Asmusr]

BTW, would you be interested in a set of native installers/launchers with embedded Java for TIImageTool like we have for Magellan? I have the license for a tool (Install4J) where that would be easy to produce. It only supports Java up to version 8 (202), but I guess that's fine. Only thing, when I look at the source code, the version of RXTXcom.jar that I was able to find doesn't have the gnu.io.DriverManager class, so which version are you using?

 

[+mizapf]

The RXTX version can be found on Github. The original author (Trent Jarvi) obviously abandoned it, and someone else picked it up, reorganized and refactored it, but soon the project fell asleep in a half-complete state. If I remember correctly, I was actually one of those who triggered the project again so it got finished.

 

https://github.com/rxtx/rxtx (development branch)

 

You need maven for building, and it is a bit tricky to pick the proper build targets, but I finally succeeded. For Linux, I have to do this:

 

mvn clean
mvn -Pwith-linux-x86_64 package
 

At the end you get some jar files, and rxtx-api/target/rxtx-api-x.x.x-development-SNAPSHOT.jar and rxtxSerial/target/rxtxSerial-x.x.x-development-SNAPSHOT.jar are those to be used. Both must be included in the classpath when starting TIMT, and for building, only the first is needed.

 

---

 

As for the installer/launcher, what does it create? The tiimagetool.jar can be double-clicked to start, provided that a Java RE is installed. Does it mean it contains the Java RE?

 

I'm not sure about the Java 8 restriction. When I build TIMT, I'm getting deprecation messages for Java 8 for some time now. So maybe I am forced to raise the minimum Java version.

[Asmusr]

22 hours ago, mizapf said:

As for the installer/launcher, what does it create? The tiimagetool.jar can be double-clicked to start, provided that a Java RE is installed. Does it mean it contains the Java RE?

 

I'm not sure about the Java 8 restriction. When I build TIMT, I'm getting deprecation messages for Java 8 for some time now. So maybe I am forced to raise the minimum Java version.

It generates an installer that installs a native executable wrapper for the jar together with a local JRE. On Windows, where this is probably most relevant, it means you get a normal exe file that you can associate with dsk files. It also works on Linux and Mac OS X, but on those platforms people are not so used to installers. Anyway, just an idea. If you need to use Java > 8, it won't work with my old version of Install4J.

[+InsaneMultitasker]

@mizapf and @Asmusr, Long post ahead. I walked through the DSR code and then ran a test or two. Hopefully, this makes some sense and is accurate.

 

Observations: 

1. For reference, the "floppy"-based bitmap is 200 bytes or 1600 bits.  Each bit represents one allocation unit (AU). 

2. An AU may represent 1,2,4, or 8 sectors; the maximum formatted capacities are therefore 1600, 3200, 6400, and 12800 sectors, respectively.

3. The Geneve DSR derives the sector per AU ratio (1:1, 2:1, 4:1, 8:1) from the total number of sectors allocated to the disk, sector 0 bytes 10-11.

 Example: If a disk is formatted to 2880 sectors, each bit in the bitmap represents 2 sectors, i.e., 2 sectors per AU.

4. The clusters / data chain pointer blocks are affected by the sector per AU ratio:

   a. The First sector N-value is multiplied/divided by the sector/AU value, but only for capacities above 3200 sectors.

   b. The Highest sector M-value (also known as logical sector offset) is not multiplied by the sector/AU value, however, the sector/AU value is used to adjust the pointer to the last sector within the current AU.

   c. As noted earlier, 720K disks don't require adjustments to the M or N values because the total number of sectors does not exceed the 12-bit cluster maximum value.  Only when the total sector count exceeds >xFFF do we need to use the ratio. 

5. The File Descriptor Index Record (FDIR) in sector 1 is not affected by the sector per AU ratio. (Note: the hard drive format DOES use the sector/AU ratio).

 

I formatted a Horizon ramdisk with the maximum 'floppy' capacity via FORM3MEG to illustrate the observations. 

Capacity is 12800 sectors.

Sectors/AU is 8

Sector 0 and 1 are allocated within the first AU, so the catalog shows one AU (8 sectors) used and 12792 sectors free.

 

Sector 0 shows total AUs: >3200 (12800 sectors)
Bitmap: confirms one AU is allocated (for sector 0 and 1)

The following test program writes 25 records to a file. The records are 255 bytes, which also means one record per sector.

Since the program writes 25 records, we know that the file should consume 25 sectors on disk.

25 records = 25 sectors (file) = 3 full AUs and one additional, final AU for the last sector. Therefore, 4 AUs are allocated (32 sectors)

The disk catalog indicates 40 sectors.  This is calculated as follows:

The File Descriptor Record (FDR) requires one sector which consumes one AU.

The file requires 25 sectors which consumes 4 AUs.

Total size:  FDR AU + file AUs = 5 AUs * 8 sectors/AU = 40 sectors.

 

Inspecting the bitmap (sector 0) we confirm that the DSR allocated one AU

for the FDR. (bitmap has changed from >01 to >03)

As expected, the DSR allocated 4 AUs (>0F, or 1111b ) for the file.

 

The File Descriptor Index Record (FDIR) points to the FDR of our test file.  The sector/AU

ratio is not used. 

 

The filename TEST5 was found in sector >8. This is the 2nd AU on disk, which corresponds to 

the updated bitmap (>01 to >03).

Flipping to HEX representation of sector >8, we find the cluster/data chain pointer block. 

 

The data chain pointer block is >40F0 >01xx

 

Byte 1: N2 N1   Byte 2: M1 N3  Byte 3: M3 M2

 

Putting the N and M values together: 

 

First sector: N3 N2 N1 ==>x040
- this value is calculated based on sectors per AU
- this disk is 8 sectors/AU
- thus the first file sector >40 * 8sec/AU = >0200

Highest Sector:  M3 M2 M1 ==  >x01F
 - This value is the highest logical sector offset. In other words, the last sector within the last AU (of this cluster)
-  Adjusted based on sector/AU ratio
-  The test file consumes 4 AUs (4 AU * 8 sectors/AU = 32 sectors).  
-  The highest offset within the last AU is >1F :  3 AU/ 8 sectors/AU + 7
-  This pointer is NOT the same as the FDR-based calculation for file size! 
- If we append a record to this same file, the M value will not change. 

- The DSR-based routines must inspect the true file size and/or highest written record when determining whether to request a new AU for the cluster.

- If this M value exceeds >xFFFF, a new cluster should be created to extend the file. 

 

Unfortunately, it seems that my testing uncovered an allocation bug. 

 

I created a new file by writing only record 1 and 4300. 

This should have generated two clusters during record pre-allocation,

for records 1-4095 and 4096-4300, respectively.

 

Instead, the DSR created a total of 8 clusters.  When I read record 4300, I get trash.  Back in

2009 there was a fix applied to the allocation routine that seems related to this problem. 

 

Lastly, I wrote another program to write 4300 records then read them all back.  I expected to

see a problem at sector 4096.  Before saving the program I tried to delete

the test data file. The system locked.  My guess is that the error trap is flawed and didn't know how

to return to BASIC when deleting a non-existent file. Weird. 

 

Thank you, Myarc, for changing the DELETE <file> command to <KILL> file. /smh

 

I am out of time for today, but I'll post a reference to the two bugs into the Geneve topic before I head out.