SpartaDOS X "Super" Sector Maps

[Pab]

I remember one of the more interesting undocumented features in the original SpartaDOS X was the ability to use what I nicknamed "super" sector maps when I wrote about them for "Current Notes." The super maps provided for rapid random access to extremely large binary files.

 

The way it worked is that if the "previous map sector" field in a file's first sector map was greater than 0, it pointed to the super map, which was a map of maps. Each entry in the super map pointed to the file's individual sector maps.

 

Normally if you wanted to POINT to a spot in a large file above the 32K point, the DOS would have to follow the links from map to map to get the sector number it needed. So if you wanted to point to byte number, say, 79,200, DOS would have to load the first sector map, then the second map, then the third to get the sector number it needed. Then it could load the appropriate sector. And the larger the file got, the longer seeks got. Around 200K a seek required 7 sector reads, for example.

 

With the super map, DOS could keep the super map loaded, and could jump right to the correct map, loading two sectors for any seek no matter how far into a file.

 

The problem with the feature was that it was never fully implemented. If you wanted to use a super sector map with a file, you had to map it out, allocate a sector for the super map, and plug it into the file's first map sector. And before you deleted or otherwise modified a file, you would need to deallocate the super map's sector since the DOS wouldn't know to do so. I wrote utilities to do both these jobs, and made extensive use of them myself when reading mail in PabQWK since most of my packets were well over 100K. I turned over my source code to FTe for use in an upcoming version when it looked like I was going to be working with them after the ICD acquisition.

 

What I want to know is, whatever happened to this functionality? Was it ever properly implemented and documented? Was it removed? Or is it still laying around there somewhere unimplemented and unused?

[+David_P]

The detailed SDX programmer's guide is at: https://sdx.atari8.info/sdx_files/4.49/SDX449_Programming_Guide_EN.pdf

[Pab]

I've read it. The Programmer's Guide doesn't discuss the SDFS layout, although the regular manual does. Neither mentions this functionality. Just curious to know if it was ever removed.

 

[_The Doctor__]

it shouldn't be removed, if it was, it might need some fixing up and re-in statement.

[Pab]

Finally have time to follow up on this.t

 

It appears that when the new routines for random-access "point" operations were written sometime after SDX 4.2, the existence of "super" maps was ignored. Thus, if a "super" map is created for a file it is ignored by the system, and its sector(s) not deallocated when the file is deleted. (They were also not deallocated under 4.2, as they were never fully implemented or properly documented.

 

However, the new routines seem to have solved the problem of "point"ing to a location deep within a large file requiring the chain of sector maps to be scanned from the beginning (which is what the super maps were meant to fix) so they aren't needed any longer.

 

So, mystery solved.

[_The Doctor__]

one still wishes a supermap module existed to allow old software that used the system to continue on, but having updated versions of software that don't need them is a great thing indeed. I am glad you are finding they made a way to not need them in a way you can use.

 

I often wished supermap data could be used to repair disks or hard drive issues when the standard filemap gets damaged/corrupted.

Edited March 28, 2023 by _The Doctor__

[drac030]

The problem with the original SEEK function (or POINT, if you prefer) in SDX was that it was badly implemented: whatever new position you wanted to set within the file, the DOS always started from the beginning of the file.

 

I.e. if your current position was 16777210 and you wanted to set it forward to 16777215 for example (or few bytes back, no matter), the DOS at the outset was setting the position 0, and then went all the way through all the sector maps to the position you wanted.

 

This worked smoothly as long as all the sector maps needed fit in the cache. But if they did not, the DOS at every seek had to re-read the entire sector map chain leading to the new position. In a longer file it was a disaster, and the problem was visible already while listing the contents of a 500k ARC archive (it may be visible earlier, if the cache is declared smaller). I guess anyone still can check how 4.2 behaves in this case.

 

The correct solution to that was not supermaps, but that what has been applied 27 September 2006: seeking not from the beginning of the file, but calculating the new position relative to the current one.

Edited March 28, 2023 by drac030

[Pab]

10 hours ago, drac030 said:

The correct solution to that was not supermaps, but that what has been applied 27 September 2006: seeking not from the beginning of the file, but calculating the new position relative to the current one.

Oh, I agree. The super maps were a kludge and the new code is a much more elegant solution. 

[flashjazzcat]

11 hours ago, drac030 said:

The correct solution to that was not supermaps, but that what has been applied 27 September 2006: seeking not from the beginning of the file, but calculating the new position relative to the current one.

Such an obvious implementation, it's surprising it was not designed this way from the outset.

[Alfred]

14 hours ago, drac030 said:

The problem with the original SEEK function (or POINT, if you prefer) in SDX was that it was badly implemented: whatever new position you wanted to set within the file, the DOS always started from the beginning of the file.

 

I.e. if your current position was 16777210 and you wanted to set it forward to 16777215 for example (or few bytes back, no matter), the DOS at the outset was setting the position 0, and then went all the way through all the sector maps to the position you wanted.

 

This worked smoothly as long as all the sector maps needed fit in the cache. But if they did not, the DOS at every seek had to re-read the entire sector map chain leading to the new position. In a longer file it was a disaster, and the problem was visible already while listing the contents of a 500k ARC archive (it may be visible earlier, if the cache is declared smaller). I guess anyone still can check how 4.2 behaves in this case.

 

The correct solution to that was not supermaps, but that what has been applied 27 September 2006: seeking not from the beginning of the file, but calculating the new position relative to the current one.

That's not entirely true, it did make a bit of an attempt to see if where it was was closer than starting over:

 

;       Absolute position within file                                   
;       -----------------------------                                   
; in:                                                                   
;       addoff  = file position                                         
                                                                        
; NOTES:                                                                
;       This routine first checks if it is closer to start at           
;       the beginning, so it does. Otherwise, it will calculate         
;       an offset and use xrpos instead.                                
                                                                        
                                                                        
pos     clc                     ; temp = filpos / 2                     
        ldx     #2                                                      
psh     lda     filpos,x                                                
        ror     a                                                       
        sta     temp,x                                                  
         dex                                                            
         bpl    psh                                                     
                                                                        
        jsr     caddtem         ; if addoff > filpos/2 then             
        bcc     isrel           ;  do relative positioning              
                                ; closer to start at begnning           
         ldy    #chasms         ; smsec = chain  (first sector map)     
         jsr    move                                                    
 

[drac030]

29 minutes ago, Alfred said:

That's not entirely true, it did make a bit of an attempt to see if where it was was closer than starting over:

The text you have pasted does not seem to come from the SpartaDOS X source code:

 

$(==================================================
        Set Absolute position within file
        ---------------------------------
in:
        addpos  = file position
$)

 

 

and a bit below:

 

        lda     chain           ; set to initial sector map
        sta     smsec
         lda    chain+1
         sta    smsec+1

 

Edited March 28, 2023 by drac030

[drac030]

4 hours ago, flashjazzcat said:

Such an obvious implementation, it's surprising it was not designed this way from the outset.

Probably a provisional code, which stayed, because they saw (I am guessing) that it worked well. And it did - as long as the caches were big enough.

 

I remember that when I first saw that - listing a long ARC archive - I thought that the ARC implementation was doing something wrong. Just when I wrote TAR, which used the seeking and on longer archives behaved the same way, I understood that the DOS itself was to blame.

Edited March 28, 2023 by drac030

[Alfred]

On 3/28/2023 at 6:07 PM, drac030 said:

 

        lda     chain           ; set to initial sector map
        sta     smsec
         lda    chain+1
         sta    smsec+1

 

Yes, the above code doesn't exist in my copy. Mine would be the precursor and possible first couple versions of SDX, say 4.17 and 4.18. I don't see any reference to super maps, so presumably that came later. Since a map sector spans 32K, presumably POINTing wasn't such a big deal except in the case of large ARC type files, Gustafson probably though the extra code wasn't worth the effort. Apart from those, how often are you going to run into a 32K+ file that you're going to do a lot of jumping around in ? BBS message board might be the only thing.

[_The Doctor__]

Databases in general, how's that VCR tape collection, well today Tomagatchi collection, or Disk Database etc etc. Mail Merge? wait no one does snail mail. Recipe database is a good one though...

[kenames99]

On 3/27/2023 at 11:50 PM, drac030 said:

The correct solution to that was not supermaps, but that what has been applied 27 September 2006: seeking not from the beginning of the file, but calculating the new position relative to the current one.

I would have thought Gustafson would have known that but, oh well. I say good fix drac030.

 

 

 

[drac030]

On 3/30/2023 at 2:51 AM, Alfred said:

Gustafson probably though the extra code wasn't worth the effort

The issue still was worth adding the supermap-kludge.

 

8 hours ago, kenames99 said:

good fix

Not only that, but it seems to work, too.

Edited March 31, 2023 by drac030