Validating to prevent "splat" files

[Chris+++]

I've been doing this for decades, but it just occurred to me that I've never run it by Commodore minds who know more than I about how disk blocks are handled by the 1541.

It couldn't hurt to ask, to see if I might be erroneous in my assumption that this is the ideal way to prevent the infamous save-with-replace bug from ruining a file.

It's been working fine for a long time, but I don't remember where I initially read about it, and I've never understood why it works -- or if I might be "overdoing it" by validating a disk every single time I use the save-with-replace command.

I run VICE 64 on True Drive Emulation, but it also worked for years on the real hardware.

 

Several times when I'm writing (and then fine-tuning) a BASIC program, I'll re-save it with the same file name, to prevent using up exorbitant space on the disk / disk image. The format I use is:

 

SAVE"@0:Title",8

 

Immediately after I get the "READY." prompt back, I enter this:

 

OPEN15,8,15,"V0":CLOSE15

 

The closing command obviously causes a pause before I get the prompt again, as 15 can't be closed until the validation is complete.

 

All I know is that it works. But what does it do? Move blocks around to get them all lined-up neatly on the floppy, similarly to latter-day defragmentation? And is there a better way to avoid "splat" files? Thanks.

 

 

Edited July 26, 2022 by Chris+++

[+OLD CS1]

It does nothing, really.  The "cure" to the SAVE@ problem is to use the file syntax you use, that is the drive number (0:) then the filename.  The bug is artifact of the dual-drive nature of the Commodore drive DOS.

 

At the same time, you hurt nothing.  Validation simply scans through the directory looking for non-scratched and properly-closed files, scans through the block-chain of each file, and marks the actual blocks occupied by the files as used in the BAM.  The end result is the directory and the BAM only reflect healthy files.

 

A Commodore 1541 manual says Validate reorganizes and (my word) "defragments" the disk.  However, I have never seen any evidence of that happening in practice.

[Chris+++]

 

Interesting! That's good to know. Thank you for the reply.

 

 

[carlsson]

There were programs though which could move fragments of files around on the disk to optimize loading speed, or even more specifically if you copied files from one disk to another, it would store them sequentially. Normally the floppy drive starts from the directory track 18 and works its way out toward the edges (tracks 1 and 35) which means on a fresh floppy, your programs will be aligned to the center tracks and as it gets more and more full, you might have to do major head movements back and forth. A copying program that instead begins allocation from track 1 and moves inwards can achieve faster loading, in particular with fastloaders as those speed up data transfer but not head movement.

 

IIRC the @0: syntax first stores a temporary copy of the file, then deletes the old copy. It means on a disk with very little room left, it would fail and may leave you with a splat file. A good deal of the information may be rescued by following track by track but if validate has emptied the BAM, you might have lost the opportunity to catch the file. Also I seem to recall that if you use random access storage of files (similar but not identical to relative files), you must not validate the disk since the BAM might not reflect what really is stored in each sector.

[Chris+++]

Wow -- good thing I don't use any type of file apart from PRG and SEQ! I appreciate the info.

 

 

[carlsson]

For a reference what random access files means:

https://archive.org/details/Commodore_1541_Disk_Drive_Users_Guide_1982-09_Commodore/page/n33/mode/2up?q=random

 

You may have stumbled across some, though I believe the usage is relatively uncommon.

[+OLD CS1]

6 hours ago, carlsson said:

Also I seem to recall that if you use random access storage of files (similar but not identical to relative files), you must not validate the disk since the BAM might not reflect what really is stored in each sector.

Or disks used for GEOS as its VLIR format allocates blocks similarly to how RELative files work.  It is interesting that, with all the work Commodore put into relative files in the DOS, they would not have coded the Validate command to accommodate.

 

I used RELative files in my paper route management program, and BBS programs use them for user databases, file section indexes, sometimes for message indexing, &c.

[Chris+++]

That's a good point -- the drive engineers at Commodore surely knew that Validate being poison to particular file types wasn't a good thing.

Yours is one of the best uses of REL I think I've ever heard of. I admit that I never reached a point at which I worked with random-access files of any kind. I think I was a bit scared of them, so I stuck with reading data the long way: having BASIC search through a SEQuential file until it found the brief "heading" it needed, in the form of a simple string. For decades, I probably wasted a lot of disk space that way.

 

[+OLD CS1]

3 hours ago, Chris+++ said:

For decades, I probably wasted a lot of disk space that way.

Not necessarily.  RELative files are great for files that need data updated frequently.  SEQuential files holding mostly-static information are just fine, and have the benefit of not taking up space unused.  That is, in a relative file, you define the record size based upon the largest sizes of your fields you expect.  If you define a record of 240 bytes, but record only consume 180 on average, then you "waste" 60 bytes per record.

 

It likely is not any faster to search one file type versus another.  For instance, searching for a username in a user database relative file, you have to position your record pointer, read in the fields, and make comparison to the requested username, repeatedly until a successful find.  For a sequential file in the same scenario, you do the same thing: input fields until you find the one you need.

 

I mitigated this in my BBS program by having an index database for usernames.  If a user entered his or her user ID number, I could jump immediately to that record in the user database.  If a user entered a username, instead, then I look up a record based upon the first letter of the username in the index, then read the fields in that record which hold active usernames with their user ID (record number.)  If the username is valid, then the program will find it in the appropriate record with its record number to directly access the user database.

 

The other neat thing about relative records and how the Commodore Kernal deals with device file access is the content they hold is not strictly defined, similarly to all other file types.  If you have a record of 240 bytes, one record can hold 240 bytes of text, another can hold 240 bytes of binary data, another can hold some combination of both.  Of course, it is up to the programmer to keep track of this stuff.

[Chris+++]

 

Well, that makes me feel a bit better. I guess RELative files are (in a way) the disk equivalent of arrays that need to be designated with DIM statements -- the space is being used up regardless of the value each "slot" contains.

I appreciate the info. I still find this stuff fascinating, 40 years after getting my first C64. Says something about the computers of that early era.

During what years did your BBS run?

 

[Casey]

While RELative files are in a sense random-access files, in that you can retrieve any record you want, they are different in Commodore's context.

 

Validate won't wreck a RELative file. You create a relative file in a similar way as you do a sequential file (i.e, with an OPEN command, PRINT#, CLOSE, etc)

 

A random-access file in Commodore's world is a file where you manage all of the track and sector use yourself, via the "B-A:", "U1:" commands and you write directly into the track and sector.  Since these are not stored in the directory at all, Validate will free any tracks/sectors that you allocated with the BLOCK-ALLOCATE command.  For this reason, I think Validate will also wreck an autoboot sector on a disk intended for use with the Commodore 128 (since that autoboot sector has to be written with direct access commands).   (I think I have this all correct)

 

GEOS had its own validate command that you could run inside GEOS that would not wreck its files as well.

[+OLD CS1]

1 hour ago, Chris+++ said:

During what years did your BBS run?

It was active from 1989 to 1991.  I ran originally on Vision BBS, then C-Net v12.  I started writing my own BBS program during this time and brought it up for about a year in 1992.  I found that my personal BBS usage conflicted with dial-in time for most users and, being I did not have funds for a third line, I elected to shut down completely.

[+OLD CS1]

2 minutes ago, Casey said:

While RELative files are in a sense random-access files, in that you can retrieve any record you want, they are different in Commodore's context.

 

Validate won't wreck a RELative file. You create a relative file in a similar way as you do a sequential file (i.e, with an OPEN command, PRINT#, CLOSE, etc)

This makes more sense.  I was pretty certain I had run Validate against a disk with RELative files.  Checking the 1541 and 1581 manuals, the warning for Validate does reference the manually-created files using the block commands and not relative files.

 

5 minutes ago, Casey said:

For this reason, I think Validate will also wreck an autoboot sector on a disk intended for use with the Commodore 128 (since that autoboot sector has to be written with direct access commands).   (I think I have this all correct)

The only autoboot file with which I am familiar is the 1581's "COPYRIGHT CBM 86" USR file which does have allocation information.  The 15x1 drives have the "&" utility loader, and it also references a real file in the file system.

 

Ah, but you mean autobooting the C128 from a floppy disk.  Yeah, track 1 block 0.  ISTR reading in "COMPUTE! Magazine" that you could allocate this with a faked file in the directory.  However, since the first two bytes of the file contain "CB" and not a valid track/block combo, the validation would likely remove that file entry and deallocate the block.

 

6 minutes ago, Casey said:

GEOS had its own validate command that you could run inside GEOS that would not wreck its files as well.

I cannot tell you the number of times we had to counsel people on using GEOS's Validate menu command rather than DOS's "V" command.  O.  M.  G.

[+Bruce-Robert Pocock]

Just to hearken back to the original post … and sorry if this is too, um, basic, but:

 

If you are concerned about SAVE not actually saving the file properly, you probably want to VERIFY the save worked…

READY.
SAVE"@0:MY FILE",8

SAVING @0:MY FILE
READY.
VERIFY"MY FILE",8

SEARCHING FOR MY FILE
VERIFYING
OK

READY.
VERIFY"OTHER FILE",8

SEARCHING FOR OTHER FILE
VERIFYING
?VERIFY ERROR
READY.

VERIFY compares the data on disk to the data in memory, ensuring that the save completed successfully. It's basically the same as LOAD, but it compares rather than replacing.

[Chris+++]

1 hour ago, Bruce-Robert Pocock said:

VERIFY compares the data on disk to the data in memory, ensuring that the save completed successfully. It's basically the same as LOAD, but it compares rather than replacing.

 

Good call -- I use VERIFY when I remember, or if I'm especially concerned about the save for any reason. Thanks for the reminder!