Any basics that do better random access files?

[Tyrop]

Are there any versions of Basic for the Atari that have the ability to do random access files similarly to Microsoft Basic (where you can read and write records that are composed of fields)?  Atari Basic’s Note and Point are cumbersome. For that matter, are there any other languages that can do random access files where you can have records that are composed of fields that are strings and fields that are numbers?

[Rybags]

None that I know of.

We're at a bit of a disadvantage in that most of our Doses don't use a FAT, but have the sector chain information at the end of each data sector.

That makes doing random access harder vs FAT where just a few sector reads can allow an index to quickly be built for an entire file (or disk)

Some Doses (3.0 at least, for what that's worth) use NOTE/POINT not as absolute but as relative offsets from start of the file, which could make things a bit easier.

NOTE/POINT in it's "absolute" form in itself is handy but there's no guarantee that e.g. a file starting at sector 100 will have it's 5th data sector at sector 104 - so the only way to know is to actually read the file and construct the index yourself.

[thorfdbg]

This is not really the issue of Basic, it is the problem of the FMS. Unfortunately, the random access capabilities of Atari DOS are rather "limited" as POINT and NOTE take an absolute sector number and byte offset within a sector, and since files are not necessarily assigned contiguous sector numbers, you cannot reliably compute from a record number the sector and byte offset. With DOS 3 (which is really just FAT in a disguise) you could as it uses byte offsets from the start of the flie as offsets, but it was already too late at this time.

 

[TGB1718]

You could, (memory permitting) allocate a block of memory and read your file into it, it's then easy to

traverse the fields and when finished write the memory back to disk.

 

You would obviously need to allocate enough memory to add new records, would be better with a machine

with expanded memory or at least a 130XE.

 

If you were using BASIC, you would need a machine code routine to access the data in expanded memory as

the BASIC program would probably be in the "RAM window" @$4000

 

[Rybags]

A sector map would probably be easiest IMO.

Depending on the situation you could just build it and keep in another file then load it in.  If there's errors relating to file # mismatch then obviously the map is probably broken due to the file moving and it could be rebuilt in that case.

Then you could have caching - keep a copy in memory of recently read records and/or frequently needed records.

As for getting records and splitting into strings and variables - with an index in hand that becomes a lot easier - though potentially you might have issues with short records if the file is appended to ?  Or do our Doses always ensure only the last record is the only possible short one?

 

"Records" - I've got 2 contexts going there - records as in the 125 byte ones that Dos gives us, and the user defined ones containing your sets of variables.

Edited December 29, 2023 by Rybags

[drac030]

41 minutes ago, Rybags said:

A sector map would probably be easiest IMO.

You guys keep reinventing SpartaDOS. No need for that, it has already been invented.

[Tyrop]

Thanks for all the responses. So what I gather is that there is no “put” or “get” record in any Basics because DOS was not originally designed that way.  But later DOSes were designed to use a FAT so it is possible to write routines to accomplish something similar. So if someone wanted to write a program back in the day that had some database functionality, it would have been much more tedious to do on an Atari versus other computers that had Microsoft Basic?  I had gotten an Atari 800 in mid 1981, wrote a simple BBS for it in Atari Basic, then got an IBM PCjr in late 1984 when I started college, and rewrote my BBS on the PCjr in Microsoft Basic using random access files for the message data. That was when I really learned just how much Microsoft Basic differed from Atari Basic. With Microsoft Basic, I was able to do much more, and I wrote a more robust BBS. But I always wondered if the Atari could do Basic just as well as the best of any other computers of the time, and that I just wasn’t using the right Basic. 

[Rybags]

CIO supports Get and Put Record and Basic also has the XIO command which can give some extended CIO functionality.

 

You could do something such as INPUT #1,A$,B,C,D$ but you'd need to ensure that the file data is suitably formatted to be received in that way - each item would need to be terminated with the EOL / Return character $9B (and of course receiving numeric data as ASCII strings can waste space in a lot of cases)

[xxl]

from what I read here, I wonder why databases were developed on systems that did not have FAT FS

[+DjayBee]

6 hours ago, xxl said:

from what I read here, I wonder why databases were developed on systems that did not have FAT FS

Perhaps people did it to prove that they can.

[pcrow]

Note and Point were a key part of Atari's design.  I'm pretty sure that's the reason they include the file number in the last three bytes of each sector; thanks to that, if you point outside the file, you'll get an error (unless it happens to be a sector used by the same directory entry that was previously deleted).  Personally, I think that was a mistake, and if you wanted to have a random-access database, you would just bypass DOS and use raw disk sectors.  I think BASIC would have been better with dropping Note/Point in favor of a command to do raw SIO sector read/write.

 

Along those lines, I'm also surprised that there was never a simple DOS that only allowed contiguous files.  In practice, most files were sequential, so you could design a file system without bitmaps and using all 128 bytes of each sector.  The downside would be needing to do a disk copy to clean up space from deleted files, so you wouldn't use it for development, but it would have been great for disks full of static files.  I'm also surprised SpartaDOS doesn't have a "sequential" flag where it skips the sector listing the sectors in the file since the file is sequential.  With that feature, it would only allocate the sector map for a file if the file grew to be non-sequential.

[Rybags]

The choice as it was is likely a memory saving thing.  Having to buffer large parts of the FAT or keep seeking back to it is inefficient vs streaming the links with the data.

It's easy to look back and say they should have done the other - but also remember the likely target system for early disk systems had 24 or 32K Ram.

[drac030]

4 hours ago, pcrow said:

Along those lines, I'm also surprised that there was never a simple DOS that only allowed contiguous files. (...) I'm also surprised SpartaDOS doesn't have a "sequential" flag where it skips the sector listing the sectors in the file since the file is sequential.  With that feature, it would only allocate the sector map for a file if the file grew to be non-sequential.

It might probably have been done so to uphold some sort of healty balance between adding much to the code's complexity (and size) for no practical gain, and not doing that.

[thorfdbg]

5 hours ago, pcrow said:

Note and Point were a key part of Atari's design.  I'm pretty sure that's the reason they include the file number in the last three bytes of each sector; thanks to that, if you point outside the file, you'll get an error (unless it happens to be a sector used by the same directory entry that was previously deleted).  Personally, I think that was a mistake, and if you wanted to have a random-access database, you would just bypass DOS and use raw disk sectors.  I think BASIC would have been better with dropping Note/Point in favor of a command to do raw SIO sector read/write.

 

Along those lines, I'm also surprised that there was never a simple DOS that only allowed contiguous files.  In practice, most files were sequential, so you could design a file system without bitmaps and using all 128 bytes of each sector.  The downside would be needing to do a disk copy to clean up space from deleted files, so you wouldn't use it for development, but it would have been great for disks full of static files.  I'm also surprised SpartaDOS doesn't have a "sequential" flag where it skips the sector listing the sectors in the file since the file is sequential.  With that feature, it would only allocate the sector map for a file if the file grew to be non-sequential.

The Dos 2.++ (and it's precursor, 2.XL) that comes with Atari++ does offer bypassing the FMS - then you have raw sector access, but of course no files:

 

10 OPEN #1,8,128,"D:-"
20 POINT #1,SECTOR,BYTE
30 PRINT #1;"RAW SECTOR I/O"
40 CLOSE #1

The important trick is the 128 as second parameter for open. This makes the entire disk accessible as a "single file" with 128 bytes per sector. Of course, no file names in this mode.

 

Contiguous files means that the sector allocation of the FMS would need to change. Currently, whenever a new sector is needed, it just checks in the VTOC for a free sector, allocates it and uses it. If you would want a contiguous file, it would need to check whether the next sector is available. What do you do if it is not? Fail with "disk full"?

[pcrow]

On 1/8/2024 at 4:48 AM, thorfdbg said:

Contiguous files means that the sector allocation of the FMS would need to change. Currently, whenever a new sector is needed, it just checks in the VTOC for a free sector, allocates it and uses it. If you would want a contiguous file, it would need to check whether the next sector is available. What do you do if it is not? Fail with "disk full"?

Yes, fail with disk full.  I was thinking of a very simple DOS used mostly for things that don't need to be modified once written.  I would guess that excluding full-disk programs, 90% of my disks would fit that criteria.  So each file would have a starting sector and size in bytes, and the file would always be sequential.  Instead of a VTOC bitmap, you would just have a next sector free number (likely stored in the next directory entry).  Deleting files would not free up space unless it was the last file in the directory, so if you reorganized, you would likely need to copy everything to a fresh disk.

 

I figure the code would be simpler, and it would make better use of the disk space.  Depending on the fragmentation, it could save 2249 bytes per disk.  Likely not worth having an incompatible DOS, but I still like the idea.

 

As to SpartaDOS having a "contiguous file" flag, that would certainly make sense.  If the next sector to be allocated isn't contiguous, it would then allocate two sectors: first the sector map for the file, and then the next sector for data.  I'm guessing that 90% of files would be contiguous, so it would eliminate a lot of sector map sectors, making most files a sector shorter.

[drac030]

33 minutes ago, pcrow said:

As to SpartaDOS having a "contiguous file" flag, that would certainly make sense.  If the next sector to be allocated isn't contiguous, it would then allocate two sectors: first the sector map for the file, and then the next sector for data.

Please provide an algorithm flow-chart of writing such a file to the disk. Especially, if after e.g. 1 MB data written it suddenly turns out that the next sector to be written is already allocated to another file and you have to skip it. And how do you detect that situation while reading it afterwards and deal with it.

 

Not to mention that an Atari program can write two (or more) files to a disk simultaneously...

Edited January 25 by drac030

[pcrow]

3 minutes ago, drac030 said:

Please provide an algorithm flow-chart of writing such a file to the disk. Especially, if after e.g. 1 MB data written it suddenly turns out that the next sector to be written is already allocated to another file and you have to skip it. And how do you detect that situation while reading it afterwards and deal with it.

 

Not to mention that an Atari program can write two (or more) files to a disk simultaneously...

Reading a file while it is being written is a problem.  Most DOS implementations set a flag on the file saying that it's open for writing that makes the file invisible until it's closed, so I don't think reading a file while it is being written is supported in most cases, though I'm not clear on SpartaDOS in particular.  But reading an open file would need to update from the directory entry when it gets to the end of what existed when the file was opened, so it would then see whether or not the file is still contiguous at that point.  What happens now if you open an 8K file for reading, read 4K, open it to append, write 8K, then read to the end.  Will the read stop at the original 8K, or will it catch the update and read the 16K?  If it gets the updated 16K file size, it would also get the change to being non-contiguous

 

When a program goes to write the next sector, and finds it's already allocated, it first has to allocate and write the sector map sector or sectors.  The contents can be easily generated, but this could require a number of sectors to be allocated and written before the write can complete.

 

Multiple files being written at the same time are not a problem, as DOS only deals with one operation at a time.  It would mean the files would quickly become non-contiguous.

 

[Another option that would completely break compatibility would be to use three extra bytes in the directory entry to have a number of contiguous sectors before accessing the sector map (two bytes for the starting sector and one or possibly two for a count, but how many files have more than 255 sectors).  But breaking compatibility is obviously a no-go, so that was only an option back when it was first designed.]

 

So the write flow chart would be:

     [is file flagged as contiguous?]    No  ----->   [Existing write path] ---> DONE
                 Yes
                  |
     [is next contiguous sector free?]   Yes ----->   [Allocate/use next sector] ---> DONE
                  No
                  |
     [allocate sector for sector map]  <----------|
                  |                               |
           [fill sector map]                      |
                  |                               |
     [More sector map sectors needed?]  Yes  -----|
                  No
                  |
         [Clear contiguous flag]
                  |
          [Existing write path] ---> DONE

 

                 

[drac030]

21 hours ago, pcrow said:

Reading a file while it is being written is a problem.

I didn't mean reading a file while it is being written. By "afterwards" I meant "after the file has been entirely written and closed". Before the first closure a file cannot be opened another time.

 

21 hours ago, pcrow said:

What happens now if you open an 8K file for reading, read 4K, open it to append, write 8K, then read to the end.  Will the read stop at the original 8K, or will it catch the update and read the 16K?

Without looking at the code: it will stop the reading at the original 8k (you opened it twice, once for read and once for append, so you have two handles and also two separate descriptors).

 

21 hours ago, pcrow said:

three extra bytes in the directory entry

Hmm, what three extra bytes in the drectory entry?
 

21 hours ago, pcrow said:

how many files have more than 255 sectors

This depends on the sector's size. Plethora of files are longer than 32k (255*128), tons of them - longer than 64k (255*256), quite a lot - longer than 128k (255*512). The real question is - is the percentage of very short files worth all the trouble?

 

21 hours ago, pcrow said:

Multiple files being written at the same time are not a problem, as DOS only deals with one operation at a time.

Yup, you can still easily generate interleaved writes to multiple files "at the same time". Also the more general question is: how many files on a disk frequently written-to are non-fragmented? Looking at my "asm source" partition, there is a ton of files revisited, edited, regenerated from time to time and the fragmentation % is probably rather high. Say, in theory, someone comes and implementes this "contiguous files" feature. What is the gain? Speed? None. Capacity? Sector maps saved, which in the end will have to be written anyways the more the disk gets fragmented. What is the loss? DOS code size/complexity - much added here. A disaster on a floppy drive, when to read a file which begins at a low track it has to seek to a high track again and again, because the system found out that the file can't be contiguous at track 33 and all the sector maps for the file are there.

 

In short, it is, IMHO, much a do about a feature that gives no practical gains. So, to refer to your words above, I do not think that "it would certainly make sense". YMMV.

 

Besides, there is no place in SpartaDOS directory entry for that flag. The only status bit, which was free, has already been allocated (status $4x = symbolic link).

 

Edited January 26 by drac030

[thorfdbg]

On 1/25/2024 at 8:06 PM, pcrow said:

RWhat happens now if you open an 8K file for reading, read 4K, open it to append, write 8K, then read to the end.  Will the read stop at the original 8K, or will it catch the update and read the 16K?                 

That is DOS dependent and depending on the open mode. On your classical Atari DOS, there is the "open for update mode", which is mode 12, and there you cannot extend the file post its current end. Thus, when an existing file is being updated, and you attempt to extend it beyond its current file end, you receive an "error 136 - EOF". You can extend such files with the open mode 9, then append to the end, but you cannot read in this mode.

 

Again, Dos 2.++ offers mode 13 which is "read, write, extend beyond its end". In this mode, you can read and write, and in case you write beyond its end, the file is extended and does not run into an EOF error. You can also "seek back", and then continue reading, but of course only up to the position where the new (most recent, updated) EOF position is located (and not where the file ended before the file was written to).

 

But, again, this is a 2.++ extension and not what your average Atari DOS would be able to do.