reading a file in BASIC

[AlexJ]

Preamble: I like programming in BASIC and I love learning (and exploiting) the differences between 8-bit and 16-bit computers. I've found it pretty easy to port simple games I've written from one platform to another but the most unique thing I've encountered is using files. Every platform seems to take a different approach and while they all seem to have OPEN # and INPUT # they all work differently and have different parameters. I am coding with Extended BASIC on the classic99 emulator.

 

Problem statement: I'm porting a simple game of mine that has a data file which is always read sequentially. I've tried on my own and asked for help on Facebook but I'm unable to get the file access to work. My file contains a stream of data bytes and I need to access them one at a time (but it's okay to load a chunk into a buffer and reference that). All values are possible (0-255) and null bytes (>00) are frequently used.

 

Things I've tried: I started with OPEN #2:"DSK2.DATAFILE",SEQUENTIAL,DISPLAY,INPUT,FIXED 1 and INPUT #2: DD$ to read one byte at a time. But when it reaches a null byte in the file it gets stuck there: LEN(DD$) is 0 instead of 1 (I expect it to contain CHR$(0)) and subsequent reads repeat this forever instead of reading the byte after the null. It is also very slow to read 43 bytes this way. I tried FIXED 43 to read a chunk at a time but found I would have to recreate the data file: I get an error opening a file created with FIXED 1. I'm not sure it would handle nulls properly so I haven't tried. I also tried INTERNAL but reading a string variable causes an error. I'm not sure reading a numeric variable will give what I want since TI uses a special floating point format internally. I also tried DISPLAY and reading a numeric variable. I don't recall the result but it was not working either.

 

What I intend to do with the data: Read either 1 byte at a time or a chunk of 43 bytes and process each one sequentially, splitting them into two 4-bit nybbles each (the game only uses values 0-15). I would prefer to maintain the same data file format as I used in other 8-bit platforms because I already have it created and copying it into the classic99 virtual disk was easy. But I can manipulate it into another format if necessary, with some effort. It's about 16KB of data and I'd likely write a program on my PC to read, process, and write the new datafile. Speed is not critical but if it doesn't require radical rearchitecting of my file I prefer suggestions of what would be faster (fastest) on TI. The reason I have this encoding instead of just a 32KB data file is because I originally wrote the game on C64 which has the slowest drive access and it was the same speed to read 43 bytes and decode them as it was to read 86 bytes. I don't know if TI will be slower with decoding or reading twice as many bytes, but for now I'm trying to use the data file exactly as is.

 

Thanks for any help you can give me.

[HOME AUTOMATION]

Some info here...

 

[AlexJ]

I have read that thread now. It wasn't helpful in my case but I learned some things.

 

When using DISPLAY everything is output as ASCII but special characters outside the printable ASCII range may break the file and you can't read it back.

When using INTERNAL everything is in binary and variables are stored in their radix-100 floating point representation.

 

Only I don't want either. DISPLAY seems close to what I want but it can't handle nulls so I can't use that without using a different file encoding. INTERNAL stores FP numbers and I only want small integers so a lot of disk is wasted. So does it mean there is no way I can use the same data file I use on TRS-80, Atari, Commodore, and DOS? It seems like it to me. If so I can change the encoding. Instead of merging two numbers 0-15 into 1 byte for a usable range of 0-255 and a 16KB file I could just store one number per byte by adding 48 to it for a usable range of 48-63 and a 32KB file. I could also then use a field length of 86 but I don't know if TI would waste disk space (remember the data is sequential, I don't need random access; Commodore random data would waste space but sequential data doesn't...not aware how TI does sequential files with fixed record length).

 

Given a data set of 15,050 numbers of 0-15 (grouped either 1 or 86 numbers at a time) that will always be read in order how would you arrange the data in a file and what parameters would you use in the OPEN # statement to do it?

Edited December 9, 2023 by AlexJ

[+Lee Stewart]

Pretty much the only way to access contiguous data (with no waste of space) in a file is to use DF128 format (DISPLAY, FIXED 128). For your 15050-nybble file, you will use 7552 bytes to store it, wasting only the last 27 bytes of file space.

 

When you want to read such data back in XB to work with the numbers, you can read it into a string with LINPUT. This gives you a 128-byte string record, with 256 hexadecimal digits to process.

 

You can parse the string with SEG$ to get the next “character”, say A$, containing the next pair of your numbers. You can get the numeric value of the number pair with ASC. You can pick off the numbers with

100 X1 = INT(ASC(A$)/16)
200 X2 = ASC(A$)-X1*16

 

[ Edit:  Line 100 was missing INT ]

 

...lee

Edited December 10, 2023 by Lee Stewart
CORRECTION

[+Lee Stewart]

6 hours ago, AlexJ said:

Given a data set of 15,050 numbers of 0-15 (grouped either 1 or 86 numbers at a time) that will always be read in order how would you arrange the data in a file and what parameters would you use in the OPEN # statement to do it?

 

The file should contain your 15050 hexadecimal digits, with no gaps. Here is XB code that will open the file, read a record, loop through the record 128 times, and pick off 2 hex digits each time:

100 OPEN #2:"DSK2.DATAFILE",SEQUENTIAL,DISPLAY,INPUT,FIXED 128

200 REM Read next 256-nybble (128-byte) record...
210 LINPUT #2:A$           ! get record

300 REM Get next 256 hex numbers
310 FOR P=1 TO 128         ! loop through record

320 REM Get next pair of numbers from A$...
330 Y=ASC(SEG$(A$,P,1))    ! get byte as a value
340 X1=INT(Y/16)           ! pick off left nybble
350 X2=Y-X1*16             ! pick off right nybble

360 REM Do something with the 2 hex digits extracted

370 NEXT P

 

...lee

[SteveB]

If you work with TI Data Files in DIS, INT, VAR or FIX format, you may use TiCodEd to create/edit them. You find the feature under Util / Edit TIFILES. 

 

Steve

 

[+FarmerPotato]

On 12/9/2023 at 4:12 PM, AlexJ said:

If so I can change the encoding. Instead of merging two numbers 0-15 into 1 byte for a usable range of 0-255 and a 16KB file I could just store one number per byte by adding 48 to it for a usable range of 48-63 and a 32KB file. I could also then use a field length of 86 but I don't know if TI would waste disk space (remember the data is sequential, I don't need random access; Commodore random data would waste space but sequential data doesn't...not aware how TI does sequential files with fixed record length).

 

Given a data set of 15,050 numbers of 0-15 (grouped either 1 or 86 numbers at a time) that will always be read in order how would you arrange the data in a file and what parameters would you use in the OPEN # statement to do it?

 

I think Lee gave you the most efficient packing.

 

Here is more background to your questions. 

 

TI (in all operating systems) packs records into a logical sector before writing it.  It does not split records between sectors. (certainly not on the 99/4A.) So there can be wasted space in each sector.  On the TI-99/4A the logical sector size is 256.

 

For VARIABLE records, there is a length byte followed by the characters (up to the declared logical record length.) VARIABLE records are packed until there is no room, then there is a byte marking end of sector.

 

For FIXED records, there is no length byte.  You can fit N = 256/L records in one sector.  With FIXED, the packing is the same whether writing SEQUENTIAL or RELATIVE.

 

So 43 is awkward!  With 2 * 86 you use 172, leave 84 unused.

 

Each file has a 256-byte sector buffer in VDP memory.  So the disk access speed isn't going to vary.  The overhead of BASIC  means you should prefer to move a longer FIXED record into one string.   (In my experience I got the most throughput by joining strings together to fill FIXED 80 records, rather than one line per VARIABLE record.)

 

DISPLAY converts variables to ASCII , numbers or strings. (After all, it's a PRINT statement.)

INTERNAL as you found, writes the binary floating point format, or the internal coding for the string (I forget how.)

In plain TI BASIC, the INPUT statement will interpret commas in the DISPLAY format, which will ruin your day unless ... heck, just avoid 34 and 44.  You are good using 48-63.  Or as Lee said, Extended Basic has LINPUT which ignores special characters.

 

 

 

[AlexJ]

Thanks all. I had missed the existence of LINPUT. When using that, it is reading the file properly, including null characters that get transfered correctly into the target string.

[apersson850]

On 12/11/2023 at 4:28 AM, FarmerPotato said:

TI (in all operating systems) packs records into a logical sector before writing it.  It does not split records between sectors. (certainly not on the 99/4A.) So there can be wasted space in each sector.  On the TI-99/4A the logical sector size is 256.

Not all operating systems. The p-system has its own file handling procedures, which aren't at all what the native operating system uses.

But that's of course a moot point when you work with BASIC.

[+FarmerPotato]

10 hours ago, apersson850 said:

Not all operating systems. The p-system has its own file handling procedures, which aren't at all what the native operating system uses.

But that's of course a moot point when you work with BASIC.

I should have said "in all TI operating systems that influenced the 99/4A DSR" meaning DX10 and other 990 operating systems.  

 

 

 

[apersson850]

Nah, it's just me nit-picking. I can add that the logical sector size (they call it block) in the p-system is 512 bytes. Thus it's two physical sectors, since the physical sector size for the 99/4A is the same as the logical sector in the normal operating system. Neither disk directory structure nor file formats are the same in the p-system.