Problems with older MS-DOS file systems vs hard drive capacity.

[youxia]

I'm trying to update & expand my PC-XT dedicated software collection. This is primarily meant to run on MiSTer's new PCXT core (though it can be also ran on DOSbox/PCem) and for that I need to use hdd images. I've created a few using PCem, Win10 and also MiSTer's Linux, then have formated with MS-DOS 6.22. The size varies from ~500MB to 1.3GB.

 

These images work as hard drives, and are bootable and can be filled with content. But one problem persists: there's a big loss of hdd space when filling them with my collection, which mostly consists of very small files. So, eg, for the initial release (v0.1) I've used a 487MB-sized hdd image and managed to only fill it with 334 MB of stuff (~30% space loss). A bigger one I need for the expaned collection, 1GB-size, only took 442MB of files, that's merely a 43% fill rate. After that the disks report running out of space (despite their original size being correctly reported in NC or fdisk). I've also tried filling the 1GB disk with big files (like 10x100MB) and in that case was able to fill it up to ~98%.

 

I read a liitle bit about this whole topic, and how this quagmire might be related to old FAT16 (which I need to use for PCXT - I think, not sure it'd be compatible with MSDOS 7.1 which can offer FAT32 ?) cluster size limits and the resulting space loss - but even then my impression was that it should account for 20-30% max of space loss (and possibly only when going over 1GB). But ultimately this is all above my pay grade, and so I'm hoping maybe some of local MS-DOS experts might have simpler explanation, or perhaps even a solution.

[wierd_w]

This is 'slack space' inefficiency of cluster size rearing its ugly head. It manifests most strongly with very small files.

 

If you have, eg, a cluster size of 4k, and write a 512 byte file, 3.5k of space is wasted.

 

If the MisTer supports emulating a dos capable network card, I would suggest setting up DOS networking then mounting a network share.

 

Alternatively, you can 'try' using Stacker/Drivespace compressed volume files, which deal with small files more efficiently (but are hard to work with on modern OSes).

 

Keep in mind that different versions of DOS also impose different maximum fixed disk size restrictions, on top of this issue.

[+OLD CS1]

Cluster Size Godzilla wrecks the place!  Yeah, so, bottom line, a small file of, say, 1k will consume a full cluster.  On FAT16 with 4k cluster sizes, that means 1,000 1k files will consume 4MB instead of 1MB.  I cannot recall which, but on a 1GB drive you will be looking at either 16k or 32k clusters, so 1,000 1k files will consume 16MB or 32MB --  a (roughly) 16:1 or 32:1 loss!

 

1k is kind-of an extreme example, but any file up to the cluster size will eat an entire cluster.  So a file of 1 byte up to 16k will occupy a 16k cluster, &c.

 

Chances are your games and programs are not built entirely from 1k files, so the amount of loss you will experience will not be as large as 16 or 32:1, but there will be loss nonetheless.

[Krebizfan]

FAT16 has clusters sizes of 8KB for 256-512MB, 16KB for 512MB-1GB, 32KB at 1GB-2GB. FAT32 defaults to 4KB sectors at those sizes so there will still be noticeable slack. FAT32 is available for an XT class system with FreeDOS and DRDOS. The downside of FAT32 is that the FAT is considerably larger. To get good performance, the FAT needs to be in memory. That takes a little work with the 128KB maximum size for FAT16 but the FAT balloons to 2 MB with a 2GB FAT32 drive. 

 

The best choice is running a compressed drive without turning on the compression. That allows clusters to be shared by multiple small files cutting the losses to slack. Compressing and decompressing files could produce a noticeable performance hit on a XT so it is best not to use that feature. 

[youxia]

Thank you for the replies. I mainly wanted to confirm this is indeed the known slack space issue and not something I'm doing wrong perhaps.

 

As for solutions, the thing is that this collection contains all the MS-DOS games from 1981-1989 and is meant for general public, so the maximum compatibility is paramount. I've heard that FreeDOS and DRDOS might have problems with that so I'm reluctant to use them.

 

As for

14 hours ago, Krebizfan said:

running a compressed drive without turning on the compression.

...how would I go about that?

 

At the moment I think perhaps the easiest solution for me would be just to eat up the loss and use two <1GB drives. The whole collection weighs 1030MB, so I hope perhaps the 16KB clusters Krebizfan mentions would be kind to me and let it fit in 2x1GB. Having 2 mounted drives also would speed up the core boot up, which atm hangs at detecting the drive for quite a few seconds.

[The Usotsuki]

DR DOS is usually fine but it diverges from MS-DOS at the 3.3 point.  I've had some issues with FreeDOS myself, and generally don't use it.  I prefer, actually, PC DOS - which diverged at the 5.0 point but still tracked most of the MS-DOS 6.x improvements.  (And MS-DOS actually picked up Interlnk from PC DOS.)

 

BTW, you can partition a larger drive as multiple logical drives if you want.

Edited December 21, 2022 by The Usotsuki

[Krebizfan]

Turns out that MS-DOS 6.22's DRVSPACE will not operate with no compression; that was a feature added in Win95's DRVSPACE3 (possibly with Win95 Plus). The choice could be selected from the GUI control. Stacker 4 could do it if operating in Max Speed mode which stops all on-the-fly compression. The version of Stacker that shipped with PC-DOS 7 allows for this. 

[+OLD CS1]

The solution to lost space due to larger cluster sizes is smaller drives or file systems.

[wierd_w]

Stacker with compression off allows suballocation.

 

It however, gobbles down a fat chunk of memory for the resident disk handler routine.

 

It also makes working with the volume with modern tools/os very painful. Nothing modern knows how to mount or use a compressed volume file. (Even with compression off, that is still the correct term.)

 

I still think using a lan share is the real way to go, honestly.

 

[youxia]

Thanks, but I guess setting a LAN share would be too much for most users, for me even. I'm trying to keep this to be as straightforward as possible; mount the vhd-boot DOS-browse from NC. The wasted space is somewhat annoying but seeing as this is an extra ~1GB it's not really a deal breaker in the age of huge SD cards and external storage.

 

Do you have some suggestions on how to create an optimal sized HDD image for this task? For now I'm kind of groping in the dark, just typing something in the "size" box in th PCem new hdd creation GUI, but not really adjusting any other parameters. Annoyingly, I have just tried to create some ~950MB sized vhds and they all show only 503MB in fdisk. But before I was able to create bigger ones, so not sure what I'm missing.

[wierd_w]

There is a maximum fat16 size restriction under msdos of ~2gb, if I recall correctly.

 

HugeFat fat16 (made with something like pqmagic) can be larger, but will have a downright obscene cluster size.

 

For maximum compatibility, i would shoot for ~2gb fat16 volumes, with additional extended dos partitions and logical volumes as needed.

 

Hitting the '503mb' issue sounds like CHS partitioning instead of LBA partitioning. The FAT partition type ID byte is different when CHS is used.

 

Traditional CHS can only see up to about 503 mb. ECHS can see up to about 8GB. Any bigger and you need real LBA modes. LBA24 can see up to about 128gb. You need LBA48 to see very large modern disks.

 

I would investigate what kind of disk controller bios / int13 routine is in the system bios, to see if echs modes are available or not.

 

See also:

 

https://tldp.org/HOWTO/Large-Disk-HOWTO-4.html

 

Incidentally, this is exactly what xt-ide project does: provide an LBA/ECHS aware INT13 handler, because XT system bios poops out at 503mb.

 

Vintage software solutions exist, such as DiskManager from Ontrack, or EZ-Bios.

 

 

Edited December 22, 2022 by wierd_w

[youxia]

The MiSTer's core is using XTIDe bios.

 

I made a 2GB hdd, but managed to fill it with only ~670 MB worth of files. This slacking is brutal. 1GB hdd only managed to take 440MB. 

 

500 MB hdd fared a bit better, and took 340MB. But - now I can't see the GAMES directory which contains the data...

 

I only need to fit 1GB of data but this is proving to be a major headache.

[youxia]

I read this, in an offical IBM document: https://www.ibm.com/support/pages/selecting-best-microsoft-file-system-fat16-fat32-ntfs-ibm-intellistation

Quote

If FAT16 is selected, the hard disk drive will be divided into 512 byte pieces called sectors. These sectors are grouped into larger pieces called clusters. Due to design limitations, the maximum number of clusters that the FAT16 system can keep track of is 65,535. Hard disk drives larger than 1.2GB therefore become very inefficient in storing data because large amounts of hard disk drive space is unused but occupied by the file allocation table. FAT16 does provide excellent performance on small hard disk drives that are less than 1.2GB. Hard disk drives larger than 1.2GB should be formatted as FAT32. Security in a FAT16 partition is limited to the volume access restrictions under the operating system supporting data security.

If the FAT16 performance is so excellent, why my 1GB disk wastes more than half of the space when simply filled up with some pretty standard files? I don't get it. I have a feeling that I'm doing something wrong after all.

[wierd_w]

I am thinking that your disk images might not be well crafted.

(Many disk image tools meant for VMs do not create images that are.. how to say... "Vintage Friendly", and tabulate out to strange geometries.  I can create a better image that is flat using linux's dd command, with an exact size in bytes, that will play ball nicer.)

 

Let me construct, partition, and format some image files for you. 

I will use pqmagic to create the partitions to very aggressively control cluster size for you, and force it to the minimum values.

 

I will do that in a few hours once I get home.

 

[wierd_w]

OK, I hamfisted it at work.

 

Here is a flat disk image that is ~2gb in size, that has a clean CHS geometry.

 

258 Cyls

255 Heads

63 Sectors

(512 bytes per sector)

 

It is partitioned with a single FAT16 partition, with a 32k cluster size. (min size for 2gb volume!)

 

2gb_1part.img.7z

 

I will make one with extended partitions of about 512mb in size, since those can be made with 8k cluster size, but give me a bit to make that happen.

Edited December 23, 2022 by wierd_w

[wierd_w]

Ignore the previous post-- I think I malformed the image mucking with it in dosbox, because of the byzantine way it takes the "-size" directive.

 

Try this instead:

 

OK, I hamfisted it at work.

 

Here are flat disk images that are ~2gb in size, that have clean CHS geometry.

 

258 Cyls

255 Heads

63 Sectors

(512 bytes per sector)

 

It is partitioned with a single FAT16 partition, with a 32k cluster size. (min size for 2gb volume!)

2gb_1part.img.7z

 

Here is one with a 510mb primary (with 8k clusters), and a 1.5gb extended dos partition containing 2 logical volumes-- the first is 1020mb in size (16k clusters), and the second is 494mb (8k clusters)

2gb_3part.img.7z

 

And here is one with a 510mb primary (with 8k clusters), a 1.5gb extended dos partition with 3 logical volumes -- Two that are 502MB, and one that is 510MB. (all 8k clustersize)

2gb_4part.img.7z

 

 

These will have:

2GB C drive

 

---
510MB C Drive
1020MB D Drive
494MB E Drive

---
510MB C Drive
502MB D Drive
502MB E Drive
510MB F Drive

respectively.
 

Edited December 23, 2022 by wierd_w

[mr_me]

2 hours ago, youxia said:

I read this, in an offical IBM document: https://www.ibm.com/support/pages/selecting-best-microsoft-file-system-fat16-fat32-ntfs-ibm-intellistation

If the FAT16 performance is so excellent, why my 1GB disk wastes more than half of the space when simply filled up with some pretty standard files? I don't get it. I have a feeling that I'm doing something wrong after all.

Performance in terms of speed not storage efficiency.  Larger clusters means fewer units to address and a smaller allocation table.  Fat32 would give you smaller clusters but more allocation units, more storage efficiency but a larger allocation table for the computer to manage, not a problem for fast computers. 

 

You're not doing anything wrong.  If you want more storage efficiency the solution is more partitions which gives you both smaller allocation units and a smaller allocation table.

[wierd_w]

1 minute ago, mr_me said:

Performance in terms of speed not storage efficiency.  Larger clusters means fewer units to address and a smaller allocation table.  Fat32 would give you smaller clusters but more allocation units, more storage efficiency but a larger allocation table for the computer to manage, not a problem for fast computers. 

 

You're not doing anything wrong.  If you want more storage efficiency the solution is more partitions which gives you both smaller allocation units and a smaller allocation table.

Yes. That is why I made him some images that are nice an clean, that have more tractable allocation unit sizes.  If he breaks up his collection so that "Games with larger files" live on the 1020MB D drive, and "Games with lots of itty bitty files" live on the 494MB E drive (for the 3 partition sample), he should be set I think.

[youxia]

Thanks a lot @wierd_w, I will try these and report back.

[youxia]

Ok, so the first one (with one big partition) behaved like the 2GB I made - just took over 600MB. The one with 3 partitions was far more useful and I managed to squeeze the whole collection on it. But when I put MS DOS 6.22 the whole thing wouldn't boot and also stopped being recognized in PCem (in my 3 configs for PCXT, P133 with DOS and PII with Win 98, using both IDE and XTIDE bioses). Also, the Win98 setup reports it as "LBA disk", not sure if it's relevant.

 

So I can use it as container now, though would prefer to maybe make it bootable - not sure what I'm doing wrong here (I was just using standard MS DOS 6.22 disks to install from A: in PCem).

 

Any chance you could make separate 250, 500 and 1GB disks when you have a moment @wierd_w? I think maybe this would be a better solution for me. The collection is sorted by-year, that's its whole point, and I can't really break it up by size. Also, mounting separate volumes could speed the boot up in MiSTer core (there are 2 slots for hdd and it mulls over quite long when one is empty during boot).

I just made a 251MB disk and managed to squeeze years 81-86 on it, and make it bootable. But would like to try disks from somebody who knows what they are doing.

 

[wierd_w]

It would probably be better if I did this with a REAL emulator, and not DosBox.

 

Creating the image is easy, getting it properly partitioned, formatted, and bootable is another.

 

 

If you don't mind me basically pirating MS DOS 6.22, I can totes make some images for you.  Hang on, I will install virtualbox, and build flat images that way.

[wierd_w]

OK. Here is a booting version of that 3 partition image.  Has Dos 6.22 installed, nothing else.  Same partition structure.

2gb_3part.img.7z

 

You can probably just fix the image you have by running

 

FDISK /MBR

 

on it, however.  Basically, the disk image lacks a boot loader in sector 0, since it was a 0-filled image.

(I had to re-learn how to build VMDK header/descriptor files for flat images, so that I could feed it to virtualbox! LOL! Anyway, the issue seems to have been the missing MBR code. Running the magic fdisk command with that switch should fix it.)

 

Edited December 24, 2022 by wierd_w

[youxia]

Alright, thanks. I will give it a go. I think I did fdisk /mbr before but not sure.

 

I'm still after these separate 250, 500 and 1GB disks, if you can be bothered

 

 

[wierd_w]

23 hours ago, youxia said:

Alright, thanks. I will give it a go. I think I did fdisk /mbr before but not sure.

 

I'm still after these separate 250, 500 and 1GB disks, if you can be bothered

 

 

OK, I will make some.  These are not perfect sizes, because of a bug in DOS that prevents full use of 256 heads and or 1024 cylinders, for sizes. Closest matches provided, for max compatibility.
They should be bootable with minimal system DOS 6.22.  They were created with dosbox, because it is just faster than manually prodding vmdk files for virtualbox, but I did test booting them with dosbox.

 

254.75mb (267125760 bytes) disk

256mb.img.7z

Single partition, 4k cluster

512 bytes/sector
1023 cylinders
255 heads
2 sectors/track

 

510mb (536870912 bytes) disk:

512mb.img.7z

Single partition, 8k cluster

512 bytes/sector
1023 cylinders
255 heads
4 sectors/track

 

 

1019mb (1068503040 bytes) disk:

1024mb.img.7z

Single partition, 16k cluster

512 bytes/sector
1023 cylinders
255 heads
8 sectors/track