How many files can be on an SD card in a FujiNet?

[kenp]

I'm using a FujiNet 1.6.  I have a 2G microSD card.  I must admit I've loaded it up with many, many, many (literally hundreds because they are so easy to setup in advance.) .atr files mostly blanks that I intend to copy physical disks to.

 

The question seems to be how many files can the FujiNet handle on an SD card?

 

I'm also running into a problem trying to mount more than 4 drives, as it were, in MyDOS.  I was certain I had mounted up to 7 in MyDOS before but now limited to 4 and the 5th returns a Error 160.  (I don't think I had the SD card setup with as many .atr files before though so maybe I should try another SD card somewhat more sparsely populated with files.)

 

Any help appreciated.  Thanks and Take care.

 

 

[_The Doctor__]

MyDOS does D1 thru D8 and a ramdisk at D9 in the normal world, If FujiNet firmware or software changes, it's not a MyDOS issue.

[kenp]

53 minutes ago, _The Doctor__ said:

MyDOS does D1 thru D8 and a ramdisk at D9 in the normal world, If FujiNet firmware or software changes, it's not a MyDOS issue.

Ah.  Ramdisk at D9.  I was finding it placed at D8 so I kept it there.  I'll try it at D9.  The would free up that D8 slot in the FujiNet.  But only if I can get more than D1 - D4 loaded up.  I was sure MyDOS was going to D7 before (and the D8 for the Ramdisk.)  I haven't tried any upgrades to the FujiNet code yet as the 1.6 it came with seemed to be sufficient for what I need.

[kenp]

Better response.  I pruned the SD card down to about 150 files from over 500 and things are responding better.  Was able to mount .ATRs at D1-D8 an the ramdisk at D9.

 

So there is some limit on how many files the FujiNet can handle on the SD card.

[_The Doctor__]

I am starting to believe the FujiNet is eating up a crap ton of ram to handle the file system translation over a number of drives. It didn't seem to do so previously but this keeps popping up all the time now. I guess everyone is going to have to keep the logs to nail it down. FujiNet is constantly in flux firmware wise so you are going to have to do updates or downgrades to find one that works best for you until they nail all this fun stuff down.

[kenp]

4 hours ago, _The Doctor__ said:

I am starting to believe the FujiNet is eating up a crap ton of ram to handle the file system translation over a number of drives. It didn't seem to do so previously but this keeps popping up all the time now. I guess everyone is going to have to keep the logs to nail it down. FujiNet is constantly in flux firmware wise so you are going to have to do updates or downgrades to find one that works best for you until they nail all this fun stuff down.

Or, I change my behaviour a bit and not try to contain everything in the world on a single SD card for the FujiNet.   I mean I did fill the 2G card with all them blank .atr files and could likely cram even more stuff into a larger card but I've always wondered how many files can the FujiNet keep in its catalogue at the same time.  (Maybe someone will have to hack a memory expansion for the FujiNet.  🙂

 

I got the bits to make an SIO2PC (USB type) cable and will do some experiments with that and setting up a TNFS server on a machine here as well.  The TNFS serve might be easier since I don't have to worry that I've wired the connections the wrong way 'round.

 

There's lots to be done yet before I mess up doing a firmware update on the FujiNet.

 

I think I saw that there is an emulation of the FujiNet for use with Altirra under Windows so i ended up with a weird thought or putting Altirra, the Fujinet emulator and the TNFS server and all the software I might use on a single NVME drive connected via USB 3.  An entire Atari development system in a little package that could attach to any windows machine.  Probably won't go that far but it's an interesting thought.

Edited March 9, 2023 by kenp

[_The Doctor__]

I wouldn't see any problem with an SD card holding so many files, if those files are .atr when put in a slot then that's not a problem either.

[kenp]

1 hour ago, _The Doctor__ said:

I wouldn't see any problem with an SD card holding so many files, if those files are .atr when put in a slot then that's not a problem either.

Like I say, I had over 500 files on the SD card.  Hundreds of them newly formatted blanks disks, as it were, for copying things to.  Ran into the problems that made me ask the question, not being able to mount images above D4, not able to find images I knew should be there, not able to boot from images that would boot before, not able to get directories from mounted images.  I was using the web interface to try and mount images, btw. 

I cleared a few hundred .atr files from the SD card, dropped from over 500 to about 150, and problems cleared.  There's a limit that is likely documented somewhere or a set of conditions but I haven't found it.

 

The more I think on it the more I want to check out the TNFS server idea.  That would make it more likely that I would not be trying to hold hundreds and hundreds of files on the SD card.  Will the FujiNet react the same way when facing hundreds and hundreds of files on a TNFS server?  We may find out soon.

 

[_The Doctor__]

A directory in the FAT32 file system can
contain up to 65,536 entries. Each file or subdirectory in the
directory will use from one to thirteen entries, depending on the
length of the file name.

Because many people use long file names today it's usually about half that size

so 32,000 to 16,000 isn't abnormal for a directory before there are issues.

400 or 500 usually doesn't make most things blink.

So the limit is unknown, I don't see it anywhere I looked.

[+DjayBee]

Why don't you use subdirectories? 

Browsing through hundreds of files in search of the "one" is a major PITx. 

[_The Doctor__]

I think that's what he did. cut down to 150?

[kenp]

10 hours ago, DjayBee said:

Why don't you use subdirectories? 

Browsing through hundreds of files in search of the "one" is a major PITx. 

They were in subdirectories.  There was a subdirectory for 92K blank atrs, a subdirectory for blank 360K atrs and subdirectories for the atrs that would store copies from the green, yellow or red dotted floppies in their most likely formatted sizes. 

( I gave the floppies in my libraries a number and a colour dot to try and keep things organized.  Of course, after almost 40 years I had to figure out what each colour was for. 🙂  AND almost half of them are contaminated with some sort of mould that will have to be cleaned before I can try to recover whatever was on them.)

The quick reduction in the number of atr files was simply by deleting a few directories, each with a hundred blank atr files, from the SD card.  As mentioned most were still just blank images so no real loss and I had a backup SD card with more copies of the blank atr files.

But it wasn't a question about how many files one can store on a SD card.  The question was meant to be how many files can the FujiNet keep work with in its catalogue at one time.  I'll likely need to get more familiar with the GIT repository system and start working these things out from the sources.

[kenp]

1 hour ago, _The Doctor__ said:

I think that's what he did. cut down to 150?

And they are still organized in directories.  I'm a big believer in directories and trying to keep things organized.

A directory for DOS boot images, a directory for utilities, a directory for games that I want to retry, and still a couple of directories for particular sized blank atrs to copy things to.  I'll have to keep track now as I do this and rename them to the correct colour and number as they get copied.

Although, doing this through a TNFS server may make more sense than copying everything to an SD card then having to take it out and back it up to a PC server as a separate step.  And the wi-fi connection from the FujiNet to the server is faster than the SIO connection anyway so no real loss in speed for the extra link.  Lots of things to try yet, it seems.

Edited March 9, 2023 by kenp
spelling counts

[+DjayBee]

4 hours ago, kenp said:

They were in subdirectories. 

OK, I had expected that it loads only the current directory into memory. 

[tschak909]

On 3/9/2023 at 3:54 PM, DjayBee said:

OK, I had expected that it loads only the current directory into memory. 

Need help. I can't debug it all.

 

If someone wants to experiment, grab the current source code tree, and e.g. experiment with main task size in main.cpp:

/*
 * This is the start/entry point for an ESP-IDF program (must use "C" linkage)
 */
extern "C"
{
    void app_main()
    {
        // cppcheck-suppress "unusedFunction"
        // Call our setup routine
        main_setup();

// Create a new high-priority task to handle the main loop
// This is assigned to CPU1; the WiFi task ends up on CPU0
#define MAIN_STACKSIZE 32768
#define MAIN_PRIORITY 20
#define MAIN_CPUAFFINITY 1
        xTaskCreatePinnedToCore(fn_service_loop, "fnLoop",
                                MAIN_STACKSIZE, nullptr, MAIN_PRIORITY, nullptr, MAIN_CPUAFFINITY);

        // Now that our main service is running, try connecting to WiFi or BlueTooth
        if (Config.get_bt_status())
        {
#ifdef BLUETOOTH_SUPPORT
            // Start SIO2BT mode if we were in it last shutdown
            fnLedManager.set(eLed::LED_BT, true); // BT LED ON
            fnBtManager.start();
#endif
        }
        else if (Config.get_wifi_enabled())
        {
            // Set up the WiFi adapter if enabled in config
            fnWiFi.start();
            // Go ahead and try reconnecting to WiFi
            fnWiFi.connect();
        }

        // Sit here twiddling our thumbs
        while (true)
            vTaskDelay(9000 / portTICK_PERIOD_MS);
    }
}

 

[tschak909]

I have made a change to dir_close(), which reclaims the memory used by the cache vector.

https://github.com/FujiNetWIFI/fujinet-platformio/commit/25bc9c4e7706e7dfc6796b52ec15f965b9098041

 

but...

 

This is probably not sufficient to solve the problem.

 

I can not crank a production build each time we are trying to work through a bug, 

 

those of you who are running into this, have the reproducible test case. I don't.

 

so if you can help by setting up a dev environment to AT LEAST HIT UPLOAD AND MONITOR, after following these instructions:

https://github.com/FujiNetWIFI/fujinet-platformio/wiki/Board-Bring-Up-Software

 

Please do. IF YOU WANT THIS BUG FIXED, THEN PLEASE HELP!

 

-Thom

Edited March 13, 2023 by tschak909