New (alt) BIOS for Ultimate 1MB/Incognito

[flashjazzcat]

So far, in my limited testing, everything's working okay! Excellent work! I hope to have more time to play in the next couple days...

Thanks for testing. That's good to hear.

 

I've had a day of pottering and playing with the new Loader and I also have found no significant faults at all. Really excellent work FJC - not just it but the whole BIOS and PBI improvements also!!!

Thanks. Insignificant faults include the fact that enabling booting to the loader in the BIOS doesn't deactivate SDX (the two options are supposed to be mutually exclusive, as per Ultimate). This is purely cosmetic, though, and doesn't affect functionality. There are probably other minor issues with changes to the menus not being accurately propagated over to the Incognito build yet (the Incognito menus for the two different hardware types are a pair of linked lists both of whose tails point to the higher menu items common to both. So, when we change from 800 to XL/XE, the head pointers are swapped over).

 

Before you started working on this latest update you did mention that one or two of the existing bugs that were present in 'MATR' would need to be addressed as the new loader would use the same code base. At some point - after you've had a very well deserved rest of course! - are you planning to reintegrate these fixes in to the parent application? I personally really like MATR now and still would probably use it in preference to the Loader if a new version were to appear, even without the putative R/W features mentioned in the TODO text.

Yes, that's what I said but there's now barely any code from MATR in the new loader, which is why it took so long. MATR's structure was totally unsuited to presenting lots of different lists of information in multiple menus, and the disk handling code underwent a complete re-write. So MATR doesn't really draw any immediate benefit from this. That said, I can appreciate the usefulness of the tool and I'll try to fix it up at some point in the future. It's useful to be able to perform mounting operations without rebooting the OS, although ideally the mounting tool would be right in the PBI ROM so you could switch volumes without even disturbing the underlying application. Sadly there's less than 1KB of PBI RAM which isn't adequate for the kind of firmware-based tool I'd want to present. Nevertheless, Ultimate/SIDE/Incognito is still - AFAIK - the only in-situ PBI solution which handles read/write ATRs right inside FAT partitions, so hopefully the inconvenience of having to reboot to the SIDE loader when reassigning mounts is offset a little.

 

I haven't noticed any issues so far, and the SIDE Loader seems quite a bit faster.

Numen is a good benchmark of the loader if you haven't tried it yet. It spends much longer unpacking in RAM than it does loading from disk. I timed uFlash (run direct from the loader) reading a 512KB ROM, and it read the whole file in 9 seconds, which is just about commensurate with SpartaDOS X reading from the HDD.

 

There are two reasons behind the speed: burst reads, and not testing DRQ for every byte read. Of course this will throw up errors on unstable hardware, so (as mentioned in the readme), I'll add the option of a slower method with retry if need be.

Edited December 15, 2015 by flashjazzcat

[spookt]

Flashed to my Incognito and rebooted. Very nice!

 

One quick question - the option 'ESC Quit' is shown at the foot of each screen but the machine doesn't seem to respond to pressing ESC. Should it?

 

Many, many thanks for your continued work Jon.

[flashjazzcat]

Good stuff.

 

One quick question - the option 'ESC Quit' is shown at the foot of each screen but the machine doesn't seem to respond to pressing ESC. Should it?

I noticed this the other night after uploading the archive, but well spotted. The instruction is a left-over from the main BIOS and needs removing.

[+Larry]

I haven't done very well in keeping up with this thread, but I briefly went through it this AM, and the latest beta for the U1MB appears to be 0.35. Is that correct?

 

Thanks,

Larry

[Roydea6]

I haven't done very well in keeping up with this thread, but I briefly went through it this AM, and the latest beta for the U1MB appears to be 0.35. Is that correct?

 

Thanks,

Larry

Actually the latest is 0.45 Just up some posts before this one... at #330..

[+Larry]

Got it. Thanks, Roy!

 

-Larry

[fujidude]

I noticed a small issue. When I'm using the machine and attempt to enter the setup program via holding help and pressing reset, it doesn't work. I am using Altirra 2.70t40. It worked perfectly with 0.35 setup, 1.4 PBI, and the older loader. All other conditions are exactly the same. Since I'm using a 65C816 @ 21MHz (and I thought you might get a little excited about that), I switched to standard 6502 @ 1.79MHz. There was no difference. This is not a show stopper problem for me at all, but thought you would appreciate knowing.

 

PS: Yes, I'm using RAW keys in Altirra.

[flashjazzcat]

Since the latest Ultimate BIOS and loader were developed using the latest versions of Altirra, and since I can consistently enter the BIOS setup screen with Help+Reset, I have no suggestions at all other than ensuring the Help key is actually selected as the setup entry hotkey in Advanced Settings. It might be set to the Start key.

[fujidude]

Nah, it is set to the HELP key. Why me... uhg.

 

I'll poke around a little more though and get back.

Edited December 15, 2015 by fujidude

[fujidude]

Okay... I'm back And I have a narrowed down case. It isn't actually a HELP <-> RESET issue, but rather it is the RESET that doesn't work right. What I have discovered, is that if two conditions are met simultaneously, the problem shows up:

1. PBI BIOS is enabled in the U1MB BIOS SETUP.
2. A cartridge is attached to the machine. I had the Action! 3.6 supercart "plugged" in.

If only one or none of the conditions is met, there is no RESET problem.

 

FJC: I'll send you a PM with a link to a self contained Altirra setup that produces this problem.

[flashjazzcat]

Okay... I'm back And I have a narrowed down case. It isn't actually a HELP <-> RESET issue, but rather it is the RESET that doesn't work right. What I have discovered, is that if two conditions are met simultaneously, the problem shows up:

• PBI BIOS is enabled in the U1MB BIOS SETUP.
• A cartridge is attached to the machine. I had the Action! 3.6 supercart "plugged" in.

If only one or none of the conditions is met, there is no RESET problem.

 

I checked out the setup you sent over and - typically - the issue was caused by a "new feature", namely rebooting to the loader by holding the SIDE button and pressing reset. This was a requested feature but presumably wasn't tested with cartridges yet. The main BIOS was polling the SIDE button and then attempting to clear the latch, and it will never clear while a cartridge other than SIDE is attached. Normally the advice would be to disable the PBI entirely when you want to use a cart other than SIDE, but of course we have the High-Speed SIO now.

 

Anyway: this appears to fix it:

 

ulbios46.zip

 

Flash this to the Ultimate main BIOS slot. I also reorganized the PBI menu a little, since the ATR swap button is dependent on the HDD being enabled, and the former is now greyed out (and forcibly disabled in the hardware register) when the HDD is off. These changes also address the other issue of the external cart disappearing even if the HDD was turned off but the button was left enabled.

 

I briefly tested this build with an Action! cart on real hardware and everything appears to work. Ideally, you wouldn't even have to remember to disable the HDD to use a different cart, but the whole cart handling thing - at the BIOS level - is rather delicate, since we can easily upset carts via careless writes to $D5xx. Thus, by the time the PBI BIOS - aware that the HDD is switched on - has probed for the SIDE cart, some flash carts may already be rendered unusable, so it's important to turn the HDD off in the BIOS.

 

If I haven't broken anything else, I'll upload an Incognito version in a day or two.

Edited December 16, 2015 by flashjazzcat

[morelenmir]

Yes, that's what I said but there's now barely any code from MATR in the new loader, which is why it took so long. MATR's structure was totally unsuited to presenting lots of different lists of information in multiple menus, and the disk handling code underwent a complete re-write. So MATR doesn't really draw any immediate benefit from this. That said, I can appreciate the usefulness of the tool and I'll try to fix it up at some point in the future. It's useful to be able to perform mounting operations without rebooting the OS, although ideally the mounting tool would be right in the PBI ROM so you could switch volumes without even disturbing the underlying application. Sadly there's less than 1KB of PBI RAM which isn't adequate for the kind of firmware-based tool I'd want to present. Nevertheless, Ultimate/SIDE/Incognito is still - AFAIK - the only in-situ PBI solution which handles read/write ATRs right inside FAT partitions, so hopefully the inconvenience of having to reboot to the SIDE loader when reassigning mounts is offset a little.

 

Is it fair to assume you are pretty much reaching the edges of the hardware capability offered by the current U1MB hardware?

 

I'll prefix my next paragraph by an emphatic assurance I am neither criticizing the existing design, not trying to push for anything new. I am talking in purely blue-sky terms. So... In an ideal world the functionality I would like in the system is that *.ATR's could be created, formatted, written to and read from while stored somewhere on the SIDE2 CF card that was also accessible directly when the CF was plugged in to a PC. On the Atari these functions would be controlled from a software application that runs in SDX without leaving the CLI - like MATR. In essence you would have something that offers many of the same features of an SIO2SD - or better yet the SIO2USB and its well-reviewed software front-end - except integrated on the existing hardware and perhaps operating at the speeds that connection allows. I genuinely cannot begin to imagine the amount of work required for that setup, obviously - otherwise I would not be talking about it!!! However, does the current version of the U1MB/SIDE2 hardware even allow this kind of functionality? If not, what extra features would they need? Is it perhaps something the Rapidus, with its faster processor and extra RAM/SRAM might make possible?

[flashjazzcat]

I shouldn't imagine there'd be much extra work involved since - apart from the in-situ creation of ATRs (immediately remedied should I add write-capability to the mini FMS, or once SDX has a write-capable FAT driver: whichever comes first), you can already do exactly what you describe. Don't forget the existing command-line mounting tool (ATRMNT) for Ultimate which requires the SDX FATFS.SYS driver. The existing SIDE loader provides (hopefully) a decent front-end, full ATR mounting/unmounting capabilities, plus hard disk partition mounting facilities which SIO2SD/USB do not permit (since they're not PBI devices), and all operating at the maximum possible IO speeds. Creating a new ATR inside the loader seems to me to be the only thing not yet accomplished.

 

And while I wouldn't presume that any limits have been reached by the current firmware aside from the limits of my psychological endurance, upgrades like Rapidus will make everything not hampered by slow RAM/ROM work much, much faster. Aside from that, my wish list would be 64KB of PBI SRAM, 32KB of PBI ROM, and a 32KB bank-switched main BIOS, but given that we have 8KB of PBI ROM, less than 1KB of PBI RAM and 16KB of main BIOS, we must simply do the best we can with what's there.

Edited December 16, 2015 by flashjazzcat

[fujidude]

FJC... the 0.46 BIOS seems to have cleared it up for my setup as well. Yes, new features always seem to come with their work cut out for them don't they?

[morelenmir]

I shouldn't imagine there'd be much extra work involved since - apart from the in-situ creation of ATRs (immediately remedied should I add write-capability to the mini FMS, or once SDX has a write-capable FAT driver: whichever comes first), you can already do exactly what you describe. Don't forget the existing command-line mounting tool (ATRMNT) for Ultimate which requires the SDX FATFS.SYS driver. The existing SIDE loader provides (hopefully) a decent front-end, full ATR mounting/unmounting capabilities, plus hard disk partition mounting facilities which SIO2SD/USB do not permit (since they're not PBI devices), and all operating at the maximum possible IO speeds. Creating a new ATR inside the loader seems to me to be the only thing not yet accomplished.

 

And while I wouldn't presume that any limits have been reached by the current firmware aside from the limits of my psychological endurance, upgrades like Rapidus will make everything not hampered by slow RAM/ROM work much, much faster. Aside from that, my wish list would be 64KB of PBI SRAM, 32KB of PBI ROM, and a 32KB bank-switched main BIOS, but given that we have 8KB of PBI ROM, less than 1KB of PBI RAM and 16KB of main BIOS, we must simply do the best we can with what's there.

 

That is very encouraging FJC! I have ground away at software projects of my own in the past, although not obviously in the Atari domain. Even something as simple-seeming as a perl script to reformat ebooks in to consistent and readable HTML is surprisingly demanding and iterative, so I can totally understand what you mean about psychological endurance, There is definitely that moment where you just want to get the thing done with - and I would say you have gone a long way beyond that with the excellent work here! This becomes even more apparent given the grueling nature of Atari ASM when compared to the massively forgiving style of modern languages. Again; a huge, huge achievement.

 

In regards a writable FAT driver for SDX... I had it out with one of the chaps who programme the SDX project some time ago and is responsible for that piece of software, I think drac030?. Apparently a FAT32, therefore long-filename compatible driver is impossible to mate with the underlying 16bit cluster numbering of the SDX filesystem. A writable FAT16 driver is certainly possible, but then you run in to problems with naming. Its a bit of a catch 22. I'm pretty sure there is a liscencing issue with FAT32 as well - in fact I think you told me that!

[flashjazzcat]

Apparently a FAT32, therefore long-filename compatible driver is impossible to mate with the underlying 16bit cluster numbering of the SDX filesystem. A writable FAT16 driver is certainly possible, but then you run in to problems with naming. Its a bit of a catch 22. I'm pretty sure there is a liscencing issue with FAT32 as well - in fact I think you told me that!

Quite: there is barely a single piece of A8 software in existence which would go near long FAT filenames. I even rejected them for the Graphical OS (for which all applications will be written with the file system specs in mind) purely because of the massive buffer requirements. Not that every filename would necessarily be 255 characters long, but all buffers would have to cater of such filename lengths, and then we have to add pathnames into that. But an 8+3 driver would be perfectly usable for an application which just wanted to create - say - a 90KB file "disk1.atr" on a FAT16 volume. That would show up just fine in the loader too.

 

If long filename support was required, the loader or command line tool could just write to the FAT directly, but this seems like a lot of coding effort for a rather niche requirement when you can just stick the card in the PC's card reader and create a bunch of blank images. LFNs are kludgy too, IMO, although it's economical and convenient to support them read-only in the FAT loader.

 

The licensing issues regarding LFN came to the fore a while ago when Microsoft took legal action against TomTom with, I think, specific reference to sections of the patent concerning the simultaneous creation of LFNs and corresponding 8+3 aliases. This had implications for the Linux OS, and I read somewhere that the FAT driver was eventually re-written to drop support for the 8+3 alias (replacing it with garbage, and thus breaking older DOS versions), thereby getting around the patent.

[morelenmir]

Quite: there is barely a single piece of A8 software in existence which would go near long FAT filenames. I even rejected them for the Graphical OS (for which all applications will be written with the file system specs in mind) purely because of the massive buffer requirements. Not that every filename would necessarily be 255 characters long, but all buffers would have to cater of such filename lengths, and then we have to add pathnames into that. But an 8+3 driver would be perfectly usable for an application which just wanted to create - say - a 90KB file "disk1.atr" on a FAT16 volume. That would show up just fine in the loader too.

 

If long filename support was required, the loader or command line tool could just write to the FAT directly, but this seems like a lot of coding effort for a rather niche requirement when you can just stick the card in the PC's card reader and create a bunch of blank images. LFNs are kludgy too, IMO, although it's economical and convenient to support them read-only in the FAT loader.

 

The licensing issues regarding LFN came to the fore a while ago when Microsoft took legal action against TomTom with, I think, specific reference to sections of the patent concerning the simultaneous creation of LFNs and corresponding 8+3 aliases. This had implications for the Linux OS, and I read somewhere that the FAT driver was eventually re-written to drop support for the 8+3 alias (replacing it with garbage, and thus breaking older DOS versions), thereby getting around the patent.

 

I don't know... I'd argue that LFN's have their place and are certainly a very comfortable feature. You make an excellent point about their implementation in FAT32 though - it is a total kludge, just part of the tower of compromises that came with Windows 95. Even back when the OS was brand new I preferred OS2 Warp and had a drawn out argument with a PC shop who refused to let me buy a copy from them instead of the - subsidized - Microsoft offering. The actual coding concerns are interesting also - I had not thought about the increased memory requirements. That factor is persuasive in itself.

 

As you mention though - when you are using the loader to run games for instance I think you absolutely have to have LFN's. The many titles would become unmanageable at 8.3 - indeed I found them exactly that when I tried to get around the 'one active CF partition' problem in windows while still wanting FATFS.SYS access in SDX through using a FAT16 SIDELoader partition. It was impossible.

[fujidude]

There once was a time when patent and copyrights were limited to a reasonable period of time. The idea was to encourage innovation by protecting profit for a time, but then also to allow the innovation to benefit society as a whole, either directly or as a basis for further innovation. Sadly, profit interests have (somehow, I can't imagine how) influenced our law makers to keep extending the period of protecting profit.

 

Such a shame.

[flashjazzcat]

Note that the LFN implementation is also present in FAT12 and FAT16, and works the same way. I certainly wouldn't argue with the usefulness or convenience of LFNs, but they're difficult to manage in a file system driver for an 8-bit machine with limited resources, and the primary issue is that all existing software expects file specs of no more than fifteen or so characters, and even SDX-aware software can safely assume a maximum path length of 64 characters. So it's pretty much pointless supporting them in a file system driver designed for use with legacy software, unless said software is somehow to be patched to cater for (much) longer pathnames.

 

Increasingly off-topic, but even using 8+3 names, FAT itself presents some challenges when it comes to sector allocation and even free space calculation. While FAT32 keeps useful information like the free cluster count (and the most recently allocated cluster number, etc) in the File System Information Sector, FAT16 and FAT12 don't, and therefore require a complete iteration of the file allocation table on first access in order to establish the number of free clusters (that's what the long pause is the first time you access a volume using FATFS.SYS). Deducing the FAT width from the BPB is a reminder of what a kludge the whole thing is. Even deducing the volume name is a two-step process, since this may exist in the boot sector or in the root directory. But the appeal of FAT is simply that it is ubiquitous and universally supported, and if you design an 8-bit OS from the ground up with 32-bit cluster addressing, you're covered for huge partition support.

[morelenmir]

Note that the LFN implementation is also present in FAT12 and FAT16, and works the same way. I certainly wouldn't argue with the usefulness or convenience of LFNs, but they're difficult to manage in a file system driver for an 8-bit machine with limited resources, and the primary issue is that all existing software expects file specs of no more than fifteen or so characters, and even SDX-aware software can safely assume a maximum path length of 64 characters. So it's pretty much pointless supporting them in a file system driver designed for use with legacy software, unless said software is somehow to be patched to cater for (much) longer pathnames.

 

Increasingly off-topic, but even using 8+3 names, FAT itself presents some challenges when it comes to sector allocation and even free space calculation. While FAT32 keeps useful information like the free cluster count (and the most recently allocated cluster number, etc) in the File System Information Sector, FAT16 and FAT12 don't, and therefore require a complete iteration of the file allocation table on first access in order to establish the number of free clusters (that's what the long pause is the first time you access a volume using FATFS.SYS). Deducing the FAT width from the BPB is a reminder of what a kludge the whole thing is. Even deducing the volume name is a two-step process, since this may exist in the boot sector or in the root directory. But the appeal of FAT is simply that it is ubiquitous and universally supported, and if you design an 8-bit OS from the ground up with 32-bit cluster addressing, you're covered for huge partition support.

 

I would be content if only the Loader - or whichever piece of software creates/manipulates disk images - was LFN aware. The usage I envisage is a FAT32 partition for SIDELoader use; largely RO, but with RW for floppy images and then a second FAT16 partition that is fully RW and visible to SDX to facilitate the transfer of files between PC and A8. Of course that encounters the '1 active partition on removable media' issue... I really must pick up an 'Industrial' CF card, but they are so expensive!

[fujidude]

 

and visible to SDX to facilitate the transfer of files between PC and A8.

 

You can use a PCLink device to facilitate file transfers between A8 and PC. Also, you can mount FAT based .VHD files in Windows and move files in and out.

[flashjazzcat]

I would be content if only the Loader - or whichever piece of software creates/manipulates disk images - was LFN aware. The usage I envisage is a FAT32 partition for SIDELoader use; largely RO, but with RW for floppy images and then a second FAT16 partition that is fully RW and visible to SDX to facilitate the transfer of files between PC and A8. Of course that encounters the '1 active partition on removable media' issue... I really must pick up an 'Industrial' CF card, but they are so expensive!

1. LFN-aware read-only loader: done.

2. Multiple FAT32/FAT16 partitions (recognized by loader): done.

3. FAT16 partition visible to SDX: done.

 

The only thing missing from the existing tools is a R/W SDX driver, which I guess is imminent anyway. The silly partition limitations of Windows can be overcome in various ways, not least by using Linux/OS X. I used my Hackintosh a lot during testing. One saving grace is that multiple partitions on VHD mirrors of CF cards show up just fine in Windows.

[morelenmir]

1. LFN-aware read-only loader: done.

2. Multiple FAT32/FAT16 partitions (recognized by loader): done.

3. FAT16 partition visible to SDX: done.

 

The only thing missing from the existing tools is a R/W SDX driver, which I guess is imminent anyway. The silly partition limitations of Windows can be overcome in various ways, not least by using Linux/OS X. I used my Hackintosh a lot during testing. One saving grace is that multiple partitions on VHD mirrors of CF cards show up just fine in Windows.

I did not realize that *.ATR images that had been mounted in the loader were themselves writable from inside SDX! I assumed the contents were RO. Excellent!

 

I did not express myself very well above. What I meant to say was; would it be easier to make the Loader able to RW the partition that is being used for *.ATR images - which practically in my case at least would be the same I was using for the RO 'games' loading, therefore FAT32 - than writing a driver for all of SDX to be aware of that partition? Really all the Loader would need to do is write when it created a new blank disk image or deleted an old one. I am not sure if that intention was in the TODO list or not.

[fujidude]

New problem with 0.46: I can no longer successfully use UFLASH as it cannot find/identify the U1MB. I would get that same error with 0.45 (and ealier) until I (temporarily) toggled "ATR Swap button" to enabled, and toggled "Hard disk" to disabled. With 0.46 these options have been changed so that I can no longer set them that way, and I presume that is why I get the issue.

 

I found this out as i was about to update my U1MB with the latest Altirra BASIC (0.48) in one of the slots.

 

Please let me know if you need a demo envoronment to see it in action.

Edited December 16, 2015 by fujidude

[flashjazzcat]

Hard disk enabled, button enabled should do it: nothing's changed in that respect. This will obscure the external cartridge.