EEPROM homebrew standard.. ideas

[LinkoVitch]

OK there has been a heated debate about this. I have not looked at how they work or either of the standards that have been put forward, but I deffinately agree that some standard that makes everyone happy and is safe would be nice.. So here are a few ideas to get the ball rolling.. not sure how feasible they are, I am sure someone will tell me.

 

1) To protect against power-off during a write, split the chip into sectors of an agreed minimum size (say 64 bytes for arguments sake). Now to improve the chance of this data being happy even after turning the machine off during a write, a sector can only be written to one at a time. So if power is lost only one sector should suffer.

 

2) To enhance the stability and protect data space could be wasted on the chip providing reduntant data, similar to RAID arrays. Either a duplicate copy of each sector halfing the size of the chip. Or perhaps treat the chip as 3 or 4 virtual units and use a RAID4 style approach across these wasting only a third or a quarter of the space. Thus if a sector is wiped out the redundant data can be used to rebuild that sector in it's last known good state. Or with a mirroring approach, it could simply be copied from the backup version (the one with a valid CRC).

 

As I said I have no idea how the actual writing of the chip takes place, these are just abstract ideas to try and get the ball rolling. I am interested in how the EEPROM stuff works, so if someone wants to give a quick heads up I'm willing to listen.

[Gunstar]

Isn't this academic? Aren't there ALREADY two (or three)seperate standards for EEPROM saves between Protector:SE, BSG and the B&C bypass cart (assuming it HAS eeprom, I don't know)? Or are we just expected to buy ALL of them and any new ones that come out? Please explain to us non-developers or those just learning to avoid anxieties.

[Stone]

1) To protect against power-off during a write, split the chip into sectors of an agreed minimum size (say 64 bytes for arguments sake).  Now to improve the chance of this data being happy even after turning the machine off during a write, a sector can only be written to one at a time.  So if power is lost only one sector should suffer.

 

A version of this that works better (as far as I can see) was proposed on ugd-jag (by 42Bastian, iirc) - it's called 'journaling' if you want to look it up, but I understand it as this:

 

Each 'slot' on your EEPROM has a set size. There is already a system in place to uniquely identify each slot as belonging to a particular program. One 'slot' of the EEPROM is reserved and can't be written to (maybe with a unique ID?). What happens is this:

 

1) A program needs to write to the EEPROM. It has already generated the data to be written (or it reads the previously existing slot and modifies the contents into the new slot's intended contents)

 

2) This new data is checksummed.

 

3) a new slot is opened, with this checksum being written into the very start of the EEPROM slot (after the slot ID), and all the rest of the data being written in after it, until the end of the slot.

 

4) When this write process is finished, the previous slot is erased, and has the 'don't write here' ID applied to it.

 

When reading back the data, the code should:

 

1) Check all the way through the EEPROM looking for instances of its ID (this can be very fast, as all the slots are a set size, so you can skip to known locations to get the slot IDs)

 

From here there are two options:

 

2a) There is only one slot with the program's ID. The required data is read from this slot, and all is well (sequence ends here)

 

2b) There are two slots with the program's ID. This implies that the Jag was interrupted (/shut down) during a write. The program should then:

 

3) Checksum both slots, and compare the results with the checksum stored at the beginning of each slot. One of the checksums will be incorrect (since not all of the data got written correctly for that slot).

 

4) Overwrite the slot that wasn't written correctly with the 'don't use this slot' marker ID. Display a message saying 'The last EEPROM save was unsuccessful/interrupted. Restoring last save'

 

5) Load the data from the intact segment. Sequence ends here.

 

The point with this method is that there's never a time when there is no copy of the data safely stored on the EEPROM, so the worst-case scenario is that the user loses the data since their last save. Will take very slightly longer to save each game, too, but that's not so bad The largest disadvantage is that you 'lose' one slot of the EEPROM.

 

This method could also be added into the ugd-jag standard very trivially (all that would have to be added to existing code is a method to ensure that the slot marked 'don't write here' is not overwritten when it's searching for a slot to use the first time it ever saves anything). In fact I think the current standard says to skip over any slot with an ID tag that isn't 'FF FF' or something similar, so this would even work with existing code.

 

I think this would be quite a helpful addition to programs that want to support it...and programs that don't will have no problems accessing the EEPROM with their existing code.

 

Hope that was understandable...and possibly even helpful

 

Stone

[Stone]

Isn't this academic? Aren't there ALREADY two (or three)seperate standards for EEPROM saves between Protector:SE, BSG and the B&C bypass cart (assuming it HAS eeprom, I don't know)? Or are we just expected to buy ALL of them and any new ones that come out? Please explain to us non-developers or those just learning to avoid anxieties.

 

B&C carts have an EEPROM now, apparently, and someone said in another thread that Bruce would replace them if yours didn't (early ones didn't). You'd have to mail and ask.

 

The biggest problem is between BSG and PSE, as I see it, is T-bird's setup of giving people their own 'allocated' spots. OK, this is fine if people only need a couple of bytes, but it does mean that they have to go through this allocation system, which presumably won't be around forever (whereas the ugd system allows any program written in the future to take a unique slot ID and then just use it, without asking permission from a higher authority first). I've also heard of people being refused slots...

 

It is perfectly possible to merge the two systems however; there's already talk of implementing a slot ID (or two) on the ugd system for a slot containing many 2-3 byte slots for different games. I think it could solve a lot of confusion if T-bird and UGD-JAG both posted their specs in some public place (here? lol) for coders to pick over.

 

Just my opinion, anyhow...I think it's quite unlikely that many games will be released that don't work on one spec or the other.

 

Stone

[LinkoVitch]

Isn't this academic? Aren't there ALREADY two (or three)seperate standards for EEPROM saves between Protector:SE, BSG and the B&C bypass cart (assuming it HAS eeprom, I don't know)? Or are we just expected to buy ALL of them and any new ones that come out? Please explain to us non-developers or those just learning to avoid anxieties.

 

As I understand it these methods of writing will work regardless of the cart in place. So there should be no need to buy a new bypass cart. I could be wrong this is new to me too, but from the heated discussion in the New developments thread I thought it may be worth trying to open a topic to try and resolve this.

 

From what I understand these methods of storing to the eprom will be implemented software side, and have no bearing on the bypass itself. (perhaps the software will have to check for certain carts like BSG and P:SE to ensure no damage is made to saves on their eproms).

[Gunstar]

thanks guys! I know this thread is intended more for current developers, but since it is in an open forum, those of use who are just starting out (learning basics) and users will be reading and such questions can cause anxieties to the ignorant. thanks for your understanding and tolerance to my "annoying little laymen questions." :wink:

[Matthias]

Hi,

 

I would suggest that the undergound-spec and the scatalogic-spec are publsihed here to have a real base for this thread.

 

Matthias

[justclaws]

Hello,

 

Isn't this academic? Aren't there ALREADY two (or three)seperate standards for EEPROM saves between Protector:SE, BSG and the B&C bypass cart (assuming it HAS eeprom, I don't know)? Or are we just expected to buy ALL of them and any new ones that come out? Please explain to us non-developers or those just learning to avoid anxieties.

Don't worry, all the carts have the same capability for a 2K E2PROM, it is just the allocation of the space that is in question. This is all done in the software of the game, so only the downloaded (JUGS or BJL) games need be changed for everything to work, likewise with CD loaded games. The biggest danger is just that if the 2 systems are not agreed, you might not be able to save games written for one system on the cart using another, without the danger of losing your game saves, Thunderbird's worry too.

 

The biggest problem is between BSG and PSE, as I see it, is T-bird's setup of giving people their own 'allocated' spots. OK, this is fine if people only need a couple of bytes, but it does mean that they have to go through this allocation system, which presumably won't be around forever (whereas the ugd system allows any program written in the future to take a unique slot ID and then just use it, without asking permission from a higher authority first). I've also heard of people being refused slots...

Well I've not heard of anybody being refused slots, Scatologic want developer support too, to sell JUGs, but requesting a slot has been something that I, personally, have been delaying doing pending a solution for the new system, in case it would not be necessary to be so "formal".

 

What I need Thunderbird is 128 bytes, on a 128 byte boundary please. My code already maps out 128 bytes for a normal Jaguar E2PROM, and so I really need that normal 128 bytes. I will send you an email if you can confirm that this is all I need to get "a slot". If I need to use special routines to access it, please let me know what to do. I cannot tell you the nature of the project at the moment, but the 128 bytes are the minimum.

 

Regards,

JustClaws.

[Thunderbird]

Hi Stone,

 

The biggest problem is between BSG and PSE, as I see it, is T-bird's setup of giving people their own 'allocated' spots. OK, this is fine if people only need a couple of bytes, but it does mean that they have to go through this allocation system, which presumably won't be around forever (whereas the ugd system allows any program written in the future to take a unique slot ID and then just use it, without asking permission from a higher authority first). I've also heard of people being refused slots...

 

The reason I did things the way I did was because this is a very complex issue and everyone has ideas for how it should be done. I didn't really want to make up a "ScatoEEPROM format" and force everyone to use it. I was partly being lazy and hoping that someone in the underground would propose a spec for something that was sophisticated like a journaling filessystem and everyone would use it.

 

I left the empty space as raw and unformatted and only made sure to at least guarantee that some specific developers would have some space reserved in my memory map, so I allocated some space to them specifically. I figured that if nobody came up with a good standard in the meantime then my default idea for assigning areas for certain programs would be sufficient. Plus, it was a good way to find out about a number of projects people were working on because they would write to request space on my EEPROM.

 

Since no software has yet to be publicly released that uses the EEPROM storage on the bypass carts, I don't have a problem with anyone coming up with an underground spec that works for everyone. I'll change my FAQ to reflect that the free space on the BSG cart follows the underground spec and get everyone who previously had reserved space to comply with the standard.

 

Since nothing new for the CD has yet been released, we can define the spec for a better way to do it, and use that instead of the schemes currently employed on all the different bypass cards.

[Thunderbird]

A version of this that works better (as far as I can see) was proposed on ugd-jag (by 42Bastian, iirc) - it's called 'journaling' if you want to look it up, but I understand it as this:

 

This is exactly the thing I am talking about. I use a version in my Linux devices called "JFFS" (Journaling Flash File System) which works great. It can be slow on huge flash chips, but it beats the pants off corrupt data.

 

2a) There is only one slot with the program's ID. The required data is read from this slot, and all is well (sequence ends here)

 

2b) There are two slots with the program's ID. This implies that the Jag was interrupted (/shut down) during a write. The program should then:

 

3) Checksum both slots, and compare the results with the checksum stored at the beginning of each slot. One of the checksums will be incorrect (since not all of the data got written correctly for that slot).

Actually, this is not entirely true. If the write fails after the new sector is validated but before the old one is invalidated, then you will have 2 valid sectors with valid checksums.

 

You need to have a counter in each sector which is incremented each time the sector is modified. If you end up with the situation where both are valid then you look at the counter and know which one was the last one written to, and discard (mark invalid) the opposite one.

 

This method could also be added into the ugd-jag standard very trivially (all that would have to be added to existing code is a method to ensure that the slot marked 'don't write here' is not overwritten when it's searching for a slot to use the first time it ever saves anything). In fact I think the current standard says to skip over any slot with an ID tag that isn't 'FF FF' or something similar, so this would even work with existing code.

 

The current format does not have any provisions for anything other than 128 byte sectors and has a fixed limit of 16 of these sectors. Much larger EEPROMS are coming on the market and there will be no way for existing software to make use of them because the standard doesn't have any way to change the sector size and is totally based on the fixed limitation.

 

Since matthias asked me about the format, I have made some small notes and did a little bit of work on a new format. The Open AdFree NonVolatile Memory Standard I have been playing with in my spare time has small reserved save area on an even page boundary at or near the bottom of the memory (I say "memory" because we need to support all forms of memory, like SRAMS. EEPROMS, etc. which are now available in sizes much larger then 2K) where the program can look and see the size and quantity of the save sectors. This means that all programs will be able to take advantage of huge save EEPROMS. Pages in memory below the JagFAT are ignored by a test (see below) and allow for skipping save-game data in obsolete formats or for skipping configuration data the memory interface may need to know about.

 

The reserved area I dubbed the "JagFAT", and is set up when a fresh EEPROM is "formatted".

 

With this in place, the program can search the first bytes of each "page" in the EEPROM and look for the JagFAT Identity Code. This location will be the start of the JagFAT File Area and the size and count of sectors in the file are is read from here. The JagFAT ID I used was the string "JAGFAT" and there are a few bytes of data following it to describe the sector size and count. The JagFAT header contains a checksum to make sure the miniscule chance that the bytes J-A-G-F-A-T- appear on some other even page beginning are caught. I know it's not likely, but the checksum reduces the odds of a different sector being seen as the JagFAT header.

 

All this is preliminary and just off the top of my head. I don't have the notes in front of me, and anything can change at any time as the spec evolves

 

This solves the problem of multiple standards between bypass cards because any existing standard can be replaced by this method and all software will know how to locate the JagFAT on any Bypass.

 

Also, one of the main disadvantages of having a large sector size is that a whole sector can be lost if a write fails. If a smaller sector size is used (like 32 bytes) the programmer can write his game so that a failed write on one sector is not goingto wipe out the entire saved settings for a player. He can put critical settings in one area and noncritical ones in another sector and reduce the chances for a save of something noncritical from wiping the whole save game.

 

For example, highscores can be kept in one small sector. "Unlocked levels" are saved in another small sector, and volume / controller preferences are kept in a different small sector. If the user trashes the sector with the volume settings, it's not going to wipe out the highscores or the unlocked levels which are a LOT harder for him to restore manually. This is another reason I think 128 byte sectors on the 2K EEPROM are too large, and that there should be a way to scale the sectors based on the size of the storage device.

[LinkoVitch]

for a bypass cart is there no way of combining the flash chips of the Data Track in there? So that the bypass could be used as a Memory Track system for official CD releases, and also as a bypass for homebrew CD releases. WOuld be cool if the homebrew stuff could use the flash RAM like official games. And also then use the structure Atari cam eup with for that.

[Stone]

for a bypass cart is there no way of combining the flash chips of the Data Track in there?  So that the bypass could be used as a Memory Track system for official CD releases, and also as a bypass for homebrew CD releases.  WOuld be cool if the homebrew stuff could use the flash RAM like official games.  And also then use the structure Atari cam eup with for that.

 

I believe some work has already been done on this, but I'm not sure exactly how far it went. But certainly, if the same chip was included and interfaced in the same way (is that even possible?) then it could be written to just like the normal Memory Track.

 

This would be unlikely to be used in homebrew games, I think (they can use the EEPROM standard when it's finished) but it'd be cool to use unreleased games that have Memory Track support. I'd be surprised if some of them didn't have it (there is a save/load option in American Hero but it's greyed out) but it'd be cool to try, anyway.

 

Stone

[Matthias]

Hello!

 

for a bypass cart is there no way of combining the flash chips of the Data Track in there?  So that the bypass could be used as a Memory Track system for official CD releases, and also as a bypass for homebrew CD releases.  WOuld be cool if the homebrew stuff could use the flash RAM like official games.  And also then use the structure Atari cam eup with for that.

 

I believe some work has already been done on this, but I'm not sure exactly how far it went. But certainly, if the same chip was included and interfaced in the same way (is that even possible?) then it could be written to just like the normal Memory Track.

 

This would be unlikely to be used in homebrew games, I think (they can use the EEPROM standard when it's finished) but it'd be cool to use unreleased games that have Memory Track support. I'd be surprised if some of them didn't have it (there is a save/load option in American Hero but it's greyed out) but it'd be cool to try, anyway.

 

Stone

 

Doing a CD-bypass / MemoryTrack-Combo is no real problem, you just have to have the proper PCB for this (or recycle Atari MT-cards).

 

In fact, my "Final Jaguar Cartridge (no TM)" includes these things and it works quite well.

 

If American Hero offers a menu-option for MemoryTrack which is currently grayed, then it may be grayed because of the fact that your CD-bypass solution just doesn't offer the MT-BIOS currently.

 

Matthias

[Stone]

Doing a CD-bypass / MemoryTrack-Combo is no real problem, you just have to have the proper PCB for this (or recycle Atari MT-cards).

 

Oh, yes, good point. You can just change the EPROM

 

If American Hero offers a menu-option for MemoryTrack which is currently grayed, then it may be grayed because of the  fact that your CD-bypass solution just doesn't offer the MT-BIOS currently.

 

Thats precisely what I thought. But Memory Track's aren't really cheap enough to hack up; not for me anyway (I only have the one, which is used for the intended purpose ). Unless anyone knows different?

 

Stone

[+rdemming]

there is a save/load option in American Hero but it's greyed out

 

Yes, the save/restore option in "American Hero" works. You just need a JaguarCD with a developer bios since with the other bypass solutions you can't insert a MemoryTrack too. That is one of the reasons I modified my JaguarCD (besides having a switch in the JaguarCD looks kinda cool )

 

You have three slots to save the game. If you look with the MemoryTrack manager the save games are save under the game name “GAMEFILM X.16” and filenames “FILM.1”, “FILM.2” and “FILM.3”. Each saved game takes 1536 bytes.

 

It seems that you save the game at the last choose point. Thus after a restore you start just before your last choose point. And when you save a game you are taken back to the beginning of the current scene.

 

If you store the game before the first choose point and then restore it, you get a couple of screens with the following texts (which you normally don’t see):

ATARI PRESENTS

AMERICAN HERO

CONCEPT BY DAVID M. SCHWARTZ

WELCOME TO

L.A.

 

Then it hangs till you press a button which takes you to the end of game menu (with more options then when you just die). If you then choose “PLAYBACK”, you see the film but get in an infinite loop at the first choose point.

 

Robert