The perfect dumper?

[+MrZarniwoop]

9 minutes ago, Thomas Jentzsch said:

Stella usually expects an event for switching between stretched and non-stretched (actually this is only a 16:9 switch for a 16:9 display), so a button would have matched better.

It may have better matched Stella, but a switch like the other rear switches matches the rest of the 2600 better. Otherwise we'd be back to the strange buttons on the Flashbacks, which are great for Stella but don't "feel" like the 2600:

 

I bet there were arguments on the design team for making the Color/BW switch a 4:3/16:9 switch!

[Blinky]

34 minutes ago, Thomas Jentzsch said:

That works well

I have no doubt of that. What I ment is if you want to create a hash of a cart with ram that is always the same from a dump, the ram should be treated as being a fixed value (or the hash should only be calculated over the rom area).

 

Ram on a cart is uninitialzed. After powering on / cart is inserted. The ram content  will each time be different. A  hash generated from the rom and ram area will be different after each power interruption / next time you play the game.

 

if the game had a custom configuration like  pal 60 setting configured for that hash then the next time you play the game the setting couldn't be restored because the hash is has changed.

Also if hashes where automatically stored in a database (regardless if any custom settings where made), the database will get poluted. Play the same game a 100 times (with a power cycle between each play) you'd get 100 hashes of the same game in the database.

 

 

[Thomas Jentzsch]

1 hour ago, Blinky said:

I have no doubt of that. What I ment is if you want to create a hash of a cart with ram that is always the same from a dump, the ram should be treated as being a fixed value (or the hash should only be calculated over the rom area).

 

Ram on a cart is uninitialzed. After powering on / cart is inserted. The ram content  will each time be different. A  hash generated from the rom and ram area will be different after each power interruption / next time you play the game.

 

if the game had a custom configuration like  pal 60 setting configured for that hash then the next time you play the game the setting couldn't be restored because the hash is has changed.

Also if hashes where automatically stored in a database (regardless if any custom settings where made), the database will get poluted. Play the same game a 100 times (with a power cycle between each play) you'd get 100 hashes of the same game in the database.

I get your point (assuming the RAM is random). And after detecting that there is RAM, this is easy.

 

Not sure about schemes like E7, where the RAM mapping is quite flexible.

 

BTW: Obviously the checksums defined in Stella's properties will not work:

• RAM will be different
• Hotspot values will different too

So Atari will have to build up a database from scratch (except for <= 4K and no RAM).

Edited August 29, 2023 by Thomas Jentzsch

[Dionoid]

On 8/28/2023 at 7:07 PM, Thomas Jentzsch said:

The dumper code for the R77 is known. It only detects 4K, F8 (8K) and F6 (16K). For that it triggers four hotspots ($1ff6..$1ff9), calculates the checksum of the dumped 3.5K (it avoids the first and the last 256 bytes) and if it is different, it knows it has detected a hotspot. Each hotspot pair ($fff6/$fff7 and $fff8/f9) clearly defines a bankswitching scheme (F6/F8). The spoiler shows the R77 code with my own comments.

This could be extended to F4 (32K) easily. And I suppose it could be done for all kind of unique hotspot addresses. But how slow will this become? Brute force will most likely not work, this has to be done methodical (e.g. a following a decision tree). And how will it detect e.g. Activision's FE? Will it handle Xin1 carts work? How about carts with a menu, where the user selects the game?

 

And Pitfall II and the ARM carts will most likely never work via a simple dumper.

 

What are your thoughts and ideas here?

So the R77 dumper code for 4K, 8K and 16K seems to be effective and fast enough. Maybe the F4 bank switching (32K) was left out in the R77 dumper, as it became to slow?

For the other bank switching schemes (FE, 3F, 3E, E0, E7, etc.) I think strobing a list of unique hotspot addresses (either brute force or using a decision tree) will make the dumper too slow. People probably expect the game to run within a few seconds after inserting the cartridge.

 

An idea: what if the dumper code creates a simple hash/fingerprint of the first bank of an inserted cartridge (e.g. only using the middle 2K of the first bank, so skipping the first and last 1K to avoid strobing hotspots and reading optional RAM). And then using a simple internal lookup, the dumper code checks in its internal database if this simple hash matches a game which uses a known bank switching scheme, so then the dumper knows exactly which logic to use to dump the cart correctly. If the simple hash isn't known in its internal database, the dumper code falls back to trying the normal F8, F6 bank switching hotspots. Of course this means that the dumper code must have an internal database containing known "first-bank" hashes for all games that use bank-switching other than the default F8, F6. But creating such a database is something that can be automated, as the rom dumps for all cartridges are available.

 

Also, when a rom-dump is created by the dumper, these roms files could be cached internally (FIFO), which means the second time you plug in a cartridge, it should start up significantly faster. And maybe even other information can be stored in the dumper's internal database, like TV-type, so the dumper could pass that information to stella (e.g. by adding _PAL60_ to the name of the dumped file for PAL60 non-ARM homebrew games).

 

Pitfall II probably can't be dumped this way. Maybe ARM homebrew games could be dumped, but that would require help from @batari

 

Thoughts?

 

 

Edited September 1, 2023 by Dionoid

[Thomas Jentzsch]

1 hour ago, Dionoid said:

So the R77 dumper code for 4K, 8K and 16K seems to be effective and fast enough. Maybe the F4 bank switching (32K) was left out in the R77 dumper, as it became to slow?

Maybe, it is quite slow already, it takes several seconds to dump 16K. Though I think there are a lot of unnecessary delays.

1 hour ago, Dionoid said:

For the other bank switching schemes (FE, 3F, 3E, E0, E7, etc.) I think strobing a list of unique hotspot addresses (either brute force or using a decision tree) will make the dumper too slow. People probably expect the game to run within a few seconds after inserting the cartridge.

Yes, a too slow reaction will further destroy the illusion.

1 hour ago, Dionoid said:

An idea: what if the dumper code creates a simple hash/fingerprint of the first bank of an inserted cartridge (e.g. only using the middle 2K of the first bank, so skipping the first and last 1K to avoid strobing hotspots and reading optional RAM).

That will only work for carts which have the same startup bank configuration. But most schemes start with random banks. So this kind of hash won't work.

1 hour ago, Dionoid said:

Also, when a rom-dump is created by the dumper, these roms files could be cached internally (FIFO), which means the second time you plug in a cartridge, it should start up significantly faster.

Yes, caching seems like a way to improve performance

1 hour ago, Dionoid said:

And maybe even other information can be stored in the dumper's internal database, like TV-type, so the dumper could pass that information to stella (e.g. by adding _PAL60_ to the name of the dumped file for PAL60 non-ARM homebrew games).

Well, if the full ROM is dumped, the Stella game properties can be used.

1 hour ago, Dionoid said:

Pitfall II probably can't be dumped this way. Maybe ARM homebrew games could be dumped, but that would require help from @batari

@batari said there is a way. But I doubt it will be made public since it also prevents pirating.

 

I just had different ideas:

1. Why do we have to dump all bytes, why not only e.g. N random bytes (assuming this is faster)? These bytes have to be spread so that enough bytes (e.g. 4) per bankswitching segment are compared. So for a scheme with 4K segments, check only N bytes anywhere. For a scheme with two 2K segments, check N bytes each in both segments.
2. Maybe we could check for more than one scheme per try? E.g. a scheme with 2K segments could be checked in parallel with a 4K segments scheme. If only the bytes in the 2K scheme have changed, we know it is the 2K scheme.

E.g. 3F/3E (hotspot $3F=0, 1) could be checked in parallel with F4/F6 (hotspots $1FFF6/$1FF7). And afterwards check if it is e.g. F4 or F6.

Edited September 1, 2023 by Thomas Jentzsch

[Dionoid]

54 minutes ago, Thomas Jentzsch said:

2 hours ago, Dionoid said:

An idea: what if the dumper code creates a simple hash/fingerprint of the first bank of an inserted cartridge (e.g. only using the middle 2K of the first bank, so skipping the first and last 1K to avoid strobing hotspots and reading optional RAM).

That will only work for carts which have the same startup bank configuration. But most schemes start with random banks. So this kind of hash won't work.

A game could have multiple hashes/fingerprints, one for each bank, in case it uses random banks. But maybe this gets more complex when a scheme starts with random 2k or 1k banks.

 

I think I need to learn a bit more on Atari 2600 bank switching schemes. I found this page: https://atariage.com/2600/programming/bankswitch_sizes.txt  Is that a good start to learn about the more complex Atari 2600 bank switching schemes?

Edited September 1, 2023 by Dionoid

[Thomas Jentzsch]

1 hour ago, Dionoid said:

I think I need to learn a bit more on Atari 2600 bank switching schemes. I found this page: https://atariage.com/2600/programming/bankswitch_sizes.txt  Is that a good start to learn about the more complex Atari 2600 bank switching schemes?

Yes, I attached another file from the same author (@kevtris). You can also look at the Cart... classes of Stella. The header files give a brie summary, the code has the details.

Atari 2600 Mappers.txt

[Thomas Jentzsch]

Here is some basic overview I just created:

 

[JetSetIlly]

9 hours ago, Thomas Jentzsch said:

Here is some basic overview I just created:

 

MVC is moviecart

 

[Thomas Jentzsch]

22 minutes ago, JetSetIlly said:

MVC is moviecart

I know. I meant that I have no clue, if it will ever become relevant and how to dump it.  

[JetSetIlly]

7 minutes ago, Thomas Jentzsch said:

I know. I meant that I have no clue, if it will ever become relevant and how to dump it.  

Oh I see. Sorry 🙂

[Thomas Jentzsch]

2 hours ago, JetSetIlly said:

Oh I see. Sorry 🙂

Absolutely no need to be sorry for you.  

[Thomas Jentzsch]

Some more thoughts:

 

Dumping should be fast and not last ~8 seconds like for the R77. The R77 uses a lot of long delays, which might result from slow dumper speed. The Atari 2600+ dumper might be faster. The 6507 can access 32K in <1/10s. So even bankswitching detection, if done smart, should not extend this time over 1s.

 

Edit: I have to correct the R77 timings. If the R77 is already on, it takes 4s to dump a cart and start Stella (the other 4s came from starting the R77 itself). And we do not know how much Stella takes within the 4s. So the dumper is not that slow.

 

Non-bankswitched games will be detected as "no bankswitching found" -> last (but they will be dumped faster than others).  

 

Stella's properties should be used (for controller types and swaps, phosphor, PAL-60...). Since they are based on a MD5 checksum, we would need a 1:1 dump of the existing one:

• problem #1: hotspots in ROM area return random values (F8, F6, F4, FA, E0, E7, F0, BF, CF, EF... are all affected by this; 3F, UA, WD, FE... are not)
• problem #2: dumped cart RAM will be random (affects all ..SC types, FA, E7, WD... where RAM is part of a ROM bank)

This means we need an alternative checksum, which skips all random bytes (or uses a fixed value). To exclude these bytes, we have to know the bankswitching type. So someone has to build a database where Stella's properties have been extended by the new checksum (if different). Stella has to be enhanced to use this checksum alternatively. For that it has to know the bankswitching type too. Only in cases where Stella's bankswitch auto detection fails (fortunately very few, mostly Xin1 carts), this will not work. The emulator will exit sooner or later then.

 

Also ROMs < 4K should be dumped/checksumed with their correct size. And not with repeated data. Else their checksum will differ from Stella's.

 

Meanwhile I have added more details to my spreadsheet.

Bankswitching.ods

Edited September 2, 2023 by Thomas Jentzsch

[alex_79]

1 hour ago, Thomas Jentzsch said:

Stella's properties should be used (for controller types and swaps, phosphor, PAL-60...). Since they are based on a MD5 checksum, we would need a 1:1 dump of the existing one:

• problem #1: hotspots in ROM area return random values (F8, F6, F4, FA, E0, E7, F0, BF, CF, EF... are all affected by this; 3F, UA, WD, FE... are not)
• problem #2: dumped cart RAM will be random (affects all ..SC types, FA, E7, WD... where RAM is part of a ROM bank)

I imagine another issue are overdumps: 2k games will most likely result in a 4k rom with the data repeated twice. Stella should detect those as well and recalculate the checksum for 2k. (or the dumper has to detect them, but it's probably better to leave all the non strictly necessary calculations to Stella to avoid too much slowdown)

Edited September 2, 2023 by alex_79

[alex_79]

Another aspect that needs to be considered is that the dumper in the 2600+ is shared for 7800 games too. And it has to detect those as well!

Edited September 2, 2023 by alex_79

[Thomas Jentzsch]

5 minutes ago, alex_79 said:

I imagine another issue are overdumps: 2k games will most likely result in a 4k rom with the data repeated twice. Stella should detect those as well and recalculate the checksum for 2k. (or the dumper has to detect them, but it's probably better to leave all the non strictly necessary calculations to Stella to avoid too much slowdown)

I had added this lately to my post above. But I agree, that's something for Stella. 

2 minutes ago, alex_79 said:

Another aspect that needs to be considered is that the dumper in the 2600+ is shared for 7800 games too. And it has to detect those as well!

I suppose the cart pins will do that, no?

Edited September 2, 2023 by Thomas Jentzsch

[alex_79]

9 minutes ago, Thomas Jentzsch said:

I suppose the cart pins will do that, no?

That's something we will discover after the units start shipping, I guess.

Unless there's some way to physically detect the extra pins (like a mechanical switch, or an optical sensor), I imagine you'll have to try to read some areas where a 7800 cart can be mapped but a 2600 cannot (with A12 LOW). That would be easier if the dumper has the pull-up resistors mentioned earlier in this thread, so that when the cart does not drive the bus you get a constant, known value.

Edited September 2, 2023 by alex_79

[Thomas Jentzsch]

19 minutes ago, alex_79 said:

That's something we will discover after the units start shipping, I guess.

I can ask my contact at Plaion. Maybe he knows.

[Thomas Jentzsch]

On 8/29/2023 at 4:03 PM, Blinky said:

Odd they use such long delays. The 6502/7 CPU has access times < 1 us so 1 us should suffice

The code looks like this:

DELAY_5us:                          ;
            CALL    DELAY_1us       ; 4
            CALL    DELAY_1us       ; 4
            CALL    DELAY_1us       ; 4
            CALL    DELAY_1us       ; 4
DELAY_1us:                          ;
            RET                     ; 2         RET, CALL 2 cycles each

So the call costs 20 cycles. According to the docs I found, 1 cycle equals 125ns (WT51F108s @24MHz), so 20 cycles ins 2500ns (2.5us). And the loop for each byte takes ~80 cycles. Which means the delay is completely superfluous. At least from ROM perspective.

[Blinky]

4 hours ago, Thomas Jentzsch said:

The code looks like

I found the dumpers source code and had a peek at it. It confirms the mcu uses pin 12 (with pull up activated) to detect carts. according to the datasheet the pull up has a typical value of 33k. Another thing I noticed is that Port C is used to read the carts data bus but the ports pullup resistors are not activated (random values when cart doesn't output data) and the dumper communicates with the host CPU over serial with a speed of roughly 115200 baud.
 

4 hours ago, Thomas Jentzsch said:

the delay is completely superfluous

It sure is. Once the address is set by the ADDR_SET_SUB subroutine even without a delay, at least  625ns have passed before the carts data is read.

9 hours ago, Thomas Jentzsch said:

I can ask my contact at Plaion. Maybe he knows.

Can you ask him about what mcu is used for the dumper too?

Edited September 2, 2023 by Blinky