Buggy R77s crashing with Stella 6.x

[DirtyHairy]

Hi folks!

 

It's been over three years now that we first released an updated firmware for the R77 that runs Stella 6. While the firmware runs fine for me and many other people, we (the Stella team) have received a rising number of reports of R77s on which it crashes frequently over the last year or so. The suspicion always has been that this is caused by bad hardware, but none one of us have a crashing console, so we couldn't actually check what is going on. Well, this has changed as an extraordinarily helpful user on github has not only opened a bug report but also soldered a header to the UART on the board and attached a USB dongle. The result is not exactly encouraging, but at least we now have a culprit and can do some more systematic testing.

 

What is the issue you ask? The lucky winner is.... faulty DRAM 😛 🙄 It seems some devices either contain bad DRAM chips, or have bad solder connections, or whatever, and this causes the device to crash hard if there is heavy load on the memory bus. And with crash hard I mean a hard kernel oops for those that are into Linux. Stella dies, the kernel oopses, and the SD slot switches and changes into a JTAG port. At this point the screen has frozen, and the system has to be power cycled. Basically a hardware crash.

 

Why doesn't this happen with the stock firmware you ask? Stella 6 uses the GPU of the H3 SOC much more aggressively than the stock firmware did, and this is increasing the load on the memory bus (the amount of data transferred). It seams the faulty units run stable enough with the stock firmware (which doesn't use GPU acceleration at all), but crash under the heavier load caused by the new firmware.

 

What can we do? Not much I am afraid. If you scour the web you'll find that DRAM issues are not uncommon with the Orange Pi design that the R77 is based on. One obvious screw to turn is DRAM clock speed, and indeed this makes divergentdeveloper's (the person on github) R77 run much more (although not 100%) stable. Of course, there is a limit to that, as setting the speed too low will cause frame drops. Another thing to try is cooling the DRAM. The chip has no heatsink in the R77, and adding one might improve stability. Other than that... sorry, you're out of luck.

 

Changing the DRAM clock implies changing the bootloader, which is part of the SD image. I have prepared four different images of a build based of Stella 6.7 (not the final one though, there are still some paddle bugs that I need to work out). You can find those images on github here and here. There builds also contain a test program that was specifically created (not by me) to test for DRAM issues. In order to run the test create a file called "memtest" (the contents are irrelevant) at the root of the SD card and turn on the R77. You will see a spinning companion cube on a grey background. If your console has bad DRAM it will eventually crash, at which point the cube will stop spinning and the background will turn red.

 

It would be awesome if the people with crashing R77s try those images at different DRAM clocks, check whether they improve the stability when running Stella and run the memtester to get a definitive answer on the system's stability at different clock speeds. Also, if you have a spare heatsink, try attaching at to the DRAM chip and check whether stability improves. Please report back here on your findings. While we won't be able to fix this issue we would like to know how widespread it is and how different clock speeds affect the buggy systems. 

Edited September 10, 2022 by DirtyHairy

[Tifany]

Anyone know what size heat sink would work for the ram chip on this? Any thoughts about adding a fan?  Just trying to head off issues - recently got an Amber Retron 77 for the bedroom TV, already had a woodgrain one for the living room that has never given me any issues.

 

 

[Thomas Jentzsch]

While a heat sink helps, I wouldn't expect too much from it. Our tester had crashes down to 408 MHz (from the original 624 MHz). At 480 MHz the console became at least almost stable (a non-ARM game ran for ~2h). But he had to go down to 302 MHz until everything seemed fully stable (running ARM games). Given some web reports regarding the Orange PI, a heat sink might improve the stable DRAM speed by one or two steps. 

 

Regarding the size, maybe this picture can help:

There isn't much space between RAM and CPU, but I would say, the larger the better. A fan might help too, but I doubt it will make much difference. Especially without a heat sink.

[DeafAtariFromKansas]

16 hours ago, Thomas Jentzsch said:

While a heat sink helps, I wouldn't expect too much from it. Our tester had crashes down to 408 MHz (from the original 624 MHz). At 480 MHz the console became at least almost stable (a non-ARM game ran for ~2h). But he had to go down to 302 MHz until everything seemed fully stable (running ARM games). Given some web reports regarding the Orange PI, a heat sink might improve the stable DRAM speed by one or two steps. 

 

Regarding the size, maybe this picture can help:

There isn't much space between RAM and CPU, but I would say, the larger the better. A fan might help too, but I doubt it will make much difference. Especially without a heat sink.

For RAM, heat sink for Raspberry PI will probably fit on this chip.

[Tifany]

4 minutes ago, DeafAtariFromKansas said:

For RAM, heat sink for Raspberry PI will probably fit on this chip.

I had the exact same idea 🙂

 

[fredtriestomakeretron77work]

On 9/10/2022 at 7:03 PM, Tifany said:

Anyone know what size heat sink would work for the ram chip on this? Any thoughts about adding a fan?  Just trying to head off issues - recently got an Amber Retron 77 for the bedroom TV, already had a woodgrain one for the living room that has never given me any issues.

If your Amber is having issues with 6.6, I'd recommend downgrading to 3.5.4 to see if it's stable, and stick with it if it is.

 

You might get less crashes with 6.6 using fans and heatsinks but nowhere close to something I would consider playable  I still get crashes while it's cold, after a minute or two of gameplay, even with a fan...  

[DeafAtariFromKansas]

2 hours ago, fredtriestomakeretron77work said:

If your Amber is having issues with 6.6, I'd recommend downgrading to 3.5.4 to see if it's stable, and stick with it if it is.

Remember many CDF and CDF+ games requires least Version 5.0 of Stella and higher, A lot of Non-CDF/CDF+ games will run under version 4.7 of Stella or lower. ☺️

Edited September 13, 2022 by DeafAtariFromKansas

[fredtriestomakeretron77work]

14 minutes ago, DeafAtariFromKansas said:

Remember many CDF and CDF+ games requires least Version 5.0 of Stella and higher, A lot of Non-CDF/CDF+ games will run under version 4.7 of Stella or lower. ☺️

Well, it's the only alternative if you want to salvage one of those bad Retrons... It's an upgrade from the original FW.

[DirtyHairy]

8 hours ago, fredtriestomakeretron77work said:

If your Amber is having issues with 6.6, I'd recommend downgrading to 3.5.4 to see if it's stable, and stick with it if it is.

 

You might get less crashes with 6.6 using fans and heatsinks but nowhere close to something I would consider playable  I still get crashes while it's cold, after a minute or two of gameplay, even with a fan...  

Please read the first post and stay on topic  Have you tried any of the builds with reduced DRAM clock that I linked? Have you tried the memtester for them?

[Thomas Jentzsch]

8 hours ago, fredtriestomakeretron77work said:

If your Amber is having issues with 6.6, I'd recommend downgrading to 3.5.4 to see if it's stable, and stick with it if it is.

That's 3.9.4.

[fredtriestomakeretron77work]

6 hours ago, DirtyHairy said:

Please read the first post and stay on topic  Have you tried any of the builds with reduced DRAM clock that I linked? Have you tried the memtester for them?

Sorry to have got carried away! trying to save peeps money on heatsinks  Yep, I tried those, I'm divergentdev on github

6 hours ago, Thomas Jentzsch said:

That's 3.9.4.

*faceplam* I should have labeled those SD cards! Can't edit original post so my mistake lives on...

 

I'll be quiet now

[DirtyHairy]

1 hour ago, fredtriestomakeretron77work said:

Sorry to have got carried away! trying to save peeps money on heatsinks  Yep, I tried those, I'm divergentdev on github

*faceplam* I should have labeled those SD cards! Can't edit original post so my mistake lives on...

 

I'll be quiet now

🤣 Thanks again for your assistance!

[Krathoon]

This is why I just ordered a woodgrain Retron 77. I assume those are the first batch. They still had them on Amazon. Also, for some reason, the Ranger controller does not work on an Atari Flashback X. I blame AtGames. It should have worked.

[Thomas Jentzsch]

14 minutes ago, Krathoon said:

I assume those are the first batch.

Fingers crossed you are right here.

[fredtriestomakeretron77work]

6 minutes ago, Krathoon said:

This is why I just ordered a woodgrain Retron 77. I assume those are the first batch. They still had them on Amazon. Also, for some reason, the Ranger controller does not work on an Atari Flashback X. I blame AtGames. It should have worked.

Mine is a Woodgrain and has the DRAM issue.  Got it from Ebay.

I'd be curious to know how it goes for you with that woodgrain from Amazon!

 

I almost ordered one the other day, since it'd be easy to return if it's not stable with the community build.

[Atariboy]

I installed Stella 6.5.3 almost exactly a year ago on my woodgrain Retron '77 (Which I had bought a year or so previously, but I dragged my feet with updating for quite a bit with my system). It has been solid as a rock.

 

Does that mean I should be fine to install Stella 6.6.0?

[Krathoon]

Do the Ranger controllers work with 3.9.4? I would think they would.

Does that version allow you to access all your roms?

Edited September 14, 2022 by Krathoon

[Thomas Jentzsch]

4 minutes ago, Atariboy said:

I installed Stella 6.5.3 almost exactly a year ago on my woodgrain Retron '77 (Which I had bought a year or so previously, but I dragged my feet with updating for quite a bit with my system). It has been solid as a rock.

 

Does that mean I should be fine to install Stella 6.6.0?

The difference is between 3.9.4 and Stella 5.x/6x. If it runs rock solid with 6.5.3 it will do so with 6.6.0 too. So if the changes are interesting for you, go for it.

[fredtriestomakeretron77work]

2 minutes ago, Krathoon said:

Do the Ranger controllers work with 3.9.4? I would think they would.

Does that version allow you to access all your roms?

Yes, it has the Stella menu and you can access all of your roms, and my paddles worked so I would guess Rangers too. 

 

6 minutes ago, Atariboy said:

I installed Stella 6.5.3 almost exactly a year ago on my woodgrain Retron '77 (Which I had bought a year or so previously, but I dragged my feet with updating for quite a bit with my system). It has been solid as a rock.

 

Does that mean I should be fine to install Stella 6.6.0?

I'm pretty sure you're safe. You would have seen issues with 6.5.3 already... It's hard to miss, in around 15 minutes of gameplay on my faulty system you usually see a crash with both these versions. 

[Thomas Jentzsch]

7 minutes ago, Krathoon said:

Do the Ranger controllers work with 3.9.4? I would think they would.

Does that version allow you to access all your roms?

Almost. 3.9.4 has a good launcher too, but it doesn't support some modern homebrews. Also the video and audio emulation has been much improved since then.

[Krathoon]

What was the version of Stella that came with the console?

[fredtriestomakeretron77work]

3 minutes ago, Krathoon said:

What was the version of Stella that came with the console?

Stella 3.7.5 according to their Web site https://www.hyperkin.com/r77 and Retro RGB https://www.retrorgb.com/retron77-firmware-updated-by-stella-devs-6-1rc1.html

[MMarcoux66]

I have 3 of the Retron 77s in the family. 1 Amber and 2 originals. Only tested the amber with the 624 image. After 10 minutes the cube locked up on the screen but the background did not turn red. I was playing games on it earlier without issues.

 

I will test on the other 2 and report back.

[Thomas Jentzsch]

1 hour ago, MMarcoux66 said:

I have 3 of the Retron 77s in the family. 1 Amber and 2 originals. Only tested the amber with the 624 image. After 10 minutes the cube locked up on the screen but the background did not turn red. I was playing games on it earlier without issues.

 

I will test on the other 2 and report back.

Thanks, that will help. When did you buy the three?

[MMarcoux66]

All 3 were bought from a local game store about 5 months ago. The 2 woodgrains came in 2 different style boxes. Not sure how long they were sitting on the shelf. All were new and unopened.

Edited September 15, 2022 by MMarcoux66