Yet another cartridge dumper

[arpruss]

I made a cart dumper for the stm32f103c8t6 blue pill boards using the libmaple-based Arduino core and my USBComposite library. The fun thing about this cart dumper is that it emulates a flash drive, so when you plug it into a USB port, you get a GAME.A26 file you can load into an emulator right from the dumper--you don't need to copy it or anything. 

 

Currently, it only supports 4K, F6, F8 and FA cartridges.

 

One thing I haven't been able to figure out is how to recognize whether a cartridge has been pulled out and another inserted, so to switch cartridges you currently need to reset the device or pull it from USB. If you have any thoughts on how to detect a cartridge easily, it would be appreciated.

[+bent_pin]

35 minutes ago, arpruss said:

One thing I haven't been able to figure out is how to recognize whether a cartridge has been pulled out and another inserted, so to switch cartridges you currently need to reset the device or pull it from USB. If you have any thoughts on how to detect a cartridge easily, it would be appreciated

Continuity between ground pins?

[Windless]

4 hours ago, arpruss said:

I made a cart dumper for the stm32f103c8t6 blue pill boards using the libmaple-based Arduino core and my USBComposite library. The fun thing about this cart dumper is that it emulates a flash drive, so when you plug it into a USB port, you get a GAME.A26 file you can load into an emulator right from the dumper--you don't need to copy it or anything. 

 

I am currently trying to make a devboard and I have plenty of bluepills... this will be a handy test tool

 

4 hours ago, arpruss said:

One thing I haven't been able to figure out is how to recognize whether a cartridge has been pulled out and another inserted, so to switch cartridges you currently need to reset the device or pull it from USB. If you have any thoughts on how to detect a cartridge easily, it would be appreciated.

I would activate the pullups and continously read the reset vector. If it's $FFF then you know the cartridge is removed. You probably want to programmacticaly disconnect the mass storage device from the OS then, so when a new cartridge is inserted it can be discovered again, else you will most probably have issues with the OS caching the whole 4K file (and probably the file allocation table also)

[Windless]

I tried to read the code, but I could not find where the FAT16FileReader is defined. Are you missing some files or is 7am too early for me to try reading C code (I am really not an expert in C ^^)

 

Also there is this I'm not sure I understand :

  digitalWrite(addressPins[12], 0);
  for (unsigned i = 0 ; i < 13 ; i++, address >>= 1) {
    digitalWrite(addressPins[i], address & 1);
  }

Is the 13 supposed to be a 12 ? IIRC the code works because you always %4096 the address before calling read(), so when i=12 pin 12 is set to 0 again, but that make the first line useless ?

 

I like the automatic hotspot detection. Since you are detecting the bankswitching scheme before initialise the MSD anyway, probably you could use this information to change the filename.

 

Thank you for sharing this !

[alex_79]

6 hours ago, bent_pin said:

Continuity between ground pins?

Not all carts have both gnd pins connected. The Retron 77 relies on that and as a result fails to detect some carts that in theory it could dump.

[arpruss]

8 hours ago, Windless said:

I tried to read the code, but I could not find where the FAT16FileReader is defined. Are you missing some files or is 7am too early for me to try reading C code (I am really not an expert in C ^^)

 

FAT16ReadOnly.cpp and FAT16ReadOnly.h are in the repo. I'm kind of proud of that code: it dynamically generates FAT-16 sectors as needed, so that although the computer sees an 8mb flash drive, the actual RAM usage is very small (we only have 20K RAM and 128K flash on the blue pill, and I currently use 7K RAM and 44K flash, so it should fit in an stm32f103c6t6 as well).

 

Thanks for catching the unnecessary A12 write. Fixed.

 

I could change the filename, but I don't really see why. I think GAME.A26 is more friendly than, say, GAME_F8.A26. Also, keeping the name the same makes it easier to set up an emulator to load directly from the fake flash drive (which is the purpose). Now, if I could set the filename using the CRC and a table of cartridges, that would be rather nice. I could, I guess, use the libretro database, which has 727 entries, and would fit into flash (though the list is missing one of my favorites: Foot Craz). But I'd also need to add support for long filenames to my FAT-16 code, since currently I only support 8+3. I'll think about it.

 

Other changes:

I fixed the bankedRead() routine to check whether we didn't inadvertently read a hotspot and switch banks. That would have affected cartridges where the last few bytes differ between banks--I don't know if there are any such cartridges.

I added a CRC-32 to the INFO.TXT file (and checked that the one cartridge I have matches the CRC-32 for it on the web, as well as the one zlib reports for the GAME.A26 file). And I changed to DOS line endings for INFO.TXT.

Edited September 4, 2023 by arpruss

[arpruss]

8 hours ago, Windless said:

I would activate the pullups and continously read the reset vector. If it's $FFF then you know the cartridge is removed. You probably want to programmacticaly disconnect the mass storage device from the OS then, so when a new cartridge is inserted it can be discovered again, else you will most probably have issues with the OS caching the whole 4K file (and probably the file allocation table also)

Good idea.

 

I earlier thought of using pullups, but I don't like the idea of the blue pill and the ROM chip fighting over the data. What if the ROM chip loses? Is that a possibility? I don't know how ROM (and EPROM) chips work on the hardware level. Is this an issue I should worry about?

 

I did just check, and the CRC is still correct with pullups activated (ditto for pulldowns, btw), but I only have one cartridge to test with.

[Windless]

2 hours ago, arpruss said:

What if the ROM chip loses?

pullups on the stm32f103 are (IIRC) 14KR. They are just pull ups, not drivers. Basically, pullups exist for the sole purpose of not letting undriven lines read random values while letting other drive them when they are connected.

 

2 hours ago, arpruss said:

I could change the filename, but I don't really see why.

Stella uses (de facto standard) file extensions that helps the emulator (and dev carts) know which scheme is used. It would be an easy way for user to know which format was detected without even needing to open the .txt file you create : https://stella-emu.github.io/docs/index.html#PropertiesCartType
d

2 hours ago, arpruss said:

I'm kind of proud of that code: it dynamically generates FAT-16 sectors as needed, so that although the computer sees an 8mb flash drive

I went the opposit direction for my dev cartridge : I declare a FAT12 SMD with only 64 sectors (32kB) plus the two required one for FAT and file name. Also the cluster is 32kB, this way you can not write a new file, only replace the existing one. (Ok, since the OS caches the access, there is no way to prevent file deletion so in the end you alway can delete a file - in cache only - and write one with a different name)

[arpruss]

1 hour ago, Windless said:

I went the opposit direction for my dev cartridge : I declare a FAT12 SMD with only 64 sectors (32kB) plus the two required one for FAT and file name. Also the cluster is 32kB, this way you can not write a new file, only replace the existing one. (Ok, since the OS caches the access, there is no way to prevent file deletion so in the end you alway can delete a file - in cache only - and write one with a different name)

I would worry about compatibility issues with such small drives. It wouldn't surprise me if either Windows or Linux dropped FAT12 support at some point.

[Windless]

FAT16 is basically an extension of FAT12, there are probably around 100 lines of code you would drop by reomving support of FAT12, and that would prevent anybody from reading a floppydisk. I don't expect that soon.

[arpruss]

Changes:

 

1. I used the @Windless pullup method to detect hotswapping (I also make sure that the reset vector is continuously readable for 2 seconds, in case the cartridge is being slowly inserted in a way that makes contact with some but not all data pins), but I do the dump without the pullup.

 

2. I use the retroarch rom database to detect rom names and I added LFN code so that the game gets a full name. 

 

3. I detect 2K cartridges by checking if the second half equals the first half.

 

4. Now it needs to be compiled for 128k flash. If you have one of the rare blue pills with only 64k flash, you can remove some or all of the roms.h data.

 

E.g.:

 

C:\users\[deleted]\ARduino\atari2600dump>dir e:
 Volume in drive E has no label.
 Volume Serial Number is 0000-0000

 Directory of E:\

01/01/2023  12:00 PM                70 INDEX.HTM
01/01/2023  12:00 PM                89 INFO.TXT
01/01/2023  12:00 PM             8,192 E.T. - The Extra-Terrestrial (USA).a26
               3 File(s)          8,351 bytes
               0 Dir(s)       8,325,120 bytes free

C:\users\[deleted]\ARduino\atari2600dump>type e:INFO.TXT
Bank switching: F8
Size: 8192
CRC-32: 6d0a475f
Game: E.T. - The Extra-Terrestrial (USA)

C:\users\alexander_pruss\ARduino\atari2600dump>

 

[arpruss]

I also added a second copy of the file, with the Stella type-identifying extension. After all, since it's a virtual drive, it doesn't cost anything (except a bit of RAM for the root directory entries) to have more copies.

[Windless]

If you have some spare time, it would be great to include some building instruction

 

For now I guessed that the .ino file means you build with the arduino IDE, So I tried to, I think I managed to find how to build for a bluepill. It could not build because USBComposite.h was not found, so I cloned this library in the "library" directory of the arduino ide but then it complains about not finding "boards.h", and there are to many of the on github for me to find the right one. This is where I gave up

[arpruss]

Yeah, I should provide some instructions.

 

I always use Roger's libmaple-based core rather than the official STM32 core, because USBComposite only works with libmaple, and libmaple is also less bloated.

 

Here are two different sets of instructions (the second is mine) how to set up Arduino for this core:

 1. https://www.instructables.com/Programming-STM32-Blue-Pill-Via-USB/

 2. https://www.instructables.com/Gamecube-Controller-USB-Adapter-and-Getting-Starte/ (see steps 1 and 2)

 

I fear something may be out of date in these instructions, and a lot of developers have jumped ship from the libmaple core to the official one. People keep asking me to make USBComposite compatible with the official core, but I don't have the energy to figure out the details of how to do that.

[arpruss]

I just added a fun little thing. I auto-generate a LAUNCH.HTM file which launches the ROM using the javatari.js emulator. 

[Windless]

I've still not managed to build it. I took notess of everything I did at last attempt :

 

This part is optionnal, use it to upload code from arduino IDE, but you can also just use arduino IDE to compile the code then flash the .bin with your usual method

• Download the HID bootloader ( stm32_binaries.zip ) from https://github.com/Serasidis/STM32_HID_Bootloader/releases
• Install it like you would usually flash a firmware on your board (e.g. using STLink V2) or use the hid_flash util you can find on previous link

Thi spart is reuqired :

• Download and install the version 1.8.19 of the Arduino IDE from http://arduino.cc/en/Main/Software.
• Run the IDE, in file -> preferences, change laguage from "os default" to "english" if you want
• In Tools -> Board : ... -> Boards manager, and search "Arduino SAM boards (Cortex-M3)" and install it (you won't use it, but this will install a working arm-none-eabi-g++ toolchain to compile the code for the bluepill)
• Download zip file containing the STM32 files from https://github.com/rogerclarkmelbourne/Arduino_STM32/archive/refs/heads/master.zip.
• Unzip it "my document\Arduino". You should have a directory named "C:\Users\<your user name>\Documents\Arduino\Arduino_STM32-master"
• Rename it to remove the "-master" part
• In this folder, you will find a /drivers/win/install_drivers.bat file : run it as administrator
• Restart arduino ide
• Open the atari2600 dumper project
• In Tools -> Board: ... -> Stm32 boards group choose "Generic STM32F1 series"
• In Tools -> Board part number:... choose the proper bluepill
• In File -> Preferences, make sure "Sketchbook location" is the path to the parent folder of your "Arduino_STM32" directory (so it should be "C:\Users\<your user name>\Documents\Arduino\" if you have a default windows install)

At this point, compiling gives and error :
 

atari2600dump:17:10: fatal error: USBComposite.h: No such file or directory
   17 | #include <USBComposite.h>
      |          ^~~~~~~~~~~~~~~~

and USBComposite is not found in the library manager.

 

I tries moving the "USBCompostite" folder in "ARDUINO_STM32..." to the "libraries" folder of my sketchbook directory, but then it can't find "Boards.h". There is a "boards.h" file but if I try using this one it misse "maple/maplelib.h"... I give up untill I find some help

Edited September 12, 2023 by Windless

[arpruss]

12 hours ago, Windless said:

• In Tools -> Board: ... -> Stm32 boards group choose "Generic STM32F1 series"
• In Tools -> Board part number:... choose the proper bluepill

Thank you for such a detailed writeup, and I'm sorry I was confusing...

 

The right section of the boards is "STM32F1 boards". (I think you might be choosing the boards for the official STM32 Arduino core.)

Then you choose "Generic STM32F103C series". 

Then in the Variant menu, choose the 128k version.

Then choose the upload type to match your bootloader / upload method.

 

Here's what it looks like for me: 

 

 

The version of USBComposite bundled with Roger's core is out of date, and I think he hasn't responded to my pull request to update it. Probably it doesn't matter for this project. But if you do want the latest, it's here: https://github.com/arpruss/USBComposite_stm32f1

 

I also now put a standalone firmware binary in the repo: https://github.com/arpruss/atari2600dumper/tree/main/binary  (It may get out of date from time to time.) You can install in the same way that you would install a bootloader (I use a USB-to-serial link), and it overwrites the bootloader. But this way you don't need to build, or even have Arduino installed.

 

I only own two carts; one is F8 and the other is DPC. Both now work. In theory, F4, F6, FA and FE should also auto-detect and work (and ditto Super Chip variants), but likely there are bugs since they are completely untested. (I emailed a local used game store asking if they might let me test with games in their collection.)

 

[Windless]

I finally managed to build just an hour ago (removing the "additionnal board management url", then it found the one I installed in "hardware").

 

My problem is now that GDB refuses to connect OpenOCD ^^ I'm starting to wonder if it would not be time for me to try something else than this programming thing I'm still so bad at after nearly 40 years

Edited September 12, 2023 by Windless

[arpruss]

6 hours ago, Windless said:

I finally managed to build just an hour ago (removing the "additionnal board management url", then it found the one I installed in "hardware").

 

My problem is now that GDB refuses to connect OpenOCD ^^ I'm starting to wonder if it would not be time for me to try something else than this programming thing I'm still so bad at after nearly 40 years

I've never used OpenOCD, and I've never used GDB for embedded projects. My usual way to debug for most platforms is to add print statements, or just put additional data in files on the emulated disc in this project.

[arpruss]

Looking over my pin assignments, they are kind of random, arranged by physical proximity more than anything else. I think I'm going to resolder and move some of the pins so that all the address pins are on port A, and all the data pins are on port B, and then I can do atomic writing of the addresses, which I think is a good idea.

[Windless]

Yes Also a good idea is to keep free then pins that could better be used for something else (e.g. : usb pins, SWD pins, the buttons on the board, ...)

[arpruss]

On 9/13/2023 at 1:45 PM, Windless said:

Yes Also a good idea is to keep free then pins that could better be used for something else (e.g. : usb pins, SWD pins, the buttons on the board, ...)

Good point. 

 

Avoiding the USB pins, I found I couldn't put all the 13 address lines on one port. I had available 12 pins on GPIOA and 13 pins on GPIOB, but if I took all 13 pins from GPIOB for the address lines, I didn't have enough 5V-tolerant pins left-over on the board for the 8 data lines. So I ended up putting 12 address lines on GPIOA, A13 on GPIOC, and the data lines all on GPIOB. So I can't write the whole address at once, but I can do it in two steps (I didn't check what the assembly code looks like).

 

I also made the cartridge removal detection code a bit more finicky, so it triggers a reboot more easily if you wobble the cartridge. I also do a second read of the whole cart during startup to make sure the cartridge is stably seated (if not, reboot). The downside of all this is that a wonky cartridge with bitrot that gives unrepeatable reads will get rejected entirely. Perhaps it would make sense to add a non-finicky mode which doesn't do cartridge stability or removal checks. 

 

Anyway, if you want to try out the revised code with some other nominally supported cartridges (right now all I have is one F8 and one DPC for testing), that would be great. I am particularly suspicious of my FE code. 

Edited September 15, 2023 by arpruss

[Windless]

for now I have a probleme where the usb enumeration fails. I've tested on 3 different (one real stm32, oe fake stm32, on "real" cs32) bluepills and they all get the same result. I don't know why but bluepills / blackpills usb enumeration (even in DFU mode) seldom work on my two computers, but using the code provided with STM32 seems to work fine (I made a mass storage device that is detected perfectly on each try using this).

 

My plan was to try reusing you dumping code with my mass storage device code tomorrow.

 

I don't have much cartridges here, will need to check if any of them uses bankswitching

[arpruss]

7 hours ago, Windless said:

for now I have a probleme where the usb enumeration fails. I've tested on 3 different (one real stm32, oe fake stm32, on "real" cs32) bluepills and they all get the same result. I don't know why but bluepills / blackpills usb enumeration (even in DFU mode) seldom work on my two computers, but using the code provided with STM32 seems to work fine (I made a mass storage device that is detected perfectly on each try using this).

Can you try one of the USBComposite library examples, e.g., simplekeyboard?

 

What OS are you using?

 

There is also a known issue with some of the bluepills having a wrong resistor value. This never game me trouble, but it may give trouble to other people: https://sites.google.com/view/makeriot2020/home/tutorials/stm32-blue-pill/using-native-usb-port-on-stm32-blue-pill

Edited September 16, 2023 by arpruss

[arpruss]

8 hours ago, Windless said:

for now I have a probleme where the usb enumeration fails.

Sorry for saying what may be obvious, but do you have a cartridge hooked up? The code scans the cartridge before creating the mass storage drive, so it can know how much space to allocate for each file, create the file names in the root directory, etc.

 

If you want to try without a cartridge (and get an empty 2k file), you can comment out these lines in setup() and loop():

  waitForCartridge();
  ...
  if (crc != romCRC(0,romSize))
    nvic_sys_reset(); // in case cartridge hasn't stabilized
  ...
  if (!detectCartridge()) nvic_sys_reset();

 

Edited September 16, 2023 by arpruss