New project - Pico based SIO device...

[woj]

Progress, much I'd say. XEX file support now works, minus the VTOC and directory sectors for the virtual disk to be DOS readable, but this is an easy fix. High speed SIO works too, down to divisor 0 (changing the active speed does not work, but I know why). All this is rather amazing to me, considering I have been coding this totally dry without any tests whatsoever for over a week, and that included code refactoring on the way. I was only two small bugs away from all initial developments working when I tested it today. What does not work, but was implemented, is sector writing. Probably another bug, but I still need to look into this.

 

In the meantime I educated myself in how to get ATX files to work, seems rather straightforward given there is a working and readable code for this available, the only unsolved part is that I am still looking for a super elegant solution to the the disk rotation counter on the Pico. PIO is the answer again here I think, but there is one tiny detail I need to clear up to fulfill the "super" part of the elegant requirement .  

[drac030]

On 7/20/2024 at 11:34 AM, woj said:

When receiving sector data (or any other additional data for any of the SIO commands), when the checksum does not match, one approach is to return 'N'ack, the other one is to go silent.

After reading this, I took a look at the SIO2BSD code and noticed an inconsistency: the "O" command (write PERCOM) was just ignoring the data frame with faulty checksum, whereas the "P" and "W" commands were reacting with 'E' in such a case. I now have adjusted both functions to return 'N' when the checksum does not match. Thanks

[woj]

8 minutes ago, drac030 said:

After reading this, I took a look at the SIO2BSD code and noticed an inconsistency: the "O" command (write PERCOM) was just ignoring the data frame with faulty checksum, whereas the "P" and "W" commands were reacting with 'E' in such a case. I now have adjusted both functions to return 'N' when the checksum does not match. Thanks

Thank you, it was your code that triggered this question 😉 and allowed me to do things right.

[drac030]

24 minutes ago, woj said:

Thank you, it was your code that triggered this question 😉 and allowed me to do things right.

Yeah, "I am a part of that power which eternally implements things wrong and eternally works good”.

[woj]

Pushing forward, writing sectors to ATRs is now fixed, the issue at hand was actually not in the SIO protocol itself, or anything close to it, the problem was accessing Pico's Flash for writing with 2 cores executing code from the Flash. Perfectly fine to do it for the one that flashes (code executes from the RAM cache), but the other core doing its Flash reading requiring stuff at the same time shits itself. 

 

I also think that I more or less figured out the HSIO things, with the U1MB PBI bios HSIO implementation it works beautifully, I'd need to test it with the original Hiasoft's patch. With SpartaDosX (I have a rather recent Beta on U1MB, but did not check for updates in some months now) it was a bit of a bumpy ride though, @drac030, perhaps you can chip in here a bit.

 

First, it seems that a drive that sits totally silent during boot is classified as not high speed, so when I turn it on after boot even with the HSIO enabled I get standard speed (there seem to be no attempt to get speed index). Pressing RESET seems to re-evaluate the status and the next access goes high-speed. RESET seems to be also required to fully switch after manually informing SDX with SIOSET 1 US (SIOSET definitely does get the speed index and the Pico emulated drive configures itself for high speed, but subsequent accesses are still standard speed until RESET is pressed). I am not saying there is anything wrong here, and my code is still wobbly, I am just trying to understand what could be possibly going on under the hood. (One possibility for what I am seeing is that INDUS.SYS is also loaded, not sure how this affects things, getting rid of it would require some SD card maneuvers). 

 

Second, I get different HSIO behaviour when the ATR is mounted read-only and this status is reported as such in get status. HSIO functions for read-only ATRs, but some SIO commands (get status presumably?) are causing the speed to go back and forth between fast and standard. Is this some sort of an attempt by SDX to see if the drive would work read/write at lower/standard speed?

 

Lastly, I did not manage to get formatting to work, but I only tried SDX's format so far, and that one was always very picky with emulated drives, I think I once tried with SDrive too and failed. Building directory works perfectly fine though.

[_The Doctor__]

Did you try SDX 4.49G as well? Download and install/flash and try it again. It really has solved a number of issues for the masses.

[woj]

8 hours ago, _The Doctor__ said:

Did you try SDX 4.49G as well? Download and install/flash and try it again. It really has solved a number of issues for the masses.

I now checked, this is what I have, 4.49g Beta from January 2022.

[+DjayBee]

On 7/25/2024 at 3:06 PM, woj said:

the only unsolved part is that I am still looking for a super elegant solution to the the disk rotation counter on the Pico

This question might be stupid but why do you need a rotational counter?

 

A faked index pulse at 288rpm can be managed with a timer and inside the ATX every sector's position is defined in 8 microsecond intervals starting from the track start.

Or is this too fast to handle it in software?

[drac030]

@woj Pressing RESET does nothing to SDX SIO settings. The only thing RESET does is to emit $FF out to the SIO connection (it is the OS which is doing that during SIO initialization). Perhaps this causes some reaction on the other side which in turn cause the effects you are observing.

 

A drive which is not accessed is classified as "undefined" initially ("-" in SIOSET). Then SDX tries to determine if the drive can be used with higher speed than 19200. IIRC, the UltraSpeed protocol is tried out first at the first command which is sent. The first one is usually STATUS, it goes through at 19200, then, when this succeeded, '?' is sent (at 19200). When the reply is correct, the drive is marked as "US" with the divisor returned. The further communication goes at that divisor, except when an error occurs, the SIO may then fall back to 19200.

 

The constant switching between turbo and normal are probably caused by the communcation errors. Needless to say I am not observing anything like that with SIO2BSD.

Edited July 27 by drac030

[woj]

8 hours ago, DjayBee said:

A faked index pulse at 288rpm

It's not only about a fake pulse at every rotation at 288 rpm, that is trivial with the repeating timer thing. But you also need to be able to query the timer to know how far you are in the rotation and wait until a particular position is reached, that becomes a tiny bit more of a problem. I do have a good idea on how to do this nicely, just that I first had to deal with other problems.

 

8 hours ago, drac030 said:

The constant switching between turbo and normal are probably caused by the communcation errors. Needless to say I am not observing anything like that with SIO2BSD.

 

We were both in fact correct - your description matches what I saw, and my implementation was correct. What was actually interjecting was the HSIO driver from the PBI bios that had something to say before SDX was doing its thing. Or even it could be that if that is enabled, then this is what SDX uses (the lack of any US and speed indexes indication from SETSIO despite the drive working at high speed would suggest that?). Long story short - disabling the HSIO driver in PBI bios brought the SDX and Pico SIO duo to their senses and they behave like they should, and each drive accessed now shows right away in SIOSET with the correct protocol and speed index. I suppose (still need to check it) that the speed alternation when doing DIR on a read-only ATR has to do with getting a 'N'ack on the PERCOM reading attempt (which I presumed would be used only for formatting and such), I should probably relax that.

 

Anyhow, all this now seems to work, on top of this I can now DOS read XEX-es through the virtually mounted ATR, and I can relocate the boot loader for XEX-es. The next big thing is the ATX support. Earlier on I also did a test accessing mounted ATRs and loading Turbo 2000 tape image both in one Pico session, as trivial as it sounds it was not at all obvious that it would work.

[+DjayBee]

1 hour ago, woj said:

The next big thing is the ATX support.

If you have use for an ATX image to verify your implementation against various protection methods, let me know.

[drac030]

1 hour ago, woj said:

Or even it could be that if that is enabled, then this is what SDX uses (the lack of any US and speed indexes indication from SETSIO despite the drive working at high speed would suggest that?).

Yes, the PBI has priority over the SIO, so if a PBI driver (parallel or not) completes the request, the serial routines are not called at all (this is how Atari is able to boot from a HDD).

 

1 hour ago, woj said:

I suppose (still need to check it) that the speed alternation when doing DIR on a read-only ATR has to do with getting a 'N'ack on the PERCOM reading attempt (which I presumed would be used only for formatting and such), I should probably relax that.

This should not have influence on the serial protocol, but I will take a look, maybe the 'N' is taken as a serial error retriggering the speed query. Anyways, the READ PERCOM is used to determine, if the drive being accessed has 512-byte sectors.

[woj]

6 minutes ago, drac030 said:

Yes, the PBI has priority over the SIO, so if a PBI driver (parallel or not) completes the request, the serial routines are not called at all (this is how Atari is able to boot from a HDD).

Good that my instincts are still working 

 

6 minutes ago, drac030 said:

This should not have influence on the serial protocol, but I will take a look, maybe the 'N' is taken as a serial error retriggering the speed query. Anyways, the READ PERCOM is used to determine, if the drive being accessed has 512-byte sectors.

 

That was actually it, when I allowed READ PERCOM to be called on read-only images the jumping speed problem disappeared. So the 'N'-ack was taken as communication error apparently. 

 

I went through a small battle of figuring out after how many command frame receiving desyncs I should toggle the speed. SIO2BSD has 4 I think, SDrive has one = toggle on first desync. While the former makes more sense to me and it worked in most of my testing, when I popped in the PoP disk weird things started happening, I am really too tired to try to dissect this, but I arrived at having it working well when toggling the speed the SDrive way. AVGCart also handled this case gracefully, just that I have no way of knowing what its procedures are... I would expect that PoP disk would have full support for US it being xBIOS based and all, but it seems only the boot loader loads at high-speed using the patched OS procedures and then all US communication goes south.

[drac030]

20 minutes ago, woj said:

That was actually it, when I allowed READ PERCOM to be called on read-only images the jumping speed problem disappeared. So the 'N'-ack was taken as communication error apparently.

Yes, I did not remember that (it has been some years) but I can now see it in the source code: actually any error except manual Break causes it to fall back to 19200.

 

Are there any US-capable drives which do not support READ PERCOM?

[woj]

43 minutes ago, drac030 said:

Are there any US-capable drives which do not support READ PERCOM?

I wouldn't know. I just thought that since I do not support 512 sector images (there is no point really with the total storage size of about 15M for all images) and otherwise PERCOM is used for formatting I can just disable it for read-only situations. But there is no good reason for this. But I still need to figure out what to do for ATR images that are not standard floppy size...

 

BTW, I now checked the retry counter in sio2bsd, it is 4, but not for the complete command frame desync, you have some more involved method for trying to save the frame while reading it, I wonder if that actually works...

[drac030]

21 hours ago, woj said:

I now checked the retry counter in sio2bsd, it is 4, but not for the complete command frame desync, you have some more involved method for trying to save the frame while reading it, I wonder if that actually works...

This code is probably an overkill, especially since the $FF Atari emits at every reset has been filtered out inside com_read().

[woj]

On 7/27/2024 at 9:51 PM, DjayBee said:

If you have use for an ATX image to verify your implementation against various protection methods, let me know.

The ATX support code transplant from SDrive-MAX was successful, the two (not sure how come I do not have more) ATX files load and work fine, so indeed I could probably use some "difficult" ATX files to test. Also some that specifically need 1050 or 810.

 

I also implemented config saving to Flash, so at this point I have only a handful of minor, some fiddly, gadgets to implement and a lot of testing to do. One bigger thing that ended up on the list is support for an SD card module (with support for hot-swapping hopefully) instead of using the internal Flash. This should be useful especially for cases when the original 2M Pico is used which does not leave much room for files, just 1M. There is a clean and good implementation for SD cards that uses the same FatFS API, and it seems I still have the set of SPI compatible GPIO pins available for this, so it would be stupid and ignorant of me not to do it. But then of course it's going to be an either-or situation. 

[HiassofT]

On 7/27/2024 at 10:15 PM, woj said:

I went through a small battle of figuring out after how many command frame receiving desyncs I should toggle the speed. SIO2BSD has 4 I think, SDrive has one = toggle on first desync.

In a lot of cases, eg when only having to deal with standard SIO speed at ~19kbit and highspeed at 50-120kbit you can make the switch at the first hard failure:

 

A command frame has to contain exactly 5 bytes, so if you receive less than or more than that while COMMAND is asserted you know you are off.

 

It's also worth to check for framing errors (i.e. the stop bit being 0 instead of 1) - if you got more than one of them during a command frame it's also a good indication you are off.

 

If you got 5 bytes but the checksum is wrong or you got a single framing error then you might stay at the current speed for another try as this could have been a temporary glitch. If the next frame fails, too, then switch speed.

 

This is about how I implemented in AtariSIO and it's been working reliably for the past 20+ years

 

so long,

 

Hias

[Peri Noid]

A new pico arrived: https://www.raspberrypi.com/products/raspberry-pi-pico-2/

Maybe it has enough power?

[woj]

16 minutes ago, Peri Noid said:

Maybe it has enough power?

Thanks! So far I do get enough power, though the ~500k of RAM would make some things easier (for instance, I currently need to limit the number of files visible to the user in each directory). Too bad it does not expand on Flash too much, with the new amount I am still fine with 250k of RAM, very fine. But for the version with SD card slot it surely would be better.

[woj]

Back on the project after a break, got some new toys, Pico2 and Pimoroni Lipo Pico, the latter has 16MB Flash, so it makes a decently sized Flash disk (the new Pico has 4MB, which is also a small improvement over the old one), the LiPo function is also nice, it would enable running the device off a battery. I also got the ATX test suite, but still had no time to look into it. Got the Pico2 to work today and already discovered a quirk - it does not like a very large number for DMA channel transfer repeat count (something I used for a nice solution for the ATX rotation counter) that worked with Pico 1, it made the core trying to set up the DMA channel freeze. One other small disappointment with Pico2 is that it still has the MicroUSB connector (I guess this is for backwards compatibility). The other thing is an SD card module, I want to have this as an option instead of the internal Flash. Because of this, it seems I will have to rewrite a bit of the program to read the directory contents (it is currently limited to something between 256-2048 files in one directory, depending on how long the file names are, this was fine (and fast) for a small Flash drive of 1-15MB, but it is going to be hugely limiting for GB-s size range of SD cards).

[woj]

The number of problems I tripped over in the last two weeks with this project is immense, but I think I arrived at a steady place again. I was finally able to run the ATX compatibility test @DjayBee, the results are in line with what i would expect considering (a) that I based the implementation entirely on SDrive-Max's one, (b) I currently only test on a PAL machine (I need SIO plugs to be able to re-plug back and forth and these are still in the mail), and (c) see below. Now I can concentrate on the ATX compatibility for the time being, the only other thing left (software wise) is SD-card interface.

 

But, among many things that kicked me in the past days are two that stand out. First, I struggled with understanding the existing implementations of the disk rotational counter (I referred mostly to SDrive-Max and atari_drive_emulator from MiSTer / EclaireXL projects). The more I looked into SDrive-Max one, the more I am convinced it is broken and works by chance / luck. In particular, the counter is based on a hardware clock running at twice the speed of the ATX assumed disk ticker, but that counter is actually counting more than 26042 rotations before it wraps back to 0 (or 1, that was another thing that gave me headache), namely 32768, or so i concluded with my understanding of the AVR. All the other procedures in the ATX implementation assume that the counter runs in a round fashion only until 26042. Long story short, because of details like this my implementation is not 100% the same as SDrive-Max one and it remains to be carefully checked how this particular thing should be (re-)done. For example, I already know that that to wait for one full retry rotation I have to wait for the first half first, then for the second half. If I wait for the whole rotation in one call, the Pico is fast enough to catch the current rotation as the target one. Just tagging @kbr here for information, I will get to the bottom of this I hope.

 

The second thing - if you are into the news about new devices and technology you might have seen there are some discussions about the silicon bug in the new Pico2s and the "Errata E9". At first I though it would not apply to my project and that "surely they would not release a product with such a serious bug in it", well, guess what, they did! The story now deepens because the initial diagnosis that the GPIO pins can be stuck high only if one uses internal pull down resistors is not true, without touching the pull downs there are also problems. All in all, this problem in both flavours (with and without pull-downs) has hit me (spent endless hours chasing software bugs that were not there) and it is now resolved with a suitably (not just any) selected external pull-down resistor for the motor activity sensing line (with the downside of increased power consumption if I understand it correctly). The really (un)entertaining part of this are the messages from the RPi engineers on the forums saying that they still investigate the problem, that they were not able to reproduce some parts of it, that it does not seem to be serious, users are doing something wrong, one should not design circuits that rely on internal pull down resistors, and they would not fix it anyway because it now will cost too much, and so on and so forth. Sadly, I see some resemblance to what has been going on with SIDE3 recently @flashjazzcat, it seems the big players have the same problems as the smaller ones  

 

 

Edited September 6 by woj

[phaeron]

13 hours ago, woj said:

The second thing - if you are into the news about new devices and technology you might have seen there are some discussions about the silicon bug in the new Pico2s and the "Errata E9". At first I though it would not apply to my project and that "surely they would not release a product with such a serious bug in it", well, guess what, they did! The story now deepens because the initial diagnosis that the GPIO pins can be stuck high only if one uses internal pull down resistors is not true, without touching the pull downs there are also problems. All in all, this problem in both flavours (with and without pull-downs) has hit me (spent endless hours chasing software bugs that were not there) and it is now resolved with a suitably (not just any) selected external pull-down resistor for the motor activity sensing line (with the downside of increased power consumption if I understand it correctly).

This is pretty sad, as the RP2350 seemed to fix so many of the shortcomings of the RP2040, but easy to use GPIO pins are really important for Pico/Pico2. Certainly seems more annoying to work around than a lot of things I've seen in the RP2040 and RP2350 errata.

 

Thankfully, the Atari 8-bit chipset doesn't have much in the way of annoying bugs. The ANTIC DMA clock running wild when you hit DMACTL or HSCROL at the wrong time is probably the most serious one, and is harder to diagnose than to work around.

13 hours ago, woj said:

The really (un)entertaining part of this are the messages from the RPi engineers on the forums saying that they still investigate the problem, that they were not able to reproduce some parts of it, that it does not seem to be serious, users are doing something wrong, one should not design circuits that rely on internal pull down resistors, and they would not fix it anyway because it now will cost too much, and so on and so forth. Sadly, I see some resemblance to what has been going on with SIDE3 recently @flashjazzcat, it seems the big players have the same problems as the smaller ones

The cost issue is the main one I sympathize with, for both large and small outfits. It is expensive to fix hardware. It's why I like to work with software, it's easier to fix and there are usually no bad smells when it breaks.

 

Hobbyist hardware is unfortunately inherently prone to risk, because if hardware is being sold hand-made or in small runs at low prices and margins, you just can't have the same expectations regarding hardware defects, especially design issues. One case that I ran into was a cart that had a bank-swap between circuits running on asynchronous clocks. The first revision turned out to have a bug that could rarely cause blown write cycles on a bank swap due to a lack of synchronization. Unfortunately the first major program I wrote for it swapped banks constantly and caused the issue to manifest in seconds instead of days. But there was no easy fix that could be applied to the affected hardware revision, it was a fundamental problem in the design that couldn't be fixed with a firmware update or soldering on a few components.

 

Communication is a different issue. One of my pet peeves is a common scenario on many discussion forums: a person who posts all the time, clogs up threads by replying to virtually every post with a subtly condescending tone, and is a master at baiting people into crossing the line into ToS violations, such as by calling other people toxic  -- and the moderators let the person get away with it repeatedly. Better forums have strong enough moderation and rules to prevent this and don't make exceptions for anyone, even those close to the moderators.

 

[flashjazzcat]

On 9/6/2024 at 3:12 PM, woj said:

Sadly, I see some resemblance to what has been going on with SIDE3 recently @flashjazzcat, it seems the big players have the same problems as the smaller ones  

Yeah: SIDE3 definitely received a Crowdstrike update.

 

Worth remembering that in the realm of 'hobbyist hardware', SIDE3 was produced by a manufacturer with a professional pick and place facility and aspirations to become the Poland's answer to PCBWay, and sells for over a hundred quid. If the hardware was released with a hard-to-correct firmware defect after all members of the 'team' had taken some time to carry out structured testing, fair enough: shit happens and I'm sure most people would be reasonable about it, but that's not how it played out, as we know.

[woj]

So I think I nailed the ATX support, with the help and support of @DjayBee, cheers! I get fully consistent results with those of Altirra, though for now I was only able to test PAL thoroughly. Despite good and fully working result I still struggle with understanding one timing parameter, that is the one of track re-calibration on an entirely missing sector. Altirra has it well documented in the sources, just that it is not that straightforward to transplant the logic into my code, I need to understand it better. For now I have it roughly approximated and the "visual" timing behavior matches sufficiently well, and this timing behavior actually does not affect any of the tests provided, so I will probably never know if I got it fully correct (not without writing my own test program for the Atari, which I am bit too lazy to do).

 

One particular parameter was also a bit of a mystery, the CRC checksum calculation constant was defined at 2ms in the SDrive implementation, regardless of the emulated drive. Altirra sources suggest (if I found the right thing) 5136us for 810 and 270us for 1050. Especially the last one seems suspiciously low, and depending on "how" I execute this delay (never mind about the "how") one particular timing result in the tests is straying a bit off from the Altirra reference results. Probably need to recheck that again.