New HDX crossplatform server, and HDX hardware design discussion

[ckoba]

(per arcadeshopper, I'm moving this here)

 

I've installed a HDX in my TI RS232 card, and have it running (for some value of "running").

 

It does not use the cables recommended by Fred, because those cables did not work in any configuration I tried. From a design perspective, it makes no sense to add what amounts to two crossover cables when the TI is already correctly wired as a DCE.

 

The Windows HDX server does work under Linux via wine, but it is very picky about the serial port hardware that is used. From what I hear, the same problem exists on Windows itself.

 

To address the software issue and the serial port compatibility issue, I have written a quick proof-of-concept of a HDX server in python (attached to this post). This implements only the read and write sector commands for DSK2PC, as well as the detection routine. This appears to run well under Linux and MacOS, with both hardware and USB serial connectors. Windows is untested; volunteers are welcome.

 

I'd like to open discussion about a HDX redesign. I did not have a pleasant time getting this running, and I'm a professional engineer. I doubt that a casual user would have the patience or experience to troubleshoot and remedy the problems faced, let alone write a HDX server to test theory against reality. My analysis of the HDX hardware, software, and protocol are at these links: http://atariage.com/forums/topic/243766-bin-repository-images-for-burning-cartridges/page-6?do=findComment&comment=3437849, http://atariage.com/forums/topic/243766-bin-repository-images-for-burning-cartridges/page-6?do=findComment&comment=3438355, and especially the last half of http://atariage.com/forums/topic/243766-bin-repository-images-for-burning-cartridges/page-6?do=findComment&comment=3438373

 

For those who don't want to wade through a few posts to clean the wheat from the chaff, I believe the following to be true:

 

* The HDX board should be reworked. The RAM should be replaced with a DSR ROM and a small amount of scratchpad RAM, glue logic to select the right chip for the right address, and the battery backup be eliminated completely.

* Doing this would remove the need to cut the IRQ line. It would also keep the battery from trying to power the entire RS232 board while powered down, resulting in a quickly dead battery (see: the MiniMemory effect)

* The HDX protocol exists as three separate entities: the one on Fred's site, the one in the HDX server helpfile, and the actual wire protocol. Discrepancies range from omission of checksums from the probe routine, errors in portions of the read-sector spec (0x00 versus 0xff in byte 2 of the response), and so forth.

 

As I've said elsewhere: although I have the skill to redesign the HDX board, I lack the motivation and resources to do so. For the latter I would need source code to the DSR to cleanly split the DSR code proper (EPROM) from the buffers (RAM). For the former I would need a personal use case for this that the HxC does not currently address, or at least a threshold number of people that want a fixed HDX board to make this worth my time.

 

Python HDX server that works for DSK2PC attached. I've started the ball rolling and mooted the point; would anyone else like to weigh in? Fred, are you willing to publish any of your source?

 

(edit: always regression-test your code after refactoring, folks)

hdx-python-v002.zip

Edited February 12, 2016 by ckoba

[+jedimatt42]

there has been repeated expression of interest in a lighter weight / cross platform HDX server. I recall folks wishing they could run it on something like an ARM based Raspberry PI.

 

From what I recall, strolling through the history on this forum, most people seem to be content/happy with the HDX board/mod to the RS232 card as is.

 

I'm sure there has been lots of debate over the years regarding if the TI RS232 port being DCE instead of DTE is proper or not. As hobby craft, I personally like having the y-splitter per Fred's diagram that yields DTE PC DB9 ports, as well has having a DB9 null-modem cabled up. This, has little to do with the design of HDX.

 

As for use cases for HDX, I actually have never used the DSK2PC function. I have Gotek drives, but only use those for software that requires being on DSK1. I use the HDX for all other data storage, using the file on a disk model. Presently the storage is on my PC gaming rig, which heats my room excessively even while doing little more than hosting files for the TI. File access is faster than with the floppy emulators ( at least with my single density floppy controller ). Downloading a file from this forum, and dropping it into your HDX storage server is so much easier than transferring a .DSK to an .HFE and then putting that on the sdcard/usb-stick.

 

I would actually like to see the server running on a raspberry pi (2), I would configure it to use a network share as storage, with that share also used by Classic99 on my laptop, when desired. I don't actually use emulation very often, but when I do I wish I had already set up the use of my network storage.

 

Of course, lacking a cross platform server, I've been tempted to just throw money at it:

http://smile.amazon.com/dp/B00X4O6GRK?psc=1

or an intel compute stick, or similar product.

 

A raspberry pi is considerably cheaper.

 

-M@

[+jedimatt42]

Oh, and pretty awesome of you to go digging into the HDX protocol taking it as far as you did!

 

-M@

[ckoba]

there has been repeated expression of interest in a lighter weight / cross platform HDX server. I recall folks wishing they could run it on something like an ARM based Raspberry PI.

I tested the DSK2PC python thingy this afternoon on the BeagleBoard, and it was perfectly happy to transfer using a crappy Chinese no-name USB port. So, successful proof-of-concept.

 

If I can get accurate protocol specs, the rest of it should be a cakewalk. If I can't, it'll take a little longer. I have a bit of a feel for how Fred implemented this now, so what's left is the transmogrification of the PC-side directory parsing into the TI format. That's going to take a lot of serial sniffing.

 

From what I recall, strolling through the history on this forum, most people seem to be content/happy with the HDX board/mod to the RS232 card as is.

*chuckle* I think maybe because they didn't really look at the circuit, nor consider the ramifications of, say, having an HDX instead of (rather than along with) a floppy drive.

 

As for use cases for HDX, I actually have never used the DSK2PC function. I have Gotek drives, but only use those for software that requires being on DSK1. I use the HDX for all other data storage, using the file on a disk model. Presently the storage is on my PC gaming rig, which heats my room excessively even while doing little more than hosting files for the TI. File access is faster than with the floppy emulators ( at least with my single density floppy controller ). Downloading a file from this forum, and dropping it into your HDX storage server is so much easier than transferring a .DSK to an .HFE and then putting that on the sdcard/usb-stick.

Right, so you have a different use-case. Although I think you'd benefit from having the DSR in ROM. How frequently do you have to refresh that DSR? What's the RAM chip on it, a 6264-alike or something else?

 

I would actually like to see the server running on a raspberry pi (2), I would configure it to use a network share as storage, with that share also used by Classic99 on my laptop, when desired. I don't actually use emulation very often, but when I do I wish I had already set up the use of my network storage.

It will (and does already, partially). I've already committed to finishing off (what I'm now calling) hdx.py. No ETA, depends on a lot of private things going on at the moment.

 

Oh, and pretty awesome of you to go digging into the HDX protocol taking it as far as you did!

No worries, that's what I do. New thing doesn't work? Well, hell, I've got this oscilloscope and a soldering iron just sitting here doing nothing ...

[HackMac]

Nice thread, nice idea!

 

For me, I don't have a TI system that runs with all components I own for now, so I haven't try out the HDX things with my Mac. The other reason why I don't have HDX running is the missing software. Which I have to do. My plan is to integrate a HDX server into the TI-Disk Manager with version 4. (The roadmap shows all plans for each main version number.) Before I'll start this, I'll like to get in contact with Fred to ask him because of any copyright issues. The greatest "problem" for implementing the HDX server into the TI-Disk Manager application is that I first have to build the hardware on the TI side.

Additionally I had plans for extending the HDX protocol in that way, the TI can also act as a server simultaneously.

 

For me it is okay to have some glue logic to mount an extra DSR ROM. So far as I know there a no other modifications to the RS232 board. Right? So this ROM can be placed anywhere. So I think it's the wrong way and we are wasting time to build a new extra board for rebuilding that we already have. Perhaps the DRS can be implemented in GPL to run from a cartridge (with support routines in the >6000 ROM area).

 

At last I ask for the implementation detail for the cross platform software. Why using a snake? Python is not very native on Windows platforms. Linux is not the problem and Mac OS is based on Unix. Although I'm not a fan of C/C++ (also not of Python), I would prefer this variant, because for a implementation in C, there is no need to install additional frameworks or runtime environments. But it is easy to wrapping this C code for using it in other environments like Java, Python or Swift based.

So, before the discussion starts, what the 'best' programming language is. We should disregard the choice of the language. It is a quasi meaningless detail. First we should ask what is to implement in software, then define the architecture (how to implement). We can do (for example) a framework (in contrast to a library) that is really independent of any OS, with any nice design patterns like delegates, observers, notifications, etc.. So different projects can take the HDX framework to fiddle its preferred UI around.

 

That's my first thoughts for now.

[Willsy]

Python is completely trivial to install in windows, and often comes pre-installed on Linux platforms. Is not a big deal.

[HackMac]

Python is completely trivial to install in windows, and often comes pre-installed on Linux platforms. Is not a big deal.

 

Yes, that could be, but you have to do extras... This is not really plus point. Using applications without any (additionally) installations, that is a user friendly behavior! Just copy an application to any location on the 'PC' and just start. Thats all a user should do.

[+Vorticon]

There was really nothing "extra" to do to install Python in Windows. The installer takes care of all the details. As Willsy said, it is trivial and doesn't get much easier than that.

From my end of things, the HDX design may not be optimal from an engineering point of view (I have to take the original comments in the first post at face value since I am not even remotely an engineer), but frankly it has worked for me and most other users here pretty much flawlessly and has made the transfer of files and whole disk images between the TI and PC absolutely trivial. I still remember the days before the HDX when I was using communication programs on both ends and a null modem cable to move files around one at a time. Yuk!

Given the above, I think we owe Fred the courtesy of perhaps a private discussion before trashing the HDX design publicly. I for one am very grateful for his efforts...

[HackMac]

As I wrote (and I think you can't read if you write this) all a user have to do is starting the application. Nothing else.

The extra is in fact installing something (except copying the program), regardless how easy it is.

That is the fact, why all wants Apple products. They keep it simple!

[Omega-TI]

Chris,

While I like my HDX, I'm open to a new product if it can offer greater abilities (user mentality again).

 

From what you've written so far:

 

1 - Speed will be increased.

2 - No more battery backup.

3 - No trace cuts! I agree with you on this aspect whole heatedly.

I was pulling my hair out when I first got the thing and could not get a

communications program to run because of the interrupt line.

 

Will you be taking suggestions from 'users' at this early stage of development? I have a couple of ideas, well maybe a few... if you give me enough time to think on it!

[Willsy]

As I wrote (and I think you can't read if you write this) all a user have to do is starting the application. Nothing else.

The extra is in fact installing something (except copying the program), regardless how easy it is.

That is the fact, why all wants Apple products. They keep it simple!

But you only install it once! Just like you install classic 99 once or win994a once. After that you double click a batch file and it's running. Emulator users are well versed in having to manage disk files and disk images and ROM images. Running a python program will not be a problem! :-) Edited February 12, 2016 by Willsy

[+InsaneMultitasker]

I noticed mention of 115,200 in one of the linked posts. If memory serves correctly, the 9902 frequency cannot by set to this value; the closest value you can achieve is ~125,000. I ran into a similar situation with 57,600 where the 9902 frequency was nearly 10% above expected value, and most modems and serial connections were unable to exchange data properly. Have you successfully tested 115,200 with the 9902?

 

The HDX DSR uses roughly 3.5K for code and the remainder for buffers, workspace, and program data. File buffers are allocated for the path.filename and control information. Unlike the disk controllers, buffers are not required for the VIB and related control blocks. A receive/transmit buffer is allocated for both the control frames and the data. There are two 4K banks that reside at 0x4000-0x4fff, controlled by CRU bit.

 

Have any of the concerns been brought to Fred's attention? Perhaps he has some solutions already in mind or has decided they are not worth pursuing. Only he would know...

[HackMac]

But you only install it once! ... Running a python program will not be a problem! :-)

Yes, of course, if it is a reason for you to use python (or .NET or other), only for that you can install additional software, you can do it. But it is nevertheless definitely not a pro for selecting this as the preferred programming language for making cross platform software. Making software that depends on various other (software) components doesn't make it better.

Of course you are one of these people who want to buy a car and 'install' the motor for yourself instead of getting in and push the pedal to the metal. The advertising tells you: Hey, you only need to install it once (hooray!) and it is so easy to install (yay!), that you wish you had a second car only for install the motor again.

Edited February 12, 2016 by HackMac

[+arcadeshopper]

Things I'd like to see HDX do:

 

CLOCK device rather than HDX1.TIME so it works with existing sw.

Ability to mount/enter disk images from the TI side so either we could go in via dm2k like you can in tidir just enter the disk image and it comes up, or mount it like HDX1.tidisk.dsk SHRUG

 

I've mentioned both of these to Fred in the past. He added the HDX1.TIME at our request.. but that doesn't work with most sw

[HackMac]

I noticed mention of 115,200 in one of the linked posts. If memory serves correctly, the 9902 frequency cannot by set to this value; the closest value you can achieve is ~125,000. I ran into a similar situation with 57,600 where the 9902 frequency was nearly 10% above expected value, and most modems and serial connections were unable to exchange data properly.

I just remember, that USB dongles (or the driver of those), which are used on Macintoshes because there are no other serial interfaces than USB, doesn't use the hardware handshake lines. This could be problematic as well.

 

A new implementation of a HDX server should handle a wide range of bit rates, so why not using 9600 bit/s?

[+arcadeshopper]

I just remember, that USB dongles (or the driver of those), which are used on Macintoshes because there are no other serial interfaces than USB, doesn't use the hardware handshake lines. This could be problematic as well.

 

A new implementation of a HDX server should handle a wide range of bit rates, so why not using 9600 bit/s?

 

Current hdx does 38400 .. why go slower? I will have to try it on the mac and see if it works the same in wine as it does in linux with my serial ports. I picked up a new one recetly off amazon that has lights on it for the signals.. Will try it out http://www.amazon.com/GearMo-Adapter-Indicators-Windows-Support/dp/B00AHYJWWG/ref=sr_1_13?ie=UTF8&qid=1454627766&sr=8-13&keywords=rs232+to+usb

[+arcadeshopper]

Nice thread, nice idea!

 

For me, I don't have a TI system that runs with all components I own for now, so I haven't try out the HDX things with my Mac. The other reason why I don't have HDX running is the missing software. Which I have to do. My plan is to integrate a HDX server into the TI-Disk Manager with version 4. (The roadmap shows all plans for each main version number.) Before I'll start this, I'll like to get in contact with Fred to ask him because of any copyright issues. The greatest "problem" for implementing the HDX server into the TI-Disk Manager application is that I first have to build the hardware on the TI side.

Additionally I had plans for extending the HDX protocol in that way, the TI can also act as a server simultaneously.

 

For me it is okay to have some glue logic to mount an extra DSR ROM. So far as I know there a no other modifications to the RS232 board. Right? So this ROM can be placed anywhere. So I think it's the wrong way and we are wasting time to build a new extra board for rebuilding that we already have. Perhaps the DRS can be implemented in GPL to run from a cartridge (with support routines in the >6000 ROM area).

 

At last I ask for the implementation detail for the cross platform software. Why using a snake? Python is not very native on Windows platforms. Linux is not the problem and Mac OS is based on Unix. Although I'm not a fan of C/C++ (also not of Python), I would prefer this variant, because for a implementation in C, there is no need to install additional frameworks or runtime environments. But it is easy to wrapping this C code for using it in other environments like Java, Python or Swift based.

So, before the discussion starts, what the 'best' programming language is. We should disregard the choice of the language. It is a quasi meaningless detail. First we should ask what is to implement in software, then define the architecture (how to implement). We can do (for example) a framework (in contrast to a library) that is really independent of any OS, with any nice design patterns like delegates, observers, notifications, etc.. So different projects can take the HDX framework to fiddle its preferred UI around.

 

That's my first thoughts for now.

 

Why do you have to build the hardware? Wouldn't it work with cfhdxs?

 

Greg

[HackMac]

Greg,

 

I don't know what cfhdxs is...

 

I think your adapter is not an Prolific PL2303-based adapter, isn't it? This chip is used in my (old?) GMUS-03. I got two of them.

There are several open source driver for it.

 

 

[edit]

I found out that there is a FTDI Chipset Inside: FT232RL

Your adapter has two 28 pin chips in. Mine only one 28 pin. But the PL2303 has pins for handshaking. So I think the USB driver(s) doesn't take advantage of these feature.

[/edit]

Edited February 12, 2016 by HackMac

[+Vorticon]

As I wrote (and I think you can't read if you write this) all a user have to do is starting the application. Nothing else.

The extra is in fact installing something (except copying the program), regardless how easy it is.

That is the fact, why all wants Apple products. They keep it simple!

I assure you I can read. And I can also assure you that my education level far exceeds yours. So please spare us your idiotic comments and try to respond politely instead.

[HackMac]

I assure you I can read. And I can also assure you that my education level far exceeds yours. So please spare us your idiotic comments and try to respond politely instead.

Oh, yes indeed. I'll do like you do, sir!

[ckoba]

Replying to everything in one post:

 

 

For me it is okay to have some glue logic to mount an extra DSR ROM. So far as I know there a no other modifications to the RS232 board. Right? So this ROM can be placed anywhere.

 

Good point, this can be done on the UberGROM. It has a UART can can (obviously) do DSR hooks. The DSR would need to be heavily rewritten, which is why source code for the DSR would be nice.

 

The choice for the serial board, as far as I can tell, was to leverage the hardware serial drive goop.

 

 

So, before the discussion starts, what the 'best' programming language is. We should disregard the choice of the language.

 

No, you're conflating two separate aspects of the project -- the software server side, which will be developed in python because it is multiplatform and interpreted. The hardware side is the HDX hardware refresh.

 

You're free to write your own server software, but the decision for my software side of this project has been done already. If you want to volunteer to write the software, then fine -- I'll step back and let you do it. Software developer gets to decide the language

 

 

From what you've written so far:

 

1 - Speed will be increased.

2 - No more battery backup.

3 - No trace cuts! I agree with you on this aspect whole heatedly.

I was pulling my hair out when I first got the thing and could not get a

communications program to run because of the interrupt line.

 

Will you be taking suggestions from 'users' at this early stage of development? I have a couple of ideas, well maybe a few... if you give me enough time to think on it!

 

Sure. That's what this thread is about. I wouldn't expect too much more speed out of it (if any); the bottleneck isn't necessarily the serial line speed, but the protocol overhead and flushing of buffers.

 

 

 

I noticed mention of 115,200 in one of the linked posts. If memory serves correctly, the 9902 frequency cannot by set to this value; the closest value you can achieve is ~125,000. I ran into a similar situation with 57,600 where the 9902 frequency was nearly 10% above expected value, and most modems and serial connections were unable to exchange data properly. Have you successfully tested 115,200 with the 9902?

 

The HDX DSR uses roughly 3.5K for code and the remainder for buffers, workspace, and program data. File buffers are allocated for the path.filename and control information. Unlike the disk controllers, buffers are not required for the VIB and related control blocks. A receive/transmit buffer is allocated for both the control frames and the data. There are two 4K banks that reside at 0x4000-0x4fff, controlled by CRU bit.

 

Have any of the concerns been brought to Fred's attention? Perhaps he has some solutions already in mind or has decided they are not worth pursuing. Only he would know...

115200 was tested over a loopback plug between /1 and /2. I didn't time it; it could have drifted, but the point of the test when I was running it was to see how fast they could go, and the speed was a tad higher than 115.2 IIRC. The point was that the HDX (and the card) are capable of greater than 38400, although 38400 may well be the highest safe speed for half-duplex no-flow-control.

 

I have not personally brought any concerns to Fred except for my request in the first post. I don't know Fred and he doesn't know me; I want to keep this discussion in the public forum.

 

 

CLOCK device rather than HDX1.TIME so it works with existing sw.

 

DSR change.

 

 

Ability to mount/enter disk images from the TI side so either we could go in via dm2k like you can in tidir just enter the disk image and it comes up, or mount it like HDX1.tidisk.dsk SHRUG

 

DSR change. The primitives appear to exist in the protocol, insofar as you can read arbitrary blocks to and from a disk image. In fact, that's exactly what DSK2PC does, and I see no reason it can't be extended as you suggest.

 

But, again, DSR change.

 

 

I just remember, that USB dongles (or the driver of those), which are used on Macintoshes because there are no other serial interfaces than USB, doesn't use the hardware handshake lines.

 

HDX doesn't use hardware handshake lines, except to check for an asserted DTR on (at least) port 2 at boot time to find the server. After that, there is no handshaking at all, either hardware or software. I've actually unplugged the cable in mid-transfer from one machine, plugged it into another, and the transfer didn't stop. Whether that is a good thing or a bad thing is up for debate, too. (I think it's a bad thing)

 

Whatever the problem is with USB serial dongles is, it is not due to "hardware handshaking". It's a server-side or OS-driver side problem. As I've said, my skeleton python implementation runs well with various USB serial dongles (FTDI, pl2303, and one I didn't recognize -- ttyACM0) on a (expletive deleted) rPi clone.

 

That's my two yen for this round of debate, I'm mostly away for the weekend, interested in seeing replies when I get back

 

-- Chris

 

Edit: multiquotes are a PITA.

Edited February 12, 2016 by ckoba

[+InsaneMultitasker]

115200 was tested over a loopback plug between /1 and /2. I didn't time it; it could have drifted, but the point of the test when I was running it was to see how fast they could go, and the speed was a tad higher than 115.2 IIRC. The point was that the HDX (and the card) are capable of greater than 38400, although 38400 may well be the highest safe speed for half-duplex no-flow-control.

That makes sense; two 9902s using the same frequency settings should be able to successfully communicate with one another.

 

The challenge occurs when you connect the RS232/9902 to a device besides another of its kind. It is for this reason the 38,400 barrier hasn't been "broken" - not because the RS232/9902 cannot handle higher speeds, but because those higher speeds don't match and sync to the expected, commonly defined rates, i.e., 57.6K and 115.2K. I'm all for speed so if there is a way to do it, I'm all for it.

[HackMac]

Good point, this can be done on the UberGROM. It has a UART can can (obviously) do DSR hooks. The DSR would need to be heavily rewritten, which is why source code for the DSR would be nice.

Well, not exactly what I try to obtain. I hoped to do a HDX connection without any new/extravagant hardware. A cartridge with EPROMS in GROM and ROM area has everyone of us in the drawer. So, on TI side the minimal configuration should be a cartridge which holds the software. An extra UART is a bit too special I think.

 

No, you're conflating two separate aspects of the project -- the software server side, which will be developed in python because it is multiplatform and interpreted. The hardware side is the HDX hardware refresh.

I do differentiate between these to sides. My interests lies on the software side, each on Personal Computer and the TI. I have no interests in new extra hardware.

On the software side my idea is NOT to use any interpreting language but rather to find an independent platform solution, not a multi platform solution. My proposal is to create a core, a framework that can be used in a Python/Java/.NET/PERL or other environment.

 

You're free to write your own server software, but the decision for my software side of this project has been done already. If you want to volunteer to write the software, then fine -- I'll step back and let you do it. Software developer gets to decide the language

With such a solution, I mentioned above, we are all free to use whatever we want. A server software should have different layers. The most abstract layer can be implemented in any environment (I named above), but we should decide to get the maximum flexibility for the most common layers. I my opinion we can't reach this goal while insisting on Python.

 

HDX doesn't use hardware handshake lines, except to check for an asserted DTR on (at least) port 2 at boot time to find the server. After that, there is no handshaking at all, either hardware or software. ... Whether that is a good thing or a bad thing is up for debate, too. (I think it's a bad thing)

I think so too.

 

Whatever the problem is with USB serial dongles is, it is not due to "hardware handshaking". It's a server-side or OS-driver side problem. As I've said, my skeleton python implementation runs well with various USB serial dongles (FTDI, pl2303, and one I didn't recognize -- ttyACM0) on a (expletive deleted) rPi clone.

Ok, I have tested my dongles while I implemented the SerialBride in TI-Disk Manager and I did my own disk image transfer program on the TI several years back. It runs, but I can't get hardware handshaking working. I'll have a closer look into the driver, I think. I'll feel much better when handshaking works, then I'll be able to get my serial EPROM burner working on my Mac.

[+Vorticon]

Oh, yes indeed. I'll do like you do, sir!

Look I'm not interested in a war of words. Let's agree to stay away from the snide comments and it will be a far more constructive discussion. Friends?

[HackMac]

We were never enemies, from my side. But I'm not on earth to be everybodys darling. Your last post tells much about you, it disgrace yourself.

Next time, when you have a problem with me, you can pm me.

Edited February 13, 2016 by HackMac