PLATOTerm for Atari - Working out tgi and ser issues.

[tschak909]

Awesome

 

Thanks for testing, btw!

 

-Thom

Edited September 13, 2018 by tschak909

[_The Doctor__]

looks like adjusting the receive window within the buffer according to baud rate will tailor the timing of xon xoff to do something useful.

 

if you go back and adjust for each of the previous revisions you've done could be one that looks the best can work again not lose data.

 

might make a set up entry to adjust the high and low water marks so you can play with this on the fly through the different versions you've made, It just might surprise you.

 

Could be a good idea to leave it in not only to test but to allow folks to adjust for their interfaces particular timings.(perhaps a tuning utility)... only thing is explaining about high and low water for the buffer and the 'window' between the two, we spent almost no time floating (I didn't see black) the idea is on data flow to get a cycle of green black red green - green black red green.

 

I think this illustrates what I was trying convey about flow and need for Atari to make pretend it has the extra bytes of a real uart.

 

I am pretty sure now that this is seen, with a little fiddling 4800 should be possible. The lantronix I used was set to software flow. without it being set the screen became a real mess Further tuning of the two values will get us the sweet spots we need.

Edited September 13, 2018 by _The Doctor__

[mr-atari]

Success at last.

 

I had to hack into your plato build to make it work....

-just to mention-

 

Patched CONTROL-X to sending out $0A.

Patched startup, not clearing page 6.

 

Pity to see that this build stills assumes the config-files are on D1, were as I load plato from D8:

Something to look into.

 

So, with the patched version:

I can setup my ESP8266 and use LiteDOS/ESP8266-driver.

Connect and use plato.

 

My driver sends out Xoff also, but is not successful enough.

How many bytes are poring in after I send $13?

To me it seems, no I measured, transmission is not stopped when I send $13.

 

Graphics look better, the issues I saw are gone.

 

What next?

Can you make things work for me, without me hacking into your build?

[tschak909]

Of course, I will make CR send an appropriate LF (line feed). Let me do that when I get home from taking my daughter to school...

 

The issues you're experiencing are what I've tried to explain in multiple forums, over and over. XON and XOFF are an in-band signalling protocol, and thus are at the mercy of connection asymmetries as well as buffering happening at both ends of the connection.

 

When you have a host computer that can send much faster than the terminal can receive, you will have a situation where trying to send the XOFF character will happen after the host has sent a bunch of data, already,

 

So you have to implement a local solution to help mitigate the problem, if you want to continue using XON or XOFF.

 

A better solution is to implement out of band signalling, what is typically called hardware handshaking, or RTS/CTS, in RS232 terminology, as these handshaking techniques are not affected by massive shifts in the transmission or reception queues.

 

-Thom

[tschak909]

@mr-atari - I have added CR/LF processing, which you need, to the TTY mode (which happens before you connect to PLATO)

 

plato-20180913-redgreen-xon32-xoff128-cr-lf.atr

 

Let me know if this works okay.

-Thom

[a8isa1]

Of course, I will make CR send an appropriate LF (line feed). Let me do that when I get home from taking my daughter to school...

 

The issues you're experiencing are what I've tried to explain in multiple forums, over and over. XON and XOFF are an in-band signalling protocol, and thus are at the mercy of connection asymmetries as well as buffering happening at both ends of the connection.

 

When you have a host computer that can send much faster than the terminal can receive, you will have a situation where trying to send the XOFF character will happen after the host has sent a bunch of data, already,

 

So you have to implement a local solution to help mitigate the problem, if you want to continue using XON or XOFF.

 

A better solution is to implement out of band signalling, what is typically called hardware handshaking, or RTS/CTS, in RS232 terminology, as these handshaking techniques are not affected by massive shifts in the transmission or reception queues.

 

-Thom

I saw these notes in Bo Zimmerman's Zimodem (v3.4) help documentation. I wonder if ATF2 is a potential solution.

 

"ATF0 : Turns on rts/cts flow control.

ATF1 : Turns on xon/xoff flow control.

ATF2 : Turns on xon/xoff flow control, sets XON mode (if necessary), and, in command mode, will immediately go to XOFF when a single connection packet is received. This is very useful when the client wants to ensure it only receives one packet to process. You can think of this as an alternative way to use xon/xoff by having XOFF automatic between packets.

ATF3 : Similar to ATF2 except that the default is XOFF, and, in command mode, a XON code from the user will immediately trigger either an empty packet response [ 0 0 0 ], or a real packet if one is available. After this, as in ATF2, XOFF is automatically set.

ATF4 : Turns off flow control for command mode"

 

-SteveS

[_The Doctor__]

Just to be clear, software flow control is a multipart situation, There were two sets of numbers at one time for is, The mode we are using is supposed to be local device xon xoff, the other set was for remote on off or for host/server control. The method everyone can use is to assert the stop early and in the buffer high water mark, the buffer continues to fill in the computers buffer as the surge is absorbed, the modems buffer fills as the x stops then sending and the modems buffer will hold and continue to fill. As the low or empty computer buffer singnals continue or send more it refills itself from the modems buffer, this is a bucket brigade and does work.

 

I have went a step further in my own configuration, now only does my lantronix act on local xon xoff, but it sends it to the remote machine as well, this makes BBS work very responsive and doesn't seem to hurt plato,

 

the buffer is 256 byte long in Plato,

the please send is happening at 32 bytes left but may be fine slightly lower depending on baud rate

the please stop I've got to much to deal with is happening at 128

that allows the surge to be absorbed by the remaining buffer area

if it passes that corruption will occur.

 

the area between send and stop is the sweet spot window in the buffer, we want the buffer to send more when a complete plato command or char bitmap (font whatever) is left, my mind keeps saying that's 24 but I don't remember, if it is the send more should happen at 24 or 48. the 128 was chosen based on watching the serial line (I couldn't watch the PBI).

All are divisible by 16 which is what the most used uart had for a buffer.

 

The send and stop should attempt to keep plato fed based on baud rate and not let the buffer overlfow. We don't want the buffer to go less than a full plato usable object in the buffer unless nothing else is coming from the host/server. We always want at least one full plato thing in the buffer when sending and we don't want to hold off data so long it drops off due to overflow or time to live for packets from host/server.

 

The buffer should be adjustable during set up during this phase to allow adjustment to each persons connection and device.

 

This has not been perfected to the byte yet as I have to request each change. It did take it from a complete jiberish mess at 2400 for me and give the result posted in the picture.

Some older device had a 1 byte on chip buffer and I had seen in other software divisible by 16 plus 1 as the buffer send norm, and the stop at divisible by 16 minus one, and the buffer topping out at what ever the buffer can hold.

 

That's what was in my head when I asked him to adjust the computers internal Plato buffer to some rough numbers, we got something, it still needs to be tuned. I am not sure if the internal Plato buffer needs to be slightly larger or not.

 

There may be parts of Plato that act outside of the buffer as well but I am not sure, that rattles around my head from older Plato softwares... I really hadn't dug into this more deeply as what I saw at 2400 failure said 'a timing/buffer issue' in my head. It's at least part of what the issue was.

 

also most modem devices tell what their xon xoff are or even let you configure it. as well as it's behavior (ignore, local, remote, local with remote pass thru)

Edited September 13, 2018 by _The Doctor__

[tschak909]

On a lark, I decided to quickly do a test version of PLATOTerm that uses IP65, and the Dragon Cart.

 

Lack of flow control really hurts it, but you can see when it's going, it's going _fast_.

 

 

-Thom

[_The Doctor__]

yep and if you use the other ethernet chipset it has some buffer and flow etc... dragon was proof of concept... and we enjoyed it. Repeatedly stated was that it was to get ethernet off the ground and others could build from there with the other chipset. Getting things to work as much in the Atari would ensure with better ethernet hardware things would only improve.

 

If you post what you've done I can play...

 

also, have you added any more bytes to the internal buffer?

other question would be serial about serial filling that buffer how often does it pull from that and process, is there a balance to between the two that sort of adjusts.

Edited September 14, 2018 by _The Doctor__

[tschak909]

I am still gravitating toward trying to put a microcontroller with a PHY onto a cartridge slot and expose that to the bus. That will get us ultimately a solid solution.

 

-Thom

 

I have increased the parsing buffer from 256 bytes to 512 bytes.

 

if you look here, in https://github.com/tschak909/platoterm64/blob/master/src/io_base.c:

/**
 * io_main() - The IO main loop
 */
void io_main(void)
{
  if (io_load_successful==false)
    return;
  
  // Drain primary serial FIFO as fast as possible.
  while (ser_get(&ch)!=SER_ERR_NO_DATA)
    {
      recv_buffer[recv_buffer_size++]=ch;
    }
  
  if (xoff_enabled==false)
    {
      if (recv_buffer_size>XOFF_THRESHOLD)
  	{
  	  io_recv_serial_flow_off();
  	}
    }
  else /* xoff_enabled==true */
    {
      if (xoff_enabled==true && recv_buffer_size<XON_THRESHOLD)
  	{
  	  io_recv_serial_flow_on();
  	}
    }

  ShowPLATO(recv_buffer,recv_buffer_size);
  recv_buffer_size=0;
}

You see that the first priority is to immediately try and fill the secondary buffer, so long as there are characters waiting. Once there's a lull in the input, the resulting character stream is pushed to ShowPLATO (which is the core of the entire terminal)

 

New build: plato-20180913-512byte-buffer.atr

 

-Thom

Edited September 14, 2018 by tschak909

[_The Doctor__]

can we make buffer 384? get more 25 and stop 127

 

hold on a sec, you have a serial buffer and a parsing buffer, my brain has farted and I got lost today, I forget what I did yesterday, thought saw yesterday.

 

there is a timing issue, not sure if the is core processing unless the buffer stop has been on a while (talking out my rear on that since idk what's going on in there)

this larger buffer allowed the dice to appear better in back gammon, and the ships to be solid in the battleship game. but processing the larger buffer either slowed things down or pass thru flow control had time to work, the drawing motion was jerky but the image was more complete!

 

I keep changing settings to see what happens. It can make things hard to follow. I should write it down to avoid confusing myself as well as others of the course of time. At least I still had some clarity yesterday and it improved things greatly. Sadly today I'm not so hot. Good days/bad days...

 

At least I helped a little, and it's definitely improved, what's the smallest buffer packet the host/server is pushing to the ethernet port anyway, I know you can send a single byte but what does it pad it out to.

Edited September 14, 2018 by _The Doctor__

[tschak909]

so, buffer[384], XON_THRESHOLD 25 XOFF_THRESHOLD 127 ?

 

-Thom

[tschak909]

New build: Buffer 384 - XON 25, XOFF 127

 

plato-20180913-384buffer-25xon-127xoff.atr

 

-Thom

[_The Doctor__]

I really should move this over to the black box, I could probably just change the high and low marks on the fly using the memory monitor, I really don't like having to make you do all the small changes. And I must apologize for my remembering everything one day, not knowing anything the next way I have been. Just remember if I frustrate... I am probably more frustrated with myself.. knowing I know and not being able to reach it.

[_The Doctor__]

oh that was bad....oh not good.

24 and 128 please but leave buffer at 384

and then

24 and 128 and buffer at 512

 

if you could those two please

 

. burn 25and127 at 384 in fire... it causes plato to have a bad char day

 

if figure I might as well ask for both at once. Since I really need to make room and set up all the machines to actually do this myself...

Edited September 14, 2018 by _The Doctor__

[_The Doctor__]

i just remembered 46 bytes is the smallest ethernet payload of data...

so our buffer for flow control should possibly be multiples of that, of course it could send less but the transport would pad it out to that....

so 46

Edited September 14, 2018 by _The Doctor__

[tschak909]

plato-20180913-384buffer-46xon-127xoff.atr

 

46 XON

127 XOFF

384 buffer

 

-Thom

[_The Doctor__]

I was thinking out loud, to find the correct upper buffer size to fall on packets without fragmentation and excess padding, this would allow for flow control to work in band provided our buffer is big enough... like I said I can't think like I did yesterday... I was thinking 552 or less it has to be divisible by 46 and divisible by 24 and even better if also divisible by 16 (this allows for all buffers and parsers to get a full chuck on each pass. avoiding waste.

the lower buffer size should be platos word size I think that's 24

that is why I was going for 24 128 and then the different buffer sizes to test with...

 

is this a plato system?

https://ple.platoweb.com/Account/SignIn

Edited September 14, 2018 by _The Doctor__

[tschak909]

Edumentum is what became of NovaNET. It is no longer a classical PLATO system, but an evolution of the code that had been passed down over many years, ported over to the web.

 

-Thom

[_The Doctor__]

oh the last posting was smoother and gammon is even more correctly drawn. pulling at 46 was a good move!

Edited September 14, 2018 by _The Doctor__

[tschak909]

meanwhile, I have made a quick demo showing just how poignant the multiple platforms from the same code-base can be:

 

PLATOTerm, many platforms, one code-base. Here showing a concrete example of many different 8-bit systems connected to http://IRATA.ONLINE to participate in a giant screen-share, complete with baton passing and messaging. #irataonline #plato

 

 

-Thom

[mr-atari]

Found this internet quote, perhaps that is an answer to my question...

18th March 2010 at 21:48 +0100

If I recall well, 576 bytes is the byte size that “any network media should be able to transmit”. I think that’s where you got your number

Btw. average packet size is meaningless. You have to look at the distribution of packet sizes. If you do that, you will notice that 2/3 are either just above 64 bytes (ACK) or very close to 1500 (payload). The rest will spread almost evenly between these two values, with majority being very close to 1000.

On averages: Put your feet in the oven and your head in the freezer. Averagely speaking, you’re fine.

[mr-atari]

@mr-atari - I have added CR/LF processing, which you need, to the TTY mode (which happens before you connect to PLATO)

 

plato-20180913-redgreen-xon32-xoff128-cr-lf.atr

 

Let me know if this works okay.

-Thom

No, sorry mate, that does not do the trick.

 

It does not send out LF.

Can you assign the inverse-key to send out LF?

That would be of help.

 

It still erases page 6, why?

 

It still continues loading/saving to drive 1.

Where as I load plato.com from a different drive.

 

also it would be very nice, then when in the (tty mode you call it), keys are processed as capitol letters.

since typos can not be corrected with BS with the plato terminal.

 

I wait for your new build.

[_The Doctor__]

we use tiny payload to emulate modems, going for the minimum payload of 46 bytes at a time is more responsive and we don't need to send a maximum transfer unit of 1500 or slightly less as is done with dsl etc... while the 46 byte payload is not the whole packet which is somewhere around 18 bytes more for all the other packet stuff. making the 64 you mentioned. we want to sync on the minimum packet size for our purposes and probably round it off to whatever the standard plato word size is 24?.. that I cant recall atm that might mean 48 would be the correct time to pull each packet for plato to get 2 words at a time since anything less than 46 will be padded out to 46 anyway, why waste a packet? Since our internet is high latency but happy to stream, more small packets slow down things that is perfect for us... if my constant obsession with 24 as a word size for plato keeps bouncing in my head is correct, I just don't remember....

 

If I were to set up my own server, I would set mtu at the minimum for whatever my equipment and isp would allow. this means many packets with high overhead slowing things down and letting us stop the flow as soon as possible, and delivering chunks of data we can use right away without fragmenting our words, Now if we are using a dragon cart I am not sure how that would help or hurt as the Atari is really doing almost all the gymnastics.

 

in any event whatever we are doing has been improving what I have been getting at 2400

Games are now playable while not perfect, and all the other stuff in most lessons/labs are working nicely

 

Can you send your drivers for r: to me mr. Atari? or however many iterations of it you wish to send, I wish to try them one at a time with plato to see how it works with the MIO, as I mentioned your driver actually controlled rts when using floppy drives in bobterm. since that is the case, you are not far from giving us the true hardware flow control that has been quoted as being needed for plato.

Edited September 14, 2018 by _The Doctor__

[_The Doctor__]

Success at last.

 

I had to hack into your plato build to make it work....

-just to mention-

 

Patched CONTROL-X to sending out $0A.

Patched startup, not clearing page 6.

 

Pity to see that this build stills assumes the config-files are on D1, were as I load plato from D8:

Something to look into.

 

So, with the patched version:

I can setup my ESP8266 and use LiteDOS/ESP8266-driver.

Connect and use plato.

 

My driver sends out Xoff also, but is not successful enough.

How many bytes are poring in after I send $13?

To me it seems, no I measured, transmission is not stopped when I send $13.

 

Graphics look better, the issues I saw are gone.

 

What next?

Can you make things work for me, without me hacking into your build?

but I though $0A and $0D are just carriage return and line feed... that isn't x on xoff ..... how many bytes poring after stop has to do with what is in the buffers set up in the Atari.. high water stop mark is hit modem shuts off flow, and then that buffer has to wind down in the Atari... remember the data should stop coming from the modem right away, if plato is full duplex and sending right away, if plato does not send it right away but sends after it is done doing whatever then more info will get by. I experiment with this quite a bit on normal BBS. I saw the lantronix was still receiving data from the internet and filling it's buffer, the rs232 analyzer showed no data flowing to the mio, when x send was received it sent the what was in the lantronix buffer even though nothing was coming from the web. That is exactly how that is suppose to work.

 

Since the wireless gizmo's everyone is using do not have hardware flow control.. and only use gnd tx rx there isn't a chance for anything other than xon xoff for everything to work

Edited September 14, 2018 by _The Doctor__