Escape Sequences for Arrow Keys
Closed this issue · 41 comments
I'm using this firmware on an ESP8266 connected to a CoCo3's serial port. Any reason why the arrow keys wouldn't send the proper escape sequence to move the cursor? Instead I see "[A", "[B", "[C", or "[D" appear on the screen whenever I strike those keys, as if ASCII character 27 is being omitted from the start of the sequence. Maybe there's a config setting I'm overlooking?
EDIT: I should add that the arrow keys work fine to move the cursor around the terminal while in command mode. It's when a connection is established they stop working.
Everything else is working great -- love the firmware.
That’s what your terminal program is sending... probably ANSI escape sequences.
Doubtful. The arrow keys work (send the correct sequences) in the same terminal program when connected to a PC running tcpser.
Yeah, Twilight Term is what I'm using. Do you remember what modes/settings worked for you? Anything else that I need to set in Zimodem?
Twilight Term does send the arrow key codes (you have to press F1 and the arrow key to move the cursor). As I've stated, it works when using tcpser -- the problem began when I switched from tcpser to the ESP8266 running this firmware.
I do agree with you -- Twilight Term is an amazing piece of software. All of SockMaster's work is extraordinary.
Allen, while I didn't have any success getting a telnet server installed on my Linux machine to test, I did manage to install a default Synchronet BBS on my Windows 10 PC and used that to test instead.
With Synchronet running, I headed to the other room to my CoCo3, loaded up Twi-Term, and ATDT'd successfully to the BBS I set up. I took the time to set up an account and do all the typical things one might do when connecting to a BBS for the first time. When prompted with a Yes/No option, I was able to successfully arrow between Yes and No. Next I tried posting a message, and within the fullscreen editor I was able to type multiple lines of text and use the arrows to move around what I had typed. Everything worked perfectly. When I closed Twi-Term and returned to the Windows 10 PC, I saw in the Synchronet logs where I posted the message and it didn't seem to report any unusual activity.
I returned to the CoCo3 and retried connecting to the BBSs I had arrow key problem with, and the problem persists... arrow keys still don't work.
Any thoughts on why escape sequences would work with some BBSs while not on others?
(I did catch the CoCo-Talk episode where he demo'd Donkey Kong on a CoCo1/2. Amazing stuff)
BBS Software would specifically have to be written to parse and support the arrow keys.
That's the point entirely; I know they do. These are BBSs where, before when using tcpser, were perfectly able to interpret the arrow keystrokes I was sending. They have menus driven by arrow keys that I'm incapable of navigating now with this firmware, but I could before with tcpser. This is the whole reason why I posted the issue.
I believe I'm starting to find a correlation though... of the "problem" BBSs, one is a Renegade BBS, and one is a Mystic BBS. I believe all the Synchronet BBSs I've tried have been fine, but I'm going to keep trying a few more to confirm.
I'm curious what options you are using. Can you tell me what your ATI5 normally reads, and also, what command you are using to connect? (your ATD... command).
Thanks.
The only thing I can think of is that somehow PETSCII is being turned on. But if that were so, you'd be complaining about a lot more than just arrow keys.
I upgraded to 3.4.1 firmware to continue testing... After further digging, I do see on a packet trace that during Telnet negotiation, the terminal is set to "Zimodem" using Suboption 250/Terminal Type. It's appears to be done on line 159 of pet2asc.ino. I am not in a position to compile and test a custom firmware, but maybe changing line 159 to return "ansi" or "vt100" would do the trick so that the BBS/telnet server side sees a terminal type it likes?
All of these terminal emulations use ASCII, its the extended functions that are supported using special ASCII sequences that make the terminal types like VT100 and ANSI a little special. For example, ASCII alone has no provisions for cursor movement using arrow keys.
CraigEMP said TCPSER works, so I did another trace using TCPSER and (by default) it sends "VT100" over the wire for Telnet Suboption 250/Terminal Type.
I'm curious what options you are using. Can you tell me what your ATI5 normally reads, and also, what command you are using to connect? (your ATD... command).
Output of ATI5:
ATB9600E1Q0V1X1F4R0&D&MS0=0S2=43S3=13S4=10S5=8S40=127S41=1S43=0S44=0S45=0S46=0S47=2S48=0S49=5S50=0S51=4S52=0S53=14S54=0S55=12S56=0S57=13S60=0&P0&O0
Command used:
ATDT name.of.bbs
Also tried just ATD name.of.bbs and there was no noticeable difference.
Well, there are 2 reasons I didn't do that, either one of which was sufficient in my mind:
- The modem shouldn't make a promise on the terminal type that the client can't keep, and that includes ASCII, since petscii is also a valid client terminal type. For example, TCPSER promising a server that someone is running a VT100 compatible client is presumptuous.
- For what it's worth, I was following how BBS clients tend to treat termtype, which is to respond with their unique application name, which is great for server programmers because you often have to cater to their unique implementations. Since the modem is in a position of not knowing anything about the host client, it seemed reasonable to do.
All that said, I think a workable answer might be to allow the termtype variable to be configurable.
Regarding ATD operating exactly the same as ATDT, well, I would need to produce some evidence of that, as it would be a far far far far bigger problem if that's true than what we are talking about here.
In fact, I wrote a full ansi telnet client that manages its own telnet codes using ATD to bypass the modem telnet responses, so I'm pretty sure that was working recently.
Regarding ATD operating exactly the same as ATDT, well, I would need to produce some evidence of that, as it would be a far far far far bigger problem if that's true than what we are talking about here.
I'll amend my statement -- Upon connecting with ATD, the terminal spits out a dozen or so nonalphanumeric characters, and then continues on with the session as if connected using ATDT. Using ATDT alone doesn't produce the nonalphanumeric characters.
BTW, I just added AT&S41 to the command set, for setting the termtype to an arbitrary string. I didn't test it at all, since I'm at work and don;t have access to my hardware. If you'd like to do the honors, though...
Use it like:
AT&S"41=ansi"
at&u=3.4.3 is returning "ERROR" after saying wait for modem restart. Do I have this command wrong?
Sorry, I assumed since you posted here that you were compiling your own. Since this is untested, I didn't release the binary --- I only updated the source code.
Oh sorry, yea I haven't yet compiled my own firmware and don't have a dev environment set up.
I've got the Arduino IDE set up with the Git version of the 8266 libraries... I've got the Zimodem 3.4.3 source compiled successfully but don't have the hardware set up right now to flash it. If no one else has it done, I'll set up the hardware to get this flashed and tested sometime later today. My workbench is a mess, I haven't flashed an ESP in a while, gotta remember what I'm doing hardware wise. I think the variable term setting via at&s41 is a great way to go. Thanks!
Wasn't going to sleep without trying ... I got 3.4.3 flashed to a NodeMCU style board and your changes appear to work. With no configuration on my part aside from connecting to wifi, when I atdt to Debian, I see TERM is zimodem; I have various emulation problems like no VT100/Ansi coding. If I disconnect and issue at&s"41=vt100", when I reconnect I get VT100/ANSI color and cursor control and TERM is set to vt100. Good stuff! I hope CraigEMP might be able to test this against the target BBS software.
Thanks dedhed -- glad we figured it out.
I compiled and uploaded the new firmware code to the ESP8266, configured at&s"41=vt100" like dedhed did, tried several BBS sessions, and unfortunately the condition has not changed. :(
No quote between s41 and the value. Was that a typo?
I typed it the same as both dedhed and Bo had it above.
Can you try AT&S”41=VT100” making VT100 in capitals. Maybe same thing with ansi and ANSI? I know it’s nit picky but otherwise:
I tried all different permutations of casing.
Are you able to take a network trace using Wireshark on your Win10 machine running the BBS software you’re testing with? And share a trace of both Zimodem and TCPSER sessions connections working and failing?
That sounds like a good next step, and maybe I'll try that this weekend when my time frees up. For today, I went ahead and set up a Mystic BBS installation like I did before with Synchronet, and logged in from the CoCo3 connected to the ESP8266 running Zimodem (the current release, not the one with the termtype modification), and I was able to browse around and arrow between the Yes/No ANSI dialogs, although every so often it seemed to omit the ASCII escape character 27 and typed [A, [B instead.
Something somewhere is causing arrow keystrokes to not translate properly, but only on certain BBSs, and only from this firmware.
If anyone cares to try for themselves (and this is not a BBS promotion, just an example case), ATDT to blackflag.acid.org and go to the login prompt after about three ANSI intro screens. If your retro computer has arrow keys, try to move the cursor down through the options. If you cannot, then it's not just me.
If you're adventurous, you might try entering AT&O1 (that's oh one, not zero one) immediately before your ATDT command, and then as SOON as you recreate it, do the +++ and ATH0&O0 (that's oh zero) to hang up and view the log. If you can capture that log in your terminal program, it may have a clue on what's going on.
You mentioned those ansi screens -- I'm kinda terrified you'll run out of log space -- if you can abort those, please do so.
Have you also tried other terminal programs?
Yes, I was able to recreate the condition in Ultimaterm, using its way of transmitting the arrows.
If you're adventurous, you might try entering AT&O1....
Always feeling adventurous! Attached is the buffer I captured from doing as you instructed. I used my test Mystic BBS setup as the guinea pig, and at the username prompt just tapped the right and left arrow keys back and forth until I got a response that wasn't the expected outcome of pressing an arrow key -- that's when I +++'d out and captured the buffer.
Hopefully this sheds some light...
(side note... this was a really fun exercise for me, capturing and saving this buffer from this ancient computer and attaching it here!)
For good measure, here's another capture that perhaps better illustrates the issue. This is from a session where the arrow keys did not work at all, and is pinpointed to where the behavior occurs, rather than the entire session.
So, since you know which keys you typed and when, maybe you can help me out.
The way to read that log is pretty simple:
SocI means socket Input (meaning remote internet data arriving at the modem). SocO means socket output (meaning data inside the modem being sent to the internet). SerI means serial data arriving into the modem from the host computer... like when you type something. SerO means serial data being sent from the modem to the computer to view on your screen.
So, I saw this a LOT in the previous log, and a few times in this one:
0A0B31 SerI: 1B
0A0B32 SocO: 1B
0A0B3C SerI: 5B
0A0B3D SocO: 5B
0A0B47 SerI: 42
0A0B47 SocO: 42
0A18FD SerI: 71
0A18FD SocO: 71
It means you typed 1B (ESC) 5B ( [ ) 42 B which is the DOWN ARROW according to the ansi spec. As you typed each character, you can see it being sent to the Socket as output a moment later. However, while I see other ansi characters arriving from the internet and being sent to your computer, I don't see 1B 5B 42 being echoed back to you.
A theory: I wonder if the remote server is expecting the entire ansi sequence to arrive in the same packet, and is dismissing it if it doesn't....
It means you typed 1B (ESC) 5B ( [ ) 42 B which is the DOWN ARROW according to the ansi spec.
I can confirm this is the last thing I pressed before capturing the first log. I was attempting to arrow down to the second option in a four-option login menu.
However, while I see other ansi characters arriving from the internet and being sent to your computer, I don't see 1B 5B 42 being echoed back to you.
Not sure I follow. Meaning potentially the sequence is being transmitted, and just simply not coming back to move the cursor on the client screen?
A theory: I wonder if the remote server is expecting the entire ansi sequence to arrive in the same packet, and is dismissing it if it doesn't....
Interesting. Glad to test any other code changes you're contemplating.
Just checking in to see if there has been any movement on this.
Bo, does the "packet" idea you mentioned earlier require a code change, or is it something that can be tested on any build by making a config change?
No to both questions. Packet divisions are determined by either the underlying ip library or by the hardware itself (not sure).
I'm facing the problem of non-working arrow keys with the Mystic BBS, and I've investigated a bit using tcpdump. It seems that the suspicion that the arrow key sequence is transmitted by Zimodem in two different packets is true, as can be seen in a packet dump:
17:44:09.226048 IP ESP-ECA123.fritz.box.49154 > octopi.fritz.box.telnet-alt: Flags [P.], seq 2:3, ack 181, win 2002, length 1
0x0000: 4500 0029 00f8 0000 ff06 d4b1 c0a8 b23d E..)...........=
0x0010: c0a8 b296 c002 0913 0000 1b50 f52f 7148 ...........P./qH
0x0020: 5018 07d2 5bf6 0000 1b00 0000 0000 P...[.........
17:44:09.273343 IP octopi.fritz.box.telnet-alt > ESP-ECA123.fritz.box.49154: Flags [.], ack 3, win 29200, length 0
0x0000: 4510 0028 ca59 4000 4006 8a41 c0a8 b296 E..(.Y@.@..A....
0x0010: c0a8 b23d 0913 c002 f52f 7148 0000 1b51 ...=...../qH...Q
0x0020: 5010 7210 e63f 0000 P.r..?..
17:44:09.275704 IP ESP-ECA123.fritz.box.49154 > octopi.fritz.box.telnet-alt: Flags [P.], seq 3:5, ack 181, win 2002, length 2
0x0000: 4500 002a 00f9 0000 ff06 d4af c0a8 b23d E..*...........=
0x0010: c0a8 b296 c002 0913 0000 1b51 f52f 7148 ...........Q./qH
0x0020: 5018 07d2 1bb3 0000 5b41 0000 0000 P.......[A....
17:44:09.275769 IP octopi.fritz.box.telnet-alt > ESP-ECA123.fritz.box.49154: Flags [.], ack 5, win 29200, length 0
0x0000: 4510 0028 ca5a 4000 4006 8a40 c0a8 b296 E..(.Z@.@..@....
0x0010: c0a8 b23d 0913 c002 f52f 7148 0000 1b53 ...=...../qH...S
0x0020: 5010 7210 e63f 0000 P.r..?..
While it is easy to argue that the Mystic BBS software should in fact not rely on the arrow keys to arrive in the same packet, it apparently does just that and the authors don't seem to be willing or able to fix this behavior. I've now looked at the Zimodem source code to see how hard it'd be to fix this here.
My proposal would be to buffer an outbound ESC character for a short while (maybe 10 ms) and then read any further characters off the serial port to collect an eventual function key escape sequence. Instead of sending the collected characters one by one, they'd then be sent to the socket in one go. This should be a relatively local change in zstream.ino, although it would need some changes around socketWrite to deal with telnet option escaping properly when writing multiple characters.
I'll try myself on a patch to this effect unless something has already been in the making to fix this issue.
Looking at the packet trace, it occurred to me that there is in fact more to this than just "Mystic BBS does it wrong". When looking at the timestamps between the first and the third packet, it can be seen that they're 50ms apart, which means that the BBS software likely sees a gap of some 50 ms between the ESC and the rest of the arrow keys as well. Given that even at modest speeds and on serial lines, characters come in at a rate of much less than one per millisecond, I'd say that 50 ms is a long gap and should be avoided when sending out a single escape sequence.