Linux Telnet Upload Problem

Question

Linux Telnet Upload Problem

Closed this issue 4 years ago · 19 comments

kjklucky commented 5 years ago

esp8266(rx tx gnd vcc) pin connected.(baud-rate 38400)

File Upload Fails or Very Slow Speed(command: rz -e) When Connected to Linux Telnet.

But Download is Good.(Command: sz -e)

How to Fix?

Answer 1 · 2019-05-30T16:40:46.000Z

What client are you using?

Answer 2 · 2019-06-02T22:46:24.000Z

I used old korean telnet client. (for Windows)

The Name is iyagi multi ver8.5 (This Program Support Zmodem).

https://github.com/jsalin/esp8266_modem <- With this source, Upload is good. But This Source Can Not Use Rs232 Full Connection. (RTS,CTS,RI,...)

Answer 3 · 2019-11-16T02:37:39.000Z

Just as an update: this issue has been reproduced, and is very frustrating. It's my top priority to fix now that I have a solid repro case, though my efforts so far have failed. It is strictly a ZModem problem, and only with uploads.

Answer 4 · 2020-01-29T18:22:14.000Z

I am not sure if my problem is related, but I am not able to upload any file to (Synchronet) BBS at all. Using zmodem protocol on TeleMate 4.20 for DOS (tried on AMD K6 PC, DosBox, VirtualBox; any baud; any flow etc.). Downloads are fast and reliable, upload is slow and always ends in failure: upload goes to 100% (without any errors), then it hangs for severals seconds and fails horribly, sending random characters to BBS causing chaos for a while. When I use WiFiModem firmware everything works fast and reliably, so I am sure it's not BBS' or TeleMate's problem. Zimodem is far more advanced and superior to WiFiModem, thus this problem bothers me a bit 😄

Answer 5 · 2020-01-29T22:37:20.000Z

So, turns out I need to ask:
Are you using the AT+SHELL zmodem client, or a client on the computer over a streaming connection?

Answer 6 · 2020-01-29T23:29:52.000Z

I am sorry I didn't mention it. I am using ESP8266 so AT+SHELL is not available for me. I am using client (with built in X/Y/Z modem protocols) over streaming connection. Tried with and without telnet translation but still no luck.

Answer 7 · 2020-01-29T23:34:02.000Z

Thanks! I'll try TeleMate 4.20 with Synchronet BBS.

Answer 8 · 2020-01-30T01:20:21.000Z

I don't know if this may help you, I have recorded output of zmodem receiver on BBS'es side + dos pc screen, it's exactly same when using real machine, not just virtual one (virtual machine has serial pass-through to ESP8266). Maybe I have just missed something in the zimodem configuration, but I can't really see what could cause this.

https://www.youtube.com/watch?v=5mXqXTK5DK8

Answer 9 · 2020-02-01T08:09:16.000Z

Hi. Can you give the latest changes a try?

Answer 10 · 2020-02-01T19:59:46.000Z

Hi, I found the problem and the solution!
Your latest changes didn't resolve the problem but it put me on right track of where the problem lies. Output of BBS' zmodem receiver started outputting wrong CRCs so data went through but not correcly, I did some research and found out that you have to start escaping IAC character (0xFF) by doubling it in the outgoing stream of data (in case of telnet escaping is enabled). I did test that by inserting if (isTelnet() && c == 0xFF) { current->write(c); } before original current->write(c); to ZStream::socketWrite procedure and finally I was able to upload file using zmodem protocol to bbs.

Answer 11 · 2020-02-03T16:51:18.000Z

@AfBu - Good change, thanks, though it now sounds like your issue was unrelated to this ticket, per se. Your issue would have prevented transfers from working at all, while the OP is about the SPEED of transfers. Leaving this open until someone with the original program reports in.

Answer 12 · 2020-02-03T17:55:34.000Z

@bozimmerman I should have created separate issue, sorry for off-topic. Thank you for cooperating on finding where my issue lied.

Answer 13 · 2020-05-03T13:49:50.000Z

Hi,

I have the same problem trying to upload files using Zmodem from a Falcon 030 with Simulant Retro Wifi SI (R2, firmware 3.4). The bug is asymmetrical. Here is a bad quality video demonstration:

https://www.youtube.com/watch?v=8zMl_FftG04&t=

So I made tcpdump captures on the telnet side (to be opened with Wireshark):

https://app.box.com/s/r817cu4zl539wwgyrc4cm1qry74sy6mb

My hypothesis is that the ESP8266 CPU isn't fast enough, in some conditions, to forge TCP/IP packets at RS-232 wire speed. By chance, this does not happen in download situation, because telnet gathers characters in big IP packets. To confirm this, I wrote a C program which uploads a binary file character by character.The upload is successful (m5dsum ok) when there is at least 4ms between each character (between TCP packets consequently).

So, maybe an optimized buffering (like shaping? or Nagle algorithm?), to avoid CPU overload, would make the Zmodem upload reliable with Zimodem?

Answer 14 · 2020-06-29T00:29:17.000Z

So, I've put code for enlarging the rx serial buffer into the latest code. Can someone let me know if this helped?

Answer 15 · 2020-07-15T20:34:27.000Z

Closing this on the assumption it's fixed. Comment if you don't believe it is.

Answer 16 · 2020-12-14T16:25:08.000Z

I'm having a similar problem with what I assume is buffer overrun issues, but in this case using Xmodem protocol. How do we adjust the buffer size within the code? What is the default?

I'm running 3.5.6. Downloads from a BBS work mostly okay, but uploads completely fail. It also seems to really mess up the CPU on the ESP module because it will sit there and retransmit (over the Telnet connection) the same outgoing blocks over and over for like 15 times, and sometimes it crashes even harder where it requires a power cycle to work again.

Answer 17 · 2020-12-16T20:01:42.000Z

Depends on the platform.
In zimodem.ino, there is this block for the 8266:
#ifdef ZIMODEM_ESP8266
HWSerial.setRxBufferSize(1024);
#endif

For the ESP32, I think I had to change something in one of the platform lib files, which I'll look up if you need it.

Answer 18 · 2020-12-16T20:49:35.000Z

I'm using an 8266. I have tried lowering the buffer to 256 and raising it to 2048, but no effect. Downloads (Remote Host --> ESP8266/Computer are slow (@115200) but generally work. Uploads (ESP8266/Computer --> Remote Host) completely fail. Using the https://github.com/jsalin/esp8266_modem firmware someone mentioned above made the download speeds MUCH faster, but didn't resolve the upload issue. And the esp8266_modem firmware above is vastly inferior to yours in terms of AT command support. Is there anything else I could try? Most of my uploads seem to fail at a repeatable location in each file, although not the same location for all. Is it possible that the Zimodem firmware isn't properly escaping some characters?

Answer 19 · 2020-12-18T07:59:54.000Z

Interesting. If this other firmware has the same upload problem, then the issue is likely down in the ESP core libs, not the firmware that's using it.
So, normally, the trick to fixing a problem like this comes down to two things:

a quick, reliable re-production of the problem (and I don't even have a tedious, unreliable one)
good logging.
The zimodem firmware has debug logging, but the 8266 does not handle it very well, due to the cramped flash space, the slow write speed, and other issues. I think, for #2, we'd probably need logging between the client serial program doing to the uploads and the modem, and then another log on the receiver side -- to see exactly what's going on. I've actually implemented X-Modem a few times in the past, so I'm pretty sure I could parse the packets if I could see them.