rshell cp fails if connection is via UART
Opened this issue · 20 comments
SD card contains only boot.py:
import pyb
pyb.usb_mode(None)
uart = pyb.UART(6, 115200)
pyb.repl_uart(uart)
Typical rshell session copying a 2.9KB file (I don't always get trace data: sometimes it just hangs)
/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> cp ttest.py /flash
Traceback (most recent call last):
File "/mnt/qnap2/python/rshell/rshell.py", line 1785, in <module>
main()
File "/mnt/qnap2/python/rshell/rshell.py", line 1782, in main
shell.cmdloop(cmd_line)
File "/mnt/qnap2/python/rshell/rshell.py", line 1084, in cmdloop
cmd.Cmd.cmdloop(self)
File "/usr/local/opt/python-3.4.3/lib/python3.4/cmd.py", line 138, in cmdloop
stop = self.onecmd(line)
File "/mnt/qnap2/python/rshell/rshell.py", line 1115, in onecmd
return cmd.Cmd.onecmd(self, line)
File "/usr/local/opt/python-3.4.3/lib/python3.4/cmd.py", line 217, in onecmd
return func(arg)
File "/mnt/qnap2/python/rshell/rshell.py", line 1353, in do_cp
if not cp(src_filename, dst_filename):
File "/mnt/qnap2/python/rshell/rshell.py", line 364, in cp
filesize, xfer_func=send_file_to_remote)
File "/mnt/qnap2/python/rshell/rshell.py", line 933, in remote
xfer_func(self, *args, **kwargs)
File "/mnt/qnap2/python/rshell/rshell.py", line 622, in send_file_to_remote
sys.stdout.write(chr(ord(char)))
TypeError: ord() expected a character, but string of length 0 found
This seems similar to using rshell with the WiPy: most functionality works except for cp.
ok - thanks - I'll look into this
Is this over a wireless link? Or using something like an FTDI connection?
Initial analysis suggests that the uart read buffer is getting messed up. The host side wrote 512 bytes.
I tried: modifying your boot.py to look like:
import pyb
pyb.usb_mode(None)
uart = pyb.UART(6, 115200, timeout_char=200, read_buf_len=600)
pyb.repl_uart(uart)
it got the first 198 bytes fine, and then it dropped 8 characters. I confirmed (using my logic analyzer) that the correct characters arrived on the Rx line.
I probably won't get much further tonight, so I thought I'd write this up to let you know what I found.
And it definitely looks like a software issue. I toggled a GPIO for each character received and the dropped characters were received.
Nope - I see some dropped IRQs. So that tells me that interupts were disabled or some higher priority handler was running.
Sorry for the delay getting back to you but we are separated by a few degrees of longitude :) I was testing using an FTDI adaptor lead connected by short jumpers to Pyboard UART6. The link seems reliable.
I opened: micropython/micropython#1555
From my logic analyzer traces it looks like interupts are being disabled to do sdcard I/O and that's causing dropped characters.
I'm going to see if I can figure out how to do UART Rx using DMA.
This should be addressed by micropython/micropython#1608
I'm still seeing this with a Pyboard connected via an FTDI adaptor. Rshell works fine except for copying files to the Pyboard SD card where it hangs. The card in a card reader shows a zero length file. This is with current firmware.
I stumbled across this last night as well on my NUCLEO 401 board. In that
case, it was copying to internal flash, and also over a non-USB serial link.
On Wed, Feb 10, 2016 at 5:07 AM, Peter Hinch notifications@github.com
wrote:
I'm still seeing this with a Pyboard connected via an FTDI adaptor. Rshell
works fine except for copying files to the Pyboard SD card where it hangs.
The card in a card reader shows a zero length file. This is with current
firmware.—
Reply to this email directly or view it on GitHub
#10 (comment).
Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com
@peterhinch Are you using rshell from this repository? This repository is now obsolete and has been replaced by: https://github.com/dhylands/rshell and you can now install rshell using pip.
Installing with pip sounds so easy. Unless you have boxes running rather old versions of Debian. I'll tackle that can of worms tomorrow.
On this relatively up to date laptop (Mint 17.1) I managed to install with pip3 (pip threw errors). The following presumably implies I'm running the right version:
axolotl adminpete # pip3 install rshell --upgrade
Requirement already up-to-date: rshell in /usr/local/lib/python3.4/dist-packages
Requirement already up-to-date: pyserial in /usr/local/lib/python3.4/dist-packages (from rshell)
Requirement already up-to-date: pyudev>=0.16 in /usr/local/lib/python3.4/dist-packages (from rshell)
Requirement already up-to-date: six in /usr/local/lib/python3.4/dist-packages (from pyudev>=0.16->rshell)
Cleaning up...
axolotl adminpete #
Version I'm running
[adminpete@axolotl]: ~
$ which rshell
/usr/local/bin/rshell
[adminpete@axolotl]: ~
Attempt to run
[adminpete@axolotl]: ~
$ rshell -p /dev/ftdi
Connecting to /dev/ftdi ...
Welcome to rshell. Use Control-D to exit.
/home/adminpete> cp rats9.py /sd
^CTraceback (most recent call last):
File "/usr/local/bin/rshell", line 9, in <module>
load_entry_point('rshell==0.0.1', 'console_scripts', 'rshell')()
File "/usr/local/lib/python3.4/dist-packages/rshell/command_line.py", line 4, in main
rshell.main.main()
File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 2254, in main
shell.cmdloop(cmd_line)
File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1311, in cmdloop
cmd.Cmd.cmdloop(self)
File "/usr/lib/python3.4/cmd.py", line 138, in cmdloop
stop = self.onecmd(line)
File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1342, in onecmd
return cmd.Cmd.onecmd(self, line)
File "/usr/lib/python3.4/cmd.py", line 217, in onecmd
return func(arg)
File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1690, in do_cp
if not cp(src_filename, dst_filename):
File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 452, in cp
filesize, xfer_func=send_file_to_remote)
File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1107, in remote
xfer_func(self, *args, **kwargs)
File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 738, in send_file_to_remote
char = dev.read(1)
File "/usr/local/lib/python3.4/dist-packages/rshell/main.py", line 1073, in read
return self.pyb.serial.read(num_bytes)
File "/usr/local/lib/python3.4/dist-packages/serial/serialposix.py", line 481, in read
ready, _, _ = select.select([self.fd], [], [], timeout)
KeyboardInterrupt
[adminpete@axolotl]: ~
$
``` (It hung on the attempt to copy). Is there a way to get rshell to issue a version number?
Good question. I don't think that there is, but I'll figure out a way to do that.
And yeah - you need to use pip3 because rshell requires python3. I guess I should make sure I say that :)
I also discovered that if you have python3, you can run python3 -m ensurepip
and that will get you pip3.
Thanks for the tip re pip3.
I'm puzzled you've closed this as I'm still experiencing the hanging problem. To try to settle the version issue I used pip3 to uninstall rshell and deleted the directories in /usr/local/lib/python3.4/dist-packages. I then reinstalled it with pip3 checking that the code had re-appeared in the same location. I then cloned your repository into ~/temp using
git clone https://github.com/dhylands/rshell.git
Finally I ran the following checks, which have convinced me I'm running the right version - but if I'm doing something dumb do please enlighten me!
[adminpete@axolotl]: ~
$ cd /usr/local/lib/python3.4/dist-packages/rshell
[adminpete@axolotl]: /usr/local/lib/python3.4/dist-packages/rshell
$ md5sum *.py
55537e0c645de5b4d323b1fb97873b4b command_line.py
8671bdf67f2ff1a301d0cd0e49e901f4 getch.py
d41d8cd98f00b204e9800998ecf8427e __init__.py
7656d1566097065c4250b531093fe2f3 main.py
da18a8d6d57694c55c1751cf53860f1d pyboard.py
[adminpete@axolotl]: /usr/local/lib/python3.4/dist-packages/rshell
$ cd ~/temp/rshell/rshell/
[adminpete@axolotl]: ~/temp/rshell/rshell
$ md5sum *.py
55537e0c645de5b4d323b1fb97873b4b command_line.py
8671bdf67f2ff1a301d0cd0e49e901f4 getch.py
d41d8cd98f00b204e9800998ecf8427e __init__.py
7656d1566097065c4250b531093fe2f3 main.py
da18a8d6d57694c55c1751cf53860f1d pyboard.py
[adminpete@axolotl]: ~/temp/rshell/rshell
$ which rshell
/usr/local/bin/rshell
[adminpete@axolotl]: ~/temp/rshell/rshell
$ alias rshell
alias rshell='rshell -p /dev/pyboard'
[adminpete@axolotl]: ~/temp/rshell/rshell
$
$ alias rsftdi
alias rsftdi='rshell -p /dev/ftdi'
[adminpete@axolotl]: /mnt/qnap2/data/Projects/MicroPython/micropython/docs
The closing wasn't intentional. I must have just clicked on the wrong button.
I just wanted to make sure you were using the pip installed version, because I seem to recall fixing a few issues. I'm homing in on the issue (I think). At least for me on the Nucleo. I suspect that your problem is somehow related. Could you try with --buffer-size=30 and see if that makes any difference?
Yes! A quick test was successful: I copied a small file to /sd, 'cat'-ed it and deleted it all OK.
[edit]
A test copying a 56KB file to and from /sd with a subsequent verify also worked, as did editing files on /sd. Looking good!
What seems to be happening is that something slowed down a bit and rshell is no longer processing buffers as quickly as it used to. rshell has a default bufer size of 512, however, the UART only has a buffer of 32 (or maybe 64). This host side sends a buffer of data and waits for an ACK before sending another buffer worth of data. If the micropython side processes the data quickly enough then it can deal with more than the UART buffer worth, however, it seems like we were really close to the edge of "running in real-time" and something has slowed down enough that the python side is taking more than a character time to process each character. All it takes is one character to be dropped and the file transfer grinds to a halt.
Now to figure out a workaround.
ctrl S ctrl Q flow control in the UART driver? Alas it would need a firmware change, but perhaps a useful option.
Another one I get occasionally on a USB connection is this "serial port /dev/pyboard closed". It's benign in that you can restart the link and carry on, but I thought it worth mentioning. The sequence below was just after dropping back from the REPL with ctrl X. After the error I hit ctrl D and fired up rshell again, without resetting the Pyboard.
/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> cp upower.py /flash
serial port /dev/pyboard closed
/mnt/qnap2/data/Projects/MicroPython/micropython-micropower>
[adminpete@axolotl]: /mnt/qnap2/data/Projects/MicroPython/micropython-micropower
$ rshell
Connecting to /dev/pyboard ...
Welcome to rshell. Use Control-D to exit.
/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> cp upower.py /flash
/mnt/qnap2/data/Projects/MicroPython/micropython-micropower> repl
Entering REPL. Use Control-X to exit.
>
MicroPython v1.6-35-ge372e83 on 2016-02-11; PYBv1.0 with STM32F405RG
Type "help()" for more information.
>>>