Follow-up for different options for file transfer.

Question

Follow-up for different options for file transfer.

Opened this issue 2 months ago · 14 comments

felixdoerre commented 2 months ago

Hi, this is follow-up for micropython/micropython#13786, to not unnecessarily clutter that issue.

So up-front I think I mixed up things in my initial comment, so to clarify the mechanisms mpremote uses:

fs_get (for direct file transfer e.g. in mpremote cp): https://github.com/micropython/micropython/blob/42eab32a36f70840c7e670a61269fb795d43991b/tools/mpremote/mpremote/transport_serial.py#L391
This logic uses the standard-python byte-array escaping mechanism. That would escape 0x04 but leave ascii characters unchanged, your logic for fs_put already has that optimization and even tests two different options for an encoding. This works with the webrepl code as is.
mpremote mount: https://github.com/micropython/micropython/blob/42eab32a36f70840c7e670a61269fb795d43991b/tools/mpremote/mpremote/transport_serial.py#L719
The mount protocol uses binary file transfer and prefixes packages with a length parameter. This requires a REPL stdout stream that is able to handle arbitrary bytes, as mount "protocol" is inherently binary. The stream is naturally segmented due to the individual read/write commands. For developing locally, I really like the mpremote mount functionality, as it allows virtually immediate "synchronization" without having to use on-device flash, and allowing to load development code directly into device RAM.

When talking about the optimization, I think I mixed things up with the mount-logic.

Answer 1 · 2024-06-26T12:11:24.000Z

Thanks much for the explanation!
I have yet to explore the benefits of mount approach in the context of ViperIDE. Interestingly, it could allow running examples / complete projects without affecting the flash FS of the device. Maybe some other use cases? Let me know if you have any ideas!

Also, I would be happy to help with testing / improving / brainstorming the changes in WebREPL.
I think ViperIDE provides a good basis for rapid prototyping and exploration.

Answer 2 · 2024-06-26T13:31:02.000Z

A quite fun extension could be to "automatically" prompt the user to install (official) packages, when they are searched for but are not in the path.

Answer 3 · 2024-06-26T13:39:06.000Z

For me the typical "mount" use-case is: I have some version of my application on the device and just want to tinker and test out some changes. So I modify only the local copy, and load it with mpremote mount (and reload modules in-place with del sys.modules[...]; import). Tinkering with the application local makes sure I don't make it worse while tinkering, worst-case I can just power cycle the device and it will boot again as before, making the webrepl available again.
Only once I'm sufficiently satisfied with the changes, I copy them to the device.

Answer 4 · 2024-06-26T13:44:28.000Z

I think I need to embed this workflow into ViperIDE.
Do you mount the root fs this way?

Answer 5 · 2024-06-26T13:47:20.000Z

mpremote mount will mount a directory /remote. And then I just add /remote to sys.path.

Answer 6 · 2024-06-27T10:43:19.000Z

I'm skimming through the https://github.com/felixdoerre/webreplv2 , the JS file is there but no html.

Answer 7 · 2024-06-27T11:27:11.000Z

Yes, I've missed to commit that as well. I haven't changed much there so it didn't feel too important, I've added it now.

I'm still thinking about what feature set would be best for the micropython-"included" webrepl-web-client. And I'm beginning to think that just having plain REPL access (with raw paste support), without file transfer might be best, and to leave more elaborate user interfaces to other projects. Then it would be as easy as overriding the "static_host" variable at runtime or even maybe selecting other webrepl-web-clients in the UI. Maybe we could have an "explicit" mpremote port allowing shared code, and then be the "included" webrepl client be a simple example on how to use it, and have it more tightly integrated into ViperIDE. That would allow you to have a micropython-compiler in the browser and get immediate feedback, at least for syntax errors.

Btw, I haven't extensively tested Viper yet, but do you allow browser-saving the WebREPL password? For me, that was a very comfortable extension. And as it is saved in the context/origin/ip of device itself, it naturally allows saving passwords per device.

Answer 8 · 2024-06-27T12:26:15.000Z

I could definitely run mpy-cross as a WebAssembly module.
Currently the WebREPL url and password are saved throughout the session. When ViperIDE is reloaded, the info vanishes.
This should be good enough

Answer 9 · 2024-06-27T12:41:13.000Z

I personally like it stored in the browser password manager, as I have multiple micropython-devices with unique webrepl passwords, so having that stored in the browser password manger is really comfortable, but sure, as a start its enough to have it only in the session. The WebREPL URL itself does not need to be stored, if the IDE runs in the device-origin, that URL is implicitly evident.

Answer 10 · 2024-06-27T12:43:43.000Z

Correct. Yes i think this will automatically become the case when ViperIDE switches from standard JS prompt to a custom modal layout

Answer 11 · 2024-06-27T12:47:06.000Z

Maybe, WebREPL python module could handle an optional url query string, the backreference to the IDE.

This way a single webrepl module would serve different tools. Not sure if it will work though

Answer 12 · 2024-06-27T12:51:37.000Z

WebREPL python module could handle an optional url query string

I would be cautious to just echo back an arbitrary url, but maybe a selection of "known-good" tools or something similar to use the "official" client as jumping pad to other clients. I would probably also want to defer that handling to the webrepl-web-client if possible.

If you have (a matching) mpy-cross running you could also offer to upload mpy files instead and/or serve mpy files via mount instead of py files. They are quite a bit smaller so you get a speedup for free. This is an extension I was thinking about for mpremote mount and I'm using that locally.

Answer 13 · 2024-06-28T11:20:51.000Z

Please check out the new way to connect to your board remotely:
https://github.com/vshymanskyy/ViperIDE/blob/main/docs/Web-REPL-Relay.md

Answer 14 · 2024-07-02T21:04:23.000Z

@felixdoerre ViperIDE now automatically runs mpy-cross V6 (directly in browser using WebAssembly) on every file save.
This is not used anywhere, it just results in a warning if compilation fails.