Keep updating while drawing
Closed this issue · 9 comments
It appears that the screen doesn't update as long as the pen is on or near the surface of the tablet, with the result that I if I'm using a drawing app (e.g., Xournal++), I can write for tens of seconds before any of it shows up on the screen. The normal ~1s latency isn't bad, but 20 seconds starts to be a problem!
I'm not sure if this is an issue with rmvncclient or with the VNC server on my computer choosing not to send an update, but the former seems more likely since it doesn't depend on actually drawing anything, only that the pen is above the screen.
Edit: A couple more experiments: if I scribble with my mouse, nothing gets updated on the tablet until I stop moving or release the mouse button, which suggests it's on the host side. However, if I use my computer as the VNC client (mirroring one part of my screen to the other), then it seems to update just fine, which suggests the problem is on the reMarkable side...
PS - Thanks for this fantastic tool! I have a feeling it will become a regular part of my workflow as I teach online this fall.
I’m glad that the tool is useful to you!
So, there is some debouncing logic built into rmvncclient, which is almost certainly at the root of the problem you describe. The current behavior is to wait until it’s been 150 ms since the last update was received from the server before sending the update to the reMarkable screen (the logic is here).
This kind of limit is necessary because screen updates from the server are not sent in one single message per update, but rather broken up into smaller successive messages containing sub-squares of the screen. If we were to send those updates directly to the reMarkable screen, it would create even more delay, because it freezes the process for ~400 ms after each sub-square update.
The compromise I have in mind would be to replace the debouncing logic with some throttling logic, i.e. to send updates every n ms to the reMarkable screen. But ideally we would need to make sure that we’ve received a full screen update, otherwise we would risk being stuck with a partial screen update for a half-second.
Ah, that makes perfect sense. And making the "right choice" between drawing immediately or waiting for more pixels requires knowing the future... I wonder if there are patterns in the tiles being sent that would give a clue. For example, I assume that for a large screen update the tiles are sent in some kind of sequential order, while updates while writing with a pen follow the pen stroke. But this probably depends on quirks of the VNC server and maybe other things.
I'll play around with it a little and see if there's a tweak that would work better for my use case.
because it freezes the process for ~400 ms after each sub-square update.
Pardon my ignorance here... I thought that updating small parts of the screen was relatively fast, and that only full-screen updates were this slow?
I did a couple quick experiments with the update_delay parameter. I found that 50ms is fast enough that it updates between words rather than between lines, and is pretty usable. 20ms was a little too fast; updates happened quickly and I could see what I was writing, but the constant screen refreshing behind the pen became distracting.
The next thing I'd like to try is "update the screen if it's been X ms since the last VNC tile OR it's been Y ms since the last screen update." I'll play around with this a bit, but it probably won't be for a few weeks.
Another thought... I switched the update mode to "direct" instead of "gc16". This eliminates the flicker, at the cost of only working for black/white. Then I turned update_delay down to 5ms and the tablet response time was quite good.
Obviously this is terrible for most use cases, but for mine (using Xournal++ to draw on slides while lecturing or writing on an online collaborative whiteboard) it's actually pretty nice. I wonder if there's a clever way to switch modes on the fly and get the best of both... More things for me to try later!
Thank you for your work! Experiments are definitely needed to find the right balance between latency and quality/refreshing artifacts. Documentation is rather scarce on E-Ink devices, I gathered what I could find into this header, but I suspect there’s more to uncover.
because it freezes the process for ~400 ms after each sub-square update.
Pardon my ignorance here... I thought that updating small parts of the screen was relatively fast, and that only full-screen updates were this slow?
I’m far from being an expert on the matter too. In my experience, the latency is not a function of the updated area but rather, as you found out, of the selected update mode. To be fair, I need to correct myself: updates to the screen are asynchronous, so they do not block the main process, however, they do block the E-Ink controller, so the net result is the same for the present use-case.
The next thing I'd like to try is "update the screen if it's been X ms since the last VNC tile OR it's been Y ms since the last screen update." I'll play around with this a bit, but it probably won't be for a few weeks.
This would be a great improvement indeed. I’m also going to think about it.
Another thought... I switched the update mode to "direct" instead of "gc16". This eliminates the flicker, at the cost of only working for black/white. Then I turned
update_delaydown to 5ms and the tablet response time was quite good.
As I understand it, rendering gray tones is what is the slowest, because the controller needs to first turn the cell black, then white, then black again for a tiny bit of time proportional to the desired lightness of the tone (this is what causes the flashing effect with mode GC16). Whereas for full white or full black, the controller can save time by only turning cells black or white (which is why update modes DU and A2 are much faster and do not cause flashes). This is described in section 3.2.1, page 8 of this document (Google Translate works reasonably well).
Obviously this is terrible for most use cases, but for mine (using Xournal++ to draw on slides while lecturing or writing on an online collaborative whiteboard) it's actually pretty nice. I wonder if there's a clever way to switch modes on the fly and get the best of both... More things for me to try later!
Reading the i.MX EPDC controller specification, I found that there’s actually an automatic mode selection built into the controller, but I wonder how effective it is (see MXCFB_SET_WAVEFORM_MODES and MXCFB_SET_AUTO_UPDATE_MODE on pages 226–227).
In any case, we could always add a flag to rmvncclient to allow users to choose the mode that best fits their use case.
Hello,
I ran into the same issue, but I have patched it to solve it in a different way: when the pen is on the screen, ie. we are drawing, then we update right away. I have also modified the code to update every 200ms instead of waiting for the server to send updates for 150ms (which can never happen if the image is very dynamic). In practice it does work well (better than the current version anyway), so I'm wondering what the issues were in the first place?
Hey @programminggamesemantics! Thanks for chiming in.
When the pen is on the screen, ie. we are drawing, then we update right away.
Sounds like an interesting compromise.
I have also modified the code to update every 200ms instead of waiting for the server to send updates for 150ms (which can never happen if the image is very dynamic)
I believe this is what @stevenbell and I were discussing above: “update the screen if it's been X ms since the last VNC tile OR it's been Y ms since the last screen update.” The only issue was that we didn’t get around to implementing it yet. :)
If you want to submit your patches as a PR, I’d be glad to review them.
Great thanks for your answer. I have something ready, i'll push it out. (The previous comment was made from a temporary account i made for a project that i forgot to sign out from)